Originally Posted by ixtab
Hmmm... just found this while looking through the ixtab.tk logs:
22.214.171.124 - - [06/Mar/2013:16:01:43 -0600] "GET /robots.txt HTTP/1.1" 200 52 "-" "Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)"
Should I be proud, amused, or worried?
Just deny that crawler in that same robots.txt file.
.htaccess them to death
re-direct them to google's db, let them crawl something time consuming.