Quote:
Originally Posted by ixtab
Hmmm... just found this while looking through the ixtab.tk logs:
72.21.217.98 - - [06/Mar/2013:16:01:43 -0600] "GET /robots.txt HTTP/1.1" 200 52 "-" "Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)"
Now... Should I be proud, amused, or worried?
|
Just deny that crawler in that same robots.txt file.
OR
.htaccess them to death
re-direct them to google's db, let them crawl something time consuming.