View Single Post
Old 03-06-2013, 09:26 PM   #391
knc1
Ex-Helpdesk Addict
knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.
 
knc1's Avatar
 
Posts: 12,221
Karma: 12451693
Join Date: Feb 2012
Location: Central Texas
Device: No PW2, KV, KOA
Quote:
Originally Posted by ixtab View Post
Hmmm... just found this while looking through the ixtab.tk logs:


72.21.217.98 - - [06/Mar/2013:16:01:43 -0600] "GET /robots.txt HTTP/1.1" 200 52 "-" "Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)"


Now... Should I be proud, amused, or worried?

Just deny that crawler in that same robots.txt file.
OR
.htaccess them to death
re-direct them to google's db, let them crawl something time consuming.

Last edited by knc1; 03-06-2013 at 09:28 PM.
knc1 is offline   Reply With Quote