View Single Post
Old 03-06-2013, 08:26 PM   #391
knc1
Embedded Cheerleader
knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.
 
knc1's Avatar
 
Posts: 6,382
Karma: 5840126
Join Date: Feb 2012
Device: Too many.
Quote:
Originally Posted by ixtab View Post
Hmmm... just found this while looking through the ixtab.tk logs:


72.21.217.98 - - [06/Mar/2013:16:01:43 -0600] "GET /robots.txt HTTP/1.1" 200 52 "-" "Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)"


Now... Should I be proud, amused, or worried?

Just deny that crawler in that same robots.txt file.
OR
.htaccess them to death
re-direct them to google's db, let them crawl something time consuming.

Last edited by knc1; 03-06-2013 at 08:28 PM.
knc1 is offline   Reply With Quote