View Single Post
Old 03-06-2013, 08:26 PM   #391
knc1
\_(ツ)_/ <- Clueless
knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.
 
knc1's Avatar
 
Posts: 11,137
Karma: 11512569
Join Date: Feb 2012
Location: Central Texas
Device: Nothing KV or newer, not interested.
Quote:
Originally Posted by ixtab View Post
Hmmm... just found this while looking through the ixtab.tk logs:


72.21.217.98 - - [06/Mar/2013:16:01:43 -0600] "GET /robots.txt HTTP/1.1" 200 52 "-" "Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)"


Now... Should I be proud, amused, or worried?

Just deny that crawler in that same robots.txt file.
OR
.htaccess them to death
re-direct them to google's db, let them crawl something time consuming.

Last edited by knc1; 03-06-2013 at 08:28 PM.
knc1 is online now   Reply With Quote