View Single Post
Old 03-06-2013, 08:26 PM   #391
knc1
\_(ツ)_/ <- Clueless
knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.
 
knc1's Avatar
 
Posts: 11,867
Karma: 12397563
Join Date: Feb 2012
Location: Central Texas
Device: No PW2, KV, KOA
Quote:
Originally Posted by ixtab View Post
Hmmm... just found this while looking through the ixtab.tk logs:


72.21.217.98 - - [06/Mar/2013:16:01:43 -0600] "GET /robots.txt HTTP/1.1" 200 52 "-" "Mozilla/5.0 (compatible; AMZNKAssocBot/4.0 +http://affiliate-program.amazon.com)"


Now... Should I be proud, amused, or worried?

Just deny that crawler in that same robots.txt file.
OR
.htaccess them to death
re-direct them to google's db, let them crawl something time consuming.

Last edited by knc1; 03-06-2013 at 08:28 PM.
knc1 is offline   Reply With Quote