Thread: WebCopier for Pocket PC 4.2 Now Available
View Single Post
Old 03-29-2005, 08:54 AM   #3
Bob Russell
Recovering Gadget Addict
Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.Bob Russell ought to be getting tired of karma fortunes by now.
 
Bob Russell's Avatar
 
Posts: 5,381
Karma: 676161
Join Date: May 2004
Location: Pittsburgh, PA
Device: iPad
Quote:
Originally Posted by hacker
Great, another spider to block... WebCopier (375 hits so far in this month's logs on my end) didn't request robots.txt once, it just proceeded to pound the server for as much as it would send, as fast as it could.

Nope, sorry... blocked. Next?
Hacker,
I'm curious, from a web site admin perspective on things, what sorts of spiders are out there and which ones you'd consider good/bad in their approach. For example, with WebCopier you mention that they didn't request robots.txt. What is that, and why should they request it? Thks!
Bob Russell is offline