Quote:
Originally Posted by acidzebra
a) those must have been really big stories, or you must have very little bandwidth, or you are mistaken in the source of the drain.
b) again, robots.txt, robots.txt, robots,txt.
c) search engines typically provide one or two sentence excerpts, unless you are referring to the google cache, and I don't know anyone who prefers to read the cache while the original site is still alive.
d) somehow I don't think you complain about the people that this search engine sends your way.
Search engines spider publicly-available content on the web, news at 11.
|
This happened back in 99 or 00 and my site wasn't the only one hit. Basically, one of the archives with over 10k stories was also hit and for them it was a much bigger deal. The way this happened was you'd search for the author or story and it didn't direct you to the Archive Index. It just opened the link right then in there from their cache (but the images were still linked back to the my site which still used bandwidth, and also the way it would load everything was different from the archive structure so instead of just pulling the 1 story someone may want it pulled 100 at the same time which ate bandwidth) Basically, for me it was more annoying but for another site with 10k stories it was a much bigger deal. Also 8 years ago bandwidth wasn't as cheap as it is now.