Quote:
|
Originally Posted by Laurens
Sunrise allows you to set the "Referer" header for each individual link through scripting. Simply set the Referer to the original URL and rewrite the link to point to the printable version. There's a lengthy explanation on this in the scripting reference.
|
You mean "forge", not set. Many content providers (myself included) are beginning to reject forged referers, if the host and the referer don't match properly. Easy to do, and I've been showing more and more content providers how to do this to help save their bandwidth and continue to service their users.
One of the main problems with feeds and feed parsers, is that they don't properly adhere to the standards (again with the standards) for caching, and just continue to pound the server for the feed over and over and over, even when it shouldn't and even when content hasn't changed. This is a larger (and growing) problem.
Its the same with these spiders run on the client side, and its the primary reason why content providers block and ban them.
Users want the content as fast as possible, and decide to slam the server to get it.
The content proiders want to give their users a responsive browsing experience, but can't if 1,000 separate spiders are slamming into their site, ignoring caching rules, robots.txt, and deep-linking, etc.
This leads to blocking, banning, and other techniques to stop the users from abusing the server's resources.
Fun times, cat and mouse and all.