Quote:
|
Originally Posted by hacker
You mean "forge", not set. Many content providers (myself included) are beginning to reject forged referers, if the host and the referer don't match properly. Easy to do, and I've been showing more and more content providers how to do this to help save their bandwidth and continue to service their users.
|
The reference explains that you should use the Referer property only as a last resort.
I've mailed with webmasters of several big site (including CNET) about the link rewriting capability and none of them had any objections. On the contrary, they understood that link rewriting actually helps to reduce the bandwidth usage. Printable versions usually have no navigation or banner images and contain the entire article. Many NYT articles, for instance, are split across multiple pages in the "normal" version, requiring multiple requests to obtain them in their entirety.
Quote:
|
Originally Posted by hacker
One of the main problems with feeds and feed parsers, is that they don't properly adhere to the standards (again with the standards) for caching, and just continue to pound the server for the feed over and over and over, even when it shouldn't and even when content hasn't changed. This is a larger (and growing) problem.
Its the same with these spiders run on the client side, and its the primary reason why content providers block and ban them.
|
Sunrise goes to great lengths to reduce bandwidth usage. It has a download cache that works with ETag, Last-Modified and Expires headers, so it can perform conditional IMS/INM requests. There's also the option to end an update prematurely if the source URL's content hasn't changed.