Quote:
Originally Posted by leistnerm
It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.
|
That's a very good point.
For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.
But I don't remember ever thinking about it on our client side. It's certainly worth a look, thanks for the suggestion.
BTW, ff.net in particular is slower than it could be--there's a hard coded 0.5sec sleep between hits (plus the slow_down_sleep_time, if set). We have first hand evidence from one of the other developers that ff.net will restrict your IP if you hit them "too fast". See
http://www.fanfiction.net/tos/ section 4E.
Jim