MobileRead Forums - View Single Post

tungsten · 02-04-2009, 09:18 AM

I'm trying to download (just once) a page that doesn't like to be downloaded.
Therefore I get loads of "too many accesses per second" errors.

How can I prevent this?
I have two suggestions:
(1) Slow down the download so something like 1 page per second. How would you do this?
(2) I can split it up into maybe 50 threads. In that case, I would have to set up a "master page" which links all these in one file. How can you do that?

Sorry to post this in an archive-thread, but where should I post this?

02-04-2009, 09:18 AM	#1
tungsten Junior Member Posts: 3 Karma: 10 Join Date: Feb 2009 Device: none	How to slow down the download/spidering? I'm trying to download (just once) a page that doesn't like to be downloaded. Therefore I get loads of "too many accesses per second" errors. How can I prevent this? I have two suggestions: (1) Slow down the download so something like 1 page per second. How would you do this? (2) I can split it up into maybe 50 threads. In that case, I would have to set up a "master page" which links all these in one file. How can you do that? Sorry to post this in an archive-thread, but where should I post this?