Metadata download create limit

[yamusing]

Is there a way to create a metadata download limit when fetching metadata from the website?
I happen to have a lot of books to fetch metadata for but happen to get IP banned from the website for sending too many requests.
I would like to limit the metadata download to be limited to 1 request every 5 seconds for example.

[theducks]

Originally Posted by yamusing

Is there a way to create a metadata download limit when fetching metadata from the website?
I happen to have a lot of books to fetch metadata for but happen to get IP banned from the website for sending too many requests.
I would like to limit the metadata download to be limited to 1 request every 5 seconds for example.

It is builtin (I believe the default is 10 seconds, but I can't find the Tweak. I did find the email one )

[kovidgoyal]

It already does that.

[yamusing]

Is there a way to set the limit? As it seems to fetch too fast and I'll try increasing it if possible

[kovidgoyal]

No, you'd have to run from source and personally, I wouldn't bother. The services keep tuning the parameters of their bot detection. Just pause once you start getting the errors and resume in a few hours.

[theducks]

Just queue as a bunch of small batches (~10). That also has the advantage if you crash of only losing that (active) batch of data. A crash loses all un posted data

[yamusing]

Originally Posted by kovidgoyal

No, you'd have to run from source and personally, I wouldn't bother. The services keep tuning the parameters of their bot detection. Just pause once you start getting the errors and resume in a few hours.

for the website there isn't a bot detection for the website as it clearly states the Load limiting for API requests "25 entries per request, 4-5 sequential requests usually okay before having to wait for ~5 seconds". When I try to download metadata at once on Calibre it clearly doesn't have a time limit between requests as I can see it finished downloading metadata for 4 books in 3 seconds in the job interface so it send request one at a time without any delay.

Is there really no way to implment it on my side for creating a download metadata limit per book?

Or if possible add in time.sleep(10) in the metadata plugin would this work?

[yamusing]

Originally Posted by theducks

Just queue as a bunch of small batches (~10). That also has the advantage if you crash of only losing that (active) batch of data. A crash loses all un posted data

I've got over 1000 of books to get metadata from so this would take a lot of time

[theducks]

Originally Posted by yamusing

I've got over 1000 of books to get metadata from so this would take a lot of time

And spotting and FIXING erroneous metadata doesn't?

[kovidgoyal]

Originally Posted by yamusing

for the website there isn't a bot detection for the website as it clearly states the Load limiting for API requests "25 entries per request, 4-5 sequential requests usually okay before having to wait for ~5 seconds". When I try to download metadata at once on Calibre it clearly doesn't have a time limit between requests as I can see it finished downloading metadata for 4 books in 3 seconds in the job interface so it send request one at a time without any delay.

Is there really no way to implment it on my side for creating a download metadata limit per book?

Or if possible add in time.sleep(10) in the metadata plugin would this work?

What website is this? None of the builtin plugins use an API. And yes if you are using a specific website then you will need to implement policies specific to it in the plugin. No time.sleep() wont work since downloads are threaded. You will need to use the rate_limit() decorator from sources/search_engines.py this is how the builtin plugins implement rate limiting.

[yamusing]

Originally Posted by kovidgoyal

What website is this? None of the builtin plugins use an API. And yes if you are using a specific website then you will need to implement policies specific to it in the plugin. No time.sleep() wont work since downloads are threaded. You will need to use the rate_limit() decorator from sources/search_engines.py this is how the builtin plugins implement rate limiting.

Would time.sleep(20) work if I add multiple seperate jobs to the metadata download queue? It would add a 20 second delay to each start of a metadata download batch right?

[yamusing]

It seems to work well when putting in the time.sleep(60) for the metadata plugin and queuing batches of 8 books each for metadata download. Its a bit less automated though if there is a better way to go about this? as I have to keep selecting 8 books and download metadata

[kovidgoyal]

I told you how you can implement site specific rate limiting in the plugin. Not sure what else you are asking.

[yamusing]

Originally Posted by kovidgoyal

I told you how you can implement site specific rate limiting in the plugin. Not sure what else you are asking.

Sorry I'm not sure how to use the rate_limit(), do I only have to implment a number in the brackets and it'll work fine?

[kovidgoyal]

Look inside the same file for examples of its use. You basically use it in a with statement that contains any calls to the remote service.