Does calibre support retry-after http headers ?

SimonMc · 12-15-2021, 05:12 AM

Hi,
A query please. I have been trying to download metadata and frequently get HTTP error 429 'Too Many requests'.

My reading implies that when the 429 is supplied sometimes it gives a 'retry-after' response which indicates when service can resume (I don't know if it's Amazon or Google but they definately limit requests in an automated fashion).

Is there any possibility, that an interval could be configured whenever a 429 is the response to allow for retries that applies to all plugins (on each request) or is this a can of worms ?

Thanks in advance

https://developer.mozilla.org/en-US/...TTP/Status/429

kovidgoyal · 12-15-2021, 05:15 AM

It's a can of worms. That's a bot prevention measure and it will be from google not amazon. amazon uses captchas.

SimonMc · 12-15-2021, 06:19 AM

Thanks for replying. I appreciate it. I just wondered if there is anything that can be done to reduce the impact such as rate limiting or sleeping using celery or something similar.

Example here: https://stackoverflow.com/a/50294531

I apologise in advance if this is an example of a newb teaching you to suck eggs. Just trying to see if the metadata retrieval process can be improved.

SimonMc · 12-15-2021, 06:45 AM

this page details some options for backing off, and using rate limiting, would any of that help do you think ?

https://stackoverflow.com/questions/...equests-python

kovidgoyal · 12-15-2021, 08:05 AM

It's already rate limited. Google simply blocks you once you do too many requsts from an IP regardless of the rate

SimonMc · 12-15-2021, 08:29 AM

If that is the case is is possible to batch them, or limit the number of metadata requests per batch ? If you start getting 429 responses, you may as well just cancel the remainder of the requests ?

Is it possible to show where in the code this is handled, just so that I can familiarise myself with it without wasting your time ?

This is what google says on it. https://developers.google.com/calend..._many_requests

cheers
Simon

kovidgoyal · 12-15-2021, 11:40 AM

It doesnt use an API, it scrapes the website. And there is basically no robust way to do bulk scraping from a single IP address, the server will always win. You just have to limit your queries to no more than a handful a day. And if you want to read the code look in the metadata/sources folder.

12-15-2021, 05:12 AM	#1
SimonMc Member Posts: 12 Karma: 10 Join Date: Mar 2015 Device: Kindle 3	Does calibre support retry-after http headers ? Hi, A query please. I have been trying to download metadata and frequently get HTTP error 429 'Too Many requests'. My reading implies that when the 429 is supplied sometimes it gives a 'retry-after' response which indicates when service can resume (I don't know if it's Amazon or Google but they definately limit requests in an automated fashion). Is there any possibility, that an interval could be configured whenever a 429 is the response to allow for retries that applies to all plugins (on each request) or is this a can of worms ? Thanks in advance https://developer.mozilla.org/en-US/...TTP/Status/429

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Auto Retry if Scheduled download fail!!!	jardenman26	Recipes	3	06-21-2018 05:28 AM
Auto Retry if Scheduled download fails	stevesout	Calibre	1	03-14-2018 11:26 PM
Minor UI improvement - retry button	whismerhill	Calibre	7	05-18-2016 01:01 AM
Retry Job?	truth1ness	Calibre	1	04-01-2015 07:26 PM
Calibre not removing headers in mobi conversion	kim-	Calibre	2	08-04-2009 11:37 AM

12-15-2021, 05:15 AM	#2
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It's a can of worms. That's a bot prevention measure and it will be from google not amazon. amazon uses captchas.

12-15-2021, 06:19 AM	#3
SimonMc Member Posts: 12 Karma: 10 Join Date: Mar 2015 Device: Kindle 3	Thanks for replying. I appreciate it. I just wondered if there is anything that can be done to reduce the impact such as rate limiting or sleeping using celery or something similar. Example here: https://stackoverflow.com/a/50294531 I apologise in advance if this is an example of a newb teaching you to suck eggs. Just trying to see if the metadata retrieval process can be improved.

12-15-2021, 06:45 AM	#4
SimonMc Member Posts: 12 Karma: 10 Join Date: Mar 2015 Device: Kindle 3	this page details some options for backing off, and using rate limiting, would any of that help do you think ? https://stackoverflow.com/questions/...equests-python

12-15-2021, 08:05 AM	#5
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It's already rate limited. Google simply blocks you once you do too many requsts from an IP regardless of the rate

12-15-2021, 08:29 AM	#6
SimonMc Member Posts: 12 Karma: 10 Join Date: Mar 2015 Device: Kindle 3	If that is the case is is possible to batch them, or limit the number of metadata requests per batch ? If you start getting 429 responses, you may as well just cancel the remainder of the requests ? Is it possible to show where in the code this is handled, just so that I can familiarise myself with it without wasting your time ? This is what google says on it. https://developers.google.com/calend..._many_requests cheers Simon

12-15-2021, 11:40 AM	#7
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It doesnt use an API, it scrapes the website. And there is basically no robust way to do bulk scraping from a single IP address, the server will always win. You just have to limit your queries to no more than a handful a day. And if you want to read the code look in the metadata/sources folder.

Advert

Advert