WSJ recipe gets "Too Many Requests" error for some articles

[etmurphy]

The WSJ recipe will download around 70% of the first articles for an issue, but the remaining articles will get only the text "Too Many Requests". Could the explanation be that archive.is rate limits requests, for example allowing only so many requests per minute? (PR 2075 in release 6.29 notes the switch to archive.is: https://github.com/kovidgoyal/calibre/pull/2075)

Assuming that rate limiting is the root cause, is there a way to configure the number of download threads or otherwise slow down Calibre downloads so that it can get all articles for each WSJ issue?

[unkn0wn]

you can always add delay = 1 # seconds but I don't think the problem can be fixed this way. The rate limit may not be IP specific. And also there are many non-calibre users that use archive.is to read wsj.

Do tell if adding delay to the recipe worked! I actually haven't faced "Too Many Requests" issue, maybe its because of me being on another side of the world or different servers here. Try VPN.

[unkn0wn]

Actually someone can fix the wsj login recipe to decrypt the articles as suggested by kovidgoyal and let the free recipe users use the archive.

https://www.mobileread.com/forums/sh...4&postcount=17

[etmurphy]

Originally Posted by unkn0wn

you can always add delay = 1 # seconds but I don't think the problem can be fixed this way. The rate limit may not be IP specific. And also there are many non-calibre users that use archive.is to read wsj.

Do tell if adding delay to the recipe worked! I actually haven't faced "Too Many Requests" issue, maybe its because of me being on another side of the world or different servers here. Try VPN.

The delay works, thank you!

[mjfriedman]

Hi! Could some kind soul explain to a liberal arts major the steps to add the above mentioned delay to the WSJ recipe?

[kovidgoyal]

Its already added to the builtin recipe. So just make sure you are using the builtin recipe not a custom one.

[mjfriedman]

I think I am. It says By Covid Goyal. But only about 3/4 of articles are downloaded. In this Saturday's edition, it conks out about halfway through the Review section. Am I doing something wrong?

[unkn0wn]

if these articles can be accessed from archive.is, it'll show up in the recipe.
adding delay only fixes "Too Many Requests" problem.

[mjfriedman]

The recipe I am using, which I believe is the ‘built-in’ one, downloads the first maybe 75% of articles. The remainder appear as blank, except for a URL…

[mjfriedman]

Should I use the recipe that requires a password (I have one) or the one titled Free?

Thank you.

[unkn0wn]

technically both the recipes are the same.

[mjfriedman]

For the weekend WSJ the first 69 articles appear correctly. The last correctly loaded article is from the Review section. It is a review of a book about Madame de Pompadour. For the next article, all that appears on my kindle is "This article was downloaded by Calibre from [article URL]." Going forward some articles appear but others produce the "downloaded by Calibre" message

Is this the case for others? Am I doing something wrong? Any fix?

Thank you as always to the more knowledgeable members of this community who help our the rest of us!

[unkn0wn]

https://www.mobileread.com/forums/sh...d.php?t=346055

[mjfriedman]

So it seems this is known issue with no fix at present. Grateful though, to get 75-80% of the paper....