NY Times fails

[mkgtu]

Following up on my suggestion to leave both recipes in calibre as built-ins, it occurs to me there may be many parts of the world where issues with the regular NYTimes servers are either not a problem or some people are having success with just tweaking the delay time in the regular recipe.
I've also found that sometimes even with a long delay the downloads stop after a while and you only get a few sections. Maybe some are content to comment out a lot of the sections so they at least get what they really need before the servers start blocking them.
So these people may prefer the direct access if it's working for them.

In my case, especially since I have kept a barely tweaked copy of the original recipe, I may well schedule both. Since the long delay that often works takes a long time, I'd schedule that one an hour earlier than the "wayback" one. Between the two I'd hope to get at least one up to date, useable copy. On days neither works I'd just spend the 99cents to buy that day's Kindle edition from Amazon if I need it.

But "giving up" doesn't seem very helpful. No offense intended. I get your frustration.

Sent from my SM-G975U using Tapatalk

[mkgtu]

Three days in a row now, Sat-Sun-Mon, Nov 19-20-21, the built in recipe using wayback machine has SUCCESSFULLY auto-downloaded the full daily issue of NYTimes.
This is at 6am PST (9am New York Time)

I can only concluded that the Internet Archive version beimg out of date for a lot of the day last Friday must have been an anomaly. Keeping fingers crossed for the future.



Sent from my SM-G975U using Tapatalk

[kovidgoyal]

That will be because of this: https://github.com/kovidgoyal/calibr...9a15c88caa3181

[mkgtu]

Thanks. That explains why it works perfectly now.

I wish I could say I completely understood all that code on github, but being a "bear of little brain" (thanks, Winnie the Pooh!), my best guess from what I saw is that the updated recipe takes the INDEX of Today's Paper from the nytimes website (presumably always up to date), and then using that as the guide, it fetches the actual content from the Internet Archive rather than the nytimes servers, thus still avoiding any "content cops" on the Times website.

Kind of like compiling a "shopping list" from the Times website but then filling your shopping basket from someplace else.

This must mean that although the Internet Archive is pretty quick to accumulate content it is a bit slower at getting it indexed.

Whatever, it seems to work.

Thanks again.

[kovidgoyal]

yes, you are correct, the index is now fetched from the nytimes server but the articles are fetched from wayback. One request to the nytimes server is not enough to trigger their bot protection.

[mkgtu]

That is really quite clever and imaginative.

And thanks for not giving up on the problem.

Sent from my SM-G975U using Tapatalk