New York Times and Wall Street Journal only work sporadically

[mkgtu]

I have these two set to be downloaded and emailed to Kindle every morning. Sometimes they work; sometimes they just hang forever. I could post logs but there are almost never any actual error messages. The downloads just get stuck on something mid-stream and the wheel keeps spinning for hours and hours.
Generally if I try them manually later in the day (after I've finally noticed the failure), they work ok.
This happens with both the NYTimes daily paper and the Web recipes. I've occasionally set both to download automatically in hopes that at least one will complete.
With regard to the Wall Street Journal, if it doesn't get stuck, I often experience what many others have seen: incomplete paper. Most days the WSJ has 7 sections. But often all I get are 4, or 5, or 6. What's missing varies. If I download manually to try to get the whole paper, I may still get fewer than I should, but the missing sections differ each time. I've often had to retry the download 5 or 6 times in a row before I finally get the whole thing.
Once in a while I'll get a real failure, with a message, like "no articles found", but generally there's no message; the download just hangs. And when it hangs, it usually seems to be stuck at an image. That's the case with both the WSJ and the NYT.

Sent from my SM-G975U using Tapatalk

[kovidgoyal]

presumably you have network issues somewhere along the path from your computer to the servers. And note that downloads all have timeouts so I dont see how they could possibly be getting stuck. If they are then, most likely something is interfering with calibre, typically a security/antivirus program.

[mkgtu]

I use Bitddfender Total Security on my PC. I'll check into it and create exceptions for calibre to see if that helps.
But it's strange that this only happens with the NYTimes (paper and web versions) and especially the Wall Street Journal.
I also schedule regular auto-downloads for other periodicals, and those others always complete without problems:
- The Economist
- The Spectator
- the Atlantic.com
- Science Daily
- New York Review of Books
- London Review of Books
- The New Yorker
- New York Magazine

Sent from my SM-G975U using Tapatalk

[mkgtu]

Today both NYT and WSJ downloaded without issue. But as to network interference I did check Bitdefender settings and found that calibre.exe and calibre-parallel.exe already had all access permissions in the firewall settings. I then added rules to allow access for calibre-complete.exe, calibre-smtp.exe, and calibre-server.exe. Not sure all of those are relevant but is there anything else I should add.

Since all other scheduled recipes work fine, I suspect the issue may have more to do with traffic jams on the publisher websites. I had always scheduled both WSJ and NYT to downloaded at 6:00am (pacific). I recently moved WSJ to 8:30am (pacific) thinking there might be less traffic. Not sure that's made a significant difference.

Since the point at which the downloads stall always seems to be at an image, I've also just tweaked both NYT and WSJ recipes to exclude images by adding "dict(name='img')" to the Remove Tags section of each. I'll watch them for a week or so to see if that helps. (I subscribe to both and can always see the photos in the phone apps or on the web.)

[slm]

Originally Posted by kovidgoyal

presumably you have network issues somewhere along the path from your computer to the servers. And note that downloads all have timeouts so I dont see how they could possibly be getting stuck. If they are then, most likely something is interfering with calibre, typically a security/antivirus program.

I have the same issue with the New York Times--some days it isn't there in the morning but it shows up as a running jobs with times of several hours. Once it was 5 hours (when I looked in the afternnoon). Whatever "timeout" is supposed to apply, clearly doesn't work in some contexts.

As with the OP, it always downloads successfully if I kill the running job and force a new download, usually in a few minutes.

[kovidgoyal]

Well all image downloads happen in the fetch_url() function in simple.py and you can check for yourself that a timeout is applied there. If there is some kind of timeout not being respected its somewhere lower down in the stack than calibre code.

[NSILMike]

I have the same issue with the NY Times. When it starts on the schedule, it runs endlessly. I stop the job and manually start it and it runs just fine, completing in less than a minute.

[NSILMike]

Originally Posted by mkgtu

Today both NYT and WSJ downloaded without issue. But as to network interference I did check Bitdefender settings and found that calibre.exe and calibre-parallel.exe already had all access permissions in the firewall settings. I then added rules to allow access for calibre-complete.exe, calibre-smtp.exe, and calibre-server.exe. Not sure all of those are relevant but is there anything else I should add.

Since all other scheduled recipes work fine, I suspect the issue may have more to do with traffic jams on the publisher websites. I had always scheduled both WSJ and NYT to downloaded at 6:00am (pacific). I recently moved WSJ to 8:30am (pacific) thinking there might be less traffic. Not sure that's made a significant difference.

Since the point at which the downloads stall always seems to be at an image, I've also just tweaked both NYT and WSJ recipes to exclude images by adding "dict(name='img')" to the Remove Tags section of each. I'll watch them for a week or so to see if that helps. (I subscribe to both and can always see the photos in the phone apps or on the web.)

I doubt it is an image issue, since when I run the NYT automatically it hangs, and when I stop that process and run it manually it downloads in a few seconds. And the time of day doesn't seem to make any difference either.

[rasteps]

I've been having the same problem for months.

I've attached a screenshot where several jobs just got stuck, 2 of which were stuck for more than 2 hours without timing out. It took several kills and restarts to finally get them downloaded.

I thought the simple solution would be to request a restart option for the Jobs dialog box, but it looks like there is a deeper problem of hanging without timing out going on.

[NSILMike]

So nobody has a solution to this?

[rasteps]

The only workaround now is to just stop and restart the job. It would be easier if you could restart jobs from the Jobs dialog box, but as I said, it seems that there might be a deeper problem of some recipes not timing out which underlies this situation.

[mkgtu]

Originally Posted by NSILMike

I doubt it is an image issue, since when I run the NYT automatically it hangs, and when I stop that process and run it manually it downloads in a few seconds. And the time of day doesn't seem to make any difference either.

I stated on Jun 5 that I would try downloading for a while without images.
Since the point at which the downloads stall always seems to be at an image, I've also just tweaked both NYT and WSJ recipes to exclude images by adding "dict(name='img')" to the Remove Tags section of each. I'll watch them for a week or so to see if that helps. (I subscribe to both and can always see the photos in the phone apps or on the web.)

I have been doing that now for almost three weeks and I find that by and large the recipes for New York Times and The Wall Street Journal download reliably each day. There are still occasional instances where the recipe gets stuck and keeps spinning forever, but not like before. The downloads are also now quite small, only about 1MB instead of 7-20; so most of the full downloads must be images! (For example, today, without images, the WSJ was .7mb, NYT daily paper .8mb, and the NYT web 1.1mb)
In addition to the Wall Street Journal, I am now also downloading both the NY Times recipes (the daily paper and the web edition). I figure if one fails, hopefully I'll get the other. So far, I usually get both without problems.

So far so good. I can live without all the images on a Kindle. I do have paid subscriptions to both and can see all the images I want in my phone apps or on the web.

There is still an occasional problem with the Wall Street Journal not downloading all the sections of the paper. This is odd. You can download the recipe five times in a row and never get the same number of sections, and even if you get the same number they aren't the same sections! But omitting the images seems to help with this, though it's still not perfect.

[mjfriedman]

I also have this problem with NYT and WSJ (only). It's usually not a big deal as one can simply relaunch the fetch... except when traveling

[mkgtu]

Agreed, you can always run the download again (and again, and again). But the real problem is when you're away from your computer or travelling and can't run it again. Not to mention the application is endlessly stuck on old downloads and may not even be able to try the next day's editions. I only allow 3 processes at a time in calibre; my PC is pretty fast, solid state with 80-90mb wifi download speed, 16gb RAM, but it's only Intel i3 (10th gen), dual core, 4 threads; so I can't afford to get all the calibre processes stuck. And, no, Kovid's comment notwithstanding, the timeouts don't seem to be working in these recipes or on these websites. Don't know why.

But creating custom recipes that omit all the images (for the NYT and WSJ) seems to have solved the problem for me. It's quicker, and I think possibly avoids traffic jams on what are likely very large and busy sites.

(Never have this problem on other sites, eg The Economist, The Spectator, Atlantic.com; all of which are much larger downloads).


Sent from my SM-G975U using Tapatalk

[NSILMike]

Very disappointing that so many have this issue and there are no fixes...