Missing Articles when Scheduling NYT Download

[bcollier]

I've found many times if I set the NYT subscription recipe to download at a certain time, I am often missing many of the front page articles, but if I am there and hit download now the articles are there.

I tested this with the debug on, the scheduled download for 7am was missing all but 2 front page articles, then I manually hit download at 7:20 and all the articles were there.

It appears as though this problem is related to pop up ads or ads of some kind, but I've never been able to see them in person. The logs are attached, both the log of the run with missing articles, and the log with all the articles, and key section of the log pasted below, any thoughts on how to fix this?

Fetching http://www.nytimes.com/2011/02/14/bu...pagewanted=all
Fetching http://www.nytimes.com/2011/02/14/wo...pagewanted=all
Fetching http://www.nytimes.com/2011/02/14/ny...pagewanted=all
1% Starting download [5 thread(s)]...
Found forwarding link: /2011/02/14/us/14giffords.html?adxnnl=1&pagewanted=all&adxnnlx=12 97684807-9QlfX3WHQH98rsuu9txtHg
Skipping ad to article at 'http://www.nytimes.com/2011/02/14/us/14giffords.html?pagewanted=all'
Found forwarding link: /2011/02/14/world/middleeast/14egypt-tunisia-protests.html?adxnnl=1&pagewanted=all&adxnnlx=1297 684807-9QlfX3WHQH98rsuu9txtHg
Skipping ad to article at 'http://www.nytimes.com/2011/02/14/world/middleeast/14egypt-tunisia-protests.html?pagewanted=all'
Found forwarding link: /2011/02/14/nyregion/14chicken.html?adxnnl=1&pagewanted=all&adxnnlx=129 7684807-9QlfX3WHQH98rsuu9txtHg
Skipping ad to article at 'http://www.nytimes.com/2011/02/14/nyregion/14chicken.html?pagewanted=all'
Found forwarding link: /2011/02/14/business/14retirees.html?adxnnl=1&pagewanted=all&adxnnlx=12 97684807-9QlfX3WHQH98rsuu9txtHg
Skipping ad to article at 'http://www.nytimes.com/2011/02/14/business/14retirees.html?pagewanted=all'
Found forwarding link: /2011/02/14/world/middleeast/14egypt.html?adxnnl=1&pagewanted=all&adxnnlx=12976 84807-9QlfX3WHQH98rsuu9txtHg
Skipping ad to article at 'http://www.nytimes.com/2011/02/14/world/middleeast/14egypt.html?pagewanted=all'
Could not fetch link http://www.nytimes.com/2011/02/14/us...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 428, in process_links
File "site-packages\calibre\web\fetch\simple.py", line 153, in get_soup
File "site-packages\calibre\ebooks\chardet\__init__.py", line 86, in xml_to_unicode
TypeError: 'NoneType' object is not callable

http://www.nytimes.com/2011/02/14/us...pagewanted=all saved to
Downloading
Fetching http://www.nytimes.com/2011/02/14/bu...pagewanted=all
Failed to download article: Word and Lyric, Giffords Labors to Speak Again from http://www.nytimes.com/2011/02/14/us...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 852, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 847, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in this log



1% Article download failed: u'Word and Lyric, Giffords Labors to Speak Again'
Could not fetch link http://www.nytimes.com/2011/02/14/ny...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 428, in process_links
File "site-packages\calibre\web\fetch\simple.py", line 153, in get_soup
File "site-packages\calibre\ebooks\chardet\__init__.py", line 86, in xml_to_unicode
TypeError: 'NoneType' object is not callable

[kovidgoyal]

It may be that the articles are not available at 7:00 AM. Try setting your scheduler for 7:20 instead.

[mkgtu]

I have had the same problem with front page articles, though the time frame is different. (Though not sure what timezone you're in) Always used to be able to get everything at 4:00 am (Pacific), never downloaded anything as late as 7:00. E.g. today got only one front page article at 4:00, but manually downloaded NYT at about 6:30 and got full front page.
Am now loading two copies of the recipe, one to download at 5:00 am, the second at 5:45, just to see what happens.

It's strange. The paid Amazon Kindle version of the paper appears quite early on the West Coast, at the same time it would hit the streets in New York. I don't wake up at 4:00 am, but it has certainly been there at 5:00 am (8:00 Eastern time). So those articles must be available someplace. And they were always available through calibre scheduled downdloads as early as 3:40 am (Pacific) a few months ago!

At one time I had thought that the problem had something to do with the "duplicates" variable. IOW that if any "International" article were on the Front Page of the paper, it would only appear in the "International" section, but not be repeated on the Front Page. But alas! Not so. The actual front page articles seem to be missing from the entire paper!

[bcollier]

Interesting, I don't think I have a simple answer to the problem, I've tried a bunch of scenarios as well. Scheduling for 7:30am Eastern Time has worked well this week with all the front page articles available. However, on Sunday I was up at 5:30 and checked the website and all the articles were up and available, so I'm not sure why they wouldn't download the rest of the week. Strange.

[dlgraves]

Hello,

Just wondering if this was ever resolved. I've been having the same problem: When i dl manually through the GUI, all the NYT articles are there, but when I run a scheduled download (via command line script) the Front Page section usually only has one or two. I get a slimmed down version to save space, but the same missing articles problem happens whether I use "exclude" or "include" to customize the script.

Anyway -- calibre is amazing, the NYT recipe is great, I wish I could just get this one kink ironed out. Any help appreciated!!

thanks
DLG

[bcollier]

I was able to resolve this problem in the recipe before, but sounds like it is a problem again. I don't have the paid NYT online subscription so I can't troubleshoot the problem.