Debugging Download Problems with the New York Times

[bcollier]

Very often I notice several articles that are listed on the New York Times "Today's Paper" section that do not make it into my Calibre download. Yesterday, for example, I noticed that the article "Looking Behind the Mug-Shot Grin of an Accused Killer" from the cover section was missing. Today I had a few minutes to check out what happened. The log says "Run -vv to see the reason", how exactly would I do that? I've seen this error in several places, I've attached a recipe to download yesterday's paper (based on the updated NYT recipe) and I've attached the entire log. The relevant section is also pasted below for conveniance.

Could not fetch link http://www.nytimes.com/2011/01/16/us...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 457, in process_links
File "site-packages\calibre\web\feeds\news.py", line 707, in _postprocess_html
File "c:\users\ben\appdata\local\temp\calibre_0.7.40_tm p_hyw7gs\calibre_0.7.40_udep_r_recipes\recipe0.py" , line 635, in postprocess_html
IndexError: list index out of range

http://www.nytimes.com/2011/01/16/us...pagewanted=all saved to c:\users\ben\appdata\local\temp\calibre_0.7.40_tmp _hyw7gs\calibre_0.7.40_xplpva_plumber\feed_0\artic le_2\16loughner.xhtml
Downloaded article: Tourists Mimic Polar Pioneers, Except With Planes and Blogs from http://www.nytimes.com/2011/01/16/wo...pagewanted=all
Failed to download article: Looking Behind the Mug-Shot Grin of an Accused Killer from http://www.nytimes.com/2011/01/16/us...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 846, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 842, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason



Failed to download the following articles:
Looking Behind the Mug-Shot Grin of an Accused Killer from The Front Page
http://www.nytimes.com/2011/01/16/us...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 846, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 842, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason

Parsing all content...
Parsing feed_0/index.html ...
Forcing feed_0/index.html into XHTML namespace
Parsing index.html ...
Forcing index.html into XHTML namespace
Parsing feed_0/article_0/index.html ...
Forcing feed_0/article_0/index.html into XHTML namespace
Parsing feed_0/article_1/index.html ...
Forcing feed_0/article_1/index.html into XHTML namespace
Parsing feed_0/article_4/index.html ...
Forcing feed_0/article_4/index.html into XHTML namespace
Parsing feed_0/article_3/index.html ...
Initial parse failed:
Traceback (most recent call last):
File "site-packages\calibre\ebooks\oeb\base.py", line 857, in first_pass
File "lxml.etree.pyx", line 2532, in lxml.etree.fromstring (src/lxml/lxml.etree.c:48634)
File "parser.pxi", line 1545, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:72245)
File "parser.pxi", line 1417, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:71041)
File "parser.pxi", line 898, in lxml.etree._BaseParser._parseUnicodeDoc (src/lxml/lxml.etree.c:67581)
File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:64257)
File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:65178)
File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64521)
XMLSyntaxError: xmlParseEntityRef: no name, line 114, column 92

[Alexis]

Originally Posted by bcollier

The log says "Run -vv to see the reason", how exactly would I do that?

This refers to an option to the ebook-convert command-line program, which takes your recipe as an argument. The -vv flag makes it verbose.

For example, you could use it to test your nytimes recipe

ebook-convert $CALIBRE_PATH/resources/recipes/nytimes_sub.recipe nyt.mobi -vv --debug-pipeline nyt_debug/ --test --output-profile=kindle --username yourusername --password yourpassowrd

The other really useful option is the --debug-pipeline one.

More details here:
http://calibre-ebook.com/user_manual...k-convert.html
and here:
http://calibre-ebook.com/user_manual...onvert-11.html

[kovidgoyal]

That's the log from a GUI fetch which already has the -vv option. Juts scroll up and you will see the actual error message.