Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 01-17-2011, 10:05 AM   #1
bcollier
Member
bcollier began at the beginning.
 
bcollier's Avatar
 
Posts: 22
Karma: 10
Join Date: Jan 2011
Device: Kindle DX
Debugging Download Problems with the New York Times

Very often I notice several articles that are listed on the New York Times "Today's Paper" section that do not make it into my Calibre download. Yesterday, for example, I noticed that the article "Looking Behind the Mug-Shot Grin of an Accused Killer" from the cover section was missing. Today I had a few minutes to check out what happened. The log says "Run -vv to see the reason", how exactly would I do that? I've seen this error in several places, I've attached a recipe to download yesterday's paper (based on the updated NYT recipe) and I've attached the entire log. The relevant section is also pasted below for conveniance.

Could not fetch link http://www.nytimes.com/2011/01/16/us...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 457, in process_links
File "site-packages\calibre\web\feeds\news.py", line 707, in _postprocess_html
File "c:\users\ben\appdata\local\temp\calibre_0.7.40_tm p_hyw7gs\calibre_0.7.40_udep_r_recipes\recipe0.py" , line 635, in postprocess_html
IndexError: list index out of range

http://www.nytimes.com/2011/01/16/us...pagewanted=all saved to c:\users\ben\appdata\local\temp\calibre_0.7.40_tmp _hyw7gs\calibre_0.7.40_xplpva_plumber\feed_0\artic le_2\16loughner.xhtml
Downloaded article: Tourists Mimic Polar Pioneers, Except With Planes and Blogs from http://www.nytimes.com/2011/01/16/wo...pagewanted=all
Failed to download article: Looking Behind the Mug-Shot Grin of an Accused Killer from http://www.nytimes.com/2011/01/16/us...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 846, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 842, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason



Failed to download the following articles:
Looking Behind the Mug-Shot Grin of an Accused Killer from The Front Page
http://www.nytimes.com/2011/01/16/us...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 846, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 842, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason

Parsing all content...
Parsing feed_0/index.html ...
Forcing feed_0/index.html into XHTML namespace
Parsing index.html ...
Forcing index.html into XHTML namespace
Parsing feed_0/article_0/index.html ...
Forcing feed_0/article_0/index.html into XHTML namespace
Parsing feed_0/article_1/index.html ...
Forcing feed_0/article_1/index.html into XHTML namespace
Parsing feed_0/article_4/index.html ...
Forcing feed_0/article_4/index.html into XHTML namespace
Parsing feed_0/article_3/index.html ...
Initial parse failed:
Traceback (most recent call last):
File "site-packages\calibre\ebooks\oeb\base.py", line 857, in first_pass
File "lxml.etree.pyx", line 2532, in lxml.etree.fromstring (src/lxml/lxml.etree.c:48634)
File "parser.pxi", line 1545, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:72245)
File "parser.pxi", line 1417, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:71041)
File "parser.pxi", line 898, in lxml.etree._BaseParser._parseUnicodeDoc (src/lxml/lxml.etree.c:67581)
File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:64257)
File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:65178)
File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64521)
XMLSyntaxError: xmlParseEntityRef: no name, line 114, column 92
Attached Files
File Type: zip NYT Back Issue Download.zip (7.6 KB, 53 views)
File Type: txt NYT Sunday Paper Missing Article Debug.txt (14.1 KB, 69 views)
bcollier is offline   Reply With Quote
Old 01-17-2011, 11:54 AM   #2
Alexis
Member
Alexis began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jan 2011
Device: Kindle
Quote:
Originally Posted by bcollier View Post
The log says "Run -vv to see the reason", how exactly would I do that?
This refers to an option to the ebook-convert command-line program, which takes your recipe as an argument. The -vv flag makes it verbose.

For example, you could use it to test your nytimes recipe
Code:
ebook-convert $CALIBRE_PATH/resources/recipes/nytimes_sub.recipe nyt.mobi -vv --debug-pipeline nyt_debug/ --test --output-profile=kindle --username yourusername --password yourpassowrd
The other really useful option is the --debug-pipeline one.

More details here:
http://calibre-ebook.com/user_manual...k-convert.html
and here:
http://calibre-ebook.com/user_manual...onvert-11.html

Last edited by Alexis; 01-17-2011 at 12:07 PM.
Alexis is offline   Reply With Quote
Old 01-17-2011, 12:07 PM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,374
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
That's the log from a GUI fetch which already has the -vv option. Juts scroll up and you will see the actual error message.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New York Times Error in .6.23 geneaber Calibre 0 11-14-2009 12:27 PM
New York Times recipe madrone26 Calibre 4 04-02-2009 01:13 PM
New York times about Kindle 2 Kris777 News 12 02-18-2009 08:51 AM
New York Times on 505 Hamza Sony Reader 21 03-03-2008 12:55 PM
New Reader Ad in New York Times TadW Sony Reader 7 07-28-2007 01:11 PM


All times are GMT -4. The time now is 07:00 PM.


MobileRead.com is a privately owned, operated and funded community.