Calibre + NYT download = inconsistent? - Page 2

jsl21 · 05-31-2010, 03:16 PM

GRiker,

Thanks for looking at this so quickly! Below is the part of the log where the failed loading of the articles is. It is the same first 4 or 5 articles that used to say "advertisement." And just like before, if I reload the same day, i get all of the articles.

Failed to download article: White House Tries to Regroup as Criticism Mounts Over Leak from http://www.nytimes.com/2010/05/31/us...allDownloading

Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason

Fetching http://www.nytimes.com/2010/05/31/wo...pagewanted=all
Processing images...
Recursion limit reached. Skipping links in http://www.nytimes.com/2010/05/31/wo...pagewanted=all
Could not fetch link http://www.nytimes.com/2010/05/31/wo...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 452, in process_links
File "site-packages\calibre\web\feeds\news.py", line 615, in _postprocess_html
AttributeError: 'NoneType' object has no attribute 'insert'

http://www.nytimes.com/2010/05/31/wo...pagewanted=all saved to c:\docume~1\allen\locals~1\temp\calibre_0.6.55_hx1 _jf_plumber\feed_0\article_1\31koreanavy.xhtml
Processing images...
Processing images...
Downloading
Fetching Recursion limit reached. Skipping links inhttp://www.nytimes.com/2010/05/31/world/asia/31flogging.html?pagewanted=all
http://www.nytimes.com/2010/05/31/bu...pagewanted=all
Recursion limit reached. Skipping links in http://www.nytimes.com/2010/05/31/bu...pagewanted=all
Failed to download article: U.S. to Aid South Korea With Naval Defense Plan from http://www.nytimes.com/2010/05/31/wo...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason

Could not fetch link http://www.nytimes.com/2010/05/31/bu...anted=allCould not fetch link
http://www.nytimes.com/2010/05/31/bu...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 452, in process_links
File "site-packages\calibre\web\feeds\news.py", line 615, in _postprocess_html
AttributeError: 'NoneType' object has no attribute 'insert'

Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 452, in process_links
File "site-packages\calibre\web\feeds\news.py", line 615, in _postprocess_html
AttributeError: 'NoneType' object has no attribute 'insert'

http://www.nytimes.com/2010/05/31/bu...pagewanted=all saved to c:\docume~1\allen\locals~1\temp\calibre_0.6.55_hx1 _jf_plumber\feed_0\article_4\31privacy.xhtml
http://www.nytimes.com/2010/05/31/bu...pagewanted=all saved to c:\docume~1\allen\locals~1\temp\calibre_0.6.55_hx1 _jf_plumber\feed_0\article_2\31memphis.xhtml
Downloading
Downloading
Fetching Fetchinghttp://www.nytimes.com/2010/05/31/world/asia/31japan.html?pagewanted=all
http://www.nytimes.com/2010/05/31/wo...pagewanted=all
Processing images...
Recursion limit reached. Skipping links in http://www.nytimes.com/2010/05/31/ny...pagewanted=all
Could not fetch link http://www.nytimes.com/2010/05/31/ny...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 452, in process_links
File "site-packages\calibre\web\feeds\news.py", line 615, in _postprocess_html
AttributeError: 'NoneType' object has no attribute 'insert'

http://www.nytimes.com/2010/05/31/ny...pagewanted=all saved to c:\docume~1\allen\locals~1\temp\calibre_0.6.55_hx1 _jf_plumber\feed_0\article_3\31gay.xhtml
Downloading
Fetching http://www.nytimes.com/2010/05/31/wo...pagewanted=all
Failed to download article: Web Start-Ups Offer Bargains for Users’ Data from http://www.nytimes.com/2010/05/31/bu...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason

Failed to download article: Blacks in Memphis Lose Decades of Economic Gains from http://www.nytimes.com/2010/05/31/bu...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason

Failed to download article: Prospective Catholic Priests Face Sexuality Hurdles from http://www.nytimes.com/2010/05/31/ny...pagewanted=all
Traceback (most recent call last):
File "site-packages\calibre\utils\threadpool.py", line 95, in run
File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article
File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason

Processing images...
Fetching http://up.nytimes.com/?d=0//12&t=2&s...gewanted%3dall
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 337, in process_images
File "site-packages\PIL\Image.py", line 1916, in open
IOError: cannot identify image file
Recursion limit reached. Skipping links in http://www.nytimes.com/2010/05/31/wo...pagewanted=all

postprocess_html()

GRiker · 05-31-2010, 03:23 PM

OK, that's useful data, I'll keep looking at it. For some reason, I don't get many ad insertions which makes it challenging to trace.

G

GRiker · 05-31-2010, 04:35 PM

@jsl21:
Since I don't seem to trigger the ads as frequently as you, please try the attached recipe, it has some extra diagnostic printouts that should help pin down the problem. Add it as a custom recipe, and run it instead of the built-in version. You will need to supply your login credentials in the Fetch news dialog.

G

jsl21 · 06-01-2010, 10:10 PM

GRiker,

I've attached the entire log from when it failed for the first few pages. I noticed it says in it that it is skipping the ad right before it says that it failed to load the page. I also know that this is going to almost impossible to debug as whenever you run it a second time in a day, there are no ads and it works perfectly. I guess my temporary fix is to schedule it to run twice in succession! Good luck!

Thanks again

GRiker · 06-02-2010, 06:26 AM

@jsl21,
The log was informative, please try this revised recipe and let me know the results.

G

maxbookworm · 06-02-2010, 07:34 AM

This may or may not be coincidence, but I am still using the older recipe and I set my download time to 7:15. I have not had blank articles all week. Maybe downloading at an odd time helps?

jsl21 · 06-02-2010, 09:18 AM

Griker,

The good news is that it worked fine today with both the most recent recipe and the one you sent the other day (run on separate computers). However, the log for both did not show that it encountered any ads! I have no idea what determines whether the ads appear or not. I'll keep monitoring going forward.

Thanks!

GRiker · 06-02-2010, 10:31 AM

jsl21:
I'm pretty sure that the latest revision fixes the problem. My guess is that it's something you can only test once a day, so let me know if you do see any problems.

Thanks for your assistance.

G

jsl21 · 06-03-2010, 10:22 PM

GRiker,

I think it worked today after getting the ads. On the first few pages you can see what looks like part of an ad for the NYT and the beginning and the end of the articles. Great job!

GRiker · 06-04-2010, 01:50 PM

jsl21: Thanks, there's still a bit of cleanup to do, but at least the text of the desired article is delivered. It's on the todo list.

G

maxbookworm · 06-26-2010, 09:14 AM

After weeks of having this work perfectly, I am having the same issue again. Nearly every article in the front page section is blank.

The output from the fetch is attached.

Thanks!

GRiker · 06-26-2010, 09:44 AM

Quote:

Originally Posted by maxbookworm

After weeks of having this work perfectly, I am having the same issue again. Nearly every article in the front page section is blank.

The output from the fetch is attached.

Thanks!

Are you running from a custom recipe or the built-in recipe?

The built-in recipe has been updated, either use it or update your custom recipe from the built-in recipe.

G

maxbookworm · 06-26-2010, 10:45 AM

I'm using the built-in recipe.

GRiker · 06-26-2010, 10:48 AM

What version of calibre? Have you updated to 0.7.5?

G

maxbookworm · 06-26-2010, 10:49 AM

Yup, I just updated this morning, and re-fetched the NYT after I updated

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Inconsistent Display of Cover	Motto	Kobo Reader	2	03-22-2011 11:36 PM
Inconsistent Metadata	windybt	Calibre	8	10-10-2010 12:31 PM
Calibre NYT News Fetch Problem	e-literacy	Calibre	1	04-01-2010 06:40 PM
Calibre, NYT and Opinion pages	astromusic	Calibre	0	03-02-2009 04:41 PM
Cannot Download Calibre 0.4.89	JSWolf	Calibre	9	10-01-2008 06:06 PM

05-31-2010, 03:16 PM	#16
jsl21 Member Posts: 17 Karma: 10 Join Date: May 2010 Device: Kindle	GRiker, Thanks for looking at this so quickly! Below is the part of the log where the failed loading of the articles is. It is the same first 4 or 5 articles that used to say "advertisement." And just like before, if I reload the same day, i get all of the articles. Failed to download article: White House Tries to Regroup as Criticism Mounts Over Leak from http://www.nytimes.com/2010/05/31/us...allDownloading Traceback (most recent call last): File "site-packages\calibre\utils\threadpool.py", line 95, in run File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article Exception: Could not fetch article. Run with -vv to see the reason Fetching http://www.nytimes.com/2010/05/31/wo...pagewanted=all Processing images... Recursion limit reached. Skipping links in http://www.nytimes.com/2010/05/31/wo...pagewanted=all Could not fetch link http://www.nytimes.com/2010/05/31/wo...pagewanted=all Traceback (most recent call last): File "site-packages\calibre\web\fetch\simple.py", line 452, in process_links File "site-packages\calibre\web\feeds\news.py", line 615, in _postprocess_html AttributeError: 'NoneType' object has no attribute 'insert' http://www.nytimes.com/2010/05/31/wo...pagewanted=all saved to c:\docume~1\allen\locals~1\temp\calibre_0.6.55_hx1 _jf_plumber\feed_0\article_1\31koreanavy.xhtml Processing images... Processing images... Downloading Fetching Recursion limit reached. Skipping links inhttp://www.nytimes.com/2010/05/31/world/asia/31flogging.html?pagewanted=all http://www.nytimes.com/2010/05/31/bu...pagewanted=all Recursion limit reached. Skipping links in http://www.nytimes.com/2010/05/31/bu...pagewanted=all Failed to download article: U.S. to Aid South Korea With Naval Defense Plan from http://www.nytimes.com/2010/05/31/wo...pagewanted=all Traceback (most recent call last): File "site-packages\calibre\utils\threadpool.py", line 95, in run File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article Exception: Could not fetch article. Run with -vv to see the reason Could not fetch link http://www.nytimes.com/2010/05/31/bu...anted=allCould not fetch link http://www.nytimes.com/2010/05/31/bu...pagewanted=all Traceback (most recent call last): File "site-packages\calibre\web\fetch\simple.py", line 452, in process_links File "site-packages\calibre\web\feeds\news.py", line 615, in _postprocess_html AttributeError: 'NoneType' object has no attribute 'insert' Traceback (most recent call last): File "site-packages\calibre\web\fetch\simple.py", line 452, in process_links File "site-packages\calibre\web\feeds\news.py", line 615, in _postprocess_html AttributeError: 'NoneType' object has no attribute 'insert' http://www.nytimes.com/2010/05/31/bu...pagewanted=all saved to c:\docume~1\allen\locals~1\temp\calibre_0.6.55_hx1 _jf_plumber\feed_0\article_4\31privacy.xhtml http://www.nytimes.com/2010/05/31/bu...pagewanted=all saved to c:\docume~1\allen\locals~1\temp\calibre_0.6.55_hx1 _jf_plumber\feed_0\article_2\31memphis.xhtml Downloading Downloading Fetching Fetchinghttp://www.nytimes.com/2010/05/31/world/asia/31japan.html?pagewanted=all http://www.nytimes.com/2010/05/31/wo...pagewanted=all Processing images... Recursion limit reached. Skipping links in http://www.nytimes.com/2010/05/31/ny...pagewanted=all Could not fetch link http://www.nytimes.com/2010/05/31/ny...pagewanted=all Traceback (most recent call last): File "site-packages\calibre\web\fetch\simple.py", line 452, in process_links File "site-packages\calibre\web\feeds\news.py", line 615, in _postprocess_html AttributeError: 'NoneType' object has no attribute 'insert' http://www.nytimes.com/2010/05/31/ny...pagewanted=all saved to c:\docume~1\allen\locals~1\temp\calibre_0.6.55_hx1 _jf_plumber\feed_0\article_3\31gay.xhtml Downloading Fetching http://www.nytimes.com/2010/05/31/wo...pagewanted=all Failed to download article: Web Start-Ups Offer Bargains for Users’ Data from http://www.nytimes.com/2010/05/31/bu...pagewanted=all Traceback (most recent call last): File "site-packages\calibre\utils\threadpool.py", line 95, in run File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article Exception: Could not fetch article. Run with -vv to see the reason Failed to download article: Blacks in Memphis Lose Decades of Economic Gains from http://www.nytimes.com/2010/05/31/bu...pagewanted=all Traceback (most recent call last): File "site-packages\calibre\utils\threadpool.py", line 95, in run File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article Exception: Could not fetch article. Run with -vv to see the reason Failed to download article: Prospective Catholic Priests Face Sexuality Hurdles from http://www.nytimes.com/2010/05/31/ny...pagewanted=all Traceback (most recent call last): File "site-packages\calibre\utils\threadpool.py", line 95, in run File "site-packages\calibre\web\feeds\news.py", line 751, in fetch_article File "site-packages\calibre\web\feeds\news.py", line 747, in _fetch_article Exception: Could not fetch article. Run with -vv to see the reason Processing images... Fetching http://up.nytimes.com/?d=0//12&t=2&s...gewanted%3dall Traceback (most recent call last): File "site-packages\calibre\web\fetch\simple.py", line 337, in process_images File "site-packages\PIL\Image.py", line 1916, in open IOError: cannot identify image file Recursion limit reached. Skipping links in http://www.nytimes.com/2010/05/31/wo...pagewanted=all postprocess_html()

05-31-2010, 03:23 PM	#17
GRiker Comparer of the Ephemeris Posts: 1,496 Karma: 424697 Join Date: Mar 2009 Device: iPad	OK, that's useful data, I'll keep looking at it. For some reason, I don't get many ad insertions which makes it challenging to trace. G

06-02-2010, 07:34 AM	#21
maxbookworm Addict Posts: 372 Karma: 1122865 Join Date: Apr 2010 Device: Kindle Voyage, Galaxy Note 2	This may or may not be coincidence, but I am still using the older recipe and I set my download time to 7:15. I have not had blank articles all week. Maybe downloading at an odd time helps?

06-02-2010, 09:18 AM	#22
jsl21 Member Posts: 17 Karma: 10 Join Date: May 2010 Device: Kindle	Griker, The good news is that it worked fine today with both the most recent recipe and the one you sent the other day (run on separate computers). However, the log for both did not show that it encountered any ads! I have no idea what determines whether the ads appear or not. I'll keep monitoring going forward. Thanks!

06-02-2010, 10:31 AM	#23
GRiker Comparer of the Ephemeris Posts: 1,496 Karma: 424697 Join Date: Mar 2009 Device: iPad	jsl21: I'm pretty sure that the latest revision fixes the problem. My guess is that it's something you can only test once a day, so let me know if you do see any problems. Thanks for your assistance. G

06-03-2010, 10:22 PM	#24
jsl21 Member Posts: 17 Karma: 10 Join Date: May 2010 Device: Kindle	GRiker, I think it worked today after getting the ads. On the first few pages you can see what looks like part of an ad for the NYT and the beginning and the end of the articles. Great job!

06-04-2010, 01:50 PM	#25
GRiker Comparer of the Ephemeris Posts: 1,496 Karma: 424697 Join Date: Mar 2009 Device: iPad	jsl21: Thanks, there's still a bit of cleanup to do, but at least the text of the desired article is delivered. It's on the todo list. G

06-26-2010, 10:45 AM	#28
maxbookworm Addict Posts: 372 Karma: 1122865 Join Date: Apr 2010 Device: Kindle Voyage, Galaxy Note 2	I'm using the built-in recipe.

06-26-2010, 10:48 AM	#29
GRiker Comparer of the Ephemeris Posts: 1,496 Karma: 424697 Join Date: Mar 2009 Device: iPad	What version of calibre? Have you updated to 0.7.5? G

06-26-2010, 10:49 AM	#30
maxbookworm Addict Posts: 372 Karma: 1122865 Join Date: Apr 2010 Device: Kindle Voyage, Galaxy Note 2	Yup, I just updated this morning, and re-fetched the NYT after I updated

Advert

Advert