Calibre + NYT download = inconsistent?

maxbookworm · 05-25-2010, 04:42 PM

Hi all,

I have been using the NYT recipe in Calibre to download the full paper (I have a subscription). This is an awesome, amazing feature and I love it!

The thing is, about half the time the paper downloads, a portion of the articles are blank (usually in the Front Page section). Generally, if I re-download once or twice, this problem corrects itself and I get the full paper. I'm guessing this isn't a Calibre issue since sometimes it works and sometimes it doesn't. But has anyone else noticed this and tied it to anything specific? Is it the NYT server? Is it my internet connection? What could be causing this behavior and are there any tricks that will consistently overcome it?

Thanks!

kovidgoyal · 05-25-2010, 08:14 PM

Try looking at the download log to see what error messages are generated when you notice blank articles. That should give us a clue on the root cause.

You can get the log by clicking the rotating spinner in the ottom right corner.

jsl21 · 05-27-2010, 08:37 AM

I am having the same problem for the past few days with the NYT. If you look, the page actually says "This is an advertisement" so I am guessing what is happening is it goes to the website and an ad is being shown first before it shows the article. I am not sure how to edit the recipe to skip the ad which I would normally do on the website by clicking on skip the ad. Any ideas? By the way, sometimes you can find the actual articles from the front page in a different section, e.g., today, may 27th, the E Coli article I couldn't see when choosing it from the front page but it was in the Business section. Today I could find 3 out of the 5 that just said "this was an advertisement"

maxbookworm · 05-27-2010, 10:04 AM

Thanks, Kovid. I will paste that log here next time it happens (of course, it was fine this morning). Jsl21 -- I don't get that "advertisement" message. It is just a blank page with the navigation links at top and article url at bottom. It's a good theory, though.

jsl21 · 05-27-2010, 10:12 AM

Kovid,

I still have the "This is an advertisement problem." I looked at the log and it seems to say that page downloaded without an error but I am not an expert so I cannot tell. I saved the log. Should I post it?

JRG · 05-27-2010, 10:23 AM

Quote:

Originally Posted by maxbookworm

Thanks, Kovid. I will paste that log here next time it happens (of course, it was fine this morning). Jsl21 -- I don't get that "advertisement" message. It is just a blank page with the navigation links at top and article url at bottom. It's a good theory, though.

That's exactly what I get for most (maybe 75%) of the articles using the BBC recipe; I've pretty much abandoned BBC downloads for that reason. But the problem occurs rarely, if ever, using the NY Times recipe.

kovidgoyal · 05-27-2010, 01:17 PM

The advertisement problem would need a workaround in the recipe, the log wont help with that. IIRC the recipe used to handle that, but maybe I'm wrong.

I'll ping GRiker the author of the recipe to take a look.

kiklop74 · 05-27-2010, 03:14 PM

I noticed that BBC changed some things on their website hence the BBC Fast recipe was not working correctly.

Here is the updated version.

JRG · 05-27-2010, 05:32 PM

Quote:

Originally Posted by kiklop74

I noticed that BBC changed some things on their website hence the BBC Fast recipe was not working correctly.

Here is the updated version.

Thank you, very much!

maxbookworm · 05-28-2010, 09:20 AM

OK jsl21, you're exactly right. I had never noticed that text before, but on the blank pages it says "This is an advertisement" and "Skip this ad" (see attached).

But usually if I re-download, the problem is fixed. Like maybe the second download gets you past the ad?

GRiker · 05-29-2010, 06:25 AM

I've submitted a patch to Kovid for both versions of the NYTimes recipes (Headlines, full paper) which should properly bypass the ads.

G

maxbookworm · 05-29-2010, 07:29 AM

Wow, you guys are amazing!

Thank you so so much for these programs. They are awesome!

jsl21 · 05-29-2010, 07:39 AM

Thanks GRiker!

maxbookworm - I also have found that rerunning it the same day will fetch a new copy without the "advertisement" problem. I guess the site remembers that you've already seen ads recently and doesn't force you to see them again

jsl21 · 05-31-2010, 08:11 AM

GRiker,

Thanks for your help. I assume that your fixes to the NYT are in 0.6.55. Now it skips the first 4-5 articles on the front page (which I am guessing are the ones that come up with the "advertisement"). and they aren't downloaded at all. Was this what is supposed to happen?

GRiker · 05-31-2010, 08:21 AM

@jsl21: I just downloaded today's paper, and I got every article listed in the index. Are you comparing to the same index at http://www.nytimes.com/pages/todayspaper/index.html?

I believe that the ads are being skipped correctly, so I'm guessing that the problem you're experiencing is something else. It could be that some articles are being updated at the time you're running the recipe, or something else causing a connection failure during the download of that article.

Take a look at the download log to see if there's any further information there.

I'll keep poking around. It's possible that I'm just not getting any ads at this time.

G

05-25-2010, 04:42 PM	#1
maxbookworm Addict Posts: 372 Karma: 1122865 Join Date: Apr 2010 Device: Kindle Voyage, Galaxy Note 2	Calibre + NYT download = inconsistent? Hi all, I have been using the NYT recipe in Calibre to download the full paper (I have a subscription). This is an awesome, amazing feature and I love it! The thing is, about half the time the paper downloads, a portion of the articles are blank (usually in the Front Page section). Generally, if I re-download once or twice, this problem corrects itself and I get the full paper. I'm guessing this isn't a Calibre issue since sometimes it works and sometimes it doesn't. But has anyone else noticed this and tied it to anything specific? Is it the NYT server? Is it my internet connection? What could be causing this behavior and are there any tricks that will consistently overcome it? Thanks!

05-28-2010, 09:20 AM	#10
maxbookworm Addict Posts: 372 Karma: 1122865 Join Date: Apr 2010 Device: Kindle Voyage, Galaxy Note 2	OK jsl21, you're exactly right. I had never noticed that text before, but on the blank pages it says "This is an advertisement" and "Skip this ad" (see attached). But usually if I re-download, the problem is fixed. Like maybe the second download gets you past the ad? Attached Thumbnails

05-31-2010, 08:21 AM	#15
GRiker Comparer of the Ephemeris Posts: 1,496 Karma: 424697 Join Date: Mar 2009 Device: iPad	@jsl21: I just downloaded today's paper, and I got every article listed in the index. Are you comparing to the same index at http://www.nytimes.com/pages/todayspaper/index.html? I believe that the ads are being skipped correctly, so I'm guessing that the problem you're experiencing is something else. It could be that some articles are being updated at the time you're running the recipe, or something else causing a connection failure during the download of that article. Take a look at the download log to see if there's any further information there. I'll keep poking around. It's possible that I'm just not getting any ads at this time. G Last edited by GRiker; 05-31-2010 at 08:53 AM.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Inconsistent Display of Cover	Motto	Kobo Reader	2	03-22-2011 11:36 PM
Inconsistent Metadata	windybt	Calibre	8	10-10-2010 12:31 PM
Calibre NYT News Fetch Problem	e-literacy	Calibre	1	04-01-2010 06:40 PM
Calibre, NYT and Opinion pages	astromusic	Calibre	0	03-02-2009 04:41 PM
Cannot Download Calibre 0.4.89	JSWolf	Calibre	9	10-01-2008 06:06 PM

05-25-2010, 08:14 PM	#2
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Try looking at the download log to see what error messages are generated when you notice blank articles. That should give us a clue on the root cause. You can get the log by clicking the rotating spinner in the ottom right corner.

05-27-2010, 08:37 AM	#3
jsl21 Member Posts: 17 Karma: 10 Join Date: May 2010 Device: Kindle	I am having the same problem for the past few days with the NYT. If you look, the page actually says "This is an advertisement" so I am guessing what is happening is it goes to the website and an ad is being shown first before it shows the article. I am not sure how to edit the recipe to skip the ad which I would normally do on the website by clicking on skip the ad. Any ideas? By the way, sometimes you can find the actual articles from the front page in a different section, e.g., today, may 27th, the E Coli article I couldn't see when choosing it from the front page but it was in the Business section. Today I could find 3 out of the 5 that just said "this was an advertisement"

05-27-2010, 10:04 AM	#4
maxbookworm Addict Posts: 372 Karma: 1122865 Join Date: Apr 2010 Device: Kindle Voyage, Galaxy Note 2	Thanks, Kovid. I will paste that log here next time it happens (of course, it was fine this morning). Jsl21 -- I don't get that "advertisement" message. It is just a blank page with the navigation links at top and article url at bottom. It's a good theory, though.

05-27-2010, 10:12 AM	#5
jsl21 Member Posts: 17 Karma: 10 Join Date: May 2010 Device: Kindle	Kovid, I still have the "This is an advertisement problem." I looked at the log and it seems to say that page downloaded without an error but I am not an expert so I cannot tell. I saved the log. Should I post it?

05-27-2010, 01:17 PM	#7
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The advertisement problem would need a workaround in the recipe, the log wont help with that. IIRC the recipe used to handle that, but maybe I'm wrong. I'll ping GRiker the author of the recipe to take a look.

05-29-2010, 06:25 AM	#11
GRiker Comparer of the Ephemeris Posts: 1,496 Karma: 424697 Join Date: Mar 2009 Device: iPad	I've submitted a patch to Kovid for both versions of the NYTimes recipes (Headlines, full paper) which should properly bypass the ads. G

05-29-2010, 07:29 AM	#12
maxbookworm Addict Posts: 372 Karma: 1122865 Join Date: Apr 2010 Device: Kindle Voyage, Galaxy Note 2	Wow, you guys are amazing! Thank you so so much for these programs. They are awesome!

05-29-2010, 07:39 AM	#13
jsl21 Member Posts: 17 Karma: 10 Join Date: May 2010 Device: Kindle	Thanks GRiker! maxbookworm - I also have found that rerunning it the same day will fetch a new copy without the "advertisement" problem. I guess the site remembers that you've already seen ads recently and doesn't force you to see them again

05-31-2010, 08:11 AM	#14
jsl21 Member Posts: 17 Karma: 10 Join Date: May 2010 Device: Kindle	GRiker, Thanks for your help. I assume that your fixes to the NYT are in 0.6.55. Now it skips the first 4-5 articles on the front page (which I am guessing are the ones that come up with the "advertisement"). and they aren't downloaded at all. Was this what is supposed to happen?