New York Times site redesign

[nelson1379]

The New York Times' website was revamped yesterday. Presumably as a result of this, the recipe now includes header cruft sprinkled inside the body of the article. Potentially, there may be other issues that I'm not aware of (not sure if 100% of article content is collected/displayed).

If someone with the proper technical expertise had the inclination to fix the recipe, I would greatly appreciate it. Thanks!

[gianfri]

Today most articles were not downloaded. Only the title and the author. I wish I had the skills to fix that but I don't. I hope that others will!

[gianfri]

Originally Posted by gianfri

Today most articles were not downloaded. Only the title and the author. I wish I had the skills to fix that but I don't. I hope that others will!

https://bugs.launchpad.net/calibre/+bug/1267549

"Actually, the full body of the article is there; it's just that now every article is prepended with a "Loading" message, a bunch of menu links, the word "Inside" and a page break. If you go to the second page, the article appears correctly."

[nelson1379]

I saw that the changelog for today's new Calibre version (1.1.19) mentions that the New York Times recipe has been improved, so if someone worked on that, thanks for the effort. However, when I ran the recipe with that new version, the problems with the resulting edition were the same as they were yesterday. Is anyone getting different results?

[NSILMike]

Interesting... It downloads fine for me on 1.1.19 but cannot open (loading flow... endlessly) while it syncs and appears to work fine on my Nexus tablet which uses calibre companion to get the file from Calibre on the PC.

Note that there are two NY Times recipes, at least the last time I looked. One uses just the name, and the other uses the name with Headlines appended as I recall.

[nelson1379]

Today's NYT edition removed much of the cruft but there are still "related links" sections interrupting several of the articles. I also had the same issue as NSILMike that it wouldn't open with Calibre but opened on my Kindle. So thanks to whoever improved the recipe... just a bit more work to perfect it.

[kovidgoyal]

Post a link to an article with related items.

[nelson1379]

Any of the news articles covering the latest New Jersey governor drama pull in "related coverage" from the sidebar. Here's an example of an article from today: http://www.nytimes.com/2014/01/11/ny...-released.html

Here's a mobi of today's edition that I put on my dropbox: https://dl.dropboxusercontent.com/u/...k%20Times.mobi

I did toggle some of the options in the recipe, like removing recursions, tech blogs, most popular articles, etc but I don't believe that to be making a difference. If you can't reproduce it on your end, I'll run it without any of those options toggled and I'll see what I get.

Thanks!

[kovidgoyal]

Looking at the recipe, as far as I can tell, the author intended for th erelated articles to be included. Set recursions = 0 if you dont want them.

[nelson1379]

It's not the related articles, appearing after the main article, that are the problem -- I know how to turn them off with the recursions variable and they've been a part of the recipe for months. It's the "related coverage" sidebar and its ilk that appear in the middle of the article and are pretty useless to anyone on an e-reader.

Another example in today's paper is the "interactive graphic" sidebar five paragraphs into this article: http://www.nytimes.com/2014/01/12/us...onopolies.html In the edition that Calibre generated today, it's not even a link, it's just a tiny picture of the graphic, the words "interactive graphic", and then in much larger text, "Who Controls the States and Where They Stand." It's not a huge deal, but it's pointless cruft that I'm sure was not intentionally pulled into the Calibre edition.

Again, not the biggest issue in the world (it was much worse when I originally posted) but I figured I'd try to inform the recipe's author about it.

[kovidgoyal]

Ah, ok, that's easily remedied.

[gianfri]

Originally Posted by nelson1379

It's not the related articles, appearing after the main article, that are the problem -- I know how to turn them off with the recursions variable and they've been a part of the recipe for months. It's the "related coverage" sidebar and its ilk that appear in the middle of the article and are pretty useless to anyone on an e-reader.

Another example in today's paper is the "interactive graphic" sidebar five paragraphs into this article: http://www.nytimes.com/2014/01/12/us...onopolies.html In the edition that Calibre generated today, it's not even a link, it's just a tiny picture of the graphic, the words "interactive graphic", and then in much larger text, "Who Controls the States and Where They Stand." It's not a huge deal, but it's pointless cruft that I'm sure was not intentionally pulled into the Calibre edition.

Again, not the biggest issue in the world (it was much worse when I originally posted) but I figured I'd try to inform the recipe's author about it.

This is a very clear description of the problem - thank you!

I addition, before the site redesign most articles were listed in their section summary with a brief synopsis of the con ten. Now most articles only list the title.

[mcnk522]

Thank upi for working on improving this recipe. Where is the best place for getting an updated recipe if we don't want to download a whole new version of calibre? Is it the github site?

[kovidgoyal]

yes .

[nelson1379]

Using the latest recipe from Github removed the remaining cruft. Thanks, Kovid.