New York Times site redesign

[NSILMike]

Originally Posted by mcnk522

Thank upi for working on improving this recipe. Where is the best place for getting an updated recipe if we don't want to download a whole new version of calibre? Is it the github site?

Interesting... this suggests a question about recipe updates. If I enable a recipe within Calibre by clicking 'fetch news' and 'schedule a news download' my assumption has been that any subsequent updates or modifications to that recipe by the author would be reflected at the next download using that recipe. Is that not correct?

[gianfri]

Originally Posted by nelson1379

Using the latest recipe from Github removed the remaining cruft. Thanks, Kovid.

This recipe appears to be only for the headlines. Is there another one updated with the full edition of the NYT? Thank you.

[pa_ubach]

Hi Gianfri,

The recipe is valid for the full edition as well. You only need to set the options accordingly. I use the following:

headlinesOnly = False

and

webEdition = True

Kovid, I have seen that the needs_subscription variable is set to False in all cases. Is this correct? I would suspect that you actually need the subscription to get the full version of the NYTimes. I have set it to 'optional'.

Maybe you also need to include this fragment of code to the definition of get_browser(self):

if self.username is not None and self.password is not None:
            br.open('http://www.nytimes.com/auth/login')
            br.form = br.forms().next()
            br['userid']   = self.username
            br['password'] = self.password
            raw = br.submit().read()
            if 'Please try again' in raw:
                raise Exception('Your username and password are incorrect')

I have included it myself from a previous version of the recipe and works well.

Now, if you are a perfectionist, there is still a small glitch. When the recursions is set to a value different than zero, some fetched articles show the numbers "1" and "2" bulleted at the end, along with a "Next Page" would-be-button. Example (note that the website doesn't show multiple pages for the article and that the URL address has the code ?pagewanted=all appended). I guess this has to do with the web redesign and only affects articles published before the redesign. So if that is the case, I don't think you need to worry about; because eventually all the articles fetched under the recursive option will have been edited using the new web technology and this problem will disappear.
Disclaimer: I am using Calibre v0.9.6 because I am using Fedora17 at my HTPC and I am too lazy to bother about compilations and don't want to worry about upgrading a system that has taken a lot of effort to get right.

By the way, Are you aware that David Pogue has left the NYTimes as columnist and has set a new venture with Yahoo? His column is really outdated nowadays. The New York Times has just hired a new Technology columnist who will be in charge of "State of the Art". He starts next month so I guess we won't know the new feed address until then.

Thanks all to paying attention to this recipe and fixing it. And THANKS Kovid for this outstanding recipe and specially for Calibre!!! I wish we had the same support for the recipes of the main newpapers in Spain!

[kovidgoyal]

The NYTimes doesn't need a subscription, since it uses a javascript paywall, which does not work for non-browser based download systems like calibre's recipe system.

[gianfri]

1. It looks like the byline is only loaded for a few articles, mostly from the front page. Most articles are only listed with their title. Before the redesign I seem to remember that most articles had bylines.

2. At least on my iPad, most (all?) articles show a wide blank gap, about half way into the article. The text is still all there, there is just a gap that can be as much as a third of a page on my device.

Overall the recipe works great, though. Still one of my favorite. Thank you so much to all of you who are working to improve it!

[randomzombie]

Hi Kovid...
Just wanted to thank you for a great service - these recipes are appreciated!
But also, a few questions:
a) How would I amend the recipe to preserve the "Today's Paper" format and organization style, a la:
http://www.nytimes.com/pages/todayspaper/index.html ?
b) What is the difference between your nytimes and nytimes_sub recipes? Is the latter for subscribers?
c) How would I incorporate other blogs instead of the Tech stories, examples such as Well (http://well.blogs.nytimes.com/) and The Stone (http://opinionator.blogs.nytimes.com...ory/the-stone/), without potentially upsetting the rest of your code?

Thanks again for the recipes as they stand!

[kovidgoyal]

These aren't my recipes, you'd have to ask the recipe authors. I just tweaked thema little to keep up with ongoing site changes at nytimes.com