Truncation of the NYTimes Headlines recipe

[Nanoox]

I've searched the forums and it doesn't appear that anyone's reported this (or perhaps it's only happening to me!).

I run the NYTimes Headlines recipe daily at 6:30am.

Since last Friday (I think -- it's been about a week), the articles cease to load as of the New York/Region section. In other words, articles are fine for the first 7-8 sections, and then articles appear blank with a "Log In to Download" message. This can even happen within the same section; e.g. the first article in the Arts section will have content, and the other two will show the "Log In" message.

This was working fine for me until last week. I believe that I have my log in information set for that recipe (and the complete NYTimes recipe which I run once a week on Fridays with no issues), but will check Calibre on my home Mac when I get back home.

Has anyone else seen this behavior? Is there perhaps some content throttling starting to be imposed by NYTimes?

Thanks.

[Nanoox]

I've checked it in Calibre and it has no username and pw field, unlike the full NYTimes recipe.

Perhaps this one needs to be updated with a username field to avoid throttling downloads?

[kovidgoyal]

Sounds like the NY Times now requires a password for the headlines articles as well, I'll change the recipe for the next release.

[Nanoox]

Thanks Kovid! I look forward to it.

[Nanoox]

And here's the log:

calibre, version 0.7.46
ERROR: Conversion Error: <b>Failed</b>: Fetch news from New York Times Headlines

Fetch news from New York Times Headlines
Resolved conversion options
calibre version: 0.7.46
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'enable_heuristics': False,
 'extra_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x108487ad0>,
 'insert_blank_line': False,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_ignore_margins': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindleOutput object at 0x108488150>,
 'page_breaks_before': None,
 'password': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'rescale_images': False,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'username': None,
 'verbose': 2}
Python function terminated unexpectedly: The "New York Times Headlines" recipe needs a username and password.
InputFormatPlugin: Recipe Input running
Traceback (most recent call last):
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 147, in main
    return run_entry_point()
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 116, in run_entry_point
    return getattr(pmod, func)()
  File "site-packages/calibre/utils/ipc/worker.py", line 110, in main
  File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 904, in run
  File "site-packages/calibre/customize/conversion.py", line 204, in __call__
  File "site-packages/calibre/web/feeds/input.py", line 101, in convert
  File "site-packages/calibre/web/feeds/news.py", line 629, in __init__
ValueError: The "New York Times Headlines" recipe needs a username and password.

[ChappyOnIce]

I get this for several whole sections on the NY Times headlines recipe. However, if I go to the website, I can access all the articles on the headlines page that show up with "Please log in" in Calibre, without needing to log in. I think there may be other issues at work than needing a login.

[ChappyOnIce]

After poking around some more, it appears that the NY Times only allows so many accesses per browser session. I used Chrome to access the headlines page and after being able to pull up several articles, I finally got a login prompt. I then got the same prompt for the articles I was previously able to open. Then I opened IE, pulled up the same headlines page and started counting as I accessed the articles. On the 12th one, I was prompted to login.

[sdow1]

The Times is in the process of erecting its paywall (which they claim is going to launch any day now), so it's possible the changes we're seeing are the result of new barriers they're erecting "under the hood" to make the paywall work without actually having to pay quite yet.

From my understanding, the "casual" browser will probably never have to pay, but the paywall is designed to get $$ from people who read the whole paper online every day, and who don't already pay for a paper subscription. I imagine that calibre users would count in the latter.

My info is probably not helpful on a technical level, but it might be worth waiting a few days, or weeks (as annoying as it is) for the paywall to actually kick in before spending a ton of time revamping the recipe - if they're making a boatload of changes to the site, it might be easier in the long run to wait for the dust to settle.