New York Times

[jazzbox]

For the last couple of days, recipes retrieve only the section headers. I checked print and web versions, book review, tech beat, sports beat. TIA

[kovidgoyal]

Looks like the nytimes has started requiring javascript and captchas. Sigh. Probably will have to get it from the wayback machine instead, although the wayback machine doesnt seem to be storing nytimes articles in a timely fashion.

Another possibility is using the graphql api though that will require some reverse engineering.

[unkn0wn]

Worked until now because of google bot headers.. looks like the bot itself got blocked.

the graphql api one looks hard.

[scottsan1]

I downloaded the Sunday NY Times today multiple times, only to retrieve section headers, as well. This is terrible.

[bllittle]

Following thread as I'm having the same issue. Hope to see a solution soon.

[kovidgoyal]

This fixes it for me, but dont know how long it will remain fixed.
https://github.com/kovidgoyal/calibr...0c145c09025cf0

[nickredding]

The New York Times and New York Times (Web) recipes already use graphql and work fine.

[kovidgoyal]

Originally Posted by nickredding

The New York Times and New York Times (Web) recipes already use graphql and work fine.

they use graphql for loading the index not article content.

[unkn0wn]

Originally Posted by kovidgoyal

This fixes it for me, but dont know how long it will remain fixed.
https://github.com/kovidgoyal/calibr...0c145c09025cf0

wow. great find... wasnt expecting this to be fixed so easily.

[nickredding]

Originally Posted by kovidgoyal

they use graphql for loading the index not article content.

True, but the article content is encapsulated in JSON within the article content so fiddling the UA is not necessary--the JSON is there with a standard UA header.

It seems that major publications are moving to a Content Management System that does this JSON encapsulation as a method of defeating simple screen scrapers. It also seems that RSS feeds are going out of style and relying on RSS in the future for indexes will be increasingly unreliable.

[unkn0wn]

Originally Posted by nickredding

fiddling the UA is not necessary--the JSON is there with a standard UA header.

yea but it requires js enabled browser, and then there's captha page for bots to fight.
with these headers it doesnt require so.

[kovidgoyal]

Originally Posted by nickredding

True, but the article content is encapsulated in JSON within the article content so fiddling the UA is not necessary--the JSON is there with a standard UA header.

It seems that major publications are moving to a Content Management System that does this JSON encapsulation as a method of defeating simple screen scrapers. It also seems that RSS feeds are going out of style and relying on RSS in the future for indexes will be increasingly unreliable.

that's irrelevant, you will get a captcha page if you try to download without an appropriate user-agent. nytimes uses captcha-delivery.com on its article pages.

[bllittle]

Originally Posted by kovidgoyal

This fixes it for me, but dont know how long it will remain fixed.
https://github.com/kovidgoyal/calibr...0c145c09025cf0

Thanks so much for the update! Its working again for me, too (at least for the moment).

[Michael-E]

The recipe appears to be broken again, at least for me. Last successfully downloaded the paper yesterday, 12 August. Today it fails no matter what tweaks I try (both recipes (web/not), resetting all options to default, etc.). Tried with and without VPN and from different IP by using personal hotspot.

These are the job details (spoiler tag used to hide the section of code before the errors appear):

Spoiler: 

Fetch news from The New York Times
Conversion options changed from defaults:
  output_profile: 'kindle_dx'
  recipe_specific_option: ['web:yes']
  verbose: 2
Resolved conversion options
calibre version: 8.8.0
{'add_alt_text_to_img': False,
 'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x1071af710>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': 'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindleDXOutput object at 0x10789fc10>,
 'page_breaks_before': None,
 'personal_doc': '[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'recipe_specific_option': ['web:yes'],
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: User-Agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Recipe specific options:
  web = yes
  days = 7
  comp = no


Using today's paper from: 2025/08/13
Failed to parse section: World with error: HTTP Error 400: Bad Request
  No articles found in section: World
Failed to parse section: U.S. with error: HTTP Error 400: Bad Request
  No articles found in section: U.S.
Failed to parse section: Politics with error: HTTP Error 400: Bad Request
  No articles found in section: Politics
Failed to parse section: New York with error: HTTP Error 400: Bad Request
  No articles found in section: New York
Failed to parse section: Business with error: HTTP Error 400: Bad Request
  No articles found in section: Business
Failed to parse section: Technology with error: HTTP Error 400: Bad Request
  No articles found in section: Technology
Failed to parse section: Sports with error: HTTP Error 400: Bad Request
  No articles found in section: Sports
Failed to parse section: Science with error: HTTP Error 400: Bad Request
  No articles found in section: Science
Failed to parse section: Health with error: HTTP Error 400: Bad Request
  No articles found in section: Health
Failed to parse section: Opinion with error: HTTP Error 400: Bad Request
  No articles found in section: Opinion
Failed to parse section: Arts with error: HTTP Error 400: Bad Request
  No articles found in section: Arts
Failed to parse section: Movies with error: HTTP Error 400: Bad Request
  No articles found in section: Movies
Failed to parse section: Music with error: HTTP Error 400: Bad Request
  No articles found in section: Music
Failed to parse section: Television with error: HTTP Error 400: Bad Request
  No articles found in section: Television
Failed to parse section: Style with error: HTTP Error 400: Bad Request
  No articles found in section: Style
Failed to parse section: Dining & Wine with error: HTTP Error 400: Bad Request
  No articles found in section: Dining & Wine
Failed to parse section: Fashion & Style with error: HTTP Error 400: Bad Request
  No articles found in section: Fashion & Style
Failed to parse section: Travel with error: HTTP Error 400: Bad Request
  No articles found in section: Travel
Failed to parse section: Education with error: HTTP Error 400: Bad Request
  No articles found in section: Education
Failed to parse section: Multimedia with error: HTTP Error 400: Bad Request
  No articles found in section: Multimedia
Failed to parse section: Obituaries with error: HTTP Error 400: Bad Request
  No articles found in section: Obituaries
Failed to parse section: Sunday Magazine with error: HTTP Error 400: Bad Request
  No articles found in section: Sunday Magazine
Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 42, in <module>
  File "site.py", line 38, in main
  File "calibre/utils/ipc/worker.py", line 213, in main
  File "calibre/gui2/convert/gui_conversion.py", line 32, in gui_convert_recipe
  File "calibre/gui2/convert/gui_conversion.py", line 26, in gui_convert
  File "calibre/ebooks/conversion/plumber.py", line 1089, in run
  File "calibre/customize/conversion.py", line 242, in __call__
  File "calibre/ebooks/conversion/plugins/recipe_input.py", line 153, in convert
  File "calibre/web/feeds/news.py", line 1121, in download
  File "calibre/web/feeds/news.py", line 1310, in build_index
ValueError: No articles found, aborting

[mkgtu]

NYT failed repeatedly today, but with a completely different error details report:

Spoiler:

Fetch news from The New York Times
Conversion options changed from defaults:
  output_profile: 'generic_eink_hd'
  verbose: 2
Resolved conversion options
calibre version: 8.8.0
{'add_alt_text_to_img': False,
 'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_max_image_size': 'none',
 'epub_toc_at_end': False,
 'epub_version': '2',
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000026B4B8BBBD0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.GenericEinkHD object at 0x0000026B4B8D7550>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'recipe_specific_option': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: User-Agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Recipe specific options:
  web = Todays Paper
  days = 7
  comp = no
Using today's paper from: 2025/08/13
Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 83, in <module>
  File "site.py", line 78, in main
  File "site.py", line 50, in run_entry_point
  File "calibre\utils\ipc\worker.py", line 213, in main
  File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
  File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
  File "calibre\ebooks\conversion\plumber.py", line 1089, in run
  File "calibre\customize\conversion.py", line 242, in __call__
  File "calibre\ebooks\conversion\plugins\recipe_input.py", line 153, in convert
  File "calibre\web\feeds\news.py", line 1121, in download
  File "calibre\web\feeds\news.py", line 1299, in build_index
  File "<string>", line 285, in parse_index
  File "<string>", line 255, in parse_todays_page
  File "<string>", line 248, in nyt_graphql_query
  File "mechanize\_mechanize.py", line 257, in open
  File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 400: Bad Request