New York Times

[kovidgoyal]

Yes, looks like the NYT servers have changed their backend again, sigh.

[mkgtu]

"That which does not kill us makes us stronger."

- Friedrich Nietzsche

[bllittle]

Yes, even though the developer claims to not support recipes, this has broken again after the latest update.

Spoiler:

Fetch news from The New York Times
Conversion options changed from defaults:
recipe_specific_option: ['comp:Yes']
output_profile: 'kindle_fire'
verbose: 2
Resolved conversion options
calibre version: 8.8.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000001FD85D1E110>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KindleFireOutput object at 0x000001FD85D2CD10>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': ['comp:Yes'],
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: User-Agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Recipe specific options:
web = Todays Paper
days = 7
comp = Yes
Using today's paper from: 2025/08/14
Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 213, in main
File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1089, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 153, in convert
File "calibre\web\feeds\news.py", line 1121, in download
File "calibre\web\feeds\news.py", line 1299, in build_index
File "<string>", line 285, in parse_index
File "<string>", line 255, in parse_todays_page
File "<string>", line 248, in nyt_graphql_query
File "mechanize\_mechanize.py", line 257, in open
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 400: Bad Request

[kovidgoyal]

https://github.com/kovidgoyal/calibr...13f3c8fdc0c007

NYTimes IT department seems to be severely underemployed.

[Michael-E]

[BillD]

I am getting errors with NYT today ... I am using the built-in recipes I believe (I have latest Calibre for Windows installed version 8.10, with no custom recipes).

Here is the log of the failure ...

Spoiler:

calibre, version 8.10.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The New York Times

Fetch news from The New York Times
Conversion options changed from defaults:
output_profile: 'kindle'
verbose: 2
Resolved conversion options
calibre version: 8.10.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000002DBCE565E50>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KindleOutput object at 0x000002DBCE590150>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: User-Agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Recipe specific options:
web = Todays Paper
days = 7
comp = no
Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 213, in main
File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1089, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 153, in convert
File "calibre\web\feeds\news.py", line 1121, in download
File "calibre\web\feeds\news.py", line 1299, in build_index
File "<string>", line 224, in parse_index
File "<string>", line 198, in parse_todays_page
File "<string>", line 310, in parse_todays_page
File "<string>", line 270, in preloaded_data
File "json\__init__.py", line 346, in loads
File "json\decoder.py", line 340, in decode
json.decoder.JSONDecodeError: Extra data: line 1 column 940561 (char 940560)

[kovidgoyal]

https://github.com/kovidgoyal/calibr...66ceee1c66cb9d

[Jia Guo]

Originally Posted by kovidgoyal

This fixes it for me, but dont know how long it will remain fixed.
https://github.com/kovidgoyal/calibr...0c145c09025cf0

hello kovidgoyal
I have been started to use and learn calibre recently .
it is super powerful to download manginess I'd like to(barron / nytimes etc)
my question is how on earth do calibre able to download it without subscribing ?
could you point out the code of the secret of download different news source needs to be subscribed and pay ?

[scottsan1]

Master Kovidgoyal,

As of this morning, September 14, 2025 this receipe (https://github.com/kovidgoyal/calibr...0c145c09025cf0) downloads section headers only. No Sunday NY Times?????????? What shall I do?

[BillD]

Originally Posted by kovidgoyal

https://github.com/kovidgoyal/calibr...66ceee1c66cb9d

Sorry ... do I download these 3 files from github and then load them into Calibre to create custom recipes? Or do I copy them to some folder to overlay the built-in recipes?

[unkn0wn]

just use the built-in recipes like you usually do. It will update itself.

[BillD]

Originally Posted by scottsan1

Master Kovidgoyal,

As of this morning, September 14, 2025 this receipe (https://github.com/kovidgoyal/calibr...0c145c09025cf0) downloads section headers only. No Sunday NY Times?????????? What shall I do?

Same here with the current NYT recipes - failing to download articles.

There are the sorts of errors I am seeing ...

Could not fetch link https://www.nytimes.com/2025/08/05/books/review/elliot-ackerman-sheepdogs.html
Traceback (most recent call last):
  File "calibre\web\fetch\simple.py", line 557, in process_links
  File "calibre\web\fetch\simple.py", line 194, in get_soup
  File "calibre\web\feeds\news.py", line 670, in preprocess_raw_html_
  File "<string>", line 132, in preprocess_raw_html
  File "calibre.web.site_parsers.nytimes", line 295, in extract_html
  File "calibre.web.site_parsers.nytimes", line 210, in json_to_html
  File "json\__init__.py", line 346, in loads
  File "json\decoder.py", line 337, in decode
  File "json\decoder.py", line 353, in raw_decode
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 20742 (char 20741)

https://www.nytimes.com/2025/08/05/books/review/elliot-ackerman-sheepdogs.html saved to 
Failed to download article: A Plane Heist Goes Outrageously Awry in This Brisk Spy Thriller from https://www.nytimes.com/2025/08/05/books/review/elliot-ackerman-sheepdogs.html
Traceback (most recent call last):
  File "calibre\utils\threadpool.py", line 100, in run
  File "calibre\web\feeds\news.py", line 1250, in fetch_article
  File "calibre\web\feeds\news.py", line 1245, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in this log

[kovidgoyal]

https://github.com/kovidgoyal/calibr...bb8bee720634bb

[scottsan1]

Bravo! success