New York Times

[kovidgoyal]

Yes, looks like the NYT servers have changed their backend again, sigh.

[mkgtu]

"That which does not kill us makes us stronger."

- Friedrich Nietzsche

[bllittle]

Yes, even though the developer claims to not support recipes, this has broken again after the latest update.

Spoiler:

Fetch news from The New York Times
Conversion options changed from defaults:
recipe_specific_option: ['comp:Yes']
output_profile: 'kindle_fire'
verbose: 2
Resolved conversion options
calibre version: 8.8.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000001FD85D1E110>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KindleFireOutput object at 0x000001FD85D2CD10>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': ['comp:Yes'],
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: User-Agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Recipe specific options:
web = Todays Paper
days = 7
comp = Yes
Using today's paper from: 2025/08/14
Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 213, in main
File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1089, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 153, in convert
File "calibre\web\feeds\news.py", line 1121, in download
File "calibre\web\feeds\news.py", line 1299, in build_index
File "<string>", line 285, in parse_index
File "<string>", line 255, in parse_todays_page
File "<string>", line 248, in nyt_graphql_query
File "mechanize\_mechanize.py", line 257, in open
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 400: Bad Request

[kovidgoyal]

https://github.com/kovidgoyal/calibr...13f3c8fdc0c007

NYTimes IT department seems to be severely underemployed.

[Michael-E]

[BillD]

I am getting errors with NYT today ... I am using the built-in recipes I believe (I have latest Calibre for Windows installed version 8.10, with no custom recipes).

Here is the log of the failure ...

Spoiler:

calibre, version 8.10.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The New York Times

Fetch news from The New York Times
Conversion options changed from defaults:
output_profile: 'kindle'
verbose: 2
Resolved conversion options
calibre version: 8.10.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000002DBCE565E50>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KindleOutput object at 0x000002DBCE590150>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: User-Agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Recipe specific options:
web = Todays Paper
days = 7
comp = no
Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 213, in main
File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1089, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 153, in convert
File "calibre\web\feeds\news.py", line 1121, in download
File "calibre\web\feeds\news.py", line 1299, in build_index
File "<string>", line 224, in parse_index
File "<string>", line 198, in parse_todays_page
File "<string>", line 310, in parse_todays_page
File "<string>", line 270, in preloaded_data
File "json\__init__.py", line 346, in loads
File "json\decoder.py", line 340, in decode
json.decoder.JSONDecodeError: Extra data: line 1 column 940561 (char 940560)

[kovidgoyal]

https://github.com/kovidgoyal/calibr...66ceee1c66cb9d

[Jia Guo]

Originally Posted by kovidgoyal

This fixes it for me, but dont know how long it will remain fixed.
https://github.com/kovidgoyal/calibr...0c145c09025cf0

hello kovidgoyal
I have been started to use and learn calibre recently .
it is super powerful to download manginess I'd like to(barron / nytimes etc)
my question is how on earth do calibre able to download it without subscribing ?
could you point out the code of the secret of download different news source needs to be subscribed and pay ?

[scottsan1]

Master Kovidgoyal,

As of this morning, September 14, 2025 this receipe (https://github.com/kovidgoyal/calibr...0c145c09025cf0) downloads section headers only. No Sunday NY Times?????????? What shall I do?

[BillD]

Originally Posted by kovidgoyal

https://github.com/kovidgoyal/calibr...66ceee1c66cb9d

Sorry ... do I download these 3 files from github and then load them into Calibre to create custom recipes? Or do I copy them to some folder to overlay the built-in recipes?

[unkn0wn]

just use the built-in recipes like you usually do. It will update itself.

[BillD]

Originally Posted by scottsan1

Master Kovidgoyal,

As of this morning, September 14, 2025 this receipe (https://github.com/kovidgoyal/calibr...0c145c09025cf0) downloads section headers only. No Sunday NY Times?????????? What shall I do?

Same here with the current NYT recipes - failing to download articles.

There are the sorts of errors I am seeing ...

Could not fetch link https://www.nytimes.com/2025/08/05/books/review/elliot-ackerman-sheepdogs.html
Traceback (most recent call last):
  File "calibre\web\fetch\simple.py", line 557, in process_links
  File "calibre\web\fetch\simple.py", line 194, in get_soup
  File "calibre\web\feeds\news.py", line 670, in preprocess_raw_html_
  File "<string>", line 132, in preprocess_raw_html
  File "calibre.web.site_parsers.nytimes", line 295, in extract_html
  File "calibre.web.site_parsers.nytimes", line 210, in json_to_html
  File "json\__init__.py", line 346, in loads
  File "json\decoder.py", line 337, in decode
  File "json\decoder.py", line 353, in raw_decode
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 20742 (char 20741)

https://www.nytimes.com/2025/08/05/books/review/elliot-ackerman-sheepdogs.html saved to 
Failed to download article: A Plane Heist Goes Outrageously Awry in This Brisk Spy Thriller from https://www.nytimes.com/2025/08/05/books/review/elliot-ackerman-sheepdogs.html
Traceback (most recent call last):
  File "calibre\utils\threadpool.py", line 100, in run
  File "calibre\web\feeds\news.py", line 1250, in fetch_article
  File "calibre\web\feeds\news.py", line 1245, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in this log

[kovidgoyal]

https://github.com/kovidgoyal/calibr...bb8bee720634bb

[scottsan1]

Bravo! success

[michailbachtin]

On both my machines (Windows 10 & Windows 11) I get the same error message when trying to download the New York Times and the New York Times Book Review.

I'd be grateful for some help.

Cheers

Here the error message:

Spoiler:

calibre, version 8.10.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The New York Times

Fetch news from The New York Times
Failed to initialize plugin: 'C:\\Users\\mrbri\\AppData\\Roaming\\calibre\\plugins\\Goodreads.zip'
Failed to initialize plugin: 'C:\\Users\\mrbri\\AppData\\Roaming\\calibre\\plugins\\Kindle hi-res covers.zip'
Conversion options changed from defaults:
  verbose: 2
  output_profile: 'generic_eink_large'
Resolved conversion options
calibre version: 8.10.0
{'add_alt_text_to_img': False,
 'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_max_image_size': 'none',
 'epub_toc_at_end': False,
 'epub_version': '2',
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x000001B8F33706D0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.GenericEinkLarge object at 0x000001B8F33DD590>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'recipe_specific_option': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:nytimes_sub
Trying to get latest version of recipe: nytimes_sub
Using user agent: User-Agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Recipe specific options:
  web = Todays Paper
  days = 7
  comp = no
Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 83, in <module>
  File "site.py", line 78, in main
  File "site.py", line 50, in run_entry_point
  File "calibre\utils\ipc\worker.py", line 213, in main
  File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
  File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
  File "calibre\ebooks\conversion\plumber.py", line 1089, in run
  File "calibre\customize\conversion.py", line 242, in __call__
  File "calibre\ebooks\conversion\plugins\recipe_input.py", line 153, in convert
  File "calibre\web\feeds\news.py", line 1121, in download
  File "calibre\web\feeds\news.py", line 1299, in build_index
  File "<string>", line 223, in parse_index
  File "<string>", line 197, in parse_todays_page
  File "<string>", line 309, in parse_todays_page
  File "<string>", line 269, in preloaded_data
  File "json\decoder.py", line 353, in raw_decode
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 491475 (char 491474)