The Economist recipe fails to fetch past editions

[xkcklzn]

I've been trying to customize a news recipe in the latest version of Calibre, both through the Advanced tab in the Fetch News dialog and using the command line --recipe-specific-option.However, both methods fail with the same error. Could anyone help me troubleshoot or guide me on the correct approach?

calibre, version 7.21.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The Economist

Fetch news from The Economist
Conversion options changed from defaults:
  recipe_specific_option: ['date:2024-11-02']
  verbose: 2
  output_profile: 'generic_eink_hd'
Resolved conversion options
calibre version: 7.21.0
{'add_alt_text_to_img': False,
 'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_max_image_size': 'none',
 'epub_toc_at_end': False,
 'epub_version': '2',
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x000001972FDB0550>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.GenericEinkHD object at 0x0000019730EC5E10>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'recipe_specific_option': ['date:2024-11-02'],
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:economist
Trying to get latest version of recipe: economist
Recipe specific options:
  res = 600
  date = 2024-11-02
Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 83, in <module>
  File "site.py", line 78, in main
  File "site.py", line 50, in run_entry_point
  File "calibre\utils\ipc\worker.py", line 215, in main
  File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
  File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "calibre\ebooks\conversion\plumber.py", line 1129, in run
  File "calibre\customize\conversion.py", line 245, in __call__
  File "calibre\ebooks\conversion\plugins\recipe_input.py", line 155, in convert
  File "calibre\web\feeds\news.py", line 1124, in download
  File "calibre\web\feeds\news.py", line 1303, in build_index
  File "<string>", line 265, in parse_index
KeyError: 'id'

[unkn0wn]

https://github.com/kovidgoyal/calibr...6a98e63766a410

[xkcklzn]

Originally Posted by unkn0wn

https://github.com/kovidgoyal/calibr...6a98e63766a410

Thanks for your help!
------------------------------
I’ve tested the built-in The Economist recipe in Calibre 7.21.0 on different systems, networks, and at various times (spanning several hours) to fetch a past edition of The Economist (2024-11-02), but I still encounter:

calibre, version 7.21.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The Economist

Fetch news from The Economist
Conversion options changed from defaults:
  recipe_specific_option: ['date:2024-11-02']
  verbose: 2
  output_profile: 'generic_eink_hd'
Resolved conversion options
calibre version: 7.21.0
{'add_alt_text_to_img': False,
 'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_max_image_size': 'none',
 'epub_toc_at_end': False,
 'epub_version': '2',
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x00000200C5EB0650>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.GenericEinkHD object at 0x00000200C6F85ED0>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'recipe_specific_option': ['date:2024-11-02'],
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1026
Using user agent: common_words/based
Using proxies: {'http': '127.0.0.1:10809', 'https': '127.0.0.1:10809', 'ftp': 'http://127.0.0.1:10809'}
Recipe specific options:
  res = 600
  date = 2024-11-02
Fetching articles from web archive.
Traceback (most recent call last):
  File "<string>", line 293, in parse_index
  File "calibre\web\feeds\news.py", line 752, in index_to_soup
  File "mechanize\_mechanize.py", line 241, in open_novisit
  File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 400: Bad Request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 83, in <module>
  File "site.py", line 78, in main
  File "site.py", line 50, in run_entry_point
  File "calibre\utils\ipc\worker.py", line 215, in main
  File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
  File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "calibre\ebooks\conversion\plumber.py", line 1129, in run
  File "calibre\customize\conversion.py", line 245, in __call__
  File "calibre\ebooks\conversion\plugins\recipe_input.py", line 155, in convert
  File "calibre\web\feeds\news.py", line 1124, in download
  File "calibre\web\feeds\news.py", line 1303, in build_index
  File "<string>", line 297, in parse_index
  File "<string>", line 475, in parse_web_index
  File "<string>", line 485, in economist_parse_web_index
AttributeError: 'dict' object has no attribute 'replace'

I suspect the issue might be related to the User-Agent string (common_words/based) being blocked by the website.

Does anyone know how to modify the recipe to use a valid browser User-Agent, or if there’s another potential cause or workaround?

Thanks for your help!

[unkn0wn]

past edition doesnt always work now with economist.. keep trying but dont spam.

I was able to fetch that particular edition.

[xkcklzn]

Huge thanks for all your incredible work! It works well.