Failure fetching The Economist: NoArticles

[Kman]

This was at the bottom of "Details" for "Fail" message fetching The Economist:

NoArticles: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

[ebilg]

here's the full log
Fetch news from The Economist
Conversion options changed from defaults:
verbose: 2
output_profile: 'generic_eink'
Resolved conversion options
calibre version: 8.16.2
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x1123dbc10>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.GenericEink object at 0x1123e7910>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:economist
Trying to get latest version of recipe: economist
Using user agent: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Recipe specific options:
res = 600
de = No
Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 42, in <module>
File "site.py", line 38, in main
File "calibre/utils/ipc/worker.py", line 213, in main
File "calibre/gui2/convert/gui_conversion.py", line 32, in gui_convert_recipe
File "calibre/gui2/convert/gui_conversion.py", line 26, in gui_convert
File "calibre/ebooks/conversion/plumber.py", line 1088, in run
File "calibre/customize/conversion.py", line 242, in __call__
File "calibre/ebooks/conversion/plugins/recipe_input.py", line 153, in convert
File "calibre/web/feeds/news.py", line 1121, in download
File "calibre/web/feeds/news.py", line 1299, in build_index
File "<string>", line 291, in parse_index
File "<string>", line 267, in economist_return_index
NoArticles: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

[kovidgoyal]

economist seems to have switched to using cloudflare bot protection which means this is likely not fixable with any reasonable amount of effort.

[xkcklzn]

The binary installer fails to fetch The Economist on Ubuntu, while the older distribution package (apt-get install calibre) continues to work fine. This will probably get blocked too once it becomes more widely used.

[kovidgoyal]

Temporary fix: https://github.com/kovidgoyal/calibr...26390daf28b690

[xiatian]

File "mechanize/_mechanize.py", line 241, in open_novisit
File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 403: Forbidden

[sriramna]

Originally Posted by kovidgoyal

Temporary fix: https://github.com/kovidgoyal/calibr...26390daf28b690

Thanks for the quick fix . Took 20 minutes now though.

[Glenn680]

Is there a similar temporary fix available for Economist Espresso? Thanks.

[kovidgoyal]

The temporary fix has already stopped working, IIRC.

[hansschnee]

hi kovid!

really appreciate your work! quick question: would a user login help to circumvent the cloudflare problem? I do have a valid economist subscription, I just want an epub file so that I can read it on my ereader.

thanks!

[kovidgoyal]

No, I dont think so. cloudflare layer happens before auth processing.

[hansschnee]

thats a pity but thanks for the swift reply!

[ebilg]

Working for me in 9.1 with the old mechanize engine. Thanks Kovid! I sent some $ from your support Calibre link.

[kovidgoyal]

The bot protection basically learns from aggregate traffic. As more people try to download the eco today/tomorrow it will start blocking them. Then it will forget again in a day or two and next week it will work for some time. That's what it looks like to me anyway. And yes, for the moment it is working.

[yoyoisbest]

Originally Posted by kovidgoyal

The bot protection basically learns from aggregate traffic. As more people try to download the eco today/tomorrow it will start blocking them. Then it will forget again in a day or two and next week it will work for some time. That's what it looks like to me anyway. And yes, for the moment it is working.

Still not working now.
Is it possible to import curl_cffi and impersonate as chrome?