Both "The Economist News" and "The Economist Espresso" keep failing

[joer2786]

It is so crazy to me because almost none of the magazine / news services offer epub formats which make reading their stuff on an e-reader impossible. I have to use calibre not because I just want the content for free but because there is no other way to conveniently get the content on my devices.

[Cough]

I have been fighting this issue for two days on my Laptop. But I was able to download the Economist as normal on my desktop. So I tried restore, M alwarebytes etc on my laptop.

Then I thought hmmmm... My desktop is old and only has 32 bit Calibre 5.44. My laptop is 64 bit Calibre 8.5. So after reading this thread I just loaded Caliber 5.44 onto my laptop.

Eureka!! It works fine. A few downsides with such an old version but at least I get my weekly read.

I just subscribed to the forum to help others.

[kovidgoyal]

That's probably because it uses an older network stack and the economists bot defenses havent yet learned to block it. If enough people use it, it will start getting blocked too. Not really a scaleable fix. I am still hoping I can get the log in working so that users who subscribe can at least get access to it.

[kovidgoyal]

At least for the time being, this fixes it, changing the network stack. We'll see if it remains fixed long term or not.

https://github.com/kovidgoyal/calibr...c42c1628d5396f

[jwhittin]

Thanks so much Kovid. Do I have to reinstall Calibre to get the new recipe? I thought it automatically used the latest online version but it's still erroring out for me so obviously I was wrong!

[kovidgoyal]

It takes an hour or two before the recipe update goes live.

[jwhittin]

Thanks, I'll set a timer and try again!

[jwhittin]

Thanks Kovid, all working for me (for now )

[yoyoisbest]

Originally Posted by kovidgoyal

That's probably because it uses an older network stack and the economists bot defenses havent yet learned to block it. If enough people use it, it will start getting blocked too. Not really a scaleable fix. I am still hoping I can get the log in working so that users who subscribe can at least get access to it.

Is it possible to do the log in process in Calibre, so people have subscription can access it?

[JaneD]

It's working again for me, thank you!

[HamMerhead]

Fixed 😁

FYI on another project (yfinance) the following Python has worked reliably to inhibit 404 errors:

from curl_cffi import requests
requests.Session(impersonate="chrome")

[xkcklzn]

Spoiler:

calibre, version 8.16.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The Economist

Fetch news from The Economist
Conversion options changed from defaults:
output_profile: 'generic_eink'
verbose: 2
Resolved conversion options
calibre version: 8.16.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000028715F32F10>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.GenericEink object at 0x0000028715F3AB10>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:economist
Trying to get latest version of recipe: economist
Using proxies: {'http': '127.0.0.1:10808', 'https': '127.0.0.1:10808', 'ftp': 'http://127.0.0.1:10808'}
Using user agent: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Recipe specific options:
res = 600
de = No
Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 213, in main
File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1088, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 153, in convert
File "calibre\web\feeds\news.py", line 1121, in download
File "calibre\web\feeds\news.py", line 1299, in build_index
File "<string>", line 291, in parse_index
File "<string>", line 267, in economist_return_index
NoArticles: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

Spoiler:

Fetch news from The Economist Espresso
Conversion options changed from defaults:
output_profile: 'generic_eink'
verbose: 2
Resolved conversion options
calibre version: 8.16.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000020808742E50>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.GenericEink object at 0x000002080874AB50>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:economist_espresso
Trying to get latest version of recipe: economist_espresso
Using proxies: {'http': '127.0.0.1:10808', 'https': '127.0.0.1:10808', 'ftp': 'http://127.0.0.1:10808'}
Using user agent: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Fetching https://www.economist.com/the-world-in-brief
Fetched https://www.economist.com/the-world-in-brief in 0.234000 seconds
Parsed https://www.economist.com/the-world-in-brief in 0.0 seconds
Processing images...
Recursion limit reached. Skipping links in https://www.economist.com/the-world-in-brief
https://www.economist.com/the-world-in-brief saved to C:\Users\Joea\AppData\Local\Temp\calibre-w_mul7te\fhislbwn_plumber\feed_0\article_0\index.x html
Downloaded article: The World in Brief from https://www.economist.com/the-world-in-brief
Parsing all content...
Parsing feed_0/index.html ...
Forcing feed_0/index.html into XHTML namespace
Parsing feed_0/article_0/index.html ...
Forcing feed_0/article_0/index.html into XHTML namespace
Parsing index.html ...
Forcing index.html into XHTML namespace
Referenced file 'feed_1/index.html' not found
Reading TOC from NCX...
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Found 5 items of level: div_1
Found 3 items of level: p_2
Found 1 items of level: div_4
Found 1 items of level: div_2
Ignoring level div_4
Ignoring level p_2
div_1 left margin stats: Counter()
div_1 right margin stats: Counter()
div_2 left margin stats: Counter()
div_2 right margin stats: Counter()
Cleaning up manifest...
Trimming unused files from manifest...
Creating EPUB Output...
Found non-unique filenames, renaming to support broken EPUB readers like FBReader, Aldiko and Stanza...
{'feed_0/article_0/index.html': 'feed_0/article_0/index_u1.html',
'index.html': 'index_u2.html'}
Splitting markup on page breaks and flow limits, if any...
Looking for large trees in feed_0/index.html...
No large trees found
Looking for large trees in feed_0/article_0/index_u1.html...
No large trees found
Looking for large trees in index_u2.html...
No large trees found
The cover image has an id != "cover". Renaming to work around bug in Nook Color
EPUB output written to C:\Users\Joea\AppData\Local\Temp\calibre-w_mul7te\8o4oz1x1_recipe_out.epub

[ebilg]

Based on the response header having cf_bm, it seems to be cloudflare bot protection.

"Cloudflare Bot Management uses machine learning, behavioral analysis, and fingerprinting to accurately classify bots."
https://www.cloudflare.com/en-ca/app...ot-management/


Cloudflare makes a lot of money from the services they provide so they have developers working on protection that is unique and probably regularly updated. This might be a lot of work for an open source project to work around

[ebilg]

It's possible that they're trying to stop actual bots and they're blocking calibre as just a bonus for them too.