Economist failed after upgrade to 9.10

[hongho71]

When I added The Economist epub into calibre from the github website, the author shows as Kovid Goyal attached to the epub which means that person is somehow successful in modifying an economist recipe in calibre to download the articles?

[xkcklzn]

The current Calibre recipe for The Economist no longer fetches articles – it fails on the index with a 403 from CloudFront, even when trying the GraphQL fallback. After some investigation, I found that the official Android app uses a completely different GraphQL endpoint and client identification headers that are still accepted.

I’ve written a standalone Python script that successfully pulls the entire weekly edition, downloads images, and builds a clean EPUB. The script doesn’t replace the Calibre recipe, but it demonstrates a method that could be adapted back into a recipe. I’m sharing it here so the community can see the approach and, if someone is inclined, update the built‑in recipe (or just use the script directly).

Key differences that make it work:

1. GraphQL endpoint
Instead of the endpoint the current recipe uses, this script calls:

https://cp2-graphql-gateway.p.aws.economist.com/graphql

with the query variables exactly as the Android app sends them.

2. Custom headers
It mimics the official Economist Android app (version 4.40.0) by setting:

User-Agent: TheEconomist-Liskov-android
accept: multipart/mixed; deferSpec=20220824, application/json
x-economist-consumer: TheEconomist-Liskov-android
x-teg-client-name: Economist-Android
x-teg-client-os: Android
x-teg-client-version: 4.40.0
These headers bypass the CloudFront block.

3. Fallback to web scraping
When GraphQL isn’t necessary or to supplement missing data, the script fetches the article page with a mobile User‑Agent (Liskov suffix) and extracts the article body from __NEXT_DATA__ or direct HTML, which also still works.

4. Robust retry / session handling
The script uses a requests.Session with a retry adapter, parallel downloads for articles and images, and proper error handling – all of which improve reliability.

Current limitation:

• Articles that rely heavily on interactive content (infographics, dynamic charts, JavaScript‑driven embeds) cannot be fully captured. The script only extracts static HTML/JSON, so interactive elements are either missing or appear as empty placeholders. If the edition contains such articles, they will not render properly in the resulting EPUB.

Disclaimer:
This is shared for educational purposes and personal use only. It’s not intended to overload the servers or to redistribute copyrighted material. Please respect The Economist’s terms of service.

[kovidgoyal]

@xkcklzn that doesnt make any sense. Your script is using the exact same graphql endpoint the recipe uses it is probably copied directly from the recipe. And looking at the code its calling scrape_economist_edition() with use_web=True which means the graphql endpoint is not being used at all. The only reason your script works is that it uses a different browser type (requests versus mechanize and different user agent) and not enough people have used it, so the anti bot protection hasnt learned to fingerprint it. You shouldnt blindly trust what AI tells you.

[kovidgoyal]

I have changed the recipe to remove the graphql which is what was being blocked and additionally to use the android app user agent to fetch the article pages. that seems to be working for the moment, no idea how long it will survive.

[JSWolf]

Originally Posted by Cough

Here is the Mobi version of this weeks issue. Seems complete. Thanx Hongho

This is copyright material and should not be attached.

[NullNix]

Originally Posted by JSWolf

This is copyright material and should not be attached.

Agreed. If you want it, buy the Economist (and then read it on your e-reader using Calibre's recipe anyway, guilt-free). It's absolutely worth the money. I for one am extremely glad that Kovid's spent so much time keeping the recipe working: it's undoubtedly the bit of Calibre that I run (automatically) most frequently, and probably the bit I'd miss most if it stopped working (since I'm weird and don't hyper-classify my books: my bookshelves are a mess too).

[jwhittin]

Originally Posted by kovidgoyal

I have changed the recipe to remove the graphql which is what was being blocked and additionally to use the android app user agent to fetch the article pages. that seems to be working for the moment, no idea how long it will survive.

I just tried a download and got the following error:

Spoiler:

calibre, version 9.11.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The Economist

Fetch news from The Economist
Conversion options changed from defaults:
output_profile: 'kindle_pw3'
verbose: 2
Resolved conversion options
calibre version: 9.11.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000022D3556C830>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x0000022D3556F230>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:economist
Trying to get latest version of recipe: economist
Using user agent: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Recipe specific options:
res = 600
Failed to get index with UA: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Retrying with UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36
Using user agent: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Falling back to GraphQL for the edition index
Failed to get index from GraphQL with error: <urlopen error [Errno 11001] getaddrinfo failed>
Traceback (most recent call last):
File "runpy.py", line 203, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 214, in main
File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1087, in run
File "calibre\customize\conversion.py", line 241, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 153, in convert
File "calibre\web\feeds\news.py", line 1121, in download
File "calibre\web\feeds\news.py", line 1299, in build_index
File "<string>", line 333, in parse_index
File "<string>", line 299, in economist_return_index
NoArticles: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

This looks like the same error that I had before. So I'm not sure if I'm using the new recipe or the old one. It's my understanding that Calibre runs the latest recipe but perhaps I am wrong. Do I need to update Calibre manually to get the latest recipe?

Or has it been blocked already?

I wish they would provide subscribers with a Kobo/Kindle version so we didn't have to go through all of this...

[jwhittin]

Originally Posted by NullNix

I for one am extremely glad that Kovid's spent so much time keeping the recipe working: it's undoubtedly the bit of Calibre that I run (automatically) most frequently, and probably the bit I'd miss most if it stopped working

+1 for me

[haha123]

I'm using the latest recipe, but I still had to comment out # browser_type = 'webengine' to make it work. Could it be that Qt WebEngine has a very distinct "browser fingerprint" that makes it easily recognized and blocked by advanced anti-bot systems?

[kovidgoyal]

no as far as I can tell its just that the anti bot systems pick up on any particular engine's fingerprint after many people use it.