Economist failed after upgrade to 9.10

[xkcklzn]

The current Calibre recipe for The Economist no longer fetches articles – it fails on the index with a 403 from CloudFront, even when trying the GraphQL fallback. After some investigation, I found that the official Android app uses a completely different GraphQL endpoint and client identification headers that are still accepted.

I’ve written a standalone Python script that successfully pulls the entire weekly edition, downloads images, and builds a clean EPUB. The script doesn’t replace the Calibre recipe, but it demonstrates a method that could be adapted back into a recipe. I’m sharing it here so the community can see the approach and, if someone is inclined, update the built‑in recipe (or just use the script directly).

Key differences that make it work:

1. GraphQL endpoint
Instead of the endpoint the current recipe uses, this script calls:

https://cp2-graphql-gateway.p.aws.economist.com/graphql

with the query variables exactly as the Android app sends them.

2. Custom headers
It mimics the official Economist Android app (version 4.40.0) by setting:

User-Agent: TheEconomist-Liskov-android
accept: multipart/mixed; deferSpec=20220824, application/json
x-economist-consumer: TheEconomist-Liskov-android
x-teg-client-name: Economist-Android
x-teg-client-os: Android
x-teg-client-version: 4.40.0
These headers bypass the CloudFront block.

3. Fallback to web scraping
When GraphQL isn’t necessary or to supplement missing data, the script fetches the article page with a mobile User‑Agent (Liskov suffix) and extracts the article body from __NEXT_DATA__ or direct HTML, which also still works.

4. Robust retry / session handling
The script uses a requests.Session with a retry adapter, parallel downloads for articles and images, and proper error handling – all of which improve reliability.

Current limitation:

• Articles that rely heavily on interactive content (infographics, dynamic charts, JavaScript‑driven embeds) cannot be fully captured. The script only extracts static HTML/JSON, so interactive elements are either missing or appear as empty placeholders. If the edition contains such articles, they will not render properly in the resulting EPUB.

Disclaimer:
This is shared for educational purposes and personal use only. It’s not intended to overload the servers or to redistribute copyrighted material. Please respect The Economist’s terms of service.

[kovidgoyal]

@xkcklzn that doesnt make any sense. Your script is using the exact same graphql endpoint the recipe uses it is probably copied directly from the recipe. And looking at the code its calling scrape_economist_edition() with use_web=True which means the graphql endpoint is not being used at all. The only reason your script works is that it uses a different browser type (requests versus mechanize and different user agent) and not enough people have used it, so the anti bot protection hasnt learned to fingerprint it. You shouldnt blindly trust what AI tells you.

[kovidgoyal]

I have changed the recipe to remove the graphql which is what was being blocked and additionally to use the android app user agent to fetch the article pages. that seems to be working for the moment, no idea how long it will survive.

[JSWolf]

Originally Posted by Cough

Here is the Mobi version of this weeks issue. Seems complete. Thanx Hongho

This is copyright material and should not be attached.

[NullNix]

Originally Posted by JSWolf

This is copyright material and should not be attached.

Agreed. If you want it, buy the Economist (and then read it on your e-reader using Calibre's recipe anyway, guilt-free). It's absolutely worth the money. I for one am extremely glad that Kovid's spent so much time keeping the recipe working: it's undoubtedly the bit of Calibre that I run (automatically) most frequently, and probably the bit I'd miss most if it stopped working (since I'm weird and don't hyper-classify my books: my bookshelves are a mess too).

[jwhittin]

Originally Posted by kovidgoyal

I have changed the recipe to remove the graphql which is what was being blocked and additionally to use the android app user agent to fetch the article pages. that seems to be working for the moment, no idea how long it will survive.

I just tried a download and got the following error:

Spoiler:

calibre, version 9.11.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The Economist

Fetch news from The Economist
Conversion options changed from defaults:
output_profile: 'kindle_pw3'
verbose: 2
Resolved conversion options
calibre version: 9.11.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000022D3556C830>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x0000022D3556F230>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:economist
Trying to get latest version of recipe: economist
Using user agent: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Recipe specific options:
res = 600
Failed to get index with UA: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Retrying with UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36
Using user agent: Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.103 Mobile Safari/537.36 Liskov
Falling back to GraphQL for the edition index
Failed to get index from GraphQL with error: <urlopen error [Errno 11001] getaddrinfo failed>
Traceback (most recent call last):
File "runpy.py", line 203, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 214, in main
File "calibre\gui2\convert\gui_conversion.py", line 32, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1087, in run
File "calibre\customize\conversion.py", line 241, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 153, in convert
File "calibre\web\feeds\news.py", line 1121, in download
File "calibre\web\feeds\news.py", line 1299, in build_index
File "<string>", line 333, in parse_index
File "<string>", line 299, in economist_return_index
NoArticles: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

This looks like the same error that I had before. So I'm not sure if I'm using the new recipe or the old one. It's my understanding that Calibre runs the latest recipe but perhaps I am wrong. Do I need to update Calibre manually to get the latest recipe?

Or has it been blocked already?

I wish they would provide subscribers with a Kobo/Kindle version so we didn't have to go through all of this...

[jwhittin]

Originally Posted by NullNix

I for one am extremely glad that Kovid's spent so much time keeping the recipe working: it's undoubtedly the bit of Calibre that I run (automatically) most frequently, and probably the bit I'd miss most if it stopped working

+1 for me

[haha123]

I'm using the latest recipe, but I still had to comment out # browser_type = 'webengine' to make it work. Could it be that Qt WebEngine has a very distinct "browser fingerprint" that makes it easily recognized and blocked by advanced anti-bot systems?

[kovidgoyal]

no as far as I can tell its just that the anti bot systems pick up on any particular engine's fingerprint after many people use it.

[eastonfmly]

I am still not able to use it. Errors out with the NoArticles: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

Are you saying that if too many folks use your recipe the ai bot at the economist stops it?

Thanks Jim

[Kman]

The Economist no longer supports Kindle subscriptions. I'm not a skilled techie. If I subscribe to The Economist, how can I generate a MOBI or EPUB file for downloading to my Kindle?

[eastonfmly]

Originally Posted by Kman

The Economist no longer supports Kindle subscriptions. I'm not a skilled techie. If I subscribe to The Economist, how can I generate a MOBI or EPUB file for downloading to my Kindle?

I dont know how. I have been using Calibre back when the only way was to pay amazon $3 a week in addition to your subscription. I have put suggestions in so many times, if this cant be worked out, I will probably end it. I dont want to use a tablet and the app on the phone is a pain to read. The kindle paperwrite was perfect, except maybe for some of the graphics and KAL.

[Kman]

Originally Posted by NullNix

Agreed. If you want it, buy the Economist (and then read it on your e-reader using Calibre's recipe anyway, guilt-free). It's absolutely worth the money. I for one am extremely glad that Kovid's spent so much time keeping the recipe working: it's undoubtedly the bit of Calibre that I run (automatically) most frequently, and probably the bit I'd miss most if it stopped working (since I'm weird and don't hyper-classify my books: my bookshelves are a mess too).

If I subscribe to The Economist, how can I generate an EPUB or MOBI file for downloading to my Kindle? From what I can tell, you either read The Economist while signed in on a browser or you download it to their Android app in a proprietary format.

[NullNix]

Originally Posted by Kman

If I subscribe to The Economist, how can I generate an EPUB or MOBI file for downloading to my Kindle? From what I can tell, you either read The Economist while signed in on a browser or you download it to their Android app in a proprietary format.

I wonder if we can somehow leverage browser automation (there's a standard thing for it whose name I forget) to do the downloading part using the actual browser, signed-in and everything? Would be a gigantic change though, basically a rewrite, but it would probably be very hard for the Economist to do anything to stop other than rate-limiting, because we'd literally be using the same machinery to read that other users are.

[Cough]

Originally Posted by JSWolf

This is copyright material and should not be attached.

After I posted I sent a note to the mods asking if I was in contravention but they didn't respond. Will remove if I can but cant seem to edit it.