The current Calibre recipe for The Economist no longer fetches articles – it fails on the index with a 403 from CloudFront, even when trying the GraphQL fallback. After some investigation, I found that the official Android app uses a completely different GraphQL endpoint and client identification headers that are still accepted.
I’ve written a standalone Python script that successfully pulls the entire weekly edition, downloads images, and builds a clean EPUB. The script doesn’t replace the Calibre recipe, but it demonstrates a method that could be adapted back into a recipe. I’m sharing it here so the community can see the approach and, if someone is inclined, update the built‑in recipe (or just use the script directly).
Key differences that make it work:
1. GraphQL endpoint
Instead of the endpoint the current recipe uses, this script calls:
Code:
https://cp2-graphql-gateway.p.aws.economist.com/graphql
with the query variables exactly as the Android app sends them.
2. Custom headers
It mimics the official Economist Android app (version 4.40.0) by setting:
Code:
User-Agent: TheEconomist-Liskov-android
accept: multipart/mixed; deferSpec=20220824, application/json
x-economist-consumer: TheEconomist-Liskov-android
x-teg-client-name: Economist-Android
x-teg-client-os: Android
x-teg-client-version: 4.40.0
These headers bypass the CloudFront block.
3. Fallback to web scraping
When GraphQL isn’t necessary or to supplement missing data, the script fetches the article page with a mobile User‑Agent (Liskov suffix) and extracts the article body from __NEXT_DATA__ or direct HTML, which also still works.
4. Robust retry / session handling
The script uses a requests.Session with a retry adapter, parallel downloads for articles and images, and proper error handling – all of which improve reliability.
Current limitation:
- Articles that rely heavily on interactive content (infographics, dynamic charts, JavaScript‑driven embeds) cannot be fully captured. The script only extracts static HTML/JSON, so interactive elements are either missing or appear as empty placeholders. If the edition contains such articles, they will not render properly in the resulting EPUB.
Disclaimer:
This is shared for educational purposes and personal use only. It’s not intended to overload the servers or to redistribute copyrighted material. Please respect The Economist’s terms of service.