Economist failed after upgrade to 9.10

[PacificNW]

Originally Posted by NullNix

I wonder if we can somehow leverage browser automation (there's a standard thing for it whose name I forget) to do the downloading part using the actual browser, signed-in and everything? Would be a gigantic change though, basically a rewrite, but it would probably be very hard for the Economist to do anything to stop other than rate-limiting, because we'd literally be using the same machinery to read that other users are.

Really beyond the concept of pure python scripting as done in Calibre recipes, but very complex web client-server interactions can be handled by the 'deno' tool. The excellent product 'yt-dlp' recently adopted it to deal with massive complexity on YouTube. Works great, but there's still an ongoing cat-and-mouse aspect to reverse engineering here, which our maintainer is clearly annoyed by. IF interested, an Example

[theducks]

Originally Posted by Cough

After I posted I sent a note to the mods asking if I was in contravention but they didn't respond. Will remove if I can but cant seem to edit it.

Many Mods are volunteers. We check messages when we have a chance.
Forum levels (like myself), only get notices by specific PM. To whome did you send the request?
The Greenies all get notices when 'report this post' is used (when needed rapid action)

[Cough]

just delete my post #28 please.

[P38eagle]

I changed my version from 9.11 to 7.26 and was able to download the Economst.

[kovidgoyal]

Originally Posted by P38eagle

I changed my version from 9.11 to 7.26 and was able to download the Economst.

And if lots of people do that it will stop working.

[NullNix]

Originally Posted by PacificNW

Really beyond the concept of pure python scripting as done in Calibre recipes, but very complex web client-server interactions can be handled by the 'deno' tool. The excellent product 'yt-dlp' recently adopted it to deal with massive complexity on YouTube. Works great, but there's still an ongoing cat-and-mouse aspect to reverse engineering here, which our maintainer is clearly annoyed by. IF interested, an Example

Surely that's where https://pypi.org/project/deno/ comes in handy. But the thing I was thinking of was https://pypi.org/project/marionette-harness/ or of course https://pypi.org/project/selenium/.

[kovidgoyal]

No the solution for this kind of thing is camoufox. And there is already code in calibre to start integrating it. However camoufox is huge several hundred MB and getting the integration to a point where its useable in recipes etc in calibre is a lot of work. https://github.com/daijro/camoufox

[Emagazine2026]

Originally Posted by kovidgoyal

No the solution for this kind of thing is camoufox. And there is already code in calibre to start integrating it. However camoufox is huge several hundred MB and getting the integration to a point where its useable in recipes etc in calibre is a lot of work. https://github.com/daijro/camoufox

Hi Kovid, Using your updated recipe, I can only fetch the section names and article lists from The Economist, but the full article content cannot be retrieved. What could be the reason for this? Thanks.

[kovidgoyal]

Repeating myself one last time. The economist has moved to using statistical fingerprint based blocking. Any changes you make to the recipe will cause it to work *for some time* until enough people use the change, at which point it will stop working again. The *only* way to fix this robustly is to use a browser engine that is impossible to statistically isolate from regular browsers. As I said above one possibility for that is camoufox. If and when I find the time to finish its integration with calibre this can be re visited until then you can keep making endless tweaks to the recipe to have the download work for yourself personally, such as changing the user agent or the browser stack etc. But any and all such changes will not work if they are used for the builtin recipe which is used by large numbers of people thereby providing an easily detected signal. For a while we got away with using the endpoint used by the economist android app, but since last week the economist has blocked that as well.

[jwhittin]

And today, for some reason, the Calibre recipe worked for me.

[kovidgoyal]

Originally Posted by jwhittin

And today, for some reason, the Calibre recipe worked for me.

There was an update to it to use the current android app UA which has changed. Lets see how long it survives.

[hilihuo]

great, the economist works this week, thanks a lot, but Scientific American is not working for two weeks: unable to find issue link