Economist recipe failing with 403 error

adfadfsasdfafafd · 12-08-2023, 02:25 AM

I'm consistently getting this error today (and I've tried from a couple of different IPs):

Quote:

Downloading recipe urn: builtin:economist
Trying to get latest version of recipe: economist
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Kindle Output profile being used, reducing image quality to keep file size below amazon email threshold
Traceback (most recent call last):
File "runpy.py", line 194, in _run_module_as_main
File "runpy.py", line 87, in _run_code
File "site.py", line 39, in <module>
File "site.py", line 35, in main
File "calibre/utils/ipc/worker.py", line 215, in main
File "calibre/gui2/convert/gui_conversion.py", line 31, in gui_convert_recipe
File "calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
File "calibre/ebooks/conversion/plumber.py", line 1108, in run
File "calibre/customize/conversion.py", line 242, in __call__
File "calibre/ebooks/conversion/plugins/recipe_input.py", line 137, in convert
File "calibre/web/feeds/news.py", line 1056, in download
File "calibre/web/feeds/news.py", line 1225, in build_index
File "<string>", line 280, in parse_index
File "calibre/web/feeds/news.py", line 706, in index_to_soup
File "mechanize/_mechanize.py", line 241, in open_novisit
File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 403: Forbidden

unkn0wn · 12-08-2023, 02:31 AM

its checking if connection is secure.

kovidgoyal · 12-08-2023, 02:51 AM

Quote:

Originally Posted by unkn0wn

its checking if connection is secure.

What it's really doing is checking if you are connecting using a browser blessed by cloudflare's "bot protection". Which means you cannot connect anymore without using a browser running javascript that happens to pass a bunch of heuristics implemented in cloudflare's JS script.

"Secure" indeed, roll eyes.

adfadfsasdfafafd · 12-08-2023, 02:53 AM

Is there a way around this?

unkn0wn · 12-08-2023, 03:21 AM

can we get __NEXT_DATA__ json through waybackmachine? like nyt!

archive.is/latest/ doesn't have it

Code:

    def print_version(self, url):
        return 'https://webcache.googleusercontent.com/search?q=cache:' + url.split('?')[0]

this one works but has rate limit.

kovidgoyal · 12-08-2023, 03:32 AM

If the google webcache works you can use it via google_get_cached_url() function that implements rate limiting globally across all calibre process/threads. But I suspect google's web cache has both rate and total requests limits.

unkn0wn · 12-08-2023, 04:11 AM

fixed. it'll work now.

adfadfsasdfafafd · 12-08-2023, 12:57 PM

Amazing, thank you!

12-08-2023, 02:31 AM	#2
unkn0wn Guru Posts: 645 Karma: 85520 Join Date: May 2021 Device: kindle	its checking if connection is secure. Attached Thumbnails

12-08-2023, 03:21 AM	#5
unkn0wn Guru Posts: 645 Karma: 85520 Join Date: May 2021 Device: kindle	can we get __NEXT_DATA__ json through waybackmachine? like nyt! archive.is/latest/ doesn't have it Code: def print_version(self, url): return 'https://webcache.googleusercontent.com/search?q=cache:' + url.split('?')[0] this one works but has rate limit. Last edited by unkn0wn; 12-08-2023 at 03:29 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
HTTP Error 403: Forbidden	enuddleyarbl	Calibre	13	07-08-2024 05:48 PM
The Economist Recipe	NSILMike	Recipes	6	05-20-2016 08:40 AM
"The New York Times" recipe failing with error	mikebw	Recipes	8	10-02-2015 06:48 PM
Economist Recipe Failing...	awitko	Recipes	2	11-07-2011 12:47 AM
Economist recipe	jdomingos76	Recipes	1	03-08-2011 09:33 AM

12-08-2023, 02:53 AM	#4
adfadfsasdfafafd Enthusiast Posts: 27 Karma: 76 Join Date: May 2014 Device: Kindle 3	Is there a way around this?

12-08-2023, 03:32 AM	#6
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	If the google webcache works you can use it via google_get_cached_url() function that implements rate limiting globally across all calibre process/threads. But I suspect google's web cache has both rate and total requests limits.

12-08-2023, 04:11 AM	#7
unkn0wn Guru Posts: 645 Karma: 85520 Join Date: May 2021 Device: kindle	fixed. it'll work now.

12-08-2023, 12:57 PM	#8
adfadfsasdfafafd Enthusiast Posts: 27 Karma: 76 Join Date: May 2014 Device: Kindle 3	Amazing, thank you!

Advert

Advert