Thread: Support for HTTP 308 redirects
View Single Post
Old 09-13-2023, 08:47 AM   #1
modified
Connoisseur
modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!modified is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!
 
Posts: 90
Karma: 50742
Join Date: Jan 2011
Device: PW5
Support for HTTP 308 redirects
Recipe fails when the url responds with a HTTP308.

Sample recipe below

Code:
from calibre.web.feeds.news import BasicNewsRecipe


class Http308RedirectRecipe(BasicNewsRecipe):
    title = "Http 308 Redirect Recipe"
    language = "en"

    def parse_index(self):
        return [
            (
                "Example",
                [
                    {
                        "url": "https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530",
                        "title": "This url responds with a HTTP 308 redirect",
                    }
                ],
            ),
        ]
Error log:
Code:
Fetching https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530
Could not fetch link https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530
Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 278, in fetch_url
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 308: Permanent Redirect
curl response:
Code:
$ curl -A 'Mozilla/5.0' -Ii 'https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530'
HTTP/2 308 
content-length: 0
location: https://www.wsj.com/politics/mccarthy-biden-impeachment-inquiry-b9cc6530
date: Wed, 13 Sep 2023 12:46:09 GMT
x-proxy-cache: BYPASS
x-cache: Miss from cloudfront
via: 1.1 53b2bbb13e5db590d598ee4e9aa9bd80.cloudfront.net (CloudFront)
x-amz-cf-pop: HKG62-C2
x-amz-cf-id: mtOFJ7gK44Weycg7uPApuysAAakLzPg2kojmOCpiNpVi_Yk1bP7qZg==
modified is offline   Reply With Quote