Support for HTTP 308 redirects

[modified]

Recipe fails when the url responds with a HTTP308.

Sample recipe below

from calibre.web.feeds.news import BasicNewsRecipe


class Http308RedirectRecipe(BasicNewsRecipe):
    title = "Http 308 Redirect Recipe"
    language = "en"

    def parse_index(self):
        return [
            (
                "Example",
                [
                    {
                        "url": "https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530",
                        "title": "This url responds with a HTTP 308 redirect",
                    }
                ],
            ),
        ]

Error log:

Fetching https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530
Could not fetch link https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530
Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 278, in fetch_url
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 308: Permanent Redirect

curl response:

$ curl -A 'Mozilla/5.0' -Ii 'https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530'
HTTP/2 308 
content-length: 0
location: https://www.wsj.com/politics/mccarthy-biden-impeachment-inquiry-b9cc6530
date: Wed, 13 Sep 2023 12:46:09 GMT
x-proxy-cache: BYPASS
x-cache: Miss from cloudfront
via: 1.1 53b2bbb13e5db590d598ee4e9aa9bd80.cloudfront.net (CloudFront)
x-amz-cf-pop: HKG62-C2
x-amz-cf-id: mtOFJ7gK44Weycg7uPApuysAAakLzPg2kojmOCpiNpVi_Yk1bP7qZg==

[kovidgoyal]

Your recipe worked fine for me in current calibre and the recipe system most definitely handles redirects. From the use of curl I am guessing you are on linux. Dont use whatever distro calibre package use the official binaries.

[modified]

Apologies for cutting too much of the log. I'm running the official 6.26.0 on macOS.

I dug around and it looks like it's because calibre is pinned to mechanize v0.4.7 but support for HTTP308 redirects is available only from v0.4.8 (commit).

$ ebook-convert 'http308.recipe' .epub --test --debug-pipeline debug -vv
Conversion options changed from defaults:
  test: (2, 2)
  debug_pipeline: 'debug'
  verbose: 2
Resolved conversion options
calibre version: 6.26.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': 'debug',
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_max_image_size': 'none',
 'epub_toc_at_end': False,
 'epub_version': '2',
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x118f1fbe0>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.OutputProfile object at 0x118f1ead0>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': (2, 2),
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'transform_html_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
1% Converting input to HTML...
InputFormatPlugin: Recipe Input running
Using custom recipe
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
1% Fetching feeds...
1% Got feeds from index page
1% Trying to download cover...
1% Generating masthead...
Synthesizing mastheadImage
1% Starting download [4 threads]...
Fetching https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530
Could not fetch link https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530
Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 278, in fetch_url
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 308: Permanent Redirect

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/web/fetch/simple.py", line 536, in process_links
  File "calibre/web/fetch/simple.py", line 283, in fetch_url
calibre.web.fetch.simple.FetchError: Permanent Redirect

https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530 saved to 
Failed to download article: HTTP 308 Direct from https://www.wsj.com/articles/mccarthy-biden-impeachment-inquiry-b9cc6530
Traceback (most recent call last):
  File "calibre/utils/threadpool.py", line 99, in run
  File "calibre/web/feeds/news.py", line 1195, in fetch_article
  File "calibre/web/feeds/news.py", line 1190, in _fetch_article
Exception: Could not fetch article. The debug traceback is available earlier in this log

[kovidgoyal]

https://github.com/kovidgoyal/calibr...ea3cba39b27598

[xiatian]

I'm having the same issue. I use the latest calibre binary (6.26.0) on linux but still had the same error "HTTP Error 308: Permanent Redirect" when converting wsj recipe.

[kovidgoyal]

yes why wouldnt you, since the fix has not yet been released.

[unkn0wn]

looks like WSJ articles wont be loading text even with the redirect fix. The amp version of the links have stopped loading content.

maybe we are back to trying this
https://www.mobileread.com/forums/sh...4&postcount=17

EDIT:
oh wait is it going to start working from the next update after the redirect is fixed?
https://github.com/unkn0w7n/calibre/...f7c539a973af9f

[kovidgoyal]

I am still getting some content in wsj free I dont subscribe so cant try the main recipe. See https://github.com/kovidgoyal/calibr...5305f9a1b4e721

[kovidgoyal]

Originally Posted by unkn0wn

oh wait is it going to start working from the next update after the redirect is fixed?
https://github.com/unkn0w7n/calibre/...f7c539a973af9f

It should but I need someone's WSJ subscription username/password to really test it.

[Hellman101]

Originally Posted by kovidgoyal

It should but I need someone's WSJ subscription username/password to really test it.

Kovid - have you been able to test? If not, I can lend you my creds to test.

[kovidgoyal]

No no one has sent me the creds, PM them to me.

[xiatian]

Originally Posted by kovidgoyal

No no one has sent me the creds, PM them to me.

Sent! You can test it.

[kovidgoyal]

I tested it and sadly only the headlines, hero image and first para are present, as I said before the rest of the content is transmitted encrypted and decrypted on client. Looks like the AMP loophole @unknown found no longer works.