Arts and Letters Daily failing

[DaySleeper]

My last successful fetch was October 24. It's encountering a 403 error, but it doesn't indicate what site it's trying to access that produces the error. It looks like it's trying to download an updated recipe from somewhere (it doesn't say where), and that's failing, as opposed to the website where the actual information resides? I haven't attempted to create or closely follow any of the news fetch recipes, so don't know what to look for. I just use the ones that Calibre offers me, and mostly they work. Here's the log:

Fetch news from Arts and Letters Daily
Conversion options changed from defaults:
verbose: 2
output_profile: 'tablet'
Resolved conversion options
calibre version: 5.7.2
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x055C7208>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.TabletOutput object at 0x055C7688>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:ald
Trying to get latest version of recipe: ald
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Traceback (most recent call last):
File "runpy.py", line 194, in _run_module_as_main
File "runpy.py", line 87, in _run_code
File "site.py", line 82, in <module>
File "site.py", line 77, in main
File "site.py", line 49, in run_entry_point
File "calibre\utils\ipc\worker.py", line 216, in main
File "calibre\gui2\convert\gui_conversion.py", line 34, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 28, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1108, in run
File "calibre\customize\conversion.py", line 245, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 139, in convert
File "calibre\web\feeds\news.py", line 1051, in download
File "calibre\web\feeds\news.py", line 1220, in build_index
File "<string>", line 34, in parse_index
File "calibre\web\feeds\news.py", line 702, in index_to_soup
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 403: Forbidden

[kovidgoyal]

It's bot detection on the ALD server, probably using HTTPS signature.

[DaySleeper]

Do you have an idea how I could repair it or a work-around? I'm using this as a built-in recipe, and don't really know how to troubleshoot.

[kovidgoyal]

The recipe will need to be rewritten to workaround the bot detection. How to do that will depend on what exactly the bot detection is.

[unkn0wn]

add this to the recipe

    def get_browser(self): 
        return BasicNewsRecipe.get_browser(self, user_agent='common_words/based')

[DaySleeper]

Originally Posted by unkn0wn

add this to the recipe

    def get_browser(self): 
        return BasicNewsRecipe.get_browser(self, user_agent='common_words/based')

Thanks! I replaced the previous version of that statement, which had two additional arguments, and that did the trick.