The Atlantic is failing now

[mr316]

Just started yesterday, throws a 403 error immediately. Are they requiring Javascript now, too?

...
File "calibre/web/feeds/news.py", line 755, in index_to_soup
File "mechanize/_mechanize.py", line 241, in open_novisit
File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 403: Forbidden

[JSWolf]

It seems a lot of recopies are going to fail because the sites are being protected.

[kovidgoyal]

Update to the preview release and it should work. Yes it has switched to needing a real browser to do the downloads as well, which the preview release provides. https://www.mobileread.com/forums/sh...d.php?t=359648

[mr316]

Thanks Kovid.

Is there a way to force the preview build using the installer script? This is what I use on Fedora -

sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.sh | sudo sh /dev/stdin

[kovidgoyal]

Use the manual install instructions at https://calibre-ebook.com/download_linux

[Kman]

Calibre's recipe for TheAtlantic.com is only fetching section headings. No articles in the Table of Contents and no articles. Screenshot attached.

[gc886]

Both recipes for The Atlantic worked for me again today and I'm running the older 8.7 version. I experienced the mentioned 403 problem previously too. I wonder if The Atlantic had done away with their protection.

[Kman]

I updated to 9.16, which I believe should automatically download and use Camoufox for fetching TheAtlantic.com.

I tried that but it failed. I immediately get a Camoufox popup with the heading "Firefox" and message "Couldn't load XPCOM". Details below.

Spoiler:

Fetch news from TheAtlantic.com
Conversion options changed from defaults:
verbose: 2
output_profile: 'kindle_pw3'
Resolved conversion options
calibre version: 9.16.0
{'add_alt_text_to_img': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000001EED2C867B0>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x000001EED2C912B0>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'recipe_specific_option': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:atlantic_com
Trying to get latest version of recipe: atlantic_com
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36
Ignoring the requested user agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36', camoufox uses one that matches the rest of its fingerprint
Warming up camoufox on: https://www.bbc.com/news, https://www.reddit.com/r/funny
Traceback (most recent call last):
File "runpy.py", line 203, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 186, in main
File "calibre\gui2\convert\gui_conversion.py", line 49, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 36, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1258, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 162, in convert
File "calibre\web\feeds\news.py", line 1015, in __init__
File "<string>", line 147, in get_browser
File "calibre\web\automate\recipes.py", line 937, in set_simple_cookie
File "calibre\web\automate\recipes.py", line 701, in request
urllib.error.URLError: <urlopen error The camoufox browser failed: The camoufox browser failed to start: The browser process exited
Browser log:
>

[kovidgoyal]

Works fineonmy windows system. That error indicates some dll problem, typically with a browser its antivirus having quarantined or deleted some dll or a system level dll conflict. Try the following

1. Check whether xul.dll, mozglue.dll, nss3.dll and lgpllibs.dll are all present in %LOCALAPPDATA%\calibre-dbin\camoufox\156.0.1-beta.36\.
2. Look in their antivirus quarantine or history for anything from that folder.
3. Double-click camoufox.exe in that folder. If the same popup appears, calibre isn't involved.
4. Delete %LOCALAPPDATA%\calibre-dbin\camoufox to force a fresh download, ideally after adding an antivirus exclusion for calibre-dbin.

[mr316]

Just want to add that 9.16 is working perfect for me on Fedora now, the Atlantic came through perfectly. Thanks Kovid!

[Kman]

Originally Posted by kovidgoyal

Works fineonmy windows system. That error indicates some dll problem, typically with a browser its antivirus having quarantined or deleted some dll or a system level dll conflict. Try the following

1. Check whether xul.dll, mozglue.dll, nss3.dll and lgpllibs.dll are all present in %LOCALAPPDATA%\calibre-dbin\camoufox\156.0.1-beta.36\.
2. Look in their antivirus quarantine or history for anything from that folder.
3. Double-click camoufox.exe in that folder. If the same popup appears, calibre isn't involved.
4. Delete %LOCALAPPDATA%\calibre-dbin\camoufox to force a fresh download, ideally after adding an antivirus exclusion for calibre-dbin.

All those .dll files are in calibre-dbin\camoufox\156.0.1-beta.36\. When I double-click camoufox.exe an empty Firefox window opens. I didn't see any quarantined files in Window Security or Malwarebytes.

I'll mess around with my security settings to see if I can fix this, but so far everything that's supposed to be there is there and camoufox.exe does open a Firefox window.

[Kman]

Originally Posted by Kman

All those .dll files are in calibre-dbin\camoufox\156.0.1-beta.36\. When I double-click camoufox.exe an empty Firefox window opens. I didn't see any quarantined files in Window Security or Malwarebytes.

I'll mess around with my security settings to see if I can fix this, but so far everything that's supposed to be there is there and camoufox.exe does open a Firefox window.

Weird. I added the calibre-dbin folder to the "Allow" list in Malwarebytes and fetching The Atlantic worked. I then removed that exclusion and fetching The Atlantic worked again.

So, I'm confused whether it was a security issue. Or perhaps just double-clicking camoufox.exe (before adding a security exclusion) somehow initialized camoufox.exe from within Calibre? Maybe Firefox behaves differently than other browsers?

About another news source: There are no fields for entering subscription information in the recipe for The Economist. Is that in the works?

[DNSB]

Originally Posted by Kman

About another news source: There are no fields for entering subscription information in the recipe for The Economist. Is that in the works?

When I tryto schedule the Economist for download, the username/password fields were present. See the attached image.