Atlantic recipe results in python error

[thisbook]

Hello,

I'm getting a python conversion error when trying to fetch news from The Atlantic. It works for other sites (eg. The New Yorker) but for some reason fails with The Atlantic stock recipe.

Spoiler:

calibre, version 1.47.0 (win32, isfrozen: True)
Conversion Error: Failed: Fetch news from The Atlantic

Fetch news from The Atlantic
Resolved conversion options
calibre version: 1.47.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000000000484F860>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutp ut object at 0x000000000484FEB8>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Python function terminated unexpectedly
'NoneType' object has no attribute 'find' (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 195, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 1040, in run
File "site-packages\calibre\customize\conversion.py", line 241, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\recipe_ input.py", line 117, in convert
File "site-packages\calibre\web\feeds\news.py", line 992, in download
File "site-packages\calibre\web\feeds\news.py", line 1157, in build_index
File "<string>", line 38, in parse_index
AttributeError: 'NoneType' object has no attribute 'find'

Happy to provide any other information to help this.

Thanks!

[kovidgoyal]

The Atlantic redesigned their website, again. I've updated the recipe, just re-run the download.

[NSILMike]

Originally Posted by kovidgoyal

The Atlantic redesigned their website, again. I've updated the recipe, just re-run the download.

I tried this recipe, and it times out (and therefore fails) at 45 minutes! Is that expected?

[kovidgoyal]

A timeout is a server issue, either the atlantic server is having issues or they have added some kind of IP blocker to prevent download of articles.

[NSILMike]

Originally Posted by kovidgoyal

A timeout is a server issue, either the atlantic server is having issues or they have added some kind of IP blocker to prevent download of articles.

Understood. However, in the past I've had to increase the time limit for the New York Times (up to around 20 minutes I think...) to get past the same issue. In this case it just seems unreasonably long. And when you updated the recipe did you see the same thing?

[kovidgoyal]

No, it worked for me.

[NSILMike]

Originally Posted by kovidgoyal

No, it worked for me.

Then I am left wondering why it takes so long for me (it repeatedly takes more than 40 minutes.) My internet connection is reasonably fast (8-10MB/s.) Is there another Calibre setting that I need to tweak?

[kovidgoyal]

No, no settings influence the speed of download. The recipe is fairly slow, because the atlantic servers are slow and the articles have a lot of large images, but it does work.

[NSILMike]

Originally Posted by kovidgoyal

No, no settings influence the speed of download. The recipe is fairly slow, because the atlantic servers are slow and the articles have a lot of large images, but it does work.

Hmmm. OK. I'll set it up very high and see what happens. FYI, it's not data intensive of course, but I can surf their website easily with no noticeable page load delays.
Edit: one item I've noticed (NY Times... The Atlantic never made it this far) needed substantial time to run the MOBI Output Plugin- seems related to the size of the download.)
Edit 2: failed at 55 minutes- only got to ~20%. now trying 75.
Edit 3: failed at 75... I give up.

[thisbook]

Thankyou for fizing the recipe, kovidgoyal. Appreciate all your effort in Calibre.

I'm also facing the timeout issues, can anyone suggest a method to get around this?

[NSILMike]

Originally Posted by thisbook

Thank you for fizing the recipe, kovidgoyal. Appreciate all your effort in Calibre.

I'm also facing the timeout issues, can anyone suggest a method to get around this?

If you saw my last edit (above) you know I timed out all the way up to 75 minutes. That's crazy. My internet speed is high, and my latency is quite good, so I have no idea how a user in India can use that recipe, and someone (me) in the same country as the source cannot. What's your location by the way?

[nickredding]

Downloading entire publications is time consuming and obsolete. The always-on internet makes that unnecessary.

[PeterT]

@nickredding: It's not feasible for all of us to have internet access whereever and whenever we want. Some of us far FAR prefer the ability to download and access material when we are off net.

[kovidgoyal]

This should take care of it:

https://github.com/kovidgoyal/calibr...3ca7fef8a43451

[PeterT]

WOW... It made quite a difference! I had tried earlier and saw the exact same issue as the others had. Now; download is done in seconds. AMAZING