Arcamax Comics recipe hangs without error

[KecajN]

Since a couple of weeks Arcamax Comics recipe usually hangs at the very beginning and does not complete. It goes through strips lists without any issues, enters "Synthesizing mastheadImage" steps at at some point stops working without any error returned. The most recent message today was:

Fetched http://arcamax.com/newspics/177/17733/1773307.gif in 1.654000 seconds

Sometimes the issue vanishes next day but it is not always the case. I guess that download procedure does not time out on server or network related error. Any ideas how can it be fixed?

[kovidgoyal]

Works for me, and the download process definitely has a timeout, one that can even be set in the reipe with timeout = whatever (the default is 120 seconds)

[KecajN]

Comic stripes downloaded yesterday, first time in two weeks. But download failed again today in exactly same manner. It hangs on this step since >20 minutes:

Processing images...
Fetching http://arcamax.com/newspics/177/17729/1772948.gif
Fetching http://arcamax.com/newspics/177/17731/1773133.gif
Fetching http://arcamax.com/newspics/177/17733/1773307.gif
Fetching http://arcamax.com/newspics/177/17734/1773442.gif
Fetched http://arcamax.com/newspics/177/17729/1772948.gif in 1.551000 seconds
Fetched http://arcamax.com/newspics/177/17727/1772715.gif in 1.587000 seconds

How can I troubleshoot this issue?

[kovidgoyal]

Run from source and instrument the download code, which is in fetch/simple.py

In paricular look in the fetch_url() function, see if the correct timeout is being passed using a print() and see what URL is hanging.

[KecajN]

My problem did not vanish entirely but usually I was able to download a comic ebook after restarting a download later. Now I have a little bit more time to work on debugging but I am not sure what you recommended. Could you be more specific?

I only run my recipe in accordance to instruction found in a Calibre manual:

ebook-convert "Arcamax Ltd_1002.recipe" .mobi --test -vv --debug-pipeline debug

It processed recipe up to creating final output but mobi file contained only 2 (out of 10) incomplete comic strips (2 rather than 7 days each). I have found the following error messages in the output:

Parsing all content...
Parsing feed_1/article_0/index.html ...
Parsing feed_0/article_0/index.html ...
Parsing index.html ...
Forcing index.html into XHTML namespace
Parsing feed_0/article_1/index.html ...
Parsing feed_1/index.html ...
Initial parse failed, using more forgiving parsers
Parsing feed_1/index.html as HTML
Parsing feed_0/index.html ...
Initial parse failed, using more forgiving parsers
Parsing feed_0/index.html as HTML
Parsing feed_1/article_1/index.html ...
Referenced file u'feed_2/index.html' not found
Referenced file u'/manifest.json' not found
Referenced file u'/thefunnies/dilbert/rss' not found
Referenced file u'feed_0/article_0/file%3a/thefunnies/dilbert/biography' not found
Referenced file u'feed_0/article_1/file%3a/thefunnies/dilbert/biography' not found
Referenced file u'feed_1/article_0/file%3a/thefunnies/bc/biography' not found
Referenced file u'/thefunnies/bc/rss' not found
Referenced file u'feed_1/article_1/file%3a/thefunnies/bc/biography' not found

Full output is attached.

[hegi]

Hi there,
got somewhat of a similar problem here.

I observed over the last ~1 year, that Arcamax downloads hangup maybe once in a month or 6 weeks. However, with a second or third try, it usually worked. - So I presumed the problem is server reachability or availability.

Since a few days the download simply hangs a couple of hours, before I terminate. Normally Arcamax takes 3 to 5 mins. max.

When I hit the kill button, I note, that the picture is similar to KecajN's post. The pics take extremely long to download.

See download log below.

Any idea, how to handle this?
Thanks
Hegi

Nachrichten abrufen von Arcamax
Failed to initialize plugin: '/home/hegi/.config/calibre/plugins/ANOBII.zip'
Failed to initialize plugin: '/home/hegi/.config/calibre/plugins/DeDRM.zip'
Conversion options changed from defaults:
verbose: 2
Resolved conversion options
calibre version: 5.8.1
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x7f44c33b8880>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': False,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.OutputProfile object at 0x7f44c33b8b50>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1016
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Finding strips for: 9 Chickweed Lane
Found: 9 Chickweed Lane for today at: http://www.arcamax.com/thefunnies/ninechickweedlane
Found: 9 Chickweed Lane for 12/26/2020 at: http://www.arcamax.com/thefunnies/ni...lane/s-2457577
Found: 9 Chickweed Lane for 12/25/2020 at: http://www.arcamax.com/thefunnies/ni...lane/s-2457270
Found: 9 Chickweed Lane for 12/24/2020 at: http://www.arcamax.com/thefunnies/ni...lane/s-2456968
Found: 9 Chickweed Lane for 12/23/2020 at: http://www.arcamax.com/thefunnies/ni...lane/s-2456558
Found: 9 Chickweed Lane for 12/22/2020 at: http://www.arcamax.com/thefunnies/ni...lane/s-2456073
Found: 9 Chickweed Lane for 12/21/2020 at: http://www.arcamax.com/thefunnies/ni...lane/s-2455597
Finding strips for: Arctic Circle
Found: Arctic Circle for today at: http://www.arcamax.com/thefunnies/arcticcircle
Found: Arctic Circle 12/26/2020 for today at: http://www.arcamax.com/thefunnies/ar...rcle/s-2457583
Found: Arctic Circle 12/25/2020 for today at: http://www.arcamax.com/thefunnies/ar...rcle/s-2457276
Found: Arctic Circle 12/24/2020 for today at: http://www.arcamax.com/thefunnies/ar...rcle/s-2456974
Found: Arctic Circle 12/23/2020 for today at: http://www.arcamax.com/thefunnies/ar...rcle/s-2456565
Found: Arctic Circle 12/22/2020 for today at: http://www.arcamax.com/thefunnies/ar...rcle/s-2456079
Found: Arctic Circle 12/21/2020 for today at: http://www.arcamax.com/thefunnies/ar...rcle/s-2455603
Finding strips for: Dilbert
Found: Dilbert for today at: http://www.arcamax.com/thefunnies/dilbert
Found: Dilbert for 12/26/2020 at: http://www.arcamax.com/thefunnies/dilbert/s-2457578
Found: Dilbert for 12/25/2020 at: http://www.arcamax.com/thefunnies/dilbert/s-2457271
Found: Dilbert for 12/24/2020 at: http://www.arcamax.com/thefunnies/dilbert/s-2456969
Found: Dilbert for 12/23/2020 at: http://www.arcamax.com/thefunnies/dilbert/s-2456559
Found: Dilbert for 12/22/2020 at: http://www.arcamax.com/thefunnies/dilbert/s-2456074
Found: Dilbert for 12/21/2020 at: http://www.arcamax.com/thefunnies/dilbert/s-2455598
Finding strips for: Garfield
Found: Garfield for today at: http://www.arcamax.com/thefunnies/garfield
Found: Garfield for 12/26/2020 at: http://www.arcamax.com/thefunnies/garfield/s-2457558
Found: Garfield for 12/25/2020 at: http://www.arcamax.com/thefunnies/garfield/s-2457254
Found: Garfield for 12/24/2020 at: http://www.arcamax.com/thefunnies/garfield/s-2456957
Found: Garfield for 12/23/2020 at: http://www.arcamax.com/thefunnies/garfield/s-2456542
Found: Garfield for 12/22/2020 at: http://www.arcamax.com/thefunnies/garfield/s-2456058
Found: Garfield for 12/21/2020 at: http://www.arcamax.com/thefunnies/garfield/s-2455586
Finding strips for: Hagar the Horrible
Found: Hagar the Horrible for today at: http://www.arcamax.com/thefunnies/hagarthehorrible
Found: Hagar the Horrible for 12/26/2020 at: http://www.arcamax.com/thefunnies/ha...ible/s-2457553
Found: Hagar the Horrible for 12/25/2020 at: http://www.arcamax.com/thefunnies/ha...ible/s-2457248
Found: Hagar the Horrible for 12/24/2020 at: http://www.arcamax.com/thefunnies/ha...ible/s-2456945
Found: Hagar the Horrible for 12/23/2020 at: http://www.arcamax.com/thefunnies/ha...ible/s-2456525
Found: Hagar the Horrible for 12/22/2020 at: http://www.arcamax.com/thefunnies/ha...ible/s-2456047
Found: Hagar the Horrible for 12/21/2020 at: http://www.arcamax.com/thefunnies/ha...ible/s-2455571
Finding strips for: Heathcliff
Found: Heathcliff for today at: http://www.arcamax.com/thefunnies/heathcliff
Found: Heathcliff for 12/26/2020 at: http://www.arcamax.com/thefunnies/heathcliff/s-2457546
Found: Heathcliff for 12/25/2020 at: http://www.arcamax.com/thefunnies/heathcliff/s-2457241
Found: Heathcliff for 12/24/2020 at: http://www.arcamax.com/thefunnies/heathcliff/s-2456938
Found: Heathcliff for 12/23/2020 at: http://www.arcamax.com/thefunnies/heathcliff/s-2456517
Found: Heathcliff for 12/22/2020 at: http://www.arcamax.com/thefunnies/heathcliff/s-2456039
Found: Heathcliff for 12/21/2020 at: http://www.arcamax.com/thefunnies/heathcliff/s-2455562
Finding strips for: Luann
Found: Luann for today at: http://www.arcamax.com/thefunnies/luann
Found: Luann for 12/26/2020 at: http://www.arcamax.com/thefunnies/luann/s-2457581
Found: Luann for 12/25/2020 at: http://www.arcamax.com/thefunnies/luann/s-2457274
Found: Luann for 12/24/2020 at: http://www.arcamax.com/thefunnies/luann/s-2456972
Found: Luann for 12/23/2020 at: http://www.arcamax.com/thefunnies/luann/s-2456562
Found: Luann for 12/22/2020 at: http://www.arcamax.com/thefunnies/luann/s-2456077
Found: Luann for 12/21/2020 at: http://www.arcamax.com/thefunnies/luann/s-2455601
Finding strips for: Rhymes with Orange
Found: Rhymes with Orange for today at: http://www.arcamax.com/thefunnies/rhymeswithorange
Found: Rhymes with Orange 12/26/2020 for today at: http://www.arcamax.com/thefunnies/rh...ange/s-2457586
Found: Rhymes with Orange 12/25/2020 for today at: http://www.arcamax.com/thefunnies/rh...ange/s-2457279
Found: Rhymes with Orange 12/24/2020 for today at: http://www.arcamax.com/thefunnies/rh...ange/s-2456977
Found: Rhymes with Orange 12/23/2020 for today at: http://www.arcamax.com/thefunnies/rh...ange/s-2456568
Found: Rhymes with Orange 12/22/2020 for today at: http://www.arcamax.com/thefunnies/rh...ange/s-2456082
Found: Rhymes with Orange 12/21/2020 for today at: http://www.arcamax.com/thefunnies/rh...ange/s-2455606
Finding strips for: Rubes
Found: Rubes for today at: http://www.arcamax.com/thefunnies/rubes
Found: Rubes for today at: http://www.arcamax.com/thefunnies/rubes/s-2457574
Found: Rubes for today at: http://www.arcamax.com/thefunnies/rubes/s-2457267
Found: Rubes for today at: http://www.arcamax.com/thefunnies/rubes/s-2456965
Found: Rubes for today at: http://www.arcamax.com/thefunnies/rubes/s-2456555
Found: Rubes for today at: http://www.arcamax.com/thefunnies/rubes/s-2456070
Found: Rubes for today at: http://www.arcamax.com/thefunnies/rubes/s-2456069
Finding strips for: Speed Bump
Found: Speed Bump for today at: http://www.arcamax.com/thefunnies/speedbump
Found: Speed Bump for 12/26/2020 at: http://www.arcamax.com/thefunnies/speedbump/s-2457544
Found: Speed Bump for 12/25/2020 at: http://www.arcamax.com/thefunnies/speedbump/s-2457239
Found: Speed Bump for 12/24/2020 at: http://www.arcamax.com/thefunnies/speedbump/s-2456936
Found: Speed Bump for 12/23/2020 at: http://www.arcamax.com/thefunnies/speedbump/s-2456515
Found: Speed Bump for 12/22/2020 at: http://www.arcamax.com/thefunnies/speedbump/s-2456037
Found: Speed Bump for 12/21/2020 at: http://www.arcamax.com/thefunnies/speedbump/s-2456036
Finding strips for: Wizard of Id
Found: Wizard of Id for today at: http://www.arcamax.com/thefunnies/wizardofid
Found: Wizard of Id for 12/26/2020 at: http://www.arcamax.com/thefunnies/wizardofid/s-2457547
Found: Wizard of Id for 12/25/2020 at: http://www.arcamax.com/thefunnies/wizardofid/s-2457242
Found: Wizard of Id for 12/24/2020 at: http://www.arcamax.com/thefunnies/wizardofid/s-2456939
Found: Wizard of Id for 12/23/2020 at: http://www.arcamax.com/thefunnies/wizardofid/s-2456518
Found: Wizard of Id for 12/22/2020 at: http://www.arcamax.com/thefunnies/wizardofid/s-2456041
Found: Wizard of Id for 12/21/2020 at: http://www.arcamax.com/thefunnies/wizardofid/s-2455564
Finding strips for: Zits
Found: Zits for today at: http://www.arcamax.com/thefunnies/zits
Found: Zits for 12/26/2020 at: http://www.arcamax.com/thefunnies/zits/s-2457538
Found: Zits for 12/25/2020 at: http://www.arcamax.com/thefunnies/zits/s-2457233
Found: Zits for 12/24/2020 at: http://www.arcamax.com/thefunnies/zits/s-2456930
Found: Zits for 12/23/2020 at: http://www.arcamax.com/thefunnies/zits/s-2456509
Found: Zits for 12/22/2020 at: http://www.arcamax.com/thefunnies/zits/s-2456029
Found: Zits for 12/21/2020 at: http://www.arcamax.com/thefunnies/zits/s-2455556
Synthesizing mastheadImage
Fetching file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/5.html
Fetching file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/3.html
Fetching file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/6.html
Fetched /tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/5.html in 0.0 seconds
Fetching file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/7.html
Fetching file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/4.html
Fetched /tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/3.html in 0.0 seconds
Fetched /tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/6.html in 0.0 seconds
Fetched /tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/7.html in 0.0 seconds
Parsed file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/3.html in 0.0 seconds
Processing images...
Parsed file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/5.html in 0.0 seconds
Processing images...
Fetched /tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/4.html in 0.1 seconds
Fetching http://arcamax.com/newspics/203/20313/2031339.gif
Parsed file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/7.html in 0.0 seconds
Fetching http://arcamax.com/newspics/203/20326/2032659.gif
Parsed file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/6.html in 0.1 seconds
Processing images...
Parsed file:///tmp/calibre_5.8.1_tmp_f0odngk_/yu_msjdnarcamax/4.html in 0.0 seconds
Processing images...
Processing images...
Fetching http://arcamax.com/newspics/202/20298/2029809.gif
Fetching http://arcamax.com/newspics/203/20320/2032092.gif
Fetching http://arcamax.com/newspics/203/20307/2030762.gif
Fetched http://arcamax.com/newspics/203/20307/2030762.gif in 1.358774 seconds
Fetched http://arcamax.com/newspics/203/20313/2031339.gif in 1.404818 seconds

[kovidgoyal]

This should take care of it: https://github.com/kovidgoyal/calibr...e5593d688af4e9

[hegi]

Hi Kovid,
that seems to do the trick ;-). Thanks a lot ... once again!

Just a general question as I write to you: Some versions back Calibre introduced for news-downloads a header & a footer with basic navigation (forward, submenu, main menus, back). Is there a way to switch this off? Ideally on a per-recipe-basis.

I've got some comics that stretch now over two pages, were I do not want this ...

Cheers! - And a happy, healty and good new year to you and Krittika down in Mumbai!
... from Germany with love

Stefan.

[kovidgoyal]

That feature has been there since the beginning. It is disabled when generating kindle periodicals, so set your utput profile to kindle and output format to azw3 or mobi.

[hegi]

Thanks Kovid, that seems to do the trick ...

[KecajN]

Originally Posted by kovidgoyal

This should take care of it: https://github.com/kovidgoyal/calibr...e5593d688af4e9

Fix to recipe made the trick for me today. I hope that it is a permanent solution.