LA Times fetch hangs interminably

[Rick's Kindle]

6 hours later and my LA Times d/l is still chugging away at 1%. Doesn't time out and doesn't progress. Any guidance would be appreciated.

Job details as follows:

Fetch news from The Los Angeles Times
Resolved conversion options
calibre version: 0.7.9
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'extra_css': None,
'font_size_mapping': None,
'footer_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'header_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x04F594B0>,
'insert_blank_line': False,
'insert_metadata': False,
'isbn': None,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'max_toc_links': 50,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindleOutput object at 0x04F59790>,
'page_breaks_before': None,
'password': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'preprocess_html': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_first_image': False,
'remove_footer': False,
'remove_header': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'rescale_images': False,
'series': None,
'series_index': None,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'use_auto_toc': False,
'username': None,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Looking for full story link in http://www.latimes.com/news/la-me-bi...,4748121.story
Looking for full story link in http://www.latimes.com/news/la-sci-h...,5183013.story
Looking for full story link in http://www.latimes.com/news/la-na-oi...,5787606.story
Looking for full story link in http://www.latimes.com/news/la-me-07...0,318452.story
Looking for full story link in http://www.latimes.com/news/la-me-li...,7075277.story
Looking for full story link in http://www.latimes.com/news/la-me-07...,1955509.story
Looking for full story link in http://www.latimes.com/news/la-me-07...,5138232.story
Looking for full story link in http://www.latimes.com/news/la-na-jo...,2225607.story
Found full story link http://www.latimes.com/news/la-na-jo...115,full.story
Looking for full story link in http://www.latimes.com/news/la-me-07...,3088449.story
Looking for full story link in http://www.latimes.com/news/local/la...,6229042.story
Found full story link http://www.latimes.com/news/local/la...310,full.story
Looking for full story link in http://www.latimes.com/news/local/la...,3128251.story
Looking for full story link in http://www.latimes.com/news/local/la...,3651606.story
Looking for full story link in http://www.latimes.com/news/local/la...0,771528.story
Looking for full story link in http://www.latimes.com/news/local/la...,3188139.story
Found full story link http://www.latimes.com/news/local/la...484,full.story
Looking for full story link in http://www.latimes.com/news/local/la...0,718439.story
Looking for full story link in http://www.latimes.com/news/local/la...,5232560.story
Looking for full story link in http://www.latimes.com/news/local/la...,5455407.story
Looking for full story link in http://www.latimes.com/news/local/me...7896235.column
Looking for full story link in http://www.latimes.com/news/local/la...,0,81442.story
Looking for full story link in http://www.latimes.com/news/health/l...,1610881.story
Looking for full story link in http://www.latimes.com/news/custom/t...,4176839.story
Looking for full story link in http://www.latimes.com/news/nationwo...,5232689.story
Looking for full story link in http://www.latimes.com/news/local/la...,3128251.story
Looking for full story link in http://www.latimes.com/news/custom/t...,2721392.story
Looking for full story link in http://www.latimes.com/business/la-f...,6089020.story
Looking for full story link in http://www.latimes.com/news/local/la...,1616500.story
Looking for full story link in http://www.latimes.com/news/health/l...,1531545.story
Found full story link http://www.latimes.com/news/health/l...357,full.story
Looking for full story link in http://www.latimes.com/news/la-fiw-0...0,106596.story
Looking for full story link in http://www.latimes.com/news/local/la...0,898989.story
Looking for full story link in http://www.latimes.com/news/local/po...,3571179.story

[Starson17]

Originally Posted by Rick's Kindle

6 hours later and my LA Times d/l is still chugging away at 1%. Doesn't time out and doesn't progress. Any guidance would be appreciated.

Mine completed in 6.5 minutes. The recipe seems OK. Are other recipes working/failing? Can you reach the links in that recipe in a browser?

[Rick's Kindle]

The recipe is the unadulterated version of the one offered within Calibre. Yeah, the links all send me to the full stories via Firefox. All other recipes (approx. 12) work beautifully.

[kovidgoyal]

Probably temporary network trouble. Try it again.

[Rick's Kindle]

Originally Posted by kovidgoyal

Probably temporary network trouble. Try it again.

Sadly, no. I've had the same issue for the week+ I've been using your program. I DID retry, however; I'm 20 minutes in and still at 1%.

It's no exaggeration to say that I'm finding Calibre life-enriching. Thank you.

[kovidgoyal]

Hmm, well that recipe jumps through a few hoops to try to get the full version of each article from the LA Times website, because of the way the website is designed. Which means it is more network intensive than most other recipes, which is probably what's causing the problem for you. I'm afraid there's no simple solution to this.

[Starson17]

Originally Posted by Rick's Kindle

Sadly, no. I've had the same issue for the week+ I've been using your program. I DID retry, however; I'm 20 minutes in and still at 1%.

Try modifying the recipe to add throttles:

simultaneous_downloads = 1
delay = 3

This will drastically slow it down, but it may help allow it to complete.

[Rick's Kindle]

Originally Posted by Starson17

Try modifying the recipe to add throttles:

simultaneous_downloads = 1
delay = 3

Thanks, I'll try this if you can tell me where in the recipe to paste that code.

Edit: I added those lines under the "Class" heading and below "max_articles_per_feed." That sound right?

[kovidgoyal]

Another thing you can do is delete the get_article_url function from the recipe, it will mean some articles will be incomplete, but the download itself should complete

[Rick's Kindle]

The throttle seems to have done the trick. I'm getting a good sampling of articles in each section (though the job details do show a number of items as not being downloaded). Thanks for your help with this.

I think I'll also try deleting the get_article_url function from a fresh recipe and see which produces a better product.

[Starson17]

Originally Posted by Rick's Kindle

Edit: I added those lines under the "Class" heading and below "max_articles_per_feed." That sound right?

Yes.