"The New York Times" recipe failing with error

[mikebw]

I'm not sure what is causing this, but The New York Times recipe consistently fails out with an error, apparently while parsing the scraped page:

Fetch news from New York Times
Resolved conversion options
calibre version: 2.39.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'dont_split_on_page_breaks': True,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'epub_flatten': False,
 'epub_inline_toc': False,
 'epub_toc_at_end': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'flow_size': 260,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x7f8169b99d10>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'no_chapters_in_toc': False,
 'no_default_epub_cover': False,
 'no_inline_navbars': False,
 'no_svg_cover': False,
 'output_profile': <calibre.customize.profiles.KindleFireOutput object at 0x7f8169ba6490>,
 'page_breaks_before': None,
 'prefer_metadata_cover': False,
 'preserve_cover_aspect_ratio': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': '',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'smarten_punctuation': False,
 'sr1_replace': '',
 'sr1_search': '',
 'sr2_replace': '',
 'sr2_search': '',
 'sr3_replace': '',
 'sr3_search': '',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Traceback (most recent call last):
  File "site.py", line 63, in main
  File "site-packages/calibre/utils/ipc/worker.py", line 193, in main
  File "site-packages/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1042, in run
  File "site-packages/calibre/customize/conversion.py", line 241, in __call__
  File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 117, in convert
  File "site-packages/calibre/web/feeds/news.py", line 1029, in download
  File "site-packages/calibre/web/feeds/news.py", line 1194, in build_index
  File "<string>", line 736, in parse_index
  File "<string>", line 676, in parse_todays_index
  File "<string>", line 565, in get_popular_articles
  File "site-packages/calibre/web/feeds/news.py", line 689, in index_to_soup
  File "site-packages/mechanize/_mechanize.py", line 199, in open_novisit
  File "site-packages/mechanize/_mechanize.py", line 255, in _mech_open
httperror_seek_wrapper: HTTP Error 404: Not Found

My impression is that this particular recipe is perhaps the most elegantly complicated and sophisticated of all built-in recipes, and I'm terrified to try debugging it.

[NSILMike]

I see it too. I think the error indication is:

Python function terminated unexpectedly
HTTP Error 404: Not Found (Error Code: 1)

Let me add also, that I ran it 3 times, and it always failed quickly (7 seconds twice and 8 seconds once.)

[kovidgoyal]

https://github.com/kovidgoyal/calibr...df121865d2ca94

[mikebw]

The stock recipe in 2.39 started working again mysteriously this morning.

Thanks for the GitHub fix version, though: we may need that.

[kovidgoyal]

A change to the recipe in github is automatically propagated to all calibre installs in about an hour.

[NSILMike]

Originally Posted by kovidgoyal

A change to the recipe in github is automatically propagated to all calibre installs in about an hour.

Wow, that is extremely fast. I thought they propagated on Friday when new Calibre versions show up...

Here is a mostly unrelated question. I had started customizing a few of my recipes (for example, to limit the age of news articles to one day on daily downloads.) In another conversation here, you let me know that my customized recipes don't get the general updates- and of course that makes a perfect sense. I wonder though, is there any trick to combine the two? I highly doubt it (other than customizing the updated recipe) but....?

[kovidgoyal]

No there is no way to combine the two.

[NSILMike]

Originally Posted by kovidgoyal

No there is no way to combine the two.

Makes sense. Thanks.

[eschwartz]

Originally Posted by NSILMike

Wow, that is extremely fast. I thought they propagated on Friday when new Calibre versions show up...

Nope -- the latest (GitHub) version of an existing recipe is automatically retrieved from the calibre website during the news download.

Here is a mostly unrelated question. I had started customizing a few of my recipes (for example, to limit the age of news articles to one day on daily downloads.) In another conversation here, you let me know that my customized recipes don't get the general updates- and of course that makes a perfect sense. I wonder though, is there any trick to combine the two? I highly doubt it (other than customizing the updated recipe) but....?

Well, you could use a diff of the customized version and apply that to the updated version.
Or save it as a private commit to the source code (see http://manual.calibre-ebook.com/develop.html) and rebase it.