Economist fetching stopped working for me yesterday

[NullNix]

I see the usual failure message for an Economist recipe failure, both with my existing antique Calibre (2.64.0) and with a current 2.78.0. The username and password work with the current Economist website via a browser, and the print edition shows a webpage as usual, with article titles. It's hard to figure out what the page Calibre's fetching looks like on account of SSL, but I wonder if, perhaps, we're being fed the beta site? (It does look like you have to set a cookie to force that, still, though, so perhaps that isn't the problem and the site has just legitimately changed.)

(Kovid: probably know about this already, since you're an Economist reader too, but just in case this isn't happening for you, what can I provide to help debug this? A dump of the HTML source of http://www.economist.com/printedition as I see it, perhaps? It would be nice if I could convince Calibre to emit a dump of the page it downloaded...)

ebook-convert "The Economist.recipe" ~/tmp/conversion/economist-$(date +"%Y-%m-%d").mobi --output-profile=kindle_voyage --username={censored} --password={censored} --keep-ligatures --smarten-punctuation --change-justification left

Resolved conversion options
calibre version: 2.64.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'even_margin_left': 5.0,
'even_margin_right': 5.0,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x7f0ddb876710>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutp ut object at 0x7f0ddb876e90>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Traceback (most recent call last):
File "/usr/bin/calibre-parallel", line 20, in <module>
sys.exit(main())
File "/usr/lib/calibre/calibre/utils/ipc/worker.py", line 190, in main
result = func(*args, **kwargs)
File "/usr/lib/calibre/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
plumber.run()
File "/usr/lib/calibre/calibre/ebooks/conversion/plumber.py", line 1074, in run
accelerators, tdir)
File "/usr/lib/calibre/calibre/customize/conversion.py", line 241, in __call__
log, accelerators)
File "/usr/lib/calibre/calibre/ebooks/conversion/plugins/recipe_input.py", line 117, in convert
ro.download()
File "/usr/lib/calibre/calibre/web/feeds/news.py", line 998, in download
res = self.build_index()
File "/usr/lib/calibre/calibre/web/feeds/news.py", line 1163, in build_index
feeds = feeds_from_index(self.parse_index(), oldest_article=self.oldest_article,
File "<string>", line 81, in parse_index
File "<string>", line 129, in economist_parse_index
Exception: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

[NullNix]

Looks like this is bug #1654763, which of course I didn't find until posting this.

Just as for the original poster, one recipe works, the other doesn't, which is mystifying because the two are *byte-for-byte identical*, as far as I can tell from the Calibre source tree.

So I guess my question has changed to: if one recipe fails and the other works, given that they have the same name, how do I run one rather than the other from ebook-convert? (I'm passing extra flags in, and doing it from cron, and I'd rather not figure out how to do all that from the GUI given that I only want e.g. the justification changes to apply to downloads of this one periodical.)

[NullNix]

OK, so I tried making a custom recipe out of the Economist recipe. One failed, the other worked, still. Copied-and-pasted them into .recipe files (where they were still byte-for-byte identical, as expected) and both worked.

This is deeply bizarre! Does failure depend on the *name of the .recipe file* or something? (I mean, maybe it's a pure timing issue and the website is sometimes working and sometimes not, but I've tried dozens of times and this behaviour is perfectly consistent, so either I'm being astonishingly unlucky or there's a bug here that does not depend on the -- identical -- recipe content, and the only other thing that differs that I can see between the two recipes is their .recipe file name.)

[kovidgoyal]

It's simply an intermittent issue with the economist servers. It has nothing to do with the recipes.

[NullNix]

Originally Posted by kovidgoyal

It's simply an intermittent issue with the economist servers. It has nothing to do with the recipes.

Astonishing ill luck on my part, then I was most unhappy for a while: a Calibre-converted ebook on a Kindle is so much more pleasant to read than a big floppy magazine...