WSJ recipe fails

[mjfriedman]

I received this message:

calibre, version 3.34.0 (darwin, embedded-python: True)
Conversion error: Failed: Fetch news from The Wall Street Journal

Fetch news from The Wall Street Journal
Conversion options changed from defaults:
verbose: 2
output_profile: u'kindle_voyage'
Resolved conversion options
calibre version: 3.34.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x1124ff1d0>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindleVoyageOutput object at 0x1124ffa10>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
Python function terminated unexpectedly: No articles found, aborting
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:wsj
Trying to get latest version of recipe: wsj
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 154, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 199, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 35, in gui_convert_recipe
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 27, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 1106, in run
File "site-packages/calibre/customize/conversion.py", line 244, in __call__
File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 136, in convert
File "site-packages/calibre/web/feeds/news.py", line 1021, in download
File "site-packages/calibre/web/feeds/news.py", line 1196, in build_index
ValueError: No articles found, aborting

[kovidgoyal]

the website has changed so the recipe will need to be re-written.

[kovidgoyal]

Well I have updated the recipe to at least get the articles from the newly redesigned print edition WSJ page. Hoever, as I am not a WSJ subscriber I have no way to test the recipe further unless someone PMs their WSJ username/password to me.

[mjfriedman]

Yes, it's working again. Thank you so much!

FWIW, some stories are missing headlines and graphics are clumsily treated (one must turn many pages to get to the end of a graphic and back to the main story, limitations, I imagine, of the format?).

But those are very minor quibbles. Thank you again for Calibre, and for making this fix!

[kovidgoyal]

to look at the article formatting issues i will need a wsj username and password -- PM me one if you want me to look at it.

[bobbysteel]

Originally Posted by kovidgoyal

to look at the article formatting issues i will need a wsj username and password -- PM me one if you want me to look at it.

sent! thanks as always Kovid for your quick help

[kovidgoyal]

I fixed the issue with the missing headlines, but I could not find any messed up graphics, the worstI saw was very large images that took up an entire page.

[mjfriedman]

Thank you again. I'll flag the next example of a poorly rendering graphic I encounter.

[mjfriedman]

For example the graphic in the U.S. News section story "Consumer Beware: With Less Trade Comes Less Choice. I totally understand that e-readers are not built to display graphics but this fairly simple graphic is chopped up and spread over five screens on my 7-inch Oasis.

Again, I understand that a better rendering may not be possible on an e-Reader, as opposed to a tablet. But flagging it for you regardless

[mjfriedman]

Only the first three paragraphs of the story "Maldives’ New Leaders Confront a Chinese-Funded Building Binge" rendered. The last words to appear were "repayment periods" followed by a map.

BTW I should say that I am super grateful for the support on this forum; I can't code or do much beyond contribute to the upkeep of Calibre and flag issues

[kovidgoyal]

the graphics being split up is really a function of the reader yo are using. Open the epub using calibre's viewer and you should see the image occupying a single full page ratherthan being chopped up over multiple pages.

[mjfriedman]

The recipe seems again to be having problems:

Fetch news from The Wall Street Journal
Conversion options changed from defaults:
verbose: 2
output_profile: u'kindle_voyage'
Resolved conversion options
calibre version: 3.34.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x116bff1d0>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindleVoyageOutput object at 0x116bffa10>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
Python function terminated unexpectedly: No articles found, aborting
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:wsj
Trying to get latest version of recipe: wsj
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 154, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 199, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 35, in gui_convert_recipe
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 27, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 1106, in run
File "site-packages/calibre/customize/conversion.py", line 244, in __call__
File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 136, in convert
File "site-packages/calibre/web/feeds/news.py", line 1021, in download
File "site-packages/calibre/web/feeds/news.py", line 1196, in build_index
ValueError: No articles found, aborting

[kovidgoyal]

that's because currently their front page is broken, wait a few hours and hopefully they will fix it. the page currently has no articles, only the wahts news section.

https://www.wsj.com/print-edition/20181130/frontpage

[louisyen0505]

calibre, version 4.1.0 (win32, embedded-python: True)
轉換錯誤: 失敗: 從The Wall Street Journal接收新聞

從The Wall Street Journal接收新聞
Conversion options changed from defaults:
verbose: 2
Resolved conversion options
calibre version: 4.1.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': u'original',
'chapter': None,
'chapter_mark': u'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'epub_version': u'2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x05DE8F30>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.OutputProfile object at 0x05E16130>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': u'',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': u'',
'sr1_search': u'',
'sr2_replace': u'',
'sr2_search': u'',
'sr3_replace': u'',
'sr3_search': u'',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:wsj
Trying to get latest version of recipe: wsj
Starting login process...
Sending login request...
Performing login callback...
P