Wall Street Journal recipe skipping A-Heds

[BobbyVan]

The A-Hed is the Journal's traditionally humorous/off-beat front page story. For some reason on the "In Today's Paper" page (http://online.wsj.com/itp?mod=WSJ_formfactor) , it's been separated from the rest of the front page stories and is now found between the "Front Section" and "What's News" sections. Consequently, the recipe has been skipping A-Heds of late.

Is there a way to update the recipe to fetch this and include it somewhere in the "Front Section" output where it belongs?

Thank you.

[kovidgoyal]

It's certainly doable, someone would have to modify the parse_index() method int he recipe to special case that article. Given that parsing that page is fairly involved, it is unlikely to be a simple change, however.

[kovidgoyal]

https://github.com/kovidgoyal/calibr...dcd95024580240

[joeindallas]

Wow - I never cease to be amazed by the timeliness of help around here! (I'm not even a WSJ reader, but still...)

[BobbyVan]

Originally Posted by kovidgoyal

https://github.com/kovidgoyal/calibr...dcd95024580240

[SunLight]

I'd love for my daily WSJ file to include A-Heads too, used the above script, and got this message:

calibre, version 2.36.0 (win32, isfrozen: True)
Conversion Error: Failed: Fetch news from The Wall Street Journal - Heads

Fetch news from The Wall Street Journal - Heads
Resolved conversion options
calibre version: 2.36.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000000004EEE390>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': False,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.GenericEink object at 0x0000000004EEE710>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
JSBrowser msg():https://a248.e.akamai.net/f/248/6767...44309183394:1: Porthole: Using built-in browser support
JSBrowser msg():https://a248.e.akamai.net/f/248/6767...44309183394:1: standalone
Reply error: https://id.wsj.com/access/pages/wsj/us/img/logo-x2.png - 2 (Connection closed)
Reply error: https://id.wsj.com/access/pages/wsj/...-select-x2.png - 2 (Connection closed)
Reply error: https://id.wsj.com/access/pages/wsj/...ok-oct2013.png - 2 (Connection closed)
Reply error: https://id.wsj.com/access/pages/wsj/...with-gPlus.png - 2 (Connection closed)
JSBrowser msg():https://a248.e.akamai.net/f/248/6767...44309185221:1: Porthole: Using built-in browser support
JSBrowser msg():https://a248.e.akamai.net/f/248/6767...44309183394:1: standalone
Python function terminated unexpectedly
'NoneType' object has no attribute 'find' (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 193, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 1042, in run
File "site-packages\calibre\customize\conversion.py", line 241, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\recipe_ input.py", line 117, in convert
File "site-packages\calibre\web\feeds\news.py", line 1029, in download
File "site-packages\calibre\web\feeds\news.py", line 1194, in build_index
File "<string>", line 97, in parse_index
AttributeError: 'NoneType' object has no attribute 'find'

[kovidgoyal]

Just use the builtin recipe, it now includes the ahed

https://github.com/kovidgoyal/calibr...d97ed00ce2d5bc

[SunLight]

Originally Posted by kovidgoyal

Just use the builtin recipe, it now includes the ahed

https://github.com/kovidgoyal/calibr...d97ed00ce2d5bc

That did it, thank you Made my day!