Washington Post recipe error

[cchutney]

Hi, I'm a Calibre user (and donator ) since version 1 - it is an absolutely wonderful service.

Lately I have noticed however that the Washington Post Recipe fails with the final line in the log being: ValueError: No articles found, aborting. Have they changed the format again?

While on the subject, the Atlantic.com recipe downloads OK, but for many articles it now shows just the same set of static photos, rather than the article that is linked from the article index. A similar problem perhaps?

I am on the latest 3.2.5 version....

[kovidgoyal]

Works for me, are you sure you are using the builtin recipe and not a custom one?

[cchutney]

Puzzling that it works for you. I just tried again from the drop-down menu and it gave the same conversion error.


calibre, version 3.25.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from The Washington Post

Fetch news from The Washington Post
Conversion options changed from defaults:
output_profile: 'kindle_pw'
verbose: 2
Resolved conversion options
calibre version: 3.25.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000025B27D74FD0>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhiteOutp ut object at 0x0000025B27DA66D8>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Python function terminated unexpectedly
No articles found, aborting (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 101, in main
File "site.py", line 78, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 195, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 26, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 1088, in run
File "site-packages\calibre\customize\conversion.py", line 244, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\recipe_ input.py", line 119, in convert
File "site-packages\calibre\web\feeds\news.py", line 1018, in download
File "site-packages\calibre\web\feeds\news.py", line 1193, in build_index
ValueError: No articles found, aborting

[bobbysteel]

I just tested and see the same no-articles-found error. Are you a paid subscriber? They seem to have gone full paywall now. Also I'm seeing there's a GDPR interstitial if you're outside the USA too which maybe mucking things up

[cchutney]

There is no field for subscriber username/pw in the recipe I have - is there a version of the recipe that allows that? Looks like otherwise the Post is a gonner. Shame. I wonder how it is that Kovid's recipe works given these problems ?

[kovidgoyal]

EDIT: Never mind, I guess this has to do with GDPR

[bobbysteel]

Originally Posted by cchutney

There is no field for subscriber username/pw in the recipe I have - is there a version of the recipe that allows that? Looks like otherwise the Post is a gonner. Shame. I wonder how it is that Kovid's recipe works given these problems ?

you might be better off finding a free proxy or using a US based VPN when downloading if it's GDPR related.

[kovidgoyal]

The recipe can probably be fixed to workaround it, but it has to be done by someone that can reproduce the problem.

[cchutney]

Thanks Bobbysteel it does indeed work via a VPN to the US. That will do as my workaround - I have an annual VPN contract, so it just needs enabling permanently on my Calibre server. Much obliged - I missed my daily dose of Jeff Bezos dinging Donald Trump.

[bobbysteel]

Happy to help @cchutney.
Kovid, seems to be IP based blocking on their end from what I can tell. All hits redirect to the paywall/login page here https://www.washingtonpost.com/gdpr-...ination=%2f%3f and it's not able to be bypassed easily AFAICT.

[kovidgoyal]

Hmm, well, in that case I'll just add a note to the recipe description saying that the website is blocked in europe.

[cchutney]

Hi Kovid, the Washington Post seems now to have put in a handler for GDPR, and the paper and its sections now download. However, at the beginning of each article, there is a message saying:

"By clicking “I agree” below, you consent to the use by us and our third-party partners of cookies and data gathered from your use of our platforms".

It seems that maybe the recipe can be fixed by clicking this box. Is it possible to investigate whether that works?

Many thanks

[kovidgoyal]

You can edit the recipe to remove that box on download. Currently it uses auto_cleanup I think. The recipe would need to be rewritten to use keep_tags/remove_tags instead. Shouldn't be hard, but since I am not ineurope, I cannot help with that.

[kovidgoyal]

I changed the recipe to not use auto clenup. So if you still see the box, all you need to do is figure out what id/class the HTML for that box has, and add it to remvoe_tags in the recipe.

[EEAndersen]

I wanted to comment - for the benefit of future readers - that my use of "pihole", a DNS advertising filter, interfered with Washington Post downloads. I suspect it has to do with the connection between Amazon and WaPo. So I am disabling pihole for 5 minutes for WaPo downloads. Appears LA Times might also be involved in blocks from pihole.