Der Spiegel (access to paid content) recipe failing

[Sheep2045]

Hiho! :-)

Starting a week ago, the recipe mentioned in the topic started failing with:

Fetch news from Der Spiegel
Resolved conversion options
calibre version: 1.23.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x814269610>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': False,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.GenericEink object at 0x814269a10>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Traceback (most recent call last):
File "/usr/local/bin/calibre-parallel", line 20, in <module>
sys.exit(main())
File "/usr/local/lib/calibre/calibre/utils/ipc/worker.py", line 192, in main
result = func(*args, **kwargs)
File "/usr/local/lib/calibre/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
plumber.run()
File "/usr/local/lib/calibre/calibre/ebooks/conversion/plumber.py", line 1035, in run
accelerators, tdir)
File "/usr/local/lib/calibre/calibre/customize/conversion.py", line 241, in __call__
log, accelerators)
File "/usr/local/lib/calibre/calibre/ebooks/conversion/plugins/recipe_input.py", line 116, in convert
ro = recipe(opts, log, self.report_progress)
File "/usr/local/lib/calibre/calibre/web/feeds/news.py", line 871, in __init__
self.browser = self.get_browser()
File "<string>", line 51, in get_browser
File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 541, in submit
return self.open(self.click(*args, **kwds))
File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 203, in open
return self._mech_open(url, data, timeout=timeout)
File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 255, in _mech_open
raise response
mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found

I checked the URLs linked to in the recipe and they work (except the masthead, which I tried commenting out to no avail - still doesn't work).

Does anyone have an idea what might be wrong?

Thanks in advance!

Regard,
sheep2045

[kovidgoyal]

You appear to be using a custom recipe, make sure you are using the builtin one.

[Sheep2045]

Thanks for the quick reply; the problem with the builtin recipe is that it doesn't support downloading paid content, only the articles freely available (which are vastly inferior).
I was hoping someone else was using the custom recipe (I don't think there is another one) and could shed some light on what's going wrong.
Judging by your last posts regarding the topic spiegel.de and recipes, they probably changed something on their website...

Regards,
Marc

[kovidgoyal]

Looking at the builtin recipe, it appears to have a login, so it should be downloading full content, but as I haven't written that recipe, I cannot be sure.

[Gustav M.]

Hi!

I have the same problem with the builtin recipe for "Der Spiegel". It worked perfectly until about 10 days ago.

@Sheep2045 Just to clarify: The built in recipe did/does provide access to the paid content. Could it be that you refer to the Spiegel Online Recipe?

@kovidgoyal How did you know that Sheep2045 was using a custom recipe? Because of the "Using custom recipe" in the log? I also get this.

Gustav

[Sheep2045]

Yeah, I somehow copied the builtin "Der Spiegel" recipe to custom recipes. I just deleted it and tried the builtin, which fails in exactly the same way.

@Gustav M.: as written above, I got confused due to me having copied the builtin to custom recipes. I was referring to the builtin that provides access to paid content.

[Gustav M.]

Hi,

I just tried to contact the author of the recipe. Let's hope he can help!

If you are using a Kindle, have you tried
https://abo.spiegel.de/de/c/kindle

It says "reine Textversion", so it is probably not the same, but perhaps it will serve as a temporary solution for you.

Gustav

[aerodynamik]

Hi Gustav, Hi everyone,

sorry, I didn't see this thread any earlier. I actually don't have an account anymore for Der Spiegel. I created the original recipe when there was a 2 week test period, and somebody here gave me their account (password temporary changed) to fix some issues.

I took a quick look without an account. What's obviously broken is the masthead URL. Before posting the completely fixed recipe I would like somebody to test the change here first or be so kind and give me temporarily their account information.

To test it out, customize the built in recipe and replace the masthead_url with 'http://upload.wikimedia.org/wikipedia/de/thumb/3/3e/Logo-der_spiegel.svg/200px-Logo-der_spiegel.svg.png'

This at least gets rid of the 404 error.

Cheers,
aero

[Sheep2045]

Hi aerodynamik,

thanks for your quick response! :-)
I just tried replacing the masthead url, but that unfortunately does not change anything regarding the 404 error (error remains exactly the same that I posted above).

I'll change my password and pm it to you.

Thanks a bunch for taking a look at this!


Cheers,
sheep2045

[kovidgoyal]

Originally Posted by Gustav M.

@kovidgoyal How did you know that Sheep2045 was using a custom recipe? Because of the "Using custom recipe" in the log? I also get this.

No, the line numbers in the error message dont match the line numbers in the builtin recipe. Using cutom recipe is always there if you download via the GUI, because that is the way the GUI works.

[aerodynamik]

Hi sheep2045,

thanks for your support and help with this. I used your account to analyze the issue.

I ran into a few issues tonight with timeouts also using my normal browser. Access to the mobile site, which is used by the recipe, is quite slow overall.

This shouldn't break the recipe however. I still get the 404 error for the login form URL, which definitely exists though. I found a dirty fix by simply ignoring the error (try / except around br.submit() in get_browser), however I am sure I can find a clean resolution for this error.

Give me a few more days. If you absolutely cannot wait try this for now instead of the br.submit() at around line 51 of the recipe:

PHP Code:

try:
    br.submit()
except:
    self.log('submit raised an error, let's ignore and continue') 


Hope this helps for now.

Cheers,
aero

[Sheep2045]

Thanks a bunch; this code:

PHP Code:

try:
    br.submit()
except:
    self.log('submit raised an error, ignore and continue') 


works as a workaround, though I'm grateful that you're looking for a proper fix!

Cheers,
sheep2045

[Gartenzwerg]

Just in case somebody is looking for a Kindle version: If you have a subscription, Der Spiegel is able to send a MOBI file directly to your Kindle. See https://abo.spiegel.de/de/c/kindle

[Sheep2045]

@Gartenzwerg: Yeah, thanks for the remainder, I think aero mentioned it as well. Unfortunately, that only works (to my understanding) without incurring further costs (for 3g delivery) if your Kindle is registered to a german Amazon account? But maybe I'm misinterpreting Amazon's terms of service regarding subscriptions?
(and it obviously won't matter if you have wifi)


Cheers,
sheep2045

[aerodynamik]

OK, here we go. They must have changed something that requires the login.html to have a valid referrer URL. Skip the try / except from before and just replace the getbrowser section with the follwing. The real change to the original recipe is the URL in br.open now containing the ?backUrl part.

PHP Code:

br = BasicNewsRecipe.get_browser(self)
        if self.username is not None and self.password is not None:
            br.open(self.PREFIX + '/meinspiegel/login.html?backUrl=' + self.PREFIX + '/spiegel/print')
            br.select_form(predicate=has_login_name)
            br['f.loginName'] = self.username
            br['f.password'] = self.password
            br.submit()
        return br 


I took a quick look at the resulting eBook. Doesn't look as nice as I remember it, e.g. unnecessary information at the end of each article, headers not clipped correctly, some articles without a header at all.

Is this just in my Kindle output or do you experience the same?