Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 03-02-2014, 05:27 PM   #1
Sheep2045
Junior Member
Sheep2045 began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Mar 2014
Device: Kindle DXG
Der Spiegel (access to paid content) recipe failing

Hiho! :-)

Starting a week ago, the recipe mentioned in the topic started failing with:

Quote:
Fetch news from Der Spiegel
Resolved conversion options
calibre version: 1.23.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x814269610>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': 'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': False,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.GenericEink object at 0x814269a10>,
'page_breaks_before': None,
'personal_doc': '[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Using custom recipe
Traceback (most recent call last):
File "/usr/local/bin/calibre-parallel", line 20, in <module>
sys.exit(main())
File "/usr/local/lib/calibre/calibre/utils/ipc/worker.py", line 192, in main
result = func(*args, **kwargs)
File "/usr/local/lib/calibre/calibre/gui2/convert/gui_conversion.py", line 25, in gui_convert
plumber.run()
File "/usr/local/lib/calibre/calibre/ebooks/conversion/plumber.py", line 1035, in run
accelerators, tdir)
File "/usr/local/lib/calibre/calibre/customize/conversion.py", line 241, in __call__
log, accelerators)
File "/usr/local/lib/calibre/calibre/ebooks/conversion/plugins/recipe_input.py", line 116, in convert
ro = recipe(opts, log, self.report_progress)
File "/usr/local/lib/calibre/calibre/web/feeds/news.py", line 871, in __init__
self.browser = self.get_browser()
File "<string>", line 51, in get_browser
File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 541, in submit
return self.open(self.click(*args, **kwds))
File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 203, in open
return self._mech_open(url, data, timeout=timeout)
File "/usr/local/lib/python2.7/site-packages/mechanize/_mechanize.py", line 255, in _mech_open
raise response
mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found
I checked the URLs linked to in the recipe and they work (except the masthead, which I tried commenting out to no avail - still doesn't work).

Does anyone have an idea what might be wrong?

Thanks in advance!

Regard,
sheep2045
Sheep2045 is offline   Reply With Quote
Old 03-02-2014, 11:24 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,433
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You appear to be using a custom recipe, make sure you are using the builtin one.
kovidgoyal is offline   Reply With Quote
 
Advertisement
Old 03-03-2014, 01:43 AM   #3
Sheep2045
Junior Member
Sheep2045 began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Mar 2014
Device: Kindle DXG
Thanks for the quick reply; the problem with the builtin recipe is that it doesn't support downloading paid content, only the articles freely available (which are vastly inferior).
I was hoping someone else was using the custom recipe (I don't think there is another one) and could shed some light on what's going wrong.
Judging by your last posts regarding the topic spiegel.de and recipes, they probably changed something on their website...

Regards,
Marc
Sheep2045 is offline   Reply With Quote
Old 03-03-2014, 01:49 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,433
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Looking at the builtin recipe, it appears to have a login, so it should be downloading full content, but as I haven't written that recipe, I cannot be sure.
kovidgoyal is offline   Reply With Quote
Old 03-04-2014, 01:25 PM   #5
Gustav M.
Junior Member
Gustav M. began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Mar 2014
Device: Sony PRS-T3
Hi!

I have the same problem with the builtin recipe for "Der Spiegel". It worked perfectly until about 10 days ago.

@Sheep2045 Just to clarify: The built in recipe did/does provide access to the paid content. Could it be that you refer to the Spiegel Online Recipe?

@kovidgoyal How did you know that Sheep2045 was using a custom recipe? Because of the "Using custom recipe" in the log? I also get this.

Gustav
Gustav M. is offline   Reply With Quote
Old 03-04-2014, 01:41 PM   #6
Sheep2045
Junior Member
Sheep2045 began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Mar 2014
Device: Kindle DXG
Yeah, I somehow copied the builtin "Der Spiegel" recipe to custom recipes. I just deleted it and tried the builtin, which fails in exactly the same way.

@Gustav M.: as written above, I got confused due to me having copied the builtin to custom recipes. I was referring to the builtin that provides access to paid content.

Last edited by Sheep2045; 03-04-2014 at 01:50 PM.
Sheep2045 is offline   Reply With Quote
Old 03-04-2014, 02:33 PM   #7
Gustav M.
Junior Member
Gustav M. began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Mar 2014
Device: Sony PRS-T3
Hi,

I just tried to contact the author of the recipe. Let's hope he can help!

If you are using a Kindle, have you tried
https://abo.spiegel.de/de/c/kindle

It says "reine Textversion", so it is probably not the same, but perhaps it will serve as a temporary solution for you.

Gustav
Gustav M. is offline   Reply With Quote
Old 03-04-2014, 03:07 PM   #8
aerodynamik
Enthusiast
aerodynamik doesn't litteraerodynamik doesn't litter
 
Posts: 41
Karma: 136
Join Date: Mar 2011
Device: Kindle Paperwhite
Hi Gustav, Hi everyone,

sorry, I didn't see this thread any earlier. I actually don't have an account anymore for Der Spiegel. I created the original recipe when there was a 2 week test period, and somebody here gave me their account (password temporary changed) to fix some issues.

I took a quick look without an account. What's obviously broken is the masthead URL. Before posting the completely fixed recipe I would like somebody to test the change here first or be so kind and give me temporarily their account information.

To test it out, customize the built in recipe and replace the masthead_url with 'http://upload.wikimedia.org/wikipedia/de/thumb/3/3e/Logo-der_spiegel.svg/200px-Logo-der_spiegel.svg.png'

This at least gets rid of the 404 error.

Cheers,
aero
aerodynamik is offline   Reply With Quote
Old 03-04-2014, 03:30 PM   #9
Sheep2045
Junior Member
Sheep2045 began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Mar 2014
Device: Kindle DXG
Hi aerodynamik,

thanks for your quick response! :-)
I just tried replacing the masthead url, but that unfortunately does not change anything regarding the 404 error (error remains exactly the same that I posted above).

I'll change my password and pm it to you.

Thanks a bunch for taking a look at this!


Cheers,
sheep2045

Last edited by Sheep2045; 03-04-2014 at 03:33 PM.
Sheep2045 is offline   Reply With Quote
Old 03-04-2014, 10:14 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,433
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by Gustav M. View Post
@kovidgoyal How did you know that Sheep2045 was using a custom recipe? Because of the "Using custom recipe" in the log? I also get this.
No, the line numbers in the error message dont match the line numbers in the builtin recipe. Using cutom recipe is always there if you download via the GUI, because that is the way the GUI works.
kovidgoyal is offline   Reply With Quote
Old 03-05-2014, 05:10 PM   #11
aerodynamik
Enthusiast
aerodynamik doesn't litteraerodynamik doesn't litter
 
Posts: 41
Karma: 136
Join Date: Mar 2011
Device: Kindle Paperwhite
Hi sheep2045,

thanks for your support and help with this. I used your account to analyze the issue.

I ran into a few issues tonight with timeouts also using my normal browser. Access to the mobile site, which is used by the recipe, is quite slow overall.

This shouldn't break the recipe however. I still get the 404 error for the login form URL, which definitely exists though. I found a dirty fix by simply ignoring the error (try / except around br.submit() in get_browser), however I am sure I can find a clean resolution for this error.

Give me a few more days. If you absolutely cannot wait try this for now instead of the br.submit() at around line 51 of the recipe:
PHP Code:
try:
    
br.submit()
except:
    
self.log('submit raised an error, let's ignore and continue') 
Hope this helps for now.

Cheers,
aero
aerodynamik is offline   Reply With Quote
Old 03-06-2014, 04:55 AM   #12
Sheep2045
Junior Member
Sheep2045 began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Mar 2014
Device: Kindle DXG
Thanks a bunch; this code:

PHP Code:
try:
    
br.submit()
except:
    
self.log('submit raised an error, ignore and continue'
works as a workaround, though I'm grateful that you're looking for a proper fix!

Cheers,
sheep2045
Sheep2045 is offline   Reply With Quote
Old 03-07-2014, 04:23 PM   #13
Gartenzwerg
Junior Member
Gartenzwerg began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Oct 2011
Device: Kindle Voyage
Just in case somebody is looking for a Kindle version: If you have a subscription, Der Spiegel is able to send a MOBI file directly to your Kindle. See https://abo.spiegel.de/de/c/kindle
Gartenzwerg is offline   Reply With Quote
Old 03-07-2014, 04:52 PM   #14
Sheep2045
Junior Member
Sheep2045 began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Mar 2014
Device: Kindle DXG
@Gartenzwerg: Yeah, thanks for the remainder, I think aero mentioned it as well. Unfortunately, that only works (to my understanding) without incurring further costs (for 3g delivery) if your Kindle is registered to a german Amazon account? But maybe I'm misinterpreting Amazon's terms of service regarding subscriptions?
(and it obviously won't matter if you have wifi)


Cheers,
sheep2045
Sheep2045 is offline   Reply With Quote
Old 03-07-2014, 05:05 PM   #15
aerodynamik
Enthusiast
aerodynamik doesn't litteraerodynamik doesn't litter
 
Posts: 41
Karma: 136
Join Date: Mar 2011
Device: Kindle Paperwhite
OK, here we go. They must have changed something that requires the login.html to have a valid referrer URL. Skip the try / except from before and just replace the getbrowser section with the follwing. The real change to the original recipe is the URL in br.open now containing the ?backUrl part.

PHP Code:
br BasicNewsRecipe.get_browser(self)
        if 
self.username is not None and self.password is not None:
            
br.open(self.PREFIX '/meinspiegel/login.html?backUrl=' self.PREFIX '/spiegel/print')
            
br.select_form(predicate=has_login_name)
            
br['f.loginName'] = self.username
            br
['f.password'] = self.password
            br
.submit()
        return 
br 
I took a quick look at the resulting eBook. Doesn't look as nice as I remember it, e.g. unnecessary information at the end of each article, headers not clipped correctly, some articles without a header at all.

Is this just in my Kindle output or do you experience the same?
aerodynamik is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Der Spiegel, Recipe Error Code:1 penguin Recipes 7 10-09-2012 05:44 PM
Recipe Request SPIEGEL/SPON Archive calyo98 Recipes 0 09-09-2012 01:11 PM
Recipe: DER SPIEGEL? ganymede Recipes 14 06-04-2012 12:49 PM
Economist Recipe Failing... awitko Recipes 2 11-07-2011 12:47 AM
Spiegel.de: Kleine Nabelschau zu eBooks auf der Buchmesse K-Thom Deutsches Forum 14 10-16-2009 02:25 PM


All times are GMT -4. The time now is 01:51 PM.


MobileRead.com is a privately owned, operated and funded community.