Calibre download of The Spectator not working

davidwright · 03-05-2020, 10:09 AM

Hi - after several years of seamless operation Calibre no longer can download my Spectator subscription onto my iMac. I have changed nothing , and my other subscription (The Economist) still works fine. Any ideas please?
Thanks

Fetch news from Spectator Magazine
Conversion options changed from defaults:
output_profile: u'kindle_pw3'
verbose: 2
Resolved conversion options
calibre version: 4.11.2
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': u'original',
'chapter': None,
'chapter_mark': u'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x10a6bfb50>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': u'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x10a6c8390>,
'page_breaks_before': None,
'personal_doc': u'[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': u'',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': u'',
'sr1_search': u'',
'sr2_replace': u'',
'sr2_search': u'',
'sr3_replace': u'',
'sr3_search': u'',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1000
Using user agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Python function terminated unexpectedly: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 187, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 121, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 209, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 36, in gui_convert_recipe
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 28, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 1110, in run
File "site-packages/calibre/customize/conversion.py", line 246, in __call__
File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 139, in convert
File "site-packages/calibre/web/feeds/news.py", line 1022, in download
File "site-packages/calibre/web/feeds/news.py", line 1190, in build_index
File "<string>", line 54, in parse_index
TypeError: 'NoneType' object has no attribute '__getitem__'

kovidgoyal · 03-05-2020, 08:43 PM

The spectator website is undergoing a redesign, going there with a browser redirects me to a beta version (beta.spectator.co.uk). SO once the redesign is finished the recipe in calibre will need to be updated for the new design. Keep an eye on it and when the redesign is over, post here and I will update the recipe.

davidwright · 03-06-2020, 12:45 AM

That's great Kovid - Thank you

davidwright · 03-12-2020, 01:01 PM

Hi Kovid - It looks like the Spectator is no longer on Beta version.....so hopefully you will be able to redo the recipe. Thanks in advance. Dave

kovidgoyal · 03-15-2020, 07:29 AM

https://github.com/kovidgoyal/calibr...82f9a70b1a0290

stupid website decided to load all content using javascript, so had to jump through a few hoops to reverse engineer it.

davidwright · 03-19-2020, 03:49 AM

Hi again Kovid - Thanks for the prompt reply.
However, I still can't seem to get it to work.
I deleted the old "Spectator" recipe, then added the new one (from "Add or edit custom news source" / "Customise built in recipe" / "English(429)" / "The Spectator" ) but i get the following error message -

Code:

Fetch news from Spectator Magazine
Conversion options changed from defaults:
  output_profile: u'kindle_pw3'
  verbose: 2
Resolved conversion options
calibre version: 4.12.0
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': u'original',
 'chapter': None,
 'chapter_mark': u'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'dehyphenate': True,
 'delete_blank_paragraphs': True,
 'disable_font_rescaling': False,
 'dont_compress': False,
 'dont_download_recipe': False,
 'duplicate_links_in_toc': False,
 'embed_all_fonts': False,
 'embed_font_family': None,
 'enable_heuristics': False,
 'expand_css': False,
 'extra_css': None,
 'extract_to': None,
 'filter_css': None,
 'fix_indents': True,
 'font_size_mapping': None,
 'format_scene_breaks': True,
 'html_unwrap_factor': 0.4,
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x112022c50>,
 'insert_blank_line': False,
 'insert_blank_line_size': 0.5,
 'insert_metadata': False,
 'isbn': None,
 'italicize_common_cases': True,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'markup_chapter_headings': True,
 'max_toc_links': 50,
 'minimum_line_height': 120.0,
 'mobi_file_type': u'old',
 'mobi_ignore_margins': False,
 'mobi_keep_original_images': False,
 'mobi_toc_at_start': False,
 'no_chapters_in_toc': False,
 'no_inline_navbars': True,
 'no_inline_toc': False,
 'output_profile': <calibre.customize.profiles.KindlePaperWhite3Output object at 0x112030490>,
 'page_breaks_before': None,
 'personal_doc': u'[PDOC]',
 'prefer_author_sort': False,
 'prefer_metadata_cover': False,
 'pretty_print': False,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_fake_margins': True,
 'remove_first_image': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'renumber_headings': True,
 'replace_scene_breaks': u'',
 'search_replace': None,
 'series': None,
 'series_index': None,
 'share_not_sync': False,
 'smarten_punctuation': False,
 'sr1_replace': u'',
 'sr1_search': u'',
 'sr2_replace': u'',
 'sr2_search': u'',
 'sr3_replace': u'',
 'sr3_search': u'',
 'start_reading_at': None,
 'subset_embedded_fonts': False,
 'tags': None,
 'test': False,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'toc_title': None,
 'transform_css_rules': None,
 'unsmarten_punctuation': False,
 'unwrap_lines': True,
 'use_auto_toc': False,
 'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1002
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Got Spectator data: {u'contentEnvironment': u'live', u'apiSecret': u'cs4b204a44bc4e88da701dbc9b', u'magazineIssueContentUrl': u'https://cdn.contentstack.io/v3/content_types/magazine_issue', u'contentUrl': u'https://cdn.contentstack.io/v3/content_types/article', u'siteUrl': u'https://www.spectator.co.uk', u'apiKey': u'bltf04078f3cf7a9c30'}
Fetching: https://cdn.contentstack.io/v3/conte...e&limit=1&only[BASE][]=url
Downloading issue: 21-03-2020
Fetching: https://cdn.contentstack.io/v3/content_types/magazine_issue/entries?environment=live&limit=1&query={"url": "/magazine/21-03-2020"}
Python function terminated unexpectedly: HTTP Error 400: Bad Request
Traceback (most recent call last):
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 187, in main
    return run_entry_point()
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 121, in run_entry_point
    return getattr(pmod, func)()
  File "site-packages/calibre/utils/ipc/worker.py", line 209, in main
  File "site-packages/calibre/gui2/convert/gui_conversion.py", line 36, in gui_convert_recipe
  File "site-packages/calibre/gui2/convert/gui_conversion.py", line 28, in gui_convert
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1110, in run
  File "site-packages/calibre/customize/conversion.py", line 246, in __call__
  File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 139, in convert
  File "site-packages/calibre/web/feeds/news.py", line 1022, in download
  File "site-packages/calibre/web/feeds/news.py", line 1190, in build_index
  File "<string>", line 76, in parse_index
  File "<string>", line 59, in get_result
  File "site-packages/mechanize/_mechanize.py", line 238, in open_novisit
  File "site-packages/mechanize/_mechanize.py", line 309, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 400: Bad Request

I had a quick check on the "github" link you provided to check that the recipe I have downloaded is the correct new one, and I think it looks the same (but I am not very computer savvy) .

Any ideas what I may be doing wrong?
(I have recently resubscribed to the magazine - maybe that means that I have a new password / username " Does that make any difference? On Calibre I can't find anywhere to input the username / password for the recipe anyway.)

Sorry to be a pain.
Thanks

kovidgoyal · 03-19-2020, 04:45 AM

Just use the normal builtin recipe, it is auto-updated.

davidwright · 03-19-2020, 07:50 AM

I am using the built in recipe choosing from the drop down menu within Calibre New Sources - but I still get the following error message -
(I am using a Mac 10.15.3 if that makes any difference)
Thanks again.
Dave

Fetch news from Spectator Magazine
Conversion options changed from defaults:
output_profile: u'kindle_pw3'
verbose: 2
Resolved conversion options
calibre version: 4.12.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': u'original',
'chapter': None,
'chapter_mark': u'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x10899ec90>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': u'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x1089aa4d0>,
'page_breaks_before': None,
'personal_doc': u'[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': u'',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': u'',
'sr1_search': u'',
'sr2_replace': u'',
'sr2_search': u'',
'sr3_replace': u'',
'sr3_search': u'',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: builtin:spectator_magazine
Trying to get latest version of recipe: spectator_magazine
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
Got Spectator data: {u'contentEnvironment': u'live', u'apiSecret': u'cs4b204a44bc4e88da701dbc9b', u'contentUrl': u'https://cdn.contentstack.io/v3/content_types/article', u'apiKey': u'bltf04078f3cf7a9c30', u'siteUrl': u'https://www.spectator.co.uk', u'magazineIssueContentUrl': u'https://cdn.contentstack.io/v3/content_types/magazine_issue'}
Fetching: https://cdn.contentstack.io/v3/conte...e&limit=1&only[BASE][]=url
Downloading issue: 21-03-2020
Fetching: https://cdn.contentstack.io/v3/content_types/magazine_issue/entries?environment=live&limit=1&query={"url": "/magazine/21-03-2020"}
Python function terminated unexpectedly: HTTP Error 400: Bad Request
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 187, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 121, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 209, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 36, in gui_convert_recipe
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 28, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 1110, in run
File "site-packages/calibre/customize/conversion.py", line 246, in __call__
File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 139, in convert
File "site-packages/calibre/web/feeds/news.py", line 1022, in download
File "site-packages/calibre/web/feeds/news.py", line 1190, in build_index
File "<string>", line 76, in parse_index
File "<string>", line 59, in get_result
File "site-packages/mechanize/_mechanize.py", line 238, in open_novisit
File "site-packages/mechanize/_mechanize.py", line 309, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 400: Bad Request

kovidgoyal · 03-19-2020, 10:16 AM

Works for me, that's an error from the server, possibly because of some geo restrictions/different behavior in different countries. Without being able to reproduce I'm afraid there is not much I can do, sorry.

davidwright · 03-19-2020, 11:54 AM

No problem. thanks for your help.

davidwright · 04-03-2020, 10:13 AM

Hi Kovid - Just to let you know that the recipe for Spectator download seems to be working again on my computer.(The only thing I have changed was an upgrade to 10.15.4 MacOS Catalina which came available last week)
USD 25 sent to you by Paypal.
Thanks for all your hard work.
Have a nice weekend
dave

kovidgoyal · 04-03-2020, 12:13 PM

Glad to hear it and thanks for the support

nano5 · 04-24-2020, 09:54 PM

It is working well, thank you!

Is it possible to retrieve the previous issue, i.e. 18-Apr, with cover article from Niall Ferguson. Thanks again.

davidwright · 07-02-2020, 01:03 AM

Hello again Kovid.
My download fo the Economist magazine is now not working. (as of several weeks).
Any chance you can have a look at it please? Thanks Dave
Fetch news from The Economist
Conversion options changed from defaults:
output_profile: u'kindle_pw3'
verbose: 2
Resolved conversion options
calibre version: 4.19.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': u'original',
'chapter': None,
'chapter_mark': u'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_compress': False,
'dont_download_recipe': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x112af8190>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'mobi_file_type': u'old',
'mobi_ignore_margins': False,
'mobi_keep_original_images': False,
'mobi_toc_at_start': False,
'no_chapters_in_toc': False,
'no_inline_navbars': True,
'no_inline_toc': False,
'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x112af8990>,
'page_breaks_before': None,
'personal_doc': u'[PDOC]',
'prefer_author_sort': False,
'prefer_metadata_cover': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': u'',
'search_replace': None,
'series': None,
'series_index': None,
'share_not_sync': False,
'smarten_punctuation': False,
'sr1_replace': u'',
'sr1_search': u'',
'sr2_replace': u'',
'sr2_search': u'',
'sr3_replace': u'',
'sr3_search': u'',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1001
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
Kindle Output profile being used, reducing image quality to keep file size below amazon email threshold
Python function terminated unexpectedly: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 187, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 121, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 209, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 36, in gui_convert_recipe
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 28, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 1110, in run
File "site-packages/calibre/customize/conversion.py", line 246, in __call__
File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 139, in convert
File "site-packages/calibre/web/feeds/news.py", line 1028, in download
File "site-packages/calibre/web/feeds/news.py", line 1196, in build_index
File "<string>", line 245, in parse_index
__builtin__.NoArticles: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

kovidgoyal · 07-02-2020, 01:05 AM

use the builtin recipe not your custom one.

03-05-2020, 10:09 AM	#1
davidwright Junior Member Posts: 8 Karma: 10 Join Date: Mar 2020 Device: kindle paperwhite	Calibre download of The Spectator not working Hi - after several years of seamless operation Calibre no longer can download my Spectator subscription onto my iMac. I have changed nothing , and my other subscription (The Economist) still works fine. Any ideas please? Thanks Fetch news from Spectator Magazine Conversion options changed from defaults: output_profile: u'kindle_pw3' verbose: 2 Resolved conversion options calibre version: 4.11.2 {'asciiize': False, 'author_sort': None, 'authors': None, 'base_font_size': 0, 'book_producer': None, 'change_justification': u'original', 'chapter': None, 'chapter_mark': u'pagebreak', 'comments': None, 'cover': None, 'debug_pipeline': None, 'dehyphenate': True, 'delete_blank_paragraphs': True, 'disable_font_rescaling': False, 'dont_compress': False, 'dont_download_recipe': False, 'duplicate_links_in_toc': False, 'embed_all_fonts': False, 'embed_font_family': None, 'enable_heuristics': False, 'expand_css': False, 'extra_css': None, 'extract_to': None, 'filter_css': None, 'fix_indents': True, 'font_size_mapping': None, 'format_scene_breaks': True, 'html_unwrap_factor': 0.4, 'input_encoding': None, 'input_profile': <calibre.customize.profiles.InputProfile object at 0x10a6bfb50>, 'insert_blank_line': False, 'insert_blank_line_size': 0.5, 'insert_metadata': False, 'isbn': None, 'italicize_common_cases': True, 'keep_ligatures': False, 'language': None, 'level1_toc': None, 'level2_toc': None, 'level3_toc': None, 'line_height': 0, 'linearize_tables': False, 'lrf': False, 'margin_bottom': 5.0, 'margin_left': 5.0, 'margin_right': 5.0, 'margin_top': 5.0, 'markup_chapter_headings': True, 'max_toc_links': 50, 'minimum_line_height': 120.0, 'mobi_file_type': u'old', 'mobi_ignore_margins': False, 'mobi_keep_original_images': False, 'mobi_toc_at_start': False, 'no_chapters_in_toc': False, 'no_inline_navbars': True, 'no_inline_toc': False, 'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x10a6c8390>, 'page_breaks_before': None, 'personal_doc': u'[PDOC]', 'prefer_author_sort': False, 'prefer_metadata_cover': False, 'pretty_print': False, 'pubdate': None, 'publisher': None, 'rating': None, 'read_metadata_from_opf': None, 'remove_fake_margins': True, 'remove_first_image': False, 'remove_paragraph_spacing': False, 'remove_paragraph_spacing_indent_size': 1.5, 'renumber_headings': True, 'replace_scene_breaks': u'', 'search_replace': None, 'series': None, 'series_index': None, 'share_not_sync': False, 'smarten_punctuation': False, 'sr1_replace': u'', 'sr1_search': u'', 'sr2_replace': u'', 'sr2_search': u'', 'sr3_replace': u'', 'sr3_search': u'', 'start_reading_at': None, 'subset_embedded_fonts': False, 'tags': None, 'test': False, 'timestamp': None, 'title': None, 'title_sort': None, 'toc_filter': None, 'toc_threshold': 6, 'toc_title': None, 'transform_css_rules': None, 'unsmarten_punctuation': False, 'unwrap_lines': True, 'use_auto_toc': False, 'verbose': 2} InputFormatPlugin: Recipe Input running Downloading recipe urn: custom:1000 Using user agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36 Python function terminated unexpectedly: 'NoneType' object has no attribute '__getitem__' Traceback (most recent call last): File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 187, in main return run_entry_point() File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 121, in run_entry_point return getattr(pmod, func)() File "site-packages/calibre/utils/ipc/worker.py", line 209, in main File "site-packages/calibre/gui2/convert/gui_conversion.py", line 36, in gui_convert_recipe File "site-packages/calibre/gui2/convert/gui_conversion.py", line 28, in gui_convert File "site-packages/calibre/ebooks/conversion/plumber.py", line 1110, in run File "site-packages/calibre/customize/conversion.py", line 246, in __call__ File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 139, in convert File "site-packages/calibre/web/feeds/news.py", line 1022, in download File "site-packages/calibre/web/feeds/news.py", line 1190, in build_index File "<string>", line 54, in parse_index TypeError: 'NoneType' object has no attribute '__getitem__'

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
The Spectator - how do I download back issues?	lennyw	Recipes	2	03-28-2017 04:12 PM
Business Spectator	soctec	Recipes	0	09-27-2012 03:29 AM
Calibre, download cover, metadata not working	mufc	Calibre	9	06-19-2011 03:25 PM
[Old thread] Issues to download news with Calibre - keeps working, but nothing hap...	davidoffpopoff	Recipes	7	05-09-2011 05:37 PM
Calibre-Wall Street Journal download not working	moosejons_dad	Calibre	0	09-24-2008 09:29 AM

03-05-2020, 08:43 PM	#2
kovidgoyal creator of calibre Posts: 43,852 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The spectator website is undergoing a redesign, going there with a browser redirects me to a beta version (beta.spectator.co.uk). SO once the redesign is finished the recipe in calibre will need to be updated for the new design. Keep an eye on it and when the redesign is over, post here and I will update the recipe.

03-06-2020, 12:45 AM	#3
davidwright Junior Member Posts: 8 Karma: 10 Join Date: Mar 2020 Device: kindle paperwhite	That's great Kovid - Thank you

03-12-2020, 01:01 PM	#4
davidwright Junior Member Posts: 8 Karma: 10 Join Date: Mar 2020 Device: kindle paperwhite	Hi Kovid - It looks like the Spectator is no longer on Beta version.....so hopefully you will be able to redo the recipe. Thanks in advance. Dave

03-15-2020, 07:29 AM	#5
kovidgoyal creator of calibre Posts: 43,852 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	https://github.com/kovidgoyal/calibr...82f9a70b1a0290 stupid website decided to load all content using javascript, so had to jump through a few hoops to reverse engineer it.

03-19-2020, 04:45 AM	#7
kovidgoyal creator of calibre Posts: 43,852 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Just use the normal builtin recipe, it is auto-updated.

03-19-2020, 07:50 AM	#8
davidwright Junior Member Posts: 8 Karma: 10 Join Date: Mar 2020 Device: kindle paperwhite	I am using the built in recipe choosing from the drop down menu within Calibre New Sources - but I still get the following error message - (I am using a Mac 10.15.3 if that makes any difference) Thanks again. Dave Fetch news from Spectator Magazine Conversion options changed from defaults: output_profile: u'kindle_pw3' verbose: 2 Resolved conversion options calibre version: 4.12.0 {'asciiize': False, 'author_sort': None, 'authors': None, 'base_font_size': 0, 'book_producer': None, 'change_justification': u'original', 'chapter': None, 'chapter_mark': u'pagebreak', 'comments': None, 'cover': None, 'debug_pipeline': None, 'dehyphenate': True, 'delete_blank_paragraphs': True, 'disable_font_rescaling': False, 'dont_compress': False, 'dont_download_recipe': False, 'duplicate_links_in_toc': False, 'embed_all_fonts': False, 'embed_font_family': None, 'enable_heuristics': False, 'expand_css': False, 'extra_css': None, 'extract_to': None, 'filter_css': None, 'fix_indents': True, 'font_size_mapping': None, 'format_scene_breaks': True, 'html_unwrap_factor': 0.4, 'input_encoding': None, 'input_profile': <calibre.customize.profiles.InputProfile object at 0x10899ec90>, 'insert_blank_line': False, 'insert_blank_line_size': 0.5, 'insert_metadata': False, 'isbn': None, 'italicize_common_cases': True, 'keep_ligatures': False, 'language': None, 'level1_toc': None, 'level2_toc': None, 'level3_toc': None, 'line_height': 0, 'linearize_tables': False, 'lrf': False, 'margin_bottom': 5.0, 'margin_left': 5.0, 'margin_right': 5.0, 'margin_top': 5.0, 'markup_chapter_headings': True, 'max_toc_links': 50, 'minimum_line_height': 120.0, 'mobi_file_type': u'old', 'mobi_ignore_margins': False, 'mobi_keep_original_images': False, 'mobi_toc_at_start': False, 'no_chapters_in_toc': False, 'no_inline_navbars': True, 'no_inline_toc': False, 'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x1089aa4d0>, 'page_breaks_before': None, 'personal_doc': u'[PDOC]', 'prefer_author_sort': False, 'prefer_metadata_cover': False, 'pretty_print': False, 'pubdate': None, 'publisher': None, 'rating': None, 'read_metadata_from_opf': None, 'remove_fake_margins': True, 'remove_first_image': False, 'remove_paragraph_spacing': False, 'remove_paragraph_spacing_indent_size': 1.5, 'renumber_headings': True, 'replace_scene_breaks': u'', 'search_replace': None, 'series': None, 'series_index': None, 'share_not_sync': False, 'smarten_punctuation': False, 'sr1_replace': u'', 'sr1_search': u'', 'sr2_replace': u'', 'sr2_search': u'', 'sr3_replace': u'', 'sr3_search': u'', 'start_reading_at': None, 'subset_embedded_fonts': False, 'tags': None, 'test': False, 'timestamp': None, 'title': None, 'title_sort': None, 'toc_filter': None, 'toc_threshold': 6, 'toc_title': None, 'transform_css_rules': None, 'unsmarten_punctuation': False, 'unwrap_lines': True, 'use_auto_toc': False, 'verbose': 2} InputFormatPlugin: Recipe Input running Downloading recipe urn: builtin:spectator_magazine Trying to get latest version of recipe: spectator_magazine Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36 Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36 Got Spectator data: {u'contentEnvironment': u'live', u'apiSecret': u'cs4b204a44bc4e88da701dbc9b', u'contentUrl': u'https://cdn.contentstack.io/v3/content_types/article', u'apiKey': u'bltf04078f3cf7a9c30', u'siteUrl': u'https://www.spectator.co.uk', u'magazineIssueContentUrl': u'https://cdn.contentstack.io/v3/content_types/magazine_issue'} Fetching: https://cdn.contentstack.io/v3/conte...e&limit=1&only[BASE][]=url Downloading issue: 21-03-2020 Fetching: https://cdn.contentstack.io/v3/content_types/magazine_issue/entries?environment=live&limit=1&query={"url": "/magazine/21-03-2020"} Python function terminated unexpectedly: HTTP Error 400: Bad Request Traceback (most recent call last): File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 187, in main return run_entry_point() File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 121, in run_entry_point return getattr(pmod, func)() File "site-packages/calibre/utils/ipc/worker.py", line 209, in main File "site-packages/calibre/gui2/convert/gui_conversion.py", line 36, in gui_convert_recipe File "site-packages/calibre/gui2/convert/gui_conversion.py", line 28, in gui_convert File "site-packages/calibre/ebooks/conversion/plumber.py", line 1110, in run File "site-packages/calibre/customize/conversion.py", line 246, in __call__ File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 139, in convert File "site-packages/calibre/web/feeds/news.py", line 1022, in download File "site-packages/calibre/web/feeds/news.py", line 1190, in build_index File "<string>", line 76, in parse_index File "<string>", line 59, in get_result File "site-packages/mechanize/_mechanize.py", line 238, in open_novisit File "site-packages/mechanize/_mechanize.py", line 309, in _mech_open mechanize._response.httperror_seek_wrapper: HTTP Error 400: Bad Request

03-19-2020, 10:16 AM	#9
kovidgoyal creator of calibre Posts: 43,852 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Works for me, that's an error from the server, possibly because of some geo restrictions/different behavior in different countries. Without being able to reproduce I'm afraid there is not much I can do, sorry.

03-19-2020, 11:54 AM	#10
davidwright Junior Member Posts: 8 Karma: 10 Join Date: Mar 2020 Device: kindle paperwhite	No problem. thanks for your help.

04-03-2020, 10:13 AM	#11
davidwright Junior Member Posts: 8 Karma: 10 Join Date: Mar 2020 Device: kindle paperwhite	Hi Kovid - Just to let you know that the recipe for Spectator download seems to be working again on my computer.(The only thing I have changed was an upgrade to 10.15.4 MacOS Catalina which came available last week) USD 25 sent to you by Paypal. Thanks for all your hard work. Have a nice weekend dave

04-03-2020, 12:13 PM	#12
kovidgoyal creator of calibre Posts: 43,852 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Glad to hear it and thanks for the support

04-24-2020, 09:54 PM	#13
nano5 Connoisseur Posts: 84 Karma: 2136220 Join Date: May 2019 Device: Kindle	It is working well, thank you! Is it possible to retrieve the previous issue, i.e. 18-Apr, with cover article from Niall Ferguson. Thanks again.

07-02-2020, 01:03 AM	#14
davidwright Junior Member Posts: 8 Karma: 10 Join Date: Mar 2020 Device: kindle paperwhite	Hello again Kovid. My download fo the Economist magazine is now not working. (as of several weeks). Any chance you can have a look at it please? Thanks Dave Fetch news from The Economist Conversion options changed from defaults: output_profile: u'kindle_pw3' verbose: 2 Resolved conversion options calibre version: 4.19.0 {'asciiize': False, 'author_sort': None, 'authors': None, 'base_font_size': 0, 'book_producer': None, 'change_justification': u'original', 'chapter': None, 'chapter_mark': u'pagebreak', 'comments': None, 'cover': None, 'debug_pipeline': None, 'dehyphenate': True, 'delete_blank_paragraphs': True, 'disable_font_rescaling': False, 'dont_compress': False, 'dont_download_recipe': False, 'duplicate_links_in_toc': False, 'embed_all_fonts': False, 'embed_font_family': None, 'enable_heuristics': False, 'expand_css': False, 'extra_css': None, 'extract_to': None, 'filter_css': None, 'fix_indents': True, 'font_size_mapping': None, 'format_scene_breaks': True, 'html_unwrap_factor': 0.4, 'input_encoding': None, 'input_profile': <calibre.customize.profiles.InputProfile object at 0x112af8190>, 'insert_blank_line': False, 'insert_blank_line_size': 0.5, 'insert_metadata': False, 'isbn': None, 'italicize_common_cases': True, 'keep_ligatures': False, 'language': None, 'level1_toc': None, 'level2_toc': None, 'level3_toc': None, 'line_height': 0, 'linearize_tables': False, 'lrf': False, 'margin_bottom': 5.0, 'margin_left': 5.0, 'margin_right': 5.0, 'margin_top': 5.0, 'markup_chapter_headings': True, 'max_toc_links': 50, 'minimum_line_height': 120.0, 'mobi_file_type': u'old', 'mobi_ignore_margins': False, 'mobi_keep_original_images': False, 'mobi_toc_at_start': False, 'no_chapters_in_toc': False, 'no_inline_navbars': True, 'no_inline_toc': False, 'output_profile': <calibre.customize.profiles.KindlePaperWhite3Outpu t object at 0x112af8990>, 'page_breaks_before': None, 'personal_doc': u'[PDOC]', 'prefer_author_sort': False, 'prefer_metadata_cover': False, 'pretty_print': False, 'pubdate': None, 'publisher': None, 'rating': None, 'read_metadata_from_opf': None, 'remove_fake_margins': True, 'remove_first_image': False, 'remove_paragraph_spacing': False, 'remove_paragraph_spacing_indent_size': 1.5, 'renumber_headings': True, 'replace_scene_breaks': u'', 'search_replace': None, 'series': None, 'series_index': None, 'share_not_sync': False, 'smarten_punctuation': False, 'sr1_replace': u'', 'sr1_search': u'', 'sr2_replace': u'', 'sr2_search': u'', 'sr3_replace': u'', 'sr3_search': u'', 'start_reading_at': None, 'subset_embedded_fonts': False, 'tags': None, 'test': False, 'timestamp': None, 'title': None, 'title_sort': None, 'toc_filter': None, 'toc_threshold': 6, 'toc_title': None, 'transform_css_rules': None, 'unsmarten_punctuation': False, 'unwrap_lines': True, 'use_auto_toc': False, 'verbose': 2} InputFormatPlugin: Recipe Input running Downloading recipe urn: custom:1001 Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36 Kindle Output profile being used, reducing image quality to keep file size below amazon email threshold Python function terminated unexpectedly: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated. Traceback (most recent call last): File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 187, in main return run_entry_point() File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 121, in run_entry_point return getattr(pmod, func)() File "site-packages/calibre/utils/ipc/worker.py", line 209, in main File "site-packages/calibre/gui2/convert/gui_conversion.py", line 36, in gui_convert_recipe File "site-packages/calibre/gui2/convert/gui_conversion.py", line 28, in gui_convert File "site-packages/calibre/ebooks/conversion/plumber.py", line 1110, in run File "site-packages/calibre/customize/conversion.py", line 246, in __call__ File "site-packages/calibre/ebooks/conversion/plugins/recipe_input.py", line 139, in convert File "site-packages/calibre/web/feeds/news.py", line 1028, in download File "site-packages/calibre/web/feeds/news.py", line 1196, in build_index File "<string>", line 245, in parse_index __builtin__.NoArticles: Could not find any articles, either the economist.com server is having trouble and you should try later or the website format has changed and the recipe needs to be updated.

07-02-2020, 01:05 AM	#15
kovidgoyal creator of calibre Posts: 43,852 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	use the builtin recipe not your custom one.