Custom Recipe returns list index out of range

[tonedeaf]

Hi all:

I've made a copy of the Scientific American recipe. I'd like to be able to download archived issues (I am a subscriber). I've tested the URL to make sure I can access it, here's the full error:

calibre, version 6.10.0 (win32, embedded-python: True)
Conversion error: Failed: Fetch news from Scientific American - monthly

Fetch news from Scientific American - monthly
Conversion options changed from defaults:
output_profile: 'kobo'
verbose: 2
Resolved conversion options
calibre version: 6.10.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000015B9AFE68C0>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KoboReaderOutput object at 0x0000015B9AFE6F50>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1001
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Traceback (most recent call last):
File "runpy.py", line 196, in _run_module_as_main
File "runpy.py", line 86, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 31, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1108, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 138, in convert
File "calibre\web\feeds\news.py", line 1058, in download
File "calibre\web\feeds\news.py", line 1227, in build_index
File "<string>", line 65, in parse_index
IndexError: list index out of range


I've made one change in my recipe. I see the post will have the indents stripped out, but they are there in the original recipe. see below:

def parse_index(self):
# Get the cover, date and issue URL
root = self.index_to_soup(
'https://www.scientificamerican.com/magazine/sa/2022/12-01', as_tree=True)
select = Select(root)
self.cover_url = [x.get('src', '') for x in select('main .store-listing__img img')][0]
url = [x.get('href', '') for x in select('main .store-listing__img a')][0]
url = absurl(url)

I've also tried this with '/' at the end of the URL, same outcome.

Should this work? to the naked eye, the archived editions look to be formatted the same as the current issue.
Any help would be appreciated.
Thanks
Kevin

[kovidgoyal]

If the underlying markup is identical it will work of it isnt it wont and you will need to adjust the logic in parse_index accordingly.

[tonedeaf]

thanks for the reply, I'll start debugging, seems there are differences between the current issue markup and archived issues. My python is basic at best, I'll enjoy learning a new language :-) thanks for all the work you do with Calibre, I've used it off and on for years, it's a great application.

[unkn0wn]

url = [x.get('href', '') for x in select('main .store-listing__img a')][0]
url = absurl(url)

try this - replace the url = absurl(url) with url = your link