The Hindu recipe is not recognizing articles from a section

[unkn0wn]

Found section: Sci-tech & Agri https://www.thehindu.com/todays-paper/tp-features/tp-sci-tech-and-agri/
Found section: Others https://www.thehindu.com/todays-paper/tp-miscellaneous/tp-others/
		Found article: Celebrations break out as farmers from Punjab, Haryana reach home
			 https://www.thehindu.com/todays-paper/tp-miscellaneous/tp-others/celebrations-break-out-as-farmers-from-punjab-haryana-reach-home/article37936952.ece
		Found article: Trinamool promises Rs. 5,000 per month for women in Goa
               .
               .
               .

It finds the section tp-sci-tech-and-agri but fails to look for the articles in that section.

all other sections and articles load perfectly.

https://www.thehindu.com/archive/print/2021/12/12/

https://github.com/kovidgoyal/calibr...s/hindu.recipe


EDIT
Okay, looks like the link doesn't open and show article links, for calibre to fetch..
https://www.thehindu.com/todays-pape...tech-and-agri/

but those articles links are present in today's paper list link

[kovidgoyal]

Thats because that page is empty on the website. Go to the scitech and agri page and see for yourself.

[unkn0wn]

mechanize._response.httperror_seek_wrapper: HTTP Error 403: Forbidden

today i got this error.. website is working fine.. I didn't want to start a new thread!

Spoiler:

Fetch news from The Hindu
Integration status: True
Conversion options changed from defaults:
output_profile: 'kindle_oasis'
verbose: 2
Resolved conversion options
calibre version: 5.33.0
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x000002650398A1C0>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.KindleOasisOutput object at 0x000002650398A760>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': None,
'transform_html_rules': None,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Downloading recipe urn: custom:1000
Using user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Traceback (most recent call last):
File "runpy.py", line 194, in _run_module_as_main
File "runpy.py", line 87, in _run_code
File "site.py", line 82, in <module>
File "site.py", line 77, in main
File "site.py", line 49, in run_entry_point
File "calibre\utils\ipc\worker.py", line 216, in main
File "calibre\gui2\convert\gui_conversion.py", line 34, in gui_convert_recipe
File "calibre\gui2\convert\gui_conversion.py", line 28, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1111, in run
File "calibre\customize\conversion.py", line 244, in __call__
File "calibre\ebooks\conversion\plugins\recipe_input.py ", line 138, in convert
File "calibre\web\feeds\news.py", line 1056, in download
File "calibre\web\feeds\news.py", line 1225, in build_index
File "<string>", line 102, in parse_index
File "calibre\web\feeds\news.py", line 706, in index_to_soup
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 403: Forbidden

[kovidgoyal]

That will be because the website is using some kind of bot detection. Someone with more time than I do will need to figure out what is needed to bypass the bot detection.

calibre-debug -c 'from calibre import browser; br = browser(); br.open("https://www.thehindu.com/todays-paper/")'     
Traceback (most recent call last):
  File "/usr/bin/calibre-debug", line 21, in <module>
    sys.exit(main())
  File "/home/kovid/work/calibre/src/calibre/debug.py", line 272, in main
    exec(opts.command)
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.10/site-packages/mechanize/_mechanize.py", line 257, in open
    return self._mech_open(url_or_request, data, timeout=timeout)
  File "/usr/lib/python3.10/site-packages/mechanize/_mechanize.py", line 313, in _mech_open
    raise response
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 403: Forbidden

[kovidgoyal]

Actually this should take care of it: https://github.com/kovidgoyal/calibr...b687d54a36e88c

[unkn0wn]

Thank you. You are a magician..