NHK News in Japanese recipe inconsistent

rasteps · 12-21-2022, 03:31 AM

There are very few recipes in Japanese so I wrote a recipe which scrapes the NHK news site. But it sometimes downloads and sometimes hang and I'm not sure why it's so inconsistent. Does anyone have any advice?

Code:

from calibre.web.feeds.news import BasicNewsRecipe


class ReutersJa(BasicNewsRecipe):

# feed source: https://www.nhk.or.jp/toppage/rss/index.html

    title = 'NHK News'
    description = 'NHK News in Japanese'
    __author__ = 'Richard A. Steps'
    use_embedded_content = False
    language = 'ja'
    max_articles_per_feed = 30
    remove_javascript = True
    auto_cleanup = True

    feeds = [(
        '主要ニュース', 'https://www.nhk.or.jp/rss/news/cat0.xml?format=xml'),
         ('社会', 'https://www.nhk.or.jp/rss/news/cat1.xml?format=xml'),
         ('科学・医療', 'https://www.nhk.or.jp/rss/news/cat3.xml?format=xml'),
         ('政治', 'https://www.nhk.or.jp/rss/news/cat4.xml?format=xml'),
         ('経済', 'https://www.nhk.or.jp/rss/news/cat5.xml?format=xml'),
         ('国際', 'https://www.nhk.or.jp/rss/news/cat6.xml?format=xml'),
         ('スポーツ', 'https://www.nhk.or.jp/rss/news/cat7.xml?format=xml'),
         ('文化・エンタメ', 'https://www.nhk.or.jp/rss/news/cat2.xml?format=xml')
    ]

kovidgoyal · 12-21-2022, 06:22 AM

Post the log when it hangs, it will most likely be some anti bot tech on the servers.

rasteps · 12-22-2022, 01:04 AM

It doesn't generate a log when it hangs. It just sits in the queue until it is canceled.

kovidgoyal · 12-22-2022, 01:13 AM

click the rotating jobs button in the bottom right corner to find the log

rasteps · 12-23-2022, 08:51 AM

Quote:

Originally Posted by kovidgoyal

click the rotating jobs button in the bottom right corner to find the log

I just realized that I could just click on the stopped job to get the log. The following in a truncated version of the original recipe that only gets the Main News section. This had been in the queue for over 17 minutes. It usually takes less than a minute to complete.

--------------------------------START OF LOG -------------------------------------

Spoiler:

kovidgoyal · 12-23-2022, 10:28 PM

Yeah it looks like bot protection. Sometimes websites will send one byte at a time with intervals when they think a bot is scraping them to prevent timeout errors and keep the bot hanging.

rasteps · 12-25-2022, 09:15 AM

Sounds like there's no workaround for this. Thanks for looking into it for me.

Can I submit this recipe as it stands then?

kovidgoyal · 12-25-2022, 10:42 AM

You could try using

Code:

    def get_browser(self):
        return BasicNewsRecipe.get_browser(self, user_agent='common_words/based')

that might make a difference. And sure you can submit the recipe.

rasteps · 12-28-2022, 05:53 PM

I added it just above the Feeds section and got the following error:

Code:

calibre, version 6.10.0
ERROR: Invalid recipe: Failed to compile the recipe, with syntax error: unexpected indent (<string>, line 18)

Where is the best place to put those lines? And so I know what I'm looking for, what are they supposed to do?

Quote:

Originally Posted by kovidgoyal

You could try using

Code:

    def get_browser(self):
        return BasicNewsRecipe.get_browser(self, user_agent='common_words/based')

that might make a difference. And sure you can submit the recipe.

kovidgoyal · 12-28-2022, 10:09 PM

Where you put them is fine, just make sure they are indented properly to line up with the rest of the things inside the recipe class.

12-25-2022, 10:42 AM	#8
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You could try using Code: def get_browser(self): return BasicNewsRecipe.get_browser(self, user_agent='common_words/based') that might make a difference. And sure you can submit the recipe.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Japanese recipe broken - MainichiDailyNews	duluoz	Recipes	2	02-27-2020 06:57 PM
NHK Easy News (Japanese News site)	beemanfunk	Recipes	1	12-25-2014 03:44 AM
Recipe for Foresight(Japanese)	tmizukami	Recipes	0	10-10-2013 10:57 PM
Fairbanks Daily News-miner News Recipe Submission	rogerx	Recipes	2	08-25-2011 07:30 PM
New Fairbanks Daily News-miner News Recipe -- Need Date inclusion only	rogerx	Recipes	5	08-24-2011 09:12 AM

12-21-2022, 06:22 AM	#2
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Post the log when it hangs, it will most likely be some anti bot tech on the servers.

12-22-2022, 01:04 AM	#3
rasteps Zealot Posts: 119 Karma: 10 Join Date: Sep 2010 Device: Kindle, Android phone	It doesn't generate a log when it hangs. It just sits in the queue until it is canceled.

12-22-2022, 01:13 AM	#4
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	click the rotating jobs button in the bottom right corner to find the log

12-23-2022, 10:28 PM	#6
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Yeah it looks like bot protection. Sometimes websites will send one byte at a time with intervals when they think a bot is scraping them to prevent timeout errors and keep the bot hanging.

12-25-2022, 09:15 AM	#7
rasteps Zealot Posts: 119 Karma: 10 Join Date: Sep 2010 Device: Kindle, Android phone	Sounds like there's no workaround for this. Thanks for looking into it for me. Can I submit this recipe as it stands then?

12-28-2022, 10:09 PM	#10
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Where you put them is fine, just make sure they are indented properly to line up with the rest of the things inside the recipe class.