BBC News feeds in Calibre

elixir · 07-21-2010, 03:28 PM

Hi all, I am a recent user of Calibre and have been using it to populate news feeds on my PRS-505.

Most of my feeds have been working fine, however, I noticed the BBC feeds show just the news headlines but when you click on the articles all are blank, with just the URL at the bottom. This is the case both when viewing in Calibre as well as on my e-Reader.

I'm running Calibre 0.7.9 and have tried both "The BBC" and "BBC News (Fast)" feeds that come with Calibre.

Is anyone else experiencing this as well or is it just a configuration issue on my end?

Thanks!

chaley · 07-21-2010, 04:36 PM

The BBC has recently seen fit to destroy all their news sites. I imagine that the recipe has not yet been adapted to the new 'improved' version.

kovidgoyal · 07-21-2010, 05:42 PM

You should open a ticket assigned to darkom who is the author of the BBC recipes. He'll take a look at them when he has the time.

elixir · 07-21-2010, 05:48 PM

Chaley, you are right, the BBC does have a 'new look'!

Kovid, will open a ticket, thx.

jessie102 · 07-22-2010, 10:21 AM

wow, it's been happening for several weeks already, and only now someone notices?
wonder how many people use this recipe

kovidgoyal · 07-22-2010, 10:26 AM

You can see recipe usage stats here:http://status.calibre-ebook.com/recipe_stats

Starson17 · 07-22-2010, 02:57 PM

Here are the two recipes, updated. If someone who used the old version will test them, they can confirm they look correct and pass them to Kovid or post here with problems. This is quick and dirty, but seems OK in my quick tests:

Without images:

Spoiler:

Code:

__license__   = 'GPL v3'
__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
'''
news.bbc.co.uk
'''
import re
from calibre.web.feeds.recipes import BasicNewsRecipe

class BBC(BasicNewsRecipe):
    title                  = 'BBC News (fast)'
    __author__             = 'Darko Miletic, Starson17'
    description            = 'News from UK. A much faster version that does not download pictures'
    oldest_article         = 2
    max_articles_per_feed  = 100
    no_stylesheets         = True
    #delay                  = 1
    use_embedded_content   = False
    encoding               = 'utf8'
    publisher              = 'BBC'
    category               = 'news, UK, world'
    language               = 'en_GB'
    publication_type       = 'newsportal'
    extra_css              = ' body{ font-family: Verdana,Helvetica,Arial,sans-serif } .introduction{font-weight: bold} .story-feature{display: block; padding: 0; border: 1px solid; width: 40%; font-size: small} .story-feature h2{text-align: center; text-transform: uppercase} '
    preprocess_regexps     = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]
    conversion_options = {
                             'comments'        : description
                            ,'tags'            : category
                            ,'language'        : language
                            ,'publisher'       : publisher
                            ,'linearize_tables': True
                         }

    keep_only_tags    = [
                       dict(name='div', attrs={'class':['story-body']})
                        ]

    remove_tags = [
                       dict(name='div', attrs={'class':['story-feature related narrow', 'share-help', 'embedded-hyper', \
                       'story-feature wide ', 'story-feature narrow']})
                       , dict(name=['img'])
                        ]

    remove_attributes = ['width','height']

    feeds          = [
                      ('News Front Page', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/front_page/rss.xml'),
                      ('Science/Nature', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/science/nature/rss.xml'),
                      ('Technology', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/technology/rss.xml'),
                      ('Entertainment', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/entertainment/rss.xml'),
                      ('Magazine', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/magazine/rss.xml'),
                      ('Business', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/business/rss.xml'),
                      ('Health', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/health/rss.xml'),
                      ('Americas', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/americas/rss.xml'),
                      ('Europe', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/europe/rss.xml'),
                      ('South Asia', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/south_asia/rss.xml'),
                      ('UK', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/rss.xml'),
                      ('Asia-Pacific', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/asia-pacific/rss.xml'),
                      ('Africa', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/africa/rss.xml'),
                    ]

With images:

Spoiler:

Code:

__license__   = 'GPL v3'
__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
'''
news.bbc.co.uk
'''
import re
from calibre.web.feeds.recipes import BasicNewsRecipe

class BBC(BasicNewsRecipe):
    title                  = 'BBC News'
    __author__             = 'Darko Miletic, Starson17'
    description            = 'News from UK. '
    oldest_article         = 2
    max_articles_per_feed  = 100
    no_stylesheets         = True
    #delay                  = 1
    use_embedded_content   = False
    encoding               = 'utf8'
    publisher              = 'BBC'
    category               = 'news, UK, world'
    language               = 'en_GB'
    publication_type       = 'newsportal'
    extra_css              = ' body{ font-family: Verdana,Helvetica,Arial,sans-serif } .introduction{font-weight: bold} .story-feature{display: block; padding: 0; border: 1px solid; width: 40%; font-size: small} .story-feature h2{text-align: center; text-transform: uppercase} '
    preprocess_regexps     = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]
    conversion_options = {
                             'comments'        : description
                            ,'tags'            : category
                            ,'language'        : language
                            ,'publisher'       : publisher
                            ,'linearize_tables': True
                         }

    keep_only_tags    = [
                       dict(name='div', attrs={'class':['story-body']})
                        ]

    remove_tags = [
                       dict(name='div', attrs={'class':['story-feature related narrow', 'share-help', 'embedded-hyper', \
                       'story-feature wide ', 'story-feature narrow']})
                        ]

    remove_attributes = ['width','height']

    feeds          = [
                      ('News Front Page', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/front_page/rss.xml'),
                      ('Science/Nature', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/science/nature/rss.xml'),
                      ('Technology', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/technology/rss.xml'),
                      ('Entertainment', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/entertainment/rss.xml'),
                      ('Magazine', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/magazine/rss.xml'),
                      ('Business', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/business/rss.xml'),
                      ('Health', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/health/rss.xml'),
                      ('Americas', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/americas/rss.xml'),
                      ('Europe', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/europe/rss.xml'),
                      ('South Asia', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/south_asia/rss.xml'),
                      ('UK', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/rss.xml'),
                      ('Asia-Pacific', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/asia-pacific/rss.xml'),
                      ('Africa', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/africa/rss.xml'),
                    ]

JRG · 07-22-2010, 06:17 PM

I just tested the "without images" version (calibre 0.7.9, Win XP SP3).

It's a definite improvement, but there are still lots of articles in the table of contents that prove to be blank.

elixir · 07-22-2010, 07:13 PM

Starson17

I tried the "with images" version, same result as JRG, still a lot of blank articles but certainly better than the older version.

Starson17 · 07-23-2010, 07:48 AM

Quote:

Originally Posted by elixir

[I]I tried the "with images" version, same result as JRG, still a lot of blank articles but certainly better than the older version.

I ran it with the --test option which only picks up two feeds and two articles from each feed. Can you point me to a specific feed/article that is not working so I can check it out?

Edit: I just ran the complete version, and looked at 20 articles in various feeds. All had content. I couldn't find any that didn't work. I don't dispute that they're there, but I couldn't find them. Post a link to an empty article and I'll find out how it differs from the working articles.

elixir · 07-23-2010, 12:58 PM

Starson17, thanks for reviewing and yes there are very few blank pages.

I also ran with the --test option, in addition I also had the -vv option for verbose output. That resulted in a few parsing errors, they are in the attached bbc-errors.txt file.

One of the errors corresponds to the article UK->"PM's use of crime figures 'is propaganda'"
Initial parse failed:
Parsing file 'feed_10/article_35/index.html' as HTML
Forcing feed_10/article_35/index.html into XHTML namespace
Forcing index.html into XHTML namespace

A few more are in the Health Section:

Iraq veteran's struggle with PTSD [Thu, 22 Jul 02:51]
Vital care lacking for mini-strokes [Thu, 22 Jul 04:19]
Ex-conjoined twins reunited with mother [Thu, 22 Jul 13:13]
Drugs robots help out hospital staff [Wed, 21 Jul 13:11]

I've uploaded my entire archive to Dropbox here:

http://dl.dropbox.com/u/1925993/calibre/bbc.zip

Starson17 · 07-23-2010, 02:15 PM

Quote:

Originally Posted by elixir

I also ran with the --test option, in addition I also had the -vv option for verbose output.

Yes, I also always run with -vv.

Quote:

That resulted in a few parsing errors, they are in the attached bbc-errors.txt file.

One of the errors corresponds to the article UK->"PM's use of crime figures 'is propaganda'"

I checked this article, and mine was also empty, with a parsing error.

I'm running some tests and will see if I can work around the parsing problem. Somewhere I read that newer versions of Beautiful soup use an HTML parser that is less tolerant of malformed code than older versions. I don't know if that plays a role.

Edit: The links to several failed articles end .stm If you can check to see if they all end that way, at least we can focus on how those articles differ.

elixir · 07-23-2010, 02:56 PM

You are onto something, yes that particular one in 'UK' sections ends in .stm

I also looked at all in the 'Health' and it seems all these articles have an embedded video - so I guess it's probably just putting in something to ignore those tags?

Starson17 · 07-23-2010, 03:30 PM

Quote:

Originally Posted by elixir

so I guess it's probably just putting in something to ignore those tags?

I was able to work around the problem, and found some others. Try these. There may still be some problems, as I only looked at the UK feed. They seem to be sort of disorganized with their page structure, so problems will have to be fixed as they are found. Please give me a link to the BBC article page with any other error reports, as it takes a long time to run this recipe. I can't just run the test version or I only get good pages.

Without images:

Spoiler:

Code:

__license__   = 'GPL v3'
__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
'''
news.bbc.co.uk
'''
import re
from calibre.web.feeds.recipes import BasicNewsRecipe

class BBC(BasicNewsRecipe):
    title                  = 'BBC News (fast)'
    __author__             = 'Darko Miletic, Starson17'
    description            = 'News from UK. A much faster version that does not download pictures'
    oldest_article         = 2
    max_articles_per_feed  = 100
    no_stylesheets         = True
    #delay                  = 1
    use_embedded_content   = False
    encoding               = 'utf8'
    publisher              = 'BBC'
    category               = 'news, UK, world'
    language               = 'en_GB'
    publication_type       = 'newsportal'
    extra_css              = ' body{ font-family: Verdana,Helvetica,Arial,sans-serif } .introduction{font-weight: bold} .story-feature{display: block; padding: 0; border: 1px solid; width: 40%; font-size: small} .story-feature h2{text-align: center; text-transform: uppercase} '
    preprocess_regexps     = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]
    conversion_options = {
                             'comments'        : description
                            ,'tags'            : category
                            ,'language'        : language
                            ,'publisher'       : publisher
                            ,'linearize_tables': True
                         }

    keep_only_tags    = [
                       dict(name='div', attrs={'class':['layout-block-a layout-block']})
                       ,dict(attrs={'class':['story-body','storybody']})
                        ]

    remove_tags = [
                       dict(name='div', attrs={'class':['story-feature related narrow', 'share-help', 'embedded-hyper', \
                       'story-feature wide ', 'story-feature narrow']})
                       , dict(name=['img'])
                        ]

    remove_attributes = ['width','height']

    feeds          = [
                      ('News Front Page', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/front_page/rss.xml'),
                      ('Science/Nature', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/science/nature/rss.xml'),
                      ('Technology', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/technology/rss.xml'),
                      ('Entertainment', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/entertainment/rss.xml'),
                      ('Magazine', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/magazine/rss.xml'),
                      ('Business', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/business/rss.xml'),
                      ('Health', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/health/rss.xml'),
                      ('Americas', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/americas/rss.xml'),
                      ('Europe', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/europe/rss.xml'),
                      ('South Asia', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/south_asia/rss.xml'),
                      ('UK', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/rss.xml'),
                      ('Asia-Pacific', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/asia-pacific/rss.xml'),
                      ('Africa', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/africa/rss.xml'),
                    ]

With images:

Spoiler:

Code:

__license__   = 'GPL v3'
__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
'''
news.bbc.co.uk
'''
import re
from calibre.web.feeds.recipes import BasicNewsRecipe

class BBC(BasicNewsRecipe):
    title                  = 'BBC News'
    __author__             = 'Darko Miletic, Starson17'
    description            = 'News from UK. '
    oldest_article         = 2
    max_articles_per_feed  = 100
    no_stylesheets         = True
    #delay                  = 1
    use_embedded_content   = False
    encoding               = 'utf8'
    publisher              = 'BBC'
    category               = 'news, UK, world'
    language               = 'en_GB'
    publication_type       = 'newsportal'
    extra_css              = ' body{ font-family: Verdana,Helvetica,Arial,sans-serif } .introduction{font-weight: bold} .story-feature{display: block; padding: 0; border: 1px solid; width: 40%; font-size: small} .story-feature h2{text-align: center; text-transform: uppercase} '
    preprocess_regexps     = [(re.compile(r'<!--.*?-->', re.DOTALL), lambda m: '')]
    conversion_options = {
                             'comments'        : description
                            ,'tags'            : category
                            ,'language'        : language
                            ,'publisher'       : publisher
                            ,'linearize_tables': True
                         }

    keep_only_tags    = [
                       dict(name='div', attrs={'class':['layout-block-a layout-block']})
                       ,dict(attrs={'class':['story-body','storybody']})
                        ]

    remove_tags = [
                       dict(name='div', attrs={'class':['story-feature related narrow', 'share-help', 'embedded-hyper', \
                       'story-feature wide ', 'story-feature narrow']})
                        ]

    remove_attributes = ['width','height']

    feeds          = [
                      ('News Front Page', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/front_page/rss.xml'),
                      ('Science/Nature', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/science/nature/rss.xml'),
                      ('Technology', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/technology/rss.xml'),
                      ('Entertainment', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/entertainment/rss.xml'),
                      ('Magazine', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/magazine/rss.xml'),
                      ('Business', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/business/rss.xml'),
                      ('Health', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/health/rss.xml'),
                      ('Americas', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/americas/rss.xml'),
                      ('Europe', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/europe/rss.xml'),
                      ('South Asia', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/south_asia/rss.xml'),
                      ('UK', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/uk_news/rss.xml'),
                      ('Asia-Pacific', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/asia-pacific/rss.xml'),
                      ('Africa', 'http://newsrss.bbc.co.uk/rss/newsonline_world_edition/africa/rss.xml'),
                    ]

elixir · 07-23-2010, 03:35 PM

Great - I will try these out and get back to you if I see any errors, and yes will include the news URL

Thanks for you help!

Edit:
I just reviewed two full sections in both versions and no problems so far!!

07-21-2010, 03:28 PM	#1
elixir Member Posts: 10 Karma: 10 Join Date: Jul 2010 Device: Currently - Sony PRS-600; Sold - Sony PRS-505	BBC News feeds in Calibre Hi all, I am a recent user of Calibre and have been using it to populate news feeds on my PRS-505. Most of my feeds have been working fine, however, I noticed the BBC feeds show just the news headlines but when you click on the articles all are blank, with just the URL at the bottom. This is the case both when viewing in Calibre as well as on my e-Reader. I'm running Calibre 0.7.9 and have tried both "The BBC" and "BBC News (Fast)" feeds that come with Calibre. Is anyone else experiencing this as well or is it just a configuration issue on my end? Thanks!

07-22-2010, 06:17 PM	#8
JRG Member Posts: 15 Karma: 10 Join Date: Mar 2010 Device: DR800SG	BBC (fast) I just tested the "without images" version (calibre 0.7.9, Win XP SP3). It's a definite improvement, but there are still lots of articles in the table of contents that prove to be blank.

07-23-2010, 03:35 PM	#15
elixir Member Posts: 10 Karma: 10 Join Date: Jul 2010 Device: Currently - Sony PRS-600; Sold - Sony PRS-505	Great - I will try these out and get back to you if I see any errors, and yes will include the news URL Thanks for you help! Edit: I just reviewed two full sections in both versions and no problems so far!! Last edited by elixir; 07-23-2010 at 03:51 PM. Reason: Testing update

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Does calibre de-dupe news feeds?	tom95521	Calibre	1	08-24-2009 08:04 AM
automatic tag changes for news feeds in calibre?	SDgirl	Sony Reader	0	06-03-2009 09:11 PM
Content For My Illiad : BBC Headline News	jæd	iRex	29	05-29-2009 02:49 AM
Using the Calibre downloaded RSS/news feeds via Mobi2IMP and Impserve	nrapallo	Fictionwise eBookwise	0	03-23-2009 11:03 PM
Problem with News Feeds	Sydney's Mom	Calibre	10	03-07-2009 02:54 PM

07-21-2010, 04:36 PM	#2
chaley Grand Sorcerer Posts: 11,742 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	The BBC has recently seen fit to destroy all their news sites. I imagine that the recipe has not yet been adapted to the new 'improved' version.

07-21-2010, 05:42 PM	#3
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You should open a ticket assigned to darkom who is the author of the BBC recipes. He'll take a look at them when he has the time.

07-21-2010, 05:48 PM	#4
elixir Member Posts: 10 Karma: 10 Join Date: Jul 2010 Device: Currently - Sony PRS-600; Sold - Sony PRS-505	Chaley, you are right, the BBC does have a 'new look'! Kovid, will open a ticket, thx.

07-22-2010, 10:21 AM	#5
jessie102 Addict Posts: 234 Karma: 6720 Join Date: Aug 2008 Device: SONY PRS505	wow, it's been happening for several weeks already, and only now someone notices? wonder how many people use this recipe

07-22-2010, 10:26 AM	#6
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You can see recipe usage stats here:http://status.calibre-ebook.com/recipe_stats

07-22-2010, 07:13 PM	#9
elixir Member Posts: 10 Karma: 10 Join Date: Jul 2010 Device: Currently - Sony PRS-600; Sold - Sony PRS-505	Starson17 I tried the "with images" version, same result as JRG, still a lot of blank articles but certainly better than the older version.

07-23-2010, 02:56 PM	#13
elixir Member Posts: 10 Karma: 10 Join Date: Jul 2010 Device: Currently - Sony PRS-600; Sold - Sony PRS-505	You are onto something, yes that particular one in 'UK' sections ends in .stm I also looked at all in the 'Health' and it seems all these articles have an embedded video - so I guess it's probably just putting in something to ignore those tags?

Advert

Advert