The Independent : Updated recipe for 2011 site redesign

[NotTaken]

As you probably know, the independent has recently updated its website and this has broken the old recipe.

Here is an initial basic recipe for the new site. I thought it would be good to make a thread for people to post improvements.

[NotTaken]

Updated the recipe to pull in the images. Do you reckon its best to limit these to one per article?

Regarding the categories, is it possible to merge them into a parent category if the number of articles is below a certain threshold?

[kovidgoyal]

If you want to do some dynamic modifications to the categories, you will need to override parse_feeds in your recipe, like this:

def parse_feeds(self):
  feeds = BasicNewsRecipe.parse_feeds(self)
  # do something to the feeds
  return feeds

[NotTaken]

Thanks kovid, I really appreciate all the work you do on calibre.

I've made a few changes to the recipe:

• Added a few extra guards to make it more robust
• Got rid of the "Caption:" label
• Added a postprocess parse to remove the captions that resulted from when an image fetch failed
• Stop adding empty captions
• Changed "feedsportal" rss feeds to Independent.co.uk equivalents

I noticed that their web server frequently times out when processing a request (probably initial teething problems with the new site). This resulted in a lot of captions being added without an image. Hopefully the post process parse should take care if this issue.

[kovidgoyal]

For the timeouts, try adding delay=1 to your recipe. That will greatly slow down the download but it might prevent the timeouts.

[NotTaken]

Originally Posted by kovidgoyal

For the timeouts, try adding delay=1 to your recipe. That will greatly slow down the download but it might prevent the timeouts.

I was getting the timeouts whilst browsing their site normally - seemed just to refuse to serve some pictures.

Slight update to the recipe to add star images to the reviews.

[mufc]

It is great to finally get a working recipe for the Independent ( my fav read) . What part of the recipe can I eliminate to remove the pictures and picture caption?
A way to get the print version would be even better
This is the difference @ the end of url

html

html?printService=print

[mufc]

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe
class AdvancedUserRecipe1289709253(BasicNewsRecipe):
title = u'The Independent My Test'
oldest_article = 7
max_articles_per_feed = 200
summary_length = 100
use_embedded_content = False
no_stylesheets = True
auto_cleanup = True
encoding = 'utf8'

remove_javascript = True

extra_css = '''
h1{font-family:Arial,sans-serif; font-weight:bold;font-size:large;}
h2{font-family:Arial,sans-serif; font-weight:normal;font-size:small;}
body{font-family:Arial,sans-serif;font-size:small;}
p{font-family:Arial,sans-serif;font-size:small;line-height: 1.2;margin-bottom: 0;margin-left: 2pt;
margin-right: 2pt;margin-top: 0;padding-left: 0;padding-right: 0;text-align: left;text-indent: 1.5em}
'''





feeds = [
(u'News - UK',
u'http://www.independent.co.uk/news/uk/?service=rss'),
(u'News - World',
u'http://www.independent.co.uk/news/world/?service=rss'),
(u'News -People Profiles',
u'http://www.independent.co.uk/news/people/profiles/?service=rss'),
(u'News - People',
u'http://www.independent.co.uk/news/people/?service=rss'),
(u'News - Media',
u'http://www.independent.co.uk/news/media/?service=rss'),
(u'Opinion',
u'http://www.independent.co.uk/opinion/?service=rss'),
(u'Sport - Football',
u'http://www.independent.co.uk/sport/football/?service=rss'),
(u'Sport - Fooball Comments',
u'http://www.independent.co.uk/sport/football/news-and-comment/?service=rss'),
(u'Life & Style - Health and Families',
u'http://www.independent.co.uk/life-style/health-and-families/?service=rss'
),
(u'Life & Style - Gadgets & Tech',
u'http://www.independent.co.uk/life-style/gadgets-and-tech/?service=rss'
),
(u'Arts & Ents - Music',
u'http://www.independent.co.uk/arts-entertainment/music/?service=rss'
),
(u'Arts & Ents - Comedy',
u'http://www.independent.co.uk/arts-entertainment/comedy/?service=rss'
)]



def print_version(self, url):
return url.replace('html', 'html?printService=print')

Thiis works on the articles I get. Problem is I am not getting many articles.
For example before trying to get the print version I was getting 100 articles for UK News. With recipe change I get 5. Some categories have none.
Before anyone states the obvious. Yes "oldest article 7" days is a bit much

[NotTaken]

Looks like they try and prevent direct linking to the print pages. To remove the images you can always change remove to True in the following piece of code:

            #images
            pattern = re.compile('slideshow')    
            if (pattern.search(item['class'])) is not None:
                remove = False

You'll get the added bonus of a subtitle and review ratings over the print version.

[mufc]

That works great. Thanks a Million !. I thought doing that would leave the photo caption but that is gone too.

[NotTaken]

Some changes:

• Filter feeds with title prefix 'Video:' - most only have one line of text
• Prevented duplicated content by setting recursions to zero and checking url existence against a list of feeds already processed
• Removed line breaks and empty paragraphs from the storyTop section as these cause unsightly white space (tries to sensibly replace line breaks between text with spaces)
• Try to fetch extra images related to the content when labeled with ...Click here for graphic... (this may need improving if the pattern changes wildy) - see this page for an example
• Added some flags up the top to disable image fetching

I was thinking about removing the advertorial articles (see here) but could not see a clean way of doing this. As far as I am aware, they are only identifiable by the text 'Advertorial Feature ' in <div class=" ... strapLine"> so I was thinking of returning None in preprocess_soup if the text was found (this causes an AttributeError exception to be raised). Can anyone think of a nicer solution?

[kovidgoyal]

Returning None in preprocess is fine. If you wish to be more explicit about it you could raise an Exception, though I dont recall if the download system discards exceptions in that method.

[NotTaken]

Thanks. A few updates:

• Fixed some flawed logic in detection of empty paragraphs in storyTop (didn't consider nesting)
• Updated article graphics regex to be more generic
• Filtered advertorial features

[NotTaken]

Fixed an issue whereby a KeyError was raised on pages with embedded flash videos. These pages also had some other crud, which I have also removed.

[dasym]

I'm afraid neither the built-in recipe nor this one is working for me for The Independent any more. It downloads very few links and typically gives error messages in the log such as:

Could not fetch link http://www.independent.co.uk/news/uk...s-6277966.html
Traceback (most recent call last):
File "site-packages\calibre\web\fetch\simple.py", line 432, in process_links
File "site-packages\calibre\web\fetch\simple.py", line 193, in get_soup
File "c:\docume~1\dave\locals~1\temp\calibre_0.8.31_tmp _uf67cn\2zkxsh_recipes\recipe0.py", line 202, in preprocess_html
File "c:\docume~1\dave\locals~1\temp\calibre_0.8.31_tmp _uf67cn\2zkxsh_recipes\recipe0.py", line 275, in _insertRatingStars
IndexError: list index out of range

I wish I could help but modifying the recipes is a little beyond me.