Custom recipes (archive, read-only)

[Starson17]

Originally Posted by roguefan99

Dunno if anyone has reported this but the Esquire reciepe is returning blank articles for me at the moment. I guess they have changed their site design so that calibre doesn't see it correctly at the moment.

Its been doing it for a few weeks now.

You may want to post this as a bug, so it eventually gets fixed. I wrote the Big Oven recipe, and that site has completely changed, so it's on my list of recipes to fix. MSNBC also has some problems - I started doing some work on that one, but ran out of time.

[Starson17]

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe
import re

class BigOven(BasicNewsRecipe):
    title               = 'BigOven'
    __author__          = 'Starson17'
    description         = 'Recipes for the Foodie in us all. Registration is free. A fake username and password just gives smaller photos.'
    language            = 'en'
    category            = 'news, food, recipes, gourmet'
    publisher           = 'Starson17'
    use_embedded_content= False
    no_stylesheets      = True
    oldest_article      = 24
    remove_javascript   = True
    remove_empty_feeds    = True
    cover_url           = 'http://www.software.com/images/products/BigOven%20Logo_177_216.JPG'
    max_articles_per_feed = 30
    needs_subscription = True

    conversion_options = {'linearize_tables'  : True
                        , 'comment'           : description
                        , 'tags'              : category
                        , 'publisher'         : publisher
                        , 'language'          : language
                        }
    
    def get_browser(self):
        br = BasicNewsRecipe.get_browser()
        if self.username is not None and self.password is not None:
            br.open('http://www.bigoven.com/account/login?ReturnUrl=/')
            br.select_form(nr=1)
            br['Email']  = self.username
            br['Password'] = self.password
            br.submit()
        return br

    remove_attributes = ['style', 'font']

    remove_tags     = [dict(name='div', attrs={'class':['ppy-caption']})
                                  ,dict(name='div', attrs={'id':['float_corner']})
                                  ]

    def preprocess_html(self, soup):
        for tag in soup.findAll(name='a', attrs={'class':['deflink']}):
          tag.replaceWith(tag.string)
        for tag in soup.findAll(name='a', text=re.compile(r'.*View Metric.*', re.DOTALL)):
          tag.parent.parent.extract()
        for tag in soup.findAll(name='a', text=re.compile(r'.*Add my own photo.*', re.DOTALL)):
          tag.parent.parent.extract()
        for tag in soup.findAll(name='div', attrs={'class':['container']}):
          if tag.find(name='h1'):
              continue
          if tag.find(name='h2', text=re.compile(r'.*Ingredients.*', re.DOTALL)):
              print 'tag found Ingred h2'
              continue
          if tag.find(name='h2', text=re.compile(r'Preparation.*', re.DOTALL)):
              print 'tag found Prep h2'
              continue
          tag.extract()
        return soup

    feeds = [(u'4 & 5 Star Rated Recipes', u'http://feeds.feedburner.com/Bigovencom-RecipeRaves?format=xml')]

Since I wrote this recipe a few weeks ago, this site has been totally redesigned, and is currently in beta.

In the redesign, they've removed a lot of the class and id tag labels previously found on this site, so the recipe has some interesting (to me) methods of removing tags by identifying text within a sub-tag and removing the parent tag.

[kiklop74]

New recipe for The Financial Express:

[baboyak]

how do you do this news source from the Philippines with clickable links (the links lead to a page which is also saved so i don't have to be connected). thanks!

http://www.inquirer.net/


and

http://www.abs-cbnnews.com/

[bhandarisaurabh]

Originally Posted by kiklop74

New recipe for The Financial Express:

thanks for the recipe,you did the wonderful job but can the recipe be not made without using feeds and for the whole page section by section

http://www.financialexpress.com/feprint.php

covering all the sections like you made it for business week magazine take your time whenever you get time please can you make this for the whole page

[kiklop74]

Originally Posted by bhandarisaurabh

thanks for the recipe,you did the wonderful job but can the recipe be not made without using feeds and for the whole page section by section

I will not enter into discussion with ungrateful people. This is all the work I'm willing to do for this site for free. That is my final word.

[nickredding]

The custom masthead seems to be broken -- I get this

Traceback (most recent call last):
  File "site-packages\calibre\web\feeds\news.py", line 1012, in download_masthead
  File "site-packages\calibre\web\feeds\news.py", line 1004, in _download_masthead
  File "site-packages\calibre\web\feeds\news.py", line 1085, in prepare_masthead_image
TypeError: create_canvas() takes exactly 3 arguments (2 given)

Does anyone have any insight into the cause of this? It happens when I set masthead_url or override the get_masthead_url method.

[kiklop74]

Originally Posted by nickredding

Does anyone have any insight into the cause of this? It happens when I set masthead_url or override the get_masthead_url method.

It is most likely a bug that you should report in calibre's trac.

Meanwhile you can circumvent this bug by implementing get_cover_url since it is called before masthead and set class field masthead_url there like this:

    def get_cover_url(self):
         #your code here
        self.masthead_url = 'some url value'
        return None

[hrickh]

I haven't been using Calibre very long for my news feeds, but I'm consistently getting the following errors for the El Pais recipe. Is anybody else having this problem? If so, is there an updated recipe?

(Sorry if I included too many error messages. I tried to just include the actual errors).

R.
==
...
File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64088)
XMLSyntaxError: Entity 'squot' not defined, line 5, column 559
...
File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64088)
XMLSyntaxError: Entity 'squot' not defined, line 5, column 395
...
File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64088)
XMLSyntaxError: Entity 'M' not defined, line 82, column 350
...
File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64088)
XMLSyntaxError: Entity 'M' not defined, line 82, column 350
...
File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64088)
XMLSyntaxError: Entity 'artes' not defined, line 52, column 27
...
Referenced file '/comunes/player/video_iframe.html%3fxref%3d20100812elpepunac_1.Ves %26conf%3d%26id%3d864532%26logo%3d%26skin%3delpais Mini%26referer%3d' not found
Referenced file 'feed_13/index.html' not found
Referenced file 'feed_4/article_3/20100812elpepieco_8/Tes' not found
Referenced file '/comunes/player/video_iframe.html%3fxref%3d20100726elpepucin_1.Ves %26conf%3d%26id%3d17177%26logo%3d%26skin%3delpaisM ini%26referer%3d' not found
Referenced file '/comunes/player/video_iframe.html%3fxref%3d20100812elpepunac_2.Ves %26conf%3d%26id%3d57471%26logo%3d%26skin%3delpaisM ini%26referer%3d' not found
Traceback (most recent call last):
File "/usr/bin/calibre-parallel", line 19, in <module>
sys.exit(main())
File "/usr/lib/calibre/calibre/utils/ipc/worker.py", line 99, in main
result = func(*args, **kwargs)
File "/usr/lib/calibre/calibre/gui2/convert/gui_conversion.py", line 24, in gui_convert
plumber.run()
File "/usr/lib/calibre/calibre/ebooks/conversion/plumber.py", line 787, in run
self.input_plugin)
File "/usr/lib/calibre/calibre/ebooks/conversion/plumber.py", line 911, in create_oebbook
reader()(oeb, path_or_stream)
File "/usr/lib/calibre/calibre/ebooks/oeb/reader.py", line 72, in __call__
self._all_from_opf(opf)
File "/usr/lib/calibre/calibre/ebooks/oeb/reader.py", line 588, in _all_from_opf
self._manifest_from_opf(opf)
File "/usr/lib/calibre/calibre/ebooks/oeb/reader.py", line 243, in _manifest_from_opf
self._manifest_add_missing(invalid)
File "/usr/lib/calibre/calibre/ebooks/oeb/reader.py", line 204, in _manifest_add_missing
if not self.oeb.container.exists(href):
File "/usr/lib/calibre/calibre/ebooks/oeb/base.py", line 433, in exists
path = os.path.join(self.rootdir, urlunquote(path))
File "/usr/lib/calibre/calibre/startup.py", line 103, in my_join
res = res.decode(encoding)
File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 18-21: invalid data

[kovidgoyal]

@nickredding: That's a bug, fixed in 0.7.14

[kiklop74]

Originally Posted by hrickh

I haven't been using Calibre very long for my news feeds, but I'm consistently getting the following errors for the El Pais recipe. Is anybody else having this problem? If so, is there an updated recipe?

I'll look into that.

Edit:
Actually which El Pais recipe? I did the printed edition recipe, and there is another one that uses rss feeds developed by Lorenzo Vigentini?

[hrickh]

Originally Posted by kiklop74

I'll look into that.

Edit:
Actually which El Pais recipe? I did the printed edition recipe, and there is another one that uses rss feeds developed by Lorenzo Vigentini?

Thanks for responding.

It's the Lorenzo Vigentini one - it was included in the Calibre install.

I'll do a search through the forum to find your recipe and try that one (is it posted here?)

R.
==

[kiklop74]

Originally Posted by hrickh

Thanks for responding.

It's the Lorenzo Vigentini one - it was included in the Calibre install.

I'll do a search through the forum to find your recipe and try that one (is it posted here?)

R.
==

It is included in latest version of Calibre. Look for El País - printed edition.

[hrickh]

Originally Posted by kiklop74

It is included in latest version of Calibre. Look for El País - printed edition.

Thanks.

Is there by chance any way to just use the recipe?

The reason I ask is I have the version installed through the Ubuntu Lucid repository (0.6.42). I've tried to uninstall that and manually install the latest version as explained elsewhere here. Turns out it needs the dev poppler-qt4 libraries, which in turn need another 60-some megs of other libraries installed.

Since I'm on a netbook, I don't even have the space to get the required libraries to install the latest Calibre version.

Thanks for any info.

R.
==

[kiklop74]

Originally Posted by hrickh

Thanks.

Is there by chance any way to just use the recipe?

Sure here is the recipe posted:

https://www.mobileread.com/forums/sho...ed#post1004098