Custom recipes (archive, read-only)

[rty]

Originally Posted by Gunnerp245

Standard calibre news recipes:

English (Thailand) Bangkok Post.
English (Singapore) Today Online - Singapore.

Each only downloads the section headings; no articles.

I just went to check Today Online website RSS feed page http://www.todayonline.com/RSS/Hotnews and found that there is no feed on the page. Maybe they are updating the page now. We can check again later.

Recipe for Bangkok Post does not work anymore. If the author doesn't fix it, I'll probably can take a look at it this weekend.

[kovidgoyal]

http auth in mechanize is really easy there are examples here:

http://wwwsearch.sourceforge.net/mechanize/

[Starson17]

Originally Posted by kovidgoyal

http auth in mechanize is really easy there are examples here:
http://wwwsearch.sourceforge.net/mechanize/

Thanks. The mechanize documentation seems particularly skimpy. I briefly read up on Google authentication, and it looked like they would occasionally issue a new CAPTCHA challenge after some failed login attempts. Not having an account, I wasn't sure how it all worked.

[DoctorOhh]

Originally Posted by Starson17

If someone wants to set up a test account, add some blogs or sites, or whatever content is needed in Google, put some "stars" or whatever they are into it, give me the user/pass (here or by PM) and help with figuring out what I'm supposed to be getting, I'll take a whack at it.

I'll setup a test account later tonight and send you what I know about its working.

[Starson17]

Originally Posted by dwanthny

I'll setup a test account later tonight and send you what I know about its working.

OK, I've set up a test environment, added some debug statements into the old GoogleReader recipe, and drafted a first attempt at the new login/auth procedure. As soon as I have a valid user/pass, I'll run some tests.

[DoctorOhh]

Originally Posted by Starson17

OK, I've set up a test environment, added some debug statements into the old GoogleReader recipe, and drafted a first attempt at the new login/auth procedure. As soon as I have a valid user/pass, I'll run some tests.

The "star" thing described never worked. I always put whatever subscriptions I wanted to download in folders. The recipes treated each folder as a different feed and created a menu item for each.

[Starson17]

Originally Posted by Starson17

As to the "Photo" issue - you want to skip articles that have that text in the link. I only know one way to do that. Perhaps someone else knows another. Basically, I know two ways to follow articles - to follow all the links in the automatically parsed feed, or to build your own feed (without the Photo links) with parse_index and then follow all of those links.

If there's another way - to follow some links, but not others, I don't know it.

I thought of "another way."

Use print_version.
Try this code:

    def print_version(self, url):
        match = re.search(r'PhotoGallery', url)
        if not match:
            return url

Print_version operates between the parsing and the fetching. The code above will check to see if "PhotoGallery" is in the URL and skip that article if it is.

[Starson17]

Originally Posted by dwanthny

The "star" thing described never worked. I always put whatever subscriptions I wanted to download in folders. The recipes treated each folder as a different feed and created a menu item for each.

I assume this was for the builtin GoogleReader recipe - correct? There's also something called GoogleReader Uber. Is that involved in this?

[rty]

My humble little gift to folks up there in Alaska: Recipe for Anchorage Daily News.

[DoctorOhh]

Originally Posted by Starson17

I assume this was for the builtin GoogleReader recipe - correct? There's also something called GoogleReader Uber. Is that involved in this?

That is correct. GoogleReader is the one I have been referring to. To save my brain I actually posted the difference between them a while back when I understood the difference.

Although the changes you make for Google Reader should work for Google reader Uber too. they both use the SID for login.

[mik991]

Hi Guys I was trying to create a custom recipe for the local newspaper, Democrat and Chronicle. I found the print page and that works very fast with calibre. My problem is once it syncs onto my nook it will only display one page worth for each story and then cut off (sometimes even the bottom portion of the next sentence). I'm sure it is most likely some code snippet I am missing.

Any help would be greatly appreciated.

Here is what I had

class AdvancedUserRecipe1278352006(BasicNewsRecipe):
    title          = u'Democrat and Chronicle'
    oldest_article = 1
    max_articles_per_feed = 100
    no_stylesheets = True

    feeds          = [(u'Breaking news and top stories', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=RSS&mime=xml'), 

(u'News', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=RSS01&mime=xml'), 

(u'Bills', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=SPORTS03&template=rss_dc&mime=xml'), 

(u'Business', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=BUSINESS&template=rss_dc&mime=xml'), 

(u'Health', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=HEALTH&template=rss_dc&mime=xml'), 

(u'Arts', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=rssdcarts&mime=xml'), 

(u'Living', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=LIVING&template=rss_dc&mime=xml'), 

(u'Opinions', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=RSS04&mime=xml'), 

(u'Sports', u'http://www.democratandchronicle.com/apps/pbcs.dll/section?category=SPORTS&template=rss_dc&mime=xml')]

    def print_version(self, url):
    	return url.replace('http://www.democratandchronicle.com/', 'http://www.democratandchronicle.com/print/')

[mikegps1]

The UK online editions of The Times and The Sunday Times are now subscription only. I have a subscription for both and would like to download on to my Sony PRS600. The existing custom recipe doesn't have the facility for sending the password - can any of the wizards on this forum help please?

The urls are:
http://www.thetimes.co.uk/tto/news/
http://www.thesundaytimes.co.uk/sto/

The Calibre program is excellent - if Sony had any sense they should be throwing large sums of money at Kovid to allow them to ship Calibre with their products instead of the inferior Reader program.

Thanks in advance.

Mike

[nook.life]

Originally Posted by Starson17

There is a <div> tag in the original that has a style with a defined width that matches the width of the image before rotation, and a text-align attribute that centers the image in that width.

I don't have a nook, but try this...

Starson17, you are the best. The recipe works perfectly now. It is centered and everything. Thank you so, so much for all of the time you put into this. For those of you who want the final recipe here it is. Be sure to thank Starson17 for all of his hard work. Thanks again!

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup
import re
import calibre.utils.PythonMagickWand as pw
import calibre.utils.PythonMagickWand

class Explosm(BasicNewsRecipe):
    title               = 'Explosm Rotated'
    __author__          = 'Starson17'
    description         = 'Explosm'
    language            = 'en'
    use_embedded_content= False
    no_stylesheets      = True
    oldest_article      = 24
    remove_javascript   = True
    remove_empty_feeds    = True
    max_articles_per_feed = 10

    feeds = [
             (u'Explosm Feed', u'http://feeds.feedburner.com/Explosm')
             ]

    keep_only_tags     = [dict(name='div', attrs={'align':'center'})]
    remove_tags = [dict(name='span'),
                   dict(name='table')]

    def postprocess_html(self, soup, first):
        #process all the images. assumes that the new html has the correct path
        for tag in soup.findAll(lambda tag: tag.name.lower()=='img' and tag.has_key('src')):
            iurl = tag['src']
            print 'resizing image' + iurl
            with pw.ImageMagick():
                img = pw.NewMagickWand()
                p = pw.NewPixelWand()
                if img < 0:
                    raise RuntimeError('Out of memory')
                if not pw.MagickReadImage(img, iurl):
                    severity = pw.ExceptionType(0)
                    msg = pw.MagickGetException(img, byref(severity))
                    raise IOError('Failed to read image from: %s: %s'
                        %(iurl, msg))
                width = pw.MagickGetImageWidth(img)
                height = pw.MagickGetImageHeight(img)
                if( width > height ) :
                    print 'Rotate image'
                    pw.MagickRotateImage(img, p, 90)
                if not pw.MagickWriteImage(img, iurl):
                    raise RuntimeError('Failed to save image to %s'%iurl)
                pw.DestroyMagickWand(img)
                for divtag in soup.findAll('div'):
                   del(divtag['style'])
        return soup

    extra_css = '''
                    h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;}
                    h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;}
                    p{font-family:Arial,Helvetica,sans-serif;font-size:small;}
                    body{font-family:Helvetica,Arial,sans-serif;font-size:small;}
		'''

[Starson17]

Originally Posted by mikegps1

The Calibre program is excellent - if Sony had any sense they should be throwing large sums of money at Kovid to allow them to ship Calibre with their products instead of the inferior Reader program.

Calibre is licensed to all under GPL v.3. That means Sony wouldn't have to pay a dime to Kovid (or me or anyone else who has contributed to the Calibre code). OTOH, they would have to comply with the terms of GPL v.3.

[Starson17]

Originally Posted by nook.life

Starson17, you are the best. The recipe works perfectly now. It is centered and everything. Thank you so, so much for all of the time you put into this.

I'm glad it's working for you. I realized after I wrote that little bit of code that there's a predefined method that I had never used that would have done it for you more easily:

remove_attributes = ['style']

As an aside, I often wish I could have the recipe accept some settings from the user, other than just the username and password. In this recipe, the images are rotated, which makes sense for someone reading on a handheld device, but not for someone who reads on a desktop. It would be nice to simply have an option to rotate images or not, instead of requiring two different recipes that are basically the same.

Most recipes have optional settings, such as the number of comics to download, the number of articles to retrieve, etc. It would be nice to have a settings screen associated with the recipe (like the user/password) that can be set by the user when they schedule a builtin. The way it works now, the user has to load the builtin recipe in the recipe editor, modify the code, save the recipe, go to the News fetcher and schedule the user modified version.