Custom recipes (archive, read-only)

[Tumaini]

Arbetaren (Swedish socialist newspaper, works great!)

class Arbetaren_SE(BasicNewsRecipe):
    title          = u'Arbetaren'
    __author__            = 'Joakim Lindskog'
    description           = 'Nyheter från Arbetaren'
    publisher             = 'Arbetaren'
    category              = 'news, politics, socialism, Sweden'
    oldest_article        = 7
    delay                 = 1
    max_articles_per_feed = 100
    no_stylesheets        = True
    use_embedded_content  = False
    encoding              = 'utf-8'
    language              = 'sv'

    conversion_options = {
                          'comment'   : description
                        , 'tags'      : category
                        , 'publisher' : publisher
                        , 'language'  : language
                        }

    keep_only_tags = [dict(name='div', attrs={'id':'article'})]
    remove_tags_before = dict(name='div', attrs={'id':'article'})
    remove_tags_after = dict(name='p',attrs={'id':'byline'})
    remove_tags = [
                     dict(name=['object','link','base']),
                     dict(name='p', attrs={'class':'print'}),
                     dict(name='a', attrs={'class':'addthis_button_compact'}),
                     dict(name='script')
                  ]

    feeds          = [(u'Nyheter', u'http://www.arbetaren.se/rss/arbetaren.rss?rev=123')]

[kiklop74]

Originally Posted by mobilewilier

Hi Kiklop

Would you be so kind as to start me off with a recipe for the South China Morning Post?

www.scmp.com

This site has a very complicated logon procedure. I have no time to work on that here is a starting point for you. You only need to resolve logon. The rest is done.

[mobilewilier]

Originally Posted by kiklop74

This site has a very complicated logon procedure. I have no time to work on that here is a starting point for you. You only need to resolve logon. The rest is done.

Thanks so much... it works!!! many many thanks

WL

[Krittika Goyal]

I had this request on facebook. if someone can do it cause I am a little busy rt now..
The East Bay Express. http://www.eastbayexpress.com/ebx/Home

Thanks

[smargo]

Hi, I am trying to make a simple recipe for the best russian language newspaper Kommersant.
Here is the recipe:

from calibre.web.feeds.news import BasicNewsRecipe
class AdvancedUserRecipe1272297716(BasicNewsRecipe):
    title          = u'Kommersant'
    oldest_article = 7
    max_articles_per_feed = 100

    feeds          = [(u'Kommersant', u'http://feeds.kommersant.ru/RSS_Export/RU/daily.xml')]


    



def print_version(self,url):

        segments = url.split('=')
        article_id = segments[1]
        newurl = 'http://www.kommersant.ru/doc-rss.aspx?DocsID=' + article_id + '&print=true'

        return newurl

but it fails with the following error log (below).

what am i doing wrong ? Thanks!

ERROR: Conversion Error: <b>Failed</b>: Fetch news from Kommersant

It seems to download the articles fine:

Fetching http://www.kommersant.ru/doc-rss.aspx?DocsID=1365119
Downloaded article: Tencent разложила DST на активы // Mail.ru, "Вконтакте" и "Одноклассникам" прописали мультипликаторы from http://www.kommersant.ru

but then fails:

lxml.etree.XMLSyntaxError: Failed to parse QName 'font-size:', line 33, column 3710

[olaf]

When running a job to create a Kindle file from a recipe, I often look at the job details to see what progress is being made. Is there any way to save the column widths of the Job Details screen? Each time I go in, I need to expand the columns to see the detail I'm looking at. It would be nice to customize the column widths and have them stay fixed after that. (The total screen size of that panel as well)

[smargo]

OK, now it's generally working,

from calibre.web.feeds.news import BasicNewsRecipe
class KommersantRecipe(BasicNewsRecipe):
    title          = u'Kommersant'
    oldest_article = 7
    max_articles_per_feed = 100
    feeds          = [(u'Kommersant', u'http://feeds.kommersant.ru/RSS_Export/RU/daily.xml')]

    def print_version(self,url):
       segments = url.split('=')
       article_id = segments[1]
       newurl = 'http://www.kommersant.ru/doc.aspx?DocsID=' + article_id + '&print=true'
       return newurl

but when I read it on Kindle, pagination does not work. When I am on the first page and press "Next pages" the aricle is skipped to the last page. What can be the problem?
Thanks all!

[kovidgoyal]

@smargo: Use

conversion_options = {'linearize_tables':True}

[smargo]

@kovidgoyal

Thanks! It workes!

Some cosmetics remains to be done, but I am happy.

[Raoul O'Malley]

is there a way to get the recipe for the IHT - Euro edition?

thanks so much

[PaxtonReader]

When I send a book back to my Kindle, is there a way to keep the original file name, without embedding the author's name to a shortened version?

[Starson17]

Multipage implemented for multiple page articles,
New feeds,
Miscellaneous advertising and junk removed.

#!/usr/bin/env  python
__license__   = 'GPL v3'
__copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
__docformat__ = 'restructuredtext en'

'''
discovermagazine.com
'''

import re
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, Tag

class DiscoverMagazine(BasicNewsRecipe):

    title = u'Discover Magazine'
    description = u'Science, Technology and the Future' 
    __author__ = 'Starson17' 
    language = 'en'

    oldest_article = 33
    max_articles_per_feed = 20
    no_stylesheets = True
    remove_javascript = True
    use_embedded_content  = False
    encoding = 'utf-8'
    extra_css = '.headline {font-size: x-large;} \n .fact {padding-top: 10pt}'
    
    remove_tags = [
                   dict(name='div', attrs={'id':['searchModule', 'mainMenu', 'tool-box']}),
                   dict(name='div', attrs={'id':['footer','teaser','already-subscriber','teaser-suite','related-articles']}),
                   dict(name='div', attrs={'class':['column']}),
                   dict(name='img', attrs={'src':'http://discovermagazine.com/onebyone.gif'})]

    remove_tags_after = [dict(name='div', attrs={'class':'listingBar'})]
   
    def append_page(self, soup, appendtag, position):
        pager = soup.find('span',attrs={'class':'next'})
        if pager:
           nexturl = pager.a['href']
           soup2 = self.index_to_soup(nexturl)
           texttag = soup2.find('div', attrs={'class':'articlebody'})
           newpos = len(texttag.contents)          
           self.append_page(soup2,texttag,newpos)
           texttag.extract()
           appendtag.insert(position,texttag)
    
    def preprocess_html(self, soup):
        mtag = '<meta http-equiv="Content-Language" content="en-US"/>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>'
        soup.head.insert(0,mtag)    
        self.append_page(soup, soup.body, 3)
        pager = soup.find('div',attrs={'class':'listingBar'})
        if pager:
           pager.extract()        
        return soup
        
    def postprocess_html(self, soup, first_fetch):
        for tag in soup.findAll(text=re.compile('^This article is a sample')):
            tag.parent.extract()
        for tag in soup.findAll(['table', 'tr', 'td']):
            tag.name = 'div'
        for tag in soup.findAll('div', attrs={'class':'discreet advert'}):
            tag.extract()
        for tag in soup.findAll('hr', attrs={'size':'1'}):
            tag.extract()
        for tag in soup.findAll('br'):
            tag.extract()
        return soup        
 
    feeds = [
             (u'Technology', u'http://discovermagazine.com/topics/technology/rss.xml'), 
             (u'Health - Medicine', u'http://discovermagazine.com/topics/health-medicine/rss.xml'), 
             (u'Mind Brain', u'http://discovermagazine.com/topics/mind-brain/rss.xml'), 
             (u'Space', u'http://discovermagazine.com/topics/space/rss.xml'), 
             (u'Human Origins', u'http://discovermagazine.com/topics/human-origins/rss.xml'), 
             (u'Living World', u'http://discovermagazine.com/topics/living-world/rss.xml'), 
             (u'Environment', u'http://discovermagazine.com/topics/environment/rss.xml'), 
             (u'Physics & Math', u'http://discovermagazine.com/topics/physics-math/rss.xml'), 
             (u"20 Things you didn't know about...", u'http://discovermagazine.com/columns/20-things-you-didnt-know/rss.xml'), 
             (u'Fuzzy Math', u'http://discovermagazine.com/columns/fuzzy-math/rss.xml'), 
             (u'The Brain', u'http://discovermagazine.com/columns/the-brain/rss.xml'), 
             (u'What is This', u'http://discovermagazine.com/columns/what-is-this/rss.xml'),
             (u'Vital Signs', u'http://discovermagazine.com/columns/vital-signs/rss.xml'), 
             (u'Think Tech', u'http://discovermagazine.com/columns/think-tech/rss.xml'),
             (u'Future Tech', u'http://discovermagazine.com/columns/future-tech/rss.xml'),
             (u'Discover Interview', u'http://discovermagazine.com/columns/discover-interview/rss.xml'),
            ]

[kiklop74]

Russian news pack:
Kommersant
Izvestia
Ria Novosti
Argumenti & fakti

[smargo]

@kiklop74

Your Kommersant recipe is great, thanks for this Russian pack!

Small wish: in the bottom of certain articles in Kommersant, there are links to the additional pages (of the same issue, they are not avaialble as links from rss). For example, on the page http://www.kommersant.ru/doc-rss.aspx?DocsID=1366511 there are links to page "2" - http://www.kommersant.ru/doc.aspx?DocsID=1366459 and page "3" - http://www.kommersant.ru/doc.aspx?DocsID=1366462. Is there any way to include these additional pages?

[kiklop74]

There is always a way. But right now I have no spare time nor will I have it in the foreseeable future. You are on your own on this one.