Custom recipes (archive, read-only)

[Starson17]

Originally Posted by funkgut

the articles still do not download.

It is weird. Can you see the articles in your browser? I'd just like to confirm that you can actually get to them. Also, do other recipes work for you?

[Blisters]

Originally Posted by kiklop74

The recipe includes images too.

kiklop74, sorry, I now realise that. No images came with the first download, but the next day they did.

I definitely owe you a beer the next time I am in Argentina!

[funkgut]

Originally Posted by Starson17

It is weird. Can you see the articles in your browser? I'd just like to confirm that you can actually get to them. Also, do other recipes work for you?

I can see them in my browser....I do have other recipes that work.......I have uploaded The Nation and The New York Post. This is why it has me so baffled.......This is the first one I created via Calibre, but the others work fine.

[gambarini]

Originally Posted by gambarini

    def print_version(self, url):
        raw = self.browser.open(url).read()
        soup = BeautifulSoup(raw.decode('utf8', 'replace'))
        print_link = soup.find('a', {'class':'printarticle'})
        return  'www.infomotori.com' + print_link['href'] 
        feeds          =     [(u'Ultime Novit\xe0', u'http://feeds.infomotori.com/ultimenovita')]

In this case the print link appeare good in debug, but the calibre return "can't download article. try -vv for details"

The print link, in this case and in one another case, is suitable only for particular articles (example: in dept reviews stripped on more pages)... The recipes, otherwise, works correctly. So, i have a group of new recipes to release:
Italian News Daily:
IlMessaggero
IlGiornale
LaStampa (Editoriali)
Italian News (Agency):
ANSA.it
ADNKronos.it

I.T. News Italian:
PuntoInformarico.it
TheAppleLounge
Androidiani

Italian News of Cars and MotoCycles:
InfoMotori.com

Uk News:
ElectronistaGadget (I.T. news)
Ansa.it (English)

What is the correct way to post all of them?
A single zip with all, or a zip for every recipes?
What is the correct values for author?

This program is great, for me, and it's the main reason to buy an e-book reader!!!!!

[ErikG]

Can I adapt an existing recipe to get the news from the book department of the french newspaper Le Monde?

http://www.lemonde.fr/livres/

I am stunned by what is possible with Calibre and my irex DR800S.
Thanks to you guys!

[Starson17]

Originally Posted by funkgut

I can see them in my browser....I do have other recipes that work

If you want to go further, you can copy your recipe into a recipe file and run it from the command line like this:
ebook-convert CatholicNews.recipe CatholicNews -vv> CatholicNews.txt

It will create html in a folder called CatholicNews and put all the error messages into CatholicNews.txt. While it worked on mine with both recipes, it did throw some errors from the source. Perhaps one of the errors is making your machine hiccup.

[Starson17]

Originally Posted by gambarini

The print link, in this case and in one another case, is suitable only for particular articles

You might want to fix it or remove the non-working parts before posting your recipes. Otherwise, you'll probably get error reports.

What is the correct way to post all of them?

You can post here or in the bug tracker (or both). If you post here, more people will see them. If you post in the bug tracker, they're less likely to get lost.

I doubt if it matters how you zip them up. You are the author, unless someone else wrote it and you're just adding to it or fixing something.

[kiklop74]

New recipe for China Daily in english:

[gambarini]

Originally Posted by Starson17

You might want to fix it or remove the non-working parts before posting your recipes. Otherwise, you'll probably get error reports.

i am near to the solution...

using the contents[x] parameter of the beautiful soup...

[lmittell]

Anyone game to concoct a recipe for the Jackson Hole News & Guide? An index of the RSS feeds is at http://www.jhnewsandguide.com/rss.php.

TIA,
Larry

[exdream]

Hi,

I have (had) a well working recipe for the textversion of sueddeutsche zeitung, which kiklop74 wrote. Today sueddeutsche changed the internal adressing of that webcontent e.g. from yesterdays adress http://www.sueddeutsche.de/sz/2010-04-27/politik/ to todays adress http://www.sueddeutsche.de/app/epape...ik/1272453301/
Would be great if you kiklop or somebody else could help me with the update.

Many Thanks,
exdream

PS.: I don't necessarily need the function INDEX = PREFIX + strftime('/sz/%Y-%m-%d/'). I can also open the content in the browser and copy this new timecode (1272453301) manually to source code of the recipe right before the download.


This is the old recipe:

__license__ = 'GPL v3'
__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
'''
www.sueddeutsche.de/sz/
'''

import urllib
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe

class SueddeutscheZeitung(BasicNewsRecipe):
title = 'Sueddeutsche Zeitung'
__author__ = 'Darko Miletic'
description = 'news from Germany'
publisher = 'Sueddeutsche Zeitung'
category = 'news, politics, Germany'
no_stylesheets = True
oldest_article = 2
encoding = 'cp1252'
needs_subscription = True
remove_empty_feeds = True
PREFIX = 'http://www.sueddeutsche.de'
# INDEX = PREFIX + strftime('/sz/%Y-%m-%d/')
INDEX = 'http://www.sueddeutsche.de/sz/2010-04-28/'
LOGIN = PREFIX + '/app/lbox/index.html'
use_embedded_content = False
masthead_url = 'http://pix.sueddeutsche.de/img/g_.gif'
language = 'de_DE'
extra_css = ' body{font-family: Arial,Helvetica,sans-serif} '

conversion_options = {
'comment' : description
, 'tags' : category
, 'publisher' : publisher
, 'language' : language
, 'linearize_tables' : True
}

remove_attributes = ['height','width']

def get_browser(self):
br = BasicNewsRecipe.get_browser()
br.open(self.INDEX)
if self.username is not None and self.password is not None:
data = urllib.urlencode({ 'login_name':self.username
,'login_passwort':self.password
,'lboxaction':'doLogin'
,'passtxt':'Passwort'
,'referer':self.INDEX
,'x':'22'
,'y':'7'
})
br.open(self.LOGIN,data)
return br

remove_tags =[
dict(attrs={'class':'hidePrint'})
,dict(name=['link','object','embed','base','iframe'])
]
remove_tags_before = dict(name='h2')
remove_tags_after = dict(attrs={'class':'author'})

feeds = [
(u'Politik' , INDEX + 'politik/' )
,(u'Seite drei' , INDEX + 'seitedrei/' )
,(u'Meinungsseite', INDEX + 'meinungsseite/')
,(u'Wissen' , INDEX + 'wissen/' )
,(u'Panorama' , INDEX + 'panorama/' )
,(u'Feuilleton' , INDEX + 'feuilleton/' )
,(u'Medien' , INDEX + 'medien/' )
,(u'Wirtschaft' , INDEX + 'wirtschaft/' )
,(u'Sport' , INDEX + 'sport/' )
,(u'Bayern' , INDEX + 'bayern/' )
,(u'Muenchen' , INDEX + 'muenchen/' )
,(u'jetzt.de' , INDEX + 'jetzt.de/' )
]

def parse_index(self):
totalfeeds = []
lfeeds = self.get_feeds()
for feedobj in lfeeds:
feedtitle, feedurl = feedobj
self.report_progress(0, _('Fetching feed')+' %s...'%(feedtitle if feedtitle else feedurl))
articles = []
soup = self.index_to_soup(feedurl)
tbl = soup.find(attrs={'class':'szprintd'})
for item in tbl.findAll(name='td',attrs={'class':'topthema'}):
atag = item.find(attrs={'class':'Titel'}).a
ptag = item.find('p')
if ptag:
stag = ptag.find('script')
if stag:
stag.extract()
url = self.PREFIX + atag['href']
title = self.tag_to_string(atag)
description = self.tag_to_string(ptag)
articles.append({
'title' :title
,'date' :strftime(self.timefmt)
,'url' :url
,'description':description
})
totalfeeds.append((feedtitle, articles))
return totalfeeds

def print_version(self, url):
return url + 'print.html'

[kiklop74]

There is really no need to send recipe which is already included into calibre. Instead you open a bug issue in calibre's trac. I'll take a look now and post update.

[kiklop74]

This site is really annoying with all these changes they do all of the time.

Updated recipe for payed content for sueddeutsche zeitung:

[yrvn]

This is my first recipe. It's for the uruguayan newspaper El Observador (Observa.com.uy) Don't know if this is of interest of someone but i share it just in case.

#!/usr/bin/env  python

__license__   = 'GPL v3'
__author__ = '2010, Yuri Alvarez<me at yurialvarez.com>'
__copyright__ = '2008, Kovid Goyal kovid@kovidgoyal.net'
'''
observa.com.uy
'''

from calibre.web.feeds.news import BasicNewsRecipe

class ObservaDigital(BasicNewsRecipe):
    title                 = 'Observa Digital'
    description           = 'Noticias de Uruguay'
    language       = 'es'
    timefmt        = '[%a, %d %b, %Y]'
    use_embedded_content  = False
    recursion             = 5
    encoding = 'utf8'
    remove_javascript = True
    no_stylesheets = True

    oldest_article        = 2
    max_articles_per_feed = 100
    keep_only_tags = [dict(id=['contenido'])]
    remove_tags = [
                     dict(name='div', attrs={'id':'contenedorVinculadas'}),
	   	     dict(name='p', attrs={'id':'nota_firma'}),
		     dict(name=['object','link'])
                  ]

    extra_css = '''
                h1{font-family:Geneva, Arial, Helvetica, sans-serif;color:#154B7A;}
                h3{font-size: 14px;color:#999999; font-family:Geneva, Arial, Helvetica, sans-serif;font-weight: bold;}
                h2{color:#666666; font-family:Geneva, Arial, Helvetica, sans-serif;font-size:small;}
                p {font-family:Arial,Helvetica,sans-serif;}
                '''
    feeds = [
               (u'Actualidad', u'http://www.observa.com.uy/RSS/actualidad.xml'),
	       (u'Deportes', u'http://www.observa.com.uy/RSS/deportes.xml'),
   	       (u'Vida', u'http://www.observa.com.uy/RSS/vida.xml'),
	       (u'Ciencia y Tecnologia', u'http://www.observa.com.uy/RSS/ciencia.xml')
	    ]

    def get_cover_url(self):
	cover_url = None
        index = 'http://www.observa.com.uy/'
        soup = self.index_to_soup(index)
        for image in soup.findAll('img',alt=True):
           if image['alt'].startswith('Tapa El Observador'):
              return image['src'].rstrip('b.jpg') + '.jpg'
        return None

    def preprocess_html(self, soup):
        for item in soup.findAll(style=True):
            del item['style']
        return soup

[exdream]

Hi Kiklop,
sorry for sending the whole recipe I didn't realize that it is part of calibres recipes now. The recipe worked fine after an import of strftime.
But I have another problem with it now. Sueddeutsche releases the textversion of the following day the evening before. With the old recipe it was possible to replace the strftime-function with a string and set the date manually to the following day. With this recipe this doesn't work anymore. Would be great if you have a solution for that for the new recipe.

Thanks a lot,
exdream