Custom recipes (archive, read-only)

[GRiker]

macsilber: It would be more helpful if you could post the recipe you're using.

dmendozadmd: a 'sticky' is a popular topic that stays in the upper list of topics in the forum, so they're easier to find. A recipe is a script that calibre uses to download the contents of a particular website, then format it for your eReader.

G

[Gomes]

Originally Posted by GRiker

Gomes,
There are RSS feeds in each section of philly.com. Follow the directions to create a custom feed, then ask for assistance if you get stuck. It's actually pretty simple.

G

I've been trying to get a clean copy for a couple of weeks with no success. Essentially, I am unable to get the print version of the stories. I've tried to go through the directions cited above, but that doesn't seem to help...What I end up with is the article with all the various menus, pictures, and comments, which makes it difficult to read at best, and takes forever for calibre to fetch and convert. Can anyone help?

And yes, I realize I'm just probably missing something obvious...

[cix3]

In a custom recipe, how do I remove multiple div classes?

For example, from this source page (http://www.tnr.com/print/article/pol...ocking-roberts), I want to remove these div classes: print-logo, print-site_name, img-left, and print-source_url.

Probably a simple syntax question, but I'm new to Python. I have tried...

    remove_tags = [dict(name='div', attrs={'class':'print-logo'})]
    remove_tags = [dict(name='div', attrs={'class':'print-site_name'})]
    remove_tags = [dict(name='div', attrs={'class':'img-left'})]
    remove_tags = [dict(name='div', attrs={'class':'print-source_url'})]

... which only removes the last div class listed (in this case, print-source_url).

This gives me a syntax error:

    remove_tags = [dict(name='div', attrs={'class':'print-logo', 'print-site_name', 'img-left', 'print-source_url'})]

What is the correct syntax?

Thanks

[kovidgoyal]

remove_tags = [dict(name='div', attrs={'class':['print-logo', 'print-site_name', ..]}]

[cix3]

Thanks... I knew it must have been something simple like that.

Your snippet as written gave me a syntax error, but adding a ) as the second to last character fixed it.

[GRiker]

gomes: Post your recipe. You will probably need to use remove_tags as cix3 has learned to get rid of the stuff you don't want.

Basically, this involves going to a sample page, examining the HTML source, isolating the stuff you don't want, then specifying a remove_tags directive as Kovid has described in his post above this one.

If you post your recipe, folks here are better able to help you refine it.

G

[cix3]

Hello,

Here's my first stab at a recipe for The New Republic (www.tnr.com). It aggregates all articles and blogs, minus the images. Enjoy!

class The_New_Republic(BasicNewsRecipe):
    title = 'The New Republic'
    __author__ = 'cix3'
    description = 'Intelligent, stimulating and rigorous examination of American politics, foreign policy and culture'
    timefmt = ' [%b %d, %Y]'

    oldest_article = 7
    max_articles_per_feed = 100

    remove_tags = [dict(name='div', attrs={'class':['print-logo', 'print-site_name', 'img-left', 'print-source_url']}), dict(name='hr', attrs={'class':'print-hr'}), dict(name='img')]

    feeds = [
        ('Politics', 'http://www.tnr.com/rss/articles/Politics'),
        ('Books and Arts', 'http://www.tnr.com/rss/articles/Books-and-Arts'),
        ('Economy', 'http://www.tnr.com/rss/articles/Economy'),
        ('Environment and Energy', 'http://www.tnr.com/rss/articles/Environment-%2526-Energy'),
        ('Health Care', 'http://www.tnr.com/rss/articles/Health-Care'),
        ('Urban Policy', 'http://www.tnr.com/rss/articles/Urban-Policy'),
        ('World', 'http://www.tnr.com/rss/articles/World'),
        ('Film', 'http://www.tnr.com/rss/articles/Film'),
        ('Books', 'http://www.tnr.com/rss/articles/books'),
        ('The Plank', 'http://www.tnr.com/rss/blogs/The-Plank'),
        ('The Treatment', 'http://www.tnr.com/rss/blogs/The-Treatment'),
        ('The Spine', 'http://www.tnr.com/rss/blogs/The-Spine'),
        ('The Stash', 'http://www.tnr.com/rss/blogs/The-Stash'),
        ('The Vine', 'http://www.tnr.com/rss/blogs/The-Vine'),
        ('The Avenue', 'http://www.tnr.com/rss/blogs/The-Avenue'),
        ('William Galston', 'http://www.tnr.com/rss/blogs/William-Galston'),
        ('Simon Johnson', 'http://www.tnr.com/rss/blogs/Simon-Johnson'),
        ('Ed Kilgore', 'http://www.tnr.com/rss/blogs/Ed-Kilgore'),
        ('Damon Linker', 'http://www.tnr.com/rss/blogs/Damon-Linker'),
        ('John McWhorter', 'http://www.tnr.com/rss/blogs/John-McWhorter')
            ]

    def print_version(self, url):
        return url.replace('http://www.tnr.com/', 'http://www.tnr.com/print/')

[bhandarisaurabh]

can anyone help me with recipe of business standard
if the url for the article is
http://www.business-standard.com/ind...?autono=369650
then print url is
http://www.business-standard.com/ind...ono=369650&tp=

[cutterjohn42]

It seems that the most recent version of the /. recipe in Calibre may have caused an auto-ban to be triggered for my IP address.

I noticed the last time that it seemed to be downloading more of the site than before, i.e. I had the article + comments, and I think that the way the site is setup that it leads to recursively downloading most of the site unless strictly limited. I used to have that problem with sitescooper and plucker and have to be very careful about limiting how much of /. was spidered to create a document for offline reading.

(This would be the version included with 0.6.11 .)

[kovidgoyal]

Open a ticket about it, I'll look at it when I have a spare moment.

[cix3]

Any idea how I can transform an article URL like this (http://www.motherjones.com/politics/...-job-van-jones) into the print URL (http://www.motherjones.com/print/27151) that I want to use for my recipe?

I'm hoping there's an easy way to find corresponding print URLs (by that 5 digit number) for articles. Rather than removing all unwanted html from the actual article...

Any ideas?

Edit: I should also note that the original article page actually splits the article into multiple pages (which I would want to combine into one article for my recipe). The print version lists the entire article.

[kovidgoyal]

Just fetch the HTML and parse it looking for the print link

[cix3]

Originally Posted by kovidgoyal

Just fetch the HTML and parse it looking for the print link

Can you give me an example of a built-in recipe that does this?

[kovidgoyal]

Cant think of one off hand but basically, it's something like this

def get_article_url(self, article):
   url = ...(from article as before)
   soup = self.index_to_soup(url)
   # do some processing on soup to find the full article link
   a = soup.find(name='a', href=True, text=re.compile(r'Full\s*Article')
   if a is not None:
      return a['href']
   return url

Stick a few print statements in there to debug things

[cix3]

Originally Posted by kovidgoyal

Cant think of one off hand but basically, it's something like this

def get_article_url(self, article):
   url = ...(from article as before)
   soup = self.index_to_soup(url)
   # do some processing on soup to find the full article link
   a = soup.find(name='a', href=True, text=re.compile(r'Full\s*Article')
   if a is not None:
      return a['href']
   return url

Stick a few print statements in there to debug things

Hmmm... that's beyond my level of expertise. I'm going to have to wait for someone else to recommend a pre-built recipe that I can copy from.

Thanks!