Custom recipes (archive, read-only)

[XanthanGum]

Hi,

I'm attaching a recipe for Think Progress, a site that serves as a forum to advance progressive ideas and policies.

Enjoy...

XG

[XanthanGum]

Hi,

I've given up on coming up with a good recipe for MIT's Technology Review at:

http://www.technologyreview.com/

Questions:

Some of the reports at MIT Technology Review are split across multiple pages. How do you deal with that?

In the middle of some of the articles, a line stating "Story continues below" occurs along with an advertisement. How do I cut that out?

The site has a print option for each article, but it uses the article id number in each of the print URLs. How would I deal with that?

I hope someone will improve my recipe and post it here so that I can see how to solve the problems I ran into.

Thanks...

XG

My recipe follows:

from calibre.web.feeds.news import BasicNewsRecipe

class MITtechnologyReview(BasicNewsRecipe):
title = u'MIT Technology Review'
__author__ = u'Xanthan Gum'
description = 'Technology news from MIT'

no_stylesheets = True

remove_tags_before = dict(id='articlebody')
remove_tags_after = dict(name='h3')

oldest_article = 7
max_articles_per_feed = 100

feeds = [(u'Top Stories', u'http://feeds.technologyreview.com/technology_review_top_stories'),
(u'Computing', u'http://feeds.technologyreview.com/technology_review_Computing'),
(u'Web', u'http://feeds.technologyreview.com/technology_review_Web'),
(u'Communications', u'http://feeds.technologyreview.com/technology_review_Communications'),
(u'Energy', u'http://feeds.technologyreview.com/technology_review_Energy'),
(u'Materials', u'http://feeds.technologyreview.com/technology_review_Materials'),
(u'Biomedicine', u'http://feeds.technologyreview.com/technology_review_Biotech'),
(u'Business', u'http://feeds.technologyreview.com/technology_review_Biztech')]

[kiklop74]

Originally Posted by XanthanGum

Hi,

I've given up on coming up with a good recipe for MIT's Technology Review at:

http://www.technologyreview.com/

Questions:

Some of the reports at MIT Technology Review are split across multiple pages. How do you deal with that?

See source of Adventure Gamers recipe which deals with multiple page articles.

[XanthanGum]

Originally Posted by kiklop74

See source of Adventure Gamers recipe which deals with multiple page articles.

kiklop,

Where can I find the recipe you suggested and any of the others in Calibre?

Thanks...

XG

[XanthanGum]

Originally Posted by XanthanGum

kiklop,

Where can I find the recipe you suggested and any of the others in Calibre?

Thanks...

XG

kiklop,

I found the recipes. Forgot that the source was posted just recently.

I will explore some, like the one you suggested.

Thanks again...

XG

[lorenzov]

Hi Brian, i did change the recipe for the CSM, but after looking at the Atlantic seems quite different and the types of content offered (current issues and 'voices' from the correspondents); as nobody seems to jump on it, i will look into it

lorenzo

[wdrwc]

Originally Posted by evanmaastrigt

Their print version is hard to get at, but I think it can be done (calibre knows some nice tricks too).

But the easy strategy is to forget the print version and just use the article from the feed. Their HTML seems to be valid, so you could use the keep_only_tags and remove_tags properties to get rid of unwanted content. There is also the preprocess_html() method to refine the result even further.

If you have further questions feel free to post them.

evanmaastright:
Thanks for the suggestion. As Krittika wrote, there is a chance for the recipe in the next release.
In the mean tiime I would like to try to go further with my first recipe.
I do not understand why their print version is hard to get at.
It's true - I can not get it with web2disk but wget dowloads it properly.
Can you please explain where is difficulty?

/wdrwc

[evanmaastrigt]

Originally Posted by wdrwc

I do not understand why their print version is hard to get at.

I do not understand it either. That is the 'sort of' part of the 'hard to get'. :-) But maybe it is just me. I tried wget as well, and the only thing I can think of is that calibre does not like the comma's in the url.

But I did not investigate that, because I saw another potential problem. For the print version you need o.a. to substitute a '1,' with a '2029020,' If that number is based on the year or changed for any other reason, the recipe will break.

And as the original article is not that hard to get into shape, I think you you should go for that.

[Krittika Goyal]

Originally Posted by evanmaastrigt

I do not understand it either. That is the 'sort of' part of the 'hard to get'. :-) But maybe it is just me. I tried wget as well, and the only thing I can think of is that calibre does not like the comma's in the url.

But I did not investigate that, because I saw another potential problem. For the print version you need o.a. to substitute a '1,' with a '2029020,' If that number is based on the year or changed for any other reason, the recipe will break.

And as the original article is not that hard to get into shape, I think you you should go for that.

Yes the problem was with the commas and kovid said he would fix it. i think he did but then I forgot about it and did not write the recipe. If someone is trying to write this recipe now it should work. If its still a problem i will try to do it on the weekend.

[lorenzov]

Hi Brian,
it might sound a stupid question, but what version of calibre are you using?
i'm usin 0.6.33 therefore can't say if it is the case for other, but i have just used the atlantic recipe as it is and it seems to work fine for me (see attached).

this recipe is very intriguing as it parses the index of the current issue page and compiles a list of articles from there.
(have to admit that i'm still trying to figure out how it does the magic...)

however i noticed that on the actual site there are some sections which allow you to use RSS feeds. i'm not sure if it is possible to mix feeds with the current recipe - Kovid will know the answer to this and those python expert out there.

lorenzo

[Krittika Goyal]

Originally Posted by lorenzov

Hi Brian,
it might sound a stupid question, but what version of calibre are you using?
i'm usin 0.6.33 therefore can't say if it is the case for other, but i have just used the atlantic recipe as it is and it seems to work fine for me (see attached).

this recipe is very intriguing as it parses the index of the current issue page and compiles a list of articles from there.
(have to admit that i'm still trying to figure out how it does the magic...)

however i noticed that on the actual site there are some sections which allow you to use RSS feeds. i'm not sure if it is possible to mix feeds with the current recipe - Kovid will know the answer to this and those python expert out there.

lorenzo

Lorenzo are you using a kindle? I wanted to fix the recipe and I mentioned it to kovid. He said that it worked fine on the kindle. He said the problem may be with the epub for the sony reader.
He said the epub can probably be viewed in calibre but not on the sony reader. I am not sure of the reason but it has something to do with the html of the page and how epub is more sensitive to it than mobi.

He said he maintains this recipe quite well usually as he reads it himself.

[Briand]

Thanks for your help Lorenzo...the recipe for The Atlantic worked as of early January...but it seemed to stop working properly in the last week or so...not sure if The Atlantic changed its site or?

Brian

[Briand]

I'm using the latest version of Calibre but as Krittika suggests maybe it's an issue with my Sony reader....would love to find a solution, as it is also one of my regular reads.

Brian

[kovidgoyal]

@Briand: I just downloaded the atlantic onto my 600 and it works fine. Try it again.

[espariz]

Hello, first many many thanks for all recipes, I would like to read on my kindle 2 this spanish paper, if you have time it would be nice if you make a recipe, many thanks!!!

http://www.nortecastilla.es/rss/feeds/