The Simplest Recipe

[tolgame]

Hello,
I am simply looking for a way to make Calibre dump the information on the RSS/XML/Atom entries without trying to do anything intelligent like following the link. I looked through the "BasicNewsRecipe" expecting to see a simple option that does this, but no luck.
Basically I just want to see more or less what I would see if I go to the feed directly with the browser, except obviously that only the recent entries would be shown.
Sorry if this is a retarded question or one that]s been asked many times already.
Thanks in advance,
Tolgame

[Starson17]

Originally Posted by tolgame

I am simply looking for a way to make Calibre dump the information on the RSS/XML/Atom entries without trying to do anything intelligent like following the link.

Set this:

use_embedded_content = True

That's the easiest to do what you want.

[tolgame]

Thanks for your answer.
Actually I had tried this but it still seemed to do a lot of processing. For example if I added more than five or six feeds it would hang somewhere in the process of collecting them, and it renames all the article links to random local files which don't exist. But now that you've told me it's the correct parameter, I'll give it another try.
If you know of a better way, which preserves hyperlinks for example, I would still be interested in hearing that too!

[Starson17]

Originally Posted by tolgame

If you know of a better way, which preserves hyperlinks for example, I would still be interested in hearing that too!

You can treat the RSS page as an article page. Create a virtual feed with parse_index that has a single article located at the RSS feed address. Then you will see the RSS page as though it was viewed in a browser, with preserved links from the RSS page.

Look at some of my comics recipes (gocomics or comics.com). They use that method to create virtual RSS feeds pointing to individual comic pages.

[tolgame]

Sorry but I have given up, come back, given up again, come back again, a couple times now. I just can't figure out this GoComics code.
Could someone please help me out? The goal is to simply go to a bunch of RSS feed pages, fetch anything new (last x days), and show that content exactly as is, no formatting required. I have no experience with calibre, though.
I was trying to do it using Starson's advice about the virtual feed, here is my pitiful attempt. What am I doing wrong?


#!/usr/bin/env python

__license__ = 'GPL v3'
__copyright__ = 'Copyright 2010 Starson17'
'''
www.gocomics.com
'''
from calibre.web.feeds.news import BasicNewsRecipe
import mechanize, re

class GoJournal(BasicNewsRecipe):
title = 'Journals'
__author__ = 'Tolga'
__version__ = '1.05'
__date__ = '19 may 2011'
description = u'My journals.'
category = 'news, comics'
language = 'en'
use_embedded_content= False
no_stylesheets = True
remove_javascript = True
cover_url = 'http://paulbuckley14059.files.wordpress.com/2008/06/calvin-and-hobbes.jpg'
remove_attributes = ['style']


def get_browser(self):
br = BasicNewsRecipe.get_browser(self)
cookies = mechanize.CookieJar()
br = mechanize.build_opener(mechanize.HTTPCookieProcess or(cookies))
br.addheaders = [('Referer','http://www.gocomics.com/')]
return br

def parse_index(self):
feeds = []
for title, url in [
# (u"9 to 5", u"http://feeds.feedburner.com/acs/achre4"),
# (u"9 to 5", u"http://feeds.feedburner.com/acs/aamick"),
]:
print 'Working on: ', title
articles = self.make_links(url)
if articles:
feeds.append((title, articles))
return feeds

def make_links(self, url):
title = 'Temp'
current_articles = [self.index_to_soup(url)]
return current_articles

[Starson17]

Originally Posted by tolgame

I was trying to do it using Starson's advice about the virtual feed, here is my pitiful attempt. What am I doing wrong?

Explain what you want to do again.
It sounds like you want to just list your RSS feeds and set

use_embedded_content = True

I gave an alternative approach, but didn't know why anyone would want to do it that way.

[tolgame]

Actually now I feel pretty stupid.
So the reason why I pursued your parse-feeds suggestion over the embedded content suggestion was, as I said: for more than a handful of feeds it would always hang somewhere while processing them, and if I put few enough feeds in for it not to crash it would change all the hyperlinks to gibberish local-file links. The problem for both cases seemed to lie with the apparently unnecessary processing that I was hoping to avoid.

But just now, to reply to your question, I made another test feed using the embedded content to try to point out those problems for a specific case, and now I'm not seeing either of those problems. Embedded content now seems to remove hyperlinks entirely (which is ok for me) and does not hang even if I add like 30 feeds.

The charitable explanation is that one of the Calibre updates since March made the improvement. The uncharitable explanation, well, let's leave it unsaid.
Thanks for your help and patience.

[Starson17]

Originally Posted by tolgame

Thanks for your help and patience.

No problem I'm glad it's solved.