Recipe without rss feed?

[nicolash]

Hi!

There is a webpage (URL) that gets updated daily. There is no rss feed.

Basically I would like to write a recipe that grabs the contents of that single URL and formats it into an epub for me once a day.

My first approach, to just add the URL to the feeds variable gives me an epub with all the contents of the URL including its html tags - with no formatting at all.

I flipped through the API documentation but only found the 'use_embedded_content' variable, which might be the correct direction? However I feel being trapped by having the URL content interpreted as rss and not as "news content".

Any clue how to process webpages without the help of rss feeds?
Thank you!

[Steven630]

So, you mean you want to download ONLY the starting page? (against using the page as TOC)

[kiklop74]

You should be able to do this with instapaper. Just create accaount and add that url to your main feed. The existing recipe should do the rest.

[nicolash]

Yes, that is the idea.
The single webpage contains all the contents (but no toc and neither a rss data).

[nicolash]

Thank you kiklop74,

Using Instapaper works and brings the page as an epub to my reader.

However starting to learn how calibre recipes look like, I would like to go on and try to figure out how I could do processing of the page by myself - that way I might be able to insert special formatting or sectioning, maybe even a table of contents.

[yosivd]

Originally Posted by kiklop74

You should be able to do this with instapaper. Just create accaount and add that url to your main feed. The existing recipe should do the rest.

Had the same problem.
Thanks

[scissors]

feed43 (feed for free) is pretty good.

[nicolash]

Okay, thanks - the idea to create a rss feed by a service or even locally on my computer might work.

However, there must be a way to do it directly wih the recipes?
I did not have much time to dig into this forum. But maybe the BeautifulSoup might be something? Or is this used for handling after a rss feed helped to get to some content page?

[nicolash]

The main issue I have with Instapaper is, that I have to visit the page manually each time I would like to get the actual copy. Without this step, I receive with every download the same content. So I guess feed43 is the better work-around for my problem. Will give feedback, after I put it to work.

[eroche]

Nicolash I was trying to do the same thing the other day. You can do what your asking using a basicnewsrecipe class and the parse_index function. The following should get you started

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe

class BBC(BasicNewsRecipe):
title = 'Latest Headlines, Business and Sport from RTE (Ireland)'
def parse_index(self):
url = 'http://www.bbc.com/'
feeds = []
articles = [({'title': 'BBC Web', 'url': url, 'description':'', 'date':''})]
feeds.append( ( 'BBC Website', articles ) )
return feeds

Basically all you need to do is inherit a BasicNewsRecipe class and then initialise a feeds variable with the correct paramaters, calibre does the rest

[nicolash]

Thanks, eroche!

Your code snippet really gets me started.

Almost everything I had to do to obtain the page as epub with proper formatting was to exchange the URL!

Will start playing around with this some more.