Only fetches 10 posts from blogs

[RickOlson]

I've got a new Kindle and am hoping to use Calibre to fetch blogs. I've tried creating custom news sources to go back 365 days and grab up to 100 posts. In all 3 that I've tried it only goes back to the last 10 posts. Essentially all 3 recipes look like this:

class AdvancedUserRecipe1290235764(BasicNewsRecipe):
title = u'Punk Rock OR'
oldest_article = 365
max_articles_per_feed = 100

feeds = [(u'Punk Rock OR', u'http://punkrockor.wordpress.com/feed/')]


The other two site had these URLs:
http://mat.tepper.cmu.edu/blog/?feed=rss2
http://greenor.wordpress.com/feed/

I'm guessing that I'm missing something obvious, but I don't see it from the other recipes or the part of the manual that I can understand. Any pointers (or solutions) will be appreciated.


Also, after the initial use, will Calibre grab only new posts, or will it grab the most recent X posts even if some were grabbed earlier. This has me a bit confused, too.

Thanks in advance--

[kovidgoyal]

The RSS feed you are using only has 10 items in it.

[RickOlson]

Originally Posted by kovidgoyal

The RSS feed you are using only has 10 items in it.

I *said* it was probably something obvious. I was unclear on the distinction between the person's blog page with many entries and the RSS feed that is shorter term. Is there a way to get the older posts with Calibre? It looks like putting the feeds into folders, starring the entries, and the using the Google Reader recipe might work as a one-off approach. Is there another Google-like RSS aggregator that works with Calibre that I should look into?

Finally, I don't know whether Calibre keeps a record of the most recent items it's grabbed. if the RSS feed is limited to the 10 most recent posts will I always get the entire feed, or just the ones since Calibre last looked?

Thanks for the help.

[kovidgoyal]

No it doesn't it will refetch any articles newer then the setting for oldest_article.

You can get older blog posts by directly parsing the html usng the parse_index function, see the user manual for how to do that.

[RickOlson]

Thanks, Kovid. Looks like I have a little project for the upcoming long weekend!

[galactus]

Originally Posted by kovidgoyal

No it doesn't it will refetch any articles newer then the setting for oldest_article.

You can get older blog posts by directly parsing the html usng the parse_index function, see the user manual for how to do that.

Hi!
I've got the same problem, since i would like to download an entire blog instead of just the latest items using the feed; i'm quite good in python so i guess i should be able to figure out by myself how to write the code, but i was wondering if someone has a generic recipe for blogs hosted by google (blogspot) and/or wordpress so i can save some time

Thanks in advance to anyone willing to help...

[paulkon]

There's a blogspot blogger that I like to read. It goes back for 5 years, and so I wanted a way to download past years.

At first I thought that Calibre had a limit on the number of articles it could download from a blog, because it wasn't getting all of the articles available. But obviously it was nowhere near this limit.

Luckily someone wrote code to do this download. If you search the Custom Recipes thread for the "Universal Blogspot Downloader" it'll get you here:
https://www.mobileread.com/forums/sho...postcount=1785
which gives you recipe code, and some tweaking instructions. You can tweak it to specify the range of years and months that you want. As written, it downloads comments, too.

I decided to remove the comments, and so only show articles. My slightly revised version is attached. All I did was add a few new entries in the "remove_tags" line, based on reading the orginal HTML carefully. Big kudos to EnergyLens for the original.

Paul

BTW if you want to have articles sorted from oldest to newest, add this line to your recipe:
reverse_article_order = True