Big RSS feeds and mobi formating

Zorz · 01-25-2010, 03:16 AM

When trying to fetch news from a customized RSS feed, it seems calibre limits me to 120 articles or 20 pages of output on my kindle. There is a lot more material in the feed. I have set max_articles_per_feed = 999, and it still stops at 120. What can I do to have a bigger file.

Also why are some feeds from calibre when displayed on the kindle in a list and others the traditional Amazon Newspaper format?

DoctorOhh · 01-25-2010, 04:21 AM

Quote:

Originally Posted by LlamaZorz

When trying to fetch news from a customized RSS feed, it seems calibre limits me to 120 articles or 20 pages of output on my kindle. There is a lot more material in the feed. I have set max_articles_per_feed = 999, and it still stops at 120. What can I do to have a bigger file.

Did you also adjust the below code?

Code:

oldest_article        = 7 #days

This sets how many days to go back in the feed. Then there is always the possibility that the feed limits you to 120.

I had one feed that stopped at 10 no matter what things were set to. But in Google reader I was able to go further back. I solved that problem by using the Google reader recipe. Initially it still only allowed 10 but as they came in I was no longer limited.

Quote:

Originally Posted by LlamaZorz

Also why are some feeds from calibre when displayed on the kindle in a list and others the traditional Amazon Newspaper format?

Not having a Kindle I don't know what the "traditional Amazon Newspaper format" is except to suggest it was an industry way of dealing with feeds. What exactly does the traditional Amazon Newspaper format look like?

For that matter what does "a list" look like? If you mean that some feeds don't have a summary after each headline, again that is controlled by the site itself.

I hope my rambling at least let you know your not alone in trying to figure all of this out.

Zorz · 01-25-2010, 05:19 AM

Quote:

Originally Posted by dwanthny

Did you also adjust the below code?

Code:

oldest_article        = 7 #days

This sets how many days to go back in the feed. Then there is always the possibility that the feed limits you to 120.

I had one feed that stopped at 10 no matter what things were set to. But in Google reader I was able to go further back. I solved that problem by using the Google reader recipe. Initially it still only allowed 10 but as they came in I was no longer limited.

Not having a Kindle I don't know what the "traditional Amazon Newspaper format" is except to suggest it was an industry way of dealing with feeds. What exactly does the traditional Amazon Newspaper format look like?

For that matter what does "a list" look like? If you mean that some feeds don't have a summary after each headline, again that is controlled by the site itself.

I hope my rambling at least let you know your not alone in trying to figure all of this out.

Well the feed I am using has no dates or anything, its just a incredibly simple rss feed. The most basic form of RSS 2.0. I have both options set to 999 and still am locked into 120 articles. I wounder if anybody can confirm greater then 120 articles?

As for the different types of layouts, this is the list option

and this is the kindle catagory layout

I have no other experience with ereaders like yourself, so I don't know what's the standard. I just want to know what form does the input need to take, to make the output look like the second picture.

GRiker · 01-25-2010, 06:25 AM

Quote:

Originally Posted by LlamaZorz

Also why are some feeds from calibre when displayed on the kindle in a list and others the traditional Amazon Newspaper format?

If your recipe has only one section, then the articles are presented as a list on the Kindle. If your recipe has more than one section, then the sections are presented, and you can navigate to view the list of articles within the sections.

G

DoctorOhh · 01-25-2010, 06:28 AM

Quote:

Originally Posted by LlamaZorz

I have no other experience with ereaders like yourself, so I don't know what's the standard. I just want to know what form does the input need to take, to make the output look like the second picture.

Here is some reading on creating recipes for yourself. I would suggest that you ask the smart folks in this thread how to get feeds formatted exactly the way you like.

Just to let you know how those of us outside the Kindle exist I have attached 5 pictures.

My Collections page 4 of 5, My news is sent to the News collection area.
What my News collection has in it.
Raleigh News and Observer feeds.
Selecting the Cover sends me to the beginning of the list of articles that appear on the cover of the newspaper.
I select the article and end up on the first page of selected article.

Good Luck

Zorz · 01-25-2010, 12:04 PM

Quote:

Originally Posted by GRiker

If your recipe has only one section, then the articles are presented as a list on the Kindle. If your recipe has more than one section, then the sections are presented, and you can navigate to view the list of articles within the sections.

G

well im not using recipes. Just the default method of downloading the rss feed. Are those different sections categories in the RSS or are they different channels possibly?

GRiker · 01-25-2010, 12:56 PM

Feeds = sections. If you have one feed, that's considered a single-section recipe. If your feed has content within that you want separated into sections, you'll need to write a custom recipe.

G

DoctorOhh · 01-26-2010, 12:13 AM

Quote:

Originally Posted by LlamaZorz

well im not using recipes. Just the default method of downloading the rss feed. Are those different sections categories in the RSS or are they different channels possibly?

The default method of downloading the rss feed using Calibre is by using recipes. Since this post is in the Calibre forum I thought you were using Calibre to download your rss.

I'm curious, what is your default method of downloading the rss feed. Whispernet maybe?

Zorz · 01-26-2010, 05:44 AM

Quote:

Originally Posted by dwanthny

The default method of downloading the rss feed using Calibre is by using recipes. Since this post is in the Calibre forum I thought you were using Calibre to download your rss.

I'm curious, what is your default method of downloading the rss feed. Whispernet maybe?

You don't need a recipe to download a rss feed in calibre, uther then its internal facilities for handling RSS feeds.. You just add a feed in the gui and calibre handles the rest. It can be absolutely any RSS feed(though ive only tried 2.0) and it will make a ebook for you. I am using calibre to download my RSS/

DoctorOhh · 01-26-2010, 06:55 AM

Quote:

Originally Posted by LlamaZorz

You don't need a recipe to download a rss feed in calibre, uther then its internal facilities for handling RSS feeds.. You just add a feed in the gui and calibre handles the rest. It can be absolutely any RSS feed(though ive only tried 2.0) and it will make a ebook for you. I am using calibre to download my RSS/

Interesting, how do you add a feed in the gui?

Zorz · 01-26-2010, 03:27 PM

Quote:

Originally Posted by dwanthny

Interesting, how do you add a feed in the gui?

Atleast with the Linux Version

Fetch News -> Add Custom News Source -> Add Feed To Recipe, -> Add/update Feed. then you are done.

I am still limited to 120 articles or 20 pages no matter what source I use.

DoctorOhh · 01-26-2010, 09:39 PM

Quote:

Originally Posted by LlamaZorz

Atleast with the Linux Version

Fetch News -> Add Custom News Source -> Add Feed To Recipe, -> Add/update Feed. then you are done.

I am still limited to 120 articles or 20 pages no matter what source I use.

I have good news and bad news.

The good news. I now understand. You are using a recipe. A basic custom recipe that you created. Knowing this I can answer your question.

The bad news. Your feed limits you to 120 articles. I have one feed that if I do what your saying I only get 10 articles. Most days that doesn't even cover the articles for the day. I have another feed that limits me to 100 articles, most days that covers it but there have been days where the content for the day has been well past 100 articles.

More good news. To get by the 10 article limit on the one feed I just subscribe to it via Google Reader. Then I use the recipe for Google Reader. Even if I wait a week and there are 203 articles they will be waiting for me to download using the Google Reader recipe.

You too can subscribe to your feed using Google reader. Initially it won't let you go back any more then the 120 they allow but it will catch all articles going forward. If you wait a week and 329 articles were supplied by that feed then they will be in your Google Reader waiting for you to use the recipe to download them. Even though the main feed will still only allow you to download the last 120.

For more insight into the Google Reader recipe check out this post.

To use this you will need a Gmail account.

Zorz · 01-26-2010, 11:03 PM

The problem is my feed doesn't limit anything it contains no dates or time data. I can use many different agregators as well as feedbooks and they work great. They get every item of the RSS feed. It is only calibre which limits me.

Here is a sample of the RSS feed. As you can see its incredibly basic and uniform
http://paste2.org/p/634868

DoctorOhh · 01-27-2010, 02:08 AM

Quote:

Originally Posted by LlamaZorz

The problem is my feed doesn't limit anything it contains no dates or time data. I can use many different agregators as well as feedbooks and they work great. They get every item of the RSS feed. It is only calibre which limits me.

Here is a sample of the RSS feed. As you can see its incredibly basic and uniform
http://paste2.org/p/634868

Update: It totally blew right past me that this was a custom feed. Maybe my rambling will help others.

Let me clarify things. In a news aggregator the first thing it does when you subscribe to a rss feed is go out and grabs all of the new items. This list of new items is what I'm referring to when I say your feed limits you to 120 new items.

The list of new items is what Calibre is able to get via the feed.

If you are using an interactive aggregator, when you scroll back past the new items it goes and asks the web site for the previous (archived) articles in the feed. When you scroll past that list it will ask for the (archived) articles before that. In very short order you can go back 1000s of articles.

Now if the rss feed didn't have the ability for a webmaster or web host (I don't know who sets it) to set a limit on the initial feed all hell would break loose, one query would return 1000s of archived articles.

Calibre recipes operate within this new items limit. If you want you can use Calibre to only get 40 of those 120 or use it to only get the articles within the set of 120 that are 4 days old or less , but you can't bust past that initial new item limit without being more interactive.

Google reader (or other aggregator) is the solution for this. There is a recipe for Google reader. You know how aggregators keep collecting every new item sent from the feed. So when the recipe goes to Google Reader it can download however many new items that Google reader has collected. Currently my Google Reader as 24 new articles from the site that limits me to 10 new items.

I hope I made some sense in this rambling of mine.

Maybe Calibre does have a 120 article limit per feed?

Zorz · 01-27-2010, 02:35 AM

My feed has no dates or other meta data other then link, title and description. So there never are new articles, it is what it is. The next time you update you get a completely different feed with absolutely no resemblance to the last. The feed is not even online, it needs to be uploaded to a ftp.

Sorry for the non traditional method, but I feel this is the best way to get news. I just use calibre to make the ebook.

Thanks for your responses though they have been enlightening.

01-25-2010, 03:16 AM	#1
Zorz Zealot Posts: 134 Karma: 36 Join Date: Nov 2009 Device: Kindle 2i	Big RSS feeds and mobi formating When trying to fetch news from a customized RSS feed, it seems calibre limits me to 120 articles or 20 pages of output on my kindle. There is a lot more material in the feed. I have set max_articles_per_feed = 999, and it still stops at 120. What can I do to have a bigger file. Also why are some feeds from calibre when displayed on the kindle in a list and others the traditional Amazon Newspaper format?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Is there a good way to convert partial rss to full rss feeds.	Zorz	Other formats	5	05-29-2010 12:17 PM
RSS feeds	peejay	PocketBook	2	04-26-2010 05:16 AM
PRS-300 RSS Feeds	denmarks	Sony Reader	1	10-06-2009 01:41 PM
How can I aggregate News/Rss feeds to .mobi	tinybilbo	Bookeen	5	11-08-2008 02:07 PM
RSS feeds	ichor	iRex	1	03-01-2008 11:30 PM

01-25-2010, 12:56 PM	#7
GRiker Comparer of the Ephemeris Posts: 1,496 Karma: 424697 Join Date: Mar 2009 Device: iPad	Feeds = sections. If you have one feed, that's considered a single-section recipe. If your feed has content within that you want separated into sections, you'll need to write a custom recipe. G

01-26-2010, 11:03 PM	#13
Zorz Zealot Posts: 134 Karma: 36 Join Date: Nov 2009 Device: Kindle 2i	The problem is my feed doesn't limit anything it contains no dates or time data. I can use many different agregators as well as feedbooks and they work great. They get every item of the RSS feed. It is only calibre which limits me. Here is a sample of the RSS feed. As you can see its incredibly basic and uniform http://paste2.org/p/634868

01-27-2010, 02:35 AM	#15
Zorz Zealot Posts: 134 Karma: 36 Join Date: Nov 2009 Device: Kindle 2i	My feed has no dates or other meta data other then link, title and description. So there never are new articles, it is what it is. The next time you update you get a completely different feed with absolutely no resemblance to the last. The feed is not even online, it needs to be uploaded to a ftp. Sorry for the non traditional method, but I feel this is the best way to get news. I just use calibre to make the ebook. Thanks for your responses though they have been enlightening.

Advert

Advert