Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 01-25-2010, 03:16 AM   #1
Zorz
Zealot
Zorz began at the beginning.
 
Posts: 134
Karma: 36
Join Date: Nov 2009
Device: Kindle 2i
Big RSS feeds and mobi formating

When trying to fetch news from a customized RSS feed, it seems calibre limits me to 120 articles or 20 pages of output on my kindle. There is a lot more material in the feed. I have set max_articles_per_feed = 999, and it still stops at 120. What can I do to have a bigger file.

Also why are some feeds from calibre when displayed on the kindle in a list and others the traditional Amazon Newspaper format?
Zorz is offline   Reply With Quote
Old 01-25-2010, 04:21 AM   #2
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,773
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by LlamaZorz View Post
When trying to fetch news from a customized RSS feed, it seems calibre limits me to 120 articles or 20 pages of output on my kindle. There is a lot more material in the feed. I have set max_articles_per_feed = 999, and it still stops at 120. What can I do to have a bigger file.
Did you also adjust the below code?

Code:
oldest_article        = 7 #days
This sets how many days to go back in the feed. Then there is always the possibility that the feed limits you to 120.

I had one feed that stopped at 10 no matter what things were set to. But in Google reader I was able to go further back. I solved that problem by using the Google reader recipe. Initially it still only allowed 10 but as they came in I was no longer limited.

Quote:
Originally Posted by LlamaZorz View Post
Also why are some feeds from calibre when displayed on the kindle in a list and others the traditional Amazon Newspaper format?
Not having a Kindle I don't know what the "traditional Amazon Newspaper format" is except to suggest it was an industry way of dealing with feeds. What exactly does the traditional Amazon Newspaper format look like?

For that matter what does "a list" look like? If you mean that some feeds don't have a summary after each headline, again that is controlled by the site itself.

I hope my rambling at least let you know your not alone in trying to figure all of this out.
DoctorOhh is offline   Reply With Quote
Old 01-25-2010, 05:19 AM   #3
Zorz
Zealot
Zorz began at the beginning.
 
Posts: 134
Karma: 36
Join Date: Nov 2009
Device: Kindle 2i
Quote:
Originally Posted by dwanthny View Post
Did you also adjust the below code?

Code:
oldest_article        = 7 #days
This sets how many days to go back in the feed. Then there is always the possibility that the feed limits you to 120.

I had one feed that stopped at 10 no matter what things were set to. But in Google reader I was able to go further back. I solved that problem by using the Google reader recipe. Initially it still only allowed 10 but as they came in I was no longer limited.



Not having a Kindle I don't know what the "traditional Amazon Newspaper format" is except to suggest it was an industry way of dealing with feeds. What exactly does the traditional Amazon Newspaper format look like?

For that matter what does "a list" look like? If you mean that some feeds don't have a summary after each headline, again that is controlled by the site itself.

I hope my rambling at least let you know your not alone in trying to figure all of this out.
Well the feed I am using has no dates or anything, its just a incredibly simple rss feed. The most basic form of RSS 2.0. I have both options set to 999 and still am locked into 120 articles. I wounder if anybody can confirm greater then 120 articles?

As for the different types of layouts, this is the list option



and this is the kindle catagory layout


I have no other experience with ereaders like yourself, so I don't know what's the standard. I just want to know what form does the input need to take, to make the output look like the second picture.

Last edited by Zorz; 01-25-2010 at 05:24 AM.
Zorz is offline   Reply With Quote
Old 01-25-2010, 06:25 AM   #4
GRiker
Comparer of the Ephemeris
GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.
 
Posts: 1,497
Karma: 424627
Join Date: Mar 2009
Device: iPad
Quote:
Originally Posted by LlamaZorz View Post
Also why are some feeds from calibre when displayed on the kindle in a list and others the traditional Amazon Newspaper format?
If your recipe has only one section, then the articles are presented as a list on the Kindle. If your recipe has more than one section, then the sections are presented, and you can navigate to view the list of articles within the sections.

G
GRiker is offline   Reply With Quote
Old 01-25-2010, 06:28 AM   #5
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,773
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by LlamaZorz View Post
I have no other experience with ereaders like yourself, so I don't know what's the standard. I just want to know what form does the input need to take, to make the output look like the second picture.
Here is some reading on creating recipes for yourself. I would suggest that you ask the smart folks in this thread how to get feeds formatted exactly the way you like.

Just to let you know how those of us outside the Kindle exist I have attached 5 pictures.
  1. My Collections page 4 of 5, My news is sent to the News collection area.
  2. What my News collection has in it.
  3. Raleigh News and Observer feeds.
  4. Selecting the Cover sends me to the beginning of the list of articles that appear on the cover of the newspaper.
  5. I select the article and end up on the first page of selected article.
Good Luck
Attached Thumbnails
Click image for larger version

Name:	2010-1-25.jpg
Views:	92
Size:	56.1 KB
ID:	44219   Click image for larger version

Name:	2010-1-25_0.jpg
Views:	88
Size:	55.3 KB
ID:	44220   Click image for larger version

Name:	2010-1-25_1.jpg
Views:	81
Size:	29.7 KB
ID:	44221   Click image for larger version

Name:	2010-1-25_2.jpg
Views:	93
Size:	70.3 KB
ID:	44222   Click image for larger version

Name:	2010-1-25_3.jpg
Views:	110
Size:	81.2 KB
ID:	44223  

Last edited by DoctorOhh; 01-25-2010 at 06:32 AM.
DoctorOhh is offline   Reply With Quote
Old 01-25-2010, 12:04 PM   #6
Zorz
Zealot
Zorz began at the beginning.
 
Posts: 134
Karma: 36
Join Date: Nov 2009
Device: Kindle 2i
Quote:
Originally Posted by GRiker View Post
If your recipe has only one section, then the articles are presented as a list on the Kindle. If your recipe has more than one section, then the sections are presented, and you can navigate to view the list of articles within the sections.

G
well im not using recipes. Just the default method of downloading the rss feed. Are those different sections categories in the RSS or are they different channels possibly?
Zorz is offline   Reply With Quote
Old 01-25-2010, 12:56 PM   #7
GRiker
Comparer of the Ephemeris
GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.
 
Posts: 1,497
Karma: 424627
Join Date: Mar 2009
Device: iPad
Feeds = sections. If you have one feed, that's considered a single-section recipe. If your feed has content within that you want separated into sections, you'll need to write a custom recipe.

G
GRiker is offline   Reply With Quote
Old 01-26-2010, 12:13 AM   #8
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,773
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by LlamaZorz View Post
well im not using recipes. Just the default method of downloading the rss feed. Are those different sections categories in the RSS or are they different channels possibly?
The default method of downloading the rss feed using Calibre is by using recipes. Since this post is in the Calibre forum I thought you were using Calibre to download your rss.

I'm curious, what is your default method of downloading the rss feed. Whispernet maybe?
DoctorOhh is offline   Reply With Quote
Old 01-26-2010, 05:44 AM   #9
Zorz
Zealot
Zorz began at the beginning.
 
Posts: 134
Karma: 36
Join Date: Nov 2009
Device: Kindle 2i
Quote:
Originally Posted by dwanthny View Post
The default method of downloading the rss feed using Calibre is by using recipes. Since this post is in the Calibre forum I thought you were using Calibre to download your rss.

I'm curious, what is your default method of downloading the rss feed. Whispernet maybe?
You don't need a recipe to download a rss feed in calibre, uther then its internal facilities for handling RSS feeds.. You just add a feed in the gui and calibre handles the rest. It can be absolutely any RSS feed(though ive only tried 2.0) and it will make a ebook for you. I am using calibre to download my RSS/
Zorz is offline   Reply With Quote
Old 01-26-2010, 06:55 AM   #10
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,773
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by LlamaZorz View Post
You don't need a recipe to download a rss feed in calibre, uther then its internal facilities for handling RSS feeds.. You just add a feed in the gui and calibre handles the rest. It can be absolutely any RSS feed(though ive only tried 2.0) and it will make a ebook for you. I am using calibre to download my RSS/
Interesting, how do you add a feed in the gui?
DoctorOhh is offline   Reply With Quote
Old 01-26-2010, 03:27 PM   #11
Zorz
Zealot
Zorz began at the beginning.
 
Posts: 134
Karma: 36
Join Date: Nov 2009
Device: Kindle 2i
Quote:
Originally Posted by dwanthny View Post
Interesting, how do you add a feed in the gui?
Atleast with the Linux Version


Fetch News -> Add Custom News Source -> Add Feed To Recipe, -> Add/update Feed. then you are done.

I am still limited to 120 articles or 20 pages no matter what source I use.
Zorz is offline   Reply With Quote
Old 01-26-2010, 09:39 PM   #12
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,773
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by LlamaZorz View Post
Atleast with the Linux Version


Fetch News -> Add Custom News Source -> Add Feed To Recipe, -> Add/update Feed. then you are done.

I am still limited to 120 articles or 20 pages no matter what source I use.
I have good news and bad news.

The good news. I now understand. You are using a recipe. A basic custom recipe that you created. Knowing this I can answer your question.

The bad news. Your feed limits you to 120 articles. I have one feed that if I do what your saying I only get 10 articles. Most days that doesn't even cover the articles for the day. I have another feed that limits me to 100 articles, most days that covers it but there have been days where the content for the day has been well past 100 articles.

More good news. To get by the 10 article limit on the one feed I just subscribe to it via Google Reader. Then I use the recipe for Google Reader. Even if I wait a week and there are 203 articles they will be waiting for me to download using the Google Reader recipe.

You too can subscribe to your feed using Google reader. Initially it won't let you go back any more then the 120 they allow but it will catch all articles going forward. If you wait a week and 329 articles were supplied by that feed then they will be in your Google Reader waiting for you to use the recipe to download them. Even though the main feed will still only allow you to download the last 120.

For more insight into the Google Reader recipe check out this post.

To use this you will need a Gmail account.

Last edited by DoctorOhh; 01-26-2010 at 09:48 PM.
DoctorOhh is offline   Reply With Quote
Old 01-26-2010, 11:03 PM   #13
Zorz
Zealot
Zorz began at the beginning.
 
Posts: 134
Karma: 36
Join Date: Nov 2009
Device: Kindle 2i
The problem is my feed doesn't limit anything it contains no dates or time data. I can use many different agregators as well as feedbooks and they work great. They get every item of the RSS feed. It is only calibre which limits me.

Here is a sample of the RSS feed. As you can see its incredibly basic and uniform
http://paste2.org/p/634868
Zorz is offline   Reply With Quote
Old 01-27-2010, 02:08 AM   #14
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,773
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by LlamaZorz View Post
The problem is my feed doesn't limit anything it contains no dates or time data. I can use many different agregators as well as feedbooks and they work great. They get every item of the RSS feed. It is only calibre which limits me.

Here is a sample of the RSS feed. As you can see its incredibly basic and uniform
http://paste2.org/p/634868
Update: It totally blew right past me that this was a custom feed. Maybe my rambling will help others.

Let me clarify things. In a news aggregator the first thing it does when you subscribe to a rss feed is go out and grabs all of the new items. This list of new items is what I'm referring to when I say your feed limits you to 120 new items.

The list of new items is what Calibre is able to get via the feed.

If you are using an interactive aggregator, when you scroll back past the new items it goes and asks the web site for the previous (archived) articles in the feed. When you scroll past that list it will ask for the (archived) articles before that. In very short order you can go back 1000s of articles.

Now if the rss feed didn't have the ability for a webmaster or web host (I don't know who sets it) to set a limit on the initial feed all hell would break loose, one query would return 1000s of archived articles.

Calibre recipes operate within this new items limit. If you want you can use Calibre to only get 40 of those 120 or use it to only get the articles within the set of 120 that are 4 days old or less , but you can't bust past that initial new item limit without being more interactive.

Google reader (or other aggregator) is the solution for this. There is a recipe for Google reader. You know how aggregators keep collecting every new item sent from the feed. So when the recipe goes to Google Reader it can download however many new items that Google reader has collected. Currently my Google Reader as 24 new articles from the site that limits me to 10 new items.

I hope I made some sense in this rambling of mine.

Maybe Calibre does have a 120 article limit per feed?

Last edited by DoctorOhh; 01-27-2010 at 02:23 AM.
DoctorOhh is offline   Reply With Quote
Old 01-27-2010, 02:35 AM   #15
Zorz
Zealot
Zorz began at the beginning.
 
Posts: 134
Karma: 36
Join Date: Nov 2009
Device: Kindle 2i
My feed has no dates or other meta data other then link, title and description. So there never are new articles, it is what it is. The next time you update you get a completely different feed with absolutely no resemblance to the last. The feed is not even online, it needs to be uploaded to a ftp.

Sorry for the non traditional method, but I feel this is the best way to get news. I just use calibre to make the ebook.

Thanks for your responses though they have been enlightening.
Zorz is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Is there a good way to convert partial rss to full rss feeds. Zorz Other formats 5 05-29-2010 12:17 PM
RSS feeds peejay PocketBook 2 04-26-2010 05:16 AM
PRS-300 RSS Feeds denmarks Sony Reader 1 10-06-2009 01:41 PM
How can I aggregate News/Rss feeds to .mobi tinybilbo Bookeen 5 11-08-2008 02:07 PM
RSS feeds ichor iRex 1 03-01-2008 11:30 PM


All times are GMT -4. The time now is 07:24 AM.


MobileRead.com is a privately owned, operated and funded community.