03-16-2011, 10:31 PM | #1 |
Junior Member
Posts: 3
Karma: 10
Join Date: Dec 2010
Device: Kindle
|
Performance of Fetching News
If this is the wrong forum, I apologize.
I have just started using Calibre (I am three days into it) and am particularly impressed with the automatic news fetching capability. I do have some comments on the performance of this and how to improve it. First, I run Calibre on a Mac and the target device is a Kindle so that is what these suggestions apply to. Also, I decided to email the results to my kindle.com email address. To see how the news feature worked and which feeds I wanted, I turned on about 50 of the built-in feeds, told Calibre to fetch them all and ran with all default settings. On my older dual-core iMac this ran for hours. Looking into this, I found two thing to speed this up. 1) In preferences, in the advanced area under miscellaneous set: "Maximum number of waiting processes" to 6 (or more if you have more than 2 cores). Also, uncheck the box that says "Limit the max. simultaneous jobs to the available CPU cores". (Note: I also tried setting this to 4 and to 8. 4 left about half of one core idle. 6 kept both cores busy with very little idle time. 8 also kept both cores busy but seemed to take a little longer to process all the feeds. The machine was idle except for Calibre during this.) My tentative rule of thumb for maximizing processing performance is to set this to: #cores*2+2 2) Under tweaks, under advanced in preferences change "# The number of seconds to wait before sending emails public_smtp_relay_delay = 301" from 301 to 0. The delay is to prevent some email servers from rejecting the mail from Calibre as spam. This does not seem to be a problem with the amazon kindle email. The first run I did took 4 additional hours to complete because Calibre was waiting 5 minutes between sending each email. With both these changes the wall clock time for fetching all the feeds, converting them and emailing them to the kindle went from about 6 hours to one hour. Anyway, I really like Calibre and hope this helps someone. Please fell free to correct me if I have anything incorrect or unclear. |
03-17-2011, 12:10 AM | #2 | |
US Navy, Retired
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
Thanks for sharing your experience. |
|
03-17-2011, 12:41 AM | #3 |
creator of calibre
Posts: 43,796
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
No the delay is implemented if you are using gmail or hotmail to send mails. It has nothing to do with amazon.
|
03-17-2011, 12:53 AM | #4 |
US Navy, Retired
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
|
03-17-2011, 01:26 AM | #5 |
Enthusiast
Posts: 43
Karma: 136
Join Date: Mar 2011
Device: Kindle Paperwhite
|
|
03-17-2011, 01:45 AM | #6 |
creator of calibre
Posts: 43,796
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
yes .
|
03-17-2011, 04:52 AM | #7 |
Wizard
Posts: 3,130
Karma: 91256
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
|
I think that this only works because the bottleneck would be the actual fetching of the news from the websites, so the different jobs are basically idling while waiting for a response from the server. You could be seeing vastly different results when converting, which is relatively CPU intensive all the time.
|
03-17-2011, 09:25 AM | #8 |
Junior Member
Posts: 3
Karma: 10
Join Date: Dec 2010
Device: Kindle
|
As I understand it, Calibre processes come in pairs; one process for conversion/download and one for adding saving. Since conversion, as you point out is far more compute intensive you need Calibre to start at least 4 processes in order to have two conversion processes running. I tried 6 and 8 ( 3 and 4 conversion processes) in order to to have at least one spare conversion process able to run when one of the others was idle, waiting for IO to complete.
The crude experiments I tried, showed that on a dual core machine that is idle except for Caliber running 3 conversion processes (6 calibre.parallel processes) kept the both cores running close to 100% until all the conversions were done. One could run more than 6 processes but this adds overhead as the cores switch from process to process. 8 processes were a little slower than 6. |
03-17-2011, 11:33 AM | #9 |
Sigil & calibre developer
Posts: 2,488
Karma: 1063785
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
|
If you don't have the preferred output format in preferences set to MOBI then the process will take longer. It defaults to EPUB so if you havent changed it it will download and geberate an EPUB then convert the file to MOBI before sending to the Kindle.
|
03-17-2011, 01:54 PM | #10 |
Junior Member
Posts: 3
Karma: 10
Join Date: Dec 2010
Device: Kindle
|
Thanks very much.
I had changed it to MOBI (I think; I should double check) but I didn't realize there was a performance implication. |
04-08-2011, 05:44 PM | #11 |
Enthusiast
Posts: 43
Karma: 136
Join Date: Mar 2011
Device: Kindle Paperwhite
|
|
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Fetching News has gone bad... | rogue_ronin | Calibre | 6 | 09-03-2010 08:41 AM |
Trouble fetching news | jpustin | Calibre | 6 | 06-28-2010 01:57 PM |
automating news fetching | zerozombie72 | Calibre | 6 | 02-16-2010 04:31 PM |
Fetching News In Calibre | Rootman | Calibre | 2 | 11-11-2009 07:06 PM |
Question about fetching the news | spoudaios | Sony Reader Dev Corner | 4 | 01-27-2008 05:01 PM |