Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 03-16-2011, 10:31 PM   #1
jkdawson
Junior Member
jkdawson began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Dec 2010
Device: Kindle
Performance of Fetching News

If this is the wrong forum, I apologize.

I have just started using Calibre (I am three days into it) and am particularly impressed with the automatic news fetching capability. I do have some comments on the performance of this and how to improve it.

First, I run Calibre on a Mac and the target device is a Kindle so that is what these suggestions apply to. Also, I decided to email the results to my kindle.com email address.

To see how the news feature worked and which feeds I wanted, I turned on about 50 of the built-in feeds, told Calibre to fetch them all and ran with all default settings. On my older dual-core iMac this ran for hours.

Looking into this, I found two thing to speed this up.

1) In preferences, in the advanced area under miscellaneous set:
"Maximum number of waiting processes" to 6 (or more if you have more than 2 cores). Also, uncheck the box that says "Limit the max. simultaneous jobs to the available CPU cores". (Note: I also tried setting this to 4 and to 8. 4 left about half of one core idle. 6 kept both cores busy with very little idle time. 8 also kept both cores busy but seemed to take a little longer to process all the feeds. The machine was idle except for Calibre during this.) My tentative rule of thumb for maximizing processing performance is to set this to: #cores*2+2

2) Under tweaks, under advanced in preferences change
"# The number of seconds to wait before sending emails

public_smtp_relay_delay = 301" from 301 to 0.

The delay is to prevent some email servers from rejecting the mail from Calibre as spam. This does not seem to be a problem with the amazon kindle email. The first run I did took 4 additional hours to complete because Calibre was waiting 5 minutes between sending each email.

With both these changes the wall clock time for fetching all the feeds, converting them and emailing them to the kindle went from about 6 hours to one hour.

Anyway, I really like Calibre and hope this helps someone. Please fell free to correct me if I have anything incorrect or unclear.
jkdawson is offline   Reply With Quote
Old 03-17-2011, 12:10 AM   #2
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,565
Karma: 12369681
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by jkdawson View Post
2) Under tweaks, under advanced in preferences change
"# The number of seconds to wait before sending emails

public_smtp_relay_delay = 301" from 301 to 0.

The delay is to prevent some email servers from rejecting the mail from Calibre as spam. This does not seem to be a problem with the amazon kindle email.
Good information. As I recall the delay was implemented because of Amazon. Maybe Amazon corrected their problem.

Thanks for sharing your experience.
DoctorOhh is offline   Reply With Quote
 
Enthusiast
Old 03-17-2011, 12:41 AM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 24,769
Karma: 4369667
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No the delay is implemented if you are using gmail or hotmail to send mails. It has nothing to do with amazon.
kovidgoyal is online now   Reply With Quote
Old 03-17-2011, 12:53 AM   #4
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,565
Karma: 12369681
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by kovidgoyal View Post
No the delay is implemented if you are using gmail or hotmail to send mails. It has nothing to do with amazon.
Understood.
DoctorOhh is offline   Reply With Quote
Old 03-17-2011, 01:26 AM   #5
aerodynamik
Enthusiast
aerodynamik doesn't litteraerodynamik doesn't litter
 
Posts: 41
Karma: 136
Join Date: Mar 2011
Device: Kindle Paperwhite
Quote:
Originally Posted by kovidgoyal View Post
No the delay is implemented if you are using gmail or hotmail to send mails. It has nothing to do with amazon.
And why exactly? Because Gmail would not allow to send out 10 emails within a minute?

Thanks in advance.

- aerodynamik
aerodynamik is offline   Reply With Quote
Old 03-17-2011, 01:45 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 24,769
Karma: 4369667
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
yes .
kovidgoyal is online now   Reply With Quote
Old 03-17-2011, 04:52 AM   #7
Manichean
Wizard
Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!Manichean My eyes! My eyes! The light is just too bright!
 
Manichean's Avatar
 
Posts: 3,130
Karma: 80446
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
Quote:
Originally Posted by jkdawson View Post
Also, uncheck the box that says "Limit the max. simultaneous jobs to the available CPU cores".
I think that this only works because the bottleneck would be the actual fetching of the news from the websites, so the different jobs are basically idling while waiting for a response from the server. You could be seeing vastly different results when converting, which is relatively CPU intensive all the time.
Manichean is offline   Reply With Quote
Old 03-17-2011, 09:25 AM   #8
jkdawson
Junior Member
jkdawson began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Dec 2010
Device: Kindle
As I understand it, Calibre processes come in pairs; one process for conversion/download and one for adding saving. Since conversion, as you point out is far more compute intensive you need Calibre to start at least 4 processes in order to have two conversion processes running. I tried 6 and 8 ( 3 and 4 conversion processes) in order to to have at least one spare conversion process able to run when one of the others was idle, waiting for IO to complete.

The crude experiments I tried, showed that on a dual core machine that is idle except for Caliber running 3 conversion processes (6 calibre.parallel processes) kept the both cores running close to 100% until all the conversions were done.

One could run more than 6 processes but this adds overhead as the cores switch from process to process. 8 processes were a little slower than 6.
jkdawson is offline   Reply With Quote
Old 03-17-2011, 11:33 AM   #9
user_none
Sigil & calibre developer
user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.user_none ought to be getting tired of karma fortunes by now.
 
user_none's Avatar
 
Posts: 2,427
Karma: 950001
Join Date: Jan 2009
Location: Florida, USA
Device: Nook STR
If you don't have the preferred output format in preferences set to MOBI then the process will take longer. It defaults to EPUB so if you havent changed it it will download and geberate an EPUB then convert the file to MOBI before sending to the Kindle.
user_none is offline   Reply With Quote
Old 03-17-2011, 01:54 PM   #10
jkdawson
Junior Member
jkdawson began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Dec 2010
Device: Kindle
Thanks very much.

I had changed it to MOBI (I think; I should double check) but I didn't realize there was a performance implication.
jkdawson is offline   Reply With Quote
Old 04-08-2011, 05:44 PM   #11
aerodynamik
Enthusiast
aerodynamik doesn't litteraerodynamik doesn't litter
 
Posts: 41
Karma: 136
Join Date: Mar 2011
Device: Kindle Paperwhite
Quote:
Originally Posted by aerodynamik View Post
And why exactly? Because Gmail would not allow to send out 10 emails within a minute?
Just thought I share this: using Gmail I changed the value to 10 seconds and have not run into any issues yet.

Cheers,
- aerodynamik
aerodynamik is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Fetching News has gone bad... rogue_ronin Calibre 6 09-03-2010 08:41 AM
Trouble fetching news jpustin Calibre 6 06-28-2010 01:57 PM
automating news fetching zerozombie72 Calibre 6 02-16-2010 04:31 PM
Fetching News In Calibre Rootman Calibre 2 11-11-2009 07:06 PM
Question about fetching the news spoudaios Sony Reader Dev Corner 4 01-27-2008 05:01 PM


All times are GMT -4. The time now is 04:24 AM.


MobileRead.com is a privately owned, operated and funded community.