Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 06-07-2013, 02:59 PM   #1
Down South
Junior Member
Down South began at the beginning.
 
Posts: 7
Karma: 10
Join Date: Jun 2013
Device: Kindle Keyboard
Large Library Performance Comparisons

I ran some performance comparisions to try and shed some light on
what was happening and possible solutions.
I used 6 drive configurations ranging in speed from an OCZ Diesel 8GB
USB flash drive to 2 Samsung 830 - 128GB SSD's in RAID 0.
I tested with and without a 1.5GB RAM drive for TEMP files and the metadata.db.

Using Process Monitor to see what Calibre was doing and Process Explorer
to see performance and contol CPU affinity allowed me to look at what was
happening when opening the library. (Sysinternals.com)

PROCESSING the data in the metadata.db file which is read in 1K chunks
is what takes all the time.
Any reasonably fast drive will give equal times to open or refresh Calibre.
Let me rephrase that. Use any drive you like. The speed will be the same.

Copying JUST the 51 MB metadata.db file from my 38,000 record library
to 6 different devices and then opening or refreshing the library
on each device took ~17 seconds (same as opening the actual library).

Underclocking and overclocking slow and speed up Calibre by
proportionate amounts of time.
CPU 50% faster 2.4GHZ -> 3.6 GHZ (3/2) = opens in 2/3 the time. ~11 secs.
CPU 2/3 the speed 1.6 GHZ = 3/2 the opening time. ~25 secs.

Calibre/SimpleSQL? uses only 1 processor/core so an Intel Q6600 (My CPU)
(4X 2.4 GHZ) is SLOWER than a single core @ 3 GHZ of the SAME family.
This can be seen in Process Explorer and CPU affinity from 1-4 CPU's
can be assigned to Calibre with benchmarks remaining the same.

Perhaps the more robust version of SQL uses Multiple CPU's?
If so I will be happy to contribute so Kovid can buy and use a
database engine that can multiprocess and open my library 3-4 times faster.

Please do NOT take my word for it. Run your own benchmarks and report back.
I live on a mountain in Mexico and only get to town for the internet every 7-14 days so please do not expect a rapid response from me.

Kovid: Thank you for providing such a fine, multifaceted program.
I KNOW you never expected it to take so much of your time and to
grow to what is has become which is a program being used far beyond it's
original design parameters.

If I remember correctly full versions of SQL ran on 8 or more CPU's and handled 100's of thousands of records with a search or update time under 5 seconds.
That is what we had in the mainframe world at clients like State Farm and AT&T.
I doubt the version of SQL in Calibre is designed to do so.
Down South is offline   Reply With Quote
Old 06-07-2013, 07:15 PM   #2
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Moderator Notice
You posted to a 2 year old thread and calibre has gone through about 100 revisions since then. Thank you for this info. I moved the post to its own thread.

Last edited by DoctorOhh; 06-07-2013 at 07:20 PM.
DoctorOhh is offline   Reply With Quote
Advert
Old 06-07-2013, 07:30 PM   #3
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 12,119
Karma: 73448614
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
While it is possible that other database engines might make use of more processing power and threads, remember that the challenge is to find a multi-platform engine. Don't forget that calibre runs on OS/X, Windows (32 and 64 bit), as well as Linux, so whatever engine is used HAS to support all of those environments.
PeterT is offline   Reply With Quote
Old 06-07-2013, 10:18 PM   #4
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,460
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by PeterT View Post
While it is possible that other database engines might make use of more processing power and threads, remember that the challenge is to find a multi-platform engine. Don't forget that calibre runs on OS/X, Windows (32 and 64 bit), as well as Linux, so whatever engine is used HAS to support all of those environments.
And it has to be free to use for the end user - donating towards buying Kovid a copy of a high performance enterprise database and the iron to run it on might be good for Kovid, but who's going to buy it for the rest of us.

Christine Lagarde could put in on her Get Out of Jail for Free card

BR
BetterRed is offline   Reply With Quote
Old 06-08-2013, 12:17 AM   #5
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
I am not sure what we are comparing here so sorry if this is not relevant.

In my large calibre library, moving from one edited fied to another takes approximately 8 seconds. In a smaller library it seems instant. However in the large library editing metadata and going from one edited record to the next seems instant and the changes show up in the GUI instantly. The only delay is when I exit from the edit metadata dialog to the GUI and it is still much shorter than the delay in moving from edited field to another field in the Gui.

Same 2 databases, converting 100 books takes upwards of 30 minutes in large library and calibre is in a non-resnsive state for about 10-30 minute after the job number is zero.
Calibre stops responding during the entire process and I am lucky to be able to play Solitaire The little icon in the corner does not whir around and clicking anywhere gives a white screen flash.

Same task in smaller library with same books takes less than 10 minutes total and both calibre and other programs are not noticeably slower. The little icon keeps whirring and I can view the job list etc.

I don't see this as a database problem, but I am not an expert.

Not complaining mind you, as it is my choice to have all my books and documents in one library.

Helen
speakingtohe is offline   Reply With Quote
Advert
Old 06-08-2013, 12:28 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,782
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@speakingtohe: Hide the Tag browser, that should improve your large library performance. After every edit, the entire tag browser has to be re-calculated, which is very slow on a large library.
kovidgoyal is online now   Reply With Quote
Old 06-08-2013, 12:47 AM   #7
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,460
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by speakingtohe View Post
Same 2 databases, converting 100 books takes upwards of 30 minutes in large library and calibre is in a non-resnsive state for about 10-30 minute after the job number is zero.
Could this behaviour be related to this thread - https://www.mobileread.com/forums/sho...d.php?t=215232 - I'm not sure if opf files are written after conversion

See recent exchange between theducks and myself

BR
BetterRed is offline   Reply With Quote
Old 06-08-2013, 01:53 AM   #8
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Originally Posted by kovidgoyal View Post
@speakingtohe: Hide the Tag browser, that should improve your large library performance. After every edit, the entire tag browser has to be re-calculated, which is very slow on a large library.
Well that made a huge difference in the speed of going from one edited field to another and the little icon is whirring away.


Thanks a great big bunch. I love the tag browser, but I guess I can hide it and unhide it. The spirit is willing but the memory is weak.)

Helen
speakingtohe is offline   Reply With Quote
Old 06-08-2013, 09:31 AM   #9
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,691
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Kovid
Have you thought of making a 'auto-hide' function similar to that available in Linux Unity or the Windows taskbar? n-seconds of no use and it tucks its self away.
theducks is offline   Reply With Quote
Old 06-08-2013, 11:37 AM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,782
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No need for needless complexity, I am working on a db backend re-write that (among other things) maintains a normalised view of the data in memory which should greatly improve the tag browsers performance. This is a long term project, though, so no promises.
kovidgoyal is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Performance issue with large library audeojude Calibre 25 12-07-2011 04:44 PM
Best way to get a large Calibre library into the PE library? Filark enTourage Archive 0 04-20-2011 10:18 PM
DX performance on large (>100MB) pdf files? Blackguard Amazon Kindle 6 06-16-2009 06:57 PM
Library size and performance MrFinch Calibre 4 01-01-2009 04:11 AM
Performance with large Libraries? itimpi Calibre 3 12-14-2008 02:46 PM


All times are GMT -4. The time now is 03:44 AM.


MobileRead.com is a privately owned, operated and funded community.