Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 03-18-2013, 10:50 PM   #1
audeojude
Enthusiast
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 25
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
Calibre performance

I thought it would be interesting to report on calibre performance. I just purchased and built a new modern computer with a lot of resources and right now have a good contrast with my old system that was a top end system from about 6 or 7 years ago.

Library size is about 120,000 books, 250 gigs, and a DB file 166 megabytes in size. The directory structure in the calibre library folder has tens of thousands of directories in it which was a huge performance hit.

Operating system is Ubuntu Linux 12.04 for both computers. Just moved the same OS drive from one computer to the next. OS drive is a 100 gig ocz 4x PCI-E ssd raid card with 4 25 gig drives on it in a stripped linux software raid configuration. 200 megs a second through put on the old computer 600 megs a second throughput with the new system.

Last system was a Intel Core2 Quad Core system running at 2.66 ghz with 8 gigs of ram with calibre living on a 3 terrabyte sata 3 internal drive that was on a intell sata 2 hard drive controller.

Opening up calibre took about 2 minutes. A search would take anywhere from a minute to a minute and a half+ This was faster than when I had the library on a older sata 2 hard drive.

New system is a Intel i7-3770T 45 watt cpu Running at 2.5 GHZ with 32 gigs of ram on a z77 motherboard with sata 3 controllers. The 3 terabyte drives have been moved over into that system.

Opening up calibre now takes about 30 seconds to 45 seconds and a search is anywhere from 15 to 40 seconds. After upgrading the system to 32 gigs of ram calibre is using just over 6 gigs of ram to run. CPU usage seems to max out at about 10 to 12 % averaged over all 8 hyper threaded cores when doing a database check or other intensive search. Several cores hitting 100% for extended periods but not slowing the overall system.

My system typically consumes about 15 watts of power idling at the desktop just doing light internet usage. Running a search or database check will pushed the cpu cores to about 10 to 12% and doubled power consumption to about 30 watts. Old system consumed about 130 watts idling.

I just found it interesting the difference in performance in calibre between the systems. I think that most of the performance gain is simply IO going from sata 2 to sata 3. I have future plans to get a large 500 gig ssd specd to around 500 megs a second. just for calibre. that should speed up access 3 or 4 fold if I'm right.

I'm way at the extreme end for number of files and size of database and calibre just chunks through it. It definitely impacts performance once the library grows past about 20000 books but even up to the 120000 I have it is reliable if a bit slow. Again I think a lot of the slowness is due to simple IO with a huge directory structure. Neither Linux nor windows like large numbers of directory's and files in a root directory. I have looked a multiple file system formats to see if I could use something other than NTFS or EXT4 but they seem to be a bit more robust and get decent average benchmarks even against some of the exotic file systems out there.


Just my thoughts on the comparison between the systems.
audeojude is offline   Reply With Quote
Old 03-18-2013, 11:34 PM   #2
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,006
Karma: 3512570
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
I don't know if you've seen this thread: http://www.mobileread.com/forums/sho....php?p=2244745

A few ideas on how to speed up calibre up a little.

I don't think that the number of files and folders is a problem at all for ext4. Compared to things like mail servers calibre is tame.

I have my library on a relatively slow Synology 411j NAS, using NFS, but the database on a local SSD and temp in RAM using tempfs. Most of the things you do in calibre, change metadata, search, tag, download metadata and so on, never touch the actual books, only the database. And if it does, most of the work/updates are done in the temp folder, not in the library folders.

Usually the normal disk caches are enough to give a significant speed boost. But it is fun to tinker some more...

Save a few thousand books to disk, and change the format on the fly and/or update metadata using a plugboard. That is a nice stress test... ;D

Last edited by Adoby; 03-18-2013 at 11:54 PM.
Adoby is offline   Reply With Quote
 
Enthusiast
Old 03-19-2013, 01:12 AM   #3
audeojude
Enthusiast
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 25
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
I just spent the last couple hours playing with it and am puzzled. I moved the metadata.db file and created a soft link it on the the pci-e flash drive that has 600 mgs a sec through put and didn't see much performance increase over the 140 meg a second sata drive in searching or doing a search.

Moved the file to a ram drive tmpfs under /tmp on ubuntu and the speed of starting calibre was very contemporary to both the hard drive and the ssd pci-e card, same for the search. So..... im confused. the ram drive is throughput can be measured in gigs a second vs 100 to 100's of megs a second but im not seeing much speed improvement at all.
audeojude is offline   Reply With Quote
Old 03-19-2013, 02:22 AM   #4
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,006
Karma: 3512570
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
The disk cache is very efficient. The whole database is cached in RAM. And the books are not accessed at all during searches and startups.

Do some tests where you change and convert ebooks. Perhaps that will be different?
Adoby is offline   Reply With Quote
Old 03-19-2013, 10:35 AM   #5
audeojude
Enthusiast
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 25
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
Yep it is efficient. I was testing it by moving 2.2 gig files to it off the PCI-E ssd card and that were from a human perspective copying instantaneously. So well in excess of 2 gigs of throughput, though I don't see how the PCI-E card was feeding the file there that fast.

really bizare I went to play with it again this morning and tweaked some settings re the tmpfs and now the system won't mount /tmp on tmpfs..

Anyway.. I will have to come back to it at a later date. Work calls.
audeojude is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
calibre performance settings? unboggling Calibre 2 08-05-2012 09:08 PM
touchscreen performance sebigbos PocketBook 6 03-21-2012 01:02 PM
Calibre performance on Mac really bad neonbible Calibre 2 10-30-2010 08:52 AM
DR800 Performance jjansen iRex 18 08-03-2010 09:10 PM
Calibre performance over the network mrwindex Calibre 11 07-30-2009 12:45 AM


All times are GMT -4. The time now is 11:00 PM.


MobileRead.com is a privately owned, operated and funded community.