03-18-2013, 10:50 PM | #1 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
Calibre performance
I thought it would be interesting to report on calibre performance. I just purchased and built a new modern computer with a lot of resources and right now have a good contrast with my old system that was a top end system from about 6 or 7 years ago.
Library size is about 120,000 books, 250 gigs, and a DB file 166 megabytes in size. The directory structure in the calibre library folder has tens of thousands of directories in it which was a huge performance hit. Operating system is Ubuntu Linux 12.04 for both computers. Just moved the same OS drive from one computer to the next. OS drive is a 100 gig ocz 4x PCI-E ssd raid card with 4 25 gig drives on it in a stripped linux software raid configuration. 200 megs a second through put on the old computer 600 megs a second throughput with the new system. Last system was a Intel Core2 Quad Core system running at 2.66 ghz with 8 gigs of ram with calibre living on a 3 terrabyte sata 3 internal drive that was on a intell sata 2 hard drive controller. Opening up calibre took about 2 minutes. A search would take anywhere from a minute to a minute and a half+ This was faster than when I had the library on a older sata 2 hard drive. New system is a Intel i7-3770T 45 watt cpu Running at 2.5 GHZ with 32 gigs of ram on a z77 motherboard with sata 3 controllers. The 3 terabyte drives have been moved over into that system. Opening up calibre now takes about 30 seconds to 45 seconds and a search is anywhere from 15 to 40 seconds. After upgrading the system to 32 gigs of ram calibre is using just over 6 gigs of ram to run. CPU usage seems to max out at about 10 to 12 % averaged over all 8 hyper threaded cores when doing a database check or other intensive search. Several cores hitting 100% for extended periods but not slowing the overall system. My system typically consumes about 15 watts of power idling at the desktop just doing light internet usage. Running a search or database check will pushed the cpu cores to about 10 to 12% and doubled power consumption to about 30 watts. Old system consumed about 130 watts idling. I just found it interesting the difference in performance in calibre between the systems. I think that most of the performance gain is simply IO going from sata 2 to sata 3. I have future plans to get a large 500 gig ssd specd to around 500 megs a second. just for calibre. that should speed up access 3 or 4 fold if I'm right. I'm way at the extreme end for number of files and size of database and calibre just chunks through it. It definitely impacts performance once the library grows past about 20000 books but even up to the 120000 I have it is reliable if a bit slow. Again I think a lot of the slowness is due to simple IO with a huge directory structure. Neither Linux nor windows like large numbers of directory's and files in a root directory. I have looked a multiple file system formats to see if I could use something other than NTFS or EXT4 but they seem to be a bit more robust and get decent average benchmarks even against some of the exotic file systems out there. Just my thoughts on the comparison between the systems. |
03-18-2013, 11:34 PM | #2 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
I don't know if you've seen this thread: https://www.mobileread.com/forums/sho....php?p=2244745
A few ideas on how to speed up calibre up a little. I don't think that the number of files and folders is a problem at all for ext4. Compared to things like mail servers calibre is tame. I have my library on a relatively slow Synology 411j NAS, using NFS, but the database on a local SSD and temp in RAM using tempfs. Most of the things you do in calibre, change metadata, search, tag, download metadata and so on, never touch the actual books, only the database. And if it does, most of the work/updates are done in the temp folder, not in the library folders. Usually the normal disk caches are enough to give a significant speed boost. But it is fun to tinker some more... Save a few thousand books to disk, and change the format on the fly and/or update metadata using a plugboard. That is a nice stress test... ;D Last edited by Adoby; 03-18-2013 at 11:54 PM. |
Advert | |
|
03-19-2013, 01:12 AM | #3 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
I just spent the last couple hours playing with it and am puzzled. I moved the metadata.db file and created a soft link it on the the pci-e flash drive that has 600 mgs a sec through put and didn't see much performance increase over the 140 meg a second sata drive in searching or doing a search.
Moved the file to a ram drive tmpfs under /tmp on ubuntu and the speed of starting calibre was very contemporary to both the hard drive and the ssd pci-e card, same for the search. So..... im confused. the ram drive is throughput can be measured in gigs a second vs 100 to 100's of megs a second but im not seeing much speed improvement at all. |
03-19-2013, 02:22 AM | #4 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
The disk cache is very efficient. The whole database is cached in RAM. And the books are not accessed at all during searches and startups.
Do some tests where you change and convert ebooks. Perhaps that will be different? |
03-19-2013, 10:35 AM | #5 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
Yep it is efficient. I was testing it by moving 2.2 gig files to it off the PCI-E ssd card and that were from a human perspective copying instantaneously. So well in excess of 2 gigs of throughput, though I don't see how the PCI-E card was feeding the file there that fast.
really bizare I went to play with it again this morning and tweaked some settings re the tmpfs and now the system won't mount /tmp on tmpfs.. Anyway.. I will have to come back to it at a later date. Work calls. |
Advert | |
|
04-06-2016, 04:40 PM | #6 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
Just a note on performance currently.
4 generation old i7 server cpu running at .5 ghz 32 gigs ram and now a 1 TB samsung ssd library size of 220000 files standard setup on calibre no special configuration. I played with special configurations but ended up removing them. searches take 6 seconds initial startup about 30 to 40 seconds import is pretty fast. 1000 books might take 10 to 20 minutes or so. Book conversion is so so with wildly varying times.... 20 seconds to more than a minute per document sometimes. calibre uses about 2 gigs of memory when running. the database metadata.db is 350 megs in size right now. in day to day usage I am very happy. It has been extremely stable and I can't remember it crashing. Only issue I have is new one within the last 4 updates or so and it is more annoying than broken. if you right click on a title and then go to edit metadata the menu flickers and you can not left click on it to select an option. however you can move the mouse over the options and they will highlight blue and you can use the enter key on the keyboard to select them. all other menus other than the edit metadata one that show when a title is right clicked on work fine with the mouse. It's just weird |
04-06-2016, 05:23 PM | #7 | |
null operator
Posts: 20,414
Karma: 25894272
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Quote:
BTW conversion times vary without obvious reason (but well understood by Kovid) - they've very rarely been known to take > 24 hours for one book - and quite often up to a hour. Its not a strictly linear process - it proceeds in steps. BR |
|
04-07-2016, 04:54 AM | #8 |
eBook Enthusiast
Posts: 85,544
Karma: 93383043
Join Date: Nov 2006
Location: UK
Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6
|
It also depends on the format. I've found that conversion from ePub to "old" Mobi regularly takes up to 3x longer than conversion from ePub to AZW3 (I have some large books that have colour illustrations that I want to view on the iPad Kindle App, which only supports old Mobi).
|
04-07-2016, 10:29 AM | #9 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
OS is Ubuntu Linux 15.10
the menu issue isn't a mouse or hard to use with mouse issue. That specific edit metadata menu acts differently than all other menus all of a sudden (last few versions calibre) it flickers in and out as if you were running the mouse back and forth over it with the mouse just sitting still on it. also left click doesn't work to select option on submenu items. They highlight like they are selected but clicking doesn't do anything. once highlighted you can hit the enter key on keyboard and it will run that menu item. all other menu items other than that one work fine. You can use mouse to select and run them, no flickering etc.. on the conversion speed. I didn't know for fact that depending on what format to what format there was a difference in conversion speed, however it is logical. Mostly all I do is epub to mobi or azw3 as I use a kindle reader. I will have to start using azw3 as my default if that is that much faster. Over all it is not an issue. Nothing like the post of hours. probably the worst I have seen is 4 or 5 minutes and normal is under a minute. Usually if I do a batch of 10 or so it might take 3 or 4 minutes. I had no idea some were having really long conversion times like described. I have never seen that even when doing from or to pdf which seems the longest to me. I just had noticed a difference of some taking 30 seconds to a minute at the fast end and 2 or 3 minutes on the long end Wasn't a complaint at all just an observation that I didn't understand. |
04-07-2016, 10:31 AM | #10 |
creator of calibre
Posts: 43,739
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
|
04-07-2016, 10:32 AM | #11 | |
eBook Enthusiast
Posts: 85,544
Karma: 93383043
Join Date: Nov 2006
Location: UK
Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6
|
Quote:
|
|
04-07-2016, 10:46 AM | #12 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
Just a thought for those with performance issues. Hardware really makes a difference. I know it is expensive to get a higher end system. I typically spend about 1000 to 1200 on a computer. i7, as much ram as the motherboard will hold 32 gigs in this case, and ssd drives for OS and stuff like calibre that has io constraints.
The upside is that i only build computers every 5 years or so. I find that I can go that or longer with a top end computer and be exceedingly happy with the performance for all my computing needs. However I don't run games so just use the built in Intell HD video. My systems would make serious game machines other than the video. Adding video would add another 400 dollars or so for something decent. So take my price range with a grain of salt if you play games. I find my life is a lot happier with a fast computer in that I don't deal with wait for it itis and frustration in using it. When I start wanting to punch my computer based on performance issues I know it is getting to be time to find or build a new one. Over the last two computers I have had it has been IO that has caused me to get frustrated. I think IO lags massively behind cpu and bus performance. So expensive computer that I have trouble affording sometimes but if I look at it stretched out over 5 years it is only 200 a year or so to make me have a higher quality of life. With my work I spend about 5 hours a day using my computer so it makes a lot of difference to me. Not trying to preach or anything but just observation on my personal perception and life as well as a couple clients I talked into using higher end computers for their employees in their offices. It made a huge productivity difference in their specific situations. If all your doing is browsing the Internet and using email or a word processor then any 500 dollar computer should make you happy. Any graphics, rendering, large database search or manipulation, or huge file transfers, or excessive tabs in a browser will be much happier with a faster larger ram computer. SSD for OS drive will double the perceived performance of any computer. My background is 20 years of owning and running a small computer it/service company. Providing Network and Server administration services, PC repair, Windows, Mac and Linux support. |
04-07-2016, 10:49 AM | #13 | |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
Quote:
thanks for the link. I will just live with it till they get it fixed on their end. it is simply annoying, it doesn't break functionality. Just makes me use the keyboard for those few functions. |
|
04-07-2016, 11:07 AM | #14 | |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
Quote:
What is the system specs for your computer that it is taking that long to do a conversion and what format to what format? how many pages? It would be interesting to get a feel for the impact of hardware and memory on calibre's performance. I guess it would be combination of how many files on the file system (OS files system speed issue), memory and architecture (32 or 64), type of hard drive (IO), type of format to format conversion issue, etc |
|
04-07-2016, 11:48 AM | #15 | |
eBook Enthusiast
Posts: 85,544
Karma: 93383043
Join Date: Nov 2006
Location: UK
Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6
|
Quote:
One of the recent long conversions I've done was the Delphi "Complete Works of Edgar Wallace". 70MB ePub which took about an hour and a half to convert to AZW3. Last edited by HarryT; 04-07-2016 at 11:56 AM. |
|
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
calibre performance settings? | unboggling | Calibre | 2 | 08-05-2012 09:08 PM |
touchscreen performance | sebigbos | PocketBook | 6 | 03-21-2012 01:02 PM |
Calibre performance on Mac really bad | neonbible | Calibre | 2 | 10-30-2010 08:52 AM |
DR800 Performance | jjansen | iRex | 18 | 08-03-2010 09:10 PM |
Calibre performance over the network | mrwindex | Calibre | 11 | 07-30-2009 12:45 AM |