Calibre performance

audeojude · 03-18-2013, 10:50 PM

I thought it would be interesting to report on calibre performance. I just purchased and built a new modern computer with a lot of resources and right now have a good contrast with my old system that was a top end system from about 6 or 7 years ago.

Library size is about 120,000 books, 250 gigs, and a DB file 166 megabytes in size. The directory structure in the calibre library folder has tens of thousands of directories in it which was a huge performance hit.

Operating system is Ubuntu Linux 12.04 for both computers. Just moved the same OS drive from one computer to the next. OS drive is a 100 gig ocz 4x PCI-E ssd raid card with 4 25 gig drives on it in a stripped linux software raid configuration. 200 megs a second through put on the old computer 600 megs a second throughput with the new system.

Last system was a Intel Core2 Quad Core system running at 2.66 ghz with 8 gigs of ram with calibre living on a 3 terrabyte sata 3 internal drive that was on a intell sata 2 hard drive controller.

Opening up calibre took about 2 minutes. A search would take anywhere from a minute to a minute and a half+ This was faster than when I had the library on a older sata 2 hard drive.

New system is a Intel i7-3770T 45 watt cpu Running at 2.5 GHZ with 32 gigs of ram on a z77 motherboard with sata 3 controllers. The 3 terabyte drives have been moved over into that system.

Opening up calibre now takes about 30 seconds to 45 seconds and a search is anywhere from 15 to 40 seconds. After upgrading the system to 32 gigs of ram calibre is using just over 6 gigs of ram to run. CPU usage seems to max out at about 10 to 12 % averaged over all 8 hyper threaded cores when doing a database check or other intensive search. Several cores hitting 100% for extended periods but not slowing the overall system.

My system typically consumes about 15 watts of power idling at the desktop just doing light internet usage. Running a search or database check will pushed the cpu cores to about 10 to 12% and doubled power consumption to about 30 watts. Old system consumed about 130 watts idling.

I just found it interesting the difference in performance in calibre between the systems. I think that most of the performance gain is simply IO going from sata 2 to sata 3. I have future plans to get a large 500 gig ssd specd to around 500 megs a second. just for calibre. that should speed up access 3 or 4 fold if I'm right.

I'm way at the extreme end for number of files and size of database and calibre just chunks through it. It definitely impacts performance once the library grows past about 20000 books but even up to the 120000 I have it is reliable if a bit slow. Again I think a lot of the slowness is due to simple IO with a huge directory structure. Neither Linux nor windows like large numbers of directory's and files in a root directory. I have looked a multiple file system formats to see if I could use something other than NTFS or EXT4 but they seem to be a bit more robust and get decent average benchmarks even against some of the exotic file systems out there.

Just my thoughts on the comparison between the systems.

Adoby · 03-18-2013, 11:34 PM

I don't know if you've seen this thread: https://www.mobileread.com/forums/sho....php?p=2244745

A few ideas on how to speed up calibre up a little.

I don't think that the number of files and folders is a problem at all for ext4. Compared to things like mail servers calibre is tame.

I have my library on a relatively slow Synology 411j NAS, using NFS, but the database on a local SSD and temp in RAM using tempfs. Most of the things you do in calibre, change metadata, search, tag, download metadata and so on, never touch the actual books, only the database. And if it does, most of the work/updates are done in the temp folder, not in the library folders.

Usually the normal disk caches are enough to give a significant speed boost. But it is fun to tinker some more...

Save a few thousand books to disk, and change the format on the fly and/or update metadata using a plugboard. That is a nice stress test... ;D

audeojude · 03-19-2013, 01:12 AM

I just spent the last couple hours playing with it and am puzzled. I moved the metadata.db file and created a soft link it on the the pci-e flash drive that has 600 mgs a sec through put and didn't see much performance increase over the 140 meg a second sata drive in searching or doing a search.

Moved the file to a ram drive tmpfs under /tmp on ubuntu and the speed of starting calibre was very contemporary to both the hard drive and the ssd pci-e card, same for the search. So..... im confused. the ram drive is throughput can be measured in gigs a second vs 100 to 100's of megs a second but im not seeing much speed improvement at all.

Adoby · 03-19-2013, 02:22 AM

The disk cache is very efficient. The whole database is cached in RAM. And the books are not accessed at all during searches and startups.

Do some tests where you change and convert ebooks. Perhaps that will be different?

audeojude · 03-19-2013, 10:35 AM

Yep it is efficient. I was testing it by moving 2.2 gig files to it off the PCI-E ssd card and that were from a human perspective copying instantaneously. So well in excess of 2 gigs of throughput, though I don't see how the PCI-E card was feeding the file there that fast.

really bizare I went to play with it again this morning and tweaked some settings re the tmpfs and now the system won't mount /tmp on tmpfs..

Anyway.. I will have to come back to it at a later date. Work calls.

audeojude · 04-06-2016, 04:40 PM

Just a note on performance currently.
4 generation old i7 server cpu running at .5 ghz
32 gigs ram
and now a 1 TB samsung ssd
library size of 220000 files
standard setup on calibre no special configuration. I played with special configurations but ended up removing them.

searches take 6 seconds

initial startup about 30 to 40 seconds

import is pretty fast. 1000 books might take 10 to 20 minutes or so.
Book conversion is so so with wildly varying times.... 20 seconds to more than a minute per document sometimes.
calibre uses about 2 gigs of memory when running.

the database metadata.db is 350 megs in size right now.

in day to day usage I am very happy. It has been extremely stable and I can't remember it crashing.

Only issue I have is new one within the last 4 updates or so and it is more annoying than broken. if you right click on a title and then go to edit metadata the menu flickers and you can not left click on it to select an option. however you can move the mouse over the options and they will highlight blue and you can use the enter key on the keyboard to select them. all other menus other than the edit metadata one that show when a title is right clicked on work fine with the mouse. It's just weird

BetterRed · 04-06-2016, 05:23 PM

Quote:

Originally Posted by audeojude

Only issue I have is new one within the last 4 updates or so and it is more annoying than broken. if you right click on a title and then go to edit metadata the menu flickers and you can not left click on it to select an option. however you can move the mouse over the options and they will highlight blue and you can use the enter key on the keyboard to select them. all other menus other than the edit metadata one that show when a title is right clicked on work fine with the mouse. It's just weird

What is your workstation OS ? It's OK here on Windows with mouse - but IMO its easier with the keyboard, given the close proximity of menu, arrow and enter keys.

BTW conversion times vary without obvious reason (but well understood by Kovid) - they've very rarely been known to take > 24 hours for one book - and quite often up to a hour. Its not a strictly linear process - it proceeds in steps.

BR

HarryT · 04-07-2016, 04:54 AM

Quote:

Originally Posted by BetterRed

BTW conversion times vary without obvious reason (but well understood by Kovid) - they've very rarely been known to take > 24 hours for one book - and quite often up to a hour. Its not a strictly linear process - it proceeds in steps.

It also depends on the format. I've found that conversion from ePub to "old" Mobi regularly takes up to 3x longer than conversion from ePub to AZW3 (I have some large books that have colour illustrations that I want to view on the iPad Kindle App, which only supports old Mobi).

audeojude · 04-07-2016, 10:29 AM

OS is Ubuntu Linux 15.10

the menu issue isn't a mouse or hard to use with mouse issue.
That specific edit metadata menu acts differently than all other menus all of a sudden (last few versions calibre) it flickers in and out as if you were running the mouse back and forth over it with the mouse just sitting still on it. also left click doesn't work to select option on submenu items. They highlight like they are selected but clicking doesn't do anything. once highlighted you can hit the enter key on keyboard and it will run that menu item. all other menu items other than that one work fine. You can use mouse to select and run them, no flickering etc..

on the conversion speed. I didn't know for fact that depending on what format to what format there was a difference in conversion speed, however it is logical. Mostly all I do is epub to mobi or azw3 as I use a kindle reader. I will have to start using azw3 as my default if that is that much faster. Over all it is not an issue. Nothing like the post of hours.

probably the worst I have seen is 4 or 5 minutes and normal is under a minute. Usually if I do a batch of 10 or so it might take 3 or 4 minutes. I had no idea some were having really long conversion times like described. I have never seen that even when doing from or to pdf which seems the longest to me. I just had noticed a difference of some taking 30 seconds to a minute at the fast end and 2 or 3 minutes on the long end

Wasn't a complaint at all just an observation that I didn't understand.

kovidgoyal · 04-07-2016, 10:31 AM

https://bugs.launchpad.net/calibre/+bug/1534936

HarryT · 04-07-2016, 10:32 AM

Quote:

Originally Posted by audeojude

on the conversion speed. I didn't know for fact that depending on what format to what format there was a difference in conversion speed, however it is logical. Mostly all I do is epub to mobi or azw3 as I use a kindle reader. I will have to start using azw3 as my default if that is that much faster. Over all it is not an issue. Nothing like the post of hours.

probably the worst I have seen is 4 or 5 minutes and normal is under a minute. Usually if I do a batch of 10 or so it might take 3 or 4 minutes. I had no idea some were having really long conversion times like described. I have never seen that even when doing from or to pdf which seems the longest to me.

I buy a lot of the omnibus editions from Delphi Classics, which can contain dozens of individual books. Conversion times of several hours aren't at all uncommon for such books. Some of them can only be converted on 64-bit versions of Calibre; the 32-bit version runs out of memory.

audeojude · 04-07-2016, 10:46 AM

Just a thought for those with performance issues. Hardware really makes a difference. I know it is expensive to get a higher end system. I typically spend about 1000 to 1200 on a computer. i7, as much ram as the motherboard will hold 32 gigs in this case, and ssd drives for OS and stuff like calibre that has io constraints.

The upside is that i only build computers every 5 years or so. I find that I can go that or longer with a top end computer and be exceedingly happy with the performance for all my computing needs.

However I don't run games so just use the built in Intell HD video. My systems would make serious game machines other than the video. Adding video would add another 400 dollars or so for something decent. So take my price range with a grain of salt if you play games.

I find my life is a lot happier with a fast computer in that I don't deal with wait for it itis and frustration in using it. When I start wanting to punch my computer based on performance issues I know it is getting to be time to find or build a new one. Over the last two computers I have had it has been IO that has caused me to get frustrated. I think IO lags massively behind cpu and bus performance.

So expensive computer that I have trouble affording sometimes but if I look at it stretched out over 5 years it is only 200 a year or so to make me have a higher quality of life. With my work I spend about 5 hours a day using my computer so it makes a lot of difference to me.

Not trying to preach or anything but just observation on my personal perception and life as well as a couple clients I talked into using higher end computers for their employees in their offices. It made a huge productivity difference in their specific situations. If all your doing is browsing the Internet and using email or a word processor then any 500 dollar computer should make you happy. Any graphics, rendering, large database search or manipulation, or huge file transfers, or excessive tabs in a browser will be much happier with a faster larger ram computer. SSD for OS drive will double the perceived performance of any computer.

My background is 20 years of owning and running a small computer it/service company. Providing Network and Server administration services, PC repair, Windows, Mac and Linux support.

audeojude · 04-07-2016, 10:49 AM

Quote:

Originally Posted by kovidgoyal

https://bugs.launchpad.net/calibre/+bug/1534936

Exactly it!!!!
thanks for the link. I will just live with it till they get it fixed on their end. it is simply annoying, it doesn't break functionality. Just makes me use the keyboard for those few functions.

audeojude · 04-07-2016, 11:07 AM

Quote:

Originally Posted by HarryT

I buy a lot of the omnibus editions from Delphi Classics, which can contain dozens of individual books. Conversion times of several hours aren't at all uncommon for such books. Some of them can only be converted on 64-bit versions of Calibre; the 32-bit version runs out of memory.

I haven't used a 32 OS in years. I couldn't do 1/4 of what I daily do on my pc with a 4 gig memory limitation. Shoot I couldn't even do it with 8 gigs. I'm not doing anything special right this moment and have 9 gigs tied up between the OS, calibre and firefox. When I start cranking virtual machines for work up it will tie up all 32 gigs once in a while. 15 to 20 gigs is about normal for me.

What is the system specs for your computer that it is taking that long to do a conversion and what format to what format? how many pages? It would be interesting to get a feel for the impact of hardware and memory on calibre's performance. I guess it would be combination of how many files on the file system (OS files system speed issue), memory and architecture (32 or 64), type of hard drive (IO), type of format to format conversion issue, etc

HarryT · 04-07-2016, 11:48 AM

Quote:

Originally Posted by audeojude

What is the system specs for your computer that it is taking that long to do a conversion and what format to what format? how many pages? It would be interesting to get a feel for the impact of hardware and memory on calibre's performance. I guess it would be combination of how many files on the file system (OS files system speed issue), memory and architecture (32 or 64), type of hard drive (IO), type of format to format conversion issue, etc

Latest gen (Skylake) quad core i7 with 16GB RAM running 64-bit Windows 10, converting ePub to AZW3. I tend to buy all the Delphi books as ePub, and then convert them to AZW3 for my Kindle, because they're easier to edit as ePubs. The Calibre conversion worker processes get up to about 6GB RAM during some conversions (I've never seen them go any higher than that). Conversion times seem to depend a lot on the complexity of the source rather than just its length: I've had books that are double the length of others convert in less time.

One of the recent long conversions I've done was the Delphi "Complete Works of Edgar Wallace". 70MB ePub which took about an hour and a half to convert to AZW3.

03-18-2013, 10:50 PM	#1
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	Calibre performance I thought it would be interesting to report on calibre performance. I just purchased and built a new modern computer with a lot of resources and right now have a good contrast with my old system that was a top end system from about 6 or 7 years ago. Library size is about 120,000 books, 250 gigs, and a DB file 166 megabytes in size. The directory structure in the calibre library folder has tens of thousands of directories in it which was a huge performance hit. Operating system is Ubuntu Linux 12.04 for both computers. Just moved the same OS drive from one computer to the next. OS drive is a 100 gig ocz 4x PCI-E ssd raid card with 4 25 gig drives on it in a stripped linux software raid configuration. 200 megs a second through put on the old computer 600 megs a second throughput with the new system. Last system was a Intel Core2 Quad Core system running at 2.66 ghz with 8 gigs of ram with calibre living on a 3 terrabyte sata 3 internal drive that was on a intell sata 2 hard drive controller. Opening up calibre took about 2 minutes. A search would take anywhere from a minute to a minute and a half+ This was faster than when I had the library on a older sata 2 hard drive. New system is a Intel i7-3770T 45 watt cpu Running at 2.5 GHZ with 32 gigs of ram on a z77 motherboard with sata 3 controllers. The 3 terabyte drives have been moved over into that system. Opening up calibre now takes about 30 seconds to 45 seconds and a search is anywhere from 15 to 40 seconds. After upgrading the system to 32 gigs of ram calibre is using just over 6 gigs of ram to run. CPU usage seems to max out at about 10 to 12 % averaged over all 8 hyper threaded cores when doing a database check or other intensive search. Several cores hitting 100% for extended periods but not slowing the overall system. My system typically consumes about 15 watts of power idling at the desktop just doing light internet usage. Running a search or database check will pushed the cpu cores to about 10 to 12% and doubled power consumption to about 30 watts. Old system consumed about 130 watts idling. I just found it interesting the difference in performance in calibre between the systems. I think that most of the performance gain is simply IO going from sata 2 to sata 3. I have future plans to get a large 500 gig ssd specd to around 500 megs a second. just for calibre. that should speed up access 3 or 4 fold if I'm right. I'm way at the extreme end for number of files and size of database and calibre just chunks through it. It definitely impacts performance once the library grows past about 20000 books but even up to the 120000 I have it is reliable if a bit slow. Again I think a lot of the slowness is due to simple IO with a huge directory structure. Neither Linux nor windows like large numbers of directory's and files in a root directory. I have looked a multiple file system formats to see if I could use something other than NTFS or EXT4 but they seem to be a bit more robust and get decent average benchmarks even against some of the exotic file systems out there. Just my thoughts on the comparison between the systems.

03-18-2013, 11:34 PM	#2
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	I don't know if you've seen this thread: https://www.mobileread.com/forums/sho....php?p=2244745 A few ideas on how to speed up calibre up a little. I don't think that the number of files and folders is a problem at all for ext4. Compared to things like mail servers calibre is tame. I have my library on a relatively slow Synology 411j NAS, using NFS, but the database on a local SSD and temp in RAM using tempfs. Most of the things you do in calibre, change metadata, search, tag, download metadata and so on, never touch the actual books, only the database. And if it does, most of the work/updates are done in the temp folder, not in the library folders. Usually the normal disk caches are enough to give a significant speed boost. But it is fun to tinker some more... Save a few thousand books to disk, and change the format on the fly and/or update metadata using a plugboard. That is a nice stress test... ;D Last edited by Adoby; 03-18-2013 at 11:54 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
calibre performance settings?	unboggling	Calibre	2	08-05-2012 09:08 PM
touchscreen performance	sebigbos	PocketBook	6	03-21-2012 01:02 PM
Calibre performance on Mac really bad	neonbible	Calibre	2	10-30-2010 08:52 AM
DR800 Performance	jjansen	iRex	18	08-03-2010 09:10 PM
Calibre performance over the network	mrwindex	Calibre	11	07-30-2009 12:45 AM

03-19-2013, 01:12 AM	#3
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	I just spent the last couple hours playing with it and am puzzled. I moved the metadata.db file and created a soft link it on the the pci-e flash drive that has 600 mgs a sec through put and didn't see much performance increase over the 140 meg a second sata drive in searching or doing a search. Moved the file to a ram drive tmpfs under /tmp on ubuntu and the speed of starting calibre was very contemporary to both the hard drive and the ssd pci-e card, same for the search. So..... im confused. the ram drive is throughput can be measured in gigs a second vs 100 to 100's of megs a second but im not seeing much speed improvement at all.

03-19-2013, 02:22 AM	#4
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	The disk cache is very efficient. The whole database is cached in RAM. And the books are not accessed at all during searches and startups. Do some tests where you change and convert ebooks. Perhaps that will be different?

03-19-2013, 10:35 AM	#5
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	Yep it is efficient. I was testing it by moving 2.2 gig files to it off the PCI-E ssd card and that were from a human perspective copying instantaneously. So well in excess of 2 gigs of throughput, though I don't see how the PCI-E card was feeding the file there that fast. really bizare I went to play with it again this morning and tweaked some settings re the tmpfs and now the system won't mount /tmp on tmpfs.. Anyway.. I will have to come back to it at a later date. Work calls.

04-06-2016, 04:40 PM	#6
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	Just a note on performance currently. 4 generation old i7 server cpu running at .5 ghz 32 gigs ram and now a 1 TB samsung ssd library size of 220000 files standard setup on calibre no special configuration. I played with special configurations but ended up removing them. searches take 6 seconds initial startup about 30 to 40 seconds import is pretty fast. 1000 books might take 10 to 20 minutes or so. Book conversion is so so with wildly varying times.... 20 seconds to more than a minute per document sometimes. calibre uses about 2 gigs of memory when running. the database metadata.db is 350 megs in size right now. in day to day usage I am very happy. It has been extremely stable and I can't remember it crashing. Only issue I have is new one within the last 4 updates or so and it is more annoying than broken. if you right click on a title and then go to edit metadata the menu flickers and you can not left click on it to select an option. however you can move the mouse over the options and they will highlight blue and you can use the enter key on the keyboard to select them. all other menus other than the edit metadata one that show when a title is right clicked on work fine with the mouse. It's just weird

04-07-2016, 10:29 AM	#9
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	OS is Ubuntu Linux 15.10 the menu issue isn't a mouse or hard to use with mouse issue. That specific edit metadata menu acts differently than all other menus all of a sudden (last few versions calibre) it flickers in and out as if you were running the mouse back and forth over it with the mouse just sitting still on it. also left click doesn't work to select option on submenu items. They highlight like they are selected but clicking doesn't do anything. once highlighted you can hit the enter key on keyboard and it will run that menu item. all other menu items other than that one work fine. You can use mouse to select and run them, no flickering etc.. on the conversion speed. I didn't know for fact that depending on what format to what format there was a difference in conversion speed, however it is logical. Mostly all I do is epub to mobi or azw3 as I use a kindle reader. I will have to start using azw3 as my default if that is that much faster. Over all it is not an issue. Nothing like the post of hours. probably the worst I have seen is 4 or 5 minutes and normal is under a minute. Usually if I do a batch of 10 or so it might take 3 or 4 minutes. I had no idea some were having really long conversion times like described. I have never seen that even when doing from or to pdf which seems the longest to me. I just had noticed a difference of some taking 30 seconds to a minute at the fast end and 2 or 3 minutes on the long end Wasn't a complaint at all just an observation that I didn't understand.

04-07-2016, 10:31 AM	#10
kovidgoyal creator of calibre Posts: 43,855 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	https://bugs.launchpad.net/calibre/+bug/1534936

04-07-2016, 10:46 AM	#12
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	Just a thought for those with performance issues. Hardware really makes a difference. I know it is expensive to get a higher end system. I typically spend about 1000 to 1200 on a computer. i7, as much ram as the motherboard will hold 32 gigs in this case, and ssd drives for OS and stuff like calibre that has io constraints. The upside is that i only build computers every 5 years or so. I find that I can go that or longer with a top end computer and be exceedingly happy with the performance for all my computing needs. However I don't run games so just use the built in Intell HD video. My systems would make serious game machines other than the video. Adding video would add another 400 dollars or so for something decent. So take my price range with a grain of salt if you play games. I find my life is a lot happier with a fast computer in that I don't deal with wait for it itis and frustration in using it. When I start wanting to punch my computer based on performance issues I know it is getting to be time to find or build a new one. Over the last two computers I have had it has been IO that has caused me to get frustrated. I think IO lags massively behind cpu and bus performance. So expensive computer that I have trouble affording sometimes but if I look at it stretched out over 5 years it is only 200 a year or so to make me have a higher quality of life. With my work I spend about 5 hours a day using my computer so it makes a lot of difference to me. Not trying to preach or anything but just observation on my personal perception and life as well as a couple clients I talked into using higher end computers for their employees in their offices. It made a huge productivity difference in their specific situations. If all your doing is browsing the Internet and using email or a word processor then any 500 dollar computer should make you happy. Any graphics, rendering, large database search or manipulation, or huge file transfers, or excessive tabs in a browser will be much happier with a faster larger ram computer. SSD for OS drive will double the perceived performance of any computer. My background is 20 years of owning and running a small computer it/service company. Providing Network and Server administration services, PC repair, Windows, Mac and Linux support.

Advert

Advert