Calibre performance

[audeojude]

hmm... ok.. so your probably running faster than me other than maybe IO unless you have an ssd your running off. I'm about 4 generations older on the cpu and mother board than you are. That 70 meg file size is huge compared to most of the stuff I convert though. I'm usually .5 meg to 2 meg in size. It might actually work out equivalent if you figure roughly a minute per meg

I just sent 40 books to my kindle and 13 needed to be converted to mobi. average file size of about .5 meg.. conversion and transfer started at 12:32 and ended at 12:43 for a run time of 11 minutes. The job que showed that each conversion took about 20 seconds. I think the send to device job ran parallel to the conversions, jumping back and forth. I will have to keep a closer eye on the details and get a better feel for the true speed over the next few weeks. Maybe do some conversion jobs and document them.

[HarryT]

I only got the new computer two weeks ago . A massive improvement over the dual-core i5 with 4GB RAM I'd been using for the previous five years!

[audeojude]

Yep, you have to be loving life right now.

just did a conversion of a .5 meg book from epub to azw3 (10 sec) same book eput to mobi (16 sec) so the mobi conversion does take a bit longer.

[Katsunami]

Originally Posted by HarryT

I only got the new computer two weeks ago . A massive improvement over the dual-core i5 with 4GB RAM I'd been using for the previous five years!

I can relate Two months ago, I've upgraded from a Q8400 Penryn and 8GB RAM (2008 vintage) to a Dell Precision notebook, which runs an 6820HQ Skylake with 16GB RAM.

The Passmark scores are (from the website):
Q8400: 3227 (100%)
6820HQ: 8619 (267%)

I don't know why my scores are better than the ones on the site. Maybe I have better mainboards with faster interconnects, or better cooling which makes the CPU's throttle less. (I can run my desktop and both of the old and new laptops at full speed indefinitely without throttling, which is often not the case for prebuilt system, especially laptops.)

However, my real scores (running Passmark myself) are:
Q8400: 3561 (100%)
6820HQ 9125 (256%)

Even on the old computer, Calibre was fast enough in general usage, with a library of close to 900 books, with many Delphi Classics in there. The one thing where I do see a very noticable difference in speed is with said Delphi Classics. They convert a lot faster. Also, on the old desktop I could only run three conversions at once (4 CPU's minus 1, so I can still do other stuff), while the new laptop can run 6 conversions ((4 CPU's - 1) * 2 for hyperthreading). Yes, Hyperthreading does gain me the possibility to run twice as many conversions, while also doing them at least 2,5x faster The 6GB of RAM also helps.

HarryT: Have you looked into the Calibre settings? It has a setting to set the number of processes. I often see the worker processes rise way over 12GB of RAM when converting multiple Delphi's at once.

[audeojude]

It had been a while since I had been on the forums so I thought I would put an update on this thread. Library is up to 228,000 items, approx 53,000 directories in library directory, and the database file is 463.6 mb. With the 1tb ssd samsung 940 or 950 i think performance is very very reasonable. I'm still using the same computer with no other changes other than operating system is now ubuntu 16.04

Though I haven't measured performance since earlier in this thread it feels as fast if not faster now than then. I think some of the calibre updates have improved performance a little bit and maybe ubuntu/OS improvements. Not sure about that.

calibre runs like a rock for me. I very seldom have performance or stability issues anymore. Using SSD to me is mandatory for any performance at all given a larger library size.

Hopefully I will be able to upgrade to a newer computer in the next couple years and move to m.2 drives. Move from the 400 to 500 megs a second to 3100 megs a second IO. It will be interesting to see what that does to performance.

[audeojude]

So almost a year later and I can see the handwriting on the wall that I will need to get a new hard drive for my calibre library. I'm down to 130 gigs free space on my 1tb samsung SSD that the library resides on. I can't see going back to a standard HD but I don't think I can afford a larger SSD. Has anyone put their large calibre library on a SSHD drive and have a feeling for comparison between a standard drive, a hybrid SSHD or a standard SSD?



Digression that really doesn't have anything to do with calibre directly, so you can just ignore if you want.

On another note I am still using my same computer that I commented on in an earlier post on this thread, and I am still really happy with it's performance in general. I built the system in march of 2013 so I am right at the 5 year mark with this computer with no upgrades other that putting in a larger SSD for the OS and a larger one for the calibre library over that period of time. (actually I got a larger 1tb drive for calibre and moved the old 512GB ssd calibre was using to the OS Drive.

I do get chip envy once in a while as I have built some nice PC's for a some of my clients with M2 drives and 20 core 3.5 to 4 GHZ i7 cpu's etc.. They are faster and I love them but I have yet to be unhappy with the performance of my 8 year old computer. Really the only thing that would make much difference to me would be more ram, the new USB 3.1 speeds or going to a M2 drive running at 3500 MB/s

Actually I would be willing to bet a current 3000 dollar computer would only give me about 30% performance gain and in reality maybe only a perceived 5 to 10 percent improvement over my 5 year old 1200 dollar computer (only circumstance this wouldn't hold true for is games). Honestly I and my customers have been really happy with long term perceived performance where they went with high end performance on purchase. Most computers 1/2 to 2/3 the cost bought in the same time frame are being replaced due to how slow they now run with all the OS updates (like going to windows 10, or even running windows 7 with all the years of service packs and updates) and other issues. You can tell the High end PC's don't run as snappy as they did 5 years ago but they still run fast enough that none of the users are bothered at all.

[ilovejedd]

I find SSDs make a pretty big difference for Calibre. SSHDs work by caching data that's used repetitively so I don't think it will speed up a lot of the tasks dealing with a ton of small files that Calibre does.

Does your PC not have a spare SATA3 port and space so you can install an extra drive? Or heck, maybe use a USB3 SSD for Calibre. I find the biggest improvement that SSDs bring isn't the 500+ MB/s sequential speed but that their random small block read/write isn't a paltry 1-3MB/s as is the case for most HDDs.

[HarryT]

I'm curious what Calibre operations are disk intensive? The only time-consuming operation I've encountered is conversion, and that seems to be dominated by CPU speed rather than disk speed.

[ilovejedd]

Originally Posted by HarryT

I'm curious what Calibre operations are disk intensive? The only time-consuming operation I've encountered is conversion, and that seems to be dominated by CPU speed rather than disk speed.

Depends on the conversion.

I remember I had one conversion where the max execution time was limited to 10 minutes. On the same computer, the operation kept timing out with VM stored on HDD. Moved the VM to SSD and problem solved (conversion finished in 6 minutes). I've occasionally had to do batch conversions on my entire fanfic library (~3K+ at the time) and the SSD shaves a couple of hours from the conversion time.

Granted, I'm working on organizing and cleaning up a new fanfic library (after just letting downloaded epubs run amok in my Dropbox) and all the library duplication (for testing) and Save to Disk I do would be a major pain on HDD.

At the very least, I'd put metadata.db on SSD.

[HarryT]

What do mean when you say that the max execution time was limited to 10 minutes?

[ilovejedd]

Originally Posted by HarryT

What do mean when you say that the max execution time was limited to 10 minutes?

I was using ebook-convert in a script and it was limited to run for just 10 minutes.

[HarryT]

Oh, OK.

The reason, BTW, I say that Calibre conversions appear to me to be CPU limited is that I always have Windows task manager running in a corner of my screen, and when a conversion is running, the "Calibre worker process" thread is at 25% CPU (ie it's running one of the four CPU cores flat-out), whereas disk usage never seems particularly high. But perhaps it does vary with the type of conversion, as you say. Nearly all the conversions I run are ePub to AZW3.

[ilovejedd]

Originally Posted by HarryT

Oh, OK.

The reason, BTW, I say that Calibre conversions appear to me to be CPU limited is that I always have Windows task manager running in a corner of my screen, and when a conversion is running, the "Calibre worker process" thread is at 25% CPU (ie it's running one of the four CPU cores flat-out), whereas disk usage never seems particularly high. But perhaps it does vary with the type of conversion, as you say. Nearly all the conversions I run are ePub to AZW3.

Mine was xhtml to ePub albeit one with hundreds of chapters. Granted, that was in VM with both the library and /tmp on the same drive. With /tmp on SSD and the Calibre library on HDD, the script probably would've worked fine.

Another thought, my library's not made up of typical books. There's a bunch of drabble fics that are 1K words or less. Should take very little CPU time to convert but small block read/write is something HDDs are slow at.

Right now, I duplicate a 4000 book library maybe 3x a day as I work on adding and cleaning metadata (thank goodness for TXT query on the MCS plugin). That's ~12,000 small files copied every time (epub, txt, jpg, opf).

[HarryT]

Yes, makes sense that conversion of large numbers of small files would be more I/O dependent, I agree.

[audeojude]

Originally Posted by ilovejedd

I find SSDs make a pretty big difference for Calibre. SSHDs work by caching data that's used repetitively so I don't think it will speed up a lot of the tasks dealing with a ton of small files that Calibre does.

Does your PC not have a spare SATA3 port and space so you can install an extra drive? Or heck, maybe use a USB3 SSD for Calibre. I find the biggest improvement that SSDs bring isn't the 500+ MB/s sequential speed but that their random small block read/write isn't a paltry 1-3MB/s as is the case for most HDDs.

Yep plenty of ports. I have been keeping the whole library on only one drive though for ease of use and backup. Have a 1tb ssd right now. Maybe another 4 months before it is full. I could probably delete a bunch of stuff off to give me more time but the amount of effort needed to do that is not going to be fun. Would rather just buy a larger drive. Sadly I don't have the money at this time to pop for a 2tb ssd. So I was wondering about a sshd. I might do it just to find out and measure the differences. Going from a standard drive i saw major reductions in time for searches, as well as adding books, metadata changes in bulk. I hate to give much of that up. Conversions I'm not so worried about as I think as spoken of here that it is more cpu bound and though fast im still only pushing a 2.5 ghz xeon

Sigh just took a moment to look at upgrading my cpu and it isn't worth it.. E5 1265 vs3 2.5 ghz cpu is only about 10% behind performance of a i7 3770 which is the fastest cpu that will go in my mother board that I can tell. Price is to much still to bother for only a 10% cpu gain. im pushing 8000ish cpu marks so can't complain for 5 year old computer. It's funny but It would cost me more now to build the same computer. Ram is higher, used cpu of the model I have is more than I spent on it new. hard drives are pretty much the only thing cheaper than then for the same item.