Calibre performance

[ilovejedd]

Originally Posted by audeojude

Yep plenty of ports. I have been keeping the whole library on only one drive though for ease of use and backup. Have a 1tb ssd right now. Maybe another 4 months before it is full. I could probably delete a bunch of stuff off to give me more time but the amount of effort needed to do that is not going to be fun. Would rather just buy a larger drive. Sadly I don't have the money at this time to pop for a 2tb ssd. So I was wondering about a sshd. I might do it just to find out and measure the differences. Going from a standard drive i saw major reductions in time for searches, as well as adding books, metadata changes in bulk. I hate to give much of that up. Conversions I'm not so worried about as I think as spoken of here that it is more cpu bound and though fast im still only pushing a 2.5 ghz xeon

Sigh just took a moment to look at upgrading my cpu and it isn't worth it.. E5 1265 vs3 2.5 ghz cpu is only about 10% behind performance of a i7 3770 which is the fastest cpu that will go in my mother board that I can tell. Price is to much still to bother for only a 10% cpu gain. im pushing 8000ish cpu marks so can't complain for 5 year old computer. It's funny but It would cost me more now to build the same computer. Ram is higher, used cpu of the model I have is more than I spent on it new. hard drives are pretty much the only thing cheaper than then for the same item.

Hmm, how about RAID-0 with another 1TB SSD (albeit be extra vigilant with backup and replication)?

[audeojude]

Originally Posted by ilovejedd

Hmm, how about RAID-0 with another 1TB SSD (albeit be extra vigilant with backup and replication)?

That is an excellent point... I might end up doing that but in general I really don't like raid 0 arrays at all. I actually have a fast pcie raid 0 on my system across 4 25gig chips running at almost 1000megs a second I use for video editing and conversions etc.. but I never leave anything on it. You have doubled your chance of data loss.

My current backup strategy is pretty lame given that I am an IT professional. I simply rsync my data manually every week or two to external hard drives stored in a safe.

I have been looking at a network backup using Urbackup. I install and support it for clients. Absolutely stellar open source backup solution. but right now my redundancy isn't where it should be much less using a raid 0.

I keep cruising ebay for a used data server or NAS I can afford with 15 to 24 TB of redundant space on it. lol. maybe i will find that deal soon.

Performance and simplicity come at a price and right now im having problems affording that. It's totally doable if I go to a much more complicated solution with less performance. That's kinda what I'm doing right now with most of it and I'm really tired of it.

edited march 2018 to change raid 1 to raid 0.. Everywhere I had said raid 1 I meant raid 0
also though not for my books I had both a backup drive and then the main drive the data was used from fail with full loss of all data the other day... I could have fixed it but was trying to buy a larger hard drive rather than just replace the smaller one that died. I put it off because of lack of money. Just the time invested in the lost data was priceless. Don't do as I did.

[ilovejedd]

Yeah, I wouldn't trust RAID-0 to be sole repository of data even for just a day.

My backup's pretty crude, too. Dropbox is pretty much my only offsite backup and is installed/replicated on two laptops and two desktops. Locally, scheduled nightly FreeFileSync with versioning to external HDD and NAS.

[audeojude]

So.. been a while and thought a update is in order as the performance thread to me has been interesting. Now on almost 10 years since first post on this thread.

Since my last update I had put an Intel optane 2tb nvme (1000 Mbs) in the computer in the old computer that has been referenced before. I am still using that system today. I put that drive in on an add on card about the time of my last post. And used it for a long time. though I don't have any hard data as I was happy and didn't think about it. it worked well and reasonably fast. I did want one of the 3500MBS nvme's but couldn't afford it. I got the one I did back there for about 1 dollar a TB.

However I have a new system gifted to me that is a 2nd gen ryzen 7 that a gamer friend retired and have moved my calibre library from the old system. When doing the move about 3 months ago I put in a 4tb crucial 3400 mbs NVME in the system as I had almost filled the prior 2TB drive.

I am running
Ryzen 7 2700x
AMD Vega 64 video card
32 Gigs Ram
Both cpu and video liquid cooled.
OS is Ubuntu 22.04
OS is on a 1TB samsung NVME on motherboard
Calibre drive is a Crucial 4TB drive in a Highpoint 4 nvme enclosure in a PCIE slot. Highpoint is old and im not running the drivers for it so am taking a bit of a performance hit using it.

here is link to drive. I paid just under 400 dollars 3 months ago.. they have it for 249 dollars right now.

https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1

NVMe (PCIe Gen3 x4) technology with up to 3500MB/s sequential reads, random read/write 650K/700K IOPS
Spacious storage up to 4TB
Performs up to 33% better than the previous generation
Solid Gen3 performance
Rated at MTTF greater than 1.5 million hours for extended longevity and reliability

I would prefer latest gen 6500 mbs drive but cant afford that.

ok now that that is all said.

I have 315,000 books in calibre
the database metadata.db is 638.3MB
There are 71,446 folders in the root of the calibre files directory
It takes 18 seconds for calibre to start with that library selected.
with an empty library it takes 3 seconds
a search now takes 2 seconds on average



Linux does not like indexing and viewing the directory that the calibre books are in.
There are 71,446 folders in the root of the calibre files directory
there are 1,330,796 items (1.1 TB in size) in the directory and just to get a properties on it that show that takes about 20 minutes.
There is another directory for comics on drive that calibre isn't handling. with about 600 gigs in it.
Drive shows 2.1 TB free
The is a disparity between what shows in calibre