How many books in a library?

[BetterRed]

My media library is 853.20 GB, 8,768 titles, 26,853 files ('format' files, covers, and opfs), the database 7,367,680 bytes. That's ~100MB per title.

It lives on a 4TB WD Black spinning platter drive. I moved the database to an empty 256GB SATA SSD drive, made no discernible difference to library performance, so I moved it to my 512GB PCIe M.2 system SSD, start times went from ~8 secs to ~4 secs, otherwise no discernible difference.

I figure the most I would gain from keeping my library databases on a high end SSD would be a minute or two a day. Needless to say I won't be bothering.

BR

[mbovenka]

Originally Posted by ilovejedd

Curious, large PDF manuals?

My metadata.db is 30MB for a 2.5GB library (if I keep just EPUB format, then library is only ~500MB).

No, that's only 1.5MB/book, roughly, 80% EPUB. My metadata.db is about 600MB. So assuming the DB size goes up linearly with the number of books (and the average book size stays the same) that would make a 6TB library 3.5 million books and a 7.2GB DB.

So I rounded up to 7.5 GB.

[ilovejedd]

Originally Posted by mbovenka

No, that's only 1.5MB/book, roughly, 80% EPUB. My metadata.db is about 600MB. So assuming the DB size goes up linearly with the number of books (and the average book size stays the same) that would make a 6TB library 3.5 million books and a 7.2GB DB.

So I rounded up to 7.5 GB.

Ah, I had thought you meant 300K as in size of metadata.db. Didn't realize you meant 300,000 books. Lol, it was 1AM local when I posted. Synapses weren't firing up properly.

[ilovejedd]

Originally Posted by BetterRed

My media library is 853.20 GB, 8,768 titles, 26,853 files ('format' files, covers, and opfs), the database 7,367,680 bytes. That's ~100MB per title.

It lives on a 4TB WD Black spinning platter drive. I moved the database to an empty 256GB SATA SSD drive, made no discernible difference to library performance, so I moved it to my 512GB PCIe M.2 system SSD, start times went from ~8 secs to ~4 secs, otherwise no discernible difference.

I figure the most I would gain from keeping my library databases on a high end SSD would be a minute or two a day. Needless to say I won't be bothering.

Thank you for the info.

I've probably got very unusual usage patterns. I don't use Calibre GUI on a daily basis but when I do use it, it's to do mass import, metadata changes (from local data) and copying. I think I spend like 2-4 hours working with the library. Even most of my conversions are I/O limited. Hence, I actually do store my full library on SSD and not just metadata.db. I tried moving my library (well, Calibre Portable) to HDD and iirc, it doubled my processing time.

[Tarana]

I have a library that is 26 gigs. Browsing is as fast as a smaller library. Searches are slower obviously due to the large amount of data to go through. The searches aren't terribly slow, just not instantaneous. There are about 76,000 files in this library of 26,000 or so ebooks. Hope that helps. I honestly didn't even know this library was that big until I tried to fit it on a partially filled flashdrive to move to my laptop. Wow!

Note: I don't have a lot of pdfs, but there are lots of illustrations and tables amongst the non-fiction.

[theducks]

Originally Posted by Tarana

I have a library that is 26 gigs. Browsing is as fast as a smaller library. Searches are slower obviously due to the large amount of data to go through. The searches aren't terribly slow, just not instantaneous. There are about 76,000 files in this library of 26,000 or so ebooks. Hope that helps. I honestly didn't even know this library was that big until I tried to fit it on a partially filled flashdrive to move to my laptop. Wow!

Note: I don't have a lot of pdfs, but there are lots of illustrations and tables amongst the non-fiction.

You need to use a crossover USB cable or a Network connection (a crossover network cable would need static IP's set. A lot of fussing) if you are pushing that much data. Even on a 100Mbps network, that is going to take time as the drive is probably 3 or 6Mbps, but it is a LOT faster than flash
FWIW, I still have some 'Laplink' cables, just no Parallel ports on any PC later than a P4

[ilovejedd]

Originally Posted by theducks

You need to use a crossover USB cable or a Network connection (a crossover network cable would need static IP's set. A lot of fussing) if you are pushing that much data. Even on a 100Mbps network, that is going to take time as the drive is probably 3 or 6Mbps, but it is a LOT faster than flash
FWIW, I still have some 'Laplink' cables, just no Parallel ports on any PC later than a P4

There are some really fast flash drives nowadays. I got ones that can do 100+MByte/s sustained a few years back and were faster than HDD at random 4K writes (SanDisk Extreme).

Don't use them much anymore though. I've got spare 128-500GB SSDs from when I upgrade internal SSDs to 1TB so I just use those paired with a $5 SATA to USB adapter for moving files between PCs sneakernet style. Significantly faster than HDDs particularly with small files.

[Tarana]

I still use flash drives, so I'll have to take a look at the fast ones for transferring large lots of data. I wonder if they have a higher failure rate? My secondary backups previously were kept at sister's house, but now I'll be using a safety deposit box at the local bank. I can fit 3 or 4 WD passports and lots of flash drives plus important papers in one box for a nominal annual cost. Of course, my sister's house was free, but she is moving out of state. Fire can happen to anybody - the only data and family photos my cousin still has from their house fire are the flashdrives he had left out in the car and the photos I had in my possession.

[stumped]

some small but large capacity flash have a bad rep. for overheating.
if one survives the initial bulk write, and does not burn your fingers when you take it out, it should be fine.

many are not suitable for leaving always connected
for backup for a safe deposit box though, I would get a small USB (pocket sized)portable hard drive, 500Gb is dirt cheap.
with those, if the controller fails, and it's not encrypted, it may be possible to break it open and read the drive via a caddy or adaptor.

ps when I was looking into home safes and stuff, I found various fireproof A4 wallets cheap on ebay, whether a flash drive in one of those would survive a fire? . could be worth looking at . search for fireproof document storage

PPS, as previous post says: SSD. if like me you paid silly money for 128GB SSDs to use in PCs, back when they were the only affordable size, they are now almost worthless, so pop one in a caddy or adaptor and copy all to that. UGreen make an excellent fast tool-free USB3 caddy .

[Tarana]

Here's my cousin's experience with fire. Even if the fire doesn't touch something, the heat will generally ruin the storage. His computer, hard drives and flashdrives in his computer area were never touched by fire or soot, but most of it either didn't work or had so many errors it was unusable. Printed photos, on the other hand, did fairly well once the soot was taken off it. So offsite is better than fireproof containers.

[lumpynose]

Originally Posted by Tarana

Printed photos, on the other hand, did fairly well once the soot was taken off it.

Years ago there was an article in Adobe's magazine about archiving and I was expecting them to say that Acrobat was the best but they said given how media formats change, CDs delaminating, etc. that, with proper storage, paper was still the best for hundreds and thousands of years.

If you're archiving on a hard drive, dvd, bd, or whatever, it would be prudent to pull them out once a year and verify that you can still read them, and the files on them.

Several iterations ago of rebuilding my system I bought a motherboard that had only EIDE hard drive connections; it was just as SATA was taking off. Then a few years later I rebuilt and (foolishly) didn't check that the new motherboard had EIDE connects and it only had SATA so I couldn't reconnect my old drives. And finding a supported PCIE EIDE card was difficult.

[PowerOffliner]

Thank you for all your answers and tipps.

I have three libraries with daily updates.
All titles are combined with meta content.
About 60.000 titles in one library it takes more time to access, change meta data, delete single titles which re double, etc. .
So I established a new library to get more speed.

The system: Panasonic CF-54, Windows 10, 16 GB RAM, Samsung 860 evo 1 TB SSD. Backup: Corsair GTX USB 1TB (USB-C).