Tips for creating/maintening a robust and OS-independent book library

[peter0conor]

Good morning,

I' m in the process to unify several book databases that I have, and create a master library database which could be more robust for corruption and last long independently of which operating system I use in the future. To be honest, I prefer a reliable book database which is stable than a database with faster performance. For those of you how have longer experience with book databases, could you please give me your recommendations about the following?

- How many files or books should I have as a limit before creating a second library?
- Should I keep the files compressed (i.e. rar, or zip) or uncompressed as pdf, epub, mobi. etc?
- Should I create a structure by folders (i.e. authors) and sub-folders (i.e. titles) with the different book formats (i.e. pdf, mobi, epub, etc) all together or should I just leave all files together in a flat structure with only one parent folder?
- Should I rename book files with a longer but more meaningfully name (i.e. Dan Brown - The Davinci Code), or use short names (i.e. Davincicode.pdf) avoiding spaces or symbols to make it more difficult to get corrupted?
What should I do to maintain the configuration files that keep my customizations (i.e. tags, covers, new columns, etc)
- Should I use specific hard drive to keep my database and access it remotely or locate the database in the same location where my OS?
Thanks for your time and support,
Sincerely,
Peter

[theducks]

Thanks for the laugh (Last long! OS independent COMPATIBLE )

As the owner of an Amazing collection of unsupported (or nearly) media formats that are either gone or going away to the point that collectors or estate sales are the only source for reproducers.
(the list is for only stuff I had mainline media for, not that I had, at one time, equipment to play that you can not easily buy needed equipment)

78 records (mono), 45 records (mono), 33 records (Mono, Stereo, SQ, QS, CD4, DBX), 8-track tape (stereo), compact Cassette tape (mono, stereo), Open reel tape (1/4in): 1-7/8 (mono), 3-3/4 (Stereo), 7-/2, 15 (Mono, Stereo, Quad), CD (home audio style players) and HA about (native format) decoder/processor boxes or amplifiers (with quad switching)

2-1/4 (120/620 film) slides, 8mm movie film, VHS tapes, Laserdisc , (you can still find Bluray players which also play DVD, but the selection is getting thin in big box stores) and the home theater receivers have lost input types for older video leads

5-1/4 floppy disc (singe and double sided. forget about the file system format part : TI, CPM: MSDOS), 3" floppies, Zip disks (100,250M), a number of cartridge tape systems that I don't even remember their names, (computer CD and DVD are still available)

Paper books: Oh wait, that problem is just because the thinner shelf material being sold these days, can't hold up my dead tree collections

Seriously. Metadata by file system (path/file name) is limited.

Calibre includes a metadata.opf which is a XML encoded (tagged text) copy of what the DB entry had (primary reason was to allow recovery). those files can be VIEWED (parsing needs a few more tricks) with any text viewer/editor.
You can make a CSV catalog (I have had spreadsheets that could import that since 1980), so that is probably going to be around another year or so ) of the metadata.
I would be more worried about physical media failure (recorded media fades or refuses to spin freely) or becoming unsupported by computers. I could not get XP to read any of my W95era 5-1/4 floppies (I installed the drive, it would boot from a DOS disk)
To recap. Don't count on longevity unless you want 33-1/3 LP's (I hear they are coming back into vogue. Almost 70 years of life, not bad )

[uli]

• Only one main Calibre database for fiction and nonfiction.

Programming or books with supplemental data/source code are zipped and added alongside.

Used to keep comics separately >100mb per file as not to bloat the library, have so few I finally imported them.

Audiobooks are still separate, I make "backup" with InAudible into individual mp3 chapters. The one filetype limit per book in Calibre means I create a m3u playlist and link it up. If I started today I'd create single m4b instead...

• Epubs, etc are already compressed. I was sorting by Author/Book before and if I ever wanted to abandon Calibre I'd just strip the Ids and be done. 10 years later still using Calibre!

• Filename when imported already avoid characters/symbols/lengths incompatibilities with other filesystems.

• Column and other customization can be backed-up. Title, Tags, Description are saved in both the metadata.db and individual opf files.

• It's recommended to use a local hard drive, networks drives at your own risk.

I have Calibre running on a server that takes regular snapshots (ZFS) and sent offsite, most modern OS can do similar. Although OneDrive, DropBox, etc might lock files and create a mess.

My server has a GUI, but you can also manage remotely with Calibre's server or something like calibre-web.

[BetterRed]

↑ ↑ ↑ ✔

See inline comments in blue

Originally Posted by peter0conor

<snip>

How many files or books should I have as a limit before creating a second library?

There is no practical limit, some users have libraries that have hundreds of thousands of 'books', I have one which has a 154,000 'items' - they are public domain 'papers' from academia etc.

My several libraries exist because they have different custom columns, and my varying usage of standard columns.


Should I keep the files compressed (i.e. rar, or zip) or uncompressed as pdf, epub, mobi. etc?

No, in many cases its a waste of time because they are already compressed, and if you did you would lose a lot of calibre functionality - conversion, view, send to device etc.

Should I create a structure by folders (i.e. authors) and sub-folders (i.e. titles) with the different book formats (i.e. pdf, mobi, epub, etc) all together or should I just leave all files together in a flat structure with only one parent
folder?

Should I rename book files with a longer but more meaningfully name (i.e. Dan Brown - The Davinci Code), or use short names (i.e. Davincicode.pdf) avoiding spaces or symbols to make it more difficult to get corrupted?

No to both, because you can't.

You need to read this ==>> Want to change folder structure or file names in the Calibre library? and the links it contains.

What should I do to maintain the configuration files that keep my customizations (i.e. tags, covers, new columns, etc)

Nothing, apart from backing them up along with your library, their location can be accessed from the Preferences->Miscellaneous


- Should I use specific hard drive to keep my database and access it remotely or locate the database in the same location where my OS?

Read this ==>> Calibre library on a network...

Calibre library(ies) are best stored on local storage, if you want to store them on remote storage you need to take precautions. Multi-terabyte USB 3 disk drives are available for under a $100, if your computer has a decent usb 3.0 controller it's performance should be as good or better than most internal SATA drives.

Remote storage is fine for backups.

BR

[Adoby]

Many good points have been made.

I would add:

• Get a good workstation. Laptop or stationary PC.
• Use calibre.
• Use epub as the main format. You can convert to any format and store all metadata with epub as the base format.
• Use ISBN to identify your books. And other widely used identifiers as available.
• Allow calibre and the OPFs to fully handle series and filenames and so on.
• Work locally, not over a LAN.
• Use a filesystem with real time bitrot protection. For instance ZFS or BTRFS with redundancy.
• Use good quality SSDs.
• Setup a good backup system. Snapshots. Multiple versions. Multiple locations. Multiple media. NAS, DAS and cloud as a minimum.
• Verify the backups regularly.
• Every year, make a full copy of everything using save to disk and catalog. Archive it using multiple media and multiple locations. Use rar-archives with error correction codes.
• Migrate to new hardware when the warranty runs out. Always! For EVERYTHING. This means that buying stuff with 5 years warranty or more is cheap and efficient.

For me this means Linux.

I don't have your requirements so I don't use ZFS or BTRFS with redundancy. I don't replace everything when the warranty runs out. And I don't make yearly archives. But other than that this is what I do. I only buy SSDs and HDDs that have 5 years warranty.

My current workstation for calibre, and more, is a Lenovo ThinkPad with 32GB RAM, a 1TB Samsung 970EVO Plus NVMe and a 2TB Crucial MX500 SATA for backups. Ubuntu Mate 9.10. I have all my finished calibre libraries on it. And incoming. Not junk and duplicates. With finished I mean that metadata is pretty normalized, the cover is nice and the book is in good shape. Can you ever fully normalize metadata for a calibre library with more than one book?

I run calibre using a script that automatically update a snapshot backup of my calibre libraries every time I run calibre. I update backup snapshots on a NAS, daily, when I charge.

[peter0conor]

Thanks to all of you for your replies. There are a lot of good tips that will help me to build my DB, and hopefully everyone else will find them useful too.
Sincerely,
Peter

[BlankEyes]

Originally Posted by Adoby

I run calibre using a script that automatically update a snapshot backup of my calibre libraries every time I run calibre. I update backup snapshots on a NAS, daily, when I charge.

Could you be so kind as to share this script? Or would it not work on MacOS?

Blankeyes

[Adoby]

It is just a bash script launching calibre and then, when calibre quits, runs rsync. So I think that you could run it fine on a Mac.

Originally it was inspired by Mike Rubel and his site "Easy Automated Snapshot-Style Backups with Linux and Rsync", but rewritten as part of a launch script for calibre.

http://www.mikerubel.org/computers/rsync_snapshots/

I wrote about this already in 2014:

https://www.mobileread.com/forums/sh...d.php?t=244228

That version simply deleted any snapshots after a certain time.

Today I have added better deletion of old snapshots. So now I can keep, for instance, 7 daily snapshots, 4 weekly and 12 monthly snapshots. Or whatever I like.

I use copies of this script for most of my backup needs. I run the scripts automatically using crontab entries on small single drive SBCs running Linux. Odroid HC2s with OMV 4 with 12TB or 16TB HDDs. And a RPi4 with two 8TB HDDs. The scripts backups folders on one NAS to another NAS. Or backup folders on one SSD on my laptop to another SSD on my laptop. Typically calibre libraries.

Here is a current version of my rsync snapshot script:

https://github.com/WikiBox/snapshot.sh

[BlankEyes]

Originally Posted by Adoby

It is just a bash script launching calibre and then, when calibre quits, runs rsync. So I think that you could run it fine on a Mac.

Originally it was inspired by Mike Rubel and his site "Easy Automated Snapshot-Style Backups with Linux and Rsync", but rewritten as part of a launch script for calibre.

http://www.mikerubel.org/computers/rsync_snapshots/

I wrote about this already in 2014:

https://www.mobileread.com/forums/sh...d.php?t=244228

That version simply deleted any snapshots after a certain time.

Today I have added better deletion of old snapshots. So now I can keep, for instance, 7 daily snapshots, 4 weekly and 12 monthly snapshots. Or whatever I like.

I use copies of this script for most of my backup needs. I run the scripts automatically using crontab entries on small single drive SBCs running Linux. Odroid HC2s with OMV 4 with 12TB or 16TB HDDs. And a RPi4 with two 8TB HDDs. The scripts backups folders on one NAS to another NAS. Or backup folders on one SSD on my laptop to another SSD on my laptop. Typically calibre libraries.

Here is a current version of my rsync snapshot script:

https://github.com/WikiBox/snapshot.sh

Thanks

On another note - will Time Machine be suffice to use in a backup situation like this?

[Adoby]

I don't know much about Time Machine. I am not a Mac user. But I assume it creates snapshots on boot?

You have to decide if that is good enough and easy to recover and so on.

[BlankEyes]

Originally Posted by Adoby

I don't know much about Time Machine. I am not a Mac user. But I assume it creates snapshots on boot?

You have to decide if that is good enough and easy to recover and so on.

Anyone else want to weigh in on this?

[BetterRed]

My understanding is that Time Machine maintains real time copies of folders and files. It's similar to Windows' Previous Versions, and the Real Time Synch tools in the GoodSync and FreeFileSync backup products.

BR

[BlankEyes]

Thanks, BetterRed and Adoby