One Calibre to rule them all: help me collapse my libraries

[radius]

I've tried to make a go of Calibre off and on for many years now, and have finally decided to go all in for good.

The problem is I have many different Calibre libraries scattered over different machines (running different OSes and where Calibre is no longer installed and who knows what the version was anyways) and in different cloud storage etc.

What is the best way to bring all of my disparate libraries together? This is several hundred books in my "test" libraries, so I'd prefer not to import the raw epub files and lose all my tags and so on if there is an easy way around it. Also, I have different fields in different libraries (e.g. one might have a "read" column, another might have a number of pages column etc)

Finally, I'd like to access this central library from multiple computers. I know not to have multiple Calibre instances running at the same time to preserve database integrity, but was wondering if there is a best practices solution for this. In the past I remember being messed up by one cloud provider adding "(1)" to the end of file names, as an example of why I am careful about this.

Thanks!

[Dal Bhat]

In the past I remember being messed up by one cloud provider

As you will be in future.

Don't put your database in an app that syncs, like Google or OneDrive. Keep it local.

[Quoth]

1. Have one computer that is the only one where stuff is added. Local calibre.

2. Use rsync or similar to copy/overwrite all changed files to a local server or NAS etc when calibre isn't running.

3. Then rsync from that to the other computers when calibre isn't running and must not have been running since 2. was done.

OR run the Calibre Content server on one computer.

Calibre or calibre library must be "local" but that can include a USB HDD, USB SSD, USB memory stick, firewire or SCSI or eSATA external drive, or SD Card etc.

You can't run calibre with a library on any kind of network drive or share.

Only save encrypted copies to Cloud as it's simply someone else's server with unknown security and backup. Best for sharing or websites or collaboration, not backups.

I have a separate USB HDD for backups (more reliable than SSD when disconnected) and the luxury of off-site (12 hour solar UPS on server wit a local UPS in a concrete garden shed via 1Gbps ethernet so in good weather indefinite power on IT). I used to leave a USB HDD with a friend and swap it.

[Quoth]

I've one Calibre install & database on a RPi4 with SD card and another on a Samsung Chromebook converted to native Linux with all user files on a 512 Gbyte micro SD card formatted ext4.

Any local storage should use the OS native filesystem, never exFat or Fat32 as they are only for gadgets or portable files.

[DNSB]

Originally Posted by Quoth

You can't run calibre with a library on any kind of network drive or share.

Minor nit. You can run calibre with a library on almost any network drive/cloud storage other than GoogleDrive. You may even get away with doing so for years. Then you will be posting the "my library is corrupted" message.

[rantanplan]

SMB/CIFS is indeed not the same as a cloud share 😬

[DNSB]

Originally Posted by rantanplan

SMB/CIFS is indeed not the same as a cloud share 😬

SMB/CIFS is network storage and SQLite can be unhappy with it though most home users are fairly safe.

[kacir]

Install a Calibre into a "central library computer" and create a new empty Master Library there.

Copy all libraries from all the devices and backups you have to that computer. Each library will have its own separate directory. This is only temporary, as you will be copying all the books to the Master Library.

Open each library in Calibre, select all the books and select "Copy books to library". Select Master Library. Calibre program will prompt you when you have additional columns in the origin library and ask you what to do. You should be able to open even very old libraries, even from different operating systems. (it worked for me on numerous occasions) In the worst case import the books from a broken library, or use Library Maintenance menu and check / rebuild the database from the backup Metadata.opf files.

When you finish copying all the books from all libraries you open Master Library and start consolidating duplicate books. When the book has the same title and author it is automatically merged with existing books when copying from a library, depending on your Preferences. You can view individual authors one-by-one and sort books by title and see duplicates, you can use several plugins for identifying duplicates, there is plugin for searching the Internet for metadata or for opening the page with info for a book or an author on various sites (goodreads, fantastic fiction and others) so you can have consistent naming for series.

Generally it is not a good idea to have *active* library files on network drives, but you should make a backup and that can be on network. Several backup copies will be even better. Some of them off-site is excellent.

You can run on a copy of Master Library on other devices / computers or you can use a server functionality of Calibre and have other devices access your Library using a simple browser.

I am on Linux and I use rsync to make a backup. There is grsync graphical user interface even for Windows. On Windows some people like to use command robocopy and it also has a third-party "graphical" user interface so you can set options better. There are other tools for synchronizing backups, depending on whether you want to have history (in case you delete something from master library and want to recover it later from a backup) or whether you want to keep a plain copy. With bigger libraries a good tool for making a backup, especially over a network, can speed things up significantly. You can also consult AI to help you with rsync or robocopy commands.

[Quoth]

Originally Posted by kacir

You can also consult AI to help you with rsync or robocopy commands.

No, search or built in help. Not AI. AI is a garbage way to search.

All the rest of the advice is fine.

For example, using Qwant (same any other search engine)

https://www.qwant.com/?q=rsync+commands

Or rsync --help in a console/terminal