Can I rsync my library?

[Cactus Chef]

My laptop was my daily driver for 7ish years, then about six months back I finally built a proper new desktop tower. I did a one-time export of my Calibre configs and libraries over to the new desktop with the assumption that it would become my daily driver, but for various reasons it continues to be my laptop. Now the library is out of sync again.

Both the laptop and the PC are running Linux. Can I safely rsync the Calibre Library directory between the two endpoints directly? This would be easier than my current method of using a flash-drive go-between (e.g. rsyncing from laptop -> flash drive, then flash-drive -> desktop). I had heard it was a bad idea to try to rsync across different OS's, but if they're both Linux, I'm assuming it should be fine?

[Quoth]

It seems fine doing a one-way rysnc, But I only change stuff on one Linux box. Also I use a preset settings via grsync that only overwrites old files and never removes anything.

You may end up with extra caltrash, or extra stuff, so occasionally I do an export, copy that to server, delete calibre library on server and laptop and then (g)rysnc.
The export is then an archive. I might delete an archives export after a new one is made.
Keep both Linux and Calibre at same revision.
I do also have a test linux box and a VM with Win10 (but only a few ebooks on that) out of curiosity.

I use my workstation which has a 4K screen and UPS. I rsync to a server. I rsync from that to a laptop if I'm going out somewhere, and then I don't change anything on Calibre.
I might edit or write LO Writer or plain text when out, and then manually copy via lan back to workstation.

I also do backups using Timeshift and direct copies to Ext4 USB HDD.

[Cactus Chef]

Originally Posted by Quoth

It seems fine doing a one-way rysnc, But I only change stuff on one Linux box. Also I use a preset settings via grsync that only overwrites old files and never removes anything.

Looking at screenshots of grsync--does it have the ability to do a verbose dry-run before running like the cmdline does?

[JSWolf]

FreeFileSync works very well and can do a one way sync. It's very easy to use. There is a Linux version.

Given that both are Linux, it will be fine.

https://freefilesync.org

[JSWolf]

Originally Posted by Cactus Chef

Looking at screenshots of grsync--does it have the ability to do a verbose dry-run before running like the cmdline does?

FreeFileSync can show you what it's going to do before it does it.

[Quoth]

Originally Posted by JSWolf

FreeFileSync can show you what it's going to do before it does it.

So can rsync / grsync

[Cactus Chef]

Just to confirm: which file is the Calibre database itself? Is it the metadata.db file? Is there anything else I need to copy over?

I installed grsync and did the laptop->flash drive->computer sync, and could see that it mostly moved only new books since my last export/restore (along with a bunch of metadata files in the ebook folders, and inexplicably some EPUBs that are very old--dunno if the folder name changed or if rsync detected some file corruption, or what). But Calibre itself is not showing any of the new books. I opened the metadata.db file in a sqlite browser and I can see none of the new books are in the data table, so I'm assuming that file failed to copy over for some reason.

[ownedbycats]

metadata.db is the database, yes.

[theducks]

The whole folder is The Library.
It is a Hybrid.

The DB MUST be in lockstep with those other files.
Those names and (ID) are stored in the DB. That is why the rule about messing under the skirt as even the tiniest change can break the linkage.

Incremental (mirror) backups, copy all changed items (the files along with the DB)

[jmurphy]

Originally Posted by Cactus Chef

which file is the Calibre database itself? Is it the metadata.db file? Is there anything else I need to copy over?

There is no single file, yes there is more to copy.
The library is the folder that contains metadata.db and any other files in that folder and all subfolders and all files under that folder.

[jmurphy]

Originally Posted by ownedbycats

metadata.db is the database, yes.

No, not at all. metadata.db is *a* database, but it is not *the* calibre database. The calibre db is all the folders and subfolders and all the files, starting from the folder containing metadata.db. THAT is the calibre db. All of it.

[ownedbycats]

Not necessarily. The CALIBRE_OVERRIDE_DATABASE_PATH environment variable allows for having the metadata.db in a separate location than the library files.

[jmurphy]

Originally Posted by ownedbycats

Not necessarily. The CALIBRE_OVERRIDE_DATABASE_PATH environment variable allows for having the metadata.db in a separate location than the library files.

Good catch, I didn't think of that.
But the point remains. The library is part of the complete database.
If you move files around inside those folders, the metadata.db will no longer match. So both the metadata.db and all the "library" files need to be from the same point in time. Simplest to just treat them *all* as "the database".

[BetterRed]

We don't refer to a bricks and mortar library as a catalogue or card index. If you asked me about a calibre 'database', I might ask which 'database'?

All calibre libraries have at least two files which are regarded as 'databases' -- metadata.db and .calnotes.notes.db, and some libraries can have another two -- full-text-search.db and metadata_pre_restore.db.

Added: Dataset would be a good word to refer to a calibre library in the abstract.

BR

[haertig]

I rsync my Calibre installation to a remote computer a couple times a day. Been doing this for years. Never ever had a problem, even when I accidentally left Calibre running on the source machine.

My specific set up:

• My primary Calibre is running on my desktop computer - Linux - as an installed application
• I rsync my primary Calibre to a remote Linux server daily via a cron script and also manually after I add any new books
• The server runs "Calibre Web" (this is third-party software, not supplied by Calibre) in a Docker container
• The server directory that holds the rsync'ed Calibre data is mounted read-only to the Calibre Web Docker container
• Calibre Web is configured to not allow any updates to the Calibre data it serves (updates wouldn't work anyway, since the data is mounted read-only, but turning off the functionality prevents error messages from being displayed)
• After the rsync completes, the script (that does the rsync) restarts the Calibre Web Docker container to pick up the changes
• All external access to my Calibre library accesses Calibre Web on the server via the Docker container using https that is reverse-proxied by Nginx (also running in a Docker container)
• Both my primary Calibre installation on my desktop and the rsync'ed copy on the server are backed up daily to a remote backup server, also Linux, using software named UrBackup, which I believe uses rsync under-the-hood for the backup

Calibre should not be running when you do the rsync IMHO. However, when I have accidentally left it running I have never experienced any problems. Even if I did, I could just shut down Calibre and re-run the rsync. Note that I am doing a one-way rsync from my desktop computer to my server computer. Rsync never updates anything on the desktop end, it only copies data over to the server. I run my primary Calibre instance on my desktop "on demand" only. When I'm done doing what I'm doing with it, I shut it down. I have no need for it to be running continuously.