Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 02-09-2024, 01:37 PM   #1
Cactus Chef
Addict
Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.
 
Cactus Chef's Avatar
 
Posts: 323
Karma: 5736038
Join Date: Apr 2019
Device: Kobo Sage, Kobo Clara HD, Galaxy Tab S5e, Kindle 4th Gen
Can I rsync my library?

My laptop was my daily driver for 7ish years, then about six months back I finally built a proper new desktop tower. I did a one-time export of my Calibre configs and libraries over to the new desktop with the assumption that it would become my daily driver, but for various reasons it continues to be my laptop. Now the library is out of sync again.

Both the laptop and the PC are running Linux. Can I safely rsync the Calibre Library directory between the two endpoints directly? This would be easier than my current method of using a flash-drive go-between (e.g. rsyncing from laptop -> flash drive, then flash-drive -> desktop). I had heard it was a bad idea to try to rsync across different OS's, but if they're both Linux, I'm assuming it should be fine?
Cactus Chef is offline   Reply With Quote
Old 02-09-2024, 02:13 PM   #2
Quoth
the rook, bossing Never.
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 11,699
Karma: 87663461
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
It seems fine doing a one-way rysnc, But I only change stuff on one Linux box. Also I use a preset settings via grsync that only overwrites old files and never removes anything.

You may end up with extra caltrash, or extra stuff, so occasionally I do an export, copy that to server, delete calibre library on server and laptop and then (g)rysnc.
The export is then an archive. I might delete an archives export after a new one is made.
Keep both Linux and Calibre at same revision.
I do also have a test linux box and a VM with Win10 (but only a few ebooks on that) out of curiosity.

I use my workstation which has a 4K screen and UPS. I rsync to a server. I rsync from that to a laptop if I'm going out somewhere, and then I don't change anything on Calibre.
I might edit or write LO Writer or plain text when out, and then manually copy via lan back to workstation.

I also do backups using Timeshift and direct copies to Ext4 USB HDD.
Quoth is offline   Reply With Quote
Advert
Old 02-09-2024, 02:25 PM   #3
Cactus Chef
Addict
Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.
 
Cactus Chef's Avatar
 
Posts: 323
Karma: 5736038
Join Date: Apr 2019
Device: Kobo Sage, Kobo Clara HD, Galaxy Tab S5e, Kindle 4th Gen
Quote:
Originally Posted by Quoth View Post
It seems fine doing a one-way rysnc, But I only change stuff on one Linux box. Also I use a preset settings via grsync that only overwrites old files and never removes anything.
Looking at screenshots of grsync--does it have the ability to do a verbose dry-run before running like the cmdline does?
Cactus Chef is offline   Reply With Quote
Old 02-09-2024, 06:57 PM   #4
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 74,669
Karma: 130140792
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
FreeFileSync works very well and can do a one way sync. It's very easy to use. There is a Linux version.

Given that both are Linux, it will be fine.

https://freefilesync.org
JSWolf is offline   Reply With Quote
Old 02-09-2024, 06:58 PM   #5
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 74,669
Karma: 130140792
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
Quote:
Originally Posted by Cactus Chef View Post
Looking at screenshots of grsync--does it have the ability to do a verbose dry-run before running like the cmdline does?
FreeFileSync can show you what it's going to do before it does it.
JSWolf is offline   Reply With Quote
Advert
Old 02-11-2024, 07:54 AM   #6
Quoth
the rook, bossing Never.
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 11,699
Karma: 87663461
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
Quote:
Originally Posted by JSWolf View Post
FreeFileSync can show you what it's going to do before it does it.
So can rsync / grsync
Quoth is offline   Reply With Quote
Old 04-28-2024, 08:58 PM   #7
Cactus Chef
Addict
Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.Cactus Chef ought to be getting tired of karma fortunes by now.
 
Cactus Chef's Avatar
 
Posts: 323
Karma: 5736038
Join Date: Apr 2019
Device: Kobo Sage, Kobo Clara HD, Galaxy Tab S5e, Kindle 4th Gen
Just to confirm: which file is the Calibre database itself? Is it the metadata.db file? Is there anything else I need to copy over?

I installed grsync and did the laptop->flash drive->computer sync, and could see that it mostly moved only new books since my last export/restore (along with a bunch of metadata files in the ebook folders, and inexplicably some EPUBs that are very old--dunno if the folder name changed or if rsync detected some file corruption, or what). But Calibre itself is not showing any of the new books. I opened the metadata.db file in a sqlite browser and I can see none of the new books are in the data table, so I'm assuming that file failed to copy over for some reason.
Cactus Chef is offline   Reply With Quote
Old 04-28-2024, 08:59 PM   #8
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,851
Karma: 62032373
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
metadata.db is the database, yes.
ownedbycats is online now   Reply With Quote
Old 04-28-2024, 09:30 PM   #9
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,985
Karma: 56143930
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
The whole folder is The Library.
It is a Hybrid.

The DB MUST be in lockstep with those other files.
Those names and (ID) are stored in the DB. That is why the rule about messing under the skirt as even the tiniest change can break the linkage.

Incremental (mirror) backups, copy all changed items (the files along with the DB)
theducks is offline   Reply With Quote
Old 04-28-2024, 09:55 PM   #10
jmurphy
Zealot
jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.
 
Posts: 100
Karma: 1133068
Join Date: Sep 2007
Device: ipaq
Quote:
Originally Posted by Cactus Chef View Post
which file is the Calibre database itself? Is it the metadata.db file? Is there anything else I need to copy over?
There is no single file, yes there is more to copy.
The library is the folder that contains metadata.db and any other files in that folder and all subfolders and all files under that folder.
jmurphy is offline   Reply With Quote
Old 04-28-2024, 09:59 PM   #11
jmurphy
Zealot
jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.
 
Posts: 100
Karma: 1133068
Join Date: Sep 2007
Device: ipaq
Quote:
Originally Posted by ownedbycats View Post
metadata.db is the database, yes.
No, not at all. metadata.db is *a* database, but it is not *the* calibre database. The calibre db is all the folders and subfolders and all the files, starting from the folder containing metadata.db. THAT is the calibre db. All of it.
jmurphy is offline   Reply With Quote
Old 04-28-2024, 10:17 PM   #12
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,851
Karma: 62032373
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Not necessarily. The CALIBRE_OVERRIDE_DATABASE_PATH environment variable allows for having the metadata.db in a separate location than the library files.
ownedbycats is online now   Reply With Quote
Old 04-28-2024, 10:24 PM   #13
jmurphy
Zealot
jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.jmurphy ought to be getting tired of karma fortunes by now.
 
Posts: 100
Karma: 1133068
Join Date: Sep 2007
Device: ipaq
Quote:
Originally Posted by ownedbycats View Post
Not necessarily. The CALIBRE_OVERRIDE_DATABASE_PATH environment variable allows for having the metadata.db in a separate location than the library files.
Good catch, I didn't think of that.
But the point remains. The library is part of the complete database.
If you move files around inside those folders, the metadata.db will no longer match. So both the metadata.db and all the "library" files need to be from the same point in time. Simplest to just treat them *all* as "the database".
jmurphy is offline   Reply With Quote
Old 04-28-2024, 11:35 PM   #14
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,691
Karma: 26966376
Join Date: Mar 2012
Location: Sydney Australia
Device: none
We don't refer to a bricks and mortar library as a catalogue or card index. If you asked me about a calibre 'database', I might ask which 'database'?

All calibre libraries have at least two files which are regarded as 'databases' -- metadata.db and .calnotes.notes.db, and some libraries can have another two -- full-text-search.db and metadata_pre_restore.db.

Added: Dataset would be a good word to refer to a calibre library in the abstract.

BR

Last edited by BetterRed; 04-29-2024 at 12:20 AM.
BetterRed is offline   Reply With Quote
Old 04-29-2024, 12:31 AM   #15
haertig
Wizard
haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.haertig ought to be getting tired of karma fortunes by now.
 
Posts: 1,758
Karma: 26012554
Join Date: Sep 2017
Device: PW3, Fire HD8 Gen7, Moto G7, Sansa Clip v2, Ruizu X26
I rsync my Calibre installation to a remote computer a couple times a day. Been doing this for years. Never ever had a problem, even when I accidentally left Calibre running on the source machine.

My specific set up:
  • My primary Calibre is running on my desktop computer - Linux - as an installed application
  • I rsync my primary Calibre to a remote Linux server daily via a cron script and also manually after I add any new books
  • The server runs "Calibre Web" (this is third-party software, not supplied by Calibre) in a Docker container
  • The server directory that holds the rsync'ed Calibre data is mounted read-only to the Calibre Web Docker container
  • Calibre Web is configured to not allow any updates to the Calibre data it serves (updates wouldn't work anyway, since the data is mounted read-only, but turning off the functionality prevents error messages from being displayed)
  • After the rsync completes, the script (that does the rsync) restarts the Calibre Web Docker container to pick up the changes
  • All external access to my Calibre library accesses Calibre Web on the server via the Docker container using https that is reverse-proxied by Nginx (also running in a Docker container)
  • Both my primary Calibre installation on my desktop and the rsync'ed copy on the server are backed up daily to a remote backup server, also Linux, using software named UrBackup, which I believe uses rsync under-the-hood for the backup
Calibre should not be running when you do the rsync IMHO. However, when I have accidentally left it running I have never experienced any problems. Even if I did, I could just shut down Calibre and re-run the rsync. Note that I am doing a one-way rsync from my desktop computer to my server computer. Rsync never updates anything on the desktop end, it only copies data over to the server. I run my primary Calibre instance on my desktop "on demand" only. When I'm done doing what I'm doing with it, I shut it down. I have no need for it to be running continuously.
haertig is offline   Reply With Quote
Reply

Tags
linux, rsync

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
KOReader and rsync dmpop KOReader 3 04-23-2022 10:37 AM
rsync PC to Kobo Lo2 Kobo Developer's Corner 3 03-25-2019 06:06 PM
Rsync on Kindle Touch leonardo84 Kindle Developer's Corner 11 10-31-2012 05:31 PM
Request Geek Tools for moving files to/from the Library: ssh/sftp/rsync/samba kenjennings enTourage Archive 6 04-06-2010 11:36 PM
rsync thetechnobear iRex 1 03-05-2008 01:57 PM


All times are GMT -4. The time now is 01:50 AM.


MobileRead.com is a privately owned, operated and funded community.