Help with Library Sync

Puni · 12-08-2023, 12:34 PM

I'm currently using multi device synchronization for my various computers, and in the process it syncs all changes between my Calibre folders between devices. However because Calibre doesn't update the metadata.db file upon edit, only upon application shutdown, I have to be careful to ensure that I don't accidentally leave Calibre open on multiple devices, otherwise the last client that closes Calibre overwrites the metadata.db file with a potentially older index, creating a race condition. I'd also like to leave a calibre instance running for the content server for mobile devices but it also suffers from this race condition and needs to be manually shut down before library management. It works well but I'd like to see if anyone else has a better approach to automated library replication across devices.

Sirtel · 12-08-2023, 02:19 PM

Calibre is not built for syncing across multiple devices. There's no good solution for that and if you do it, you'll run the risk of corrupting your libraries.

Puni · 12-08-2023, 02:49 PM

Yeah, I'm aware it wasn't designed for it, but the synchronization use case happens when using a large variety of common tools (Google Drive, OneDrive, Resilio Sync, Offline folders, etc.) so I'm curious what others have tried. One option is to just let replication happen and not run Calibre from more than one device, but I haven't found any really great windows clients that work better than Calibre (which doesn't have a read only mode). I'm considering just making the calibre libraries read only on the filesystem but looking for cleaner solutions.

Sirtel · 12-08-2023, 03:14 PM

Quote:

Originally Posted by Puni

Yeah, I'm aware it wasn't designed for it, but the synchronization use case happens when using a large variety of common tools (Google Drive, OneDrive, Resilio Sync, Offline folders, etc.) so I'm curious what others have tried. One option is to just let replication happen and not run Calibre from more than one device, but I haven't found any really great windows clients that work better than Calibre (which doesn't have a read only mode). I'm considering just making the calibre libraries read only on the filesystem but looking for cleaner solutions.

Personally I have used Dropbox (I still use it for backup, but not for library syncing any longer), and no matter how careful I was, I still got conflicted database and book folders. I had to run the library check every time I synced across more than one device. So I don't recommend doing it.

Years ago I tried OneDrive, which ended in OneDrive starting to randomly delete my files. And Google Drive is not compatible with the calibre library structure at all (as in, it'll mess the library up pretty quickly).

theducks · 12-08-2023, 03:50 PM

Calibre is more than book files (which sync usually handles).
There is a DATA BASE (DB) that must MATCH. That is where you foot shoot yourself by multiple syncs.

Puni · 12-08-2023, 05:04 PM

One major problem with Calibre is that it needs to be used as a client in windows cases because it renames UTF-8 book titles to ASCII so they're often unidentifiable in the filesystem, which makes the DB and using the Calibre client pretty important. The mobile epub clients do a pretty good job of using the epub tags for presentation, but opening Epubs from Windows explorer is pretty bad. As far as I can tell, Calibre tries to rename all my Japanese files into chinese ASCII romanization once imported into the Calibre library structure, so I can really only find what i'm looking for from the Calibre UI.

theducks · 12-08-2023, 05:18 PM

Then why are you using it at all if it is so painful to use

Puni · 12-08-2023, 05:22 PM

Just because something is imperfect that's not a reason to abandon it, you look for solutions. I actually didn't realize it was renaming all my books until months later when I had already migrated a lot of content into it, so you could say I'm also invested into the platform. The application supports UTF-8 just fine internally, the editor, sorting etc.

Puni · 12-08-2023, 05:28 PM

Quote:

Originally Posted by theducks

Then why are you using it at all if it is so painful to use

It's still the best tool for the job, and I'm looking to continue using it for library management. Internally, it supports UTF-8 just fine, sorting, editing etc. It just garbles filenames and author names for non-english books.

BetterRed · 12-08-2023, 05:50 PM

Quote:

Originally Posted by Puni

It's still the best tool for the job, and I'm looking to continue using it for library management. Internally, it supports UTF-8 just fine, sorting, editing etc. It just garbles filenames and author names for non-english books.

Calbre also garbles English author names and titles, albeit not as drastically, but more so if it's set for middle-out truncation.

Click image for larger version

Name: Screenshot 2023-12-09 094712.jpg
Views: 404
Size: 174.8 KB
ID: 205088

BR

slm · 12-08-2023, 11:43 PM

I long ago took to running Calibre itself on only one computer. All the files on that computer appear on several others (because they are in a Dropbox folder) but I also don't make any changes to the Calibre folder on any other computer.

I access Calibre on the other computers using the server--which works well for finding books and downloading books. I add files to Calibre from other computers by adding them to a folder ("Add2Calibre") in my dropbox folder but not inside the Calibre folder. I set Calibre to automatically add files in that folder. (Add a file to that folder on another computer results in the file appearing in the folder on the Calibre computer and then Calibre noticing the added file.)

This is the strategy I use. Maybe it will work for you.

Puni · 12-08-2023, 11:58 PM

Quote:

Originally Posted by slm

I long ago took to running Calibre itself on only one computer. All the files on that computer appear on several others (because they are in a Dropbox folder) but I also don't make any changes to the Calibre folder on any other computer.

I access Calibre on the other computers using the server--which works well for finding books and downloading books. I add files to Calibre from other computers by adding them to a folder ("Add2Calibre") in my dropbox folder but not inside the Calibre folder. I set Calibre to automatically add files in that folder. (Add a file to that folder on another computer results in the file appearing in the folder on the Calibre computer and then Calibre noticing the added file.)

This is the strategy I use. Maybe it will work for you.

That sounds pretty good, I'll try moving my calibre management server to a linux docker container, share out the storage path over SMB, and have my PCs autoimport from the share. Thanks

Puni · 12-09-2023, 12:16 AM

Quote:

Originally Posted by BetterRed

Calbre also garbles English author names and titles, albeit not as drastically, but more so if it's set for middle-out truncation.

Attachment 205088

BR

Thanks that's lead to a few answers. Since automatic romanization of DBCS characters can easily turn a single Japanese kanji to 6-8 characters in (what looks like) Pinyin, it's going to overrun the NTFS limit really fast and middle-out truncation of that is making it a hot mess. I'm looking at the author's comments on UTF-8 settings in the OS but it's not clear how that corresponds to Windows 11 controls, which has separate language controls for IME, Locales, as well as Explorer localization only. UTF-8 is natively supported in the FS but I'm suspecting he is referring to locale which changes how shell applications handle data input.

BetterRed · 12-09-2023, 01:48 AM

Quote:

Originally Posted by Puni

Thanks that's lead to a few answers. Since automatic romanization of DBCS characters can easily turn a single Japanese kanji to 6-8 characters in (what looks like) Pinyin, it's going to overrun the NTFS limit really fast and middle-out truncation of that is making it a hot mess. I'm looking at the author's comments on UTF-8 settings in the OS but it's not clear how that corresponds to Windows 11 controls, which has separate language controls for IME, Locales, as well as Explorer localization only. UTF-8 is natively supported in the FS but I'm suspecting he is referring to locale which changes how shell applications handle data input.

I suspect you mean "the Windows limit".

BR

Puni · 12-09-2023, 04:47 AM

Quote:

Originally Posted by BetterRed

I suspect you mean "the Windows limit".

BR

I haven't tried calibre in Linux, perhaps that might be a different experience when adding, going to be a weekend project.

12-08-2023, 12:34 PM	#1
Puni Member Posts: 11 Karma: 10 Join Date: Dec 2023 Device: Moon Reader	Help with Library Sync I'm currently using multi device synchronization for my various computers, and in the process it syncs all changes between my Calibre folders between devices. However because Calibre doesn't update the metadata.db file upon edit, only upon application shutdown, I have to be careful to ensure that I don't accidentally leave Calibre open on multiple devices, otherwise the last client that closes Calibre overwrites the metadata.db file with a potentially older index, creating a race condition. I'd also like to leave a calibre instance running for the content server for mobile devices but it also suffers from this race condition and needs to be manually shut down before library management. It works well but I'd like to see if anyone else has a better approach to automated library replication across devices.

12-08-2023, 05:22 PM	#8
Puni Member Posts: 11 Karma: 10 Join Date: Dec 2023 Device: Moon Reader	Just because something is imperfect that's not a reason to abandon it, you look for solutions. I actually didn't realize it was renaming all my books until months later when I had already migrated a lot of content into it, so you could say I'm also invested into the platform. The application supports UTF-8 just fine internally, the editor, sorting etc. Last edited by Puni; 12-08-2023 at 05:26 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Library and sync problems	stodgeboy	Library Management	3	01-18-2018 03:41 AM
How to sync library after exported library	chantw0968	Library Management	3	10-30-2017 08:40 PM
Sync Database with Library	Mythlandia	Library Management	3	07-07-2014 11:54 AM
Touch Library and sync problems	leslie_ms	Kobo Reader	3	04-12-2012 09:54 PM
Sync Library on two computers	BrianG	Calibre	8	01-04-2010 06:59 PM

12-08-2023, 02:19 PM	#2
Sirtel Grand Sorcerer Posts: 13,426 Karma: 238915019 Join Date: Jan 2014 Location: Estonia Device: Kobo Sage & Libra 2	Calibre is not built for syncing across multiple devices. There's no good solution for that and if you do it, you'll run the risk of corrupting your libraries.

12-08-2023, 02:49 PM	#3
Puni Member Posts: 11 Karma: 10 Join Date: Dec 2023 Device: Moon Reader	Yeah, I'm aware it wasn't designed for it, but the synchronization use case happens when using a large variety of common tools (Google Drive, OneDrive, Resilio Sync, Offline folders, etc.) so I'm curious what others have tried. One option is to just let replication happen and not run Calibre from more than one device, but I haven't found any really great windows clients that work better than Calibre (which doesn't have a read only mode). I'm considering just making the calibre libraries read only on the filesystem but looking for cleaner solutions.

12-08-2023, 03:50 PM	#5
theducks Well trained by Cats Posts: 31,009 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Calibre is more than book files (which sync usually handles). There is a DATA BASE (DB) that must MATCH. That is where you foot shoot yourself by multiple syncs.

12-08-2023, 05:04 PM	#6
Puni Member Posts: 11 Karma: 10 Join Date: Dec 2023 Device: Moon Reader	One major problem with Calibre is that it needs to be used as a client in windows cases because it renames UTF-8 book titles to ASCII so they're often unidentifiable in the filesystem, which makes the DB and using the Calibre client pretty important. The mobile epub clients do a pretty good job of using the epub tags for presentation, but opening Epubs from Windows explorer is pretty bad. As far as I can tell, Calibre tries to rename all my Japanese files into chinese ASCII romanization once imported into the Calibre library structure, so I can really only find what i'm looking for from the Calibre UI.

12-08-2023, 05:18 PM	#7
theducks Well trained by Cats Posts: 31,009 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Then why are you using it at all if it is so painful to use

12-08-2023, 11:43 PM	#11
slm Fool Posts: 468 Karma: 4113712 Join Date: Feb 2003 Device: Kindle: Voyage,PW1,KOA, Kobo: Clara Colour, Nook GLP, Pocketbook verse	I long ago took to running Calibre itself on only one computer. All the files on that computer appear on several others (because they are in a Dropbox folder) but I also don't make any changes to the Calibre folder on any other computer. I access Calibre on the other computers using the server--which works well for finding books and downloading books. I add files to Calibre from other computers by adding them to a folder ("Add2Calibre") in my dropbox folder but not inside the Calibre folder. I set Calibre to automatically add files in that folder. (Add a file to that folder on another computer results in the file appearing in the folder on the Calibre computer and then Calibre noticing the added file.) This is the strategy I use. Maybe it will work for you.

Advert

Advert