Planning for My Next Crash

[slm]

Well, I've read the advice never to run Calibre and a cloud-sync program like Dropbox or Box at the same time. And despite that advice, I have stored my Calibre library in a dropbox-synced folder on my hard drive for the last fifteen or so years.

All of the files stay physically present on the hard drive. I never modify a file in the library in any fashion except through Calibre but I do retrieve some files form the cloud version for other uses.

So far, fingers-crossed, no issues.

I understand that I have been warned that this is will someday cause a mess, and I believe that warning. In the hope that when that disaster happens I will have a strategy for recovery, I'd really like a technical explanation of what Calibre is doing that might result in the Calibre library becoming messed up and what kind of mess might occur. For example, is the damage likely to be to the database files or is there a chance that some of the ebook files themselves will be corrupted?

(In passing, I should mention that I've always been impressed with Calibre's file handling. It seems to be quite fault-tolerant.)

Many thanks.

[DNSB]

For the most part the issues are with the SQLite code. On the SQLite website, you can find SQLite Over a Network, Caveats and Considerations article for information from the horse's mouth, so to speak.

The other item I've posted a couple of time before is:

Not Recommended Over Network Filesystems:

While it is technically possible to place a SQLite database file on a network share, it is strongly discouraged by the developers.

Performance Issues: Latency on most network filesystems significantly degrades performance.

Data Corruption Risk: Network filesystems often have buggy or unreliable file-locking implementations, which can lead to database corruption if multiple processes try to write simultaneously.

Concurrency Limits: SQLite allows unlimited simultaneous readers, but only one writer at a time per database file. This model works well with local file system locks, but is problematic over a network.

With the higher speeds and greater reliability of modern WAN connections, it is safer than it was even 5 years ago. OTOH, it is a risk and years of working in IT made me risk adverse.

If you browse through the number of times over the years, that people have not only lost their database but their entire book collection using calibre over a network connection (mostly cloud services but the occasional local NAS box), you might also want to take a look at storing your library locally.

[slm]

Thank you but I'm missing the connection: CAlibre's SQlite is never accessing anything but the local drive--All of the files exist on the local drive. A copy exists in the cloud and the Dropbox program (often) reads the files on the local drive and copies some of them.

Why would SQlite care that another network is sometimes copying the local file?

[DNSB]

Unless you have reliable file locking, the file that is being copied to DropBox could be modified while being copied which, shall we say, is rather ungood. See the data corruption item in the quote I posted. This is why stopping syncing is rather critical to using a calibre library with a live network connection.

[slm]

Thank you. That makes sense and tells me where to look for trouble.

[ShellShock]

I think there are two fundamentally different access scenarios here:

Calibre library on network storage/cloud storage/NAS

The Calibre library is stored entirely remotely and there is no local copy, so file I/O from the local Calibre process is over the network/internet. This is what the SQLite web site is warning against. I don't have any experience of this but I am sure their warnings are valid.

Calibre library on local storage

The Calibre library is stored entirely on the local file system, and all file I/O from the local Calibre process is to the local file system, as normal. In addition, there is a separate file syncing process (e.g., Dropbox) that synchronises the library files with a remote copy e.g., in the cloud.

This is the scenario I use and am familiar with. Here the situation is more nuanced. I use it to partly backup my library, but mainly to enable me to download epubs in the library (synced to the Dropbox cloud) to my IPad for reading, using an IOS Dropbox client.

We can assume that Calibre needs full access to the files in its library, for reads and writes, and will use shared as well as exclusive locks (e.g., for writes).

Ideally, we want the syncing process to only use shared reads on the library so as not to interfere with Calibre. I take the risk that, as long as I only change library files using Calibre, then Dropbox will detect these changes and use a shared read to copy the changed files to the Dropbox cloud.

As I am only interested in accessing the epubs in the library from other devices, I have recently excluded the Calibre database (metadata.db) and other related files from Dropbox synchronisation. How to ignore files in Dropbox is explained here https://help.dropbox.com/sync/ignored-files. Of course, this requires an alternative backup mechanism for these excluded files, as it is no longer using Dropbox as a backup.

It is also possible to pause Dropbox synchronisation whilst using Calibre, to be double-sure.

I consider the trade-off of potential file corruption versus the convenience of remote access to my ebooks is worth it, but others should decide this for themselves.

[DNSB]

Originally Posted by ShellShock

Calibre library on local storage

The Calibre library is stored entirely on the local file system, and all file I/O from the local Calibre process is to the local file system, as normal. In addition, there is a separate file syncing process (e.g., Dropbox) that synchronises the library files with a remote copy e.g., in the cloud.

This is the scenario I use and am familiar with. Here the situation is more nuanced. I use it to partly backup my library, but mainly to enable me to download epubs in the library (synced to the Dropbox cloud) to my IPad for reading, using an IOS Dropbox client.
We can assume that Calibre needs full access to the files in its library, for reads and writes, and will use shared as well as exclusive locks (e.g., for writes).

One issue being that file locking involving a network connection can be very iffy.

Originally Posted by ShellShock

Ideally, we want the syncing process to only use shared reads on the library so as not to interfere with Calibre. I take the risk that, as long as I only change library files using Calibre, then Dropbox will detect these changes and use a shared read to copy the changed files to the Dropbox cloud.

It is also possible to pause Dropbox synchronisation whilst using Calibre, to be double-sure.

I consider the trade-off of potential file corruption versus the convenience of remote access to my ebooks is worth it, but others should decide this for themselves.

Stopping synchronization between a local copy and a remote copy while using calibre is, currently, the only process I would consider safe. Unfortunately most cloud services use a 2 way sync. Everyone should get to choose the level of risk they are comfortable with. I would prefer if the complaints when the bear stops dancing were sent to /dev/null.

I can access my cloud backups from anywhere but those are copies that are only touched by the backup process. Since I can access Dropbox or Google Drive directly from my ereaders, that those copies might be a day or two out of date is not a big worry to me.