Calibre 9 and Box Cloud Storage

[Sirtel]

Originally Posted by Mr_H_Poirot

Wow, 180,000 books!?

I'm really interested why people have such big libraries given one can't read that many books in a lifetime. Is it just that some users like to 'collect' electronic books and have them available 'in case' you want to read them?

Or is there a concern that books on standard e-books or gutenberg or wherever will become unavailable at some point and so won't be downloadable when you want to read them? So you are downloading vast amounts of such books now to safeguard against a future risk? [I find that standard e-books is often updating and improving books as people read them and find the odd error in formatting. So I wait until I want to read it as then I'll have the most polished version]

Or something else?

In my case, both reasons apply. I like to collect ebooks, and yes, I'm concerned many of them might no longer be available one day.

That said, most of my ebooks are actually purchases of modern works, not public domain stuff (although I have quite a lot of those too). Some are saved webpages, free fiction published online and the like, things I have formatted and converted to epubs. But most are just purchases.

[Stevex]

Originally Posted by DNSB

So you are saying that OneDrive and calibre are accessing the metadata.db database at the same time and that is not simultaneous access? Simultaneous access does not need to be done from a different machines when multiple processes on your local machine can screw the pooch without needing outside help.

No of course they never access the file at the same time, You guys seriously dont understand how OneDrive works.
When Calibre opens a file and it's a onedrive file, the OS passes the open to OneDrive which either passes the local cache copy of the file, or downloads the file, and the the OS passes the file handle to calibre. From that point on, it's a single use local file.
When Calibre closes the file, then the o/s hands it back to to OneDrive which syncs it back to the cloud.
OneDrive is never accessing any file at the same time as calibre, or using network drives, or anything like that.

Pretty much the worst that can happen is when you close your device before OneDrive finishes uploading. Then the cloud copies of some files can be "stale". So if you say start an edit of a book on on PC, save it but shut the PC straight down, then start later editing the same book on a different PC, you are starting from before you did the first edit. In 15 years, I did that maybe half a dozen times.

[Sirtel]

Actually, that's not the worst that can happen. You've just been lucky. I used OneDrive a decade ago or so, and ended up with some of my book files simply deleted by OneDrive. And just recently there was a post where the poster's entire database got deleted.

The rule of thumb is never to run any syncing tool at the same time as calibre.

[DNSB]

A bit oddly, I was under the impression that One Drive synced files in real time. To quote from Microsoft's documentation:

If you add, change, or delete a file or folder in your OneDrive folder, the file or folder is added, changed, or deleted on the OneDrive website and vice versa. You can work with your synced files directly in File Explorer and access your files even when you’re offline. Whenever you’re online, any changes that you or others make will sync automatically.

In my admittedly limited testing, with a small calibre library synced to One Drive, changes I made were synced in real time. Not using only the local file only until I finished but syncing in real <expletive deleted> time. I could click on the One Drive icon and pause syncing while I had calibre open or I could the calibre library to a directory that was not being synced and then copy it back afterwards. I played with batch files to do both of those but then went back to the safety of local use only with a backup to multiple locations when calibre was not running. If you want to take a look at the network activity, WireShark comes in handy.

BTW, if you don't mind my asking, what is your IT background?

[Stevex]

Originally Posted by DNSB

A bit oddly, I was under the impression that One Drive synced files in real time. To quote from Microsoft's documentation:


In my admittedly limited testing, with a small calibre library synced to One Drive, changes I made were synced in real time. Not using only the local file only until I finished but syncing in real <expletive deleted> time. I could click on the One Drive icon and pause syncing while I had calibre open or I could the calibre library to a directory that was not being synced and then copy it back afterwards. I played with batch files to do both of those but then went back to the safety of local use only with a backup to multiple locations when calibre was not running. If you want to take a look at the network activity, WireShark comes in handy.

BTW, if you don't mind my asking, what is your IT background?

I started off coding assembler in octal using punched cards - since you ask!
After that I was IT director, then a contractor and Microsoft MVP with C++ and Java. And now DB management on SQL Server / Azure. I don't know if that's too much information...

Anyway, just for the future, the internet interface to OneDrive has (a) a waste basket of deleted files, going back 30 days on the basic plan I think, and (b) a version history for each file. So you can go back to yesterday's, or last week's, db file if you have a problem. So these problems people had that you were talking about, should always be recoverable really.

Note I wasn't suggesting in any way not to have backups - I have an offsite hard disk backup of all my calibre libraries, which I make every month or so, though as I say I never needed to use it.

PS And I know about Wireshark, of course. But it's much easier to see what OneDrive is doing by logging into its web interface and watching the files version themselves

[Sirtel]

Originally Posted by Stevex

Anyway, just for the future, the internet interface to OneDrive has (a) a waste basket of deleted files, going back 30 days on the basic plan I think, and (b) a version history for each file. So you can go back to yesterday's, or last week's, db file if you have a problem. So these problems people had that you were talking about, should always be recoverable really.

That's if you notice in time that some of your files have been deleted. Many people probably wouldn't. They might notice weeks, months or even years later.

You're free to do with your own files what you want, of course. But using a calibre library with any syncing tool running is not recommended by the developer, so we should refrain from suggesting it to people as a safe way to manage their library. And there have been plenty of posts over the years about folks messing up their libraries with cloud drives.