Coping with huge discrepancies between original and restored database

[Beanxx]

Dear Folks--
For a long time I have been maintaining my Calibre library across several machines using Dropbox. Recently I have become increasingly aware that the file structure in Dropbox does not correspond well to the files shown in the Calibre library view. So I made a backup, and then ran a restore database.

The result of the restore were pretty frightening. It looked very different from the library as previously shown. More than a thousand books disappeared. From checking a sample, it look like there are media files for most of these in the backed-up database. In addition, more than a thousand titles which were not previously visible appeared. So the restored library is very different than the saved library.

I've been reading forum posts and the Manual and I am pretty sure I know what I did wrong. I was unaware that when using Dropbox you can not run Calibre on more than one machine at a time. I am sure I have done this as often as not over a long period of time.

I'm not going to make that mistake any more, and I'll poke about to see if there is a better way to maintain multiple synced local copies. But I am looking for advice and instruction on what the heck I do now. Should I try to merge the backed-up and restored libraries? Should I delete on of the DB files and import the folder structure? Or what?

I really do not want to throw away hundreds or thousands of books and the hundreds of hours I have spent fixing the metadata for them.

Sympathetic advice gratefully welcomed.

Sincerely, Andrew

[kovidgoyal]

Create a new empty library then right click the add books button and choose add books one per folder and add your old library.

[eschwartz]

Adding the books to a new library as Kovid suggested should re-import all the books currently in your Dropbox folder .


In future, it is absolutely critical that you do a complete sync before and after you run calibre -- and pause Dropbox while calibre is running.
As long as you do that, it should be perfectly safe. I have been doing that for a very long time, and my database remains very much intact. And it gets synced between computers.

You just need to be careful, that's all.

[Beanxx]

Originally Posted by kovidgoyal

Create a new empty library then right click the add books button and choose add books one per folder and add your old library.

Hi, Kovid, thanks!

Just to make sure I understand the procedure you recommend, do you think I should do this with both my backup and rebuilt library into one empty new library? Or just the rebuilt one and then import it into the backup? Or do it to both and then import one into the other?

In case it matters, the rebuilt library has a substantially greater number of items, including a lot of scholarly papers with faulty metadata. The backup has more of the things I care most about (the books), and has reliable metadata for the books it contains.

I think this teaches me that Dropbox is not the right sync approach for me. First, I have a bunch of other things in Dropbox, not just my calibre library, so turning off synch frequently seems likely to cause me trouble. Second, I am interested in exploring the calibre web server (mainly for use with a windows tablet which is my main book reader but which has neither wireless capacity nor enough storage for my whole library) which, as I understand it, requires that calibre be left running on the host computer.

Do you have a preferred or recommended approach to maintaining synchronized local copies of one's library across multiple machines?

Again, many, many thanks.
Peace & joy, Andrew

[eschwartz]

The database only lists the books it successfully recovered, but all the files should remain intact. So I expect that both libraries have the same files, but their metadata.db's each list a different subset of those files.
Restore database cannot recover non-existent files, so everything is still in the backup.

Dropbox probably created numerous "_____'s conflicted copy at %date", which messed up the database tracking.


Re-adding from one library folder should suffice, as the Adding books process does not expect a specific format for the filenames. It will add all books, whether they showed up in the database or not.

[eschwartz]

As for syncing solutions, all services share the same fault.

But if you only modify the library from one computer, you could use a file-syncing program to do a mirror sync into Dropbox, or use something like a Bittorrent Sync read-only folder.