How to Duplicate My Calibre Library on Multiple Computers?

[chilady1]

PilotBob - I did get it to work and the problem was the metadata file which many said over a few posts. As soon as I got the right metadata file loaded, all was good. However, I have discovered a new issue. I updated some books on my netbook in Calibre however, the books did not update when I opened up Calibre on my desktop. I have both Calibre libraries pointed to my Dropbox account so I assumed that the files were updating on Dropbox. Any suggestions?

[Sydney's Mom]

Did you have only one library open at a time? Remember, you have to exit on the task bar as well.

[Starson17]

Originally Posted by chilady1

PilotBob - I did get it to work and the problem was the metadata file which many said over a few posts. As soon as I got the right metadata file loaded, all was good.

How did you know you got the right metadata.db file loaded? If you have an older version inadvertently in use, you could have a big chunk of your books missing, and unless you notice that, you've got a problem. If you fail to notice, and add new books, it can turn into a real mess to untangle.

If you're using dropbox and multiple machines, please make sure you also have good backups.

[chilady1]

Originally Posted by Sydney's Mom

Did you have only one library open at a time? Remember, you have to exit on the task bar as well.

Yup, reading and following the advice of all on this thread, I have ensured that I NEVER have more than one instance of Calibre opened at a time. I have checked the files on Dropbox and I see for example a new book added is in fact located on Dropbox. When I open up Calibre on my netbook, I see the book that was added. When I open up Calibre on my desktop, the new book added to Dropbox is not showing up. Shouldn't Calibre pick up there is a new folder with a new book? That is what is confusing me.

[chilady1]

Originally Posted by Starson17

How did you know you got the right metadata.db file loaded? If you have an older version inadvertently in use, you could have a big chunk of your books missing, and unless you notice that, you've got a problem. If you fail to notice, and add new books, it can turn into a real mess to untangle.

If you're using dropbox and multiple machines, please make sure you also have good backups.

Starson - great advice and yes, it is the correct metadata, I did check against a backup done recently. And yes, I do backups of my Calibre library on a regular basis. You could say that I have a little OCD about backing up this library so I always make sure of that. I want to THANK YOU along with others like PilotBob for your really good discussions around this issue. Syndey's Mom for asking the originally question which is exactly the same issue I was having. I do love MobileRead because typically whatever question you have, someone else has already asked it.

[itimpi]

Originally Posted by chilady1

Yup, reading and following the advice of all on this thread, I have ensured that I NEVER have more than one instance of Calibre opened at a time. I have checked the files on Dropbox and I see for example a new book added is in fact located on Dropbox. When I open up Calibre on my netbook, I see the book that was added. When I open up Calibre on my desktop, the new book added to Dropbox is not showing up. Shouldn't Calibre pick up there is a new folder with a new book? That is what is confusing me.

Calibre does not detect the new folder. Instead it relies on the metadata.db file having been updated correctly. If you are not seeing the new book that implies the changed metadata.db file has not been replicated locally. That is why it is critical that Calibre is closed on the other machine so that Dropbox can replicate the metadata.db changes.

[Starson17]

Originally Posted by chilady1

Starson - great advice and yes, it is the correct metadata, I did check against a backup done recently.

Great. I worry about dropbox users who might take this setup lightly. Calibre is pretty careful with our collections, but the dropbox configuration requires the user to take extra precautions to avoid messing up Calibre. Dropbox is a great setup for those who are being careful.

[DoctorOhh]

Originally Posted by chilady1

PilotBob - I did get it to work and the problem was the metadata file which many said over a few posts. As soon as I got the right metadata file loaded, all was good. However, I have discovered a new issue. I updated some books on my netbook in Calibre however, the books did not update when I opened up Calibre on my desktop. I have both Calibre libraries pointed to my Dropbox account so I assumed that the files were updating on Dropbox. Any suggestions?

My metadata.db file is around 6meg in size and this file is locked while Calibre is opened. In order for it to update Dropbox you must exit Calibre and wait until Dropbox updates the file before you shutdown the computer or change computers.

If you shutdown Computer-A before the 6 meg file can uploaded to Dropbox, when you go to Computer-B you will find the work you did on Computer-A missing. The updated metadata.db file still exists on you Computer-A but because it never had a chance to update Dropbox it is not available to Computer-B.

If you exit Computer-B and go to Computer-A and start Calibre you'll find the changes missing here too because Dropbox has updated your machine with the most recent updated file (from Computer-B). Exit Calibre before considering the following.

The metadata.db file with all of your changes didn't disappear it is now in the same directory as your metadata.db but has been labeled as a "conflicted copy" on my machine my last "conflicted copy" was named metadata (Computer-A's conflicted copy 2010-06-08).db Read these two (one, two) posts for more info.

At this point I can rename metadata.db to metadata.old.db and rename my "conflicted copy" version of metadata.db to metadata.db. Now when I restart Calibre all of the work that was missing is back again.

[Starson17]

Originally Posted by dwanthny

If you shutdown Computer-A before the 6 meg file can uploaded to Dropbox, when you go to Computer-B you will find the work you did on Computer-A missing. The updated metadata.db file still exists on you Computer-A but because it never had a chance to update Dropbox it is not available to Computer-B.

Don't do any work on Computer-B at this point. If you do, you get a third copy of metadata.db. The first copy is what was done on both machines before the conflict. The second is what was done on both plus what was done only on Computer-A, but not yet known to Computer-B. The third is like the second, but includes what was done only on Computer-B, but not yet known to Computer-A.

Recovery from this is possible, but tricky. There will be no metadata.db file that is correct, and no way for dropbox to help. There will be no file yet available that has what is needed. You will need to run one of the last two metadata files, export the stuff known only to one machine, then run the other metadata and import the stuff you exported.

[DoctorOhh]

Originally Posted by Starson17

Don't do any work on Computer-B at this point. If you do, you get a third copy of metadata.db. The first copy is what was done on both machines before the conflict. The second is what was done on both plus what was done only on Computer-A, but not yet known to Computer-B. The third is like the second, but includes what was done only on Computer-B, but not yet known to Computer-A.

Recovery from this is possible, but tricky. There will be no metadata.db file that is correct, and no way for dropbox to help. There will be no file yet available that has what is needed. You will need to run one of the last two metadata files, export the stuff known only to one machine, then run the other metadata and import the stuff you exported.

True, once you are on Computer-B and realize you have missing books or recipes then you should halt any updating of calibre or choose to redo your changes.

Redoing your changes may cause duplicate books in the in calibre's folder structure. Duplicates won't cause any problems unless you suffer fron OCD and continually search for problems in the pattern.

[chilady1]

First, hate to sound like a broken record but THANK YOU to you all for such great advice and suggestions. Eveything is working accordingly. I think what happened was Calibre on my netbook wasn't closed - as a matter of fact I know it didn't because I had the setting marked to keep it in my tray. I have removed that setting so when I close Calibre on any machine, it closes out completely. I removed one of the books I put in last night, re-added it back into Calibre - giving Dropbox enough time to synch. Once it completed, I went to my desktop and LO AND BEHOLD, it showed up in Calibre. I am so happy that I have worked this out because now I can access my library from both computers and this is a terrific thing.

[pilotbob]

Keep in mind... Dropbox IS a backup. It keeps back copies/versions of the files it syncs.

Second... DON'T shut down computers if you can avoid it... I leave all my PCs up and running. (Yes... we have plenty of coal in this country.)

Anyway... as long as you close calibre on a PC when you aren't using it... you should be ok. Dropbox syncs at a block level so it doesn't upload the full 6MBs (if that is the size of the file) it just uploads the change delta which is very small and happens very fast.

BOb

[Starson17]

Originally Posted by pilotbob

Keep in mind... Dropbox IS a backup. It keeps back copies/versions of the files it syncs.

How many forks will it keep backed up after a conflict?

[pilotbob]

Originally Posted by Starson17

How many forks will it keep backed up after a conflict?

From their Web site FAQ:

"Dropbox saves a history of all deleted and previous versions of files for 30 days for all Dropbox accounts by default. If you have the Pack-Rat add-on, Dropbox saves those files for as long as you have the Pack-Rat add-on. With Pack-Rat, you never have to worry about losing an old version of a file."

Pack-Rat is a pro account feature. I'm not sure if it cost extra or not. I just checked pack rat is an additional $40 per year.

BOb

[Starson17]

Originally Posted by pilotbob

From their Web site FAQ:

"Dropbox saves a history of all deleted and previous versions of files for 30 days for all Dropbox accounts by default. If you have the Pack-Rat add-on, Dropbox saves those files for as long as you have the Pack-Rat add-on. With Pack-Rat, you never have to worry about losing an old version of a file."

Pack-Rat is a pro account feature. I'm not sure if it cost extra or not. I just checked pack rat is an additional $40 per year.

BOb

That sounds like it refers to the single computer - single backup aspect of Dropbox. Each file is treated as a single object in normal circumstances. That's a perfect backup if only one computer is changing that file, or if all computers are in synch. However, in the multiple computer file conflict scenario (a "fork"), there is no defined correct backup for all computers - only for each single computer - and that's the scenario I was worrying about.

Ideally, one will manually resynch when a conflict is detected before things get out of hand (a process that requires some smarts and knowledge of the files Calibre uses in its library), but what does it do when multiple inconsistent files begin to build up on multiple computers as a result of a conflict? Is it possible to get each computer restored to its different state if one fails?

Does it mark conflicted file 1 (STATUS 1) as belonging to computer 1 and conflicted file 1 (STATUS 2) as belonging to computer 2, then keep 30 day backups of both, or what?

A related question is: "What do you see in Calibre during the conflict state?" Do you see your computer 2 metadata.db with all the books you added on computer 2 after the fork, or do you lose those and see only the files added on computer 1? To ask that a different way, I know that there will be a conflict file generated for metadata.db on computer 2. Is that file the computer 1 metadata.db or the computer 2 metadata.db?

Similarly, once the fork occurs, you begin to generate folders using calibre ids. They will be replicated from computer 2 to computer 1 (the folders have no conflict, as they are new on computer 2 and have no corresponding folder on computer 1). If you try to manually synchronize the two machines after the fork, you may be asking Calibre to generate folders that already exist, and that will fail. What happens when Calibre does that?

(I'm not trying to convince you, or anyone else, not to use dropbox - it's a great setup, provided its used wisely, and you remain sync'd, I'm just trying to figure out the limitations, risks and problems of using it.)