Calibre and conflict of individual books OPF files

[Josieb1]

I have my Calibre in Dropbox. It's accesable by three different PCs. I've had quite a few conflicts in the main metadata files (among the five that sit in the main database) and I've always deleted the conflicts and Calibre always looks okay. I do have 6500 books in it so it's a large database, but generally everything looks okay.

I've started to notice that individual books have conflicts in their OPF files. I only spotted these when I right clicked to view the book folder in Windows, usually to use Send to Kindle. Where I see them I delete them.

Is it possible to run some kind of integrity check where Calibre can flagged up all the individual books with conflict OPF files? Or is there some other way of clearing up the database?

[BetterRed]

@Josieb1 - the conflicts are almost certainly due to the fact that you run the calibre library manager and the dropbox client concurrently - see FAQ - I am getting errors with my calibre library on a networked drive/NAS?

Consider using the calibre Content Server to make your books available on other computers. Run calibre on a single computer and access it via the Content Server or a Remote Desktop solution.

If you must share the actual library, use a file syncing tool like DropBox or rsync instead of a networked drive. If you are using a file-syncing tool it is essential that you make sure that both calibre and the file syncing tool do not try to access the calibre library at the same time. In other words, do not run the file syncing tool and calibre at the same time.

Even with these tools there is danger of data corruption/loss, so only do this if you are willing to live with that risk.

Basically that means suspending Dropbox before starting calibre. Before resuming Dropbox I recommend running the calibre-db --backup_metadata command, that will flush any pending opf writes.

Aside - a search of the manual for 'dropbox' puts the FAQ I've cited as the first item, and a thread search for 'dropbox' in the MR Calibre forum should find a number of threads with the same advice.

BR

[Josieb1]

Thanks for posting BetterRed. Unfortunately I don't understand most of what you said.

I wanted my Calibre library in Dropbox specifically so it could be accessed and updated using whatever pc I happened to have turned on. I've moved to a Mac as my W7 pc has flaky USB ports and is running slow/hot, and while the Mac is good i I do tend to use Hubbys pc most as I rarely use my Mac during the week. I don't know what a networked drive is really, I certainly don't have one, all our pics are independent they just all access the same Dropbox folder.

I did a search of my Calibre folder and it threw up nearly 1400 conflicted opf files going back to July last year. I deleted them all and, touch wood, calibre seems fine. I'm taking a back up now. I'll keep an eye on it. If it becomes a big issue I'll have to move calibre to my Mac and just keep a back up in Dropbox.

[Josieb1]

Originally Posted by BetterRed

Aside - a search of the manual for 'dropbox' puts the FAQ I've cited as the first item, and a thread search for 'dropbox' in the MR Calibre forum should find a number of threads with the same advice.

BR

I'm going away shortly but when I get back I'll try and read up on the threads. As you said it's all about risk. I need to decide if it's acceptable.

[BetterRed]

Originally Posted by Josieb1

I deleted them all and, touch wood, calibre seems fine. I'm taking a back up now. I'll keep an eye on it. If it becomes a big issue I'll have to move calibre to my Mac and just keep a back up in Dropbox.

Josieb1 - by deleting all the opf files you have created a library which is irrecoverable should its database become corrupted/unusable. Running calibre and dropbox concurrently increases the chance of the database becoming corrupted/unusable. If dropbox syncs its copies of the opf files back to the PC then in the event of database corruption and recovery you will in all likelihood lose some of your metadata.

A Calibre library is not a single file, it is a folder containing a database (metadata.db) and a hierarchy of associated sub folders and contents, it must be managed as a whole - no cherry picking allowed. Calibre knows how its libraries work - Dropbox hasn't clue.

Regarding your pictures, assuming you're not using an application such as iMatch or Extensis to manage them, then comparisons with Calibre are debatable at best.

As I wrote before - to minimise risk of trashing your Dropbox located calibre library adopt the following process:

Suspend Dropbox before starting calibre. Before resuming Dropbox run calibre-db --backup_metadata command to flush any pending opf writes. Best way to that is via a script appropriate to the operating system you're using - something along these lines

If Suspend Dropbox is OK
    If Run Calibre until exit is OK
         If Run calibredb --backup_metadata is OK 
              Resume Dropbox

BR

[Josieb1]

Thanks again for replying. As I said before I don't understand most of what you are saying. I don't know how or when to run the code you've quoted and I'm not sure why you mentioned pictures either. I understand that my Calibre library is a sum of its parts and not a single entity. I haven't deleted all the OPF files only the conflicts. I'll take a look again tomorrow and decide how best to proceed.

[BetterRed]

I mentioned your pictures and Dropbox because you did, with the implication that you don't have have any similar problems with your pictures.

That 'code' is just a set of ' formal instructions' that echoes what I previously wrote.

The 'conflicts' are an artefact of dropbox. To avoid them don't run calibre and dropbox at the same time - it's as simple as that.

IIRC you suspend/resume Dropbox via its right click menu. Doing it in a script would just save you the effort of doing each step manually - and having to remember them.

Anyway I'm just repeating myself so ...

BR

[theducks]

Josie1

If you have deleted all the metadata.OPF (used by DB as backup)

Right-click: Library Maintenance: Backup Status: Click Queue all book...

This take time (~3 sec per book), so let it run ( same place as the button, shows progress)

[eschwartz]

It's pretty simple. To elaborate on what BetterRed said:

1. Start Dropbox. Wait for it to fully sync.
2. Pause Dropbox.
3. Start calibre. Do whatever you wanted to do with it.
4. Close calibre.
5. Un-pause Dropbox. Wait for it to fully sync.

...

You can make sure that happens automatically with one action, by using a Windows batch file or OSX shellscript. And using the aforementioned `calibredb --backup_metadata` command can provide additional security and peace of mind.
But that assumes you are comfortable with scripting.
And clearly you are not, which means it was rather counterproductive for BetterRed scare you off with overly-complicated instructions in the first place.

Rest assured -- it is by no means required. Just follow the manual, GUI instructions I provided above.

[eschwartz]

Additionally, you can resolve your current mess by going to Library Maintenance ==> Check Library and checking for extraneous files.

If there are extraneous OPF files, just delete them and then use Backup Status ==> Queue all book as theducks suggested to regenerate the necessary OPF files.
If there are other extraneous files, they might be something you want to keep, so double-check before deleting.

[Josieb1]

Originally Posted by BetterRed

I mentioned your pictures and Dropbox because you did, with the implication that you don't have have any similar problems with your pictures.

That 'code' is just a set of ' formal instructions' that echoes what I previously wrote.

The 'conflicts' are an artefact of dropbox. To avoid them don't run calibre and dropbox at the same time - it's as simple as that.

IIRC you suspend/resume Dropbox via its right click menu. Doing it in a script would just save you the effort of doing each step manually - and having to remember them.

Anyway I'm just repeating myself so ...

BR

Ahh, sorry about that, I went back and read my second post. The word pics is a typo, I meant PCs. Yes I do have pictures in Dropbox but we have no conflicts with them. Apologies for the confusion.

Yes I think I understand what you are saying about suspending Dropbox while I use Calibre to add or edit items, you mean pause the sync. At least I think you did. I will have to remember to do it as I wouldn't have a clue where or hoe to load a script, plus I have both windows 7 and a MAC to deal with.

I'm going to let all PCs sync to Dropbox (and back down again) and see what I have. If all looks okay I'm going to count myself lucky and not worry about conflicts anymore, just try and only use one of to maintain Dropbox and the other two for enquirys only.

Thanks again for your help and advice

[Josieb1]

Originally Posted by theducks

Josie1

If you have deleted all the metadata.OPF (used by DB as backup)

Right-click: Library Maintenance: Backup Status: Click Queue all book...

This take time (~3 sec per book), so let it run ( same place as the button, shows progress)

I only deleted conflicted OPFs, the latest of those was dated 02/06. From what I've seen when I noticed them on individual books they all had a OPF not listed as a conflict with the 08/06 date. I need to check all the entries but from what I saw when I went into Calibre afterwords everything looked okay.

If not I've got a back up pre removal of conflicts so I can re sync that one. Thank you for helping

[Josieb1]

Originally Posted by eschwartz

Additionally, you can resolve your current mess by going to Library Maintenance ==> Check Library and checking for extraneous files.

If there are extraneous OPF files, just delete them and then use Backup Status ==> Queue all book as theducks suggested to regenerate the necessary OPF files.
If there are other extraneous files, they might be something you want to keep, so double-check before deleting.

Thank you for helping . I'll do that later today

[Josieb1]

Originally Posted by eschwartz

It's pretty simple. To elaborate on what BetterRed said:

1. Start Dropbox. Wait for it to fully sync.
2. Pause Dropbox.
3. Start calibre. Do whatever you wanted to do with it.
4. Close calibre.
5. Un-pause Dropbox. Wait for it to fully sync.

...

You can make sure that happens automatically with one action, by using a Windows batch file or OSX shellscript. And using the aforementioned `calibredb --backup_metadata` command can provide additional security and peace of mind.
But that assumes you are comfortable with scripting.
And clearly you are not, which means it was rather counterproductive for BetterRed scare you off with overly-complicated instructions in the first place.

Rest assured -- it is by no means required. Just follow the manual, GUI instructions I provided above.

Code and me don't get on, which is ironic as I system test for a living . But yes I'm not comfortable with scripting generally, and petrified on a new MAC.

I understand the pause syncing bit, it's remembering to do it, but it does make sense. Thanks again.