Dropbox and Calibre metadata

DavidTC · 02-04-2016, 10:48 PM

First, let me preface this by saying I know all about 'one computer at a time' rule. I only have one instance of Calibre, running on Windows 10. (All the other Dropbox access is via read-only mobile apps.) There are no changes happening that are not from that one instance of Calibre.

However, often, when updating metadata, or when adding a new fanfic, I often get an error saying that Calibre could not write metadata. This error goes away if I pause Dropbox. I'm pretty certain this is because Dropbox is currently uploading the .opf file, because Dropbox's 'recently changed' list always has metadata.opf at the top after it happens.

I...do not understand what is going on here. Surely Dropbox is only *reading* the file when uploading, and will let others update it. If not, I'd be having problems in all sorts of other apps when saving files.

Also, this is some amazing timing to keep happening so often on such small files (Which really should be instantly read and written.), and never happening once with my metadata.db! That's the thing that is really confusing me...these are tiny 20k files, they should be locked by Calibre, instantly written to disk, and then unlocked. And then locked by Dropbox (Or actually not!), instantly pulled into memory, and unlocked, and then uploaded.

And while it doesn't happen on *every* metadata change, but but it isn't some once-in-a-blue-moon race condition where all the stars misalign perfectly...I can make it happen by changing the metadata on a dozen books.

I saw a mention the other day when trying to figure this out that Calibre does some sort of check to see if someone else is using a file before it writes to it. I think there might be something weird going on there.

DavidTC · 02-04-2016, 11:01 PM

Actually, the thing I *really* don't understand is how this is causing a Calibre problem.

Even if there was some sort of conflict, Dropbox would have absolutely no reason to have a random file locked *before* it was changed.

So what must be happening is that Calibre writes to the file, *pauses*, and then tries to write again but see that Dropbox has the file open for reading. I...can't figure out why Calibre would be doing that on a metadata change.

eschwartz · 02-04-2016, 11:02 PM

It is because of file locking.

For what it's worth, although I usually pause Dropbox while calibre is running, I sometimes forget.

I have never gotten those errors.
I use linux, so the files aren't locked (Dropbox can read them while calibre writes them).

Coincidence? I think not...

AnemicOak · 02-04-2016, 11:06 PM

EDIT: Nevermind. I see eschwartz has got you covered.

DavidTC · 02-05-2016, 11:02 AM

Yes, I *know* it's because of file locking.

I am asking two things:

1) Why is Calibre opening the file, replacing the contents, closing it (At which point Dropbox steps in and tries to read it) and then Calibre tries to open it *again* a split second later? Why does Calibre not just keep it open while changing it, which would result in Dropbox waiting?

Like I said, this isn't happening because I repeatedly change the metadata of a single file, and I'm getting caught because my previous edit hasn't been uploaded yet. I would understand *that* happening, and be recommending solution #2 below. But that *doesn't* happen.

It's happening when I edit the metadata of a bunch of books at once. Which means, as far as I can guess, that Calibre is opening and closing metadata.opf from 1-20 (in order?), and dropbox starts reading them...and then Calibre opens *all those files again*, presumably in order again, and thus trips over Dropbox, which had made it to file number 8 or whatever.

Why is opening files again?! What is this second pass that trips over anything that is reading recently modified files?!

2) Considering this weird double-update system that Calibre seems to want to use of editing files, if Calibre can't open a file, why can't Calibre just *move that update request to the end up of the queue* instead of throwing an error, and thus try again a second later?

Or, heck, write out a metadata.opf.new file and move it over a second later?

DavidTC · 02-05-2016, 11:18 AM

Now I'm wondering if this might be something to do with the fanfic plugin, because that is actually where it seems to happen most often, usually when adding or updating using the plugin.

Perhaps Calibre is changing something, and then the plugin is.

02-04-2016, 10:48 PM	#1
DavidTC Connoisseur Posts: 77 Karma: 10 Join Date: Sep 2011 Device: Nook, Boox C67ML	Dropbox and Calibre metadata First, let me preface this by saying I know all about 'one computer at a time' rule. I only have one instance of Calibre, running on Windows 10. (All the other Dropbox access is via read-only mobile apps.) There are no changes happening that are not from that one instance of Calibre. However, often, when updating metadata, or when adding a new fanfic, I often get an error saying that Calibre could not write metadata. This error goes away if I pause Dropbox. I'm pretty certain this is because Dropbox is currently uploading the .opf file, because Dropbox's 'recently changed' list always has metadata.opf at the top after it happens. I...do not understand what is going on here. Surely Dropbox is only reading the file when uploading, and will let others update it. If not, I'd be having problems in all sorts of other apps when saving files. Also, this is some amazing timing to keep happening so often on such small files (Which really should be instantly read and written.), and never happening once with my metadata.db! That's the thing that is really confusing me...these are tiny 20k files, they should be locked by Calibre, instantly written to disk, and then unlocked. And then locked by Dropbox (Or actually not!), instantly pulled into memory, and unlocked, and then uploaded. And while it doesn't happen on every metadata change, but but it isn't some once-in-a-blue-moon race condition where all the stars misalign perfectly...I can make it happen by changing the metadata on a dozen books. I saw a mention the other day when trying to figure this out that Calibre does some sort of check to see if someone else is using a file before it writes to it. I think there might be something weird going on there.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre Cloud Pro and Metadata.db Sync Dropbox	da_jane	Related Tools	24	02-16-2014 07:19 PM
Calibre and Dropbox	deannah	Library Management	1	07-30-2013 05:45 PM
Minor Dropbox Metadata mishap	evans	Related Tools	1	02-17-2012 10:03 PM
How to use Dropbox with Calibre	BAD18	Related Tools	5	11-09-2011 08:52 AM
Using dropbox and Calibre	totaltech	Related Tools	7	04-29-2011 02:28 PM

02-04-2016, 11:01 PM	#2
DavidTC Connoisseur Posts: 77 Karma: 10 Join Date: Sep 2011 Device: Nook, Boox C67ML	Actually, the thing I really don't understand is how this is causing a Calibre problem. Even if there was some sort of conflict, Dropbox would have absolutely no reason to have a random file locked before it was changed. So what must be happening is that Calibre writes to the file, pauses, and then tries to write again but see that Dropbox has the file open for reading. I...can't figure out why Calibre would be doing that on a metadata change.

02-04-2016, 11:02 PM	#3
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	It is because of file locking. For what it's worth, although I usually pause Dropbox while calibre is running, I sometimes forget. I have never gotten those errors. I use linux, so the files aren't locked (Dropbox can read them while calibre writes them). Coincidence? I think not...

02-04-2016, 11:06 PM	#4
AnemicOak Bookaholic Posts: 14,391 Karma: 54969924 Join Date: Oct 2007 Location: Minnesota Device: iPad Mini 4, AuraHD, iPhone XR +	EDIT: Nevermind. I see eschwartz has got you covered.

02-05-2016, 11:02 AM	#5
DavidTC Connoisseur Posts: 77 Karma: 10 Join Date: Sep 2011 Device: Nook, Boox C67ML	Yes, I know it's because of file locking. I am asking two things: 1) Why is Calibre opening the file, replacing the contents, closing it (At which point Dropbox steps in and tries to read it) and then Calibre tries to open it again a split second later? Why does Calibre not just keep it open while changing it, which would result in Dropbox waiting? Like I said, this isn't happening because I repeatedly change the metadata of a single file, and I'm getting caught because my previous edit hasn't been uploaded yet. I would understand that happening, and be recommending solution #2 below. But that doesn't happen. It's happening when I edit the metadata of a bunch of books at once. Which means, as far as I can guess, that Calibre is opening and closing metadata.opf from 1-20 (in order?), and dropbox starts reading them...and then Calibre opens all those files again, presumably in order again, and thus trips over Dropbox, which had made it to file number 8 or whatever. Why is opening files again?! What is this second pass that trips over anything that is reading recently modified files?! 2) Considering this weird double-update system that Calibre seems to want to use of editing files, if Calibre can't open a file, why can't Calibre just move that update request to the end up of the queue instead of throwing an error, and thus try again a second later? Or, heck, write out a metadata.opf.new file and move it over a second later?

02-05-2016, 11:18 AM	#6
DavidTC Connoisseur Posts: 77 Karma: 10 Join Date: Sep 2011 Device: Nook, Boox C67ML	Now I'm wondering if this might be something to do with the fanfic plugin, because that is actually where it seems to happen most often, usually when adding or updating using the plugin. Perhaps Calibre is changing something, and then the plugin is.

Advert

Advert