Why does Calibre keep losing books, and how can I stop it?

[theducks]

Some sort of 'Old File Cleanup' coming along?
Be sure to exclude you library path

Rogue A/V scan? (I would expect that log to show the activity or find the files in the Quarantine)

[goldilocks]

Originally Posted by Mex5150

Well, yes and no. They are in a folder that is normally shared on/in DropBox, however I start Calibre with a script that cuts autosync runs Calibre, then restarts autosync when I quit out of Calibre. A habit I got into using Scrivener (that gets very upset with autosync) so anything I use that needs (or may need) a steady non-autosync dive on a drive that is normally shared, I use this script with. So effectively as far as Calibre is concerned, it's a completely normal, non-networked, non-autosynced drive.

I have just tested the start-up script and can confirm autosync is stopped for the duration of Calibre running. So no closer to a solution then ;^<

Originally Posted by Mex5150

When Calibre is not running an android device will occasionally access one of the libraries directly, but I can't see how this would be the culprit as books also vanish on libraries it doesn't have access to.

In reference to the above comments:
When the android device accesses Calibre I assume Dropbox could be syncing?
If so, lets say Dropbox is accessing one of the libraries that is not accessible to the android device ... does this mean it is still safe for the android device to access another Calibre library at the same time? I mean, just because it is a different library, I would think another device accessing Calibre (the same program) at the same time, could mess up the sync with Dropbox and whoops! there goes your eBook into Neverland!
Just so you know ... I am not a technical person and get weird looks about comments I sometimes make.

[BetterRed]

@goldilocks - when calibre is running it is recommended that syncing (whether to a cloud service like Dropbox or a locally via OSX's TimeMachine or FFS's Realtime Sync) be turned off. Which is what Mex5150 has said he does.

But your point about accessing the libraries directly via Android may be relevant.

It implies that a 'program' running on Android is accessing a library on an Ext4 volume mounted on a Ubuntu 16.04 LTS system. If it's to get some files from the library and put them on the Android device and the Android program does a Move rather a Copy - whooshka the file is gone from the library on the Ext4 volume.

Originally Posted by Goldilocks

Just so you know ... I am not a technical person and get weird looks about comments I sometimes make.

Not only out of the mouths of babes and sucklings, but out of the mouths of fools and cheats, we may often get our truest lessons.

1859, Oliver Wendell Holmes, The Professor at the Breakfast Table, ch. 1:

BR

[goldilocks]

Originally Posted by BetterRed

@goldilocks - when calibre is running it is recommended that syncing (whether to a cloud service like Dropbox or a locally via OSX's TimeMachine or FFS's Realtime Sync) be turned off. Which is what Mex5150 has said he does.

BR,

Yes, I know and that is why I tried to point out a different, possible scenario.

But, even with Calibre closed, I would think it would be dangerous to have 2 other programs (Dropbox/Android)? accessing its' data at the same time.

I don't know much about Ubuntu nor about using Android. But I think the last quote is supposed to be a compliment...so, thank-you.

[eschwartz]

On Windows, it would actually be bad for both calibre and another program to access the same file even for read-only purposes.

Though on Ubuntu (or any linux) as long as they aren't both writing to it it should be all right.


Hopefully Android isn't actually modifying or deleting files on another computer it is accessing.

[BetterRed]

Originally Posted by goldilocks

BR,

Yes, I know and that is why I tried to point out a different, possible scenario.

But, even with Calibre closed, I would think it would be dangerous to have 2 other programs (Dropbox/Android)? accessing its' data at the same time.

@goldilocks - Accessing a calibre library (i.e. the folders and files) directly from an android device via an android file manager app (eg ES File Manager) is intrinsically no different to accessing a calibre library from a local file manager - files can be moved from one place to another, after which they will not be where they were.

Originally Posted by goldilocks

I don't know much about Ubuntu nor about using Android. But I think the last quote is supposed to be a compliment...so, thank-you.

It was - meant as compliment with a twist

==================================

Originally Posted by eschwartz

On Windows, it would actually be bad for both calibre and another program to access the same file even for read-only purposes.

Why, I do it all the time, I often have an epub open in calibre viewer whilst editing it Sigil. Granted, I don't do anything in the viewer that would cause it to write to the epub, and I don't expect the viewer to reflect the changes I make in Sigil in real time.

Originally Posted by eschwartz

Hopefully Android isn't actually modifying or deleting files on another computer it is accessing.

It wouldn't be Android itself doing it - it would be a file manager app such as ES File Mangler

We should probably shut up now, and wait for the OP to respond

BR

[kovidgoyal]

@BR: Just FYI both the viewer and the editor do not keep a file open while viewing/editing it. They actually work on an exploded version of the file placed in the temp directory. That is why it works to open the same epub file in multiple calibre programs on windows. This is in contrast to say acrobat reader which keeps a PDF file it is viewing opened and therefore locked.

[BetterRed]

I know, Sigils the same.

Another reason not to use Acrobat Reader Although I suspect other PDF readers may be similar.

BTW: it seems that Office 2010 doesn't hold locks on open files either, I can convert a DOCX that's open in Word that I haven't saved - and then puzzle why the changes I just made in the DOCX aren't in the EPUB I couldn't do that with Office 2007 - calibre would throw an IO error. Trying to get a straight answer from MS on such issues is like extracting hens teeth - no glitz, no glamour - no answers forthcoming.

BR

[kovidgoyal]

PSA: If you use the viewer and editor simultaneously on EPUB files, be sure to turn off the saving of bookmarks in the epub file in the viewer preferences, to make sure that the viewer does not write to the epub file.

[Mex5150]

Hi,

OK, may have solved the problem, but first there seems to be a lot of confusion on how I actually use this set-up.

Calibre is run from a script that stops DropBox from syncing until after I quit out of Calibre. This is on the Linux machine, while I am doing this the ONLY thing that accesses the library file structure it Calibre itself, and anything that calibre opens itself. When I quit Calibre the script reinstates the DropBox sync.

I have two android devices that access a couple of the libraries, a rooted Nook touchlight, and (very occasionally) my phone, both access the files from the local copy of the libraries on DropBox (i.e. not the one living on the ext4 drive Calibre accesses). Nothing is moved, changed, or deleted. I don't think the files are locked, but even if they are this is irrelevant as A) Calibre is not running when I access them and B) even if it was I wouldn't be using it to do anything to that specific book anyway. Also the problem happens in libraries that are not touched outside of Calibre, so I can see no way this can be the cause of the issue.

The misplaced/missing books are not limited to *.pdf files, but they are the easiest to notice. How I work with these is:
* Do a cover/metadata update,
* Open the file to see it it worth converting (i.e. if it's text or just pictures of text),
* Close the PDF viewer,
* If the conversion is worthwhile, convert it (this is where I notice it's missing if it is),
* Move onto the next book

It is important to note that Calibre does not write to the database while the pdf is open, this should avoid any filelock situations.

Anyway back to the issue.

I tried moving the libraries to a never synced dir that is not connected to DropBox in any way (not even one way sync), and yup, it happened again. So I think that rules out both the external access and the sync being the problem.

Now the interesting bit. I had to rebuild the system due to something else completely unrelated to software problems. Since then (although I haven't had a huge amount of time to try things out) nothing has disappeared. It's tricky to say if the problem is resolved completely or not as it was intermittent anyway. This would suggest though something connected to the local install rather than the environment it's running in. However, I seem to recall an issue with another python based application which would place the blame squarely at the feet of the python set-up.

Hopefully this is the end of the problem, I will report back either when something disappears again or after a good while has passed and it looks like it really is fixed.


~Mex

[eschwartz]

I doubt it had anything to do with your system python. You were running v2.47, which was never published to Ubuntu, therefore you must've used the official standalone binaries.


So, gremlins?

That is not a very satisfying answer but I suppose if it's the best we have, then the important thing is that the problem is gone.

[theducks]

I see nothing wrong with those steps.

As long as you do not change the Author or Title metadata, calibre only writes to the DB (and updates metadata.opf in the book folder). Viewing should not cause issues.

Having the Content server active, does not cause Calibre issues.

@ Eswartz (you are more the guru here)
ext4? wasn't there some quirks? (I use ext3)

[eschwartz]

New OS installs default to ext4.

It is stable -- has been for a while now.

...

I use btrfs -- if you are thinking of the delayed allocation controversy, you really shouldn't be asking me to weigh in on ext4.

[BetterRed]

Originally Posted by Mex5150

The misplaced/missing books are not limited to *.pdf files, but they are the easiest to notice. How I work with these is:
* Do a cover/metadata update,
* Open the file to see it it worth converting (i.e. if it's text or just pictures of text),
* Close the PDF viewer,
* If the conversion is worthwhile, convert it (this is where I notice it's missing if it is),
* Move onto the next book

My process with PDF's is v similar - have a look to see if worth converting, mine are often riddled with sidebars, tables, infographics etc which usually means they're not suitable.

But it suggests the 'viewer' had a role to play in the disappearance - not that it did it off its own bat, but that it did something to trigger something else into disappearing the file.

Did your rebuild necessitate recreating the software environment - OS, file system, dropping/replacing daemons etc, etc.

BR

[Mex5150]

Originally Posted by eschwartz

I doubt it had anything to do with your system python. You were running v2.47, which was never published to Ubuntu, therefore you must've used the official standalone binaries.

So, gremlins?

In that case I'm rather confused

That is not a very satisfying answer but I suppose if it's the best we have, then the important thing is that the problem is gone.

I've been on the new set-up for just under a fortnight, and done the remainder of the tidy-up/editing I needed, so far no more vanishing files, so (fingers crossed) it looks like it may be fixed.

Originally Posted by BetterRed

But it suggests the 'viewer' had a role to play in the disappearance - not that it did it off its own bat, but that it did something to trigger something else into disappearing the file.

If this only happened with pdf files I'd agree, but as it also happens occasionally with epub files (I just notice the pdf's quicker as I'm doing something with them at the time rather than noticing later).

No idea what was wrong, I had the same problem over many updates, the system rebuild appears to have fixed things though.


~Mex