MobileRead Forums - View Single Post - Why does Calibre keep losing books, and how can I stop it?

Mex5150 · 01-08-2016, 08:54 AM

Hi,

OK, may have solved the problem, but first there seems to be a lot of confusion on how I actually use this set-up.

Calibre is run from a script that stops DropBox from syncing until after I quit out of Calibre. This is on the Linux machine, while I am doing this the ONLY thing that accesses the library file structure it Calibre itself, and anything that calibre opens itself. When I quit Calibre the script reinstates the DropBox sync.

I have two android devices that access a couple of the libraries, a rooted Nook touchlight, and (very occasionally) my phone, both access the files from the local copy of the libraries on DropBox (i.e. not the one living on the ext4 drive Calibre accesses). Nothing is moved, changed, or deleted. I don't think the files are locked, but even if they are this is irrelevant as A) Calibre is not running when I access them and B) even if it was I wouldn't be using it to do anything to that specific book anyway. Also the problem happens in libraries that are not touched outside of Calibre, so I can see no way this can be the cause of the issue.

The misplaced/missing books are not limited to *.pdf files, but they are the easiest to notice. How I work with these is:
* Do a cover/metadata update,
* Open the file to see it it worth converting (i.e. if it's text or just pictures of text),
* Close the PDF viewer,
* If the conversion is worthwhile, convert it (this is where I notice it's missing if it is),
* Move onto the next book

It is important to note that Calibre does not write to the database while the pdf is open, this should avoid any filelock situations.

Anyway back to the issue.

I tried moving the libraries to a never synced dir that is not connected to DropBox in any way (not even one way sync), and yup, it happened again. So I think that rules out both the external access and the sync being the problem.

Now the interesting bit. I had to rebuild the system due to something else completely unrelated to software problems. Since then (although I haven't had a huge amount of time to try things out) nothing has disappeared. It's tricky to say if the problem is resolved completely or not as it was intermittent anyway. This would suggest though something connected to the local install rather than the environment it's running in. However, I seem to recall an issue with another python based application which would place the blame squarely at the feet of the python set-up.

Hopefully this is the end of the problem, I will report back either when something disappears again or after a good while has passed and it looks like it really is fixed.

~Mex

01-08-2016, 08:54 AM	#25
Mex5150 Banned Posts: 23 Karma: 7476 Join Date: Feb 2013 Location: To the left of your body, to the right of your mind, lost in a riddle. Device: Android phone, Nook Touch Simple Glowlight	Hi, OK, may have solved the problem, but first there seems to be a lot of confusion on how I actually use this set-up. Calibre is run from a script that stops DropBox from syncing until after I quit out of Calibre. This is on the Linux machine, while I am doing this the ONLY thing that accesses the library file structure it Calibre itself, and anything that calibre opens itself. When I quit Calibre the script reinstates the DropBox sync. I have two android devices that access a couple of the libraries, a rooted Nook touchlight, and (very occasionally) my phone, both access the files from the local copy of the libraries on DropBox (i.e. not the one living on the ext4 drive Calibre accesses). Nothing is moved, changed, or deleted. I don't think the files are locked, but even if they are this is irrelevant as A) Calibre is not running when I access them and B) even if it was I wouldn't be using it to do anything to that specific book anyway. Also the problem happens in libraries that are not touched outside of Calibre, so I can see no way this can be the cause of the issue. The misplaced/missing books are not limited to .pdf files, but they are the easiest to notice. How I work with these is: Do a cover/metadata update, * Open the file to see it it worth converting (i.e. if it's text or just pictures of text), * Close the PDF viewer, * If the conversion is worthwhile, convert it (this is where I notice it's missing if it is), * Move onto the next book It is important to note that Calibre does not write to the database while the pdf is open, this should avoid any filelock situations. Anyway back to the issue. I tried moving the libraries to a never synced dir that is not connected to DropBox in any way (not even one way sync), and yup, it happened again. So I think that rules out both the external access and the sync being the problem. Now the interesting bit. I had to rebuild the system due to something else completely unrelated to software problems. Since then (although I haven't had a huge amount of time to try things out) nothing has disappeared. It's tricky to say if the problem is resolved completely or not as it was intermittent anyway. This would suggest though something connected to the local install rather than the environment it's running in. However, I seem to recall an issue with another python based application which would place the blame squarely at the feet of the python set-up. Hopefully this is the end of the problem, I will report back either when something disappears again or after a good while has passed and it looks like it really is fixed. ~Mex