Why does Calibre keep losing books, and how can I stop it?

[Mex5150]

Hi Guys,

This has been happening on and off for years, but I've reached my frustration level with it.

Calibre frequently loses books, while retaining the meta data. It doesn't happen with every book, but it does happen enough to be annoying. Here's a recent example: I'm updating and converting a lot of books at the moment, this is something that just happened
1) Open the pdf to see if it's suitable for conversion to ePub (ie consists of actual text rather than just pictures of text).
2) Update meta data to get the cover I want etc
3) hit convert book
The problem is, on 3 I'm told the book format doesn't exist, even though I just looked at it seconds ago.

Library Maintenance -> Restore Database takes ages to do nothing.
Library Maintenance -> Check Library tells me there are missing books (which is why I went there in the first place).
If I go and search the HDD I can often (but not always) find the missing book(s), sometimes in a case mismatch, sometimes elsewhere, which means I've got to move them elsewhere then re-import.

The library is not on a network drive (I know this can cause problems).
Although I do on occasion access the files directly from the drive, this is not the problem here (as it also happens to newly imported books that have never been accessed directly).

Any ideas on why this keeps happening, and more importantly how to stop it from happening in the future?

~Mex

[chaley]

Originally Posted by Mex5150

The library is not on a network drive (I know this can cause problems).
Although I do on occasion access the files directly from the drive, this is not the problem here (as it also happens to newly imported books that have never been accessed directly).

Any ideas on why this keeps happening, and more importantly how to stop it from happening in the future?

~Mex

The fact that your library is on a network drive is exactly why it is happening. Many (perhaps most) network file systems are broken in some way. The brokenness usually appears when renaming files and folders, something that calibre does whenever it adds books or changes their titles or authors.

The solutions: live with it or move the library off the NAS.

[aleyx]

Originally Posted by chaley

The fact that your library is on a network drive is exactly why it is happening.

You may have misread. He said:

Originally Posted by Mex5150

The library is not on a network drive (I know this can cause problems).

I don't think it's the network.

What's your OS? Calibre version?

N.

[BookJunkieLI]

Are you using it with Dropbox or Google Drive? Do you have any kind of autosync software running?

[chaley]

@aleyx: my bad. You are right -- I didn't see the "not".

That said, the fact that the OP sees case differences in lost books implies that there is a problem doing renaming. Using a NAS is one reason this can happen. Another reason is a mismatch between the file system and the operating system. Problems like this will arise if you run windows using a case-sensitive file system, or if you run linux with a non case-sensitive file system, or if you are using a portable drive and moving it between operating systems. I don't have enough experience with macs to speculate whether case sensitivity problems might arise.

Bottom line: be sure that the file system type is native for the OS you are running, and do not move your library storage device between dissimilar operating systems.

[Mex5150]

Originally Posted by chaley

The fact that your library is on a network drive ...

As others have pointed out, I said I was NOT on a networked drive, I mentioned it because I know it can cause problems, easy mistake to make though, as it is often the culprit ;^>

Originally Posted by aleyx

What's your OS? Calibre version?

Sorry, should have said that, Doh! I’m Running Linux (Ubuntu 16.04 LTS to be precise), Calibre is 2.47.0, the drive they are on is formatted to ext4

Originally Posted by BookJunkieLI

Are you using it with Dropbox...

Well, yes and no. They are in a folder that is normally shared on/in DropBox, however I start Calibre with a script that cuts autosync runs Calibre, then restarts autosync when I quit out of Calibre. A habit I got into using Scrivener (that gets very upset with autosync) so anything I use that needs (or may need) a steady non-autosync dive on a drive that is normally shared, I use this script with. So effectively as far as Calibre is concerned, it's a completely normal, non-networked, non-autosynced drive.

I have just tested the start-up script and can confirm autosync is stopped for the duration of Calibre running. So no closer to a solution then ;^<

Originally Posted by chaley

That said, the fact that the OP sees case differences in lost books implies that there is a problem doing renaming.

Sometimes it does do this with renaming (I think I can recall an instance of that happening), but normally it's when I hit convert. It could be that that's doing it, or it could be something before hand that I don't notice until convert tells me the book is missing though.

Using a NAS is one reason this can happen. Another reason is a mismatch between the file system and the operating system. Problems like this will arise if you run windows using a case-sensitive file system, or if you run Linux with a non case-sensitive file system, or if you are using a portable drive and moving it between operating systems. I don't have enough experience with macs to speculate whether case sensitivity problems might arise.

Linux file system accessed by a Linux OS. Permanent non-removable (well not without a screwdriver LOL) drive. I do have several other OS's on the machine, but they are all safely tucked away on virtual machines that don't have access to that particular drive.

Bottom line: be sure that the file system type is native for the OS you are running, and do not move your library storage device between dissimilar operating systems.

When Calibre is not running an android device will occasionally access one of the libraries directly, but I can't see how this would be the culprit as books also vanish on libraries it doesn't have access to.

Don't know about you guys, but I really stumped by it ;^/


~Mex

[JSWolf]

Try moving the library out of the directory that gets synced and stop accessing the libraries directly outside of Calibre. See if your books stop going missing.

[dwig]

Originally Posted by Mex5150

...
1) Open the pdf ...

in what app?

[theducks]

Originally Posted by dwig

in what app?

Why would this matter, as long as the file was not saved (where the name might get tampered with) or deleted from the app?

My normal Calibre (Win & Linux) configuration : (Preferences: Behavior: <no tick next to PDF> ) View uses the OS to supply the PDF viewer. That way, you do not need to even peek inside the library folder

[Mex5150]

Originally Posted by JSWolf

Try moving the library out of the directory that gets synced

I'll give it a go, but as I said as far as Calibre is concerned it's not on a drive that gets synced, all synchronisation is stopped while calibre is running, then restarted after it stops.

and stop accessing the libraries directly outside of Calibre. See if your books stop going missing.

I know many people here consider touching the files with anything other than Calibre the deadliest of all cardinal sins, but as I said, this is not the issue as it happens in libraries that are never accessed outside of Calibre as well as some that are.

Originally Posted by dwig

in what app?

The default (Adobe Reader 9) opened by clicking the link in Calibre, but this is just one of the ways I notice the problem, it doesn't precede every instance of books vanishing, so again, unlikely to be connected.

Due to Christmas/Hogmanay I've not had the time to investigate this any further, but if I ever do resolve the issue, I'll drop back and report.

~Mex

[BetterRed]

@Mex5150 - maybe you could run a 'watch for deletes' monitor on the library tree, see ==>> Monitor file system activity with inotify - probably filter by the format types (PDF, EPUB, MOBI etc), log or pop a message when delete happens

BTW I'm with you on the library folders - I dive in and out of them all the time. But I never add new items (folders or files), or delete or rename existing items. And I've never lost anything, or had Check Library report significant errors.

BR

[JSWolf]

Originally Posted by Mex5150

I'll give it a go, but as I said as far as Calibre is concerned it's not on a drive that gets synced, all synchronisation is stopped while calibre is running, then restarted after it stops.

But it does get synced and if anything is wrong on the other end, your end gets screwed. So really, having your library be in a synced directory is a really really bad idea.

What you can do that would work rather well is make a copy of your library to the synced directory and use FreeFileSync to keep the synced copy in sync with the working copy. That would be the safest way to keep your working library safe. I've been using Calibre a good while and I've never had any eBooks go missing just like that. I do think it's related to your syncing.

[PeterT]

Like Jon, I do a one way sync using AllwaysPro to my local DropBox, and then let DropBox sync as it likes.

[eschwartz]

It is certainly true that the safest way to use calibre with Dropbox/other is to do a one-way mirror into the Dropbox folder.

But it is also (probably) irrelevant.
If you are careful, as the OP has indicated, one-way sync is not needed to prevent Dropbox/calibre editing conflicts. The most likely source of failure. Although extra safeguards are always good.

For the record, I have used calibre a long time without doing a one-way sync and never once had problems.

[JSWolf]

Originally Posted by eschwartz

It is certainly true that the safest way to use calibre with Dropbox/other is to do a one-way mirror into the Dropbox folder.

But it is also (probably) irrelevant.
If you are careful, as the OP has indicated, one-way sync is not needed to prevent Dropbox/calibre editing conflicts. The most likely source of failure. Although extra safeguards are always good.

For the record, I have used calibre a long time without doing a one-way sync and never once had problems.

But, I've never before read of anyone having eBooks vanish from a Calibre library just like that. The only thing different that I can tell given the OP's information is the use of a sync directory. Careful or not, something isn't working properly. So move out of the sync directory and see if books get lost just like that or the problem is solved. It's just easy to use FreeFileSync to make a copy in the sync directory. The only way to try to solve the problem is to try ifferent things and the first thing to try is no working library in a sync directory.