Disappearing Books

[Adoby]

As long as no restores was done, time capsule shouldn't delete or bork anything.

[madeleinelevy]

Any other ideas?

[Sabardeyn]

Since you asked, this is a completely wild guess...

You said the Library was stored on an SSD. I know that some of the early SSDs had an issue with file read/writes/deletes because of the way an SSD handles things. For instance, it would mark a file deleted and then wait a variable amount of time before actually removing the file. This timing delay was causing issues for lots of SSD owners and the system was improved in the next generation of SSD products; it's something built into the hardware (BIOS equivalent), I believe.

So perhaps there is some kind of issue here between either the Mac OS/SSD/Calibre? Or you've got an older SSD with the timing issue?

I would suggest starting by finding out the make & model of your SSD and checking online if there are any known issues with it specifically. As well as checking for your version of the Mac OS having SSD errors (in general). If that doesn't result in any viable avenues for further research, continue reading.


To verify it's an SSD related issue, test whether this error continues to occur on a 2nd Library on another drive (different make & model SSD, HDD, USB Flashdrive, internal/external, etc).

I don't recall you mentioning how quickly you saw errors, or a possible trigger for the problem. So the testing time might be substantial (months), as you use this 2nd Library normally. The more/longer you test things, the greater the chance you'll encounter the trigger and track the problem down.

Use the 2nd Library as normal, ignoring the 1st Library completely. Do all your normal work in the 2nd Library. Back up the 2nd Library daily so you don't lose any work (2-3 copies of backups in case errors don't appear for a day or so). Check daily (before and after doing any edits?) to see if the errors you saw with your 1st Library are occurring with your 2nd Library. If you do see errors, try to think what you did to the ebook that has the error. By checking daily you can spot the error quickly and stand a better chance at remembering what you did that might have triggered things.

I realize I'm suggesting a long and complicated process, but as I said, it is the only thing I can think of. Perhaps someone else has a better way to handle this testing?

[Sabardeyn]

Sorry, but looking back over the posts in this thread, something else occurred to me. You say different ebook formats are affected, but not which ones specifically.

Maybe this is an antivirus issue? I know that older PDFs can contain offensive scripting (ie, macro viruses, like MS Office used to experience years ago). While it is unlikely that multiple formats had a virus, it's still something to check. (Unless it was multiple formats of one ebook.)

[Inferno]

May be a dumb question but did you happen to run a duplicate file check and forget to exclude your library directory?

[BetterRed]

Originally Posted by Inferno

May be a dumb question but did you happen to run a duplicate file check and forget to exclude your library directory?

Equally dumb question - but why would you do that?

BR

[Inferno]

BetterRed, I sometimes run a dup file check on my external. I have a really bad habit of backing up too much junk and not looking at it for years so I compare my various external drives to find the duplicates. I usually only "check for duplicates" not check and delete.

[madeleinelevy]

Will do the SSD homework, just need to find the time to set my mind to it.

Time Capsule does it's own thing, and when I back up on my external hard drive, first I erase the old Calibre file present in it, so no chance of duplicates.

No antivirus when the books disappeared. Installed one yesterday, and no virus was found.

[BetterRed]

Originally Posted by madeleinelevy

Will do the SSD homework, just need to find the time to set my mind to it.

Time Capsule does it's own thing, and when I back up on my external hard drive, first I erase the old Calibre file present in it, so no chance of duplicates.

A calibre library is a folder of folders of folders and many files - not a single file. Can you elaborate of what you do.

No antivirus when the books disappeared. Installed one yesterday, and no virus was found.

@madeleinelevy - As I understand Time Capsule it inter-operates with Apple's Time Machine. And I'm pretty sure Time Machine can be set up to synchronise continuously.

In that scenario Time Capsule/Machine would essentially be doing the same on your local network as services such as Dropbox, OneDrive etc can do across the public wide area network. The 'official' recommendation is to pause those services whilst calibre is in use, restart them when done with calibre and don't restart calibre until they've finished synchronising the data.

I suggest you adopt that recommendation for your Time Capsule/Machine installation for a while to see what happens.

I doubt its the SSD, those problems with some early models would probably be manifesting with other data, not just calibre libraries, I assume you have things other than ebook libraries on it

BR

[madeleinelevy]

There are no known issues with the make and model of my SSD card.

I understand the concept of working from a different drive. However there are more issues to be considered. One is that I don't have an internal hard drive and I would have to use an external one. Another would be that, well, what would we try to do here? If the aim is to duplicate the same conditions with a different drive, then I would have to make my Time Capsule back up the external hard drive rather than my computer. Considering that a few months elapsed between the first and the second lose of data, it make me a bit reluctant to go down this road.

So, for the moment I will check weekly for missing formats and make two backups. Time Capsule and Calibre will never run simultaneously again.

I will report back if it happens again under these new conditions.
Thanks everyone.

[madeleinelevy]

Oh, and the Calibre file instead of folder was an obvious blunder.

[maybedog]

I've had this happen, too, and I've come up with several causes, most user error. One reason is what I think is a bug but is probably considered user error:

When I combine two books, Calibre doesn't check if the book is still in the directory. So if I select the book first that is a record without a corresponding book and then the other, Calibre just deletes the second format. [Example: I've lost the book already. The empty book folder is still there. I recover the book and just readd it. Now I have two records. I select the first one and the second one and choose combine and delete. Instead of making sure there is at least one copy, Calibre will delete the second copy so I still am left without a copy. I have to be sure to select the second one, first.] Since Calibre is supposed to *combine*, I expected Calibre to check and make sure all the formats were there. The reason I did it this way was because the original record had all the revised metadata. If I choose the second one first, then I lose much of the revised metadata as it is overwritten by the original metadata. To get around this, I have to manually add the book to the right folder. But the problem is that if the book is a different file size, like I've modified it externally to remove unused media files, the metadata isn't updated and Calibre has the wrong information.

The other two things I've come up with so far are that I was using a cloud drive to back up my files. This ran in the background and somehow deleted the files while syncing. However, this has happened with two different cloud storage sites which isn't supposed to happen and doesn't happen with any other kind of file, so I wonder if there's some bug in Calibre. I don't know what it could be, though.

The other cause is when I update metadata. Sometimes Calibre can't delete the old directory. So I might see both a "J.E. Doe" and "J. E. Doe" folders which are hard to distinguish at first glance. If I have other authors with initials starting with "J", I may miss one of these in the list. Or I switch the authors and the book is then moved to the folder for the author that is now listed first. The book is still there, just looks like it's missing when I look under the wrong name.

[theducks]

Is it a bug, when you do something (leave the cloud syncing, run on a NAS....) when the user guide/FAQ for a program says don't?

Calibre was designed to run on the typical, home desktop/laptop, not on some data center SQL server.

Calibre does multiple transactions that must happen in sequence.
Running other applications that may block/delay the sequence is asking for problems that could be avoided.

[itimpi]

Originally Posted by maybedog

When I combine two books, Calibre doesn't check if the book is still in the directory. So if I select the book first that is a record without a corresponding book and then the other, Calibre just deletes the second format. [Example: I've lost the book already. The empty book folder is still there. I recover the book and just readd it. Now I have two records. I select the first one and the second one and choose combine and delete. Instead of making sure there is at least one copy, Calibre will delete the second copy so I still am left without a copy. I have to be sure to select the second one, first.] Since Calibre is supposed to *combine*, I expected Calibre to check and make sure all the formats were there. The reason I did it this way was because the original record had all the revised metadata. If I choose the second one first, then I lose much of the revised metadata as it is overwritten by the original metadata. To get around this, I have to manually add the book to the right folder. But the problem is that if the book is a different file size, like I've modified it externally to remove unused media files, the metadata isn't updated and Calibre has the wrong information

I regularly combine book entries and have never lost a book format regardless of the order. Not sure why you should be seeing a problem.

The other two things I've come up with so far are that I was using a cloud drive to back up my files. This ran in the background and somehow deleted the files while syncing. However, this has happened with two different cloud storage sites which isn't supposed to happen and doesn't happen with any other kind of file, so I wonder if there's some bug in Calibre. I don't know what it could be, though.

You do not mention the cloud services? For instance this WILL happen if you use GoogleDrive. It is a known deficiency in the way GoogleDrive handles folder names that have numbers in brackets at the end of folder names (which Calibre uses).

[Rbneader]

I've also had Calibre randomly delete books on me. Wasn't using any cloud services, just a internal HD. It was very frustrating. I wasn't able to pin down the precise cause, and no one was able to fix the issue.

I now import new books into Calibre, fix the metadata, and export them immediately to my book storage folders. So far this is easy to manage (despite having several thousand books) and reliable. I save fiction by author and nonfiction by subject.

ETA: Were you copying books through the Calibre tools? The one time I was able to catch it in action the problem seemed to happen when I was moving books between Calibre Libraries. That was only 1 of the 10+ books I lost, though.