Calibre doesn't see books, randomly deletes them

[Rbneader]

Hello!

I'm having a very strange problem and would love some help.

Over the past few weeks, I've noticed that Calibre has
1) Missed some books that are still in the library folders. The Calibre program doesn't see them, but they do exist in the Calibre Library folder.

and separately (maybe it's related, IDK)

2) has started randomly deleting books.

I'm using Calibre Portable if that helps.

I can't figure out how to replicate either of these issues, and I'm leery of trying because this has already deleted 5 of the No. 1 Ladies Detective Agency books while I wasn't looking.

Anyone have any ideas on how to prevent this from happening in the future? At this point I think I'm going to have to redownload all the missing books, which is a pain but doable. I definitely don't want to worry about this happening again in the future though. Is Calibre subject to come kind of corruption, or ID confusion problem?

[itimpi]

Never heard of such behaviour!

You are not by any chance using a cloud service to store your Calibre library? Some of them (in particular GoogleDrive) can interfere with Calibre.

[Rbneader]

No, this is all on my local machine.

I'm completely baffled by this too. I'm in the middle of exporting my books up and setting up an alternate Calibre installation to try and stop whatever is happening. But how this happened is still a headscratcher. I mean, I thought Calibre uses the file structure as a database, so how did it not see the folders that were there? And where did my books go? Was there some kind of ID confusion and Calibre auto-merged things?

[itimpi]

Calibre will only see files/folders that are referenced in its metadata.db database. You can run the Calibre Library Maintenance function to identify mismatches between the database and the underlying files in the Calibre library folder.

[Rbneader]

Originally Posted by itimpi

Calibre will only see files/folders that are referenced in its metadata.db database. You can run the Calibre Library Maintenance function to identify mismatches between the database and the underlying files in the Calibre library folder.

Ahah! This gives a lengthy list of 'Extra Titles', which sounds like problem #1. There doesn't seem to be any way to fix them except for deletion, though. Or re-importing.

[BetterRed]

Originally Posted by Rbneader

I thought Calibre uses the file structure as a database, so how did it not see the folders that were there?

Aaah, Calibre does NOT use the file structure as its database. I'm wondering if you've been changing the files and folders via your file manager.

Calibre uses the file system as a place to put books, much like a bricks & mortar library uses its shelves. When your librarian wants to see if the library has a book, whether its out on loan, where it is in the library etc, then reference is made to the library index/catalogue

The <library>/<author>/<title> folders you see are the equivalent of the library shelves, the metadata.db file in the <library> folder is the equivalent of the catalogue/index - they are closely coupled via the Calibre GUI code.

There is one 'golden rule' with Calibre, which is, don't make DIRECT changes to the names of folders or files within a library, or remove or add any folders or files to a library via your file manager or via the command line etc.

The Calibre GUI provides rich set of features (some would say too rich) to change library & author names, book titles etc, add additional formats directly or via conversion etc etc. Ordinarily there is no need to visit the library folders but if you do then just follow the 'golden rule' and all will be fine.

If you digress from the 'golden rule' then things will go wrong and the library wont function properly. Its like if people went into a bricks & mortar library, and took books without checking them out, moved books around on the shelves, swapped covers, put extraneous books on the shelves.

Did you watch the Calibre video - https://www.mobileread.com/forums/sho...d.php?t=118596 - if not then I recommend it.

BR

[Rbneader]

Originally Posted by BetterRed

Aaah, Calibre does NOT use the file structure as its database. I'm wondering if you've been changing the files and folders via your file manager.

Nope.

[theducks]

Internal HD or
Removable Media? Disable caching (a way to improve performance, but risk data). The DB data and the folder names need to stay in lock-step

[BetterRed]

Originally Posted by Rbneader

Is Calibre subject to come kind of corruption, or ID confusion problem?

Not that I know of, and I can't recall anyone reporting random disappearing books

If you open a book format file in an editor program, and that editor program does NOT lock the book format file and then whilst editing said book you change the author or title then that can cause the database and the folder names to get out of lock step

IIRC Sigil does/did not lock the epubs being edited. The Irfanview image editor is another program that does not lock the files it has open.

Word locks the files it is editing, and Calibre will throw an exception if it detects the lock when you try to change author or title. PDF xChnage also locks files it has open which Calibre detects

Whenever I've found errors in the second stage of the Check Library (Extra Titles etc), then they've always attributable to me changing an author or title whilst editing a format or cover file with a non-locking editor or me changing or moving things in the library folders.

I don't get many of these errors since I trained myself to avoid the pitfalls, such as getting the author and book titles 100% correct as soon as the book arrives before doing anything else.

BR

[Rbneader]

This is happening again. It happened when I chose the 'Copy and delete' option to move books from one library to another. Don't know why it worked for 50 books and not the 51st, but one of the books that I know I had is missing. I didn't edit the book in Sigil or open it in another program.

At this point I think I need to stop relying on Calibre for library management, because it's deleted at least 10 of my books. And those are just the ones I've caught!

ETA: The library check doesn't find any errors. The book file and metadata are both gone. The only reason I found this was because I put the book in question on my ereader 2 days ago, before copying it to the new library, so when I looked for it in my Calibre library and didn't see it I knew there was a problem.

Anyone have any ideas on how to find other books Calibre deleted?

[BetterRed]

Originally Posted by Rbneader

Anyone have any ideas on how to find other books Calibre deleted?

Try Recycle bin/Rubbish tin/Trash can, or a find file by name - on windows, agent ransack is pretty good.


BR

[Adoby]

It is overwhelmingly more likely that it is some form of user error that caused calibre to loose track of books, than it is a bug or error in calibre that is the cause.

Somehow files in the calibre library has been updated, renamed, restored or otherwise changed in a way that made the database metadata.db out of sync with the books stored in the library. The paths and the names stored in metadata.db don't match the actual files. This is discovered only when calibre attempts to do something with the books. So there may very well be a very long time span between the error that caused the problem and the discovery of the problem. That is why it is good to run the database maintenance tools now and then, especially after the library has been updated or changed using any other tools than the GUI.

Possible causes are partial restores of backups, overwriting one library with another. Perhaps a restore of an old metadata.db without replacement of the books. Or a partial restore of book files from backups. Copying the library between different operating systems and/or filesystems that have different case sensitivity. Or using some utilities that updated files or paths in the library.

When calibre is backed up and restored it should be handled as a single monolithic entity. Never partially restored or restored over an existing library.

The problems should hopefully be easy to fix by running the calibre maintenance tools, using the GUI to copy the entire library to a new library, and manually add the books left behind in the old library to the new library. But metadata may be lost.

Please note that the problem will most likely come back again and again, if the assumption is that calibre was the cause. The same user error will then most likely be repeated again. So do try to figure out the actual cause.

[theducks]

Synching the Library folder while Calibre is active.

Multiple users/sessions connected to the Library

Multiple OS used on the same Library (file name handling differences)

Using a Network drive for the Library. (Some folk get away with it. Others don't)

[Rbneader]

Originally Posted by Adoby

It is overwhelmingly more likely that it is some form of user error that caused calibre to loose track of books, than it is a bug or error in calibre that is the cause.

Somehow files in the calibre library has been updated, renamed, restored or otherwise changed in a way that made the database metadata.db out of sync with the books stored in the library. The paths and the names stored in metadata.db don't match the actual files. This is discovered only when calibre attempts to do something with the books. So there may very well be a very long time span between the error that caused the problem and the discovery of the problem. That is why it is good to run the database maintenance tools now and then, especially after the library has been updated or changed using any other tools than the GUI.

Possible causes are partial restores of backups, overwriting one library with another. Perhaps a restore of an old metadata.db without replacement of the books. Or a partial restore of book files from backups. Copying the library between different operating systems and/or filesystems that have different case sensitivity. Or using some utilities that updated files or paths in the library.

When calibre is backed up and restored it should be handled as a single monolithic entity. Never partially restored or restored over an existing library.

The problems should hopefully be easy to fix by running the calibre maintenance tools, using the GUI to copy the entire library to a new library, and manually add the books left behind in the old library to the new library. But metadata may be lost.

Please note that the problem will most likely come back again and again, if the assumption is that calibre was the cause. The same user error will then most likely be repeated again. So do try to figure out the actual cause.

You didn't read my post.

It's not that Calibre lost the filepaths. It's that Calibre has lost the files themselves. The book file is gone. I know how to fix Calibre losing the filepaths. What I don't know is how to stop Calibre from unpredictably deleting the book files themselves.

Also, I ran all the Library Check tools, which found nothing wrong.

The book was there in Library A, I picked 'Copy to Library B, delete after moving', then looked for it the next day and it wasn't in any of my libraries. So I'm pretty sure it was something Calibre did, probably when it copied the book over.

The most logical idea seems to me to be that somehow the copying went wrong, Calibre didn't realize that, and then deleted the copy from Library A like normal. This would fit the observed behavior so far. I'm not sure how to reproduce it, though. The book in question was one of many books that was copied over at the same time, and the rest seem to have made it through alright.

Until this is figured out I'm setting up a backup folder structure that will be my actual library. It's less convenient, but a library that unpredictably deletes files is a library I can't rely on. I'm also setting up a whole new portable Calibre installation, in case it was just something about my old installation.

[Rbneader]

Originally Posted by theducks

Synching the Library folder while Calibre is active.

Multiple users/sessions connected to the Library

Multiple OS used on the same Library (file name handling differences)

Using a Network drive for the Library. (Some folk get away with it. Others don't)

The last 3 don't apply here. Is the first item talking about synching to Dropbox/another online backup site, or about something else?