Corrupted calibre library metadata - Need help

[Dailyread]

I was browsing books on my connected device (kobo) and clicked "Refresh cached metadata" after that I suppose something went awry (Inside calibre) and my on device (computer) library shortened from 248 to merely 39 books

then I ran "Library check" and it lists all the missing books as deletable



Tried restoring the library, but no use. All the "Missing folders" have calibre metadata file in them, is there any way to rebuild the database using it? I don't want to go through all the books and assign tags, status and new columns, and would like to avoid it if possible.

Also tried file recovery (thinking calibre should have deleted or edited the old metadata file) but that also resulted in nothing, there were no old metadata.db files in the directory

• The library was (is) on my local disk
• There is no Virtual library filter on my calibre library
• I did not touch any folder except for deleting a folder on my kobo (After deleting the book from my calibre library)

Is there any way to get the old library back, I successfully created another library and have all the books in it, but the old metadata and columns are gone.

Any help is appreciated. Thanks

[kovidgoyal]

Start a new empty library, re-create whatever custom columns you had using the same names, then right click the add books button and choose add books from folders select the old library folder and then select one book per folder.

[Dailyread]

Originally Posted by kovidgoyal

Start a new empty library, re-create whatever custom columns you had using the same names, then right click the add books button and choose add books from folders select the old library folder and then select one book per folder.

Hey, legend! Thanks a lot (For replying and for the software). This one solved my issue with some caveats of missing covers and data in general, and out of sync books with the kobo, but saved me a great deal of headache nonetheless.

Is there anything that I could do to prevent this in the future? (Apart from keeping weekly backups of the whole library?)

[theducks]

Originally Posted by Dailyread

Hey, legend! Thanks a lot (For replying and for the software). This one solved my issue with some caveats of missing covers and data in general, and out of sync books with the kobo, but saved me a great deal of headache nonetheless.

Is there anything that I could do to prevent this in the future? (Apart from keeping weekly backups of the whole library?)

Discover what is being done to YOUR library that is doing the data damage.
We can put Hardware down low on the list as that usually damages/loses files.

Most common(s):
2-way data sync Only mirror (1 way, a shadow of the Main only)

Network sharing / NAS. Various things need to happen in lock step. Networks can insert delays. 2 Users on a share results in Keystone Kops data. Who writes what fails. Calibre is Single User access. If you need remote access, look at the using content server. that is the way to go for remotes.
Using the OS 'copy' or Explorer to make any changes at the file system level.

[BetterRed]

Originally Posted by Dailyread

Hey, legend! Thanks a lot (For replying and for the software). This one solved my issue with some caveats of missing covers and data in general, and out of sync books with the kobo, but saved me a great deal of headache nonetheless.

Is there anything that I could do to prevent this in the future? (Apart from keeping weekly backups of the whole library?)

Run Check library and fix any problems weekly, I've never had a problem that wasn't preventable or my blunder.

And then do an incremental one-way backup of your librar(y/ies) and your configuration data ==>> Comparison of file synchronization software - Wikipedia - rsynch, freefilesync are popular amongst the calibre community.

BR

[Dailyread]

Originally Posted by theducks

Discover what is being done to YOUR library that is doing the data damage.
We can put Hardware down low on the list as that usually damages/loses files.

Most common(s):
2-way data sync Only mirror (1 way, a shadow of the Main only)

Network sharing / NAS. Various things need to happen in lock step. Networks can insert delays. 2 Users on a share results in Keystone Kops data. Who writes what fails. Calibre is Single User access. If you need remote access, look at the using content server. that is the way to go for remotes.
Using the OS 'copy' or Explorer to make any changes at the file system level.

you are on point, the hardware is not the issue, The lib is on a brand-new HDD

While I did not get the whole picture of the later paragraph, as far as I understood, you are saying that I might have done something that copied the eReader data to my calibre library, resulting in the metadata corruption?

About the network sharing, I only use wired connection to transfer ebooks from calibre, never used wireless connection, and I am a solo user.

The most probable oopsie that I remember doing is clicking this button. I hadn't used it in the past, but yesterday I deleted two books from the eReader (First through calibre and then from the explorer) and clicked this button. It was after this that I noticed missing books


Another major culprit could be that I use koReader and connect to the eReader while in KoReader and not in nickle (Connecting while in nickle interface) when it syncs changes after disconnecting. I just read that calibre relies on nickle's database to maintain its own database for the eReader. Perhaps that lead to database issue when I clicked that "Update cached metadata on the device" button.

Other than these two, I never interact with the library directly, just through calibre.

[theducks]

You have me confused.
You said the LIBRARY is being corrupted, but you point to the DEVICE (Koreader) icon and the menu that update THE DEVICE metadata.
That should never change THE LIBRARY.

[Dailyread]

Originally Posted by theducks

You have me confused.
You said the LIBRARY is being corrupted, but you point to the DEVICE (Koreader) icon and the menu that update THE DEVICE metadata.
That should never change THE LIBRARY.

Yes, the Library (on my pc) got corrupted, this is the last button I clicked before it happened. It could have been for any reason (idk, not much experience with calibre, just importing, editing metadata and exporting), but this is the last thing I remember doing before noticing that books were missing.
It was fine when I was in the library view, then switched to the device view and clicked this button. I noticed that all the books on my eReader (in the device view) were missing that "in library" checkmark. So whatever went wrong, definitely went wrong when I was in the device view and not in Library view

I don't know what caused the metadata corruption for both, the books (The OPF metadata inside the books' folders) and Calibre metadata.db.
When I imported my books from the previous library (Corrupted one) to my new library, the books were missing covers, tags and custom columns (A combination of these, some had missing covers, some were missing tags, and so on).

Right now, I have rebuilt my library in calibre and cleared everything on my kobo as well (If it had anything to do with either Kobo or something in Calibre), and set up FreeFileSync for my calibre library folder to avoid this scenario (whatever be the cause).

Hopefully that does it.

[theducks]

AFAIK Switching to/from device view does NOT write to the Library.
That leads me to thinking you have a filtered view.

Always pay attention to the status line. It offers clues as to what (filtering) is going on. and what count will be affected by further actions.
How to read. It varies by what mode (the positions also move).
the words indicate what the numbers represent

The first part is the version of Calibre that is running (stays put )
### books, is the Library VIEW size

# selected, is how many some later actions will affect <<< important when doing any bulk change.

##### total size is the Library when there is any filter. If the filter has no matches, the status is 0 of #### total