HELP! Metadata possibly corrupted but does not show in "check library"

[brainpaladin]

Hi, I'm a frequent and loyal user of Calibre.

Unfortunately, I ran into this problem and do not know what is the cause.

On the bottom left corner (beside Calibre version number), there is a number indicating the number of books in the current library. (Let's call this Number X. )

When I click "library metadata backup status" and "queue all books for backup", the number of remaining books to be backed up is more than the number mentioned above. (Let's call this Number Y. )


My first thought is that:
Number Y takes into account multiple formats. But checking using another library, it is not true. (the other library has books with multiple formats, but when i queue all books for backup, number x = number y.

I have run "check database" multiple times with no problems.

I have also used the calibre portable version (to eliminate possible issues with my local installation) but get the same results.

What could be the reason?



Thank you for any help!

[theducks]

According to your Image, you are not looking at a VL or restricted view.

Did you make 2 or more Global (affects many books) changes to the Metadata?
This may be a circular update queue. Change 1 followed by Change 2.

I guess we will need to wait for Kovid

[kovidgoyal]

Post the metadata.db file from your calibre library folder and I will take a look.

[brainpaladin]

@theducks

Yup that is correct. I don't use virtual libraries (maybe not yet). Thanks for reminding me about making big changes- i forgot to mention: earlier in the day I used duplicate finder plugin to search for binary identical files. I enabled the option to automatically remove binary duplicates (refer to screenshot1).

Thereafter, I am left with lots of book records with no formats. I used "formats:false" to search for these records and deleted all of them. They are not in recycle bin (ie. deleted permanently)



@kovidgoyal

Thanks for your reply. I posted my metadata.db here: https://app.box.com/s/wz031kzyp4wohkx3gonc00shautbblqi

because of the size limitation.


Please take note of what I mentioned above (using duplicate finder plugin).


I figured you are probably very busy being the founder of calibre, so I created a copy of metadata.db and browsed it with DBbrowser to try and figure things out and maybe contribute to solving my own problem.

I noticed 2 oddities:

1. The table "books" contain 88765 entries, while the table "data" contains 92715 entries. (screenshot2)

At first I believe the number next to the version number is the actual number of books, and the number in the pending metadata backup window is erroneous. Now I don't know which to believe..

2. There is a table named "books_plugin_data". (screenshot3)

Seems to be remnants left by plugin duplicate finder? Are they supposed to be there?

Can my metadata file be repaired? What advice do you have?


Btw @theducks and @kovidgoyal:

thank you for taking the time to help me. The support in this forum easily surpasses paid programs, but then again I don't know of any paid program that can rival Calibre.

[theducks]

You are welcome.
It is a pleasure to work on problems where details are provided up front.

[kovidgoyal]

The number on the status bar is correct. The number of records to be rwitten by the backup can be larger than the number of books, because of the way it works. Basically when anything changes, the book id is stored in the duirty table. So a single book id can appear multiple times. And when the backup file is written all the entries witht hat book id are cleared at once.

[kovidgoyal]

And another mechanism for that number to go out of sync (which is probablyt he mechanism in play in your case) is that when deleting books, that table is not updated, instead the deleted ids are simply dropped when the time to back them up comes. So since you deleted lots of book records the dirtied table will contain book ids that have been deleted but that the backup service has not got to as yet.

[brainpaladin]

Thanks kovid. So does that mean i need run the backup once. After that, my metadata file should be clean, and running the backup a second time would show both numbers to be the same?

[kovidgoyal]

You dont need to do anything, it will run automatically. Once it has completed, if you start it again, the numbers will match.

[brainpaladin]

I see, thank you for your help! So I just need to leave the program open with that library active?

[kovidgoyal]

yes. though you dont have to, it will just happen in the background using normal usage.

[brainpaladin]

I see I see, thank you so much for your help!