HELP! Metadata possibly corrupted but does not show in "check library"

brainpaladin · 03-03-2017, 09:42 AM

Hi, I'm a frequent and loyal user of Calibre.

Unfortunately, I ran into this problem and do not know what is the cause.

On the bottom left corner (beside Calibre version number), there is a number indicating the number of books in the current library. (Let's call this Number X. )

When I click "library metadata backup status" and "queue all books for backup", the number of remaining books to be backed up is more than the number mentioned above. (Let's call this Number Y. )

My first thought is that:
Number Y takes into account multiple formats. But checking using another library, it is not true. (the other library has books with multiple formats, but when i queue all books for backup, number x = number y.

I have run "check database" multiple times with no problems.

I have also used the calibre portable version (to eliminate possible issues with my local installation) but get the same results.

What could be the reason?

Thank you for any help!

theducks · 03-03-2017, 12:16 PM

According to your Image, you are not looking at a VL or restricted view.

Did you make 2 or more Global (affects many books) changes to the Metadata?
This may be a circular update queue. Change 1 followed by Change 2.

I guess we will need to wait for Kovid

kovidgoyal · 03-03-2017, 12:49 PM

Post the metadata.db file from your calibre library folder and I will take a look.

brainpaladin · 03-03-2017, 01:57 PM

@theducks

Yup that is correct. I don't use virtual libraries (maybe not yet). Thanks for reminding me about making big changes- i forgot to mention: earlier in the day I used duplicate finder plugin to search for binary identical files. I enabled the option to automatically remove binary duplicates (refer to screenshot1).

Thereafter, I am left with lots of book records with no formats. I used "formats:false" to search for these records and deleted all of them. They are not in recycle bin (ie. deleted permanently)

@kovidgoyal

Thanks for your reply. I posted my metadata.db here: https://app.box.com/s/wz031kzyp4wohkx3gonc00shautbblqi

because of the size limitation.

Please take note of what I mentioned above (using duplicate finder plugin).

I figured you are probably very busy being the founder of calibre, so I created a copy of metadata.db and browsed it with DBbrowser to try and figure things out and maybe contribute to solving my own problem.

I noticed 2 oddities:

1. The table "books" contain 88765 entries, while the table "data" contains 92715 entries. (screenshot2)

At first I believe the number next to the version number is the actual number of books, and the number in the pending metadata backup window is erroneous. Now I don't know which to believe..

2. There is a table named "books_plugin_data". (screenshot3)

Seems to be remnants left by plugin duplicate finder? Are they supposed to be there?

Can my metadata file be repaired? What advice do you have?

Btw @theducks and @kovidgoyal:

thank you for taking the time to help me. The support in this forum easily surpasses paid programs, but then again I don't know of any paid program that can rival Calibre.

theducks · 03-03-2017, 02:01 PM

You are welcome.
It is a pleasure to work on problems where details are provided up front.

kovidgoyal · 03-03-2017, 11:51 PM

The number on the status bar is correct. The number of records to be rwitten by the backup can be larger than the number of books, because of the way it works. Basically when anything changes, the book id is stored in the duirty table. So a single book id can appear multiple times. And when the backup file is written all the entries witht hat book id are cleared at once.

kovidgoyal · 03-04-2017, 12:06 AM

And another mechanism for that number to go out of sync (which is probablyt he mechanism in play in your case) is that when deleting books, that table is not updated, instead the deleted ids are simply dropped when the time to back them up comes. So since you deleted lots of book records the dirtied table will contain book ids that have been deleted but that the backup service has not got to as yet.

brainpaladin · 03-04-2017, 12:35 AM

Thanks kovid. So does that mean i need run the backup once. After that, my metadata file should be clean, and running the backup a second time would show both numbers to be the same?

kovidgoyal · 03-04-2017, 01:03 AM

You dont need to do anything, it will run automatically. Once it has completed, if you start it again, the numbers will match.

brainpaladin · 03-04-2017, 01:18 AM

I see, thank you for your help! So I just need to leave the program open with that library active?

kovidgoyal · 03-04-2017, 02:10 AM

yes. though you dont have to, it will just happen in the background using normal usage.

brainpaladin · 03-04-2017, 02:31 AM

I see I see, thank you so much for your help!

03-03-2017, 09:42 AM	#1
brainpaladin Junior Member Posts: 5 Karma: 10 Join Date: Mar 2017 Device: none	HELP! Metadata possibly corrupted but does not show in "check library" Hi, I'm a frequent and loyal user of Calibre. Unfortunately, I ran into this problem and do not know what is the cause. On the bottom left corner (beside Calibre version number), there is a number indicating the number of books in the current library. (Let's call this Number X. ) When I click "library metadata backup status" and "queue all books for backup", the number of remaining books to be backed up is more than the number mentioned above. (Let's call this Number Y. ) My first thought is that: Number Y takes into account multiple formats. But checking using another library, it is not true. (the other library has books with multiple formats, but when i queue all books for backup, number x = number y. I have run "check database" multiple times with no problems. I have also used the calibre portable version (to eliminate possible issues with my local installation) but get the same results. What could be the reason? Thank you for any help! Attached Thumbnails Last edited by brainpaladin; 03-03-2017 at 10:02 AM.

03-03-2017, 01:57 PM	#4
brainpaladin Junior Member Posts: 5 Karma: 10 Join Date: Mar 2017 Device: none	@theducks Yup that is correct. I don't use virtual libraries (maybe not yet). Thanks for reminding me about making big changes- i forgot to mention: earlier in the day I used duplicate finder plugin to search for binary identical files. I enabled the option to automatically remove binary duplicates (refer to screenshot1). Thereafter, I am left with lots of book records with no formats. I used "formats:false" to search for these records and deleted all of them. They are not in recycle bin (ie. deleted permanently) @kovidgoyal Thanks for your reply. I posted my metadata.db here: https://app.box.com/s/wz031kzyp4wohkx3gonc00shautbblqi because of the size limitation. Please take note of what I mentioned above (using duplicate finder plugin). I figured you are probably very busy being the founder of calibre, so I created a copy of metadata.db and browsed it with DBbrowser to try and figure things out and maybe contribute to solving my own problem. I noticed 2 oddities: 1. The table "books" contain 88765 entries, while the table "data" contains 92715 entries. (screenshot2) At first I believe the number next to the version number is the actual number of books, and the number in the pending metadata backup window is erroneous. Now I don't know which to believe.. 2. There is a table named "books_plugin_data". (screenshot3) Seems to be remnants left by plugin duplicate finder? Are they supposed to be there? Can my metadata file be repaired? What advice do you have? Btw @theducks and @kovidgoyal: thank you for taking the time to help me. The support in this forum easily surpasses paid programs, but then again I don't know of any paid program that can rival Calibre. Attached Thumbnails

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Bugs "Sanity Check epub" and "Failed Release date" on V 0.91	qkiazd	Sigil	5	12-03-2015 04:01 AM
How to fix "Extra Titles" in Database in " Check Library - Problems Found	millan	Library Management	33	07-16-2015 07:12 AM
Select multiple items in "Check library results" window	Pepin33	Library Management	2	08-01-2012 11:10 AM
Save Log in "Check Library-Problems Found"	BetterRed	Calibre	1	07-25-2012 05:47 PM
"Locked" 505 will not show in eBook Library	SoCal Bob	Sony Reader	1	10-16-2007 12:57 PM

03-03-2017, 12:16 PM	#2
theducks Well trained by Cats Posts: 30,957 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	According to your Image, you are not looking at a VL or restricted view. Did you make 2 or more Global (affects many books) changes to the Metadata? This may be a circular update queue. Change 1 followed by Change 2. I guess we will need to wait for Kovid

03-03-2017, 12:49 PM	#3
kovidgoyal creator of calibre Posts: 45,259 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Post the metadata.db file from your calibre library folder and I will take a look.

03-03-2017, 02:01 PM	#5
theducks Well trained by Cats Posts: 30,957 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	You are welcome. It is a pleasure to work on problems where details are provided up front.

03-03-2017, 11:51 PM	#6
kovidgoyal creator of calibre Posts: 45,259 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The number on the status bar is correct. The number of records to be rwitten by the backup can be larger than the number of books, because of the way it works. Basically when anything changes, the book id is stored in the duirty table. So a single book id can appear multiple times. And when the backup file is written all the entries witht hat book id are cleared at once.

03-04-2017, 12:06 AM	#7
kovidgoyal creator of calibre Posts: 45,259 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	And another mechanism for that number to go out of sync (which is probablyt he mechanism in play in your case) is that when deleting books, that table is not updated, instead the deleted ids are simply dropped when the time to back them up comes. So since you deleted lots of book records the dirtied table will contain book ids that have been deleted but that the backup service has not got to as yet.

03-04-2017, 12:35 AM	#8
brainpaladin Junior Member Posts: 5 Karma: 10 Join Date: Mar 2017 Device: none	Thanks kovid. So does that mean i need run the backup once. After that, my metadata file should be clean, and running the backup a second time would show both numbers to be the same?

03-04-2017, 01:03 AM	#9
kovidgoyal creator of calibre Posts: 45,259 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You dont need to do anything, it will run automatically. Once it has completed, if you start it again, the numbers will match.

03-04-2017, 01:18 AM	#10
brainpaladin Junior Member Posts: 5 Karma: 10 Join Date: Mar 2017 Device: none	I see, thank you for your help! So I just need to leave the program open with that library active?

03-04-2017, 02:10 AM	#11
kovidgoyal creator of calibre Posts: 45,259 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	yes. though you dont have to, it will just happen in the background using normal usage.

03-04-2017, 02:31 AM	#12
brainpaladin Junior Member Posts: 5 Karma: 10 Join Date: Mar 2017 Device: none	I see I see, thank you so much for your help!

Advert

Advert