Restore Database at 100%

[Arro]

Hello, all!

I've an interesting problem on which I was hoping someone might shed some light.

I am running Calibre 2.70.0.0 x64 under Windows 10 on an i3 with 4GB RAM and a (nearly full) 1TB hard disk.

During my early days using Calibre a good number of years ago, not certain how Calibre defined "duplicate", I set it to import duplicates. Consequently, I ended up with a large number of books which in retrospect I must agree really were true duplicates. So, using Calibre or a plugin, I recently asked the program to delete every duplicate except the oldest. This seemed to have worked, but then I noticed that, when attempting to open some books, I was returned an error saying that the book was not available in any format. A quick direct check of the library showed at least one copy of the book was indeed there. The first step using the Calibre database check or a plugin - I'm not sure which for reasons that should become clear - came up with no database errors. The second, more extensive check returned a number of issues, some of which were "fixable" (and were fixed) while others gave the only available option as deleting the offending books. I did not want to delete any books, because at that point I might have had only one copy. Although some books might have had multiple formats, I couldn't be certain. In addition, I was not certain exactly what Calibre was proposing to delete.

Now I must confess to having broken the cardinal rule of Calibre in manipulating the Calibre Library directly.

Having noticed a large number of folders containing only a cover and an opf file with no other file, I programmatically deleted every folder containing only "cover.*" and "metadata.opf". If a parent folder became empty as a result, it was deleted recursively. I had hoped, as well, that this would get rid of some of the truly bizarre author name folders from the 2000 to 2010 decade. It seems to have done so, at least in part, but not entirely.

I expected the folder deletions would cause database issues, but Calibre popped right back up, reporting my library as having been thinned by about 50,000 books. But…I still had the "no available formats" error described above.
At that point I decided to rebuild the database using Calibre. I was aware this was going to take some time. It started roughly 5 days ago and seems to have stopped at 100% on the "Quick Start Guide".

calibre.exe and calibre-parallel.exe are still active processes, but are using only a small amount of memory with neither disk nor CPU activity. No Calibre worker processes are running.

The most pressing issue presently is the always-on-top "Restoring database" dialogue box. It gives me only the option to "Cancel". It also makes it a bit more difficult to report exactly what I did and with which plugins, because I can't retrace from the main Calibre interface. I will update this post when this issue is resolved. In the meantime, my apologies for all imprecision.

In the Calibre Library itself I have two files: "metadata.db", last modified about 30 hours ago and a much larger "metadata_pre_restore.db" last modified at the time Calibre began the database restoration days ago.

I did back up the entire library before I began thinning duplicates, so I can simply revert back to it if all goes amiss.

Does anyone have any suggestions as to what has happened and how to proceed? Is there a calibre.log or equivalent anywhere any longer?

The restore process seems finished. But…….do I press the button?



Thanks to everyone!

[BetterRed]

@Arro - the Check library tool has three phases. Phase one (vacuuming) optimises the library by removing deleted data - I have never know it to fail. Phase two checks the internal integrity of the database - likewise, I have never know it to fail. Phase three checks the database against the author and book folder and their contents - and reports inconsistencies, this where 99.99% of problems show up. I run it once a week, then if there are any inconsistencies they are few in number, and I will probably remember the circumstances of their creation

The easiest solution may be to restore the pre-thinning backup, then run Check Library and save the report. Then change the Add Books duplicate settings and create a new library from the existing library using the Copy to library and delete tool. Then use what's left behind in the old library to resolve the issues in the Check library report. And as a final step run the Find Duplicates optional PI to catch any stray duplicates.

BR

[Arro]

the Check library tool has three phases

Agreed. There seems to be a fourth preliminary phase, but I have no idea what it is doing. It lasts under 5 minutes. Likewise, the first (vacuuming) phase is relatively fast – perhaps 30 minutes.

I believe Calibre is at the end of phase two or three. I am guessing it is at the end of phase three given

1.) the length of time the tool has been running.

2.) the fact that all processes aside from calbre.exe and calibre-parallel.exe have ended and calbre.exe and calibre-parallel.exe are clearly dormant and have been so for some time after having used nearly all of the disk's read/write capacity (97%-99%), about half to two-thirds of available memory, and surprisingly little CPU – perhaps 20%, on average, for a week.

3.) that the tool has stopped at the most recent book added to the library. After a Calibre update, which was done just before thinning, John Schember's "Quick Start Guide" is always added to the main Calibre Library.

4.) the Check Library dialogue box is at exactly 100%.

The above are all my own observations and may be entirely irrelevant. In truth, I have very little knowledge of Calibre's workings, but think the hallmark of a very well-written program is one which runs practically flawlessly "out of the box" and has a GUI which is as intuitive to the user as ever could be expected. This is Calibre including its plugins in a nutshell.

The easiest solution may be to restore the pre-thinning backup

This is good advice and will almost certainly do no harm. It does not tell me what went wrong, though. It also sets the Library back to the pre-thinning, pre-restore state – square one.

What about copying the older "metadata_pre_restore.db" to another location, ending Calibre-associated processes outside of Calibre by using Windows Task Manager or TASKKILL /F and then deleting the old "metadata_pre_restore.db" directly? This will leave the most recently modified "metadata.db" and a "metadata_db_prefs_backup.json" file which was modified just before the database restore, about a week ago. The "_backup" in the file name suggests Calibre has another .json with identical preferences somewhere, so I could probably back up "metadata_db_prefs_backup.json" to be on the safe side and delete it as well.

At this point, nothing associated with Calibre should be running. The only database in the main Calibre Library will be the most recent "metadata.db". I'll then wipe all temp directories, reboot, restart Calibre, and see what happens.

Does anyone see any problems (or absence of any advantages) in the procedure above?

I still have a library backup created before starting the clean-up, so I don't think I can do any permanent damage and might even learn something in the process, as unlikely as my history would suggest this could be...

Cheers!

[BetterRed]

I can't say if your idea will work, and I am not prepared to set up a scenario to test it - maybe someone else will be prepared to do that.

When someone's in a hole I don't throw them a spade to dig a deeper hole, I throw them line and haul them out of the hole. I've given you a solution for cleaning up 'libraries-in-an-dreadful-mess' that has been used to recovered 'messed up libraries' by many users in the past. That's all I am prepared to do.

BR

[Arro]

Originally Posted by BetterRed

I can't say if your idea will work, and I am not prepared to set up a scenario to test it - maybe someone else will be prepared to do that.

When someone's in a hole I don't throw them a spade to dig a deeper hole, I throw them line and haul them out of the hole. I've given you a solution for cleaning up 'libraries-in-an-dreadful-mess' that has been used to recovered 'messed up libraries' by many users in the past. That's all I am prepared to do.

BR

You dwell with wisdom and prudence.

For whatever reason, from beans to nuts, "Restore database" completed stage 3 but did not exit. The most recent "metadata.db" was apparently complete and intact. Calibre detects no errors. "metadata_pre_restore.db" seems to be a backup made by Calibre of Calibre Library's "metadata.db" when "Restore database" commences. In any case, it is very similar to the "metadata.db" included in the backup I made of the Calibre Library before thinning.

Killing Calibre processes and removing "metadata_pre_restore.db" and "metadata_db_prefs_backup.json" from the library caused no damage in this specific scenario at the time I did it and with the system I used.

The method I proposed is unorthodox and I do not recommend anyone else attempt this.

It saved me a week or so, but I still do not know why Restore did not exit as expected - I'm low on disk space, I have a lot of Windows management services turned off and secpol restricts access to others and to various areas of my drive...it really could have been anything.

It worked...this time...but thanks for taking the time to think about the issue and for your cautious advice.

Arro