Calibre Database rebuild

[mitch13]

Thanks Kovid and BookJunkieLI. Will certainly try your option BookJunkieLI

[mitch13]

Hi BookJunkieLI,
Tried what you suggested but books remaining (did not move but copied) were the total old library.

Still had the same problems - I thought I followed your suggestion exactly but obviously I must have made a mistake. Any Suggestions?

mitch13

[BetterRed]

Originally Posted by mitch13

Hi BookJunkieLI,
Tried what you suggested but books remaining (did not move but copied) were the total old library.

Still had the same problems - I thought I followed your suggestion exactly but obviously I must have made a mistake. Any Suggestions?

@mitch13 - Did you use calibre or your file manager to do the move.

BookJunkieLI's suggestion requires that you use calibre to do the move, not the file manager.

To move books with calibre, select all the books in the calibre book list (Ctrl/A), right click selected list, select Copy to library-><target library> (delete after copy) e.g:



The books left behind are the 'stranded' books, i.e. books which are not in the database. These must be added manually to the 'new library'.

The golden rule of calibre library management is -- Don't use your file manager in the library folder to do anything but look, so no deleting, no adding, no renaming, no moving or copying of folders or files in the library.

BR

[mitch13]

Hi BookJunkieL1,
Tried that but did not work. Ended up with library with only about 4 books in it. The old library was the same! This processing took well over a day.
Since went back to zero books and "added" all books from my library. Another day and a bit and this then had several thousand books in the library but not in the database.

Getting very frustrated!!

Decide to accept this new library at this stage but am desperate to be able to identify the actual books that are not in the db. By this I don't just mean their title but the actual book in the Calibre library. As there (I am sure) will be duplicates but need to know which one to delete?

mitch13

[BookJunkieLI]

Originally Posted by mitch13

Hi BookJunkieL1,
Tried that but did not work. Ended up with library with only about 4 books in it. The old library was the same! This processing took well over a day.
Since went back to zero books and "added" all books from my library. Another day and a bit and this then had several thousand books in the library but not in the database.

Getting very frustrated!!

Decide to accept this new library at this stage but am desperate to be able to identify the actual books that are not in the db. By this I don't just mean their title but the actual book in the Calibre library. As there (I am sure) will be duplicates but need to know which one to delete?

mitch13

I can completely understand that this is frustrating. I've gone through it myself. Both when books suddenly 'disappeared' from the library and when something got so screwed up ::coughDropboxcough:: that Calibre wouldn't even open my library at all.

Now, before going any further there's a couple of things I'm going to have you check.

First, do you have the original back-up that you restored from? If so, don't touch it. Keep it completely separate.

Second, how did you add the books to Calibre this last time around? Did you use the Add Books icon or did you use File Explorer and physically move the file folders into Calibre's filesystem? If you used Add Books, great. If you used File Explorer then, yeah, Calibre is going to have all sorts of issues recognizing the books. You will need to start over an add the books using the Add Books icon - work in small batches so you can make sure things are going correctly.

Third, assuming you used Add Books, go to the Search box and type in the following and click Go:

formats:false

Any books listed do not have a format ie an ebook file attached to them. Calibre can recognize the metadata but if you run a Check Library it will not find an ebook listed in the database and will return an error.

If there is no format listed then you need to go to the file folders that you added the books from and check to see what files are listed. A good book will have at least an ebook file (epub, mobi, pdf, etc), and possibly an opf file and a cover.jpg file. If all that is listed is an opf file then all Calibre could add is the metadata information to the library. It won't find the ebook in the database because there was no ebook format to add/import in the first place.

And this brings me to why I asked if you still have the original backup from before this whole mess began. You need to check the files there to see if epub/mobi/pdf/etc files are available or if it's just listing opf files. If there is an ebook file listed you can re-add them and Calibre will recognize it as a valid book. If your original backup is just listing opf files then you had an issue even before the original failure and you will need to re-download those books from wherever you originally got them from.

[mitch13]

BookJunkieL1,
Thanks for your prompt reply.
1, I do have an original backup.
2. Books added via Calibre - never use explorer on calibre library.
3. Used the format:false on current library and it showed around 12 items. A number where mp3 audio books so OK. There were a few books (around 6) but of no importance so deleted them via Calibre.

Checked my backup library and, although a large number of duplicates, the random check of books had a mobi and a metadata.opf file. Even one that was listed in the "Extra Titles" problem list had both mobi and opf!!

Really confusing

[BookJunkieLI]

Okay, that is actually a step in the right direction.

Make a backup of your library. Even as messed up as it currently is it's still a good idea to make a backup. Keep it separate from the original backup.

Next, when you run Check Library what are the error messages that you get? Is it all Extra Title errors or are there others?

Starting with just the Extra Title errors, pick one and using Explorer navigate to the corresponding file folders within the Calibre library filesytem. NOTE: It's best to do this with Calibre closed. What files are listed in the folder? Is it just an ebook file and an opf file? Are there multiple copies of the ebook file?
--If it's just one ebook file and an opf file move the entire book folder out of the Calibre library folder. Remember where you moved it to.
---Next, in Calibre use Add Book from single directory to add the book that you just moved out of Calibre. What happens? Was it added back in properly?

If there are multiple ebook and/or metadata.opf files things are going to be a bit more complicated and time consuming. Same if you are also getting Extra Book Formats or Unknown Files in Books error messages.

Go through the Extra Titles with a just a single ebook file first and see what happens. We can then tackle the others.

[mitch13]

Many thanks for that. Will begin working on that.
When I run library check the majority are Extra Title errors around 22,000! I know there are a lot of duplicates now. Hard to believe this library after many years work was almost error free!
Around 50 Extra Authors, 3x Missing formats and 3x Missing cover files.

As suggested I will try one book first and follow it through

regards

mitch13

[mitch13]

Hi BookJunkieL1 - I am back again.
Did as you said and after a lot of work have solved all the Extra Title errors. Also solved all the Missing Covers errors.
Am now left with around 100 Missing Book formats.
Not quite sure how I solve these yet but at least the incredible number of errors is down considerably.
Many thanks for your help and suggestions. Any clues on the way to solve the Missing formats would be good.
Actually the missing Covers was part of a similar problem - those without covers in the list also had no formats. What I cannot understand is the there are no books "without formats"showing in my Calibre library! One book I looked at has one entry in the library but three in the Calibre Library listing!

Cheers

mitch13

[mitch13]

Hello BookJunkieL1,

Pleased to advise, reasons unknown, my library is now all clear of errors.

After I fixed the Missing covers virtually all problems disappeared on a reboot. Minor ones left now also fixed.

Many thanks for your help

mitch13

[BookJunkieLI]

I'm glad to hear things sorted themselves out for you. Sometimes solving one error fixes others, particularly when it comes to covers.

Now that it's all working properly make sure you back-up on a regular basis and consider a backup scheme that doesn't just overwrite itself. Nothing worse than thinking you have a good backup only to discover the issue started a week or a month ago.

[BetterRed]

And consider using a backup scheme that will backup up only what's changed since last backup - known as incremental and/or differential backups. Depending on library size, volatility and backup media it can be the difference between a taking a few minutes and a few hours.

BR