Corrupt Metadata Library

[Adam KH]

I have a problem with the metadatabase on one of my libraries. I use multiple libraries and “switch” those libraries when I need to process the books in each library.

This has not caused a problem until recently when the response time on metadata opening and update slowed to a crawl with one library.

I have converted all the libraries bar one back to to Portable Calbre which has been fine.

The problem library contains 45,000 books. I run a library check but it returns no errors. I recently imported 5,000 books from into a new library and that resulted in approximately 200 books with out any metadata (I used the import of folders and subfolders which should contain the book, cover and OPF file.

Is there are way of resolving this as importing into Calibre portable will probably take 2 or 3 days with questionable results.

All my books are contained on a USB drive and backed up to a NAS.

Thanks

[theducks]

Please say the Calibre has never connected to the NAS copy.
That can cause corruption. Not MIGHT, but WHEN.

As to moving (working) Libraries from one drive to another. That is correct (x)COPY using the OS. Xcopy is run from a command prompt and tends to run faster)

Rules:
1) the destination Library folder is empty (or only has the 2 original files that get overwritten)
2) Calibre (server too) is shut down

[BetterRed]

Have you run any checks on the external usb drive - could be a flaky drive, cable or interface.

BR

[Adam KH]

Originally Posted by theducks

Please say the Calibre has never connected to the NAS copy.
That can cause corruption. Not MIGHT, but WHEN.

As to moving (working) Libraries from one drive to another. That is correct (x)COPY using the OS. Xcopy is run from a command prompt and tends to run faster)

Rules:
1) the destination Library folder is empty (or only has the 2 original files that get overwritten)
2) Calibre (server too) is shut down

No Calibre is not connected to the NAS Copy i9t is ony connected to the Library held on a USB Drive.

I use Goodsynch to copy the Library from the USB drive to the NAS. I only do this when Calibre is closed.

[Adam KH]

Yesterday I:

Converted all the other Libraries to Calibre portable. I had previously been using Calibre Portable for all Libraries.

I restored the NAS backup of the Large Library (using Goodsynch) to the USB Drive and ran that Library from Calibre on the PC.

I did a Check Library from Calibre and now errors were found (which was the same result I had from a Library Check on 18 August).

I did however note that 1 book added on 18 August did not have the correct Metadata but on 18 August I had deleted that book via Calibre).

My concern now is:

1. Are there orphan books in the Library Folder that are not in the metadata library?

2. How can I identify those orphan books?

The Check Library did not identify any orphan books.

The Library performance is still very slow.

[BetterRed]

Originally Posted by Adam KH

Yesterday I:

Converted all the other Libraries to Calibre portable. I had previously been using Calibre Portable for all Libraries.

What does this 'convert' mean?

The version 6.3 calibre.exe files in my c:\program files\calibre 2 and in d:\calibre\portables\version6\calibre are identical, as is everything else. The essential difference is that portable expects the configuration data and libraries to be within the install folder. The calibre-portable.exe program is a wrapper that reflects that setup to calibre.exe.

I often access the same libraries from installed and portable calibre by dropping symlinks to the library folders into the portable install folder.

Originally Posted by Adam KH

I restored the NAS backup of the Large Library (using Goodsynch) to the USB Drive and ran that Library from Calibre on the PC.

I did a Check Library from Calibre and now errors were found (which was the same result I had from a Library Check on 18 August).

I did however note that 1 book added on 18 August did not have the correct Metadata but on 18 August I had deleted that book via Calibre).

My concern now is:

1. Are there orphan books in the Library Folder that are not in the metadata library?

2. How can I identify those orphan books?

The Check Library did not identify any orphan books.

The Library performance is still very slow.

FTR - I also use Goodsynch.

What does that last sentence mean - slow at what? Scrolling the list of books, viewing a book, updating metadata?

Do you have another drive to which you can copy the Large Library?

If you copy the Large Library\metadata.db file to an empty folder on your system drive you could have calibre access that folder as a library - that might give you a clue as whether the problem is database or device related.

Oh, and how is the USB drive formatted?

BR

[Adam KH]

Originally Posted by BetterRed

Have you run any checks on the external usb drive - could be a flaky drive, cable or interface.

BR

Yes, I use Hard Disk Sentinel 5.61 Pro to monitor all my drives.
I ran a short self test with no errors found. I did not run an extended self test because it is a 4TB drive and that would take 10 hours. A seek time test showed an average of 18ms. The SMART indicators show no warnings.

[Adam KH]

Originally Posted by BetterRed

What does this 'convert' mean?

.....

FTR - I also use Goodsynch.

What does that last sentence mean - slow at what? Scrolling the list of books, viewing a book, updating metadata?

Do you have another drive to which you can copy the Large Library?

If you copy the Large Library\metadata.db file to an empty folder on your system drive you could have calibre access that folder as a library - that might give you a clue as whether the problem is database or device related.

Oh, and how is the USB drive formatted?

BR

The external drive is a Samsung 4TB drive formatted to NTFS. The library in question is 674GB. I only have 432GB available on my secondary 1TB internal drive.

The USB Drive is set for "Quick Removal" which disables write caching on the disk.

It is slow in scrolling and opening the Metadata and especially slow at saving the metadata.

I do have an alternate USB drive. I normally backup to a secondary USB drive every two months. Back up to my QNAP NAS occurs every time I add new books or update a book's metadata. Once per week I back up to a Synology NAS. I have OCD on losing data!

All backups are done using Goodsynch.

Tonight I will first change the USB cable and if that makes no difference I will try the Secondary USB back up. I will also carry out tests on the USB port. I will then try a different USB port.

The secondary backup was last carried out two months ago which is well before I encountered performance issues. If there is a change in performance it would then point to a failing USB drive.

[BetterRed]

Originally Posted by Adam KH

/
It is slow in scrolling and opening the Metadata and especially slow at saving the metadata.

That suggests it's a database related issue. When calibre is running the library metadata is memory resident, writes of the book folder metadata.opf (backup) files are done in a separate low priority thread (lazy writes) - if you do a bulk change that affects a 1,000 books those writes won't impact performance.

Do you have any custom columns that use file system template functions, such as formats_modtimes, formats_paths etc.

Did you try copying the Large Library database to an empty folder on another drive and then access that library via calibre - yes, it will work without the author and title folders.

BR

[Adam KH]

Originally Posted by BetterRed

That suggests it's a database related issue. ....

BR

I copied 5,000 books to another folder (Authors A-C) and imported into a new database using "Add from Folders and Sub-folders". That resulted in at least 80 (if not more) without updated metadata. They came in with the date of import.

I select one, switched back to the original Library, searched for it and it correctly showed the metadata for the book. On opening the containing folder for the book there was no OPF file just the cover and the epub book.

I checked a second book and again no OPF file.

I need to find a resolution for this or find books which do not have an OPF file in their folder.

[Adam KH]

I checked the number of OPF files in the library and this was 41,157 but the metadata say that there are 41,265 books. So I am missing 108 OPF files.

[BetterRed]

backup_metadata

The primary purpose of the metadata.opf files in the book title folders is to enable a corrupted database to be restored. They can also be used when books are added from a folder that was/is a calibre library folder.

Missing OPF file in the Book folders will not effect performance.

BR

[Comfy.n]

Is FTS indexing enabled?

[Adam KH]

Originally Posted by Comfy.n

Is FTS indexing enabled?

Dont understand

[Comfy.n]

see attached screenshot:

FTS (Full text search) indexing, even in slow speed, can cause a general slow response, depending on the system. IIRC, FTS indexing is on by default in Calibre recent versions. (edit: it isn't)

So you could try this: disable FTS indexing. (it's the FT icon on the left side of the gear icon in the search bar)