Libra CorruptError: database disk image is malformed

[Quoth]

Kobo Libra, *622 firmware.

The first time I connected it read all the titles. It only gave an error trying to read the annotations via Kobo Utilities.

I ejected and went to menu for the book and selected Export Highlights (the optional extra menu item that is enabled in .conf). That only had the first 1/4 of the book annotations. Going into the book and looking at annotations showed the last 3/4s of annotations gone.

Then reconnecting to Calibre I can't "communicate" at all. No titles listed.
I have sqliteman and DB browser for SQlite. But I'm only familiar with full SQLserver tools and years ago.

I've made a copy of KoboReader.sqlite and BookReader.sqlite

I searched here but couldn't see a thread on fixing the database.

a) I assume the last 3/4 of annotations are gone, or might some sort of repair work?

b) How is it fixed? There are no other unsaved annotations and all books are on Calibre.

c) I suppose all reading positions would be lost on a nuke it type repair, Not a big problem

calibre, version 4.12.0
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 90, in run
File "site-packages/calibre/gui2/device.py", line 513, in _books
File "site-packages/calibre/devices/kobo/driver.py", line 1958, in books
File "src/cursor.c", line 236, in resetcursor
CorruptError: CorruptError: database disk image is malformed

[Quoth]

I searched with Google rather than Forum search and found
"The database on the device is corrupt. Unless you have a backup you can use, the fix is sign out of the Kobo account on the device and then back in."

Any backup is likely ancient. Something I forgot about!
Thoughts:
A backup likely would be before I annotated/proofed that book anyway and I always import the annotations with Calibre Kobo Utilities, copy& paste as a text file and make a back up that immediately before starting another book.

Edit
I've logged out and logged in and Calibre is doing something. Collections all empty.

I looked at copy of database with DB browser for SQlite and did an export then import.
Except either it's too broken, or I need to edit the text as original is about 150M and "new" version about 92k!

Calibre now working with it, of course no notes.

[Quoth]

Ah, I'll need to restore my .conf file!

[davidfor]

Originally Posted by Quoth

I searched with Google rather than Forum search and found
"The database on the device is corrupt. Unless you have a backup you can use, the fix is sign out of the Kobo account on the device and then back in."

To the best of my knowledge, there is no way to recover the database from the error you are seeing. I believe that will only be seen when the actual table data is corrupted. You can recover from some problems, but, mainly ones related to the indexes or free pages in the file.

Any backup is likely ancient. Something I forgot about!

You can automate it with my Kobo Utilities plugin for whenever the device is connected

Thoughts:
A backup likely would be before I annotated/proofed that book anyway and I always import the annotations with Calibre Kobo Utilities, copy& paste as a text file and make a back up that immediately before starting another book.

Edit
I've logged out and logged in and Calibre is doing something. Collections all empty.

If you are using epubs, then the annotations might be in the "Digital Editions/Annotations" directory. These will be read when the book is opened and they are merged with whatever is in the database. Recent firmware makes the database the primary source, but, if there are no entries in the database, whatever is in the file is use.

I looked at copy of database with DB browser for SQlite and did an export then import.
Except either it's too broken, or I need to edit the text as original is about 150M and "new" version about 92k!

If you tried to do that from the corrupt database, that is pretty much what I would expect. It will only have exported the data it could reach. I had another look at this recently because my wife's Forma had a corrupt database. That was one of the things I tried. The signout was the only thing that worked. That plus turning the backup on again.

[Quoth]

"If you are using epubs, then the annotations might be in the "Digital Editions/Annotations" directory."
Which may have been replaced with a file to stop the old bug. Not sure if that is still needed.

Thanks, David. All much like I thought. Because of how I use the ereader a database backup is in reality rarely going to save more than a few reading positions as I'm usually connecting to Calibre to fetch a fresh set of notes/annotations that didn't exist on the last connection.

A quick check in "Digital Editions":
A file called Annotations (empty)
A directory, Manifest, with some book titles as directories with xml files. Only a handful of the 2000+ books. Some are books no longer on the Kobo. What is it for?

[davidfor]

Originally Posted by Quoth

A quick check in "Digital Editions":
A file called Annotations (empty)
A directory, Manifest, with some book titles as directories with xml files. Only a handful of the 2000+ books. Some are books no longer on the Kobo. What is it for?

The "Digital Editions" directory is used by the RMSDK and ADE when it is connected. Changing the "Annotations" directory to a file was a workaround for some issues with the annotating that was fixed in one of the recent firmware versions.

[Quoth]

Originally Posted by davidfor

The "Digital Editions" directory is used by the RMSDK and ADE when it is connected. Changing the "Annotations" directory to a file was a workaround for some issues with the annotating that was fixed in one of the recent firmware versions.

I wasn't sure it was fixed. I knew it was for ADE.
What's RMSDK and why does the Manifest directory have the few dozen files when I've only ever connected to Linux Mint via USB and Calibre?

Thanks.

[davidfor]

Originally Posted by Quoth

I wasn't sure it was fixed. I knew it was for ADE.
What's RMSDK and why does the Manifest directory have the few dozen files when I've only ever connected to Linux Mint via USB and Calibre?

The Adobe RMSDK is what is used to read ePubs in any ereader or app that supports Adobe DRM. It is maintaining the files in that directory. ADE will also use it when connected.

[Quoth]

I've never connected ADE and I've no epub titles with DRM. So the Kobo has created the XML files. Yet I can't see on what basis, unless it's just titles I've opened since a particular FW update?

[davidfor]

Originally Posted by Quoth

I've never connected ADE and I've no epub titles with DRM. So the Kobo has created the XML files. Yet I can't see on what basis, unless it's just titles I've opened since a particular FW update?

I know I said the RMSDK is used to read books with Adobe DRM, it is also used to read all epubs on a Kobo device. Hence, you should have files in there for every epub that you have opened on the device. And PDFs, as it is used for that as well.

[Quoth]

So perhaps it "reset it" at one of the firmware updates, or when I logged out and back in, though it has too many since I re-did the account, and I think some I haven't read recently. Very odd selection.
I don't think I've any kepubs now.

[davidfor]

Originally Posted by Quoth

So perhaps it "reset it" at one of the firmware updates, or when I logged out and back in, though it has too many since I re-did the account, and I think some I haven't read recently. Very odd selection.
I don't think I've any kepubs now.

I don't know what the exact rules are. My expectation was that there would be files for each book opened. But, looking at one of my devices, I cannot quickly see a file other than the annotations files, that was created later than December last year. That suggests a change last year, but, it might have been earlier, as I occasionally use ADE and there are a lot created on the same date.

[Quoth]

Thanks.
I suspected it wasn't simple.