Clara HD database disk image error

[retro]

The problem I'm having with my Clara started a few months ago and despite all my efforts it keeps coming back.
My Clara was working fine for several months but sometime after the update earlier this summer it started having problems.
I added a few new books from Calibre and the covers did not load. Connecting repeatedly didn't make a difference, but when I used the Kobo Utility to force the covers, they showed up. This went on for a week or so until one day I finished a book and selected remove. It did not delete. I tried a couple of more books that would not delete either. When I connected to Calibre version 4.23.0, this message popped up.

ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 90, in run
File "site-packages\calibre\gui2\device.py", line 513, in _books
File "site-packages\calibre\devices\kobo\driver.py", line 1981, in books
File "site-packages\calibre\devices\kobo\driver.py", line 1852, in get_bookshelvesforbook
File "c:\t\t\apsw-47cp24d7\src\cursor.c", line 236, in resetcursor
CorruptError: CorruptError: database disk image is malformed

I did a factory reset, let it upgrade, applied patches and all was well. A few weeks later I started having issues with covers not loading again.

Since I had upgraded to 32gb micro sd, I installed the original image to a new micro sd, let it upgrade to the latest version, applied patches and all was well.

Earlier this week, covers quit loading and today when I connected to Calibre (still on 4.23 - KoboTouchextended driver) I got the database error again.

Since I've had this happen with 2 different sd cards, they should't be the problem.

Maybe I've done something squirrelly with the patching without knowing??

Any ideas would be appreciated.
Thanks!

[geek1011]

How do you connect/disconnect the device? Do books open correctly? Do you ever have errors when copying books to the device directly? Also, do you have Plato, KOReader, KSM, or vlavsovsoft installed?

[retro]

Originally Posted by geek1011

How do you connect/disconnect the device? Do books open correctly? Do you ever have errors when copying books to the device directly? Also, do you have Plato, KOReader, KSM, or vlavsovsoft installed?

I connect with a usb cable and always disconnect device with calibre before unplugging.
Even when I get the database error, I can add books to the device without errors.
No Plato, KOReader, KSM, or vlavsovsoft.
Miniclock is installed and about 12 patches.

[geek1011]

That's odd. If you open the database in an external editor (e.g. sqlitebrowser or the sqlite3 cli) and run "PRAGMA integrity_check;", what does it say?

[retro]

Originally Posted by geek1011

That's odd. If you open the database in an external editor (e.g. sqlitebrowser or the sqlite3 cli) and run "PRAGMA integrity_check;", what does it say?

Execution finished without errors.
Result: 1 rows returned in 45ms
At line 1:
PRAGMA integrity_check;

I used sqlitebrowser, opened the KoboReader.sqlite, ran it in the sql window and again under tools. I guess that the right way.
Looks like it's ok, but calibre is still throwing the error.

Ok, I tried to open BookReader.sqlite and got 'database is malformed' and would not open.
Not sure what would cause that?

[geek1011]

Originally Posted by retro

Execution finished without errors.
Result: 1 rows returned in 45ms
At line 1:
PRAGMA integrity_check;

I used sqlitebrowser, opened the KoboReader.sqlite, ran it in the sql window and again under tools. I guess that the right way.
Looks like it's ok, but calibre is still throwing the error.

Ok, I tried to open BookReader.sqlite and got 'database is malformed' and would not open.
Not sure what would cause that?

BookReader.sqlite is encrypted. Regarding the other database, I can't say much from here since I'm not familiar with Calibre. @davidfor might know more about that.

[davidfor]

Originally Posted by retro

Execution finished without errors.
Result: 1 rows returned in 45ms
At line 1:
PRAGMA integrity_check;

I used sqlitebrowser, opened the KoboReader.sqlite, ran it in the sql window and again under tools. I guess that the right way.
Looks like it's ok, but calibre is still throwing the error.

Are you saying you ran the above and got the error from calibre in the same connection session? That shouldn't be possible. Do you have the Kobo Utilities plugin installed in calibre? If so, when you next see the above error in calibre, can you run the database check the plugin has? It is just running "PRAGMA integrity_check", but in the same context as the access from calibre.

Ok, I tried to open BookReader.sqlite and got 'database is malformed' and would not open.
Not sure what would cause that?

As @geek1011 said, this is encrypted and is not used for what you are looking at.

[retro]

Originally Posted by davidfor

Are you saying you ran the above and got the error from calibre in the same connection session? That shouldn't be possible. Do you have the Kobo Utilities plugin installed in calibre? If so, when you next see the above error in calibre, can you run the database check the plugin has? It is just running "PRAGMA integrity_check", but in the same context as the access from calibre.

No, I did not have calibre open when I used SQLitedatabasebrowser. I have run this a number of times directly on the KoboReader.sqlite on the Kobo and still it shows no errors. The error pops up when I connect with calibre open. The Kobo Utilties plugin is installed and this is what the check reported.

calibre, version 4.23.0
ERROR: Unhandled exception: <b>CorruptError</b>:CorruptError: database disk image is malformed

calibre 4.23 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.18362-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 34, 0) && DeDRM (6, 8, 0) && Package KFX (from KFX Input) (1, 34, 0) && Find Duplicates (1, 7, 2) && KindleUnpack - The Plugin (0, 82, 2) && Kobo Utilities (2, 11, 5) && EpubMerge (2, 11, 0) && KFX metadata reader (from KFX Input) (1, 34, 0) && KFX Input (1, 34, 0) && Modify ePub (1, 4, 1) && Import List (1, 5, 1) && EpubSplit (2, 9, 0) && Manage Sony x50 Reader Book List (1, 2, 11) && Count Pages (1, 10, 0) && KoboTouchExtended (3, 4, 1) && Quality Check (1, 10, 1)
Traceback (most recent call last):
File "calibre_plugins.koboutilities.action", line 1605, in check_device_database
File "calibre_plugins.koboutilities.action", line 3440, in _check_device_database
File "calibre_plugins.koboutilities.action", line 4995, in check_device_database
File "c:\t\t\apsw-47cp24d7\src\cursor.c", line 236, in resetcursor
CorruptError: CorruptError: database disk image is malformed

Originally Posted by davidfor

As @geek1011 said, this is encrypted and is not used for what you are looking at.

I have a 2nd Clara and I copied the BookReader.sqlite from both readers to my computer and tried to open them in SQLitedatabasebrowser. The problem one would not open saying it was mal formed. But the copy from the 2nd Clara popped up a window asking for a password for the encryption.

[davidfor]

Originally Posted by retro

No, I did not have calibre open when I used SQLitedatabasebrowser. I have run this a number of times directly on the KoboReader.sqlite on the Kobo and still it shows no errors.

No, I meant, did you run the check and calibre in the same session. You don't want to open the databases twice at the same time, but, if you connect, do the check and then run calibre, do you see the error? Or if you see the error in calibre, does the check work when you close calibre?

The error pops up when I connect with calibre open.

Do you mean this happens only if calibre is open when you connect the device? Does it happen if calibre is closed and you open it after the Windows notifies you the device is connected? If so, that has been reported before, but not for a long time. It suggests something is blocking access to the files on the device for a little while. An antivirus program is the obvious culprit, but it could be something else.

The Kobo Utilties plugin is installed and this is what the check reported.

calibre, version 4.23.0
ERROR: Unhandled exception: <b>CorruptError</b>:CorruptError: database disk image is malformed

calibre 4.23 [64bit] embedded-python: True is64bit: True
Windows-10-10.0.18362-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 34, 0) && DeDRM (6, 8, 0) && Package KFX (from KFX Input) (1, 34, 0) && Find Duplicates (1, 7, 2) && KindleUnpack - The Plugin (0, 82, 2) && Kobo Utilities (2, 11, 5) && EpubMerge (2, 11, 0) && KFX metadata reader (from KFX Input) (1, 34, 0) && KFX Input (1, 34, 0) && Modify ePub (1, 4, 1) && Import List (1, 5, 1) && EpubSplit (2, 9, 0) && Manage Sony x50 Reader Book List (1, 2, 11) && Count Pages (1, 10, 0) && KoboTouchExtended (3, 4, 1) && Quality Check (1, 10, 1)
Traceback (most recent call last):
File "calibre_plugins.koboutilities.action", line 1605, in check_device_database
File "calibre_plugins.koboutilities.action", line 3440, in _check_device_database
File "calibre_plugins.koboutilities.action", line 4995, in check_device_database
File "c:\t\t\apsw-47cp24d7\src\cursor.c", line 236, in resetcursor
CorruptError: CorruptError: database disk image is malformed

Same basic error, so it is seems to be blocking this.

I have a 2nd Clara and I copied the BookReader.sqlite from both readers to my computer and tried to open them in SQLitedatabasebrowser. The problem one would not open saying it was mal formed. But the copy from the 2nd Clara popped up a window asking for a password for the encryption.

This database has nothing to do with it. The driver does not look at it at all. And as it is encrypted, we can't look at it.

[retro]

Originally Posted by davidfor

No, I meant, did you run the check and calibre in the same session. You don't want to open the databases twice at the same time, but, if you connect, do the check and then run calibre, do you see the error? Or if you see the error in calibre, does the check work when you close calibre?

I have not run both at the same time.
Yes, I can do the check that says no errors then open calibre and get the error, close calibre, do the check and it says no errors.

Originally Posted by davidfor

Do you mean this happens only if calibre is open when you connect the device? Does it happen if calibre is closed and you open it after the Windows notifies you the device is connected? If so, that has been reported before, but not for a long time. It suggests something is blocking access to the files on the device for a little while. An antivirus program is the obvious culprit, but it could be something else.

I usually open calibre before connecting but trying both ways, I always get the error. I would think if something like antivirus was blocking, it would do it on both Claras. The 2nd one has no problems.

Originally Posted by davidfor

This database has nothing to do with it. The driver does not look at it at all. And as it is encrypted, we can't look at it.

Just seemed odd to me that one can be accessed and the other is said to be malformed.

[davidfor]

Originally Posted by retro

I have not run both at the same time.
Yes, I can do the check that says no errors then open calibre and get the error, close calibre, do the check and it says no errors.



I usually open calibre before connecting but trying both ways, I always get the error. I would think if something like antivirus was blocking, it would do it on both Claras. The 2nd one has no problems.

I didn't notice there was another device that did work. But, the antivirus might be seeing something different about the devices.

Just seemed odd to me that one can be accessed and the other is said to be malformed.

Yes, it is strange. Is the error you see when using calibre always the same? The first post shows an error getting the collections from the device. Is the last line in the error for file "site-packages\calibre\devices\kobo\driver.py" always the same?

Could you run the following SQL statement using SQLitedatabasebrowser? This accesses the table the line shown is querying.

select ShelfName
from ShelfContent
where _IsDeleted = 'false'
and ShelfName is not null

And then:

select ShelfName
from ShelfContent
where _IsDeleted = 'false'
and ShelfName is not null
order by ContentId

And can you run calibre in debug mode and post the log? That might tell me something. Such a is the error happening on the first book.

And would it be possible to see the database? Don't post it, upload it somewhere and PM me the link to it. It will be interesting to see if I can see anything going on with it.

[retro]

Originally Posted by davidfor

I didn't notice there was another device that did work. But, the antivirus might be seeing something different about the devices.


Yes, it is strange. Is the error you see when using calibre always the same? The first post shows an error getting the collections from the device. Is the last line in the error for file "site-packages\calibre\devices\kobo\driver.py" always the same?

Could you run the following SQL statement using SQLitedatabasebrowser? This accesses the table the line shown is querying.

select ShelfName
from ShelfContent
where _IsDeleted = 'false'
and ShelfName is not null

And then:

select ShelfName
from ShelfContent
where _IsDeleted = 'false'
and ShelfName is not null
order by ContentId

And can you run calibre in debug mode and post the log? That might tell me something. Such a is the error happening on the first book.

And would it be possible to see the database? Don't post it, upload it somewhere and PM me the link to it. It will be interesting to see if I can see anything going on with it.

All the connecting and disconnecting must have pushed it over the edge and the reader quit being recognized and wouldn't delete books.
I did a factory reset, didn't lose my patches, just added the fonts and books back and everything is fine again.
One thing I did try after the reset was to open the BookReader.sqlite and got the popup for the encryption password unlike the malformed error like before. I made a copy of the file to try replacing when it happens again. I'm sure I will be able to try some of your other suggestions too.

I can't remember exactly when this first started earlier this summer so it could possibly be an issue with calibre and another version might fix it. I'm just not ready to move to 5 yet.

Thanks for trying to help!

[davidfor]

Originally Posted by retro

All the connecting and disconnecting must have pushed it over the edge and the reader quit being recognized and wouldn't delete books.
I did a factory reset, didn't lose my patches, just added the fonts and books back and everything is fine again.

That suggests something was going on and the way that calibre access the database. My Glo HD has will connect to my main laptop and I can look at the disc. But, run calibre, and it an error is raised by both calibre and Windows. There might have been something along those lines. Possibly just reformatting the book partition was enough.

One thing I did try after the reset was to open the BookReader.sqlite and got the popup for the encryption password unlike the malformed error like before. I made a copy of the file to try replacing when it happens again. I'm sure I will be able to try some of your other suggestions too.

Reading back, I hadn't noticed that the copy you took of this gave an different error. That makes me more confident that it was a file system corruption of some sort. Hopefully it isn't a failing SD card. If it happens again, I'd replace the card.

As to BookReader.sqlite, forget about it. It is completely and utterly uninvolved in anything that you have been describing. It will only be used if you have a Kobo Plus account. Calibre does absolutely nothing with it. It has absolutely no idea the file exists. If you think something is wrong with it, just delete it. The device will recreate it.

I can't remember exactly when this first started earlier this summer so it could possibly be an issue with calibre and another version might fix it. I'm just not ready to move to 5 yet.

I don't think it is a software issue. I think it is more likely that the file system was corrupt somehow, or possibly the SD card is failing.

[geek1011]

Originally Posted by davidfor

As to BookReader.sqlite, forget about it. It is completely and utterly uninvolved in anything that you have been describing. It will only be used if you have a Kobo Plus account.

Or OverDrive.

[retro]

Originally Posted by davidfor

That suggests something was going on and the way that calibre access the database. My Glo HD has will connect to my main laptop and I can look at the disc. But, run calibre, and it an error is raised by both calibre and Windows. There might have been something along those lines. Possibly just reformatting the book partition was enough.

Reading back, I hadn't noticed that the copy you took of this gave an different error. That makes me more confident that it was a file system corruption of some sort. Hopefully it isn't a failing SD card. If it happens again, I'd replace the card.

As to BookReader.sqlite, forget about it. It is completely and utterly uninvolved in anything that you have been describing. It will only be used if you have a Kobo Plus account. Calibre does absolutely nothing with it. It has absolutely no idea the file exists. If you think something is wrong with it, just delete it. The device will recreate it.


I don't think it is a software issue. I think it is more likely that the file system was corrupt somehow, or possibly the SD card is failing.

Actually, in the first post I said I did replace the SD card after the first time this happened because it looked like corruption. Also, I wasn't sure a factory reset would rewrite files that were corrupted, so I used the original image and let it upgrade through Kobo.

The fact that it's progressive, first not loading covers, next not deleting books, eventually not being able to manage at all with calibre looks like corruption. This time I did not look to see if I could access the drive, didn't seem like it mattered when a reset was inevitable.

Thanks for helping.