Database is corrupt - a question

[John F]

Originally Posted by LordP

In my scenario, I plugged the device in, and the database was declared corrupt, I unplugged it, rebooted, and then it came up as OK.

...

Who said it was corrupt? Windows, Kobo?

[LordP]

Originally Posted by John F

Who said it was corrupt? Windows, Kobo?

Actually it was Calibre when I first connected the device. And also, when I use KoboUtilities -> Database -> Check the device database.

Are there other ways I can be told the database is corrupt?

How would Windows do that? As far as I know it doesn't check the device - it merely connects it ----- No?

In all my years of using Readers that is the only thing that has ever given me that warning.

I think I'm getting even more confused than when I began this. LOL.

[compurandom]

Originally Posted by PeterT

In hindsight I think the device might still have been importing data after the wireless sync. This raises the question though of how to tell this has completed; unlike the post sideloading sync there is no status message displayed.

That's the bug... it shoudln't allow usb connection when there is stuff still actively writing the database.

It should either kill that stuff, or wait for it to finish before offering to connect the USB.
Usually it does wait but not always.

That's why rebooting before connecting helps. I'd also recommend turning off wifi *before* rebooting, otherwise it might start a wifi sync after rebooting which will also cause issues.

If you know you just downloaded a bunch of new stuff, it wouldn't hurt to wait a few minutes for things to settle.

And ignore the advise about rolling back firmware. All the firmwares before 4.41 are worse for this problem.

[LordP]

Originally Posted by compurandom

That's the bug... it shoudln't allow usb connection when there is stuff still actively writing the database.

It should either kill that stuff, or wait for it to finish before offering to connect the USB.
Usually it does wait but not always.

That's why rebooting before connecting helps. I'd also recommend turning off wifi *before* rebooting, otherwise it might start a wifi sync after rebooting which will also cause issues.

If you know you just downloaded a bunch of new stuff, it wouldn't hurt to wait a few minutes for things to settle.

And ignore the advise about rolling back firmware. All the firmwares before 4.41 are worse for this problem.

OK. That makes things a bit simpler. PeterT has a different issue I think.

Mine is a bit different.

Boiled down it's simply this: Is the database corrupt or not?

Does anyone know what the device does on reboot? Does it rewrite the database? That would be a huge piece of the puzzle - no?

Thanks.

[compurandom]

Originally Posted by LordP

Boiled down it's simply this: Is the database corrupt or not?

It's hard to say. The database can be fine but the data in it corrupt and that causes issues too, but it's not detectable with database tools.

The only way to be 100% sure your database isn't corrupt is to delete it and let the kobo rebuild it. Obviously that's a solution of last resort, and I do it only when my kobo's misbehavior starts becoming intolerable.

Note also that there are *two* databases, and one of them is encrypted, so we can't even check that one. If the encrypted database is corrupt, the online accounts (kobo, overdrive at least) may not work or sync correctly.

Does anyone know what the device does on reboot? Does it rewrite the database?

I doubt it does anything special on reboot. What makes the reboot help is that background things that were keeping the database open are now gone. Assuming it doesn't immediately start a sync anyway.

[PeterT]

The database is not rewritten on reboot. If the corruption is too bad (no idea how bad it had to be) it is deleted and a new one is created with your being forced to do a reset up of the device.

The closest you can get to knowing from the device itself is subtle signs like the reading position not being retained across reboots.

Within calibre it is detected in two distinct locations. One is from within the kobo device driver when it attempts to open the database. This is the check you will see after the device has been connected. The other is if the Kobo Utilities plugin has been installed where it issues a "PRAGMA INTEGRITY_CHECK" to perform an integrity check of the database, it's internal structures and contents.

Sent from my Pixel 7a using Tapatalk

[compurandom]

Originally Posted by PeterT

The database is not rewritten on reboot. If the corruption is too bad (no idea how bad it had to be) it is deleted and a new one is created with your being forced to do a reset up of the device.

In other words, if you don't delete the corrupt database, eventually the kobo will delete it for you.

I suspect the threshold is when it is unable to open either one of the databases at all, or when it's so corrupt some basic query fails.

The closest you can get to knowing from the device itself is subtle signs like the reading position not being retained across reboots.

Ya, that's a bad sign, and one of the first.

Some others I've noticed:

• Can't delete books
• Can't delete or add books on the home screen, or the old set reappears after changing it
• Can't sync overdrive or kobo store to get new books
• Import books stops working, so new books dropped on the kobo don't show up
• Forgets saved wifi passwords and won't resave them
• Phantom books that have been deleted from the device stay in the book list and can't be opened
• Shelves can't be changed

I'm sure there's others, and which of these you might see depends on what parts of the databases are corrupted.

Note that reading position not being saved is different from the reading position shifting by a few pages when opening and closing a book. That's likely a different bug.

[JSWolf]

Originally Posted by LordP

I have done that in the past and may do it again.

I have another stupid question, or two, though.

In my scenario, I plugged the device in, and the database was declared corrupt, I unplugged it, rebooted, and then it came up as OK.

So, was the database corrupted or not?
If it was corrupted, does the device rebuild the database when it reboots?

I very rarely reboot, almost never.

Also, to clarify what I do when detaching the Clara from my PC: I disconnect through Calibre, I also check that it's disconnected from the USB in the system tray. I wait for the device to turn the screen on, fully lit, and then go through the process of importing and back to the home screen, again fully lit - ONLY THEN do I unplug it.

If that method corrupts the database, well, then I guess I'm screwed because I can't think of anything else I could possibly do.

One thing to do is to setup Kobo Utilities to keep track of your reading positions. That way if the database ever gets too corrupt, you have where you left off on last USB connect.

Also use Kobo Utilities to make a backup of your database on connect. It won't backup a corrupt database.

With the current firmware, you'll be best to reboot before connecting to the computer.

One idea I've come up with is using NickelMenu to get a directory listing of the where the database is. That way you can see if it's open or not and it not, connect without a reboot and if it's open, reboot first.

[LordP]

Guys, you are ALL missing the point.

JSwolf is going off the deep end, I think

It's really simple - is the database corrupt, OR NOT?

The software says on thing, but rebooting helps: WHY??????????????????????????????????????????????? ??????????????????????????????????

Sorry but it seems to me that no one is LISTENING!!

Is it the first thing or the second thing?

[Aleron Ives]

The purpose of rebooting is to close any processes that might be accessing the database. This prevents the database from becoming corrupted. If your database is already corrupt, rebooting doesn't help. You must either restore a backup of the database, or delete it and start over. Once the database is corrupt, you can't fix it.

[John F]

Originally Posted by LordP

Actually it was Calibre when I first connected the device. And also, when I use KoboUtilities -> Database -> Check the device database.

Are there other ways I can be told the database is corrupt?

How would Windows do that? As far as I know it doesn't check the device - it merely connects it ----- No?

In all my years of using Readers that is the only thing that has ever given me that warning.

I think I'm getting even more confused than when I began this. LOL.

There was some version of Windows or Kobo that would report there was a "problem" when the reader was mounted by Windows. I seem to recall that some bit wasn't getting set/reset on dismount. I haven't seen the problem in a long time.

Sorry for the confusion.

[JSWolf]

Originally Posted by LordP

Guys, you are ALL missing the point.

JSwolf is going off the deep end, I think

It's really simple - is the database corrupt, OR NOT?

The software says on thing, but rebooting helps: WHY??????????????????????????????????????????????? ??????????????????????????????????

Sorry but it seems to me that no one is LISTENING!!

Is it the first thing or the second thing?

I read what you posted. I have no idea why the database was reported to be corrupted but a reboot seemed to fix it. The only guess I can make is that when you rebooted, the system was able to fix the database.

My idea of checking if the database is open or not is the only way you'll know the state of the database unless you reboot before connecting to the USB port.

[JSWolf]

Originally Posted by John F

There was some version of Windows or Kobo that would report there was a "problem" when the reader was mounted by Windows. I seem to recall that some bit wasn't getting set/reset on dismount. I haven't seen the problem in a long time.

Sorry for the confusion.

That was for the filesystem, not the database.

[DNSB]

If it is similar to the last database corruption issue, the problem was that the databases were not properly closed before the USB connection was established. In a file explorer, you would see files called KoboReader.sqlite-wal, KoboReader.sqlite-shm, BookReader.sqlite-wal BookReader.sqlite-shm (not necessarily both pairs but even one pair indicates that database was not properly closed) in the .kobo directory.

My workaround was to force a reboot using NickelMenu to ensure the database was closed before the USB connection.

What I also found was the if I got the database corrupted message and immediately ejected, the database will be closed which also merges the -wal and -shm file so the next connection will not show a corrupt database. If any attempt to write to the database is made, this will leave the database in a corrupt state. For me using Windows, this meant closing calibre immediately and then ejecting by right-clicking on the Kobo drive in File Explorer and then on Eject.

I suspect this is why the OP saw the now I'm corrupted, now I'm not corrupted behaviour.

[PeterT]

Originally Posted by LordP

Guys, you are ALL missing the point.

JSwolf is going off the deep end, I think

It's really simple - is the database corrupt, OR NOT?

The software says on thing, but rebooting helps: WHY??????????????????????????????????????????????? ??????????????????????????????????

Sorry but it seems to me that no one is LISTENING!!

Is it the first thing or the second thing?

This sounds similar to the times that the Kobo claims that a book is DRMed and it's not. Again here a reboot solves the issue. A programming error causing a memory leak or some internal data structure within Nickel getting corrupted.

Sent from my Pixel 7a using Tapatalk