Database is corrupt - a question

[compurandom]

Originally Posted by LordP

Guys, you are ALL missing the point.

JSwolf is going off the deep end, I think

I didn't miss the point, I ignored JSwolf. He's always off the deep end.

Originally Posted by John F

There was some version of Windows or Kobo that would report there was a "problem" when the reader was mounted by Windows.

That's for filesystem corruption, which probably accompanies database corruption, and for the same reason -- something kept it open, two operating systems trying to write to the same disk at the same time. So yes, wrong thing, right reason anyway.

Originally Posted by DNSB

What I also found was the if I got the database corrupted message and immediately ejected, the database will be closed which also merges the -wal and -shm file so the next connection will not show a corrupt database.

That's very valid but you gotta get lucky! You have to eject and reboot before anything else especially calibre starts scribbling on the database.

JSwolf suggested looking for the WAL files from nickelmenu before plugging in USB, which is on the right track, but as usual, dead wrong. The problem is that as long as the kobo is in its GUI and not in usb mode, the database is open, so it will always be open then.

Similarly, once calibre has detected the device, the driver opens the database, and then it's too late to check. What we need is the driver itself, before opening the database, to check for the wal files and then refuse to open the device until it is rebooted... but even I might be off base there, because maybe it already does that and that's still not enough.

Maybe the issue is that the kobo OS has the database open but the wal file only exists in cache memory and hasn't been written to disk yet, so the OS calibre is in doesn't see it.

And then to make matters worse, calibre scribbles on the database, but the kobo OS still has the cached blocks in memory it wrote, which are now out of date, and it writes them back anyway, so now you have a database with a few old blocks written to it from before the USB mode was activated and a few blocks from calibre that were written while in USB mode, and it's all mixed up and corrupt...

If you reboot before doing all this, it doesn't happen.

If you don't reboot, then the kobo might keep cached blocks from before calibre wrote the database, and things might look ok for a while but the copy of the database blocks on disk and in kobo memory might not be in sync.

If you reboot afterwards, then the kobo memory is cleared and database blocks in memory will be fully in sync, corrupt or not. So rebooting afterwards might look like it helps but it doesn't actually prevent corruption, because it's probably too late. I think sometimes the kobo devices have bad ram in them which gets bit flips and then the kobo starts acting strange... and in those cases, rebooting helps a lot.

[DNSB]

Originally Posted by compurandom

That's very valid but you gotta get lucky! You have to eject and reboot before anything else especially calibre starts scribbling on the database.

As far as I can tell, as long as the message was on screen, calibre wasn't trying to write to the database. So as I said, close calibre immediately and eject the Kobo. I ran ~14 tests and did not get database corruption doing this. OTOH, it was simply easier to reboot my Kobo before connecting.

Originally Posted by compurandom

Maybe the issue is that the kobo OS has the database open but the wal file only exists in cache memory and hasn't been written to disk yet, so the OS calibre is in doesn't see it.

From the documentation, the -wal and -shm files are written to disk and not stored in memory. The problem is that they have to be written to the database before the database is closed though the SQLite code seems to handle the database not being properly closed so the -wal and -shm files will be merged when the database is re-opened unless the database has been hooped in the meantime.

[JSWolf]

How about a plugin for calibre that runs before anything else runs that checks if the database is closed. Would that help to not have to reboot before every connection to the computer?

[PeterT]

Jon; as soon as the USB cable is connected and the option selected to mount the drive to the host, if the database was open, corruption will occur. This is regardless of whether or not calibre or any program on the host has accessed the drive.

[DNSB]

Not quite right, Peter. What I found is if as soon as the database corrupt message shows up, I closed calibre and then safely ejected my Kobo, the database was not corrupted. I did use a reboot to force the database to close before trying to reconnect.

I suppose you could mount your Kobo, look at the .kobo directory and then safely eject and reboot if sqlite-wal/sqlite-shm files are present. If not, run calibre as usual. I played with several ideas when my Sage was running firmware that was prone to database corruption and found a simple reboot before connecting was safer and faster than connecting, checking for the journal files and ejecting and rebooting if the journal files were present.