File or database corruption when using USB

[DNSB]

Rather interesting how the OP managed to conflate the issue with Kindles randomly removing all sideloaded content which has nothing to do with USB connections and the Kobo database corruption issue which is due to the database not being closed before a USB connection is initiated.

[Quoth]

Indeed even the Kobo issue may not exactly be USB (and thus USB protocol/mode is irrelevant), but purely the system initiating the USB connection and locking the screen without closing the database first. It's a quite strange bug because it seems to afflict certain users, no matter what incantations or OS, other than rebooting. Yet others are fine if they simply make sure the 'home' / 'my books' screen is showing. It also might be that sync has a negative impact and SideloadedMode=true has a positive impact.
I can't see that using MTP or Network USB would stop it happening. It's not specifically misuse of the USB by the host or user, it's a bug in the software that not everyone is suffering, which is no doubt why it's taking time for Kobo to solve it.

[DNSB]

Nothing like an erratic and likely race issue to make locating a bug fun (for certain values of fun).

[Renate]

Originally Posted by DNSB

Rather interesting how the OP managed to conflate the issue with Kindles...

Kindles apparently still use UMS. Therefore the issues are the same as with Kobos. Whether Amazon wipes Kindle sideloaded content is immaterial to me.

[DNSB]

Originally Posted by Renate

Kindles apparently still use UMS. Therefore the issues are the same as with Kobos. Whether Amazon wipes Kindle sideloaded content is immaterial to me.

Sadly, that is what the whole discussion which you spun this one out from was concerned with.

[Renate]

Originally Posted by DNSB

Sadly, that is what the whole discussion which you spun this one out from was concerned with.

Then maybe you should start a thread on that.

[Sirtel]

Originally Posted by Renate

Then maybe you should start a thread on that.

There are multiple threads on it already.

[ownedbycats]

Originally Posted by DNSB

Rather interesting how the OP managed to conflate the issue with Kindles randomly removing all sideloaded content which has nothing to do with USB connections and the Kobo database corruption issue which is due to the database not being closed before a USB connection is initiated.

And the Kobo issue has some pretty easy workarounds; make sure you're on homescreen, reboot the device before connecting, use Kobo Utilities to make backups of the db.

Plus we know Kobo is planning to fix it.

Originally Posted by Renate

Then maybe you should start a thread on that.

What the hell is this thread for then?

[Quoth]

Originally Posted by DNSB

Nothing like an erratic and likely race issue to make locating a bug fun (for certain values of fun).

Yes, the database corruption issue seems like that. It would still happen if the Kobos used MTP and the database is available to the user. Or maybe even if the database wasn't exposed.

I suspect that while Kindle's OS is multitasking and multiuser the the database and ebook import system is not, so it makes sense that the Kindle Scribe 'locks' the screen even though it uses MTP.

Similarly while my Andriod things don't lock the screen, I don't use the GUI on them and also close all apps before connecting USB for MTP. You just don't know how Google PlayBooks, PocketBook, Viber, Kindle app etc actually works.

Unlike MSDE, MS SQL, Oracle, DB2, MySQL, MariaDB the SQLight, just like Access or Excel data, is simply files and an API. So two processes / apps accessing the SQLight database will corrupt it, though with Kobo and USB, normally you are only accessing ebook files and cover image files while USB is connected and the Kobo accesses the database when you disconnect USB. It could even be that it's that process that causes the damage because it hasn't checked what last state was, or the process last using the database didn't close it before locking screen and opening USB connection.

The fact that the database corruption can occur just by copying a single epub file to the Kobo, without touching the database files, shows it's not specifically a USB issue and that it could occur even if the SQLight files were on the system partition that's not shared. The database import occurs after USB is disconnected, so as the exposed files show the issue is not the kind of USB connection but something amiss when the USB connection request is triggered.

I always wait a little before clicking on the connect message when I plug in USB cable. I've no idea if that helps, but I don't have the problem. I also always thought 'auto-connect' sounded like a risky developer tool. Android used to autoconnect and last few versions I have it asks even though a default mode is set.
ADB needs Developer mode on for many Android devices and that's not recommended for general use, nor is ADB which I've used as an Android developer and to do funky command line fast transfers or web cam/photo capture modes. ADB is not a system for regular users.

[DNSB]

Originally Posted by Quoth

Unlike MSDE, MS SQL, Oracle, DB2, MySQL, MariaDB the SQLight, just like Access or Excel data, is simply files and an API. So two processes / apps accessing the SQLight database will corrupt it, though with Kobo and USB, normally you are only accessing ebook files and cover image files while USB is connected and the Kobo accesses the database when you disconnect USB. It could even be that it's that process that causes the damage because it hasn't checked what last state was, or the process last using the database didn't close it before locking screen and opening USB connection.

The issue for me is that one or both of the databases are not properly closed. I not only see KoboReader.sqlite and BookReader.sqlite when database corruption is reported but the KoboReader.sqlite-wal and KoboReader.sqlite-shm files and/or the BookReader.sqlite-wal and BookReader.sqlite-shm files. Since the write ahead logging and shared memory files will be removed during the database close operation, this says the database(s) were not properly closed before the USB connection. This also leaves the dirty bit set so Windows pops up the "an error was seen, do you want to scan and fix the drive" message

Originally Posted by Quoth

The fact that the database corruption can occur just by copying a single epub file to the Kobo, without touching the database files, shows it's not specifically a USB issue and that it could occur even if the SQLight files were on the system partition that's not shared. The database import occurs after USB is disconnected, so as the exposed files show the issue is not the kind of USB connection but something amiss when the USB connection request is triggered..

Again, calibre pops up the error message when attempting to open the KoboReader.sqlite as part of mounting the Kobo since it will write to the file.

Originally Posted by Quoth

I always wait a little before clicking on the connect message when I plug in USB cable. I've no idea if that helps, but I don't have the problem. I also always thought 'auto-connect' sounded like a risky developer tool. Android used to autoconnect and last few versions I have it asks even though a default mode is set.

For what it's worth, I've set the sleep timeout to 15 minutes and telneting to my Kobo Sage shows the -wal and -shm file present after 12 minutes of idle time. So far all I can say is that the issue happens most often on my Sage after I have been reading a book, my Clara HD and others trigger the issue very rarely.

[ownedbycats]

Originally Posted by Quoth

I always wait a little before clicking on the connect message when I plug in USB cable. I've no idea if that helps, but I don't have the problem. I also always thought 'auto-connect' sounded like a risky developer tool. Android used to autoconnect and last few versions I have it asks even though a default mode is set.

I use auto-connect without issue, though I make sure I'm on the homescreen first.

I use NickelMenu to reboot following the import of new books - I've mentioned that some of my random instances of the db being left open were a reconnect immediately following import. It'll automatically reconnect itself right afterwards, which is convenient.