Kobo Glo SQLite Database

[jbaumann]

Hi,

my next steps:
- re-enabled KSM and started Nickel without fmon
- plugged in the reader and checked the database (ok)
- added a few books with calibre
- checked the database (ok)
- unplugged the reader and let Nickel process the new books
- plugged the reader back in and checked the database. It was corrupted.

So, KSM corrupts the database even without fmon. What kinds of scripts are running and how could they do this? Is there any way I could provide debugging info or do some additional tests that could shed some light on this?

I would very much like to have KSM on my reader, but in the worst case I would have to find another way to start koreader (I need the reflow capability...). What could that be?

Cheers, Joe
PS: I have a very large collection with 475 books in calibre, which is not created on the reader. What is the maximum number of books that the Kobo Glo accepts in a collection?

[Ken Maltby]

Originally Posted by jbaumann

Hi,

my next steps:
- re-enabled KSM and started Nickel without fmon
- plugged in the reader and checked the database (ok)
- added a few books with calibre
- checked the database (ok)
- unplugged the reader and let Nickel process the new books
- plugged the reader back in and checked the database. It was corrupted.

So, KSM corrupts the database even without fmon. What kinds of scripts are running and how could they do this? Is there any way I could provide debugging info or do some additional tests that could shed some light on this?

That sequence looks more like the "corruption" came about when Nickel processed the calibre additions.

Have you tried the "AdvBoot" launcher I mentioned?

Luck;
Ken

[tshering]

Originally Posted by jbaumann

So, KSM corrupts the database even without fmon. What kinds of scripts are running and how could they do this? Is there any way I could provide debugging info or do some additional tests that could shed some light on this?

Alarmed by your repeated database corruption, I just loaded a bunch of book on my reader:

• started nickel via "start nickel" from ksm
• connected to PC and copied a punch of books to the reader
• disconnected from PC and let nickel process the books
• exited nickel to KSM
• checked the integrity of the database (ok)

Since I do not use calibre, I do not know and therefore I am asking: What does calibre when adding books apart from simply copying them to the device?

Maybe you could also check whether adding the books without calibre also results in a corrupted database on your device. Depending on the result we might think of further possibilities. Currently, I have no idea what we could check.

As for the scripts, as I said, they are doing basically nothing apart from waiting that nickel exists.

[davidfor]

I detailed the updates made by the KoboTouch driver earlier in the thread. I am using this most days on one or other of my four Kobos. I rarely see corrupted database.

Calibre also updates the metadata and cover in the book, but I doubt that is the problem.

But, disabling the driver and using "Save to disk" is probably the simplest way for a calibre user to get a lot of books to the device.

[Jellby]

I've also had similar problems with my Aura, but I couldn't decide KSM (or starting Nickel through KSM) was the culprit. Could you repeat your test with KSM disabled a few more times? In my experience, the corruptions are random, and I have to do my transfers in small batches (sometimes 1 by 1), with backups between them and often in several tries.

[tshering]

Originally Posted by davidfor

I detailed the updates made by the KoboTouch driver earlier in the thread.

I see. So when talking about "adding books by calibre" it always means that the KoboTouch driver is involved. (I did not know this for sure).
Then one difference between transferring books with and without calibre would be, that after ejecting and disconnecting, in one case the database on disk and the copy in reading cache are identical, and in the other case they are different. I am not sure whether this thought leads to somewhere.

@jbaumann, one thing I was remainded by this post is that if vlasovsoft is installed on your reader, depending on how you installed it, there might be running instances of fmon, even if KSM does not start them. I am not sure however whether this really makes a difference.

[jbaumann]

Hi,

sorry that I omitted that. Yes, the Kobotouch driver is involved whenever I use calibre to copy books to the reader.

Here is what I have done today:

Spoiler:

DB ok
Reboot with KSM
Plug in without calibre
DB ok
Unplug
Plug in
DB ok
Copy 1 book by hand
unplug
reader updates
plug in
DB ok
start calibre (no books copied)
unplug
plug in
DB ok
KoboUtilities - update Metadata
Unplug
plug in
DB ok
copy 1 book with calibre
unplug
plug in (hung up on connection)
plug in
DB ok
added 100 books with calibre
unplug
reader scans
plug in
DB problem
wrong # of entries in index analytics_events_timestamp
wrong # of entries in index sqlite_autoindex_AnalyticsEvents_1
copied over old version (after copying the books)
unplug
reader scans (all books - very long time - hung up at 83%)
Reset
KSM - USB Enable
Copy valid DB
Turn off KSM
Reboot
Plug in
DB ok, additional books not added (update not triggered?)
copied books again
Unplugged
plug in
DB ok
unplug
start KSM and USB
copy 100 books
unplug
reboot without KSM
Nickel does not scan
plug in
DB ok
unplug
Nickel does scan
plug in
DB ok

So, if I copy a very low number of books (1 in my experiments), nothing untoward happens. As soon as I copy a relevant number of books, an active KSM leads to a corrupted database.

If I copy a valid database file using the usb connection under KSM and then reboot without KSM, everything works fine. I have to trigger a scan with Nickel by plugging the reader in though to recognize new books copied to the device. Don't know how to do it directly.

If I copy 100 books under KSM and then again reboot without KSM to let Nickel scan the new books, it works fine as well (with the limitation of Nickel recognizing the new books only after another cycle of plugging and unplugging the reader).

Now, it seems that the process of copying the data itself works fine with KSM enabled, in KSM and in Nickel.

The subsequent scan of new books and the modification of the database does not work correctly with KSM enabled.

Important: Without KSM I can copy large numbers of books onto the reader without problems.

The next experiment could be to install fmon directly. Any recommendations about which package to use? And should I uninstall KSM first or is it enough to have it configured as never to run?

This way I could find out whether fmon works without problems. Earlier experiments showed already that KSM leads to problems even with fmon disabled.

Any other ideas what I should test? And how could I determine the KSM problem in more detail (without first having to understand the source)?

Cheers, Joe

[PeterT]

Actually you had an issue here

plug in
DB ok
copy 1 book with calibre
unplug
plug in (hung up on connection)
plug in
DB ok

[tshering]

Originally Posted by jbaumann

The next experiment could be to install fmon directly. Any recommendations about which package to use? And should I uninstall KSM first or is it enough to have it configured as never to run?

This way I could find out whether fmon works without problems. Earlier experiments showed already that KSM leads to problems even with fmon disabled.

I think if it is certain that no instance of fmon was running, when the corruption occurred, then there is no need to deal further with fmon.

So the issue seems to occur when (1) KSM is active and (2) nickel has to deal with a newly modified database and (3) at the same time with newly added books. I do not see what exactly is the problem with this setting. But maybe we can change it a little, and see what happens:

• Maybe you could start with a new booted device (so that nickel was not yet running)
• select enable usb id like nickel.sh
• add new books with calibre
• eject, disable and disconnect
• start nickel
• make nickel process the new books by opening simulate_usb_connection.png (see this post for the details); or connect and disconnect really.

I am curious whether this makes a difference.

The issue mentioned by Peter is also alarming.

[jbaumann]

Hi,

ok, did that. Here are the results:

Spoiler:

Newly booted device
select enable usb id like nickel (what is the difference to the other?)
add new books
eject, disable and disconnect
start nickel (everything ok)
simulate usb connection worked
short flash of the display
no books anymore
reboot to ksm
check database
DB ok (which is totally strange)
reboot without ksm
new books not there
plug in
unplug
Books are there
plug in
db ok

So, my first question: What is the difference between the two types of usb connections?

Second, what happened directly after simulating the usb connection?

It seems as if Nickel simply lost its db connection and could no longer find the books. It also didn't touch the database, it seems because the db was ok after rebooting (I didn't check file size or content, should have done that).

Curiouser and Curiouser...

Cheers, Joachim

[jbaumann]

Hi Ken,

regarding the "AdvBoot" launcher: That one lets me decide only at boot time whether to use koreader or nickel, right? I would very much like to have an icon on the normal user interface with which to start koreader, or even better, have it start automatically when opening PDF files...

Cheers, Joe

[tshering]

Originally Posted by jbaumann

So, my first question: What is the difference between the two types of usb connections?

Connecting "like nickel" uses the same identification strings as nickel, that means the correct FW version number and the correct model number. The "normal KSM usb connect" puts always 0.0.0 as FW version number, and the Kobo Touch model number.

What is happening is odd.

Originally Posted by jbaumann

simulate usb connection worked
short flash of the display

When you say "simulate usb connection worked", does this mean that you saw the process screen with the progress bar? Or was there only the short flash of the display?

Originally Posted by jbaumann

no books anymore

This is not the expeted result, of course.

Originally Posted by jbaumann

reboot without ksm
new books not there
plug in
unplug
Books are there

This shows that the simulation of the usb connection did not actually work. When trying to simulate the connection, did you manage to press "connect" during the first couple of seconds? And did then appear the screen with the progress bar? I guess at that point something went wrong.

Edit: To sum it up, the actually effective steps in this case were:

Newly booted device
select enable usb id like nickel (what is the difference to the other?)
add new books
eject, disable and disconnect

reboot without ksm (should not matter, the important point is to start nickel)
new books not there
plug in
unplug
Books are there
plug in
db ok

[jbaumann]

The simulation seemed to work ok i.e., I got the connect dialog and pressed it in time (ca. 1 second after it popped up). Then a few seconds it showed the "connected" screen.

Instead of the progress bar I got this short flash and then the normal Nickel screen. Before starting this experiment I already had some books on the reader. But Nickel showed none of them. I thought the database had been be corrupted.

After a reboot though the database was ok and Nickel again knew about the books. Plugging in and then unplugging the reader brought up the progress bar and Nickel had, after finishing the scan, all the books accessible.

Cheers, Joe

[tshering]

Originally Posted by jbaumann

The simulation seemed to work ok i.e., I got the connect dialog and pressed it in time (ca. 1 second after it popped up). Then a few seconds it showed the "connected" screen.

Instead of the progress bar I got this short flash and then the normal Nickel screen. Before starting this experiment I already had some books on the reader. But Nickel showed none of them. I thought the database had been be corrupted.

After a reboot though the database was ok and Nickel again knew about the books. Plugging in and then unplugging the reader brought up the progress bar and Nickel had, after finishing the scan, all the books accessible.

Cheers, Joe

In any case, the simulation did not have the wished for effect, namely processing the new books. Instead nickel processed them only after the real usb connect and disconnect sequence.

Strangely, it can also happen that nickel does not process newly added books after a real connect/disconnect too. I have seen this some weeks ago happening to my wife's Glo, which is running on the original firmware without any modifications. After copying books, ejecting and disconnecting, nickel did not process the books. Even after repeated connecting/unconnecting and powering on and off nothing happened. Then after some further trials, all of a sudden nickel processed the books.

[frostschutz]

nickel has a way to quickly check whether books were added. It probably relies on some timestamp somewhere and unless the time changed it won't walk the entire filesystem tree looking for new files. I haven't actually investigated this in detail though, so I'm just guessing.