Kobo Glo SQLite Database

[tshering]

jbaumann,

I prepared a test version of KSM. If you like, please install and test it. If the database corruption issue is solved by that, I will start preparing KSM v 08 on this basis.

Installation:
1) If you have KSM currently installed

• copy KoboRoot.tgz to .kobo
• select (in KSM home) "handle update" > "install update" (do not select: "install partial update")

2) If you do not have KSM currently installed (But I presuppose that kbmenupngs is there from a previous installation, otherwise you are missing the features connected to it.)

• copy KoboRoot.tgz to .kobo

After the reader restarts:

• select "info" (in order to see whether you are currently running the test version of 2015.08.03)
• if the test version is running, select: "tools" > "activate" > "set runmenu settings.msh" > "always" (I have preset it to "alternateMenu" just to feel more safe)
• add books to the device as you would usually do, and check the integrity of the database

Information:
- With this package, I replace the current on-animator.sh (which might have been modified by installing vlasovsoft/pbchess) by the original file in order to prevent instances of fmon running from the user partition.
- When usb is enabled, you cannot launch koreader or vlasovsoft launcher. These menu items will reappear again the main menu when you disable usb.

Warning: If there is already a large mod installed at the system partition, this might be a problem!

[frostschutz]

Are the Qt libs and fonts already on the Kobo not compatible to the ones you're shipping? If they could be used instead you'd save 20MB of space (of 100MB available, so it's not really a pressing issue I guess).

[tshering]

Originally Posted by frostschutz

Are the Qt libs and fonts already on the Kobo not compatible to the ones you're shipping? If they could be used instead you'd save 20MB of space (of 100MB available, so it's not really a pressing issue I guess).

Thank you for the suggestion! The libraries do not seem to be compatible. The fonts are certainly overkill (actually I do not have any of them on my reader) and I could replace them by a single small font file. I do not dare to rely on the shipped fonts, since I am afraid at a certain point not a single non-obfuscated/encrypted font well be left.

[tshering]

continuing the last post ...

After pointing KSM to the shipped font directory, I suddenly recall that I tested this already once. The KSM applications do not run. I guess they hang while trying to use one of the encrypted fonts.

There is a further reason to keep the KSM fonts apart from the shipped ones, see this post.

[jbaumann]

Ok. newest experiment is the same as before, but now everything worked according to plan.

Spoiler:

a newly booted device with KSM
select enable usb id like nickel.sh
add new books with calibre
db ok
eject, disable and disconnect
start nickel
make nickel process the new books by opening simulate_usb_connection.png
progress bar ran
some additional books were there
reboot without KSM
plug in
db was ok
unplug
starts again to scan remaining books
plug in
db ok
unplug
no additional scanning

I don't get it... Why some books, and the others only after the reboot without KSM?

Cheers, Joe

[jbaumann]

Oops. too much of an assumption. The rescan with the additional books might also have happened if I had plugged in again, or after a reboot even with KSM...

[tshering]

In order to keep things simple, I would do:

• start nickel from KSM
• connect via usb
• add books with calibre
• eject and disconnect
• check the database after newly connecting from within nickel or maybe at a later point

[tshering]

Because of the only partial success of scanning the newly added books I overlooked almost the main point: the db did not get corrupted.
But maybe this was just luck. I think I wait for further (success) reports before I take further steps towards KSM 8. Thank you for the tests so far.

[jbaumann]

Oh, my last posts overlapped with your posts, and were still done with the "old" version of KSM. I will try the same with the new version you provided tomorrow, and then go on to the next test you proposed.

Btw., I do not have installed anything besides KSM (and koreader) after the factory reset. So there shouldn't be a second fmon instance running.

Cheers, Joe

[tshering]

Originally Posted by jbaumann

Oh, my last posts overlapped with your posts, and were still done with the "old" version of KSM.

I understand.

Originally Posted by jbaumann

I will try the same with the new version you provided tomorrow, and then go on to the next test you proposed.

Thank you!

Originally Posted by jbaumann

Btw., I do not have installed anything besides KSM (and koreader) after the factory reset. So there shouldn't be a second fmon instance running.

I thought maybe some other users might also try the new version and contribute their results, and therefore I provided the clean animation script.

Edit: Maybe I should mention, that with the test version you should not set runsettings to "never". Since everything is on the system partition, you cannot easily modify the settings without KSM running.

[jbaumann]

Ok, I wasn't able to wait until tomorrow :-)

Here are the results of my test:

Spoiler:

info 2015.08.03
plug in KSM
db ok
copy books with calibre
db ok
unplug
disable usb
start nickel
no scan for books
simulate usb connection
no scan
db no longer seen by reader (as before in one test, no books, and usb_connection graphics no longer executes anything)
plug in
db file size the same as before starting nickel, still ok
unplug
starts processing all books (might have been to impatient)
reset
KSM enable USB like nickel
db ok (file is larger)
disable USB
start Nickel
plug in
db ok (slightly smaller)
unplug
scanning all books again (might have been to impatient)
reset
KSM enable usb
copy db from before "simulate usb connection"
unplug
start Nickel
plug in
unplug
scanning new books (started like scanning from the beginning but got faster than that)
plug in
db ok

If you ignore my resets because I was to impatient to wait for it to scan correctly, then two observations can be made:
- simulating the usb connection does not work reliably (meaning there are side-effects that disorient Nickel)
- copying books in KSM and then letting them being scanned by Nickel (via plugging in and unplugging again) seems to work ok with this version of KSM (when before this led to database corruption)

Next test will be to put books on the reader using Nickel.

Cheers, Joachim
PS: Good to know that I shouldn't fully turn off KSM.

[jbaumann]

And here the latest test:

Spoiler:

plug in
db ok
copy number of books with calibre
db ok
unplug
process books
plug in
db ok
unplug

Right now I would say that everything works without problems, but I will do some more tests.

Fantastic.

What exactly has changed?

Cheers, Joe

[tshering]

Originally Posted by jbaumann

And here the latest test:

Spoiler:

plug in
db ok
copy number of books with calibre
db ok
unplug
process books
plug in
db ok
unplug

Right now I would say that everything works without problems, but I will do some more tests.

Fantastic.

If this test result is representative, it is fantastic. But is it? I would like to know how the general success/failure rate was with the previous and current KSM versions. I saw two or three people reporting that KSM corrupts the database (apart from those cases where several things went wrong so that nickel would exit almost immediately after starting). Compared to the downloads of the KSM package, this would suggest, that the issue was not very much experienced. But of course, I have no idea how many people who download the package actually install it and keep KSM, or report issues, or how many users use the reader in a way that increases the chanced of this issue occurring.
I guess what I want to say is that I am glad that it works for you now, but that this might be a case of good luck.
I would like to underline that the scenario of the latest test (connecting while nickel is running) is the one that is important for our case. I do not care so much at the moment whether simulating an usb connection is reliable.

Originally Posted by jbaumann

What exactly has changed?

All executable files of KSM are now on the system partition (and not on the user partition) so that a clean unmount is possible. I knew that I should have done this from the beginning, but I treated it for the possibility that users can simply do changes to KSM without having to have easy access to the system partition.

Thank you for your tests again! I am curious about your further experiences.

[Lilarcor]

I do had the problem since I tried ksm06 , I gave it up again while I used ksm07. Now I will wait for ksm08

[jbaumann]

Next test:

Spoiler:

Reboot
KSM
Continue to Nickel
plug in (Nickel)
db ok
copy lots of books with calibre
db OK
unplug
process
plug in (Nickel)
db ok
copy loads and loads of books with calibre
db ok
unplug
process
plug in
db ok

I slowly start to believe that it works. I copied a very large number of books (nearly 500) to the reader and everything worked fine. I will test some more in the coming days but for the moment I assume that it works. Yay :-)

So, a few small practical questions:
- Can I delete .adds/kbmenu?
- How do I configure that clicking on the koreader icon the koreader program is actually started directly? And how do I configure that on leaving koreader Nickel is automatically restarted? I'm not really clear on the different options here.

Cheers, Joe