Glo Triggering a library refresh automatically

[davidfor]

Originally Posted by kido.resuri

What trigger could be used? How to (easily) check if the book is added?

Off the top of my head I don't know. What I do know is that when a book is added by the firmware there are multiple inserts into the content, volume_content, Event, AnalyticsEvents and Activity tables. A downloaded book can also get rows in content_keys, volume_tabs, Reviews, Bookmark and ShelfContent (the last two will mean the book has been on a Kobo device or app before).

Potentially, there would need to be a trigger for each of these tables. They would probably be "BEFORE INSERT" and fiddle with the various columns in the table which include the file path. The exact trigger wouldn't be hard to work out.

[kido.resuri]

Thanks, I don't know much about sql triggers, so it seems to be a good time to learn!

[kido.resuri]

I read about triggers, and according to that, the database change should be easy to do with sql triggers.
Executing something like this [b]untested[/u] sql script on boot should do the update when virtual sd card is mounted and nickel updates it's library.

!WARNING! THE FOLLOWING IS ONLY THEORETICAL! IT WAS NOT TESTED ON A KOBO DEVICE! EDIT: It was tested. Buggy, but maybe safe.

This should be saved to a file "/root/library-virtual-sd.trigger"

-- usage:
--    $ sqlite3 /mnt/onboard/.kobo/KoboReader.sqlite < library-virtual-sd.trigger
DROP TRIGGER IF EXISTS update_library_virtual_content;
DROP TRIGGER IF EXISTS update_library_virtual_activity;

CREATE TRIGGER IF NOT EXISTS update_library_virtual_content AFTER INSERT ON content
BEGIN
 UPDATE content SET ContentID = REPLACE(ContentID,'/mnt/sd/virtual/','/mnt/onboard/virtualSD/'), BookID = REPLACE(BookID,'/mnt/sd/virtual/','/mnt/onboard/virtualSD/') WHERE rowid = new.rowid;
END;

CREATE TRIGGER IF NOT EXISTS update_library_virtual_activity AFTER INSERT ON Activity
BEGIN
 UPDATE Activity SET Id = REPLACE(Id,'/mnt/sd/virtual/','/mnt/onboard/virtualSD/') WHERE rowid = new.rowid;
END;

Then a script containing the following should be executed on boot once:

sqlite3 /mnt/onboard/.kobo/KoboReader.sqlite < /root/library-virtual-sd.trigger

The ebook files should be placed in /mnt/onboard/virtualSD/ and each newly added ebook should have a link copy in /mnt/onboard/.mysd/
Then mount the sd using the previously discussed method:

mount --bind /mnt/onboard/.mysd/ /mnt/sd/virtual
echo sd add /dev/mmcblk1p1 >> /tmp/nickel-hardware-status

I didn't test this yet, but in theory it should work.

In this case, /mnt/onboard/.mysd/ is a temp dir for links copies of the new content. It should be cleared on every boot and/or by the script that copy the content to the eReader, so only new content is added to the library. Using a dir in /mnt/onboard/ for the real files should be ok, as the content is already in the library, so nickel shouldn't add them on next USB connection. These are all untested yet.


EDIT: I tried the trigger offline, with SQLite Expert Personal, with this insert:

INSERT INTO content (ContentID,BookID,ContentType,MimeType,___UserID) VALUES ('file:///mnt/sd/virtual/testbook2.epub','/mnt/sd/virtual/testbook2.epub',6,'application/epub+zip','adobe_user');

The trigger works, and changes /mnt/sd/virtual/ to /mnt/onboard/virtualSD/ as expected. It seems that the trigger needs to be added only once, it will remain in the database until removed. Still need to be tested on Kobo.

[kido.resuri]

I tried it on Kobo. The ebook file needs to be duplicated, symlink can't be used. Sadly, the database caching of nickel prevents it to work properly. The newly added book cannot be opened untill next restart. Needs some more work..

[kido.resuri]

So far, so good. I experimented with this method a bit.
First of all, let me explain (if it was not clear yet) why I want to update the database entries for virtual sd card library content.
- Content loaded from SD (let it be real or virtual) is only available while the SD is loaded. A vritual SD should be loaded on every boot for its content to be accessible anytime.
- When a real SD card is inserted while a virtual one exists, the virtual SD's loaded content should be removed from the library.
- Nickel must process the SD each time it is inserted (real or virtual), making it slow startup if the virtual is mounted on every boot (or whenever).
- Such content should loose it's Shelf (Collection) associations (I'm not sure about this at all).
That's why I'm trying to solve this somehow.
As mentioned before, and much earlier in other threads, nickel only reads from the database when it's starting. The update can be easily done with the sql triggers, but nickel will still use the original path, even more, it will use this path to access the database and load the required files - which won't work, as the trigger already updated the values. So currently a reboot is the only solution. Nickel adds an Event table entry after the content is added to the database. This could be used to detect the addition by nickel, but shell scripts can't be run from sql triggers, so there must be some kind of shell script written which queries the changes for each newly added book. Maybe someone else finds another, better method to wait for nickel to finish reading content from the virtual SD card.

[davidfor]

@kido.resuri: Have you actually looked at what happens when you swap cards?

The way it works is:

• When you first insert a card, all books on it are added to the database. The "ExternalID" is set to the partition serial number for each book.
• If you eject the card and reinsert it, the device looks for changes. New books are added, missing books are removed and changed books (different file size) are removed and then treated as new books. Unchanged books keep their reading status, collections and bookmarks.
• If you insert a different card, all books that were on the previous SD card are removed from the database.
• If something does happen that there are books in the database for two different cards, only those for the current card (current External) are displayed.
• If there is no card, books with an External are not displayed in the lists.
• After a reboot, the external card is processed once nickel has restarted. (Note: for the Touch, this doesn't happen, you have to eject the card after nickel has started. This is a very long standing bug.)

If you are simulating an external SD card, you would not want to lose the reading status, bookmarks and collections for books on it. Or as you put in a real card.

[kido.resuri]

Thanks davidfor, I didn't test what happens when I swap cards, as I don't have a real sd card to test with. The books on sd loosing their collections, reading status, etc. was only a theory, and as i stated, I wasn't that sure about that part. I'm happy that they won't lose these. It should be tested what happens when the virtual SD is still mounted and a real card is inserted.

Anyhow, It would look a bit strange for the user if a real sd is inserted and the "whole" library would disappear (assuming all other books were transfered via the virtual sd method) untill the real sd is removed. I'm not going to use an sd card actually as it is inconvenient on the H2O, but someone else might want to.

[kido.resuri]

I'm testing with a real sd card, and it's behavior is not satisfying.
I can't tell how this works on other devices, but on a H2O, it works this way:
If I'm adding a real SD card, then it's content is processed by nickel. The new books appear in the library. I put some in collections, open them, make annotations, etc. If I remove the card and later insert it again, the collections, read status and annotations remain. If then I unplug it, and mount a virtual sd to /mnt/sd/ using:

mount --bind /mnt/onboard/.mysd/ /mnt/sd
echo sd add /dev/mmcblk1p1 >> /tmp/nickel-hardware-status

Then it's content is processed by nickel, the same applies as a real sd card. If I "unplug" it with these commands:

umount /mnt/sd/
echo sd remove /dev/mmcblk1p1 >> /tmp/nickel-hardware-status

Then the books are removed from the library. If I later mount the virtual sd, then all the read status, annotations, etc. remain as with the real card.
BUT! If I mount the real sd after the virtual, or the virtual after the real, all the read status, annotations, etc. are removed from both the virtual and real sd, so I can't mix them.
If I'm mounting the virtual one to any subdir in /mnt/sd/, than both sd's read status, etc. are not kept, also if the virtual sd is not removed before inserting the real sd then the read stats, etc. are removed from both.
If I'm updating the database when virtual SD is used, then everything is kept, and treated like any other sideloaded content - after a nickel restart.

If anyone else could try these on other devices, please post your experience!

[davidfor]

@kido.resuri: What you described is exactly what you should have expected based on what I described.

And it will work the same on all Kobo devices running the same firmware version. The firmware is the same. The only differences are in the kernel and drivers that are devices specific. And the Touch has some different fonts and graphics. If you unpack the firmware packages for a version and do a file comparison, the files are identical.

Having said that, what I wrote is how it has behaved for a long time. But, I did retest as I hadn't been using an external SD card for a while. My tests were largely with a Touch, but I did check them on a Glo and Aura H2O as well. All of them are running 4.2.8432.

[kido.resuri]

davidfor, then I misunderstood what you wrote. Thought you meant that the books' reading status, etc. on external SD cards are kept and loaded as each SD card is inserted, so one can safely change cards and keep on reading, etc. Reading your above post again now reveals what you meant. But also that is why I'm working on a workaround solution. I made sql triggers for each table in the database to log every change in the database made by nickel during an SD card insert, to find a way to catch the finish of it. I have it in my mind now, will work on it next if I have time. Basicly a script doing sql query in a loop until the newly added books appear in the database. After that, it will do a reboot, resultin in all new books will be sideloaded on the internal card.

[davidfor]

OK, I changed the third point in my post to be a bit clearer.

If you want to allow real SD cards to be used, you will need to move the files. But, Kobo doesn't seem to be putting external slots in the new devices, that might not be worth worrying about. Or write to the real card and let the user eject and insert it to process the books.

The other thing to consider is what happens when you connect to a PC. Will the device present the correct drives? Will calibre be able to find the books?

[anarcat]

thanks so much for all your research and feedback! I'm not sure where I stand with the database stuff - it seems to me hairy to start messing around in there...

let me know if i should change the summary to point to a solution, nevertheless...

Right now, I am still using the "fake-usb-connect" hack that pops up a dialog asking the user confirmation. it's ugly, but it works, but technically (the library is re-read) but also, and more interestingly, from a usability perspective: I tell people (in the documentation and in person) that it's a hack, and they just need to tap that and things will work. so it's kind of okay.

my next hurdle, at this point, is to fine-tune the *run* trigger, ie. when do i run my sync program. right now it triggers on any udev aactivity:

    KERNEL=="eth*", RUN+="/usr/local/bin/wallabako-run"
    KERNEL=="wlan*", RUN+="/usr/local/bin/wallabako-run"

... which is not so great. first, it also runs when the connection goes down (which is silly) and second, it requires the user to "turn wifi off and on again" which is klunky. the first reaction of the user, in my test here, was to tap that neat little "sync" tile on the home screen - is there a way to hook into that?

[anarcat]

note that the plato author showed me a few tricks that might be useful to others here, and that I want to include in wallabako:

Apparently, there's a file in /tmp/nickel-hardware-status, created in /etc/init.d/rcS that Nickel reads from. A few scripts in the Kobo distro also write to it:

/etc/udhcpc.d/default.script
/usr/local/Kobo/ntpd.sh
/usr/local/Kobo/udev/ac
/usr/local/Kobo/udev/plug
/usr/local/Kobo/udev/sd

The udev scripts are triggered by rules defined in /lib/udev/rules.d/kobo.rules. Plato reads from this FIFO to get notified when the device is plugged to or unplugged from a power source.

[NiLuJe]

We still haven't found a better way to rescan the Library than to fake an USBMS session, though (as the sdcard bind-mount trick may have some unwanted side-effects for people actually using their sd cards ).

(But on that note, you can now automate the "Connect" click either via a Nickel setting, or via FBInk. See UNCaGED for an example of this very use-case).

----

If you like doing things your way, you can also hook into udev directly, instead of relying on the Nickel FIFO .