How to trigger a book update on the Kobo...

[fastrobot]

Hi,


From a hacking perspective;

I've got scripts on my kobo which are automatically downloading/generating/translating ebooks while the Kobo is NOT plugged into USB , and putting them in the normal place on /mnt/onboard as if I had copied them from my linux PC over to the reader; However, not even rebooting will cause the Kobo reader to "discover" these new books and allow me to read them. I have to plug in the USB cord, mount the reader on my PC, and unmount it -- before the reader suddenly detects them.

That stinks, as it means I have to do the hokey pokey, disable telnet -- unload modules -- etc. etc. etc. before I can make all that happen.

How can I fool the reader into thinking I have plugged the USB cord in, mounted the reader, and unmounted it -- using a shell script / without physically doing any of it?

Is there a simple way?

[boriar]

KSM by tshering have a script to simulate the plug and triggered base update. You can give it a try
That's the code:

echo usb plug add >> /tmp/nickel-hardware-status
sleep 5
echo usb plug remove >> /tmp/nickel-hardware-status

It work for me

[fastrobot]

Odd, I must be jinxed.

I upload a pdf onto the reader through telnet, and excute the script lines you show.
The "computer" connection popup happens on the kobo, and refuses to go away.
So I say connect...
and then run the sleep and disconnect, but it still says connected to computer.
So eventually, I just pull the usb cord out.

I go to 'library' on the Kobo, and not only is the new pdf/book not there -- but it says 'library empty', and all my normal books are gone.

So, I press "sync", and it goes through some motions and all my normal library DRM books reappear, but the pdf I uploaded (Python_Programming_eBook_reader.pdf) does not show up in my library. It's a normal unprotected wikimedia commons pdf, and I can view it on my PC without issues -- so it's not a DRM protection issue.

What am I missing?

I can plug in and unplug the USB cable, which is all the script is supposedly trying to emulate; but that doesn't make it find the book either.

I've not noticed the Kobo readier doesn't findi new books unless I *mount* the kobo reader, and unmount it from my PC -- which I can't do while the telnet client is installed over USB.
Is there a way to simulate 'mounting the reader ?

or perhaps, running sqlite on the database in .the /mount/.kobo directory, and adding some kind of entry that lets it know the system needs to be scanned ?

My external SD card, I know when I run CFDISK on it, and save the partition table -- that Kobo erroneously thinks it was ejected and reinserted, and will scan and mount the first partition again.
So, I know that there is probably a way to fake a udev or dbus event -- which causes the reader to believe that /mnt/onboard was unmounted and remounted....

I can probably solve it in a few days, but I was just thinking that someone else has probably already solved this issue in the past, and asking might be quicker than the school of hard knocks....

[frostschutz]

It's possible that it won't work with a USB cable connected. Maybe you could strace to find out how nickel detects this and then do something about it. I telnet through wifi. But still it's only half a solution since you have to confirm on the touchscreen. Although that could have its advantages.

Another way is to emulate ext. SD but your books have to be on ext. SD (real or emulated) for that. This does not exist for /mnt/onboard since that card can't be taken out (and the int. cardreader has no media change detection, for my magic memory upgrade mod which changes internal media I had to cheat, but no way nickel would handle).

Maybe you can convince nickel to run the database upgrade code path with a ld library preload script somehow, what the nightmode does; if there is a better way to trigger this from the outside, I don't know of it.

> it says 'library empty', and all my normal books are gone.

Connecting to PC must umount the filesystems on the Kobo; if you're using telnet and you have anything running still working off the mounted fs, you might see filesystem corruption due to failed umounts and such.

[boriar]

Well, I rarely use telnet and never when use that script. I use it with my sideload png and epub and worked like a charm. Can it be caused by telnet running?

You can try that one (not mine, author is tshering). That one is used to mount and unmount from KSM like it was nickel:

#!/bin/sh

versionfilename=/mnt/onboard/.kobo/version

#PRODUCT_ID is already exported by other script
VERSION=$(awk -F"," '{print $3}' $versionfilename)
SN=$(awk -F"," '{print $1}' $versionfilename)

MODULE_LOADED=`lsmod | grep -c g_file_storage`


VENDOR_ID=0x2237

DEV="/dev/mmcblk1p1"
if [ -e /dev/mmcblk1p1 ]; then
  LUNS=/dev/mmcblk0p3,/dev/mmcblk1p1
else
  LUNS=/dev/mmcblk0p3
fi

PARAMS="vendor=$VENDOR_ID product=$PRODUCT_ID vendor_id=Kobo product_id=eReader-$VERSION SN=$SN"
sync
echo 3 > /proc/sys/vm/drop_caches
#umount -l /mnt/onboard
umount -l /mnt/sd

/sbin/insmod /drivers/$PLATFORM/usb/gadget/arcotg_udc.ko
sleep 2

/sbin/insmod /drivers/$PLATFORM/usb/gadget/g_file_storage.ko file=$LUNS stall=1 removable=1 $PARAMS
sleep 1

[tshering]

This is how the pretend_usb_connect_and_unconnect script is supposed to work:

• the device must NOT be connected via USB
• run the script
• select quickly! (or increase the sleep value in the script) "connect"
• wait until the pop up closes, the "connected and charging" message appears, and disappears, ...

I just tested this again by downloading a file from www.gutenberg.org via wifi, then starting nickel and running the script.

If nickel fails for some reason to read new files into the database (this can also happen if you use it the regular way), nickel might refuse to read them at further trials too. Touching those files helps (sometimes?).

[Ken Maltby]

@tshering; I wonder if a "silent" version of that, at the very beginning of the KoboRoot.tgz to install the fmon icons, might make the install a little more "fool" proof? It seems that people are still having problems installing working fmon .pngs.

Luck;
Ken

[tshering]

Originally Posted by Ken Maltby

@tshering; I wonder if a "silent" version of that, at the very beginning of the KoboRoot.tgz to install the fmon icons, might make the install a little more "fool" proof? It seems that people are still having problems installing working fmon .pngs.

Luck;
Ken

Ken, I guess I am missing your point. Could you please elaborate a little on this. As for a "silent" version, I am not sure whether this could be easily done.

[Ken Maltby]

Originally Posted by tshering

Ken, I guess I am missing your point. Could you please elaborate a little on this. As for a "silent" version, I am not sure whether this could be easily done.

Right now we try to have the user copy the .png files, into a place that will be visible to the Kobo/nickel processing, then have them safely eject the USB connection - where upon the processing should occur. While this should be working, somehow people still end up with non-working icons/pngs.

As long as we need to use a KoboRoot.tgz to complete the process of installing working icons anyway, and you have the ability to "trigger" the nickel processing, couldn't you properly place the .png files and trigger the nickel processing, before conducting the rest of the installation that the KoboRoot.tgz was originally doing. That way it could be done properly, for the user. Eliminating a potential user error.

Luck;
Ken

[fastrobot]

Originally Posted by tshering

If nickel fails for some reason to read new files into the database (this can also happen if you use it the regular way), nickel might refuse to read them at further trials too. Touching those files helps (sometimes?).

Thank you so much for warning me -- I know I would have managed to waste hours on this otherwise. I will make sure to rename the pdf file to something else before trying again.... There's nothing worse than to a make a mistake once and have it stuck on a bug forever and not knowing about it.

[davidfor]

Originally Posted by fastrobot

Thank you so much for warning me -- I know I would have managed to waste hours on this otherwise. I will make sure to rename the pdf file to something else before trying again.... There's nothing worse than to a make a mistake once and have it stuck on a bug forever and not knowing about it.

I'm not sure that tshering is right, or at least I don't know what the mechanism is. As far as I can see, if you put a book on the device when it is connected to the PC, it will be processed on disconnect. It will be processed each time processing is done until the book is added to the database or it is removed. And until it is successfully processed, it blocks other books from being processed.

There might be a timestamp involved. That would explain why a touch had the books processed. If so, I don't know where it is. There is a "LastSyncTime" in the config file, but I have always assumed that is for the server sync. "__SyncTime" in the content table is the time the book was processed, so it might just be the maximum value of that. Or it could be in memory or on the root partition somewhere. In any case, it shouldn't be hard to come up with a test case for this.

One thing that hasn't been mentioned is that downloading a book with the browser will get it added to the library. From memory, the browser is running as a separate process, so there has to be some communications between it and nickel. If you can work out what that is, it might be usable.

Looking at the strings file I produced from libnickel, I can see:

void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&)
void SyncFileSystemCommand::pruneSideLoadedFiles(QStringList&, const QList<QString>&, QVector<Volume>&)
void SyncFileSystemCommand::parseFB2(const QString&) const
void SyncFileSystemCommand::parseKepub(const QString&) const
void SyncFileSystemCommand::parseTextDocument(const QString&)
virtual void SyncFileSystemCommand::execute()

I think the second one is used to check if the known sideloaded books are still there or have changed. But, the first one looks interesting. And the message, "BUG! Someone ran findFilesToParse with a wrong folder path:" suggests it is the right method.

Of course, you could always add the book directly to the database. You have to parse the metadata and ToC and build the rows, but it is possible. I wrote some code a long time ago to do it in the calibre driver, but was never happy with it, so decided to leave it to the device. I think I understand the database better now, and do have code to update the ToC for some books.

And a last thought, as you have devices with external SD cards, you could put the books there and unmount/mount it somehow. Or fake it using a directory on the internal card. Someone did something like this for a google drive sync. I don't know if it still works or not.

[tshering]

Originally Posted by davidfor

I'm not sure that tshering is right, or at least I don't know what the mechanism is.

Nor am I sure about this. I said "nickel might refuse to read them at further trials too. Touching those files helps (sometimes?)" in order to prevent possible troubles, but did not intend to make any strong claims. On touching as a means to get files processed by nickel that otherwise would not be processed you might cf. the two post by PeterT in this thread. By own sporadic trials with touching and overwriting files were not unequivocally successful however.

[davidfor]

Originally Posted by tshering

Nor am I sure about this. I said "nickel might refuse to read them at further trials too. Touching those files helps (sometimes?)" in order to prevent possible troubles, but did not intend to make any strong claims. On touching as a means to get files processed by nickel that otherwise would not be processed you might cf. the two post by PeterT in this thread. By own sporadic trials with touching and overwriting files were not unequivocally successful however.

Sorry about that, I didn't read your post properly.

In any case, I did a quick test. I put an epub in the .kobo directory so it wouldn't get processed. Then I telneted in and copied it to the root and then used touch to set the date to last year. Then a connect and disconnect and it was processed. A single test isn't conclusive, but it agrees with all the behaviour I have seen.

[fastrobot]

Originally Posted by davidfor

One thing that hasn't been mentioned is that downloading a book with the browser will get it added to the library. From memory, the browser is running as a separate process, so there has to be some communications between it and nickel. If you can work out what that is, it might be usable.

Working directly with the database is a possibility; I have installed sqlite3, but as nickel reads and writes to the database and keeps it open all the time -- there is a certain amount of danger in stopping nickel when it might happen to be accessing the database, and then corrupting it. Nickel does re-read the database before writing, so the odds of corruption are probably small -- but it would be nice to have a better solution.

The web browser is likely a safer approach....


I launched nickel with strace -t -o /mnt/home/ni.log /user/local/Kobo/nickel -platform kobo -skipFrontLoad to follow your advice.

Nickel runs, and then I launched the web browser to get a random book; eg:
http://azardi.infogridpacific.com/resources.html
the "recollections of war" book at the top, and then said view it.

And now the bad news: The first instance of "recoll" in the strace, is as a string already in memory:

20:38:18 stat64("/mnt/onboard/recollections-of-wartime.epub", 0x7ea1e300) = -1 ENOENT (No such file or directory)
20:38:18 open("/mnt/onboard/recollections-of-wartime.epub", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE|O_CLOEXEC, 0666) = 26
20:38:18 fcntl64(26, F_SETFD, FD_CLOEXEC) = 0
20:38:18 brk(0x3066000)                 = 0x3066000
20:38:18 brk(0x30b2000)                 = 0x30b2000
20:38:18 mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3b023000
20:38:18 mremap(0x3b023000, 1052672, 2101248, MREMAP_MAYMOVE) = 0x3b023000
20:38:18 mremap(0x3b023000, 2101248, 4198400, MREMAP_MAYMOVE) = 0x3b023000
20:38:18 mremap(0x3b023000, 4198400, 8392704, MREMAP_MAYMOVE) = 0x3b023000
20:38:18 mremap(0x3b023000, 8392704, 16781312, MREMAP_MAYMOVE) = 0x3b023000
20:38:18 mremap(0x3b023000, 16781312, 33558528, MREMAP_MAYMOVE) = 0x3b023000
20:38:18 brk(0x2392000)                 = 0x2392000
20:38:18 fstat64(26, {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
20:38:18 write(26, "PK\3\4\24\0\0\0\0\0\362\n0Coa\253,\24\0\0\0\24\0\0\0\10\0\0\0mi"..., 29676824) = 29676824
20:38:18 munmap(0x3b023000, 33558528)   = 0
20:38:18 close(26)                      = 0

The name wasn't passed in to nickel from outside, or read from a socket or anything.
I only see occasional netlink calls, and a continual polling of descriptors 6,11,12 via pselect6 before this point in the strace, but no reads from a file descriptor that supply a name....

So, I think there might be shared memory being used between nickel and the web-browser (I'm not very familiar with strace, but this looks like threaded operation or shared memory.)

Since this is a trace of an open for 'write', then I assume we are seeing the web browser save the file via. nickel using shared memory.

The next time the filename gets mentioned, it's being writev() to a file descriptor 35 which was never opened by nickel anywhere in the strace.

20:39:13 writev(35, [{"l\1\0\1\32\0\0\0\f\0\0\0\177\0\0\0\1\1o\0\25\0\0\0/org/fre"..., 144}, {"\25\0\0\0com.kobo.adobe.Reader\0", 26}], 2) = 170
20:39:13 read(35, 0x163fa48, 2048)      = -1 EAGAIN (Resource temporarily unavailable)
20:39:13 writev(35, [{"l\1\1\1\210\0\0\0\r\0\0\0\177\0\0\0\1\1o\0\25\0\0\0/org/fre"..., 144}, {"\203\0\0\0type='signal',sender='org.fr"..., 136}], 2) = 280
...
20:39:14 writev(35, [{"l\1\0\0016\0\0\0\24\0\0\0g\0\0\0\1\1o\0\1\0\0\0/\0\0\0\0\0\0\0"..., 120}, {"1\0\0\0file:///mnt/onboard/recollec"..., 54}], 2) = 174

It's also notable that it's usrd as a url to the file, and several adobe style sheets open immediately after it, so this looks like the adobe reader opening the file. There were also several signals sent over a socket, with send(), so this looks like a GUI interface.



Unless the web browser's source code is in the respository, this looks pretty hopeless to me. It's doing it all internally, and not over kernel calls.

To be able to do something like that, will probably require me to to write a plugin that talked to nickel and passed it a file-name via shared memory. I think there are some poker plugins for nickel in the repository, so there might be enough information to trick it into looking for an epub that was supposedly 'downloaded' even if it wasn't. But what little I have found on writing plugins for nickel, basically says it's nearly impossible because its so badly documented.

But now the really good news:

I can see that after I download that random epub book using the web server, that nickel suddenly found my thinkpython.epub book that it had been stuboornly ignoring up and until now, especially with the USB cable plugged in and telnet running.

So, there is a fairly easy and reliable way to trick the kobo into finding new books; eg:

Put a file on / called getbooks.html that looks like:

<HTML>
<A href="file:///refresh.epub">refresh my booklist now</A>
</HTML>

and then just add a dummy epub book called refresh.epub, which I will simply delete every time I 'upload' it.... or write a shell script to generate the web page dynamically, with a link to the new book already on the hard drive, but under a different directory...

#!/bin/sh
echo "<HTML><A href=\"file:///$1\">refresh my booklist now</A></HTML>"  > /getbook.html
#rm $1  # optionally remove original book as it will be copied to the /mnt/onboard

Just type, file:///getbook.html into the web browser's address bar... tap on refresh my booklist now -- and it works!

[frostschutz]

This parsing and stuff is done by an instance of /usr/local/Kobo/adobehost(-launcher.sh), it has three modes (reader, parser, search). That's probably your fd 35. You might get a more complete picture if you kill adobehost first (nickel re-starts it as needed) and then strace -ff nickel.

Interesting idea with the browser.

Modifying the database: dunno if it works for books, it does not for collections, nickel does not re-read until it leaves USB host mode. For the books table I know it changes rowIDs of books around a lot (basically everytime you open a book it gets a new rowID) and I don't know what assumptions nickel makes about those so... I'd rather not risk it.