Offline Wikipedia on Kobo (Guide)

[nhedgehog]

Originally Posted by ams

@nhedgehog: in Library.xml, I needed to replace indexPath="/index/..." with indexPath="./index/..." to make full text search work (using the directory structure from your example)

Ups, thanks for the feedback, I changed it. New tutorial is prepared (decided to keep kiwix folder structure and change the library.xml location in wikipedia.sh to:
file1="/mnt/onboard/wikipedia/library/library.xml"
to minimize errors with the libraries. New folder structure, so that the path in library.xml must not be changed:

\wikipedia\content\gutenberg_de.zim
\wikipedia\content\rezeptewiki_de.zim
\wikipedia\library\library.xml
and if fulltext search is wished::
\wikipedia\index\rezeptewiki_de.zim.idx\some_index _files_here

but I'm waiting for editing privileges on https://www.e-reader-forum.de/ebook-...anderen-kobos/

[tshering]

Originally Posted by ams

@tshering: thanks, using your suggestion in #56 I got it running. Is there an easy way to automatically run the 'start wikipedia.sh' each time KSM boots?

Maybe it would be better to start the Kiwix server only when starting nickel, and killing it when exiting nickel. I was planning to create a folder for scripts to be run before launching nickel, and another folder for scripts to be run after exiting nickel, anyway.

One problem I see is that the server is run from the user partition, which can be dangerous when connecting to the pc via usb. I guess, KSM refuses to connect via usb, after you start the server.
Moving kiwix-serve to the system partition might also be a problem, since space is very limited. (I guess the ifconfig provided of the package is not necessary, the onboard version will do.)

For the time being I would start and kill ("killall kiwix-serve") manually.

[tshering]

Originally Posted by nhedgehog

@tshering:
Thanks for the clarification. I wasn't aware that KSM is manipulating the rcS too. ifconfig needs root to run. Do KSM-user scripts have root permissions on the Kobos?

Yes they have.

Originally Posted by nhedgehog

I will update the tutorial later to avoid accidental KSM stops.

KSM users should install the KoboRoot.tgz by selecting "install partial update" (rather than "install update"), this prevents rcS to be overwritten. And then they need a way to start the server (and maybe also to kill it; see the previous post.)

[tshering]

@nhedgehog
I finally installed your package (without rcS and ifconfig) and the cook book wiki on my Touch FW 4.6.9995. The server was starting and I did not get any error message in the log file. However, nothing appeared in the web browser. I thought I might just have done some typo with the index file, and did not worry about it (I had no intention to read the cook book anyway).

Then something different attracted my attention. Since the file size of kiwix-serve is a problem for KSM users (if they want to put it on the system partition), I looked for a smaller one. I took the one from http://wiki.kiwix.org/wiki/Software, and all of a sudden I could see the cook book wiki in the web browser. I tried several times with your and the wiki.kiwix.org's version. Only the wiki.kiwix.org's version worked for me.

[nhedgehog]

Strange, I know this version, it is quite old but much smaller than the newer one. I used it on my Bookeen device before. Don't know the differences between the versions.
I guess due to the smaller RAM size from the Kobo Touch it is working better on that device. My version comes from the nightly builds (forgot to mention it):
http://download.kiwix.org/nightly/
kiwix-tools_armhf*- I took the version 2017.10.13
They didn't compile the arm version for some years but started compiling it again this year.
But you did configure the loopback interface via:

ifconfig lo 127.0.0.1 up

or not?

Edit: I'm using the version I provided on my Touch HD reading ebooks only with Coolreader but starting into Nickel for some weeks now (FW.3.19.5761). I did not experience any negative issues coming from runnig kiwix-serve in the background.

[tshering]

Originally Posted by nhedgehog

But you did configure the loopback interface via:

ifconfig lo 127.0.0.1 up

or not?

Is this a serious question?

[nhedgehog]

Originally Posted by tshering

Is this a serious question?

Yes since I don't know if this is activated on the glow devices. I did not find ifconfig on my Glo HD, thats why I integrated it. I am new to Kobo devices, Bookeen and Pocketbook had it disabled too.

[tshering]

Originally Posted by nhedgehog

Edit: I'm using the version I provided on my Touch HD reading ebooks only with Coolreader but starting into Nickel for some weeks now (FW.3.19.5761). I did not experience any negative issues coming from runnig kiwix-serve in the background.

The problem is that the user partition cannot be cleanly unmounted in this situation. Users of the Kobo Utilities Plugin for calibre are most likely to see the effect of this (a corruption of the data base).

[tshering]

Originally Posted by nhedgehog

Yes since I don't know if this is activated on the glow devices. I did not find ifconfig on my Glo HD, thats why I integrated it. I am new to Kobo devices, Bookeen and Pocketbook had it disabled too.

I just thought it is obvious that I did it, since I reported that I had success with the old/small version of kiwix-serve. ifconfig is provided by busybox, and pointed at by /sbin/ifconfig.

[nhedgehog]

Originally Posted by tshering

The problem is that the user partition cannot be cleanly unmounted in this situation. Users of the Kobo Utilities Plugin for calibre are most likely to see the effect of this (a corruption of the data base).

Thats not something to neglect.
I don't use the Kobo Utilities Plugin for calibre, so I guess that helped.
What do you suggest?
Adding some safety measure into /usr/local/Kobo/udev/usb or is this called to late?

[tshering]

Originally Posted by nhedgehog

Thats not something to neglect.
I don't use the Kobo Utilities Plugin for calibre, so I guess that helped.
What do you suggest?
Adding some safety measure into /usr/local/Kobo/udev/usb or is this called to late?

I think this should work. KSM however prevents this file from being overwritten when the user is updating by "install partial update." I will add your code to KSM when it is ready.

[nhedgehog]

I did change the installation file and managed to stop and start kiwix-serve during USB connect. Since "/usr/local/Kobo/udev/usb" changed during recent updates, there are 2 installation files now, depending on the firmware version. If the wrong version is installed, USB connection ist lost. To prevent accidental bricking of devices I wrote an extensive tutorial in German and English. Hope that someone will find it useful. Went back to the older version of kiwix-serve, since I couldn't find any flaws in it and it seems to be supporting more devices. Everything can be downloaded at:
http://bit.ly/2zlYaCL

[h87booky]

Hi, here's my solution.

I use the excellent webportal local web server plugin. https://www.mobileread.com/forums/sh...d.php?t=262353
I put all Kiwix binaries and zim files in a ext4 second partition on a 128 Go SD-Card (yes, you can, never tried 256 Go because I don't have one).
The Kobo browser is patched to work offline.

With telnet or ssh

# mkdir /mnt/wikipedia

The content of /usr/local/Kobo/udev/sd (to mount the wikipedia partition)

#!/bin/sh

if [ "$STARTUP" == "1" ]; then
  exit;
fi

# check whether nickel is running
[ "$(pidof nickel | wc -w)" != "0" ] && nickelisrunning=true || nickelisrunning=false

MODULE_LOADED=`lsmod | grep -c g_file_storage`
if [ $MODULE_LOADED -gt 0 ]; then
  if [ "$ACTION" == "remove" ]; then
    [ "$nickelisrunning" == "true" ] && echo sd $ACTION $DEVNAME >> /tmp/nickel-hardware-status
    ACTION=remove /usr/local/Kobo/udev/plug
    sleep 5
    exit
  fi
fi

DEV="/dev/mmcblk1p1"
MOUNTARGS="noatime,nodiratime,shortname=mixed,utf8"

if [ "$DEV" == "$DEVNAME" ]; then
  if [ "$ACTION" == "add" ]; then
    dosfsck -a -w $DEVNAME
    mount /dev/mmcblk1p2 /mnt/wikipedia
    mount -r -t vfat -o $MOUNTARGS $DEVNAME /mnt/sd
    if [ $? != 0 ]; then
      [ "$nickelisrunning" == "true" ] && echo sd mount fail $DEVNAME >> /tmp/nickel-hardware-status &
    else
      [ "$nickelisrunning" == "true" ] && echo sd $ACTION $DEVNAME >> /tmp/nickel-hardware-status &
      [ -e "/mnt/sd/autoscript.sh" ] && sh /mnt/sd/autoscript.sh
    fi
  fi
fi

if [ "$ACTION" == "remove" ]; then
  [ "$nickelisrunning" == "true" ] && echo sd $ACTION $DEVNAME >> /tmp/nickel-hardware-status &
  umount -l /mnt/sd
  umount -l /mnt/wikipedia
  sync
fi

The content of my "kiwix" file inside webportal/cgi-bin/

#!/bin/sh

# --- Constants: ---

TMPFILE=/tmp/servicemenu

ZIMEXT=zim
ZIMFOLDER=/mnt/wikipedia/
KIWIXM=/mnt/wikipedia/kiwix-manage
KIWIXS=/mnt/wikipedia/kiwix-serve
WIKIXML=/mnt/wikipedia/wiki.xml

# --- Helpers: ---

_chunk()
{
    CHUNK="$1"
    printf "%X\r\n" ${#CHUNK}
    printf "%s\r\n" "$CHUNK"
}

# htmlspecialchars
_htmlspecialchars()
{
    echo -n "$1" | sed -r -e 's@&@\&amp;@g' -e 's@[""]@\&quot;@g' -e 's@<@\&lt;@g' -e 's@>@\&gt;@g'
}

# urlencode
_urlencode()
{
    echo -n "$1" | hexdump -v -e '/1 "%02x"' | sed -r -e 's@..@%&@g'
}

# parse a=b&c=d data
_parse()
{
    PREFIX=$1
    DATA=$2

    while [ "$DATA" != "" ]
    do
        FIELD="`echo "$DATA" | cut -d '&' -f 1`"
        DATA="${DATA:$((${#FIELD}+1))}"

        key="`echo "$FIELD" | cut -d '=' -f 1 | sed -r -e s@[^a-zA-Z0-9_]@_@g`"
        value="`echo "$FIELD" | cut -s -d '=' -f 2-'`"

        if [ "$key" != "" ]
        then
            eval "$PREFIX""$key"'="`httpd -d "$value"`"'
        fi
    done
}

# print header
_header()
{
    echo -n -e 'HTTP/1.1 200 OK\r\n'
    echo -n -e 'CONTENT-Type: text/html; charset=UTF-8\r\n'
    echo -n -e 'Transfer-Encoding: chunked\r\n'
    echo -n -e 'Cache-Control: no-cache, must-revalidate\r\n'
    echo -n -e 'Expires: Thu, 01 Jan 1970 01:10:00 +0100\r\n'
    echo -n -e '\r\n'
    # echo -n -e 'CONTENT-Type: text/html; charset=UTF-8\r\n\r\n'
    _chunk "`printf "%4096s" " "`"
    _chunk '<html><head><style type="text/css">'

    _chunk 'input, textarea, select {
border: 1px solid #555;
padding: 0.5em;
font-size: 15px;
line-height: 1.2em;
width: 95%;
background: #fff;
background: -webkit-gradient(linear, left top, left bottom, from(#fff), to(#ccc));
-webkit-appearance: none;
-webkit-box-shadow: 1px 1px 1px #fff;
-webkit-border-radius: 0.5em;
}
'

    _chunk '
#progress {
  margin: 0px auto;
  width: 60%;
  height: 30px;
  border: 8px double #222;
  overflow: hidden;
  background: #fff;
}
#progressbar {
  width: 0%;
  height: 30px;
  border-right: 4px solid #000;
  background: #aaa;
  position: relative;
  text-align: center;
  font-size: 25px;
}'

    _chunk '</style></head><body>'
    _chunk '<h1>Kiwix Menu</h1>'
}

# print footer
_footer()
{
    _chunk '<hr>'

    # extra links
    query="$1"
    shift
    text="$1"
    shift

    while [ "$text" != "" ]
    do
        _chunk '<a href="?'$query'">'$text'</a> | '
        query="$1"
        shift
        text="$1"
        shift
    done

    # standard footer
    _chunk '<a href="?">Back to Kiwix Menu</a>'

    _chunk '</body></html>'
    _chunk '' # terminator
    exit
}

# print error message
# argument: msg
_die()
{
    _chunk "<p>Error: $1</p>"
    _footer
    exit
}

# print yesno dialog
# argument: question yesquery noquery
_yesno()
{
    _chunk "<p> $1 </p>"
    #_chunk '<a href="?'$2'">Yes</a> <a href="?'$3'">No</a>'
    _footer "$2" Yes "$3" No
    exit
}

# print progress bar
_progressbar()
{
    TOTAL="$1"
    CURRENT="$2"

    percent=$(($CURRENT*100/$TOTAL))

    _chunk '<div id="progress"><div style="width: '$percent'% !important;" id="progressbar">'$percent'%</div></div>'
}

# --- Views: ---

_view()
{
    case "$GETaction" in
        "")
            _view_list
            ;;
        "info")
            _view_info
            ;;
        "kiwixon")                                                                        
            _view_kiwixon                                                    
            ;;
        "kiwixoff")                                                                        
            _view_kiwixoff                                                                                                                                                       
            ;;
        "kiwixupdate")                                                                        
            _view_kiwixupdate                                                                                                     
            ;;
        *)
            _die "Unknown action: $GETaction"
            ;;
    esac
}

_view_list()
{
    _chunk '<ul>'
    _chunk '<li><a href="?action=info">Info</a></li>'
    _chunk '</ul>'
    _chunk '<br/>'
    _chunk '<br/>'
    _chunk '<ul>'
    _chunk '<li><a href="?action=kiwixon">Start Kiwix Server</a></li>'
    _chunk '<br/>'
    _chunk '<li><a href="?action=kiwixoff">Stop Kiwix Server</a></li>'
    _chunk '<br/>'
    _chunk '<li><a href="?action=kiwixupdate">Update Kiwix Library</a></li>'
    _chunk '</ul>'
    _chunk '<br/>'
    _chunk '<br/>'
}

_view_info()
{
    _chunk '<h4>CPU</h4>'
    _chunk "<pre>$(_htmlspecialchars "$(cat /proc/cpuinfo)")</pre>"

    _chunk '<h4>RAM</h4>'
    _chunk "<pre>$(_htmlspecialchars "$(free -m)")</pre>"

    _chunk '<h4>HDD</h4>'
    _chunk "<pre>$(_htmlspecialchars "$(df -h)")</pre>"

    _chunk '<h4>WIFI</h4>'
    _chunk "<pre>$(_htmlspecialchars "$(iwconfig)")</pre>"
    _chunk "<pre>$(_htmlspecialchars "$(ifconfig)")</pre>"
}

_view_kiwixon()
{
    if pidof -s kiwix-serve > /dev/null; then
        _chunk "Kiwix is already running!"
    else
    _chunk "<p>Kiwix starting...</p>"
        $KIWIXS --daemon --port=8888 --library $WIKIXML
        sleep 5
        if pidof -s kiwix-serve > /dev/null; then
            _chunk "Kiwix has started!"
        else
            _chunk "Error!"
        fi
    fi
}

_view_kiwixoff()
{
    if pidof -s kiwix-serve > /dev/null; then
        _chunk "<p>Kiwix stopping...</p>"
        killall -q kiwix-serve
        sleep 5
        if pidof -s kiwix-serve > /dev/null; then
            _chunk "<p>Retrying...</p>"
            killall -q kiwix-serve
            sleep 5
        else
            _chunk "<p>Kiwix has stopped.</p>"
        fi
    else
        _chunk "<p>Already stopped.</p>"
    fi
}

_view_kiwixupdate()
{
    _chunk "<p>Please wait...</p>"
    rm -f $WIKIXML
    for ZIMCONTENT in $ZIMFOLDER*; do
        if [ "${ZIMCONTENT}" != "${ZIMCONTENT%.${ZIMEXT}}" ];then
            $KIWIXM $WIKIXML add $ZIMCONTENT
        fi
    done
    _chunk "<p>Done.</p>"
}

# --- Main: ---

_parse GET "$QUERY_STRING"
read POST_STRING
_parse POST "$POST_STRING"
_header
_view
_footer

# --- End of file. ---

How it works?

I open the browser (the default home page is webportal).
I choose the kiwix menu, then I can launch kiwix-serve and go to the bookmark 127.0.0.1:8888 or refresh the list of my zim files with kiwix-manage and generate a new xml file.

Because the zim files are on a ext4 partitions I don't have to split them.

To acess the zim files with built-in Kobo reader on the good old 3.19.5761 kernel, you can do this:

# mkdir /mnt/onboard/wikipedia

Then in /usr/local/Kobo/udev/sd add something like

mount /dev/mmcblk1p2 /mnt/wikipedia
mount --bind /mnt/wikipedia /mnt/onboard/wikipedia

Maybe it's very dirty, but I like it this way.

[nhedgehog]

I like your version, thanks for sharing.

[h87booky]

I forgot to mention a little bug.

Kobo database forgets big zim files (I suppose > 4 Go?) after each reboot or switch to KOReader. I have to plugin USB or simulate an USB connection with KSM to resync, then Kobo rediscover the files. No problem with small zim files. Also, the sizes for these big files are completely wrong, for some with a big negative value.

However, at the end it works, Kobo builtin reader displays the montruous 62 Go wikipedia_en_all_novid without any slowdown.

This bug doesn't affect the kiwixserve/browser method.