MobileRead Forums - View Single Post - Clara HD Crosscompile a web browser for the KOReader terminal

elinkser · 06-02-2024, 04:51 PM

W3M - GRAPHICAL BROWSER FOR FBPAD

The w3m terminal browser we installed from Alpine Linux to the /mnt/onboard/.adds/koreader/ here ( https://www.mobileread.com/forums/sh...91&postcount=3 ) and migrated to the /mnt/onboard/.adds/kordir/ folder here ( https://www.mobileread.com/forums/sh...6&postcount=14 ) is capable of showing inline images.

It just needs an external image displayer, w3mimgdisplay, which we will crosscompile for fbink and rename w3mimgfbink.

w3mimgdisplay has the following input parameters:
(Thanks to https://blog.z3bra.org/2014/01/images-in-terminal.html for the heads up.)

Code:

/*
* w3mimg protocol
*  0  1  2 ....
* +--+--+--+--+ ...... +--+--+
* |op|; |args             |\n|
* +--+--+--+--+ .......+--+--+
*
* args is separeted by ';'
* op   args
*  0;  params          draw image
*  1;  params          redraw image
*  2;  -none-          terminate drawing
*  3;  -none-          sync drawing
*  4;  -none-          nop, sync communication
*                      response '\n'
*  5;  path            get size of image,
*                      response "<width> <height>\n"
*  6;  params(6)       clear image
*
* params
*      <n>;<x>;<y>;<w>;<h>;<sx>;<sy>;<sw>;<sh>;<path>
* params(6)
*      <x>;<y>;<w>;<h>
*
*/
	/*
		+_______screen_____+
		|       ^             |
		|       |offset_y|
		|       v              |
		|      +image |
		|<____>|         |    |
		|offset_x|       |    |
		|        |       |    |
		|        +_______+     |
		+___________|_________+
		                     |
		                     v
		+________image_______+
		|            ^             |
		|            | shift_y |
		|            v            |
		|       +- view port + ^  |
		|<___>|          |      |     |
		|shift_x |      | height|
		|         |  |  |      |
		|    +______+    v   |
		|   <-  width ->    |
		+_____________________+

	*/

***
***
***

First build w3m on the desktop.
You will build a binary called mktable that runs on the desktop to help crosscompile the arm binary for kobo (you will see later on.)

tats/w3m
https://github.com/tats/w3m/tree/master

$ wget https://github.com/tats/w3m/archive/...ads/master.zip

Now create a directory to build the arm binary:

$ mkdir websw

$ source ~/koxtoolchain/refs/x-compile.sh kobo env bare
(This is the koreader toolchain as was covered here https://www.mobileread.com/forums/sh...16&postcount=4 .
Note the frequent reference to the $HOME/x-tools/ folder.
That is where the koreader crosscompile libs and headers are installed.)

$ cd websw/

Now we will procede to crosscompile the whole w3m package, but we will only need the w3mimgdisplay binary, since the the main browser app w3m we got from Alpine Linux works fine.

***
***
***

giflib
$ wget https://sourceforge.net/projects/gif...b-5.2.2.tar.gz

$ tar zxvf giflib-5.2.2.tar.gz

$ cd giflib-5.2.2/

$ CC=arm-kobo-linux-gnueabihf-gcc make

$ cd ..

***

imlib2
$ wget https://downloads.sourceforge.net/en...-1.12.2.tar.xz

$ tar Jxvf imlib2-1.12.2.tar.xz

$ cd imlib2-1.12.2/

$ ./configure --without-x --without-tiff --without-webp --host=arm-kobo-linux-gnueabihf CPPFLAGS=" -I$HOME/x-tools/usr/include/ -I$HOME/websw/giflib-5.2.2/ " LDFLAGS=" -L$HOME/x-tools/usr/lib/ -L$HOME/websw/giflib-5.2.2/ -lgif " FREETYPE_CFLAGS=" -I$HOME/x-tools/usr/include/freetype2/ " FREETYPE_LIBS=" -L$HOME/x-tools/usr/lib/ -lfreetype " PNG_CFLAGS=" -I$HOME/x-tools/usr/include/ " PNG_LIBS=" -L$HOME/x-tools/usr/lib/ -lpng " JPEG_CFLAGS=" -I$HOME/x-tools/usr/include/ " JPEG_LIBS=" -L$HOME/x-tools/usr/lib/ -ljpeg " ZLIB_CFLAGS=" -I$HOME/x-tools/usr/include/ " ZLIB_LIBS=" -L$HOME/x-tools/usr/lib/ -lz " LIBS=" -lrt "

$ ls -l $HOME/x-tools/arm-kobo-linux-gnueabihf/arm-kobo-linux-gnueabihf/sysroot/lib/libdl*
-r-xr-xr-x 106036 libdl-2.15.so
lrwxrwxrwx 2023 libdl.so.2 -> libdl-2.15.so

$ export DLDIR=$HOME/x-tools/arm-kobo-linux-gnueabihf/arm-kobo-linux-gnueabihf/sysroot/lib/

(What just happened here is we are making sure we are building for the ARM libdl, and not the desktop one.)

$ nano -l Makefile

Code:

...
275 DLOPEN_LIBS = -L$(DLDIR) -ldl
...

$ make

$ cd ..

***

gc 8.2.2-r2
https://hboehm.info/gc/

$ wget https://hboehm.info/gc/gc_source/lib...s-7.8.0.tar.gz

$ tar zxvf libatomic_ops-7.8.0.tar.gz

$ cd libatomic_ops-7.8.0/

$ ./configure --host=arm-kobo-linux-gnueabihf

$ make

$ cd ..

***

$ wget https://hboehm.info/gc/gc_source/gc-8.2.4.tar.gz

$ tar zxvf gc-8.2.4.tar.gz

$ cd gc-8.2.4/

$ ./configure CPPFLAGS=" -I$HOME/websw/libatomic_ops-7.8.0/src/ " LDFLAGS=" -L$HOME/websw/libatomic_ops-7.8.0/src/.libs/ " --with-libatomic-ops=check --host=arm-kobo-linux-gnueabihf

$ nano -l Makefile

Code:

...
 982 THREADDLLIBS = -lpthread -lrt -L$(DLDIR) -ldl
...

$ make

$ cd ..

***

openssl
libssl3 3.1.4-r6

$ wget https://www.openssl.org/source/openssl-3.1.5.tar.gz

$ tar zxvf openssl-3.1.5.tar.gz

$ cd openssl-3.1.5/

$ ./Configure no-ssl2 no-ssl3 --cross-compile-prefix=arm-kobo-linux-gnueabihf- linux-armv4

$ nano -l Makefile

Code:

...
3198 AR=$(CROSS_COMPILE)ar
3199 ARFLAGS= qc
3200 RANLIB=$(CROSS_COMPILE)gcc-ranlib
3201 RC= $(CROSS_COMPILE)windres
3202 RCFLAGS= 
...
3222 CNF_EX_LIBS= -L$(DLDIR) -ldl -pthread -latomic
...

$ make

$ cd ..

***

ncurses
https://invisible-island.net/ncurses/

$ tar zxvf ncurses-6.3.tar.gz

$ cd ncurses-6.3/

$ ./configure --with-static --with-shared --with-cxx-shared --without-ada --without-gpm --without-sysmouse --enable-widec --disable-big-core --host=arm-kobo-linux-gnueabihf

$ make

$ cd ..

***

tats/w3m
https://github.com/tats/w3m/tree/master

$ wget https://github.com/tats/w3m/archive/...ads/master.zip

$ mv master.zip w3m-master.zip

$ unzip w3m-master.zip

$ cd w3m-master/

$ mkdir FBInk

$ cp -r ../FBInk-v1.25.0/Release FBInk/

$ cp ../FBInk-v1.25.0/fbink.h FBInk/

$ nano -l configure
[CODE]
...
7643 { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: Imlib2 is not installed. Install Imlib2$
7644 $as_echo "$as_me: WARNING: Imlib2 is not installed. Install Imlib2 (version >= 1.0.5)" >&2;}
7645 have_imlib2="yes"
...
9084 if test x"$gclibdir" = xno; then
9085 /* as_fn_error $? "libgc not found" "$LINENO" 5 */
9086 fi
...
9690 /* as_fn_error $? "cannot check setpgrp when cross compiling" "$LINENO" 5 */
...
/CODE]

(A kludge to get get configure to run. Kids please don't copy - read a proper programming book instead.)

$ ./configure --enable-image=fb --with-imagelib=imlib2 --disable-nls --disable-xface --disable-mouse --with-termlib=ncurses --with-gc=$HOME/websw/gc-8.2.4/.libs SSL_CFLAGS=" -I$HOME/websw/openssl-3.1.5/include/ " SSL_LIBS=" -L$HOME/websw/openssl-3.1.5/ " CFLAGS=" -g " CPPFLAGS=" -I$HOME/websw/imlib2-1.12.2/src/lib/ -I$HOME/websw/gc-8.2.4/include/ -I$HOME/websw/openssl-3.1.5/include/ -I$HOME/websw/ncurses-6.3/include/ -I$HOME/x-tools/usr/include/ -I$HOME/x-tools/usr/include/freetype2/ " LDFLAGS=" -L$HOME/websw/imlib2-1.12.2/src/lib/.libs/ -L$HOME/websw/gc-8.2.4/.libs/ -L$HOME/websw/openssl-3.1.5/ -L$HOME/websw/ncurses-6.3/lib/ -L$HOME/x-tools/usr/lib/ -L$HOME/websw/giflib-5.2.2/ -lgif -LFBInk/Release/ " --host=arm-kobo-linux-gnueabihf

$ nano -l w3mimg/Makefile

Code:

 14 SUBDIRS=fb

$ export DLDIR=$HOME/x-tools/arm-kobo-linux-gnueabihf/arm-kobo-linux-gnueabihf/sysroot/lib/

$ nano -l Makefile

Code:

...
 48 LIBS = -lm  -lncursesw -lz -lfreetype -lfbink -lImlib2 -lgc  -L$(DLDIR) -ldl 
...

$ nano -l config.h

Code:

...
159 #define SETPGRP() setpgrp()
...

(another kludge!)

$ nano -l w3mimg/fb/fb.c

Code:

...
  25 #include "fb.h"
  26
  27 #include "FBInk/fbink.h"
  28
  29 FBInkConfig fbink_cfg = {0};
  30
  31 FBInkConfig* cfg(){
  32     return &fbink_cfg;
  33 }
...
 140 /*    if ((fbfp = open(fbdev, O_RDWR)) == -1) {
 141         fprintf(stderr, "open %s error\n", fbdev);
 142         goto ERR_END;
 143     } */
 144     fbfp = fbink_open();
 145     if (fbfp < 0) {   
 146         fprintf(stderr, "%s error\n", fbdev);
 147         goto ERR_END;
 148     }
 149     FBInkRect cls_rect = { 0 };
 150     
 151 #if defined(__linux__)
 152     if (fb_fscrn_get(fbfp, &fscinfo)) {
...
 409     }
 410     fbink_refresh( fbfp, y, x, width, height,cfg() );
 411     return 0;
 412 }
...

(Porting to fbink)

$ nano -l w3mimg/fb/fb_w3mimg.c

Code:

...
179 /*
180     if (!check_tty_console(getenv("W3M_TTY")) &&
181         strncmp("fbterm", getenv("TERM"), 6) != 0 &&
182         strncmp("jfbterm", getenv("TERM"), 7) != 0) {
183         fprintf(stderr, "w3mimgdisplay/fb: tty is not console\n");
184         goto error;
185     }
186 */
...
187     if (fb_open())
188         goto error;
189
190     int percentDisplay;
191     if (getenv("FBPDF_PERCENT") != NULL) {
192         percentDisplay = atoi(getenv("FBPDF_PERCENT"));
193         if (percentDisplay < 1 || percentDisplay > 99)
194             percentDisplay = 100;
195     } else {
196         percentDisplay = 100;
197     }
198     wop->height = fb_height() * percentDisplay / 100;
199     wop->width = fb_width();
200     
201     wop->init = w3mfb_init;
...

(making sure images only fill up the fbpad part of the display.)

$ nano -l w3mimgdisplay.c

Code:

...
 68         fflush(stdout);
...

(dimension data got buffered so have to flush it to output, or inline imaging not enabled on kobo fbpad.)

$ make
./mktable 100 functable.tab > functable.c
/lib/ld-linux-armhf.so.3: No such file or directory
make: *** [Makefile:178: functable.c] Error 255

This error is because make is attempting to use the arm compiler to generate the mktable binary to use during the build, which won't run because whatever it builds with the arm compiler won't run during the make process since that is on the desktop.
So, copy the mktable binary from a previous desktop (non crosscompiled) build:

$ cp ../w3m-master/mktable .

$ make

$ cp ../giflib-5.2.2/libgif.so .

$ cp ../imlib2-1.12.2/src/lib/.libs/libImlib2.so.1.12.2 libImlib2.so.1

$ mkdir imlib2

$ mkdir imlib2/filters

$ mkdir imlib2/loaders

$ cp ../imlib2-1.12.2/src/modules/filters/.libs/*.so imlib2/filters/

$ cp ../imlib2-1.12.2/src/modules/loaders/.libs/*.so imlib2/loaders/

Copy the libImlib2.so.1 lib to the /mnt/onboard/.adds/kordir/libs/ folder of your kobo.

Copy the imlib2 folder to the /mnt/onboard/.adds/kordir/libs/ folder of your kobo.

$ cp w3mimgdisplay w3mimgfbink

Copy the w3mimgfbink binary to the /mnt/onboard/.adds/kordir/scripts/ folder of your kobo.

ON YOUR KOBO:

Setup the path:

# . /korenv.sh

Create a new user named w3muser so that you can browse as nonroot:

# adduser w3muser
Changing password for w3muser
New password:

# addgroup fbgroup

# adduser w3muser fbgroup

# chgrp fbgroup /dev/fb0

(Looks like you have to redo that last chgrp command as root each session.)

Create a startup script:

# nano -l /w3menv.sh

Code:

 1 #!/bin/sh
 2
 3 export PATH=/mnt/onboard/.adds/kordir/scripts:/mnt/onboard/.adds/koreader/plugins/terminal.koplugin/:/mnt/onboard/.adds/koreader/scripts:$PATH
 4 export LD_LIBRARY_PATH=/mnt/onboard/.adds/kordir/libs:/mnt/onboard/.adds/koreader/libs:$LD_LIBRARY_PATH
 5 export TERM=screen
 6 export FBPDF_PERCENT=58
 7 export IMLIB2_LOADER_PATH=/mnt/onboard/.adds/kordir/libs/imlib2/loaders
 8 export HOME=/home/w3muser/
 9 cd $HOME
10
11 alias ls='ls --color=never'
12 chgrp fbgroup /dev/fb0

Run the startup script as root:

# . /w3menv.sh

Switch to the w3muser user:

# su w3muser

Run the startup script as user:

$ . /w3menv.sh

Check that the library path was set:

$ echo $IMLIB2_LOADER_PATH

Test the w3mimgfbink binary on tux.png in same folder:

$ echo -e '0;1;0;0;200;160;;;;;tux.png\n4;\n3;' | /mnt/onboard/.adds/kordir/scripts/w3mimgfbink 2>/dev/null

You can try placing it at 400,400:

$ echo -e '0;1;400;400;200;160;;;;;tux.png\n4;\n3;' | /mnt/onboard/.adds/kordir/scripts/w3mimgfbink 2>/dev/null

You can try making it double the height:

$ echo -e '0;1;400;400;200;320;;;;;tux.png\n4;\n3;' | /mnt/onboard/.adds/kordir/scripts/w3mimgfbink 2>/dev/null

Now run w3m and set the options:

$ ./w3m -v

Enter 'o' for options.

Set options as follows:

Code:

...
Display inline images                                                  (*)YES  ( )NO
Display pseudo-ALTs for inline images with no ALT or TITLE string      (*)YES  ( )NO
Load inline images automatically                                       (*)YES  ( )NO
Maximum processes for parallel image loading                           [4                   ]
Use external image viewer                                              (*)YES  ( )NO
Scale of image (%)                                                     [150                 ]
Inline image display method                                            [external command   ]
External command to display image                                      [/mnt/onboard/.adds/kordir/scripts/w3mimgfbink]
...

Tab down to [OK] and press return to save.
q to quit.

Now you have to run fbpad/fbpad2 (from either onscreen or remote SSH terminal keyboard) to see the inline images working:

From fbpad, run the startup script as root, switch to the w3muser user and run the startup script as user, as before:

# . /w3menv.sh

# su w3muser

$ . /w3menv.sh
(ignore the chgrp error message)

You can now run the Alpine Linux w3m binary on the downloaded google page to see the inline google image:

$ w3m mywgets/index.html

OR, if WiFi is on, e.g.:

$ w3m mobileread.com

Like elinks, it is space/b to page down/up, tab to traverse the links, q to quit, but U to open a URL.

W3m Shortcuts cheatsheet
https://cheatsheetfactory.geekyhacker.com/linux/w3m
⇧r Reload
⇧h Hotkeys
⇧t Open a tab
⌃q Close a tab
⇧u Go to a url
⇧{} Switch between tabs
⌃h Show history
⇧b Exit history
⇧i View image
Esci Save image
c Show which page you are on
u View a URL of a link
Escb View bookmarks
Esca To bookmark
<> Scroll left and right
o Open options/preferences
[number Go to link number
Escl Show all links in a page in pop up form to select (with going to link)

REMEMBER YOU ONLY HAVE AROUND 50MB STORAGE SPACE ON THE ROOT PARTION, SO DELETE ALL THE CACHE FILES IN /home/w3muser/.w3m AFTER EACH SESSION!!!
* CORRECTION: the w3mxxxx files that fill up the .w3m/ folder are just temp files, not cache files, according to the FAQ ( https://stuff.mit.edu/afs/sipb.mit.e...c/w3m/FAQ.html ).
If you open up a second SSH terminal, you can witness these w3mxxxx files multiply and eat up precious storage space as you browse.
Forunately, they are automatically deleted, unless you run out of space and w3m crashes, for example.

So why are we running it in the root partition then?
There is plenty of space in the user /mnt/onboard/ partition.
It is because inline imaging is not enabled on kobo fbpad if w3m can't do a symlink, and guess what the vfat user partition can't do?
Symlinks!
So either modify w3m source to not need symlinks, or run in the linux filesystem based root partition.
For now that is what we are doing.
And since we are there, we use the fine-grained permissions capability to run as nonroot, for a more secure browse. Yay!
Note: if you want to recompile w3m, it is much easier to do it in a chroot - and that way you keep library compatibility with Alpine as well.

***
***
***

* TIPS *

To get bookmarks working:

# cd /usr/lib

# ln -s /mnt/onboard/.adds/kordir/usr/lib/w3m w3m

To get helpkey 'H' working:

# cp /mnt/onboard/.adds/kordir/usr/lib/w3m/w3mhelp.html /usr/share/w3m/w3mhelp-w3m_en.html

To create a 500MB disk image on /mnt/onboard to act as a larger .w3m/ folder that is also formatted as a Linux filesystem.

# dd if=/dev/zero of=/mnt/onboard/w3m.img bs=1M count=512

# mke2fs -F /mnt/onboard/w3m.img

# cd /home/w3muser

# mount /mnt/onboard/w3m.img .w3m

# ls -l .w3m/

# chown -R w3muser:w3muser .w3m

# su w3muser

$ . /w3menv.sh

$ w3m -v

$ exit

# umount .w3m

* DON'T FORGET TO UNMOUNT THE DISK IMAGE AS ROOT WHEN DONE BROWSING (AS IN THAT LAST COMMAND) *
(Possible data corruption risk if connecting USB cable and clicking 'connect' while loopfile still mounted was mentioned here: https://www.mobileread.com/forums/sh...02&postcount=9 )

***
***
***