Adding Lua support to KUAL-system

[knc1]

Very, very long past due, an update to the KUAL-system layout.
(Note: The BitBucket repository is still several years out-of-date.)
The new public repository is at: https://github.com/janetizzy/KUAL-buttons

Background
This adds the ability to use the series 5 device's built-in Lua with our own scripts and add-in modules.
Lua is a 'mid-level' scripting language, used in many situations where both scripting ability and performance is required (such as in games, WoW for one), see:
https://www.lua.org/about.html
The Timm Müller mentioned there is also one of the authors of the tekUI graphics package.
(Which has been adapted to the Kindles and is ready for testing, see:
https://www.mobileread.com/forums/sho...9&postcount=15
and following.

The series 5 devices have Lua included within the factory image because the Kindle's window manager is Awesome, which is configured and programed in Lua, see:
https://awesome.naquadah.org/

There is now a KUAL menu extension which uses Lua scripting that can serve as a working, worked, example, see: https://www.mobileread.com/forums/sho...d.php?t=271929
One of the things demonstrated there is the writing of the report in HTML that is compatible with the Kindle's native reader.

KOreader is also scripted in Lua, but it provides its own, embedded, just-in-time compiler rather than use the system provided interpreter.

Usage for script authors:
To make this work along with the factory supplied Lua and Lua modules requires two things:
*) In your 'button directory', along side the config.xml and menu.json files, add a copy of the esys.lua file attached below.
(This directory is the 'current working directory' while your scripts are running.)

*) As the top line of your script(s) that are being called from menu.json write:

#!/usr/bin/lua -lesys

-- Whatever you would otherwise write follows

Usage for users (other than script authors):
Ignore the esys.lua thing, the script author will include it with their button extension.
The other three, just un-zip the contents into the top of visible USB storage.

Installation (general):
Un-archive the three lua-base-* archives to the topmost level of visible USB storage.
If writing anything for this Kindle/Lua include the esys.lua file in whatever is the current working directory of your scripting and the splat-bang line as shown above as the first line of your script.

Attachments
The archives *.tar.gz and *.zip of the same basename, contain the same files.
Both formats are provided to ease the installation when using a PC to un-archive them.
The Kindle itself, has the software de-compressors installed to handle either format.

Just be sure that the un-archive tool you use:
*) Does not do EoL conversions
*) Un-archives to the directory structure contained in the archive
(the usual default)
*) That the tool is set to 'update' existing directories and files
(the usual default)

[knc1]

Well the time has come - I need to put this post to work.

The 'spoilered' details from the first post:

Spoiler:

As shown, a place in the esys tree is reserved to also support LuaJit.

The first addition to the Kindle's Lua resources adds the POSIX module and the bitop module (required by POSIX under Lua, but not under LuaJit).

Both are borrowed from Debian's Jessie build for ARMel (soft float).

core2quad lua-posix $ tree esys
esys
└── usr
    ├── lib
    │   ├── lua5.1
    │   │   ├── bit.so
    │   │   └── posix_c.so
    │   └── luajit
    └── share
        ├── doc
        │   ├── lua-bitop
        │   │   ├── changelog.Debian.gz
        │   │   └── copyright
        │   └── lua-posix
        │       ├── changelog.Debian.gz
        │       ├── changelog.gz
        │       └── copyright
        ├── lua5.1
        │   └── posix.lua
        └── luajit

10 directories, 8 files

The second addition to the Kindle's Lua resources adds the PenLight module and the two required modules, lua-expat and lua-filesystem.
The module lua-expat in turn requires the system library libexpat.so, which I am not including in the hope that the one installed on the Kindle is a new enough version.

All are borrowed from Debian's Jessie build for ARMel (soft float).

core2quad lua-penlight $ tree esys
esys
└── usr
    ├── lib
    │   ├── lua5.1
    │   │   ├── lfs.so
    │   │   └── lxp.so
    │   └── luajit
    └── share
        ├── doc
        │    ├── lua-expat
        │    │    ├── changelog.Debian.gz
        │    │    └── copyright
        │    ├── lua-filesystem
        │    │    ├── changelog.Debian.gz
        │    │    └── copyright
        │    └── lua-penlight
        │        ├── changelog.Debian.gz
        │        └── copyright
        ├── lua5.1
        │    ├── lxp
        │    │    └── lom.lua
        │    └── pl
        │        ├── app.lua
        │        ├── array2d.lua
        │        ├── class.lua
        │        ├── compat.lua
        │        ├── comprehension.lua
        │        ├── config.lua
        │        ├── data.lua
        │        ├── Date.lua
        │        ├── dir.lua
        │        ├── file.lua
        │        ├── func.lua
        │        ├── import_into.lua
        │        ├── init.lua
        │        ├── input.lua
        │        ├── lapp.lua
        │        ├── lexer.lua
        │        ├── List.lua
        │        ├── luabalanced.lua
        │        ├── Map.lua
        │        ├── MultiMap.lua
        │        ├── operator.lua
        │        ├── OrderedMap.lua
        │        ├── path.lua
        │        ├── permute.lua
        │        ├── pretty.lua
        │        ├── seq.lua
        │        ├── Set.lua
        │        ├── sip.lua
        │        ├── strict.lua
        │        ├── stringio.lua
        │        ├── stringx.lua
        │        ├── tablex.lua
        │        ├── template.lua
        │        ├── test.lua
        │        ├── text.lua
        │        ├── utils.lua
        │        └── xml.lua
        └── luajit

13 directories, 46 files

One of the additions to the KUAL menu will be a "Debug Helper" sub-menu.
The first helper added to that sub-menu will be buttons to enable and disable the factory debug, search bar, shortcuts.

The buttons for selecting the cpu governor will also be moved there from "Helper+".
With the addition of a cpu controls report button (as requested elsewhere here).

That addition to the KUAL extensions will look like this:

core2quad Kindle $ tree extensions
extensions
└── DebugHelper
    ├── bin
    │   └── debug-shortcuts.lua
    ├── config.xml
    ├── esys.lua
    └── menu.json

2 directories, 4 files

Note that the usual KUAL extension layout template has an added file, the file named: esys.lua

That file "hot fixes" the Lua search paths that Lab126's build of Lua screwed up and at the same time, adds our esys paths.

The contents of that file do not need to be changed, it can just be copied as-is to any other button extension implemented in Lua.
BUT
That file can include any strange and wonderful Lua initialization that your button set implementation requires.

To make all of this work, more or less transparently, requires that your Lua script file contain the following first line:

#!/usr/bin/lua -lesys

-- Whatever you would otherwise write follows

For example:
That first line in the debug-shortcuts.lua script file above will cause the Kindle's, Lab126 build of Lua, to pre-load (and execute) the script named: esys.lua found in the current directory.

When KUAL executes one of these extension buttons, the current working directory is: extensions/DebugHelper.

In other words, the Lua initial configuration file resides right alongside the menu configuration files in each menu extension directory.

Edit 1:
*) Boiler-plate esys.lua file corrected and tested. Works as intended.
*) Added base module tree for POSIX module. Minimal testing - at least it doesn't seg-fault (yeah! Debian).

Edit 2:
*) Tested most functions in the POSIX module.
*) Updated the esys archive to also include lua-filesystem (borrowed from Debian/Jessie)

Edit 3:
*) Split the base modules into two parts (and moved lfs to the second part).
*) Once again, fixed my esys.lua initialization file (I had over-simplified it).
*) Even lxp and lxp.lom are working, so it must be finding the system libexpat.so library 'good enough'.

Edit 4:
*) Reverted to lfs-1.6.2 from lfs-1.6.3 for PenLight support.

There now, that **should** be the basics of handling any Linux text file or application output (some of that stuff in /proc/* and /sys/* is extra funky).
Also with basic support of xml and html i/o.
Lab126 already provided us with json encode/decode.

[NiLuJe]

Nice!

For those crazy people who actually like the Lua syntax... ;p.

[knc1]

Originally Posted by NiLuJe

Nice!

For those crazy people who actually like the Lua syntax... ;p .

I can only give the usual grounds:
"One shouldn't stop using a hammer when one is enjoying oneself."

[knc1]

Originally Posted by NiLuJe

Nice!

For those crazy people who actually like the Lua syntax... ;p.

Not completely crazy (yet) -
People want an updated firewall for Big Brother -
That takes WireShark -
WireShark is scriptable -
The scripting language is Lua.

Plus, its outputs are text files, sometimes structured text files.

So there really is a reason to my current madness.

[knc1]

Originally Posted by NiLuJe

Nice!

For those crazy people who actually like the Lua syntax... ;p.

Just for any Python lovers in the audience ('> ' is the interactive prompt).
Adding the PinLight loader (all PinLight functions are 'load on demand'), then extending the standard string library:

PHP Code:

> require'pl'
> stringx.import()
> url = 'http://amazon.com/customer/complaints.html'
> ut = url:split('/')
> for k, v in ipairs(ut) do print(k, v) end
1       http:
2
3       amazon.com
4       customer
5       complaints.html
> ut[1] = 'https:'
> tt = ut:concat('/')
> print(tt)
https://amazon.com/customer/complaints.html 


[knc1]

Did I really post above (in post #1) that lua-filesystem was working?

PHP Code:

> lfs = require'lfs'
error loading module 'lfs' from file '/mnt/us/esys/usr/lib/lua5.1/lfs.so':
        /mnt/us/esys/usr/lib/lua5.1/lfs.so: undefined symbol: luaL_setfuncs
stack traceback:
        [C]: ?
        [C]: in function 'require'
        stdin:1: in main chunk
        [C]: ?
> os.exit()

core2quad lua-5.3.2 $ grep -r 'luaL_setf' *
src/lauxlib.h:LUALIB_API void (luaL_setfuncs) (lua_State *L, const luaL_Reg *l, int nup);
src/lauxlib.h:#define luaL_newlib(L,l)  \
src/lauxlib.h:  (luaL_checkversion(L), luaL_newlibtable(L,l), luaL_setfuncs(L,l,0))
src/lauxlib.c:LUALIB_API void luaL_setfuncs (lua_State *L, const luaL_Reg *l, int nup) {

core2quad lua-5.2.4 $ grep -r 'luaL_setf' *
src/lauxlib.h:LUALIB_API void (luaL_setfuncs) (lua_State *L, const luaL_Reg *l, int nup);
src/lauxlib.h:#define luaL_newlib(L,l)    (luaL_newlibtable(L,l), luaL_setfuncs(L,l,0))
src/lauxlib.c:LUALIB_API void luaL_setfuncs (lua_State *L, const luaL_Reg *l, int nup) {

# and for here, there be dragons:

core2quad lua-5.1.5 $ grep -r 'luaL_setf' *
core2quad lua-5.1.5 $ grep -r 'luaL_newlib' *
? ? ? 


I can't find anything in the lua-filesystem public repository about supporting lua-5.1.x
(Which is crazy, lfs has been around longer than 5.1.0, even though the public repo. doesn't show it. I just need a really old, lfs build. Its the public repo. that is too new.)

I had to have been looking at a terminal open into my PC rather than into my test Kindle when I checked it before.
Because there is just no way it can work without the code written into the sources.

There are also a lot of hits in doc/manual for those queries, so I guess it is time somebody (me or at Debian) to RTFM.

[NiLuJe]

We're using it in KOReader w/ LuaJIT, which is mostly 5.1, so check that?

[knc1]

Originally Posted by NiLuJe

We're using it in KOReader w/ LuaJIT, which is mostly 5.1, so check that?

Yeah - that is a bigger question, for someone else.
LuaJIT(.org) has promised their user community to continue 5.1 support.
That decision was made based on: "there is so much code written for 5.1 that it will take forever to update all of it".
Well, forever has come.

Meanwhile, I have a plan -

First rule out PEBCAK - the debian archive I used contains lfs for both 5.1 and 5.2 :
Make sure that I just didn't click on the wrong *.so when moving things to the Kindle/Lua layout.

Then, find a copy of lfs (lua-filesystem-1.5.?) that builds against lua-5.1, then backport the function entries from lua-filesystem-1.6.?.

Meanwhile, keeping my own lfs repo that **does** span the changes from 5.1 -> 5.2

such is this morning's plan - as I drown in the Debian source archives from the turn of the century.

I suppose the Python community is facing similar issues in the 2.7 -> 3.0 change.

Whoot!
Debian-3.0 has (had) Lua-4.0 available (but no lfs), so the chances of my finding lua-5.1 has just increased.

Debian-4.0 introduced lua-expat, lua-curl, and lua-filesystem for lua-5.1
Now to find those sources.
At the worst, I should have them on one of my computers or backup disks.

Got'em:

core2quad lua-filesystem $ ls -1 *.orig.*
lua-filesystem_1.2.orig.tar.gz
lua-filesystem_1.4.1.orig.tar.gz
lua-filesystem_1.4.2.orig.tar.gz
lua-filesystem_1.5.0+16+g84f1af5.orig.tar.gz
lua-filesystem_1.6.2.orig.tar.gz
lua-filesystem_1.6.3.orig.tar.gz

Now to spend the day bashing my head against that wall.

New home is at:
https://github.com/janetizzy/lua51-filesystem

Now I just have to figure out where support for lua-5.1 went to between lfs-1.6.2 and lfs-1.6.3
At least that is what the README says happened.

Update:
I reverted to lfs-1.6.2 for our Kindle/Lua and now:

[root@kindle testme]# lua -lesys
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> lfs = require'lfs'
> os.exit()

There are a lot of 'code fixes' in lfs-1.6.3 that would be nice to have, but for now it will be lfs-1.6.2

[knc1]

Posted an updated (downdated?) lua-base-PenLight archive using lfs-1.6.2 rather than lfs-1.6.3
No change in archive name, but ....
There was only one copy downloaded (google?), so no harm, no foul.

[knc1]

Progress - the script reads:

/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

And outputs this string:

PHP Code:

<dl>
    <dt style='font-style:bold'>Scaling Available Governors</dt>
    <dd><ul>
        <li>ondemand</li>
        <li>userspace</li>
        <li>powersave</li>
        <li>performance</li>
    </ul></dd>
</dl> 


None too shabby for a first script using the PenLight library.

Which will look like this in the final page:
Scaling Available Governors

• ondemand
• userspace
• powersave
• performance

[knc1]

The scripting for the 'cpu report' button's script is done.

No telling what it looks like on **your** browser (the file doesn't define its own CSS), but it looks decent on the Kindle:
http://kaynine.info/pw3/cpu_report.html

It has one error (an excess character in a status value) and one layout error (since the script can not properly report name:value pairs).
But its 99 44/100 % done
In fact, it may be as done as this script will ever get.

[knc1]

Next -
Make a set of buttons to enable/disable the search-bar shortcuts for the factory's system reports.

[knc1]

Updated the first post attachments to include both *.tar.gz and *.zip format archives.
Updated the first post attachments to include a 3rd 'lua-base-' archive, lua-base-md5.* -
this addition will be required by the next release version of the cpuReport file generator.

[piika]

hi knc1

attached the Lua lib tekUI for GUI.
It's compiled on kt2 with default options. All its external dependencies are already on the kindle.

There are 2 changes made:

1. fontsizes (variable) in tek/ui/class/display.lua have all values doubled.

2. src/display_x11/display_x11_api.c (attached) only add 1 line: else { XStoreName(..) }

As the installed path is different from the paths in your esys.lua , I update the esys.lua to add the new paths. (attached)