Geekmaster's adventures in K1 spelunking

[knc1]

The (implied) question being answered was: "My application has a slow start-up from KUAL, I have to use . . .".

No reference made to amount of memory being used by cvm, either before, during, or after.

[geekmaster]

I am recompressing bunny.gmv to bunny.gmv.xz as follows:
xz -z9 bunny.gmv

For some reason the xv in the busybox-armv5l was not happy with the previous version. If this fails, I will see about building the xv in usbnet, because it worked with gmplay on my k5. For my pre-dithered videos, xz compresses them to about one-fourth the size of gzip versions.

[geekmaster]

Originally Posted by knc1

The (implied) question being answered was: "My application has a slow start-up from KUAL, I have to use . . .".

No reference made to amount of memory being used by cvm, either before, during, or after.

My application does not have a slow startup, though in some versions I did wait a bit before starting the native app to let cvm stop updating the title bar before I take over the framebuffer. I do remember that particular question being asked nor implied, but I have a tendency to construct and interpret language a bit more formally, with fewer hidden assumptions and unexpressed implications (like Spock from Star Trek, or Sheldon Cooper from The Big Bang Theory). This was a problem in various forums in the past, before I overcame my burning desire to defend my honor and to refute (perceived) false accusations. Such things no longer matter, most of the time. Life is too short...

Anyway, I will have gmplay playing movies (with sound) on my K1 (and newer kindles) very soon (though I have chores to do, so hobby time is about to go on hold).

[geekmaster]

Originally Posted by NiLuJe

(Just a quick reminder that the K1 doesn't support Kindlets ).

Aware of that. No kindlets unless somebody ADDS such support, anyway. I do not plan to do that, though I may make my native mode launcher menu KUAL-compatible to some extent. There is no reason to NOT add java support (and lua too) to a K1, is there?

[geekmaster]

Now this is strange. Even the bunny.gmv.gz that is a fresh download, working with gmplay (and zcat) on a K5, fails on the K1. For some reason, zcat reports "bad magic" when decompressing the video. I just compressed xv and bzip2 versions using my linux mint, but the .gz version did not change.

I could TRY using the zcat in the busybox-armv4l instead of /bin/zcat, but its xv already failed on me. I am curious why the (pre-build) decompressors are failing on the K1... I have yet to try bzcat though -- I will do that now. Perhaps it is because I compressed the files with -9 compression?

Hmm... bzcat "decompression failed" too. Maybe old versions do not support -9 compression?

Okay, time to copy the 300MB raw uncompressed video to my K1, and just use "cat"... Oops, K1 only has a 180MB userstore -- need to find my SD card and put vids there.

[knc1]

Originally Posted by geekmaster

Aware of that. No kindlets unless somebody ADDS such support, anyway. I do not plan to do that, though I may make my native mode launcher menu KUAL-compatible to some extent. There is no reason to NOT add java support (and lua too) to a K1, is there?

It should be possible to build Lua for the K1 - -
Once you have figured out building dynamically linked applications for the K1.

(Lua depends on using dlopen - it can only be built static for a specific purpose (I.E: all modules are statically linked in) ).

I don't recall if Java pre-dates ARMv4 or not, I think it does.
Maybe a little searching for ARMv4JT<...> would help answer that question. (I think Jazerle (sp?) implies thumb also)

[geekmaster]

Originally Posted by knc1

It should be possible to build Lua for the K1 - -
Once you have figured out building dynamically linked applications for the K1.

(Lua depends on using dlopen - it can only be built static for a specific purpose (I.E: all modules are statically linked in) ).

I don't recall if Java pre-dates ARMv4 or not, I think it does.
Maybe a little searching for ARMv4JT<...> would help answer that question. (I think Jazerle (sp?) implies thumb also)

The K1 is armv5te, not armv5tej, so no java byte-code interpreter BUILT IN. However, that is no reason to prevent a software-based JVM, like on x86 hardware, AFAIK.

[knc1]

Originally Posted by geekmaster

The K1 is armv5te, not armv5tej, so no java byte-code interpreter BUILT IN. However, that is no reason to prevent a software-based JVM, like on x86 hardware, AFAIK.

Not if the source was written with ARMv4 (which your building for) in mind.

The comment about 'j' option was just to coordinate the dates.
The sources of the hardware assisted JVM are still under NDA.
So only the relative dates of JVM introduction and ARMv4 being in use are possibly relevant.

[geekmaster]

Java runs on arduino, so it can run on my K1.

http://forum.arduino.cc/index.php?topic=263727.0

[geekmaster]

After modifying gmplay (my kindle video player) to use "/dev/fb/0" if "/dev/fb0" fails, it is now playing video on my K1. Though "playing" is relative. I am not packing the dithered pixels into 2-bit pixels, so the pixels are displaying double-width. And the video is negative (like the K4 in one of its modes). And it is only displaying about one frame every 2.5 seconds (significantly slower than on the DX).

I can probably speed it up if I shrink the video and only do partial updates of that smaller image. I already know I can do that on the DX and DXG, from how fast "newtrix" displays eink updates when the "random temporal dither" demo shrinks the work area way down. I can do that for video too.

However, I am actually surprised that the ioctl video updates (two versions) in gmplay work on the K1, and that gmplay did not do the fallback to eips '' (which does not work on the K1).

I need to tweak the "formula" so it treats the K1 like the K4, and inverts the video so no longer a negative image, and to do 2-bit packing so the pixels are no longer double width (displaying only the bottom half of the rotated image).

Considering that gmplay had to have tweaks for each kindle model, and for various firmware versions, and for both main and diags boots, I was quite surprised to see it work with no changes on the PW1 (and later), and now also on the K1 by just changing the file name of the video framebuffer.

When I do the output display size adjustments for higher speed, the DX and DXG should be a lot faster too. I guess what surprises me most is that display updates are even triggered by the existing code, considering that it does not use the "update_display" /proc file.

EDIT: For this test, I piped the UNCOMPRESSED bunny.gmv into gmplay, because for some reason the built-in (busybox) zcat and bzcat (and the xz busybox-armv4l downloaded from busybox.net) ALL fail on the K1 when trying to decompress bunny.gmv.gz, or bunny.gmv,bz2, or bunny.gmv.xz, even though all those compression formats decompress just fine on newer kindles. I had to put the uncompressed bunny.gmv on SD card because it was much too big (286MB) for the 180MB K1 userstore. Interestingly, it my pre-dithered video is so "predictable" that it compresses all the way down to 15MB using xz (16MB for some versions of vx), about 20x compression ratio. I find it very strange that ALL the K1 decompressors claim the various formats of compressed files are "corrupt". What's up with that? Could it be that I used maximum (-9) compression, and the K1 decompressors are just too old to support that?

[NiLuJe]

@geekmaster: If you were using -9, it *might* be an OOM issue?

[geekmaster]

Originally Posted by NiLuJe

@geekmaster: If you were using -9, it *might* be an OOM issue?

Good point. Putting a swap file on SD card might resolve Out-Of-Memory issues, but with SD access SO SLOW on the K1, it may be a lot better to just use a lower compression rate (smaller compression block sizes). I cannot do swap over nbd when the K1 has no USB networking (or wifi), eh? And corkscrewing nbd through the amazon Whispernet proxy would be unwise (for many reasons), besides certainly being slower and higher latency than the SD card.

Regarding the low video frame rate on my K1, I already pointed out how the SD card is PAINFULLY SLOW when mounted in the K1. I have not done speed tests, but it is HUGELY FASTER to remove the SD card and connect it directly to my computer to transfer files. I suspect that either the K1 is doing 1-bit serial access (SPI) to the SD card, or the mmc kernel driver has the same problems it did when I "fixed" it (made it MUCH faster, then lost my source code) a decade ago: http://uanr.com/mmc/

Beware that a bad power supply can kill your work drive and ALL your backup hard drives. I should have backed up to CD and/or DVD back then...

Another problem is that gmplay naively does simple blocking reads and writes (and ioctl calls), to keep the code short and to the point (i.e. a demo). I could do async calls with double buffering, so the slow SD access overlaps the slow eink updates (but there was no need to add such extra complexity until now, discounting the slowness on DX/DXG).

It all depends on how I want to invest my limited hobby time (unless somebody is willing to PAY me to do kindle development).

[geekmaster]

Instead of using the huge busybox-armv4l (with tons of features, as listed previously in this thread), which was downloaded from busybox.net (only used so far to decompress xz archives), I see that busybox in the aboriginal armv4l rootfs is tiny (few functions) and still supports xz decompression:

PHP Code:

(armv4l:1) /home $ ls -al /bin/busybox
-rwxr-xr-x 1 root root 354648 2016-01-11 18:49 /bin/busybox

(armv4l:1) /home $ busybox
BusyBox v1.24.1 (2016-01-11 12:47:37 CST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list
   or: function [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as.

Currently defined functions:
    [, [[, awk, bunzip2, bzcat, bzip2, dd, diff, expr, fdisk, ftpd, ftpget,
    ftpput, gunzip, gzip, hush, less, pgrep, ping, pkill, route, sh,
    sha512sum, tar, test, tr, unxz, vi, wget, xzcat, zcat 


This should save a lot of memory (needed for decompression buffers), with a much smaller static-linked busybox occupying scarce memory resources. Too bad busybox on vfat cannot execute directly from OneNAND (because of the STL filesystem virtualization), which could save a lot of valuable RAM during execution.

[geekmaster]

Just to put these all in one place, here are all the busybox versions I found that work on my K1:

PHP Code:

#> ls -al /mnt/us/bin
drwxr-xr-x    2 root     root         2048 May 15 15:37 .
drwxr-xr-x   12 root     root         4096 May 15 16:12 ..
-rwxr-xr-x    1 root     root       354648 May 15 15:36 busybox
-rwxr-xr-x    1 root     root      1072032 May  6 15:08 busybox-armv4l
-rwxr-xr-x    1 root     root       620140 May 15 15:55 busybox_k1
-rwxr-xr-x    1 root     root        44328 May 13 22:47 gmplay
-rwxr-xr-x    1 root     root          288 Mar 24 21:25 gmv.sh

#> /mnt/us/bin/busybox_k1
BusyBox v1.01 (2008.03.31-00:04+0000) multi-call binary

Usage: busybox [function] [arguments]...
   or: [function] [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as!

Currently defined functions:
    [, addgroup, adduser, adjtimex, ash, awk, basename, bunzip2, busybox,
    bzcat, cat, chgrp, chmod, chown, chroot, clear, cmp, cp, crond,
    crontab, cut, date, dd, delgroup, deluser, df, dirname, dmesg,
    du, echo, env, expr, false, fdisk, fgrep, find, freeramdisk, getty,
    grep, gunzip, gzip, halt, head, hexdump, hostname, hwclock, id,
    ifconfig, ifdown, ifup, init, install, ip, ipaddr, iplink, iproute,
    kill, killall, klogd, linuxrc, ln, logger, login, logname, logread,
    losetup, ls, makedevs, md5sum, mkdir, mknod, mktemp, more, mount,
    mv, nc, netstat, passwd, pidof, ping, pipe_progress, pivot_root,
    printf, pwd, rdate, readlink, realpath, reboot, renice, reset,
    rm, rmdir, route, run-parts, sed, seq, sh, sleep, sort, start-stop-daemon,
    strings, stty, su, sulogin, sync, sysctl, syslogd, tail, tar,
    tee, telnet, test, time, touch, tr, traceroute, true, tty, udhcpc,
    umount, uname, uniq, uptime, usleep, vi, wc, wget, which, who,
    whoami, xargs, yes, zcat

#> /mnt/us/bin/busybox-armv4l
BusyBox v1.21.1 (2013-07-08 10:00:38 CDT) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2012.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list[-full]
   or: busybox --install [-s] [DIR]
   or: function [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as.

Currently defined functions:
    [, [[, acpid, add-shell, addgroup, adduser, adjtimex, arp, arping, ash,
    awk, base64, basename, beep, blkid, blockdev, bootchartd, brctl,
    bunzip2, bzcat, bzip2, cal, cat, catv, chat, chattr, chgrp, chmod,
    chown, chpasswd, chpst, chroot, chrt, chvt, cksum, clear, cmp, comm,
    conspy, cp, cpio, crond, crontab, cryptpw, cttyhack, cut, date, dc, dd,
    deallocvt, delgroup, deluser, depmod, devmem, df, dhcprelay, diff,
    dirname, dmesg, dnsd, dnsdomainname, dos2unix, du, dumpkmap,
    dumpleases, echo, ed, egrep, eject, env, envdir, envuidgid, ether-wake,
    expand, expr, fakeidentd, false, fbset, fbsplash, fdflush, fdformat,
    fdisk, fgconsole, fgrep, find, findfs, flock, fold, free, freeramdisk,
    fsck, fsck.minix, fsync, ftpd, ftpget, ftpput, fuser, getopt, getty,
    grep, groups, gunzip, gzip, halt, hd, hdparm, head, hexdump, hostid,
    hostname, httpd, hush, hwclock, id, ifconfig, ifdown, ifenslave,
    ifplugd, ifup, inetd, init, insmod, install, ionice, iostat, ip,
    ipaddr, ipcalc, ipcrm, ipcs, iplink, iproute, iprule, iptunnel,
    kbd_mode, kill, killall, killall5, klogd, last, less, linux32, linux64,
    linuxrc, ln, loadfont, loadkmap, logger, login, logname, logread,
    losetup, lpd, lpq, lpr, ls, lsattr, lsmod, lsof, lspci, lsusb, lzcat,
    lzma, lzop, lzopcat, makedevs, makemime, man, md5sum, mdev, mesg,
    microcom, mkdir, mkdosfs, mke2fs, mkfifo, mkfs.ext2, mkfs.minix,
    mkfs.vfat, mknod, mkpasswd, mkswap, mktemp, modinfo, modprobe, more,
    mount, mountpoint, mpstat, mt, mv, nameif, nanddump, nandwrite,
    nbd-client, nc, netstat, nice, nmeter, nohup, nslookup, ntpd, od,
    openvt, passwd, patch, pgrep, pidof, ping, ping6, pipe_progress,
    pivot_root, pkill, pmap, popmaildir, poweroff, powertop, printenv,
    printf, ps, pscan, pstree, pwd, pwdx, raidautorun, rdate, rdev,
    readahead, readlink, readprofile, realpath, reboot, reformime,
    remove-shell, renice, reset, resize, rev, rm, rmdir, rmmod, route, rpm,
    rpm2cpio, rtcwake, run-parts, runlevel, runsv, runsvdir, rx, script,
    scriptreplay, sed, sendmail, seq, setarch, setconsole, setfont,
    setkeycodes, setlogcons, setserial, setsid, setuidgid, sh, sha1sum,
    sha256sum, sha3sum, sha512sum, showkey, slattach, sleep, smemcap,
    softlimit, sort, split, start-stop-daemon, stat, strings, stty, su,
    sulogin, sum, sv, svlogd, swapoff, swapon, switch_root, sync, sysctl,
    syslogd, tac, tail, tar, tcpsvd, tee, telnet, telnetd, test, tftp,
    tftpd, time, timeout, top, touch, tr, traceroute, traceroute6, true,
    tty, ttysize, tunctl, udhcpc, udhcpd, udpsvd, umount, uname, unexpand,
    uniq, unix2dos, unlzma, unlzop, unxz, unzip, uptime, users, usleep,
    uudecode, uuencode, vconfig, vi, vlock, volname, wall, watch, watchdog,
    wc, wget, which, who, whoami, whois, xargs, xz, xzcat, yes, zcat, zcip

#> /mnt/us/bin/busybox
BusyBox v1.24.1 (2016-01-11 12:47:37 CST) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list
   or: function [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as.

Currently defined functions:
    [, [[, awk, bunzip2, bzcat, bzip2, dd, diff, expr, fdisk, ftpd, ftpget,
    ftpput, gunzip, gzip, hush, less, pgrep, ping, pkill, route, sh,
    sha512sum, tar, test, tr, unxz, vi, wget, xzcat, zcat 


The busybox-armv4l was downloaded from busybox.net, and that filename IS the name of the download. The busybox_k1 is a copy of /bin/busybox (FW 1.2) from the K1. And the version of interest, busybox, was copied from /bin/busybox from the mounted rootfs of aboriginal linux system-rootfs.

As can easily be seen, the busybox_k1 had no xv function, and the busybox-armv4l was more than a megabyte larger than the aborignal armv4l busybox. That memory savings more than makes up for the maximum buffer size needed for xv -9 compression (unless xv uses double-buffering, of course). We shall see. I am about to test with -4 compression.

[geekmaster]

Woohoo! xz-compressed (level -4) is playing video on my K1. There is no problem using 900KB compression buffers if you use a version of busybox that is more than 1MB smaller than the version that fails. That huge full-featured busybox-v4l is fine and dandy, provided you only do stuff that needs very little RAM.

And now Big Buck Bunny is playing at about 2.5FPS (it will say how many measured FPS in the log file when in finishes), reading COMPRESSED data from SD card. I will try again with -9 compression instead of -4 compression, and that should increase the framerate (still limited by sequential waits for SD card reads AND framebuffer updates). Eventually, though it will make my code bigger, I will do double buffering, so I can do simultaneous waitings (as opposed to much less efficient simultaneous waits). Asynchronous I/O has its benefits when extra speed is essential, as is using multithreaded execution (the simplest way to go when we add sound playback into the mix).

So yeah, as mentioned in that 10-year-old "mmc" link I provided recently, reading from really slow SD card interfaces is much less of a problem if you are reading highly compressed data, and pre-dithered video compresses quite highly, which is a good thing when using eink displays.

I have chores to do now -- I will fix my "double-width pixel" problem by adding 2-bit pixels packing to gmplay, and then I will port my changes into my "video plus sound" version of gmplay.

EDIT: Big Buck Bunny finished played, and it reported on competion:
1163 frames in 621.7 secs = 1.9 FPS
I thought it was faster than that, but that means that my estimated speed was probably also slower for the uncompressed video playing from SD card. Now I am going to try -9 xz compressed playback. The videos are copying to my SD card (slowly via the kindle on USB) now.

EDIT2: Nope! With -9 compression it is reporting "xzcat: corrupted data" just like before. Perhaps the busybox xz function *is* double-buffering (which makes sense for a production program) and it needs almost 2MB to support a pair of 900KB compression buffers, and my smaller busybox only freed up about half that. That seems like an inappropriate error message though, in this case. I guess I could binary-search compression buffer sizes to see what works, but I may end up using something that will break on the next firmware update (if it leaves less free memory). I will just stick with -4 compression for now (diminishing returns, and other things to do with my limited time).

EDIT3: I suspect that at some point I *will* add a swapfile (or swap partition on my SD card), and then perhaps I will revisit using xz -9 compression on my kindle videos, to see it it *can* work on my K1. I do not see any other way to free up RAM, because I am already running in the "update install" environment (though I *could* free up some by having my install script "exec /mnt/us/RUNME.sh" instead of "sh /mnt/us/RUNME.sh", but then my install script would have no opportunity to clean up after itself -- but I can design for that).