Geekmaster's adventures in K1 spelunking

[geekmaster]

Originally Posted by donB006

DId you get the K1 rootfs image unsquashed? Did you find the remaining files in /usr/default someplace and get those added to your K1 copy for a complete system? That should allow you to chroot into the virtual K1 and run your actual K1 programs except with the modern kernel imitating a K1 kernel. That is as far as I got.

I could not unsquashfs in my linux mint 17 because it says unsquashfs not installed, but I can do "apt-get install squash-tools" to get it. I tried that, but like so many other things in mint 17, apt-get reports "deprecated or not found". Synaptic Package manager also cannot find it. So no, no extraction (except on Win32 7-zip, to NTFS).

However, no luck finding out how to config aboriginal to built ucLibc with those missing symbols. The only place found is in "automatically generated, do not edit" files. And no "configure" script either, but a wrapper C program (he says uclibc config broken). No obvious way to set those flags in a re-generated file...

Studying the scripts DID lead me to armv4l cross-tools that work on x86 linux, built as an intermediate stage needed in a full aboriginal build. He says to only use them on distcc hosts. He says to build target apps in the qemu emulated custom kernel and rootfs only (for maximum target compatibility, especially for complex builds).

Despite those warnings, I used the "not for general use" crosstools in x86 linux to compile a "Hello World" that ran fine on my K1. If that fails on a complex build, I can fall back to his recommended "fully encapsulated" build environment. The crosstools are at least 10x faster (but it feels like 100x). Though again, elsewhere in the docs, he suggests that to use cross-tools, you should use these primitive cross tools to build better more robust cross-tools (these used crude hacks to override hard-coded paths in the host gcc that built them).

So at least that is another option. What I already did instead of unsquashfs is to untar the tarballs of my K1 rootfs (from the device, not the update package), so that problem can be side-stepped for now...

[geekmaster]

My goal is still to build kernel modules, so I can add usbnetwork support (and more) to the K1.

[knc1]

The Linux version is called: p7zip
http://www.7-zip.org/download.html
Hint: scroll down, past the Windoze packages.

[geekmaster]

Originally Posted by knc1

The Linux version is called: p7zip
http://www.7-zip.org/download.html
Hint: scroll down, past the Windoze packages.

When I need that. But my K1 tarballs worked fine instead, untarred into the aboriginal build environment (primarily /libs), and there were many K1 libs not already in aboriginal linux.

I plan to use the aboriginal build tools to build the K1 source tree, eventually (including the included toolchain). RSN, if real life doesn't intrude too soon...

[NiLuJe]

As I'm sure you're aware, you'll indeed need to match the GCC version used if you want to build kernel modules without having to ship a full kernel+modules set.

[geekmaster]

Originally Posted by NiLuJe

As I'm sure you're aware, you'll indeed need to match the GCC version used if you want to build kernel modules without having to ship a full kernel+modules set.

Yep! Which is the reason Rob Landley chose to package a binary qemu in his "full source" build -- he considered the need to install gcc 3.x on your HOST computer to be an unreasonable demand for building qemu from source. Not only kernel modules, but apps as well can be dependent on specific gcc versions...

Also why LKMPG (linux kernel module programming guide) STRONGLY recommends building your own matching kernel (though it does go into specifics for those deperate souls who MUST match a pre-existing kernel version, such as on the K1).

That is also why I mentioned building the toochain included in the kindle GPL source distribution. For kernel modules...

But thanks for the reminder -- You never know what holes I have in supposedly "obvious" (for you) knowledge -- or what I used to know but forgot, while treading this "unfamiliar" territory. Heck, even a lot of MY old code is completely unfamilar to me these days. What was old is new again...

[geekmaster]

I noticed in the aboriginal build scripts that busybox, toolbox (non-GPL busybox replacement), AND gcc are static builds, so I should be able to do native gcc buids in my K1 (perhaps with a swapfile on SD card where it should do negligable flash wear damage). But armv4l-gcc is already ungodly slow in qemu on a fast PC though -- so on a K1, hmm. That makes the faster tcc seem like a better idea if only for build speed.

But the tcc build failed with those missing symbols, which is WHY I am reading the build scripts (which told me about the much faster "special use" arm-gcc cross-tools used as part of the build process) for aboriginal linux from source.

[donB006]

I have compiled a lot of kernels. The original 2.6.10 for the K1 had the usb gadget for storage so I wonder if you could compile a custom 2.6.10 arm kernel and just borrow and add the custom g_ether module to your K1 kernel. There are hints the K1 almost had USB networking in the /etc/network/interfaces and someplace else I forget.

If you can build a correct kernel or gcc/binutils then you should be able to build anything you want.

I didn't intend that you build the gcc in your K1 only build a gcc that would work in the K1 as far as compatibilty is concerned. I was actually thinking the perfect gcc should be buildable using the K1 filesystem (with added gcc/binutils/headers) in the qemu system emulator or maybe just binfmt chroot.

Seems now like you are getting closer. Good luck.

[geekmaster]

I am still a bit confused about why armv4l static binaries will NOT run on later kindles. It seems to me (intuitively) that static apps ought to do ONLY standard system calls, which SHOULD be backward compatible, and that newer ARM chips SHOULD also be backward compatible. Finding what is choking newer kindles running old apps is not something I want to waste time on though, so "all kindle" apps may need a launcher script that selects which of two binaries to run based on kindle serial number prefix (B001 and B101 for K1 binary, else K2+ binary). Icky but expedicious...

Also, some of these binaries are huge "multiplex" binaries (like busybox) which contain many "programs" all bundled into one huge file. The main entry point is a multiplexor that selects which embedded function to call based on parameter zero (the filename, which is argv[0], or $0 in multiplex shell scripts). The busybox function is special in that it shifts the input parameters left and calls the main multiplexor again, or if no parameters, it displays usage and an embedded function list.

On a filesystem that supports symlinks, it is typical to create symlinks to busybox, one for each embedded function. Busybox sees the filename it was called under, and multiplexes accordingly. When storing a /bin folder on vfat (/mnt/us), there are no symlinks allowed, so it is common to copy /bin and let all those symlinks get replaced with renamed copies of the huge busybox file.

However, I plan to do it differently because the K1 has limited userstore space. I will replace symlinks with scripts that just do "/mnt/us/bin/busybox basename $0 $1 $2 $3 $4 $5 $6 $7 $8 $9", which should behave just like a symlink, but be vfat compatible. There is a precedent to this with /lib, which on some kindles contains a bunch of symlinks to a large multiplex library, and on other kindles contains text files containing a path to the multiplex library (i.e. a library loader script). You can see this in the replacement "bind" libs included in my tcc compiler package. Obviously, those loader scripts could work fine on vfat, just like the (untested) executable script replacements for symlinks mentioned above.

The reason I was thinking about this is that I want to copy a lot of stuff from my armv4l build environment to my kindle, without eating too much of its limited userstore (i.e. the shared flash drive space at /mnt/us). Instead of many copies of huge multiplex executables and libraries stored on vfat, just one of each, and a text file to substitute for each symlink. That seems (intuitively) non-icky and expedicious.

[knc1]

Busybox handles file systems without symlink or hard link support.

It is a build time/installation option -
each function becomes a one line script.

Each of the configuration choices in the config system has a 'help' entry.
Somehow you must have missed reading the ones about the installation choices.

= = = = =

Intel had a full architecture license - not just a license to use ARM produced cores.

And Intel kept that license when they sold xScale to Marvell - -
Marvell had to be compatible with xScale (since Intel did not re-sell the license) but that is as far as compatibility went.

It was that full architecture license that they based their switch from CISC to RISC (internally) with the Pentium (IIRC) processor.
....
Well, whatever the public model was, Intel has been building RISC processors (with a CISC -> RISC decoder glued to the front bus) for over twenty years now.
Welcome to the 21st century.

= = = =

Post one of your "runs on K1 but does not run on anything else ARM" binary examples.
Do us a favor and do not strip it.

= = = =

Edit:
Go to:
https://www.mobileread.com/forums/sho...d.php?t=240616
Download:
base-0.3-a8_armhf_vfp3d16.tar.gz
Browse the esys/bin, esys/sbin parts of the file tree.

There you will find about 160 Busybox functions that do not use hard links, symbolic links, or copies of the Busybox binary as examples.
(No, I didn't do those, they where produced by the Busybox install command.)

For the express hint (the esys/bin/bunzip2 file contents):

#!/mnt/us/esys/bin/busybox
bunzip2

That thread is also the worked example of concurrently using multiple system libraries (Kindle soft-float and Debian hard-float) on an ELF system.
That 'base system' will run Debian/Jessie binaries in /mnt/us/esys/** with a bit of PatchELF magic to the downloaded Debian binaries.
(Debian/Jessie ARM standard is hard-float, the Kindle system libraries are not.)

- - - -

Less than a hundred downloads in two years -
Not one of my more popular projects.

[geekmaster]

I did not build busybox. It is in the aboriginal linux rootfs images I downloaded from landley.net. I was just planning to copy existing bits & pieces from aboriginal to my K1.

Though after those missing symbols, it looks like I may have to build aboriginal from source after all. No luck yet finding how to configure aboriginal to keep those symbols (uses a build script, not a configure script, and the missing symbol "enables" are only found in temp "do not edit" config files).

One of the armv4l binaries that runs on K1 and not K2+ is the busybox-armv4l I mentioned previously, but it is stripped. Also, the binaries from aboriginal-ar4vl rootfs are stripped. So I compiled a non-stripped "hello world" (attached tarball). I tested it in multiple environments.

Working in aboriginal-arm4vl build environent, x86 linux mint with binfmt and qemu-arm-static, on the K1.

NOT working on K2 (hangs) or on PW3 (hangs). Well, this is new behavior -- no segfault or "Illegal instruction". Just a good old hung app. On earlier PW3 firmware, this armv4l "hello" program segfaulted as I recall. Same build environment, but fresh compile. New behavior. Never tested on the K2 before now. I expected a segfault on K2 and PW3, but just hanging instead.

EDIT: I read your edit. I browsed the download file. I see that you can also PIPE the name of a command into busybox, apparently (i.e. feed it a script file with it being the script handler). Interesting, and more interesting that it can build those symlink-replacement scripts for you too. Same idea I had (modelled on library loader scripts) but better execution.

[knc1]

I got it.
But having system problems here, not sure when I will get everything working again.

= = = =

I seem to recall that some versions of Busybox used a one liner:
#!/mnt/us/esys/busybox $@
or something like that.

[geekmaster]

Originally Posted by knc1

I got it.
But having system problems here, not sure when I will get everything working again.

= = = =

I seem to recall that some versions of Busybox used a one liner:
#!/mnt/us/esys/busybox $@
or something like that.

Ahh... Yes! $@ is the whole command line -- I never used that personally, but I certainly will in the future... And making busybox be the script processor would work better than an infinite recursion of /bin/sh when /bin/sh is also a script that loads busybox (but never quite gets that far), eh?

I hope we can figure out how to build apps that run on ALL eink kindles. That problem with segfault and/or just hanging is annoying. I wonder what causes it? Need to learn to use gdb I suppose...

[knc1]

gdb?
Heard of it, but have never used it.
(Could say that things I write always work, but it would be a lie.)

- - - -

My external 1Tbyte array is trying to recover -
OpenSUSE Tumbleweed is trying to download and install 876 updates -
A rainstorm is making my Internet connection act like a yoyo -

I may just take an early mid-day nap.

[geekmaster]

Originally Posted by knc1

gdb?
Heard of it, but have never used it.
(Could say that things I write always work, but it would be a lie.)

- - - -

My external 1Tbyte array is trying to recover -
OpenSUSE Tumbleweed is trying to download and install 876 updates -
A rainstorm is making my Internet connection act like a yoyo -

I may just take an early mid-day nap.

My debugger of choice has been a ton of ifdef-wrapped printf's scattered through the code (removed before the "production" release). I just set some flags in order to "verbose" (or breakpoint) a selected chunk of code. But sometimes instead of ifdefs I use live code to check argv[] "debug" switches, in more "dynamic" debugging cases.

Although I have single-stepped device drivers with an appropriate softICE -- and I have also attached hardare logic anaylzers to hardware to assist with finding archane software timing glitches. But usually? Printf is my debugger of choice. A "real" debugger just gets in the way for the sort of things I like to do...