Benchmark. Benchmarking Ciphers. (And a handy test of the scripty wangle WIP)

[twobob]

Originally Posted by NiLuJe

@twobob: The easier way to do that is probably to tell whatever you're using to build your toolchain (buildroot? ct-ng?) to use *this* specific glibc version when building your x-tc. (That does imply rebuilding *everything*, the toolchain, and each of every other stuff you might have built ifor your staged sysroot).

I rather thought you might say that.

Hmm not sure I can point BR at an older glibc version but Ill go find out. possibly... : )

I do have a gentoo installation on my machine now (err but I still can get my x server to boot so I shelved it) perhaps one say I will have a play on there also - one day.

thanks for everyones help. Just attempting to implement knc1's fiddle now.

[knc1]

Well, after all of this discussion, it reads as if a path forward has been decided on. That's fine.

No need for the "knc1 hack" using the loader manually, with a manually set list of libraries and with LD_DEBUG set to print a verbose report of what is happening . . . .

It would probably only report what you have deduced (the hard way).

- - - -

What with all of these detail problems, it is a bit hard to step back and look at the overall picture - -

This system that the decades of *nix work has come up with is actually pretty solid.

About 99% of applications build and run across a wide mixture of ages and version of system libraries.

Its just the other 1% that prove to be the P.I.T.A. things to "get right".

A 6 year old tool-chain, a decade old system library - and hundreds of the newest "bleeding edge" version softwares build and run?
Yup, that is really an exceptional design accomplishment.

[twobob]

Mindblowing really.

Honestly expected far less success. I would be intrigued to know if I can get more built with the more recent c version. A lot of the them choked on C errors IIRC... (well obviously errors in c I mean language specific looking deaths)

So that's a big driver for me in terms of 'should I bother?'

As a test I just renamed the extra libc.so.6.NEW & libc-2.3.6.so.NEW, which results in all the apps loading beautifully - that I have tested so far.

EDIT: Just using:

source start.sh

where start.sh is

#!/bin/sh
export LD_LIBRARY_PATH=/usr/lib:/lib:/mnt/us/usr/lib:/mnt/us/lib
export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/mnt/us/usr/bin:/mnt/us/bin
export LD_PRELOAD=
export SDL_VIDEODRIVER=aalib
export HOME=/mnt/us/home/root

Perhaps this is a non-issue in one sense. I'm using the oldest version ever. Therefore no one has a version older than me.

Older > newer in terms of 'likely to work' on all devices. ermm.. hmm... I'll think.

But yeah. gobsmacker.

[NiLuJe]

@twobob/knc1: Yeah, the glibc's ABI backwards compatibility has been pretty solid, in my experience, without too much burden . That's why your current binaries should work with the Kindle's newer glibc. (Keep in mind that glibc isn't only libc.so, though, there a whole bunch of other core library in there: libm, libcrypt and libdl being the most prominent of those).

For a stark comparison, libstdc++ is a whole different kettle of fish. There *is* ABI compatibility, but there have been so much change in the API over the years that you usually *need* to ship it/static link it to make sure it won't blow up, especially on older systems...
There's still a libstdc++-v3 package available in nearly every distro out there to provide support for older dynamically linked binary/proprietary C++ apps for this very reason.

[twobob]

erm one thought.

when we run the apps whats to stop us simply saying:

In a shell script that lives in the same directory as the binary.

export LD_LIBRARY_PATH=$PWD/libs:/usr/lib:/lib

Then reverting when it dies?

Then I can shove the libs in a nice tidy relative directory (assuming they don't do anything suicidal on FAT like generate sym links in the directory) and we can all go home for Tea

Thoughts on that? the Tea is non optional.
Other beverages will be served for heretics.

Seem's simple and elegant. there must be a flaw. : )

[twobob]

okay so to this end let's use an app. fftw-wisdom is one I was wanting to play with : )

For obvious reasons.

so step 1) use readelf to find dependant stuffs.

(I've tidied the output up)

[root@kindle bin]# readelf -d fftw-wisdom
     22566:	calling init: /lib/libc.so.6
     22566:	calling init: /lib/libgcc_s.so.1
     22566:	calling init: /mnt/us/usr/lib/libz.so.1
     22566:	initialize program: readelf
     22566:	transferring control: readelf

Dynamic section at offset 0xa00c contains 27 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libfftw3.so.3]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x0000000f (RPATH)                      Library rpath: [/home/you/BLDS/buildroot/build/fftw-3.3/.libs]
     22566:	calling fini: readelf [0]
     22566:	calling fini: /mnt/us/usr/lib/libz.so.1 [0]
     22566:	calling fini: /lib/libgcc_s.so.1 [0]
     22566:	calling fini: /lib/libc.so.6 [0]
     22566:

That's with the LD_DEBUG mods giving us some input too.
Ugly RPATH there. : \



So anyways, I figure.

b) copy the stuff to a test folder and nerf my handsets env variables.

then c) launch it via a VERY simple one or two liner script. perhaps a generic one accepting the app as an argument, or if that is too much typing just one per app.

the requirement would be that the $PWD is the executing directory and perhaps this too could be coerced within the script. Certainly a [ -e] or something won't hurt.

okay well I'm hooking this up now.

libs in /libs folder
app in root.
script in root.

#!/bin/sh

######################################
#suck, squeeze, bang, blow?
######################################

#One way
#env LD_LIBRARY_PATH=$PWD:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/sbin\
#./fftw-wisdom

#Another way
/lib/ld-linux.so.3 --library-path $PWD/lib:/usr/lib:/lib \
--inhibit-rpath libfftw3.so.3 ./fftw-wisdom

Cheers.

[knc1]

You may have to over-ride that library rpath (it is still set for the buildroot environment) in you launcher script.

Too late, too tired, to remember how to do that.

[twobob]

Originally Posted by knc1

You may have to over-ride that library rpath (it is still set for the buildroot environment) in you launcher script.

Too late, too tired, to remember how to do that.

/lib/ld-linux.so.3 --library-path $PWD/lib:/lib/usr/lib --inhibit-rpath libfftw3.so.3 readelf -d ./fftw-wisdom


I'll check the scope of the declaration.

[NiLuJe]

@twobob: That's essentially what most binary/proprietary apps/games distributed out of a PMS do. Use a wrapper script that tweaks the searchpaths to use their bundled libs over the system's.

You don't even strictly need to reset the env later if you're not doing anything else, since you're tweaking the env of *this* process and its potential children, and this process just returns at the end of the script.

(You can also just one-line it using the neat little 'env' tool from coreutils (busybox has an applet for it, too. No idea if it's enabled on the K3). env BLAH=blah BLUH=bluh /bin/blah)

[twobob]

oh it is - I've used it I believe. EDIT YEP: happy days then : )

Ill package this up then and give it a test.


Usage: env [-iu] [-] [name=value]... [command]

Print the current environment or run a program after setting
up the specified environment

Options:
-, -i Start with an empty environment
-u Remove variable from the environment

Nice.

[twobob]

FFTW (3.3) is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, i.e. the discrete cosine/sine transforms or DCT/DST).

We believe that FFTW, which is free software, should become the FFT library of choice for most applications.

(who don't mind using a GPL 2) http://www.gnu.org/licenses/gpl-2.0.html


Binaries, Script, Src's + license enc.

usage ./fftw.sh (you can call it from anywhere)


./fftw.sh contains:

#!/bin/sh

# Copyright (c) 2003, 2007-11 Matteo Frigo
# Copyright (c) 2003, 2007-11 Massachusetts Institute of Technology
#
# http://www.gnu.org/licenses/gpl-2.0.html
#

cd $( dirname "$0" )

#One way
/lib/ld-linux.so.3 --library-path $PWD/lib:/usr/lib:/lib --inhibit-rpath libfftw3.so.3 ./fftw-wisdom

#Another way
#env LD_LIBRARY_PATH=$( dirname "$0" )/lib:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/sbin\
#./fftw-wisdom

Surely some fun to be had with that
Hopefully THIS is a decent way to package things.

Obviously some launcher stuff might be nicer in the long term for appropriate apps.

: D

[knc1]

Aside from the connected use of "cd ..." and "$PWD" -
You almost have it down to a one-liner:

 #!/lib/ld-linux.so.3 --library-path $PWD/lib:/usr/lib:/lib --inhibit-rpath libfftw3.so.3 ./fftw-wisdom

For the purposes of conversation here, let us say that the above code implements the requirements of your chosen distribution policy (it does/will).
Whatever the final version of that "policy implementor" becomes, the above demonstrates the overall behavior.

(Sorry if this reads like I am preaching/teaching or something else, no insult intended to anyone, not even if the above meets the GM whitespace standard.)

Next (rhetorical) question:
How to avoid writing (or having the build system write) one of those above for each and every new application?
In answer, start by reading what is written above in the prototype launcher glue.

What does the first two characters, the Splat-Bang, in a *nix script do?
It causes the loading mechanism used to start an executable to search for a program (other than the elf loader) to interpret the rest of the file.

Now, ask yourself, what mechanism does binfmt_misc provide?
It is a loading mechanism that selects a program to interpret the rest of the file based on either extension or magic number.
The "program" may be any executable, including a script "wrapper".

Ah, so, as the light begins to dawn ....
Now we can see why that crazy old fool was asking for mod binfmt_misc.

Suppose that the above single line glue code began with a six character magic number:

 #!ld.3 --library-path $PWD/lib:/usr/lib:/lib --inhibit-rpath libfftw3.so.3 ./fftw-wisdom

Now all you need is one "wrapper" that binfmt_misc can call when it detects a new executable that needs: "launcher glue".
That wrapper can be used to determine the variables of the "glue code" launcher.

For a more advanced usage, to get a Kindle to "swallow" anything we add-in, see the following reference.

When reading the reference: http://www.kernel.org/doc/Documentation/binfmt_misc.txt

You will find examples of how binfmt_misc can read the elf header of the file to be run.
One of the first things in the elf header is the interpreter to be used and some other flag bits.

I.E: Not only can we "glue in" applications using a mis-matched glibc, we can "glue in" applications that aren't even built against glibc of any version.
(the uClibc loader has a different name, only I need to re-build the uClibc used in the emulator because the current one was built without a required option).

Note: The binfmt_misc mecanisum is the base upon which a loader for signed binaries can be built.

Amazon/lab126 requires a signed update file to make changes to the installed system but **assumes** anything found on the file system was the result of such a signed installation process.

I.E: The Kindles call the OTA updater by name, but as far as its coding can tell, the thing with the "OTA updater" name might churn butter.
If the thing with the "OTA updater" name was a signed binary, things would not be as easy to modify as they are.
Not impossilbe to modify (when using binfmt_misc), just extra insurance that the binary was from a trusted source and intended for a specific Kindle make/model/firmware.
And the test would be made each time the load attempt was made.

[twobob]

Originally Posted by knc1

...Sorry if this reads like I am preaching/teaching or something else, no insult intended to anyone...

You are not. this is awesome. I will undertake forthwith.

[twobob]

err by that i mean after i sleep. been awake almost 48h now. after!