Tools Checking for required libs. Exporting Binaries "easily"

[twobob]

Have you ever wished you could just export a binary and it's supporting libs. Or at the very least some likely candidate supporting libs.

well I did

The following works OOTB on a touch ( I think?) feedback gratefully accepted...

It needs a few fixups for the 3 as the dated busybox doesn't swallow colon substrings
REF: http://landley.net/mantis/mantis-2924.html

For the Touch and Paperwhite (and the 3 and 4 with the right files and linkages)

#!/bin/bash
FAILS=""
MAINEXPORTDIR="/mnt/us/export"
TESTUSRLIB="/usr/lib/"
TESTLIB="/lib/"
BASEUS="/mnt/base-us/"

# fail on missing victim
if [ -z ${@} ]
then
  echo "hmm I need a victim...  usage: filechecker <binary_file>"
exit 1
fi
# fail on missing victim
if [ ! -e ${@} ]
then
  echo "hmm I cannot find:  ${@}    - you sure? check the file exists."
exit 1
fi


FILES=$(readelf -d ${@} |awk -F\[ ' /Shared library/ {print $2}' |  cut -d"]" -f1) 
BINARY=$(basename ${@})
EXPORTDIR="${MAINEXPORTDIR}""/""${BINARY}"
mkdir -p $EXPORTDIR/bin
mkdir -p $EXPORTDIR/lib
cd $EXPORTDIR

# copy the binary we are interested in
cp -f ${@} "${EXPORTDIR}/bin/${BINARY}"
echo "copying ${@} to ${EXPORTDIR}/bin/${BINARY}"

LIBCOUNT=$(ls -al /lib/* | wc -l)
USRLIBCOUNT=$(ls -al /usr/lib/* | wc -l)
TOTCOUNT=$(ls -Ra1d /* | wc -l)
echo "Searching ${USRLIBCOUNT} user libs, ${LIBCOUNT} libs, ${TOTCOUNT} extended dirs"

for file in $FILES
do echo
MATCH=0
echo $file

#LOOSERESULTS=$(find / -iname '*'"$file"'*')
FASTLIBRESULTS=$(find /lib -iname '*'"$file"'*')

for result in $FASTLIBRESULTS
do
if [[ "${result:0:5}" == $TESTLIB ]]; then
echo "kindle supported at $result"
# dont copy the native stuff
MATCH=1
break
fi
done

if [ "$MATCH" == "1" ]; then
echo "/lib support detected"
continue
fi

FASTUSRLIBRESULTS=$(find /usr/lib -iname '*'"$file"'*')

for result in $FASTUSRLIBRESULTS
do
if [[ "${result:0:9}" == $TESTUSRLIB ]]; then
echo "kindle supported at $result"
# dont copy the native stuff
MATCH=1
break
fi
done

if [ "$MATCH" == "1" ]; then
echo "/usr/lib support detected"
continue
fi

EXACTRESULTS=$(find / -iname '*'"$file")

for result in $EXACTRESULTS
do
if [[ "${result:0:13}" == $BASEUS || "${result}" == $EXPORTDIR/lib/${file} ]] ; then

# dont bother with duplicates or exports
continue
fi

echo "non native result "${result}

cp -f ${result} $EXPORTDIR/lib/${file}
# just copy the first non-native instance for now...
MATCH=1
break
#fi
done

# we got this far
if [  "$MATCH" == "0"  ]; then
FAIL="$FAIL""${file} "
fi
done

#remove empty lib dir
if [[ "$(ls -1 $EXPORTDIR/lib | wc -l)" == "0"  ]] ; then
rmdir $EXPORTDIR/lib
fi 

#remove empty bin dir if we really screwed up (on an old version.)
if [[ "$(ls -1 $EXPORTDIR/bin | wc -l)" == "0"  ]] ; then
rmdir $EXPORTDIR/bin
fi 

#report failures
if [  "$FAIL" != ""  ]; then
echo "Failure of some files: $FAIL"
echo "Failed files that have no source for export are: $FAIL" >> $EXPORTDIR/failure.log
echo "Your binary wont work without them, please attempt to locate a copy of them and put them on your device then export again" >> $EXPORTDIR/failure.log
else
touch $EXPORTDIR/export.log
echo "Export complete at $EXPORTDIR ensure you run $BINARY like: cd $EXPORTDIR/bin; LD_LIBRARY_PATH=../lib:\$LD_LIBRARY_PATH ./$BINARY" | tee $EXPORTDIR/export.log
fi

[root@kindle 600x800]# filechecker /mnt/us/bin/nano

copying /mnt/us/bin/nano to /mnt/us/export/nano/bin/nano
Searching 937 user libs, 85 libs, 17 extended dirs

libncurses.so.5
kindle supported at /usr/lib/libncurses.so.5
/usr/lib support detected

libc.so.6
kindle supported at /lib/libc.so.6
/lib support detected

Export complete at /mnt/us/export/nano ensure you run nano like: LD_LIBRARY_PATH=../lib ./nano

If you don't know what this is for you don't need it.

Everyone likes a good bash improvement thread. feel free to pick holes.

[twobob]

notes: I use bash rather than ash for this. (bash is available for the 3 and the 5)

if you get
/mnt/us/bin/filechecker: line 42: syntax error: Bad substitution
you need to check your bash is being correctly referenced.

ah c'mon everyone likes bashing other peoples rubbishy scripting Chip in!

it should be methodised at the very least. : ) No? : D


ISSUES FOR THE 3:
The sub-string substitution needs reworking for the really ancient busy box on the 3.
I am enclosing egrep full version to facilitate this rewrite.

The ancient busybox find does not support iname

Either: I have enclosed a binary build of find that should do that job.
invoke it directly to bypass busybox like
/mnt/us/wherever/find /location/to/search -iname '*'"$file"'*'

OR: change the ln -s link, or add an alias, or something to get it referencing find and bash.
worst case. overwrite them. not recommended.

[Kai771]

@twobob

Awesome work, mate.

[twobob]

Originally Posted by Kai771

@twobob

Awesome work, mate.

Working on the 3 version now

Spoiler:

- egrep should fix the "sh ]]" errors with a quick rewrite.

the -iname errors (I have a custom busybox) may be fixed by replacing with -name (yet to test that)

in my case an override like

EXACTRESULTS=$(busybox find / -iname '*'"$file")

works to invoke the replacement busybox.

I can include the updated busybox (in fact I will just go find it)

Done see below

[twobob]

okay... using the files from post two. (bash and find are the biggies)

no changes in the code.

on a 3

(let's give it a tough one)

[root@kindle root]# filechecker /mnt/us/usr/bin/matchbox-desktop

copying /mnt/us/usr/bin/matchbox-desktop to /mnt/us/export/matchbox-desktop/bin/matchbox-desktop

Searching 468 user libs, 58 libs, 16 extended dirs

Spoiler:

libmb.so.1
non native result /mnt/us/usr/lib/libmb.so.1

libXft.so.2
non native result /mnt/us/usr/lib/libXft.so.2

libXrender.so.1
non native result /mnt/us/usr/lib/libXrender.so.1

libfontconfig.so.1
kindle supported at /usr/lib/libfontconfig.so.1.3.0
/usr/lib support detected

libfreetype.so.6
kindle supported at /usr/lib/libfreetype.so.6.3.20
/usr/lib support detected

libpng14.so.14
non native result /mnt/us/usr/lib/libpng14.so.14

libjpeg.so.8
non native result /mnt/us/usr/lib/libjpeg.so.8

libX11.so.6
non native result /mnt/us/usr/lib/libX11.so.6

libXext.so.6
non native result /mnt/us/usr/lib/libXext.so.6

libz.so.1
kindle supported at /usr/lib/libz.so.1
/usr/lib support detected

libc.so.6
kindle supported at /lib/libc.so.6
/lib support detected

libdl.so.2
kindle supported at /lib/libdl.so.2
/lib support detected

Export complete at /mnt/us/export/matchbox-desktop
ensure you run matchbox-desktop like: LD_LIBRARY_PATH=../lib ./matchbox-desktop

result so yeah. just update your old versions of things in a safe way.
over-ride, not overwrite

(although I overwrote my various find and egrep calls with the bins just to see if it would blow the device up and it seems fine) but really just invoke it by path or something.

but - at a pinch - you can just overwrite /bin/bash and /usr/bin/find
WARNING!! They are symbolic links that point to busybox
RENAME THEM TO name.old (like /bin/bah.old) before you attempt to replace them or you will overwrite your busybox...

: ) Nice

[twobob]

so.. and why did I want this?

on kindle 5: first mount debian

[root@kindle bin]# mount -o loop=/dev/loop/debian,noatime -t ext3 /mnt/base-us/debian.ext3 /mnt/debian
(just mount the ext3 file, not even chroot - that's all that is required)

Let's extract something - err.. debian's wget!

[root@kindle root]# filechecker /mnt/debian/usr/bin/wget

copying /mnt/debian/usr/bin/wget to /mnt/us/export/wget/bin/wget
Searching 937 user libs, 85 libs, 17 extended dirs

Spoiler:

libgnutls.so.26
kindle supported at /usr/lib/libgnutls.so.26
/usr/lib support detected

libgcrypt.so.11
kindle supported at /usr/lib/libgcrypt.so.11
/usr/lib support detected

libgpg-error.so.0
kindle supported at /usr/lib/libgpg-error.so.0.3.0
/usr/lib support detected

libz.so.1
kindle supported at /usr/lib/libz.so.1.2.3
/usr/lib support detected

libidn.so.11
non native result /mnt/us/extensions/gnash/lib/gnash/libidn.so.11

libuuid.so.1
kindle supported at /lib/libuuid.so.1
/lib support detected

librt.so.1
kindle supported at /lib/librt.so.1
/lib support detected

libgcc_s.so.1
kindle supported at /lib/libgcc_s.so.1
/lib support detected

libc.so.6
kindle supported at /lib/libc.so.6
/lib support detected

ld-linux.so.3
kindle supported at /lib/ld-linux.so.3
/lib support detected

Export complete at /mnt/us/export/wget ensure you run wget like: cd /mnt/us/export/wget/bin; LD_LIBRARY_PATH=../lib:$LD_LIBRARY_PATH ./wget

[root@kindle bin] # cd /mnt/us/export/wget/bin; LD_LIBRARY_PATH=../lib:$LD_LIBRARY_PATH ./wget --help

GNU Wget 1.14, a non-interactive network retriever.
Usage: wget [OPTION]... [URL]...

Spoiler:

Mandatory arguments to long options are mandatory for short options too.

Startup:
-V, --version display the version of Wget and exit.
-h, --help print this help.
-b, --background go to background after startup.
-e, --execute=COMMAND execute a `.wgetrc'-style command.

Logging and input file:
-o, --output-file=FILE log messages to FILE.
-a, --append-output=FILE append messages to FILE.
-d, --debug print lots of debugging information.
-q, --quiet quiet (no output).
-v, --verbose be verbose (this is the default).
-nv, --no-verbose turn off verboseness, without being quiet.
--report-speed=TYPE Output bandwidth as TYPE. TYPE can be bits.
-i, --input-file=FILE download URLs found in local or external FILE.
-F, --force-html treat input file as HTML.
-B, --base=URL resolves HTML input-file links (-i -F)
relative to URL.
--config=FILE Specify config file to use.

Download:
-t, --tries=NUMBER set number of retries to NUMBER (0 unlimits).
--retry-connrefused retry even if connection is refused.
-O, --output-document=FILE write documents to FILE.
-nc, --no-clobber skip downloads that would download to
existing files (overwriting them).
-c, --continue resume getting a partially-downloaded file.
--progress=TYPE select progress gauge type.
-N, --timestamping don't re-retrieve files unless newer than
local.
--no-use-server-timestamps don't set the local file's timestamp by
the one on the server.
-S, --server-response print server response.
--spider don't download anything.
-T, --timeout=SECONDS set all timeout values to SECONDS.
--dns-timeout=SECS set the DNS lookup timeout to SECS.
--connect-timeout=SECS set the connect timeout to SECS.
--read-timeout=SECS set the read timeout to SECS.
-w, --wait=SECONDS wait SECONDS between retrievals.
--waitretry=SECONDS wait 1..SECONDS between retries of a retrieval.
--random-wait wait from 0.5*WAIT...1.5*WAIT secs between retrievals.
--no-proxy explicitly turn off proxy.
-Q, --quota=NUMBER set retrieval quota to NUMBER.
--bind-address=ADDRESS bind to ADDRESS (hostname or IP) on local host.
--limit-rate=RATE limit download rate to RATE.
--no-dns-cache disable caching DNS lookups.
--restrict-file-names=OS restrict chars in file names to ones OS allows.
--ignore-case ignore case when matching files/directories.
-4, --inet4-only connect only to IPv4 addresses.
-6, --inet6-only connect only to IPv6 addresses.
--prefer-family=FAMILY connect first to addresses of specified family,
one of IPv6, IPv4, or none.
--user=USER set both ftp and http user to USER.
--password=PASS set both ftp and http password to PASS.
--ask-password prompt for passwords.
--no-iri turn off IRI support.
--local-encoding=ENC use ENC as the local encoding for IRIs.
--remote-encoding=ENC use ENC as the default remote encoding.
--unlink remove file before clobber.

Directories:
-nd, --no-directories don't create directories.
-x, --force-directories force creation of directories.
-nH, --no-host-directories don't create host directories.
--protocol-directories use protocol name in directories.
-P, --directory-prefix=PREFIX save files to PREFIX/...
--cut-dirs=NUMBER ignore NUMBER remote directory components.

HTTP options:
--http-user=USER set http user to USER.
--http-password=PASS set http password to PASS.
--no-cache disallow server-cached data.
--default-page=NAME Change the default page name (normally
this is `index.html'.).
-E, --adjust-extension save HTML/CSS documents with proper extensions.
--ignore-length ignore `Content-Length' header field.
--header=STRING insert STRING among the headers.
--max-redirect maximum redirections allowed per page.
--proxy-user=USER set USER as proxy username.
--proxy-password=PASS set PASS as proxy password.
--referer=URL include `Referer: URL' header in HTTP request.
--save-headers save the HTTP headers to file.
-U, --user-agent=AGENT identify as AGENT instead of Wget/VERSION.
--no-http-keep-alive disable HTTP keep-alive (persistent connections).
--no-cookies don't use cookies.
--load-cookies=FILE load cookies from FILE before session.
--save-cookies=FILE save cookies to FILE after session.
--keep-session-cookies load and save session (non-permanent) cookies.
--post-data=STRING use the POST method; send STRING as the data.
--post-file=FILE use the POST method; send contents of FILE.
--content-disposition honor the Content-Disposition header when
choosing local file names (EXPERIMENTAL).
--content-on-error output the received content on server errors.
--auth-no-challenge send Basic HTTP authentication information
without first waiting for the server's
challenge.

HTTPS (SSL/TLS) options:
--secure-protocol=PR choose secure protocol, one of auto, SSLv2,
SSLv3, and TLSv1.
--no-check-certificate don't validate the server's certificate.
--certificate=FILE client certificate file.
--certificate-type=TYPE client certificate type, PEM or DER.
--private-key=FILE private key file.
--private-key-type=TYPE private key type, PEM or DER.
--ca-certificate=FILE file with the bundle of CA's.
--ca-directory=DIR directory where hash list of CA's is stored.
--random-file=FILE file with random data for seeding the SSL PRNG.
--egd-file=FILE file naming the EGD socket with random data.

FTP options:
--ftp-user=USER set ftp user to USER.
--ftp-password=PASS set ftp password to PASS.
--no-remove-listing don't remove `.listing' files.
--no-glob turn off FTP file name globbing.
--no-passive-ftp disable the "passive" transfer mode.
--preserve-permissions preserve remote file permissions.
--retr-symlinks when recursing, get linked-to files (not dir).

WARC options:
--warc-file=FILENAME save request/response data to a .warc.gz file.
--warc-header=STRING insert STRING into the warcinfo record.
--warc-max-size=NUMBER set maximum size of WARC files to NUMBER.
--warc-cdx write CDX index files.
--warc-dedup=FILENAME do not store records listed in this CDX file.
--no-warc-compression do not compress WARC files with GZIP.
--no-warc-digests do not calculate SHA1 digests.
--no-warc-keep-log do not store the log file in a WARC record.
--warc-tempdir=DIRECTORY location for temporary files created by the
WARC writer.

Recursive download:
-r, --recursive specify recursive download.
-l, --level=NUMBER maximum recursion depth (inf or 0 for infinite).
--delete-after delete files locally after downloading them.
-k, --convert-links make links in downloaded HTML or CSS point to
local files.
-K, --backup-converted before converting file X, back up as X.orig.
-m, --mirror shortcut for -N -r -l inf --no-remove-listing.
-p, --page-requisites get all images, etc. needed to display HTML page.
--strict-comments turn on strict (SGML) handling of HTML comments.

Recursive accept/reject:
-A, --accept=LIST comma-separated list of accepted extensions.
-R, --reject=LIST comma-separated list of rejected extensions.
--accept-regex=REGEX regex matching accepted URLs.
--reject-regex=REGEX regex matching rejected URLs.
--regex-type=TYPE regex type (posix).
-D, --domains=LIST comma-separated list of accepted domains.
--exclude-domains=LIST comma-separated list of rejected domains.
--follow-ftp follow FTP links from HTML documents.
--follow-tags=LIST comma-separated list of followed HTML tags.
--ignore-tags=LIST comma-separated list of ignored HTML tags.
-H, --span-hosts go to foreign hosts when recursive.
-L, --relative follow relative links only.
-I, --include-directories=LIST list of allowed directories.
--trust-server-names use the name specified by the redirection
url last component.
-X, --exclude-directories=LIST list of excluded directories.
-np, --no-parent don't ascend to the parent directory.

helpful debian. pwned.

NOTE: THIS IS NOT A MAGIC BULLET. all manner of shared files may be missing. This only identifies the libs... But it's a start.

[twobob]

Lets do one more and see what might go wrong.

I will do debians file

(apt-get install file inside debian if you don't have it)

(mount the ext.3 if it isn't already)
[root@kindle bin]# mount -o loop=/dev/loop/debian,noatime -t ext3 /mnt/base-us/debian.ext3 /mnt/debian

okay good to go

[root@kindle bin]# filechecker /mnt/debian/usr/bin/file

copying /mnt/debian/usr/bin/file to /mnt/us/export/file/bin/file
Searching 937 user libs, 85 libs, 17 extended dirs

Spoiler:

libmagic.so.1
non native result /mnt/us/usr/lib/arm-linux-gnueabi/libmagic.so.1

libz.so.1
kindle supported at /usr/lib/libz.so.1.2.3
/usr/lib support detected

libc.so.6
kindle supported at /lib/libc.so.6
/lib support detected

ld-linux.so.3
kindle supported at /lib/ld-linux.so.3
/lib support detected

Export complete at /mnt/us/export/file ensure you run file like: cd /mnt/us/export/file/bin; LD_LIBRARY_PATH=../lib:$LD_LIBRARY_PATH ./file

let's test it

[root@kindle bin]# cd /mnt/us/export/file/bin; LD_LIBRARY_PATH=../lib:$LD_LIBRARY_PATH ./file ./file

/etc/magic, 0: Warning: using regular magic file `/usr/share/misc/magic'
file: could not find any magic files!

ro ro! it don't work!!!
let's just hack the resources on to the root (they are tiny)
[root@kindle bin]# mntroot rw
[root@kindle bin]# mkdir -p /usr/share/misc/magic/
[root@kindle bin]# cp /mnt/debian/usr/share/misc/magic.mgc /usr/share/misc/magic.mgc
[root@kindle bin]# mntroot ro

and test it again

[root@kindle bin]# cd /mnt/us/export/file/bin; LD_LIBRARY_PATH=../lib:$LD_LIBRARY_PATH ./file ./file

./file: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0xb47ce3b5fc4138f5cc958deeeabeab1b1bb782a5, stripped

sweet. so yeah - that is the kind of thing that might go wrong.
not too tricky all in.

no doubt there will be cases where walking the required resource tree would be nice. parse a strace log maybe? : D

sed would have been a better solution for accessing the resources on the FAT drive.
patching the file binary to accept the new location

Anyways - hope this helps other forgetful developers.

HOW DO I MAKE INSTALL???

on my system (where my PATH includes /mnt/us/usr/bin and /mnt/us/usr/lib by default) to do a "make install" I would just

[root@kindle bin]# cp -r /mnt/us/export/file/* /mnt/us/usr/

where file is the name of the exported application. You could do something similar

[knc1]

Reads like a generally useful tool for anyone that wants to "extended-purpose" a Kindle.
Nice job.

[brianinmaine]

Originally Posted by twobob

HOW DO I MAKE INSTALL???

on my system (where my PATH includes /mnt/us/usr/bin and /mnt/us/usr/lib by default) to do a "make install" I would just

[root@kindle bin]# cp -r /mnt/us/export/file/* /mnt/us/usr/

I would like to know, how do you change your path? Do you just mntroot rw and change /etc/profile? or do you source another file there? I just did a nice kubrick style restore (worked perfect, thanks!), so I would like to keep my medling with the root fs to a minimum.

This whole business of exporting sure looks fun to me.

[twobob]

there are plenty of options...

probably the easiest/most flexible is:

add

if [ -e "/mnt/us/.bashrc" ]; then
    . /mnt/us/.bashrc
fi

to the END of your /etc/profile then you have a nice R/W area to play in with minimal rootfs changes.

simply stuff your additions in the FAT .bashrc (that you then create)

I do that generally.


I would add that this is not a completely bullet-proof solution to exporting. just a best-start tool to save some time.

There may be resources that are missed, even libs from time to time seem to be missed.
But in the main it does the job I need.

[brianinmaine]

Thanks! I know everyone has a personal setup, would you be willing to post your /mnt/us/.bashrc as a starting point? Do you just adjust path and libraries? Do you have any cool functions in there? Anyone else reading have anything to add?

[twobob]

pretty certain I have posted it around and about before.

With the forthcoming advent of kual-system I will refrain from giving system advice until I can be certain what that will ultimately bring.

In essence include user libs and bins would be the first application of such a file as-is right now.


Something along the lines of:

export LD_LIBRARY_PATH=\
/mnt/us/usr/lib/arm-linux-gnueabi:\
/mnt/us/lib:\
/mnt/us/usr/lib:\
/usr/lib:\
/lib

export PATH=/mnt/us/usr/local/bin:\
/mnt/us/usr/bin:\
/mnt/us/usr/games:\
/mnt/us/bin:\
/mnt/us/usr/sbin:\
/mnt/us/sbin:\
/usr/local/bin:\
/usr/bin:\
/bin:\
/usr/sbin:\
/sbin

Obviously that is total overkill but you get the picture.

[brianinmaine]

Thank you for making it so easy for me, I appreciate your time!

[knc1]

Originally Posted by brianinmaine

Thank you for making it so easy for me, I appreciate your time!

Don't leave happy - at least not just yet.

As twobob wrote, there is an active project underway to bring system resources into the USB storage extensions/* tree.

When that happens, nearly all of those paths will change (and a lot of the support work disappear).

In fact, there is a whole lot of things "in progress" to extend "KUAL support" -
We (twobob and I) have not given up answering questions, just been otherwise busy.