ct-ng is moving rapidly on, how do we keep it useful?

[twobob]

Hi, I'll update this post with some numbers and version but sufficed to say the the current Stable and Git version of the CrossTools-ng Toolchain builder no longer supports many of the various default options we would like for the Kindle Keyboard.

I just ran through another build of a TC for the KK and it built in about 14 mins. but it took a good couple of hours to decide what might be decent settings given the shift away from older parts.

So. I will shift to here (and maybe the wiki one day) the list of tweaks that are required to keep ct-ng producing relevant and non-explosive code for the kindle keyboard.

There are quite a few devs that use this configuration so feel free to chip in.

[INFO ] Cleaning-up the toolchain's directory: done in 0.59s (at 13:32)
[INFO ] Build completed at 20121026.044000
[INFO ] (elapsed: 13:31.87)
[INFO ] Finishing installation (may take a few seconds)...

I'll start. The oldest Kernel is one point version too high. However choosing the oldest is your only option. (aside from making the Amazon one fly)

I'll try x-compiling a few bits and see what drops out, and note the various configurations along the way.

Niluje, Knc1 and GM amongst others, have already exhaustively offered neat sets of default flags that I will transplant here form the BR thread.

Doing this for the touch is a breeze. Let's make the KK users life a bit easier too ey.

NOTE: The 2007q3 CS TC does a fine job in many cases. Just so you know.

Thanks for listening. updates to follow.

EDIT:

Here is a big guide of what should work with what for arm http://www.kegel.com/crosstool/cross....43/buildlogs/

[knc1]

Actually, both Buildroot and Crosstool-ng make it easy to keep a kernel version selection that has been dropped.

Just ask the repository manager (Buildroot == git, ct-ng == hg) for the changeset that took it out, then two choices:
Just revert that change, or
Hand enter it again.

Both projects are careful to keep such changes (version changes, of anything) compartmentalize as a single changeset.
The naming conventions are such that there should not be any name clashes, not even in the 'patch' directories.

Of course, you will probably want to name a local 'branch' of the main code stream before you do that. But local changes should be on a named 'branch' anyway.

[NiLuJe]

@twobob: Never had an issue building a K3 TC with ct-ng (hg tip) :?

I do use a slightly more recent glibc version to be able to use a recent GCC version, so that means jumping through a few hoops in regard to CFLAGS/CXXFLAGS later on, but nothing insurmountable...

KINDLE_TC="K3"

ARCH_FLAGS="-march=armv6j -mtune=arm1136jf-s -mfpu=vfp"
CROSS_TC="arm-kindle-linux-gnueabi"
TC_BUILD_DIR="/home/niluje/Kindle/CrossTool/Build_${KINDLE_TC}"

export CROSS_PREFIX="${CROSS_TC}-"
export PATH="/home/niluje/x-tools/${CROSS_TC}/bin:${PATH}"

## NOTE: See http://gcc.gnu.org/gcc-4.7/changes.html & http://comments.gmane.org/gmane.linux.linaro.devel/12115 & http://comments.gmane.org/gmane.linux.ports.arm.kernel/117863
## But, basically, if you want to build a Kernel, backport https://github.com/mirrors/linux/commit/8428e84d42179c2a00f5f6450866e70d802d1d05 [it's not in FW 2.5.8/3.4/4.1.0/5.1.2],
## or build your Kernel with -mno-unaligned-access
## You might also want to backport https://github.com/mirrors/linux/commit/088c01f1e39dbe93a13e0b00f4532ed8b79d35f4 if you intend to roll your own Kernel.
## For those interested, basically, if your kernel has this: https://github.com/mirrors/linux/commit/baa745a3378046ca1c5477495df6ccbec7690428 then you're safe in userland.
## (That's the commit merged in 2.6.28 that the GCC docs refer to).
## It's in FW 3.x/4.x/5.x, so we're good on *some* Kindles. However, it's *NOT* in FW 2.x, and the trap handler defaults to ignoring unaligned access faults.
## I haven't seen any *actual* issues yet, but the counter does increment...
## So, to be on the safe side, let's use -mno-unaligned-access on the K3 TC, to avoid going kablooey in weird & interesting ways on FW 2.x... ;)
ARM_NO_UNALIGNED_ACCESS="-mno-unaligned-access"

## NOTE: When linking dynamically, disable GCC 4.3/Glibc 2.8 fortify & stack-smashing protection support to avoid pulling symbols requiring GLIBC_2.8 or GCC_4.3
BASE_CFLAGS="-O2 -ffast-math ${ARCH_FLAGS} ${ARM_NO_UNALIGNED_ACCESS} -pipe -fomit-frame-pointer -flto -fuse-linker-plugin -fno-stack-protector -U_FORTIFY_SOURCE"
NOLTO_CFLAGS="-O2 -ffast-math ${ARCH_FLAGS} ${ARM_NO_UNALIGNED_ACCESS} -pipe -fomit-frame-pointer -fno-stack-protector -U_FORTIFY_SOURCE"
## FIXME: Disable LTO for now, it breaks a few key parts of what we build right now (OpenSSH) ;)
BASE_CFLAGS="${NOLTO_CFLAGS}"
export CFLAGS="${BASE_CFLAGS}"
export CXXFLAGS="${BASE_CFLAGS}"
BASE_CPPFLAGS="-I${TC_BUILD_DIR}/include"
export CPPFLAGS="${BASE_CPPFLAGS}"
BASE_LDFLAGS="-L${TC_BUILD_DIR}/lib -Wl,-O1 -Wl,--as-needed"
export LDFLAGS="${BASE_LDFLAGS}"

BASE_HACKDIR="${SVN_ROOT}/Configs/trunk/Kindle/Hacks"

BASE_PKG_CONFIG_PATH="${TC_BUILD_DIR}/lib/pkgconfig"
BASE_PKG_CONFIG_LIBDIR="${TC_BUILD_DIR}/lib/pkgconfig"
export PKG_CONFIG_PATH="${BASE_PKG_CONFIG_PATH}"
export PKG_CONFIG_LIBDIR="${BASE_PKG_CONFIG_LIBDIR}"

(For reference, all of this comes from my build script).

[twobob]

hmm. well I will give it a go your way. Not turned out as I might have hoped using the stable release.

I've setup the environment near identical to yours now aside (from the underlying OS).

I'll run through the steps and assuming success, will publish confirmation and maybe a step by step.

Cheers.

[NiLuJe]

I forgot to mention the need to use make >= 3.82 on the host, and making sure glibc 2.9 will get patched for make >= 3.82 (see the top of my build script). It might not be needed for every setup/configs, but it failed to build in softfp mode with older make releases on my box.

[twobob]

Originally Posted by NiLuJe

I forgot to mention the need to use make >= 3.82 on the host, and making sure glibc 2.9 will get patched for make >= 3.82 (see the top of my build script). It might not be needed for every setup/configs, but it failed to build in softfp mode with older make releases on my box.

I did notice that. I'll dig about for the patch. we don't all have portage

Err. well its odd. I seem to think I am getting the build wrong somehow. I'll do it again rather than labour with a not right build.

Thanks again

[NiLuJe]

It's not Gentoo specific, it's actually a patch used by ct-ng that just happen to be only applied to a newer glibc version by default . (Meaning it's shipped with ct-ng, not the portage tree ).

[twobob]

okay so I threw the configuration at it. and it dies ?

[INFO ] Installing PPL for host
[ERROR] make[9]: *** [ppl_c_implementation_common.lo] Error 1
[ERROR] make[9]: *** [ppl_c_Polyhedron.lo] Error 1
[ERROR] make[9]: *** [ppl_c_Grid.lo] Error 1
[ERROR] make[9]: *** [ppl_c_Rational_Box.lo] Error 1
[ERROR] make[9]: *** [ppl_c_BD_Shape_mpz_class.lo] Error 1
[ERROR] make[9]: *** [ppl_c_Octagonal_Shape_mpz_class.lo] Error 1
[ERROR] make[9]: *** [ppl_c_Octagonal_Shape_mpq_class.lo] Error 1
[ERROR] make[8]: *** [all-recursive] Error 1
[ERROR] make[7]: *** [all] Error 2
[ERROR] make[6]: *** [all-recursive] Error 1
[ERROR] make[5]: *** [all] Error 2
[ERROR] make[4]: *** [all-recursive] Error 1
[ERROR] make[3]: *** [all] Error 2
[ERROR]
[ERROR] >>
[ERROR] >> Build failed in step 'Installing PPL for host'
[ERROR] >> called in step '(top-level)'
[ERROR] >>
[ERROR] >> Error happened in: CT_DoExecLog[scripts/functions@172]
[ERROR] >> called from: do_ppl_backend[scripts/build/companion_libs/ppl.sh@105]
[ERROR] >> called from: do_ppl_for_host[scripts/build/companion_libs/ppl.sh@59]
[ERROR] >> called from: main[scripts/crosstool-NG.sh@598]
[ERROR] >>
[ERROR] >> For more info on this error, look at the file: 'build.log'
[ERROR] >> There is a list of known issues, some with workarounds, in:
[ERROR] >> '/home/me/GIT/br_ct-ng-k3-test/output/host/usr/share/doc/crosstool-ng/ct-ng.1.16.0/B - Known issues.txt'
[ERROR]
[ERROR] (elapsed: 3:37.90)
[03:39] / make[2]: *** [build] Error 2
make[1]: *** [build.8] Error 2

so erm.. I guess I'll head scratch a bit more. maybe pull the version of PPL down by one? or did you say it was needed at that ver for Linaro? Will read up.

ho hum

EDIT: Erm. maybe I found the problem...
hokay. I thought 4gb might be enough? headscratch?
(goes to check swaps are actually turned on)

[ALL ] virtual memory exhausted: Cannot allocate memory
[ALL ] virtual memory exhausted: Cannot allocate memory
[ALL ] virtual memory exhausted: Cannot allocate memory
[ALL ] virtual memory exhausted: Cannot allocate memory
[ALL ] virtual memory exhausted: Cannot allocate memory
[ALL ] virtual memory exhausted: Cannot allocate memory
[ALL ] virtual memory exhausted: Cannot allocate memory

EDIT EDIT:

Yup confirmed. big 1.5 -2 Gb spike during PPL install. swaps were turned off duh!

on now

[twobob]

Originally Posted by NiLuJe

It's not Gentoo specific, it's actually a patch used by ct-ng that just happen to be only applied to a newer glibc version by default . (Meaning it's shipped with ct-ng, not the portage tree ).

ah yeah. the repo stuff I spotted that when I was cruising the site.

I'll check it's being done but I have another fish to fry I think.

EDIT: Okay I fixed my stupid swap foof.

TC building with exactly your specifications now via BR. (just for sanities sake)
rumbling along nicely now it has that extra 4mb !!! of space it wanted. ppfttt.

[twobob]

well shoot. there's a few hours I'll never get back.

[INFO ] Installing cross-gdb
[ERROR] configure: error: python is missing or unusable
[ERROR] make[4]: *** [configure-gdb] Error 1
[ERROR] make[3]: *** [all] Error 2
[ERROR]
[ERROR] >>
[ERROR] >> Build failed in step 'Installing cross-gdb'
[ERROR] >> called in step '(top-level)'
[ERROR] >>
[ERROR] >> Error happened in: CT_DoExecLog[scripts/functions@172]
[ERROR] >> called from: do_debug_gdb_build[scripts/build/debug/300-gdb.sh@159]
[ERROR] >> called from: do_debug[scripts/build/debug.sh@35]
[ERROR] >> called from: main[scripts/crosstool-NG.sh@598]
[ERROR] >>
[ERROR] >> For more info on this error, look at the file: 'build.log'
[ERROR] >> There is a list of known issues, some with workarounds, in:
[ERROR] >> '/home/simon/GIT/br_ct-ng-k3-test/output/host/usr/share/doc/crosstool-ng/ct-ng.1.16.0/B - Known issues.txt'
[ERROR]
[ERROR] (elapsed: 25:21.80)
[25:22] / make[2]: *** [build] Error 2
make[1]: *** [build.8] Error 2
make: *** [/home/simon/GIT/br_ct-ng-k3-test/output/stamps/ct-ng-toolchain-built] Error 2
make: Leaving directory `/home/simon/GIT/br_ct-ng-k3-test'

I assure you the python is 2.7, 2.7.3 in fact.

sigh. honestly.

back to the drawing board (and yes I did it twice to be certain)

the annoying this is:

[INFO ] ================================================== ===============
[INFO ] Installing final compiler
[INFO ] Installing final compiler: done in 321.22s (at 24:28)
[INFO ] ================================================== ===============
[INFO ] Installing cross-gdb

It has seemingly finished the main toolchain when it dies.

hmm.

[twobob]

I'll try one more time. without gdb see where it gets me.

Asleep probably.

EDIT:

Nope I made it. just.
Ran to completion this time. gdb does not want to play. meh.

I'll live without it for now.
So works without gdb. result. now: to go build that again by hand outside Buildroot.

Tomorrow perhaps. thanks for your help, I have a shiny new BR k3 TC anyways for my pains. appreciated.

[NiLuJe]

@twobob: Aw, crap, I forgot about that, sorry. I managed to pull the config from the *one* time I built a full gdb env (cross/native) >_<"... You can of course kill it from the config, because, yeah, it depends on Python (among other things) on the host, which I know I have, because, well, Gentoo :P .

[twobob]

hmm well I slept on it. I do now have python (2.7) on the host... Trying to decide if I will use the gdb or if I should just push on.

I'm leaning toward "push on". meh. At least I know I'm not going mad and I didn't bother the Buildroot mailing list with it.

Thanks mate. was an edu-ma-cation in log diving and patience . never a bad thing

[twobob]

Originally Posted by NiLuJe

@twobob: Aw, crap, I forgot about that, sorry. I managed to pull the config from the *one* time I built a full gdb env (cross/native) >_<"... You can of course kill it from the config, because, yeah, it depends on Python (among other things) on the host, which I know I have, because, well, Gentoo :P .

Just so I understand 100% correctly. Are we saying Python on the TC's "Host" build area. Because with just a single Config.in tweak pushed up the main BR branch that could then work for everyone...

I ask because obviously I have python on my "machine". therefore I am assuming it was testing for a syrooted /usr/bin/python?

Thanks, it would be nice to list that one as "understood" and I could probably fix BR long term if this IS the case.

Much obliged

Someting like ~/GIT/br_ct-ng-k3-test/toolchain/toolchain-crosstool-ng/Config.in

	config BR2_TOOLCHAIN_CTNG_GDB
		bool "Are you going to try to build GDB?"
		select BR2_PACKAGE_PYTHON
		depends on BR2_TOOLCHAIN_CTNG_glibc
                depends on BR2_PACKAGE_PYTHON

or similar. (would check that) cheers.

[NiLuJe]

Nope, I meant the host (the box you're building the stuff on), not the target (the Kindle) ;-). Well, for the cross gdb at least. The native one might need/use python at runtime, but not at build time :-).