newtrix - geekmaster's new tricks

[knc1]

Originally Posted by jmseight

1) I understand that I can compile in the Kindle with gcc with Debian and OptWare. Debian istaking up 500MB in my Kindle and I am seriously thinking about deleting. I am sure I don't want to take another 250MB for OptWare. Is there a more resource efficient way.

Yes, build your own "minimum system" including only those things __you__ need (and/or want).

Here is how (presumes a native build environment):
http://www.linuxfromscratch.org/lfs/read.html

If cross-compiling your new system, start here:
http://trac.cross-lfs.org/
Although you might still find that referencing the native build book (first link) will be helpful. The CLFS project often assumes you are already familar with LFS.

Less than a 100mb system is relatively easy, special pupose systems less than 10mb are possible.

- - - -

Since this thread might be read by those without system development experience...

The main LFS book, based on a native build, is probably the most informative.

With the limited amount of RAM on a Kindle, you will not get very far unless you setup a swap file or partition and enable swapping.

Enabling swap with the swap resource placed on the internal eMMC device is not recommended. It is rather hard to replace if you wear it out.
The directions for using an external swap resource with the Kindle is a bit too much for someone at the "just learning" stage.

(Plus, the CPU instruction rate is a bit low for building an entire system - although I have done such on my own NSLU-2 at 1/2 the Kindle clock speed.)

So using a machine with a bit more computer resources than a Kindle is a good idea.

And if you happen to have an x86 based "powerhouse" machine - then you can do a native build under a virtual ARM environment.
There is an app for that:
http://landley.net/aboriginal/about.html

One of Rob's objectives is to be able to do a full LFS build in any of the generated virtual environments. So this stands a good chance of working at any time but be sure to read his current release notes before starting.
His "release notes" can usually be found under "news":
http://landley.net/aboriginal/

He provides a binary image for the purpose of doing an LFS build, see:
http://landley.net/aboriginal/downlo...ages/binaries/

[geekmaster]

Originally Posted by xueyou2

It is awesome.I like to try technical work.It brings surprise.

Thanks. A "pat on the back" now and then is my only compensation for all this hard work, besides my own personal satisfaction.

[geekmaster]

Originally Posted by knc1

Less than a 100mb system is relatively easy, special purpose systems less than 10mb are possible.

A good first step for new developers is to install my tcc package (which I will update with the missing eink header files).

When ready to go to the next step, it would be much better to use a small loop mount with a minimal gcc build environment that does much better code optimization (as suggested by knc1) so you can compile programs that run much faster than those compiled with tcc.

But running the CodeSourcery cross-compiler is faster and better than either of the above methods, but requires that you "scp" the compiled file(s) over to the kindle to test them (unless you add that to the CodeBlocks compile). I have CodeSourcery installed in the CodeBlocks IDE in both my Linux Mint (Ubuntu-based debian distro), and in my Windows XP. It was easy to setup on both systems. I suppose that *somebody* should write some instructions on how to install CodeSourcery into CodeBlocks on both systems.

I like to write the preliminary code using CodeBlocks, and then for interactive debugging, the edit/compile/test sequence is much faster using tcc running on the kindle in an interactive session (where I use the nano editor). Then after debugging, you can "scp" the revised source code out of the kindle and back into the CodeBlocks project folder. Then release a final package with a fast binary that was cross-compiled with CodeSourcery.

[knc1]

There are differences between building a user application(s) and the building of an entire system from scratch.

Readers of this thread should keep that in mind when wondering why the differences in the suggestions you read here.

The suggestions you choose to follow depend on the task you intend to accomplish, not a matter of "the right way" or "the wrong way" - just different ways for different goals.

[geekmaster]

Originally Posted by kiri87

I had some issues compiling this on my Kindle 3.
First, the files einkfb.h & mxcfb.h were missing. The I read I should find them in the Amazon Source codes; to compile, I needed the files from Kindle_src_5.1.0.
Second, I received a

undefined symbol '__invalid_size_argument_for_IOC'

error, which I fixed by adding

unsigned int __invalid_size_argument_for_IOC;

I hope this may help someone, especially any other Linux noob like me.

This is listed as a bug in the linux ioctl.h header file, which gets optimized out in ALL CASES by the gcc compiler so you never see this with gcc. It can be a problem with other compilers (including tcc), where it causes a linker error. See here: https://bugzilla.redhat.com/show_bug.cgi?id=473947

This was fixed in that header file for newer linux versions, but the kindles use the old header file with that known bug.

Thanks for the quick fix. It works. Now that I am using ioctl() in my eink functions, I need to add that to my eink global vars.

EDIT: This bug is one reason why many developers (including myself) do not like to include any files from subfolders inside the include folder. What I should do is go back to copying the structures I need from the include files into my program source file, instead of using those include files. Including the "standard" C header files if fine, but the deeper ones can be buggy (and slow down the compiler to parse all the header crap you are not using).

EDIT 2: Now that I have compiled this with tcc, I see that it still runs many times slower than compiled with "arm-linux-gcc -O3". I can only imagine how much slower it would be without the hand-optimizations. Well, at least tcc is convenient for development and testing, even if it produces only sub-optimal results. Graphics and animation are NOT areas where you want to use non-optimizing compilers. For example, the "goodbye" demo runs with fluid animation when compiled with gcc, but has a much lower framerate when compiled with tcc.

[geekmaster]

Originally Posted by knc1

There are differences between building a user application(s) and the building of an entire system from scratch.

Readers of this thread should keep that in mind when wondering why the differences in the suggestions you read here.

The suggestions you choose to follow depend on the task you intend to accomplish, not a matter of "the right way" or "the wrong way" - just different ways for different goals.

When building full apps that have library dependencies, cross compiling gets a LOT more complicated. When building kernel modules (device drivers and such), you not only need to link to the correct library versions, but you may also need to compile with the same version of gcc as was used to build the kernel. Installing multiple versions of CodeSourcery complicates using an IDE to run it. Supporting multiple kindle models gets difficult.

Because I want programs that work on all eink kindles, I tend to avoid library and header file dependencies when I can.

Even trying to install stuff that was already compiled correctly can be difficult when you have all kinds of version conflicts and missing dependencies while using apt-get.

Just when you think you have it mostly figured out, new problems crop up. Simple demos are fun and easy, but full complex systems start falling apart if you are using the wrong build tools.

In the end, there really is no RIGHT way or WRONG way. When I mention such things, I tend to use them humorously (in quotes, followed by a "grin" icon ). What is important is that what you create does the job you need it to do, regardless of WHAT tools you used to make it. If you were so inclined, you could even make /bin/sh do framebuffer graphics, despite the fact that C can be so much faster. As it turns out, I have gotten request to do more "eink algorithmic art" demos, which are claimed to be more "fun" than C programming. If doing graphics with /bin/sh, hexdump and dd is not using the "wrong" tools for the job, I do not know what could be worse (except perhaps using a dull spoon to do heart surgery).

[geekmaster]

Originally Posted by kiri87

...
I did notice, though, there's some unusual issues with the dithering, at least on my Kindle3. That part of code is too complex for me to try fixing it.
First, the dither16 (screen attached, no 2) shows some black where there should be white (on the top, for example)
Second, the dither2 (screen attached, no 3) has a very subtle bug, that can observed easily for grays close to 50%. I attached a resized & cropped a portion of the screenshot to show the bug (screen attached, no 4). Basically, one column in 8 is off by 1 pixel.

After a lot of looking for patterns in different size dither tables, using Karnaugh maps, I figured out a logic function that can generate any size ordered dither table. It works fine up to a 4x4 dither used for 256-color mode, but beyond that it seems to have a subtle bug. Debugging these things is an iterative process, where you keep getting closer until it is "good enough" to do the job intended (and demos do not need to be very accurate). I need to do some speed tests to see whether I should use it during init to create a dither table, or if it is fast enough to use while dithering. At least when compiling with tcc, the dither table wins it would seem (not unexpected, seeing that the recursive dither function calls itself twice for every pixel, and that doubles again at each recursive call). I am impressed by how simple I got that function down to (at least *I* am impressed). I will do a demo eventually.

Anyway, the reason for this post is that this work showed where the "bug" is in dithering as described by kiri87. Dithering works by conditionally rounding the bottom four bits (not used by 4-bit eink hardware) which sometimes adds one to the visible upper 16 bits, depending on the position in the dither matrix. But there is a general problem with dithering in that it converts a 0-255 range to a 0-256 range because 0 sometimes becomes 1, but 255 sometimes becomes 256). And of course, the value 256 white become 0 black when put into an 8-bit pixel. There are two common solutions (often ignored causing the same bug in other code): use a ceiling function to clip values over 255 to 255, or rescale the values *255/256. Rescaling is the mathematically "correct" approach, but consumes more CPU (and therefore more battery).

That "one column in 8 off by 1 pixel" bug mentioned above is also the same range scaling issue due to dithering creating an extra color value. All those extra pixels push up into causing white to dither up into black again. And rescaling to fix the range issues slows down the animation.

EDIT: Rescaling fixed the problems in the dither, and for demos that wait until time for the next eink update, it runs full speed even calculating dither for every pixel and not using a dither table. Cool...

[dave2008]

Great work as always! BTW, where did you managed to get those Eink controller documents?

[geekmaster]

Originally Posted by dave2008

Great work as always! BTW, where did you managed to get those Eink controller documents?

Not very detailed, but enough to "read between the lines" based on past experience. When you have a lot of experience with a lot of hardware, little hints can give away the whole secret.

Basically, I went from the device teardowns that identified the chips (epson) on K3, and backtracked to crossreference the part number and find a PDF with that part.

For the K4 and K5, I found eink literature from Freescale, which identified the eink controller integrated into the iMX508 SoC, and from there backtracked to a project for the illiad that has partial support for the same eink controller.

I found the "not known in this forum" secret ioctl for the k4/k5 (buried in very complex code) and extracted the essential ioctl parameters, then simplified it by "defaulting" a lot of stuff.

The freescale eink controller has a LOT of untapped potential here that we can explore (especially with a custom kernel)...

Basically, after multiple failed attempts in past months, I decided I really needed it now, and just kept looking (pretty much continuously for two days) until I found the secrets I needed to write code.

[dave2008]

Originally Posted by geekmaster

Not very detailed, but enough to "read between the lines" based on past experience. When you have a lot of experience with a lot of hardware, little hints can give away the whole secret.

Basically, I went from the device teardowns that identified the chips (epson) on K3, and backtracked to crossreference the part number and find a PDF with that part.

For the K4 and K5, I found eink literature from Freescale, which identified the eink controller integrated into the iMX508 SoC, and from their backtracked to a project for the illiad that has partial support for the same eink controller.

I found the "not known in this forum" secret ioctl for the k4/k5 (buried in very complex code) and extracted the essential ioctl parameters, then simplified it by "defaulting" a lot of stuff.

The freescale eink controller has a LOT of untapped potential here that we can explore (especially with a custom kernel)...

Basically, after multiple failed attempts in past months, I decided I really needed it now, and just kept looking (pretty much continuously for two days) until I found the secrets I needed to write code.

I see... I previously thought you just got these documents by calling Amazon ;P

That's a nice hack

[geekmaster]

Originally Posted by dave2008

I see... I previously thought you just got these documents by calling Amazon ;P

That's a nice hack

I just downloaded the documents when I knew what to look for. The problem is that they say WHAT they can do, but NOT HOW to do it. It takes some hardware and software development experience for little clues to be useful at a deep technical level. The little clues tell you WHICH piece of information that you ALREADY KNOW is the right one to use in this case, or at least provide MORE relevant keywords to help narrow your search...

[geekmaster]

I just upgraded my K5(Touch) to 5.1.0, and now the ioctl() call for eink updates DOES NOT WORK! What's up with that? That was a major find, and now they broke it? Bummer... I will have to find a workaround to make it work again.

EDIT: At least amazon did not break the eink update ioctl() in diags mode (yet). The only flag I guessed on was setting "MARKER" to 1 because value 0 did not work. Perhaps I have to find a different value for that. Any ideas what value to use?

[knc1]

Originally Posted by geekmaster

I just upgraded my K5(Touch) to 5.1.0, and now the ioctl() call for eink updates DOES NOT WORK! What's up with that? That was a major find, and now they broke it? Bummer... I will have to find a workaround to make it work again.

Your correct.
The ioctl() commands, once released, are supposed to be engraved in stone.

The addition of a new (or changed) one is supposed to overlap the continued existance of the one being replaced.

So the maintainer of the e-ink driver screwed the pooch on letting that change(s) in.

[geekmaster]

Originally Posted by knc1

Your correct.
The ioctl() commands, once released, are supposed to be engraved in stone.

The addition of a new (or changed) one is supposed to overlap the continued existance of the one being replaced.

So the maintainer of the e-ink driver screwed the pooch on letting that change(s) in.

I guessed on what to set for the MARKER value in the structure passed to the ioctl() call. I suppose if I used an undocumented value that worked previously, there is no guarantee that it would continue working. The problem is that, there is no example code that I can find that calls this with parameters needed for the kindle (I modeled the call after code used for the illiad ebook).

From the GPL source code I determined the correct values for all fields except MARKER. Value 0 did not work, but value 1 DID work (but apparently not any more). I suppose I need to look in the kernel code for how it is used. The header file does not define any values for it that I can see.

[geekmaster]

After going through the kernel mxcfb driver source and trying a bunch of different options (including the values used by eips captured with strace), nothing works, so I guess we are back to system("eips ''"); like in the earlier demos (at least that still works, and it is fast too).

It looks like the only "portable" (and reliable) method of doing eink updates is to call eips ''.

I am right back where I was the first few times I tried to get ioctl() updates working, except this time they WERE working.