Virtual Development

[knc1]

Over the years there have been a number of threads published on the subject of creating a development system for Kindle software.

Two of the most recently commented on:
https://www.mobileread.com/forums/sh...d.php?t=277236
https://www.mobileread.com/forums/sh...d.php?t=286561
twobob also posted the systems that he made for porting software to the Kindles.

I need such a thing on my own (newly installed) system, so I am trying a different approach.

A "Virtual Appliance" approach that contains a ready to use developement system for this specific purpose.
This system is built by a friend of mine, JaneTizzy, who is not a member here (her field is chemistry, not computers).
The appliance is posted in a number of file formats, which should cover most of the virtualizations in common use.

This is a WIP, like all of my projects (most of which never get finished).

What we have in version 0.0.2 is just the virtual development host.
It is not (yet) equip'd with ARM emulation or ARM chroot environments. Although qemu is present and binfmt_misc is mounted and enabled but here are no assignments registered with binfmt_misc (yet).
Ref: Binfmt_Misc Doc.
The usual native development tools are installed.

It does have the SuSE package management installed (zypper), for adding the things that I forgot.
But I would like some feedback on what people find (missing, wrong, incomplete, etc.).

An example of the feedback I am hoping to get:
I forgot to install any text editor other than VIM (which I hate).
I did remember to install OpenSSH, which (along with scp, sshfs, etc.) communicates with the host system (outside of VirtualBox) just fine.

Spoiler:

You fix the missing nano (or anything else) by:

$ su -
password:
# zypper in nano
# exit

The images are all built to run on an Intel, 64bit, single core, host (not the IA64, give me some slack).
The downloads are about 400Mbyte and when uncompressed the system file is about 1.4Gbyte.
The virtual hard disk is defined as an 8Gbyte drive, so leave it room to grow.

Both the root and the home directories are installed on LVM logical volumes.
The intent is to allow the user to shift free space around as required.
The build system defaulted all of the free space to the system tree. Not exactly what I intended, I wanted the free space left 'free'. Phooey, one more thing to fix in a later version (I'll tell the build system to make three logical volumes, and then delete one of them. ).

The above plan addresses a common question in a new system layout:
"How much will the 'system' section grow?"
and
"How much will the 'user home directory' grow?".
Part of the answer to those questions depend on the user's work habits and where they like to keep files (in home directory, or in system option directories).

Since both are under LVM control, it is possible to use LVM and the file system tools to resize the logical volumes and their file systems as required (you can do that 'hot' from inside the running machine).

The hard drive layout looks like this (/dev/sda1 is swap):

Spoiler:

  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               systemVG
  PV Size               7.77 GiB / not usable 0   
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              1988
  Free PE               0
  Allocated PE          1988
  PV UUID               iEQh2c-U01y-Nfq1-gY7g-wm6Y-3GNG-ftgD4e
   
  --- Volume group ---
  VG Name               systemVG
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               7.77 GiB
  PE Size               4.00 MiB
  Total PE              1988
  Alloc PE / Size       1988 / 7.77 GiB
  Free  PE / Size       0 / 0   
  VG UUID               EpnR72-OLnP-7cn2-dnfR-NGxC-Q6eZ-3BS978
   
  --- Logical volume ---
  LV Path                /dev/systemVG/LVhome
  LV Name                LVhome
  VG Name                systemVG
  LV UUID                75Ckw9-bJvo-WM6n-WUn2-WJ5L-YyoG-sUwk7D
  LV Write Access        read/write
  LV Creation host, time (none), 2017-07-12 13:21:17 -0500
  LV Status              available
  # open                 1
  LV Size                1.95 GiB
  Current LE             500
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           254:0
   
  --- Logical volume ---
  LV Path                /dev/systemVG/LVRoot
  LV Name                LVRoot
  VG Name                systemVG
  LV UUID                rq2cGd-nGFM-x4Ds-QOYs-Xl8T-JlgV-z8ofbh
  LV Write Access        read/write
  LV Creation host, time (none), 2017-07-12 13:21:17 -0500
  LV Status              available
  # open                 1
  LV Size                5.81 GiB
  Current LE             1488
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           254:1

Note:
The VirtualBox, Linux, guest additions have not been installed in this image.
See: https://www.virtualbox.org/manual/ch04.html
You will want to install those into the image in order to access files on the host system and to make the display window a reasonable size.
An even bigger: NOTE:
The kernel is too new, or the module in GuestAdditions is too old to install.
Well, I wanted to install alternate kernels anyway (I know it will install in 3.6.x) ...

Directions for installing the appliance in a VirtualBox instance:
https://susestudio.com/help/use/virtualbox.html
Those are a bit out of date. VB version 5.x only imports *.ovf images.
For the 5.x version of the image, just point the 'Import Appliance' thingy at the file and turn it loose.
The older versions of VB import the *.vdmk images.

Directions for installing the appliance in a VMware instance:
https://susestudio.com/help/use/vmware.html
Somebody that owns VMware will have to try that out and report back.

I have never seen VMware, but I assume it is similar to VirtualBox -
A folder in the host system can be assigned to another virtual hard disk drive and that used for additional workspace (or added to LVM's collection of physical storage volumes).
(Once you install the 'Guest Additions' mentioned above.)

The low-down on the image formats:
https://susestudio.com/help/create/a...e-formats.html

And for those situations that are not covered by the above, there is also a LiveCD version available.

The full course menu:
https://susestudio.com/help/use/index.html

For the curious, the appliance was built using the SuSE Studio build system (which took all of 8 minutes to build it). The system is based on the current OpenSuse stable release.

There are only two users:
User: root
Pass: linux
Auto-login is enabled, but just in case:
User: tux
Pass: linux

The first boot is rather slow, but don't panic, it is not hung up, it is doing "run once" chores.
Recent VirtualBox and VMware (ovf)
Older VirtualBox and VMware
LiveCD

And on a final note, how to get out of it (there isn't any sudo installed):

$> su -
password:
#> shutdown -h now
bye, bye world

[encol]

Originally Posted by knc1

Over the years there have been a number of threads published on the subject of creating a development system for Kindle software.

Two of the most recently commented on:
https://www.mobileread.com/forums/sh...d.php?t=277236
https://www.mobileread.com/forums/sh...d.php?t=286561
twobob also posted the systems that he made for porting software to the Kindles.

I need such a thing on my own (newly installed) system, so I am trying a different approach...

Thanks for sharing
What differences/advantages has your approach compared to the others you mentioned?

[knc1]

Originally Posted by encol

Thanks for sharing
What differences/advantages has your approach compared to the others you mentioned?

I am looking for the answer to that question myself.
If I reach my goal: "A ready to use plug-in appliance for Kindle Development", it will save other people a lot of time doing their own set-up.

So far, the only thing I needed to supply was the decision making involved in choosing what packages to include.
The SuSE (a.k.a: Novell, a.k.a: Micro Focus) Studio build system is almost a point-and-click system builder.
https://susestudio.com/appliance/create_new

The fun part will come with the addition of the ARM chroot and ARM cross-compile environments **inside** of the virtual machine.
Needless to say, they don't have that automated (or even thought of, far as I can tell).

But the system does have a way to run 'post build' scripts and to add-in overlays.
Which is probably how the Debian base stuff that matches the Kindle system versions will get laid in there (pun intended).

[encol]

I don't know if can be suitable for your goals, but some time ago i found this (ixtab was one of the contributors)
I tried to make it work, but i got some configuration errors, so i let it go due to lack of time.
Anyway, it looks interesting

[knc1]

Originally Posted by encol

I don't know if can be suitable for your goals, but some time ago i found this (ixtab was one of the contributors)
I tried to make it work, but i got some configuration errors, so i let it go due to lack of time.
Anyway, it looks interesting

Ignoring any puns on the subject of 'family planning', this one might have some use:
https://github.com/famly/plan
Because so much of producing some bits (no pun intended) of code is repetitive setup.

Meanwhile, today, I will be working on getting custom chroot environments for cross-compiling and native compiling injected into the SuSE Studio build workflow.

= = = =

This is another of my projects started here that will probable die an early death due to lack of interest on the part of the community.
We are that short of contributors here with time on their hands . . . .

[knc1]

Its my thread, I'll drag it a little bit off-topic.
Although the topic does involve one of the (free) SuSE services, SuSE Studio.

MicroFocus :: The company that might be seeing itself as the early 20th century Standard Oil:
2014: http://www.pcworld.com/article/26843...2-billion.html
2017: https://www.ft.com/content/976c93f8-...30f58?mhq5j=e2
20xx: Will IBM be next?

ah, wait a minute on that one -
Back in 2004, IBM spent $50 million to purchase preferred shares of Novell.
Preferred shares that where (are?) convertible into 8 million common shares of Novell.
https://www.microfocus.com/about/pre...bm_investment/

That 2014 deal above was an exchange of all public Novell shares for shares in the merged company.

I.E: IBM is one of the owners of Micro Focus.
Perhaps not the majority owner, but probably enough to slow down any thoughts MF might have of a buy-out or take over.

Meanwhile, back to work @ SuSE Studio - before someone buys it or sells it and it gets shut it down.

[knc1]

of note (unless I build my own):

mszick:bin> file qemu-arm-static
qemu-arm-static: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux),
statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=6b1020f44ae6614487917693a95a497dbc27857a, stripped

mszick:bin> cd -
/run/media/mszick/Misc_Archive/qemu-deb/amd64/data/usr/bin

mszick:bin> file qemu-arm-static
qemu-arm-static: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux),
statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=c460e428e5f20e519164e4093e364c8b7e31bda2, stripped

Which means it requires AT LEAST a linux kernel 2.6.32 or newer to run.
The Kindles where using 2.6.31 back in the days of PW-3, 5.6.1.1 (what the PW-3 was released with).

So the host kernel running the virtual development machine can not be as old as what Lab126 was using, back in the day (a couple years ago).

The need to run the actual Kindle kernel when running things under the emulation only occures in the case of the software running making system calls to entry points that only exist in their build of the kernel.
(I think there are some - at least as recently as the 5.2 firmware days.)

Ah, only a minor speed bump in the development process.
Only the KT and the PW1 are running 5.6.1.1 or earlier firmware.
The devices prior to those will also be left out in the cold, as far as emulation under the control of a Lab126 kernel build.

= = = =

I am also working on a 32bit VM image that can be run on 32bit Intel PCs.
Have not (yet) looked at a PPC VM image (so you Mac owners that haven't yet upgraded to OSx on Intel might be left out in the cold along with the keyboard Kindles).

[knc1]

Coercing the SuSE Studio build system to install Debian in a chroot environment is a bit of a challenge.
You would think that they only wanted you to install SuSE software.

Today's plan:
Three users (in addition to 'root'):
tux -> logs into the host side of the system (they can schroot into either of the build systems).

The build systems (currently) will be 32bit, armel (soft float) exclusively, regardless of the host being either 32bit or 64bit.
tuxn -> logs into the native build (chrooted) system (they can exit out of it).
tuxc -> logs into the cross-compile build (chrooted) system (they can exit out of it).

In addition to being a 32bit environment, the chrooted systems will be old enough versions that there should not be any conflicts between the version of libc they are running and the version(s) of libc that the Kindles are (or where) running).

I think that means I have to support back to 5.3.7.3 - the most recent firmware run by the oldest touchscreen Kindle.

Once that is done -
If there is demand enough, I could include one of twobob's development environments that are old enough to build for at least some of the keyboard models.

Currently, from the amount of feed back I see -
This project will grow cold as soon as I have an image that suits myself.
If there are not users to support, why spend time going past the 'works for me' stage.

[encol]

do you plan to support armhf in future?

[knc1]

Originally Posted by encol

do you plan to support armhf in future?

On the Kindle?
I supported it in the past with approximately zero response.

It was a working demonstration of how to run two ELF systems concurrently (in this case armel and armelhf).
See thread by me, "ARMhf on Kindle" or something like that, it has "ARMhf" in the title for certain.

= = = =

But in direct answer to your question -
Undecided, I want to keep them small.
I am, at the moment, in the midst of creating Debian environments.
And I will be making both soft float (for the Kindles) and hard float (for the general public - Kindles are perhaps the only ARM device still using soft float ).
I just don't know how to incorporate something for everybody.
But I'll have them on hand, anyway.

[encol]

Originally Posted by knc1

Kindles are perhaps the only ARM device still using soft float ).

But why? Aren't ARMv7 armhf?

cat /proc/cpuinfo
Processor       : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 1987.37
Features        : swp half thumb fastmult vfp edsp neon vfpv3

readelf -A /proc/self/exe
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "4T"
  Tag_CPU_arch: v4T

[knc1]

Originally Posted by encol

But why? Aren't ARMv7 armhf?

cat /proc/cpuinfo
Processor       : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 1987.37
Features        : swp half thumb fastmult vfp edsp neon vfpv3

readelf -A /proc/self/exe
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "4T"
  Tag_CPU_arch: v4T

No, they are still running softfloat code.
The earlier Kindles, with ARMv6 processor had neon and vfp -
but it wasn't used.

You need to pass the compiler:
-mfloat-abi=softfp
If you need to use any of the Amazon libraries. *
https://gcc.gnu.org/onlinedocs/gcc-7...ml#ARM-Options
Yes, I know that is for gcc-7.1 but that quote has been the same since at least gcc-3.1

The 'tag' area at the end of the binary is a report of the limits the compiler had set for its code generation.
It has nothing to do with the code generated. yeah, that's the gcc gang at work.

* More specifically, if you have to pass floating point arguments from hardfp code to softfp subroutine.
Then things go sideways.
But if you never pass floating point arguments, you can mix-and-match and maybe never crash.
I honestly don't recall of the caller saved and callee saved register sets match in the to floating point ABIs.

If you use: LD_DEBUG=libs with my ARMhf examples, you will see that it is calling into one (or two) Amazon soft float libraries.
Of interest is Lua (or LuaJit whichever I built) - the only number type in that language is double.
It wouldn't run long if ARMhf wasn't working on the Kindle processors.

[nasser]

To run armhf code, do Kindles have the FP unit in their hardware?

[knc1]

Originally Posted by nasser

To run armhf code, do Kindles have the FP unit in their hardware?

Yes.
Both floating point and vector processors.
That is what the 'hf' indicates, hardware floating point (in this case, also a vector processor).

PW2 and newer, In brief:
i.MX6SL with EPD

Gory details:
data sheet
Picture on page 7.
(If you think a 160+ page data sheet is something, you should read the reference manual - it has War and Peace beat out by a long shot.)

The KT and the PW1 use an earlier model SoC, but they also have hf and vector co-processors (with an earlier, smaller, instruction set, IIRC only integer SIMD).

I posted the reference manual here in the past.
Maybe in my threads about the PW3 or KT2 when they where new.

I also posted a KUAL extension here for selecting which cpu governor the kernel uses.
Pick the 'Performance' one and a modern Kindle can be quite impressive.

[nasser]

knc1,

You're saying KT, PW1, PW2, PW3, KT2, KT3, KV, KOA are ALL armhf compatible.
In that case, would CLI applications built on a debian-armhf image run OK on these Kindles?