opkg, opkg GUI and plans on package repository for Kindle Touch

[Nyoxi]

Hi everyone! I have a big plan but little time and things don't go as fast as I would like them to. That's why I decided to make this post so you know what it's easier to track how the things are going.

Note at the beginning: I have Kindle Touch and so what I did and what I'm interested in considers KT. But even though you're interested in other Kindles keep reading as this might be interesting for you too.

Long story short: I would like to see a working package manager on kindle and have package repository that could be used also by non-geeks to install hacks/apps/games/etc. I was thinking that user would just use jailbreak to drop in the package manager and later install packages with the manager (either downloaded by wifi or uploaded on kindle from computer).

Now I realize that this is in fact a long run and will take some time to achieve. But I was quite surprised that there's no such initiative been going on (or considered). So I have decided to start working on that (I am expecting your comments about this "vision" of mine).

Now first things first. What you need to consider before you can even start something like this:

1. a package manager
2. GUI frontend that could be easily operated with the "limited" input capabilities of Kindles
3. a place where to install the packages
4. a common way of launching the installed applications

ad 1)

I am well aware that people are using the update feature of Kindles' to install stuff. I don't like this option for several reasons:

• you need one .bin to install something and .bin to uninstall
• the forced restarts after update are annoying and could be avoided in many cases
• tracking installed files with shell scripts seems old-fashioned and is prone to errors
• file conflicts or package conflicts are hard to handle
• gamble with developer keys

and you could find more. Opposed to that with package management system you don't have this issues, more over you get:

• dependency tracking
• you see what is in the repository and what you can install (no need to search the forums)
• easily install packages from net (thorough wifi)
• easy updates from net (through wifi)

For small devices it should be OK to have something lightweight. People here have already played with ipkg (simple debian like package manager) from optware. The ipkg developement has been discontinued but there is also it's fork opkg that can be considered a successor.

I have compiled it to a package (see the end of the post for the links) and also have statically linked version that I use to bootstrap the packaging system). And it works (you can e.g. install stuff from optware repositories with it). No problem there except for one thing. Now it is set to store it's files in /var/local (the only writable partition besides /mnt/us). Sadly this now seems like a poor choice as KT bricks when the partition gets full (can't find the thread about it now). But this will get solved by addressing the point 3).

Now you may ask why bother with opkg, why not use the available ipkg and optware. There are known issues with the ipkg solution (like wrong wget arguments) and as I have already said it's been discontinued. As for the optware, it is good for geeks and you can install whatever you know if you know what you're doing (I did that too). But sooner or later you will run into problems like missing files that are expected to exist in the system (remember that the packages were meant for different system) or wrong versions of the existing files. So in a long run it's not an option.

From now on I'll try to be more brief.

ad 2)

I have started on this and made Gtk+2 frontend to the opkg. You can find the pre-alpha version in links below. There are still flaws to be addressed, but the concept works.

If you would want to try on KT you will also need a few hacks to gtkrc (because it's poorly set up) that I can't find at the moment. I will add this later.

ad 3)

Now this is a tricky one. My first idea is do mount rw/ro during install and put files on the root partition (but it seems a little dirty). The only writable partitions are /var/local and /mnt/us and there's not much space on /var/local. As for /mnt/us it's FAT which means:

• no permissions
• no symlinks

and I hate the idea that the files could be accessed after connecting the device to the computer:

• user or some application may (accidentaly) screw something up
• if you have it for your own purposes it's ok, but if we want it to be common approach used by all users it's a potential security hole and therefor bad practice.

A loop device placed in /mnt/us and mounted on /opt is an option. It solves the first problem, but the issue with potential security hole remains. So what I want to try is use a FS encryption. Further more nueva suggested using an overlay filesystem on root and I find this idea reather interesting.

ad 4)

Unlike with other Kindles where there seem to be tons of different ways to launch apps this seems quite simple for KT. To my knowledge there's just one (yifanlu's) launcher, is that correct? From what I have read about it so far I like the design so I'll try to make a package for this soon too.

So for me the next steps are:

1. check the overlay kernel module
2. check the encryption
3. prepare tools to create the loop FS (that would also allow changing it's size)
4. finish the GUI
5. check Openembed as a possible platform
6. make more packages

If anyone is willing to give a helping hand don't hesitate.

I am personally interested only in Kindle Touch (because that's what I have) but in my wild imagination I don't see a reason why this couldn't work for all kindles. (You can distinguish packages by architecture field if necessary.) What more, maybe even a OS replacement for different kinds of readers (not just Kindles). But let's not get too excited yet.

If you managed to read all this you are either very interested or very opposed and want to tell me something really ugly. So now is the time to relieve the tension by leaving a comment.

Links:

Statically compiled version of opkg is here:
http://nyoxi.net/files/kindle/opkg-bootstrap.tar.bz2
EDIT: But don't use that, I suggest you use the attached installer for opkg and overlay (for writable root filesystem).

My packages are here (not much there yet):
http://nyoxi.net/files/kindle/packages/

The opkg GUI sources and build scripts for packages are here:
http://hg.nyoxi.net/

EDIT: I have attached archive with opkg & root fs overlay (see this thread https://www.mobileread.com/forums/sho...d.php?t=170261). To install just unpack it into userstore (under boostrap directory) and run runme.sh.
If you have already installed opkg with my original script you should do the following to remove it first:

mntroot rw
opkg-cl remove '*'
opkg-cl remove --force-remove '*'
opkg-cl remove --force-removal-of-essential-packages '*'
rm -fr /var/local/opkg /usr/lib/opkg/
mntroot ro

That will uninstall all the packages you have already installed and remove the opkg itself. After each opkg-cl you should check for lines "Not deleting modified conffile <some_file>" and if you modified that file on purpose then store the files somewhere so you can put them back on the overlay. Either way you should remove the files from the real root filesytem.

When the runme.sh has finished you have to restart the device. After that run opkg-cl update and you're ready to rock&roll again. Edit /etc/opkg/opkg.conf to your liking, now it contains the nslu2 Optware repository.

[fbdev]

kite makes it very easy to install onto the kindle 3.

just drop a self-extracting script into kite/ondrop and unplug the kindle.

the script will run; changes to the device will be immediate.

no reboot.

additionally, it has the capacity to run scripts at boot time.

just drop a script in kite/onboot and it will run every time the device starts.

[JustAMan]

Nyoxi,
Good to read this somewhere except IRC, keep up the good work man!

BTW I don't get what you mean by "security hole" if you do non-encrypted loop mount. It still would be almost always mounted, right? As I think this would be the place where you store installed stuff. Thus it would always be accessible from the KT to any application that has sufficient rights (however I believe everything there runs as root... if I were you I'd better look into running user stuff under non-root account first as it seems like much bigger hole than that somebody could plug Kindle to Linux PC and mount your loop partition and change it... if they're skilled enough to do this they're almost certainly skilled enough to get SSH over USB and mess with things on Kindle themselves)

And I would certainly be against installing anything from packaging system to root FS. If something went wrong with this soft and it broke my KT I can remove this loop mount file from USB drive, reboot and be safe, but if it's installed right into root FS I'm doomed. That is, I fully support your selected option of loop partition.

That possible breakage not that important for update.bin technology yet because most devs test their update.bins manually before uploading, but when you create some public repo there may be some, say, nightly updates and such.

P.S. I know Linux a bit, I know programming a bit (do not know GTK but I can learn if needed). I'm interested in giving you a hand when I have spare time. And I think there're many people around here who could give a hand, too.
So feel free to point where you need some help. I think that would benefit the project

[JustAMan]

Any progress or help needed? Seems like this topic was quiet for quite a long time...

[knc1]

Originally Posted by JustAMan

Any progress or help needed? Seems like this topic was quiet for quite a long time...

Not too surprising, have you checked out the vast number of Kindle projects in the MobileRead development support site?
https://dev.mobileread.com/trac/hub/

[JustAMan]

Nope. I wasn't browsing this site a lot, only using its forum. Thanks for pointing!

[knc1]

Originally Posted by JustAMan

Nope. I wasn't browsing this site a lot, only using its forum. Thanks for pointing!

Sorry
I should have put a "sarcasum" icon on that statement.

The "News" section hasn't been updated in four years;
The "Kindle" section is still empty, after being up for six years.

Going by that, you could probably just set your browser to check this thread once every ten years or so to see if anything has happened.

[JustAMan]

Yeah, I saw what you meant when I actually had time to look at the page That was like 10 minutes ago, way after I answered to your post

[Nyoxi]

Originally Posted by JustAMan

Any progress or help needed? Seems like this topic was quiet for quite a long time...

Sorry, no progress. I've been busy with other things, hopefully this week will be better.

Originally Posted by JustAMan

Nyoxi,
BTW I don't get what you mean by "security hole" if you do non-encrypted loop mount. It still would be almost always mounted, right? As I think this would be the place where you store installed stuff. Thus it would always be accessible from the KT to any application that has sufficient rights (however I believe everything there runs as root... if I were you I'd better look into running user stuff under non-root account first as it seems like much bigger hole than that somebody could plug Kindle to Linux PC and mount your loop partition and change it... if they're skilled enough to do this they're almost certainly skilled enough to get SSH over USB and mess with things on Kindle themselves)

I wasn't talking about danger from the applications, but of course running them not under the root account is something we should definitely consider. Neither did I mean the case where somebody takes your kindle to do evil stuff to it (because at that point there's not much anybody can do about it). My point was only concerning malicious software on the host computer.

But you made me think again about all this encryption stuff and whether it's really worth the effort. Because with those data.tar.gz jailbreaks all you have to do is trick the user to restart their kindle. And that's not so difficult.

And I would certainly be against installing anything from packaging system to root FS. If something went wrong with this soft and it broke my KT I can remove this loop mount file from USB drive, reboot and be safe, but if it's installed right into root FS I'm doomed. That is, I fully support your selected option of loop partition.

Yes, I totally agree. Writing directly to the root FS could be dangerous. Also being able to fix things when something breaks just by deleting the loop file is great. But in some cases you cannot avoid modifying the files in root FS and that's why having the root FS overlay is so beautiful. It's a win-win situation: you are able to modify the files on root FS without touching the root FS.

That possible breakage not that important for update.bin technology yet because most devs test their update.bins manually before uploading, but when you create some public repo there may be some, say, nightly updates and such.

Of course this is something we will have to deal with. My idea is to have more than one repository. One repository with tested and safe apps/hack that common (non-geek) users could use. Only selected group of reliable people would be able to push packages into this repository. Second some testing repo for experienced users who are not afraid to experiment. It would contain stuff that is known (more or less) to work. Also testing of new versions of applications would happen here. And third, some highly experimental repository for all the new hacks/apps. Or devs could make their own small repositories instead of having the third repo. That's my idea of it.

P.S. I know Linux a bit, I know programming a bit (do not know GTK but I can learn if needed). I'm interested in giving you a hand when I have spare time. And I think there're many people around here who could give a hand, too.
So feel free to point where you need some help. I think that would benefit the project

I think having good experience with Linux and shell scripting is essential at this point. What you (or somebody) could do to help is:

1. Look for the buildroot that we could use. Something that would be easy to set up and which would provide good support for writing package build scripts. I was thinking about Openembeded. Taking the framework from the OpenWrt and strip it down to things that are necessary or useful to us is also an option. Or if someone has any other tips feel free to entertain me.
   
2. Maybe write an init script that would load the necessary modules and mount the loop file. It might do some sanity checks to make sure it's safe to mount the loop file or check for some file in /mnt/us that would allow user to skip the mounting. (Or maybe even something else ... thoughts/suggestions?)

[JustAMan]

Originally Posted by Nyoxi

I think having good experience with Linux and shell scripting is essential at this point. What you (or somebody) could do to help is:

1. Look for the buildroot that we could use. Something that would be easy to set up and which would provide good support for writing package build scripts. I was thinking about Openembeded. Taking the framework from the OpenWrt and strip it down to things that are necessary or useful to us is also an option. Or if someone has any other tips feel free to entertain me.
   
2. Maybe write an init script that would load the necessary modules and mount the loop file. It might do some sanity checks to make sure it's safe to mount the loop file or check for some file in /mnt/us that would allow user to skip the mounting. (Or maybe even something else ... thoughts/suggestions?)

I'll have a look at those links. I don't have much experience with C++ GUI programming (I've started as Delphi programmer as a hobby, and now I'm Python guy on my work).

I can also write basic init.d scripts (and most of them are really basic ), however I think we should refer to them as "upstart" scripts (KT uses upstart, right?).

What I propose is having some kind of site/wiki that describes what needs to be done and advertise this somehow so people could go there and help.

BTW, as for "stripping the framework" from OpenWrt (I didn't read the links yet) - is it really needed to be stripped? I think if it would have some more MBs it's okay - we can afford this at least at the start because we'll save developers' time (and seeing we don't have many developers out there I think it's really critical to spend as little time as possible to get this working safe & reliable).

[Nyoxi]

I have finally managed to get the overlay filesystem working. More here: https://www.mobileread.com/forums/sho...76&postcount=1

This brings us another big step forward.

@JustAMan: did you find out anything usefull about the openembeded tool?

Later when the overlay is more tested I will create single install tool that will setup the overlay and put opkg there.

[JustAMan]

Nope, didn't have time yet to mess with this... A bit busy at my work.

[knc1]

@nyoxi
Are you running your Kindle image under QEMU where you have full control of what happens?

Do you ever use HG?
My HG repository supports public-published, public-unpublished, and per-user-unpublished repos.

PM me if interested in other details.

[Nyoxi]

Originally Posted by knc1

@nyoxi
Are you running your Kindle image under QEMU where you have full control of what happens?

No. I work on the device. And I have debian arm in qemu for compiling. But I would like to switch to crosscompiling at some point because the emulation is rather slow.

Originally Posted by knc1

Do you ever use HG?

Yes, my repositories are HG. (See the links at the end post #1)

[Nyoxi]

Good news everyone! It looks like knc1 is able to supply us with the repository for packages (and more!) so this thing will have a place to live.

I'll start looking at the Openembeded to see what magick we can do with that.

Later next week I will prepare single installer for the overlay+opkg combo.

What is still in the alpha stage is the GTK+ frontend. I wasn't able to do any coding on that lately and I don't expect this will change much. Fortunatelly this isn't a pressing issue because I expect you pepole can handle console app. But if there's a C developer skilled in GTK+ feel free to give a helping hand.