Overlay for root filesystem

[Nyoxi]

Experimental

I have finally created a working overlay filesystem for the root filesystem. For those who have no idea what that means it allows to mount a writable filesystem over the root filesystem that is read-only. With that you can then create files and directories on the root filesystem without actually modifying the filesystem. Thus you keep the original filesystem unchanged and you can safely revert back simply by removing the overlay.

Currently it works only with Kindle Touch (both 5.0.0 and 5.0.3), other Kindles were not tested.
EDIT: Don't even try that on other Kindles. It won't work (for one thing they use upstart instead of SysV init).

To test this un-tar the attached files into /mnt/us (wile keeping the directory they're in) and run the included runme.sh (manually or use a jailbreak). After the operation finishes you will have bootstrap-overlay.log in /mnt/us too.

On real / it will change only /sbin/init (adds a line to run /sbin/pre_init) and creates a new file /sbin/pre_init. It will then create the filesystem for the overlay in /mnt/us/rootfs.img. The initial size is 30MB, you can use resize2fs to expand it later if needed.

I tried to make the startup process as safe as possible. But it's still experimental so I expect everyone to have at least a ssh access and basic linux experience.

If something goes wrong you can simply remove (rename) the rootfs.img file and restart the device.

There are also two files that you can create to prevent using the overlay:
/NO_PRE_INIT
/mnt/us/NO_OVERLAY

If you decide to use the overlay again, be sure to remove both files (NO_PRE_INIT will be created automatically if NO_OVERLAY exists or rootfs.img is missing) if you decide to use the overlay again.

Final notes:

Sometimes when comes the moment your screen flashes and progress bar should appear it takes quite long. Before the progress bar appears your screen is just black. But for me it never took more than 20 seconds.

The sources to build the mini_fo kernel module come from OpenWRT project (they include patches for newer kernels)

You can find all the files together with the build script for mini_fo here: http://hg.nyoxi.net/kindle-packages/

EDIT: Later when the overlay is more tested I plan to create single install tool that will setup the overlay and put opkg there so that anyone can install new packages easily. (More about opkg here: https://www.mobileread.com/forums/sho...d.php?t=167579)

EDIT: I forgot to mention a fact that it is safe to run the runme.sh multiple times. You can safely use just to create new overlay filesystem.
EDIT: But don't do that while you are using another overlay because md5sum checks will fail.

[hostar]

According to notes in runme.sh,
patch utility for ARM could be downloaded here.
I didn't test the utility, I'm writing this just because there might be that patching option.

Also: good job Nyoxi

[Nyoxi]

Originally Posted by hostar

patch utility for ARM could be downloaded here.

Thanks for the tip. Of course I could use my own patch. But I like to keep the packages self-sufficient as much as possible. Especially without mysterious binaries.

[geekmaster]

Requested md5sums for k4 overlay support: (attachment removed, no longer need it here).

[Nyoxi]

Ok, K4 seems to be way too different from K5. I can't do it without having the device. Later today or tomorrow I will make a post about the technical aspect of setting up the overlay so somebody can cach up on that.

[dave2008]

Originally Posted by Nyoxi

Ok, K4 seems to be way too different from K5. I can't do it without having the device. Later today or tomorrow I will make a post about the technical aspect of setting up the overlay so somebody can cach up on that.

That's awesome! Hope it can be ported to K3. Can't wait to try it on my kindle.

[Nyoxi]

I have updated the first post because there was a small mistake in system_monitor.conf patch. I have also forgotten to mention that while you can use the script to regenerate rootfs.img you can't do that while other overlay is in use because md5sum checks will fail.

[Nyoxi]

This post deals with the technical aspects of setting up the overlay and taking it down. So if you're just curious or you want to use it on other Kindles then read on. Otherwise just move along or it will make your head hurt.

Note on naming:

• read only root (FS) is the original filesystem mounted on / that everyone has now
• overlay image (or overlay filesystem) is the filesystem we have stored in file on /mnt/us containing all the modifications we would like to have on read only FS
• writable root (FS) is the virtual filesystem created by combining read only root with overlay image

For simplicity I will not post any shell code here, for that refer to the files. You should not have a problem finding what you need there.

Before the init

To properly setup the overlay on the root FS (or changing the root FS in general) you have to do it at an early stage. And early really means early -- before the init kicks in. To do that you have two choices:

1. do it in initrd before init is called
2. replace the init with your own and call the real init later

The first option is more difficult since it means patching the cpio archive "hidden" in /dev/mmcblk0. Option 2) is easier, so we choose to replace real init with our shell script that would do all the dirty work. Moreover, the real init already is the shell script! What a surprise. The /sbin/init on K5 checks whether it is supposed to use upstart or (older) SysV init and then calls the real init (which is /sbin/init.exe). Another advantage of using second option is that we have the real root FS already to our disposal.
EDIT: Removed the note that you need /bin/sh in initrd for 2), in fact you would need it for 1). For 2) you have /bin/sh from the root filesystem.

To keep things simple I decided to put our script into /sbin/pre_init and the only change to /sbin/init is execution of /sbin/pre_init right before /sbin/init.exe is called.

Do not forget that this is tricky task. And believe me, you don't want to let Kindle start with only a half of the job done. You can bet on it that it will have some issue with that. That means you have to track all you do so you can roll back later in case of an error.

To setup the writable root FS you have to do the following steps:

1. You will need a /proc otherwise mount will refuse to work (busybox?)
2. Next you have to make yourself a /dev with everything you will need before init starts udev. Apart from loop devices or partitions this also means /dev/console and /dev/null, and if you want to write on the eink display also /dev/fb0.
3. After that you can start mounting. The first thing you need is /var/local because it contains file mntus.params necessary to mount /mnt/base-us. For those who don't know, mmcblk0p4 does not contain /mnt/base-us directly. It contains also a partition table and then the filesystem.
4. When you have mntus.params you can mount /mnt/base-us that contains the overlay image. It's better to let 'mntus --fsck mount' prepare the loop device and mount it for us so we don't have to deal with it. However note that 'mntus umount' will just umount the filesystem, but will not free the /dev/loop/0 loop device! (!)
5. Before we mount image with an overlay it is a good idea to check it for errors with fsck. But wait! You say that last mount was done in the future?! But how is that ... ah right, the clock is screwed up (!). One problem is that kindle uses it's own epoch and changes the date during boot but doesn't bother storing the new date to hardware clock. The second problem is that when you do a hard reset of your device the clock is reset. Good thing is we can tell e2fsck to ignore the problem with 'mounted in the future', the bad thing is that the only way to do that is by using a configuration file (no command line option). Rather than creating new file on read only root I decided to feed the configuration file to e2fsck on the input (through /dev/stdin).
6. When the overlay image exists and is clean we can mount it to /overlay.
7. We load mini_fo kernel module contained on the image and mount the writable root on /root-rw created as combination of read only root (on /) and our overlay image (on /overlay).
8. This is correct time to do any clean up necessary, for us this means umounting /var/local. Unfortunately we can't umount /dev because it is in use (remember the loop device for /mnt/base-us?). And as for the /proc ... we will need it later.
9. Now we're almost ready to change the root to our new writable /root-rw by invoking pivot_root. Before we do that we have to remount the procfs from /proc to /root-rw/proc. Again, we do that so that we can call mount later. After the procfs is where we need it we call 'pivot_root /root-rw /root-rw/root-ro'. This will set new / to be located where /root-rw used to be and place old / where /root-rw/root-ro used to be. In plain English: we will have our writable root on / and the old read only root on /root-ro.
10. The only thing left to do is move all previously mounted filesystems (now located on /root-ro/dev, /root-ro/overlay and /root-ro/mnt/base-us) to correct locations: /dev, /overlay, /mnt/base-us

When all is done it's time to invoke the real init. But instead of using the path /sbin/init.exe which points to our writable root we use /root-ro/sbin/init.exe. That way the binary /sbin/init.exe on our writable root is not used and we can properly remove the writable root later during shutdown.


Safety:

To stay safe in case something bad happens I decided to introduce two sentinel files. We don't want users to get stuck with unusable devices, do we? The first file is /NO_PRE_INIT. If this file exists pre_init is not executed and the control is returned to /sbin/init. The file is created automatically at the beginning of pre_init and removed before pre_init pivots the root. If something bad happens it will contain the error message.

Second file is /mnt/base-us/NO_OVERLAY. If it exists the overlay is not mounted and (after proper cleanup) the control is again returned to /sbin/init. It is automatically created before NO_PRE_INIT is removed and it is removed after successful startup (see below).

You are free to create any of these files by hand to disable the whole process.

At this point existence of NO_OVERLAY also leads to the existence of NO_PRE_INIT. This is mainly to discourage the commoners without Linux experience (and no ssh access). After the whole thing is properly tested we can change things to just keep NO_OVERLAY so user can remove it from the computer by USB mount.


The startup and normal use

EDIT: I forgot to mention that: It is strange, but the Kindle software polutes the root filesystem with files that could prevent proper startup. This is most likely because it doesn't expect writable filesystem and what would silently fail now works. It is good to remove such files before anything else is started. One place to do that might be pre_init, but I rather chose to do it in first init job that is executed (in our case system.conf).

Remember that we have already mounted /mnt/base-us ourselves. Startup job called filesystems.conf doesn't know that and tries to call 'mntus' to mount it. This will fail and the job get's confused and dies. We don't want that. Neither we want anyone to umount the filesystem because we're using it. Solution is easy: replace 'mntus' with fake script that does nothing.

Unfortunately by creating the writable root we have created an environment the kindle software was not ready for. The main problem being that lot of things likes to remount the root writable, do a couple of things and remount back to read only. The problem is two-fold. First we would like to keep the root writable and second mini_fo filesystem doesn't even support remounting. Again replacing '/usr/sbin/mntroot' with a fake script that does nothing solves most of the troubles. But there is still a problem -- /etc/upstart/firsttime script called by system.conf upstart job and system.conf itself. Both try to call 'mount' directly and don't rely on 'mntroot'. So we patch system.conf not to do the remount and not to call '/etc/upstart/firsttime' (it's not a first boot anyway, and it may screw things we did on purpose).

After the framework is up upstart emits the 'framework_ready' event. This triggers our after_boot.conf job which removes NO_OVERLAY sentinel. I assume that at this point all should be OK and user can use USB mount if necessary.

NOTE: Sometimes the NO_OVERLAY is not removed. I don't know yet whether the after_boot.conf is not started at all, or it fails for some reason. (I don't even know how to reproduce this.)

Shutdown

To keep our overlay filesystem free of errors it is a necessary to properly umount the overlay. For that we have to switch back to the read only filesystem.

The upstart job that handles power-off and restarts is called shutdown.conf. In it's original version it doesn't do much besides stopping framework and showing a neat image. For us this is not enough. First we need to stop all other upstart jobs that are still running. This also includes the system.conf script and there lies another issue. This job is expected to never stop and when it does system_monitor.conf job kicks in and causes a restart. Again, we don't like that so we patch the job to ignore situations when system.conf stops but shutdown.conf is running.

After all the upstart jobs are stopped we have to make sure no other programs are running. The only things we want to keep is ourselves, our parent process and the init. The rest gets sent TERM signal and later KILL if necessary.
EDIT: After we kill what we can we try to umount all that we can.

When no other pocess is running we again remount the /proc to /root-ro/proc and pivot the root back to /root-ro. After that we have to free the shutdown.conf file (!) so that we can umount the writable root fs. This is done by starting a script 'shutdown_real'. That handles the rest of umounting and moves all the mounts we can't remove to /root-ro. It also does all the other things shutdown.conf was supposed to do. But, simply running 'shudown_real' will not do. That would change one open file for another. We have to move it onto another filesystem first. However most of them were already umounted, and modifying some filesystem is not nice either. Unfortunately at this point the only tmpfs available to our disposal is /root-ro/dev. It's not clean, but I have decided to use it anyway. That is, the 'shutdown_real' is first copied to /root-ro/dev and then started from there in the background. That way the upstart job can terminate.

While all the steps are important, I have tried to point out the few most important things with (!) in the text.

It took me quite a time to write this post, hopefully it wasn't in vain.

[geekmaster]

Originally Posted by Nyoxi

This post deals with the technical aspects of setting up the overlay and taking it down. So if you're just curious or you want to use it on other Kindles then read on. Otherwise just move along or it will make your head hurt.
...
It took me quite a time to write this post, hopefully it wasn't in vain.

This post is very well done and very appreciated by me, which by itself should make it NOT in vain. I certainly plan to use this.

Bravo! Kudos! Thank you! Teşekkür ederim! Спасибо большое! (and all that stuff).

[Nyoxi]

After the much needed sleep I went through my previous post about technical aspects. I have corrected a few misconceptions and added some missing info.

[geekmaster]

You can do "dd if=/dev/zero of=/dev/mmcblk0p3" before and after updates if you want to. The startup scripts format it when it fails to mount, and copy stuff from /opt. That means any updates that could affect /var/local are just COPIES of data in /opt. Look in the startup scripts. You should save user settings before an update (collections database, locale, timezone, etc) and restore them after an update and /var/local rebuild.

This is just a suggestion of things you can check into (especially the scripts that rebuild /var/local). Of course it would need more research and development before you use it, but it may simplify the OTA update problem. Removing the /var/local partition (after backing up user data such as his collections database, locale and timezone settings) to force a rebuild is perhaps more useful for the "bricked kindle recovery" methods I have been developing, but I though it might be worth studying for this application as well.

EDIT: If Post #12 should be before this post as it says, then the text in Post #12 can be moved to the end of Post #10. That would get this thread back in sync, and I can remove this message. Also, you can read (and search) the IRC archive if that may help you remember some missing details:
http://kindle-synchrone.dotcloud.com/

[Nyoxi]

The previous post (#11) is somewhat out of sync. It was supposed to follow this post (which however nobody posted yet).

On IRC we were discussing with nueva the possibility of using OTA updates with the overlay. The issue here is that the OTA update can patch files on root filesystem, files in /var/local, kernel and possibly also uboot. This could result in only part of your system being updated after you remove the overlay. The problem seems to be in whatever scenario you use.

1. If you [b]don't[b] disable overlay before the update: it changes both the root file system
   and files in /var/local then you will have inconsistency at the moment you remove your overlay (e.g. to use a clean one after something breaks.
2. If you disable overlay before the update: you
   may not see some of the updated files because you have them modified on the overlay and they are overshadowing the updated files on read-only root.

Right now we seem to have one only option: disable the overlay, apply update, create new clean overlay.

At some later point we can make use of the reconstruction of /var/local ... but I can't remember what was the use. I realy had an idea here that I can't remember right now (I should have written all this sooner).

[Nyoxi]

knc1 suggested here using different filesystem for the overlay instead of mini_fo: https://www.mobileread.com/forums/sho...4&postcount=60

Althoug kindle kernel is older as he says we still could be able to use older version of auFS. From the auFS site

linux kernel version aufs version
2.6.31 ... 2.6.34 aufs2.1, Unsupported since Aug 2011

EDIT: I forgot to mention that KT uses 2.6.31 kernel.

I have no objection to using something else. I have never worked with overlay filesystem and I made use of mini_fo only because nueva suggested it and because OpenWRT guys seem to be using it happily. The problem with mini_fo however is that it hasn't been updated by the author for last 5 years and it only works because OpenWRT has buch of it's own patches to support newer kernels.

So if you have any other ideas feel free to share.

[eureka]

I've installed overlay and using it for some days without any errors or fails. Thanks!

[knc1]

Originally Posted by Nyoxi

knc1 suggested here using different filesystem for the overlay instead of mini_fo: https://www.mobileread.com/forums/sho...4&postcount=60

Althoug kindle kernel is older as he says we still could be able to use older version of auFS. From the auFS site

EDIT: I forgot to mention that KT uses 2.6.31 kernel.

Some Kindle models are not so lucky but according to the FreeScale git repo, they are maintaining a 2.6.38 version (which would be a nice, general purpose, upgrade to all Kindle models).

For a pair of tutorials on using auFS, try:
http://drpbox.knetconnect.com/aufs/
They are each "single file" html pages, read on-line or wget your own copy.

The "part 3" tutorial is still only in draft form, not yet posted.
It shows how to modify the layers while the auFS filesystem stack is in use (hot).