02-25-2012, 11:03 AM | #1 |
Connoisseur
Posts: 55
Karma: 124493
Join Date: Jan 2012
Device: Kindle Touch
|
Overlay for root filesystem
Experimental
I have finally created a working overlay filesystem for the root filesystem. For those who have no idea what that means it allows to mount a writable filesystem over the root filesystem that is read-only. With that you can then create files and directories on the root filesystem without actually modifying the filesystem. Thus you keep the original filesystem unchanged and you can safely revert back simply by removing the overlay. Currently it works only with Kindle Touch (both 5.0.0 and 5.0.3), other Kindles were not tested. EDIT: Don't even try that on other Kindles. It won't work (for one thing they use upstart instead of SysV init). To test this un-tar the attached files into /mnt/us (wile keeping the directory they're in) and run the included runme.sh (manually or use a jailbreak). After the operation finishes you will have bootstrap-overlay.log in /mnt/us too. On real / it will change only /sbin/init (adds a line to run /sbin/pre_init) and creates a new file /sbin/pre_init. It will then create the filesystem for the overlay in /mnt/us/rootfs.img. The initial size is 30MB, you can use resize2fs to expand it later if needed. I tried to make the startup process as safe as possible. But it's still experimental so I expect everyone to have at least a ssh access and basic linux experience. If something goes wrong you can simply remove (rename) the rootfs.img file and restart the device. There are also two files that you can create to prevent using the overlay: /NO_PRE_INIT /mnt/us/NO_OVERLAY If you decide to use the overlay again, be sure to remove both files (NO_PRE_INIT will be created automatically if NO_OVERLAY exists or rootfs.img is missing) if you decide to use the overlay again. Final notes: Sometimes when comes the moment your screen flashes and progress bar should appear it takes quite long. Before the progress bar appears your screen is just black. But for me it never took more than 20 seconds. The sources to build the mini_fo kernel module come from OpenWRT project (they include patches for newer kernels) You can find all the files together with the build script for mini_fo here: http://hg.nyoxi.net/kindle-packages/ EDIT: Later when the overlay is more tested I plan to create single install tool that will setup the overlay and put opkg there so that anyone can install new packages easily. (More about opkg here: https://www.mobileread.com/forums/sho...d.php?t=167579) EDIT: I forgot to mention a fact that it is safe to run the runme.sh multiple times. You can safely use just to create new overlay filesystem. EDIT: But don't do that while you are using another overlay because md5sum checks will fail. Last edited by Nyoxi; 02-25-2012 at 05:56 PM. Reason: don't try on other kindles |
02-25-2012, 11:53 AM | #2 |
Zealot
Posts: 138
Karma: 12324
Join Date: Dec 2011
Location: CZ
Device: Kindle 4 non-touch
|
According to notes in runme.sh,
patch utility for ARM could be downloaded here. I didn't test the utility, I'm writing this just because there might be that patching option. Also: good job Nyoxi Last edited by hostar; 02-25-2012 at 11:55 AM. |
Advert | |
|
02-25-2012, 12:33 PM | #3 | |
Connoisseur
Posts: 55
Karma: 124493
Join Date: Jan 2012
Device: Kindle Touch
|
Quote:
|
|
02-25-2012, 01:20 PM | #4 |
Carpe diem, c'est la vie.
Posts: 6,433
Karma: 10773668
Join Date: Nov 2011
Location: Multiverse 6627A
Device: K1 to PW3
|
Requested md5sums for k4 overlay support: (attachment removed, no longer need it here).
Last edited by geekmaster; 02-25-2012 at 01:34 PM. |
02-25-2012, 01:59 PM | #5 |
Connoisseur
Posts: 55
Karma: 124493
Join Date: Jan 2012
Device: Kindle Touch
|
Ok, K4 seems to be way too different from K5. I can't do it without having the device. Later today or tomorrow I will make a post about the technical aspect of setting up the overlay so somebody can cach up on that.
|
Advert | |
|
02-25-2012, 04:12 PM | #6 |
Addict
Posts: 251
Karma: 183457
Join Date: Jan 2012
Device: k3G, KDXG, AuraHD
|
That's awesome! Hope it can be ported to K3. Can't wait to try it on my kindle.
Last edited by dave2008; 02-25-2012 at 11:23 PM. |
02-25-2012, 05:35 PM | #7 |
Connoisseur
Posts: 55
Karma: 124493
Join Date: Jan 2012
Device: Kindle Touch
|
I have updated the first post because there was a small mistake in system_monitor.conf patch. I have also forgotten to mention that while you can use the script to regenerate rootfs.img you can't do that while other overlay is in use because md5sum checks will fail.
|
02-25-2012, 05:54 PM | #8 |
Connoisseur
Posts: 55
Karma: 124493
Join Date: Jan 2012
Device: Kindle Touch
|
Technical details of the overlay mounting proces
This post deals with the technical aspects of setting up the overlay and taking it down. So if you're just curious or you want to use it on other Kindles then read on. Otherwise just move along or it will make your head hurt.
Note on naming:
For simplicity I will not post any shell code here, for that refer to the files. You should not have a problem finding what you need there. Before the init To properly setup the overlay on the root FS (or changing the root FS in general) you have to do it at an early stage. And early really means early -- before the init kicks in. To do that you have two choices:
The first option is more difficult since it means patching the cpio archive "hidden" in /dev/mmcblk0. Option 2) is easier, so we choose to replace real init with our shell script that would do all the dirty work. Moreover, the real init already is the shell script! What a surprise. The /sbin/init on K5 checks whether it is supposed to use upstart or (older) SysV init and then calls the real init (which is /sbin/init.exe). Another advantage of using second option is that we have the real root FS already to our disposal. EDIT: Removed the note that you need /bin/sh in initrd for 2), in fact you would need it for 1). For 2) you have /bin/sh from the root filesystem. To keep things simple I decided to put our script into /sbin/pre_init and the only change to /sbin/init is execution of /sbin/pre_init right before /sbin/init.exe is called. Do not forget that this is tricky task. And believe me, you don't want to let Kindle start with only a half of the job done. You can bet on it that it will have some issue with that. That means you have to track all you do so you can roll back later in case of an error. To setup the writable root FS you have to do the following steps:
When all is done it's time to invoke the real init. But instead of using the path /sbin/init.exe which points to our writable root we use /root-ro/sbin/init.exe. That way the binary /sbin/init.exe on our writable root is not used and we can properly remove the writable root later during shutdown. Safety: To stay safe in case something bad happens I decided to introduce two sentinel files. We don't want users to get stuck with unusable devices, do we? The first file is /NO_PRE_INIT. If this file exists pre_init is not executed and the control is returned to /sbin/init. The file is created automatically at the beginning of pre_init and removed before pre_init pivots the root. If something bad happens it will contain the error message. Second file is /mnt/base-us/NO_OVERLAY. If it exists the overlay is not mounted and (after proper cleanup) the control is again returned to /sbin/init. It is automatically created before NO_PRE_INIT is removed and it is removed after successful startup (see below). You are free to create any of these files by hand to disable the whole process. At this point existence of NO_OVERLAY also leads to the existence of NO_PRE_INIT. This is mainly to discourage the commoners without Linux experience (and no ssh access). After the whole thing is properly tested we can change things to just keep NO_OVERLAY so user can remove it from the computer by USB mount. The startup and normal use EDIT: I forgot to mention that: It is strange, but the Kindle software polutes the root filesystem with files that could prevent proper startup. This is most likely because it doesn't expect writable filesystem and what would silently fail now works. It is good to remove such files before anything else is started. One place to do that might be pre_init, but I rather chose to do it in first init job that is executed (in our case system.conf). Remember that we have already mounted /mnt/base-us ourselves. Startup job called filesystems.conf doesn't know that and tries to call 'mntus' to mount it. This will fail and the job get's confused and dies. We don't want that. Neither we want anyone to umount the filesystem because we're using it. Solution is easy: replace 'mntus' with fake script that does nothing. Unfortunately by creating the writable root we have created an environment the kindle software was not ready for. The main problem being that lot of things likes to remount the root writable, do a couple of things and remount back to read only. The problem is two-fold. First we would like to keep the root writable and second mini_fo filesystem doesn't even support remounting. Again replacing '/usr/sbin/mntroot' with a fake script that does nothing solves most of the troubles. But there is still a problem -- /etc/upstart/firsttime script called by system.conf upstart job and system.conf itself. Both try to call 'mount' directly and don't rely on 'mntroot'. So we patch system.conf not to do the remount and not to call '/etc/upstart/firsttime' (it's not a first boot anyway, and it may screw things we did on purpose). After the framework is up upstart emits the 'framework_ready' event. This triggers our after_boot.conf job which removes NO_OVERLAY sentinel. I assume that at this point all should be OK and user can use USB mount if necessary. NOTE: Sometimes the NO_OVERLAY is not removed. I don't know yet whether the after_boot.conf is not started at all, or it fails for some reason. (I don't even know how to reproduce this.) Shutdown To keep our overlay filesystem free of errors it is a necessary to properly umount the overlay. For that we have to switch back to the read only filesystem. The upstart job that handles power-off and restarts is called shutdown.conf. In it's original version it doesn't do much besides stopping framework and showing a neat image. For us this is not enough. First we need to stop all other upstart jobs that are still running. This also includes the system.conf script and there lies another issue. This job is expected to never stop and when it does system_monitor.conf job kicks in and causes a restart. Again, we don't like that so we patch the job to ignore situations when system.conf stops but shutdown.conf is running. After all the upstart jobs are stopped we have to make sure no other programs are running. The only things we want to keep is ourselves, our parent process and the init. The rest gets sent TERM signal and later KILL if necessary. EDIT: After we kill what we can we try to umount all that we can. When no other pocess is running we again remount the /proc to /root-ro/proc and pivot the root back to /root-ro. After that we have to free the shutdown.conf file (!) so that we can umount the writable root fs. This is done by starting a script 'shutdown_real'. That handles the rest of umounting and moves all the mounts we can't remove to /root-ro. It also does all the other things shutdown.conf was supposed to do. But, simply running 'shudown_real' will not do. That would change one open file for another. We have to move it onto another filesystem first. However most of them were already umounted, and modifying some filesystem is not nice either. Unfortunately at this point the only tmpfs available to our disposal is /root-ro/dev. It's not clean, but I have decided to use it anyway. That is, the 'shutdown_real' is first copied to /root-ro/dev and then started from there in the background. That way the upstart job can terminate. While all the steps are important, I have tried to point out the few most important things with (!) in the text. It took me quite a time to write this post, hopefully it wasn't in vain. Last edited by Nyoxi; 02-26-2012 at 09:56 AM. Reason: fixing typos |
02-25-2012, 11:37 PM | #9 | |
Carpe diem, c'est la vie.
Posts: 6,433
Karma: 10773668
Join Date: Nov 2011
Location: Multiverse 6627A
Device: K1 to PW3
|
Great Job!
Quote:
Bravo! Kudos! Thank you! Teşekkür ederim! Спасибо большое! (and all that stuff). Last edited by geekmaster; 02-25-2012 at 11:46 PM. |
|
02-26-2012, 09:01 AM | #10 |
Connoisseur
Posts: 55
Karma: 124493
Join Date: Jan 2012
Device: Kindle Touch
|
After the much needed sleep I went through my previous post about technical aspects. I have corrected a few misconceptions and added some missing info.
|
02-26-2012, 11:12 AM | #11 |
Carpe diem, c'est la vie.
Posts: 6,433
Karma: 10773668
Join Date: Nov 2011
Location: Multiverse 6627A
Device: K1 to PW3
|
You can do "dd if=/dev/zero of=/dev/mmcblk0p3" before and after updates if you want to. The startup scripts format it when it fails to mount, and copy stuff from /opt. That means any updates that could affect /var/local are just COPIES of data in /opt. Look in the startup scripts. You should save user settings before an update (collections database, locale, timezone, etc) and restore them after an update and /var/local rebuild.
This is just a suggestion of things you can check into (especially the scripts that rebuild /var/local). Of course it would need more research and development before you use it, but it may simplify the OTA update problem. Removing the /var/local partition (after backing up user data such as his collections database, locale and timezone settings) to force a rebuild is perhaps more useful for the "bricked kindle recovery" methods I have been developing, but I though it might be worth studying for this application as well. EDIT: If Post #12 should be before this post as it says, then the text in Post #12 can be moved to the end of Post #10. That would get this thread back in sync, and I can remove this message. Also, you can read (and search) the IRC archive if that may help you remember some missing details: http://kindle-synchrone.dotcloud.com/ Last edited by geekmaster; 02-29-2012 at 07:54 AM. Reason: add thread synchronization proposal |
02-29-2012, 07:09 AM | #12 |
Connoisseur
Posts: 55
Karma: 124493
Join Date: Jan 2012
Device: Kindle Touch
|
The previous post (#11) is somewhat out of sync. It was supposed to follow this post (which however nobody posted yet).
On IRC we were discussing with nueva the possibility of using OTA updates with the overlay. The issue here is that the OTA update can patch files on root filesystem, files in /var/local, kernel and possibly also uboot. This could result in only part of your system being updated after you remove the overlay. The problem seems to be in whatever scenario you use.
Right now we seem to have one only option: disable the overlay, apply update, create new clean overlay. At some later point we can make use of the reconstruction of /var/local ... but I can't remember what was the use. I realy had an idea here that I can't remember right now (I should have written all this sooner). |
02-29-2012, 07:13 AM | #13 | |
Connoisseur
Posts: 55
Karma: 124493
Join Date: Jan 2012
Device: Kindle Touch
|
knc1 suggested here using different filesystem for the overlay instead of mini_fo: https://www.mobileread.com/forums/sho...4&postcount=60
Althoug kindle kernel is older as he says we still could be able to use older version of auFS. From the auFS site Quote:
I have no objection to using something else. I have never worked with overlay filesystem and I made use of mini_fo only because nueva suggested it and because OpenWRT guys seem to be using it happily. The problem with mini_fo however is that it hasn't been updated by the author for last 5 years and it only works because OpenWRT has buch of it's own patches to support newer kernels. So if you have any other ideas feel free to share. Last edited by Nyoxi; 02-29-2012 at 07:14 AM. Reason: kernel version |
|
02-29-2012, 08:56 AM | #14 |
but forgot what it's like
Posts: 741
Karma: 2345678
Join Date: Dec 2011
Location: north (by northwest)
Device: Kindle Touch
|
I've installed overlay and using it for some days without any errors or fails. Thanks!
|
02-29-2012, 09:56 AM | #15 | |
Going Viral
Posts: 17,212
Karma: 18210809
Join Date: Feb 2012
Location: Central Texas
Device: No K1, PW2, KV, KOA
|
Quote:
For a pair of tutorials on using auFS, try: http://drpbox.knetconnect.com/aufs/ They are each "single file" html pages, read on-line or wget your own copy. The "part 3" tutorial is still only in draft form, not yet posted. It shows how to modify the layers while the auFS filesystem stack is in use (hot). Last edited by knc1; 02-29-2012 at 09:58 AM. |
|
Tags |
mini_fo, overlay, root filesystem |
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Creating the Apple Read Aloud Media Overlay | sgtgrom | ePub | 1 | 08-08-2011 01:53 AM |
Accessories Kindle 3 number-key overlay? | tovare | Amazon Kindle | 25 | 02-21-2011 05:30 PM |
To Root, or not to Root... that is the question | t3l01v | Barnes & Noble NOOK | 8 | 01-24-2011 06:54 PM |
Development Alternate root method / "1-click root" | Oneiros | enTourage Archive | 0 | 09-06-2010 02:04 PM |
DR800 root filesystem contents | Mr. X | iRex | 2 | 03-05-2010 07:31 AM |