Notes on Kindle NFS Boot

[katadelos]

I've been meaning to write this up properly for a while now but never seem to find the time. Until then, here's some notes and example scripts to assist with the process of booting a rootfs over NFS via USBNet. You definitely want to have a device with a soldered/permanently attached serial connection before attempting this.

Prerequisites:
You need to build a kernel using a custom initramfs entrypoint script to mount the NFS share early in the boot process:

Spoiler:

#!/bin/busybox ash
/bin/busybox --install -s

USB_PATH="/sys/devices/system/wario_charger/wario_charger0/connected"

kecho() {
  echo $1 > /dev/kmsg
}

setup_mounts() {
  mount -n -t sysfs none /sys
  mount -t proc proc /proc
}

setup_usb() {
  echo "Loading kernel modules"
  insmod /lib/modules/fsl_otg_arc.ko
  insmod /lib/modules/arcotg_udc.ko
  insmod /lib/modules/ehci-hcd.ko 
}

setup_mounts

setup_usb
insmod /lib/modules/g_ether.ko host_addr=00:11:22:33:44:55
ifconfig usb0 192.168.15.244

mkdir /mnt/rootfs
BOARD_ID=$(cat /proc/board_id)
ROOTFS=$(wget -qO - http://192.168.15.201:8000/$BOARD_ID)
PREFIX="/srv"
if [ "$ROOTFS" != "${ROOTFS#$PREFIX}" ]; then
  echo "Attempting to boot over NFS"
  mount -v -t nfs -o port=2049,nolock,proto=tcp 192.168.15.201:$ROOTFS /mnt/rootfs
  exec switch_root -c /dev/console /mnt/rootfs /sbin/init
else
  echo "Attempting to boot from rootfs"
  ifconfig usb0 down
  rmmod  g_ether
  rmmod  ehci-hcd
  rmmod  arcotg_udc
  sleep 2
  mount /dev/mmcblk0p1 /mnt/rootfs
  exec switch_root -c /dev/console /mnt/rootfs /sbin/init
fi

A means of automatically configuring the USBNet interface is also necessary:

Spoiler:

# netplan example for Ubuntu 20.04
network:
  version: 2
  renderer: networkd
  ethernets:
    enx001122334455:
      link-local: []
      dhcp4: false 
      addresses:
        - 192.168.15.201/24

Setting up NFS share:
A "stock" rootfs image won't boot successfully and it's useful to be able to make changes that can be removed later, so we need to set up a fairly complicated mount point:

mkdir -p /data/kindle-rootfs/KT2
mkdir -p /srv/{rootfs,union,bootmod}/KT2-5.12.2.1
kindletool extract update_kindle_5.12.2.1.bin /tmp/update
gzip -d /tmp/update/rootfs.img.gz
mv rootfs.img /data/kindle-rootfs/KT2/5.12.2.1.img

# We use unionfs-fuse as standard unionfs doesn't play nice over NFS
sudo unionfs-fuse -o cow,max_files=32768 \
	-o allow_other,noforget,suid,dev,nonempty \
	/srv/user/KT2-5.12.2.1=RW:\
	/srv/bootmod/KT2-5.12.2.1=RO:\
	/srv/rootfs/KT2-5.12.2.1=RO \
	/srv/union/KT2-5.12.2.1

TL;DR it's a unionfs with 3 layers:

1. user - temporary changes made on device
2. bootmod - read only layer containing files and binaries needed on device
3. rootfs - mount point for "stock" image

Once that's done, we can export the directory structure as an NFS share:

Spoiler:

# in /etc/exports
/srv/union/KT2-5.12.2.1 *(rw,sync,insecure,no_subtree_check,no_root_squash,fsid=0)

These options are probably not great/are insecure but if it's stupid and it works...

We also need a way of letting the device know which share to mount - this is a cut down version of the server that I use for this:

Spoiler:

import json
from http.server import HTTPServer, BaseHTTPRequestHandler

class KindleMetadataServer(HTTPServer):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.load_config_file()
        
    def load_config_file(self):
        with open("config.json") as config:
            self.board_data = json.loads(config.read())

class KindleMetaHandler(BaseHTTPRequestHandler):  
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        if self.path[1:] in self.server.board_data:
            self.wfile.write(self.server.board_data[self.path[1:]].encode())
        else:
            self.wfile.write(b'Nah')


httpd = KindleMetadataServer(('0.0.0.0', 8000), KindleMetaHandler)
httpd.serve_forever()

... and also a config file containing serial number/mount paths:

Spoiler:

# config.json
{
    "0600000000000000": "/srv/union/KT2-x-test",
    "0600000000000000": "/srv/union/KT2-5.14.1-testing",
    "0061000000000000": "/srv/union/KT1-5.3.7.3",
    "0480000000000000": "/srv/union/KV-5.14.1"
}

Bootmods:
As alluded to, a stock rootfs image won't boot by default - upstart scripts will need to be tweaked and the like. Here's what the structure of my KT2 bootmods folder looks like:

Spoiler:

.:
etc  usr

./etc:
dropbear  passwd  upstart

./etc/dropbear:
dropbear_rsa_host_key

./etc/upstart:
dropbear.conf  filesystems_userstore.conf  langpicker.conf  x11vnc.conf

./usr:
bin  sbin

./usr/bin:
x11vnc

./usr/sbin:
dropbear  mntroot

All things combined, this will allow the device to boot fully and bring up an SSH server and x11vnc instance (yes, I've also been sitting on an x11vnc KUAL plugin that I really should release...). A couple of these are stock upstart scripts with minor tweaks, diff the originals from a stock rootfs image if you want to see the differences.

dropbear.conf:

Spoiler:

description "Dropbear SSH server"
author "Peter Molnar <hello @petermolnar.eu>"

start on starting volumd
stop on (stopped volumd or ota-update)

#respawn
#respawn limit 10 5
umask 022

env DROPBEAR_PORT=22
env DROPBEAR_RSAKEY=/etc/dropbear/dropbear_rsa_host_key
env DROPBEAR_RECEIVE_WINDOW=65535

pre-start script
        test -x /usr/sbin/dropbear || { stop; exit 1; }
end script

exec /usr/sbin/dropbear -F -r $DROPBEAR_RSAKEY -p $DROPBEAR_PORT -W $DROPBEAR_RECEIVE_WINDOW

filesystems_userstore.conf

Spoiler:

# Userstore is dependent on display: JTWO-6549

# temp fix for cognac. Userstore is not dependent on display for cognac.
start on started filesystems_var_local # and display_ready
stop on stopping filesystems

export LANG LC_ALL

emits mounted_userstore

pre-start script
  source /etc/upstart/functions
  source /etc/sysconfig/mntus

  /etc/upstart/userstore start

  #Copy data from WAF to Mesquite if required
  /etc/upstart/mesquite_copy

  f_emit mounted_userstore
  f_milestone fs50
end script


post-stop script
  source /etc/upstart/functions

  f_log I filesystems_userstore unmount_userstore "Unmounting UserStore"
  /etc/upstart/userstore stop
end script

langpicker.conf

Spoiler:

# start language picker after blanket has loaded the langpicker module

start on x_setup_ready and blanket_loaded_langpicker
stop on stopping x

emits langpicker_ready
emits langpicker_load_complete

env LOCALE_FILE=/var/local/system/locale
env ASR_TUTORIAL_COMPLETED_FILE=/var/local/system/asr_tutorial_launched
env FIRST_BOOT_FILE=/var/local/system/factory_fresh
env BLACKLIST=/opt/amazon/locale.blacklist

env LIBC_FATAL_STDERR_=1
env STDERR=/tmp/langpicker.err

task

script
    source /etc/upstart/env
    source /etc/upstart/functions

    if [ "$(devcap-get-feature -a frontlight)" -eq "1" ]; then
      lipc-set-prop com.lab126.powerd flStartup 1        
    fi

    

    # see if ASR tutorial launched flag exists and 
    # delete the existing temp locale file already selected.
    if [ -e $ASR_TUTORIAL_COMPLETED_FILE ] && [ -e $LOCALE_FILE ] ; then
        rm -rf $LOCALE_FILE
    fi

    # see if we need to pick a language.
    # if the locale exists, then just exit
    if [ -e $LOCALE_FILE ]; then
        LANG=`awk -F'=' '/LANG/{print $2}' $LOCALE_FILE`
        LC_ALL=`awk -F'=' '/LC_ALL/{print $2}' $LOCALE_FILE`
        f_log I locale read "lang=$LANG,lc_all=$LC_ALL" "Retrieved Language"
        f_emit langpicker_ready LANG_PICKER_SCREEN_SHOWN=0
        exit 0
    fi

    # Did not find first boot file. This means that the device is booting in the factory line 
    # before the device has ever entered shipping mode. The lang picker should not be shown in this case.
    # Just set the locale to en-US.
    if [ ! -e $FIRST_BOOT_FILE ]; then
        # send the event to langpicker module to install the language
	lipc-send-event com.lab126.blanket.langpicker changeLocale -s "en-US"
	# wait for the localeChange event
	lipc-wait-event com.lab126.locale localeChange
	f_emit langpicker_ready LANG_PICKER_SCREEN_SHOWN=0
	exit 0
    fi

    # N.B. - only if we seem to have a valid input device.
    # the last input device in /etc/xorg.conf.* is the pointer, which is what we need.
    # we get "/dev/input/event3" - the extra eval is to strip quotes
    set $(grep /dev/input /etc/xorg.conf)
    INPUT=$(eval echo $(eval echo \$$#))

    if [ -e $INPUT -o -e /mnt/base-us/ENABLE_VNC ]; then
        # Start the pillow
        # TODO What if pillow fails to start ?        
        # setlocale(LC_ALL, "") depends on environment variable LC_ALL.
        export LC_ALL=en_US.UTF-8
        su framework -c "exec pillowd -NoDefaultPillowCase 2>> $STDERR"
        #Wait till pillow is ready before setting lipc property
        lipc-wait-event -s 30 com.lab126.pillow pillowReady
        # send the pick, wait for the change
        f_log I locale pick "" "Picking Language"
        lipc-set-prop com.lab126.pillow applicationWindow '{"name":"pickLocale","clientParams":{"show":true}}'
        f_emit langpicker_load_complete
        
        f_log I locale wait
        lipc-wait-event com.lab126.locale localeChange
        # Kill the pillow here, so that it boots up again with proper language.
        killall -q -s KILL pillowd || true
        f_log I locale set "" "User Picked Language"
    else
        f_log I locale no_input "" "no input device, defaulting"
        echo -e "LANG=en_US.UTF-8\nLC_ALL=en_US.UTF-8" > $LOCALE_FILE
    fi

    LANG=`awk -F'=' '/LANG/{print $2}' $LOCALE_FILE`
    LC_ALL=`awk -F'=' '/LC_ALL/{print $2}' $LOCALE_FILE`
    f_log I locale read "lang=$LANG,lc_all=$LC_ALL" "Set Language"
    f_emit langpicker_ready LANG_PICKER_SCREEN_SHOWN=1
end script

post-stop script
    source /etc/upstart/functions
    libc_scan
end script

x11vnc.conf

Spoiler:

# VNC server for X11 display
                                               
start on started x
stop on stopping x     
                                       
env STDERR=/tmp/x11vnc.err
respawn                     
                            
expect fork                 
exec /usr/bin/x11vnc -cursor arrow --loop --forever 2>> $STDERR

We also need to blank the root password:

Spoiler:

root::0:0:root:/tmp/root:/bin/sh
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:100:sync:/bin:/bin/sync
operator:x:37:37:Operator:/var:/bin/sh
sshd:x:103:99:Operator:/var:/bin/sh
messagebus:x:92:92:messagebus:/tmp:/bin/false
nobody:x:99:99:nobody:/tmp:/bin/sh
default:x:1000:1000:Default non-root user:/dev/null:/bin/sh
framework:x:9000:150:Framework User:/tmp/framework:/bin/sh
guestuser:x:1002:1001:Guest user:/tmp/guestuser:/bin/sh

... and edit /usr/bin/mntroot so that it doesn't break now that we're booting from NFS

Spoiler:

#!/bin/sh
_FUNCTIONS=/etc/rc.d/functions
[ -f ${_FUNCTIONS} ] && . ${_FUNCTIONS}

CURRENT_STATE_RO=0

MOUNT_BIN="/bin/mount"
MOUNT_ARGS_RO="-o remount,ro"
MOUNT_ARGS_RW="-o remount,rw"

MSG_RO="read-only"
MSG_RW="writeable"
MSG_MAKING="Making root filesystem "
MSG_REMOUNT="Done re-mounting root filesystem as "
MSG_FAILREMOUNT="Re-mounting root filesystem failed"
MSG_NTBD="Nothing to be done"

current_state()
{
	if [ -z $(cat /etc/mtab | awk '/192.168.15.201/ { print $4 }' | grep 'ro,') ]; then
	  CURRENT_STATE_RO=0;
	  msg "Currently $MSG_RO"
	else
	  CURRENT_STATE_RO=1;
	fi
}

mount_ro()
{
	msg "$MSG_MAKING$MSG_RO" I
	current_state
	if [ $CURRENT_STATE_RO -eq 0 ]; then 
		$MOUNT_BIN $MOUNT_ARGS_RO /
		if [ $? -eq 0 ]; then
			msg "$MSG_REMOUNT$MSG_RO" D 
		else
			msg "$MSG_FAILREMOUNT" E
		fi
	else
		msg "$MSG_NTBD" D
	fi
}

mount_rw()
{
	msg "$MSG_MAKING$MSG_RW" I
	current_state
	if [ $CURRENT_STATE_RO -eq 1 ]; then 
		$MOUNT_BIN $MOUNT_ARGS_RW /
		if [ $? -eq 0 ]; then
			msg "$MSG_REMOUNT$MSG_RW" D 
		else
			msg "$MSG_FAILREMOUNT" E
		fi	
	else
		msg "$MSG_NTBD" D
	fi
}


case "$1" in

    ro)
		mount_ro
        ;;
	rw)
		mount_rw
		;;
	help|-h|--help)
		msg "Usage: $0 (ro|rw)" I
        ;;
	*)
		current_state
		{ [ $CURRENT_STATE_RO -eq 1 ] && exit 5 ; } || exit 6
       ;;

esac

exit 0