getkernels - "get kernel image files" script

[geekmaster]

UPDATE: 1.4 adds the 64-byte flash header length to the file length before rounding to the next 1024-byte block size. Now we need to remake affected kernel images in pastebin (such as 4.0.1).

getkernels

Use this script to make backup copies of kernel images from an mmcblk0.img backup file, or directly from the mmc device. You can flash these images with fastboot.

usage:

./getkernels mmcblk0.img : get kernels from file
./getkernels /dev/mmcblk0 : get kernels from mmc device

output:

main_kernel.img and diags_kernel.img

supported devices:

DX/DXG/K3/K4/Touch. K3 and earlier do not have diags_kernel.

You can see the iterative design process, from linux "one-liner" scripts, through the current version, by clicking the "Show" buttons and comparing the code below.

getkernels v1.4

Spoiler:

getkernels v1.4

PHP Code:

#!/bin/sh
#===================================================================================
# getkernels v1.4 - get kernel images from kindle mmc device or backup image file
# Usage:
#   getkernels /dev/mmcblk0 : get linux kernel(s) from mmc
#   getkernels mmcblk0.img : get linux kernel(s) from file
# Copyright (C) 2012 by geekmaster
# Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php
#-----------------------------------------------------------------------------------
IN="$1" # Input Name
[[ "$IN" == "" ]]&& head -n6 $0|tail -n4 && exit # if no params, show usage

BS=1024 # Block Size
S1=$((0x41000/BS)) # Start 1 (main, block number)
S2=$((0xe41000/BS)) # Start 2 (diags, block number)
DN=/dev/null
MK=main_kernel.img; DK=diags_kernel.img                        
EM="No linux kernels found in $IN" # Error Main
ED="No diags kernel found in $IN" # Error Diags
echo "Searching for linux kernels in $IN ..."                        
HD="$(dd if=$IN bs=$BS skip=$S1 count=1 2>$DN|hexdump -C|grep -B2 Linux-|cut -b-60)"
[[ "$HD" == "" ]]&& echo $EM && exit                                                 
set $(echo $HD); [[ $# -lt 16 ]]&& echo $EM && exit
shift 13; BO=$(((64+0x$1$2$3$4+$BS-1)/$BS))
dd if=$IN of=$MK bs=$BS skip=$S1 count=$BO && echo "~> $MK"
HD="$(dd if=$IN bs=$BS skip=$S2 count=1 2>$DN|hexdump -C|grep -B2 Linux-|cut -b-60)" 
[[ "$HD" == "" ]]&& echo $ED && exit                                                 
set $(echo $HD); [[ $# -lt 16 ]]&& echo $ED && exit
shift 13; BO=$(((64+0x$1$2$3$4+$BS-1)/$BS))
dd if=$IN of=$DK bs=$BS skip=$S2 count=$BO && echo "~> $DK" 


This version 1.4 adds the length of the flash header (64 bytes) before rounding up to the next 1024 bytes, so that the firmware images will never be truncated.

getkernels v1.3

Spoiler:

getkernels v1.3

PHP Code:

#!/bin/sh
#===================================================================================
# getkernels v1.3 - get kernel images from kindle mmc device or backup image file
# Usage:
#   getkernels /dev/mmcblk0 : get linux kernel(s) from mmc
#   getkernels mmcblk0.img : get linux kernel(s) from file
# Copyright (C) 2012 by geekmaster
# Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php
#-----------------------------------------------------------------------------------
IN="$1"; [[ "$IN" == "" ]]&& head -n6 $0|tail -n4 && exit
BS=1024; BI=$((32*1024*1024/BS)); S1=$((0x41000/BS)); S2=$((0xe41000/BS)) 
DN=/dev/null; KF=_kernel.img; MK=main$KF; DK=diags$KF; R=1                         
EM="No linux kernels found in $IN"; ED="No diags kernel found in $IN"
echo "Searching for linux kernels in $IN ..."                        
HD="$(dd if=$IN bs=$BS skip=$S1 count=$R 2>$DN|hexdump -C|grep -B2 Linux-|cut -b-60)"
[[ "$HD" == "" ]]&& echo $EM && exit                                                 
set $(echo $HD); [[ $# -lt 16 ]]&& echo $EM && exit
shift 13; BO=$(((0x$1$2$3$4+$BS-1)/$BS))
dd if=$IN of=$MK bs=$BS skip=$S1 count=$BO && echo "~> $MK"
HD="$(dd if=$IN bs=$BS skip=$S2 count=$R 2>$DN|hexdump -C|grep -B2 Linux-|cut -b-60)" 
[[ "$HD" == "" ]]&& echo $ED && exit                                                 
set $(echo $HD); [[ $# -lt 16 ]]&& echo $ED && exit
shift 13; BO=$(((0x$1$2$3$4+$BS-1)/$BS))
dd if=$IN of=$DK bs=$BS skip=$S2 count=$BO && echo "~> $DK" 


This version 1.3 fixed a "start address" glitch that had crept into 1.2, but can sometimes return a kernel image that could be up to 64 bytes too short because the flash header length does not include the 64-byte flash header. This version contains some left-over code from version 1.2, which actually searched the entire mmc for the start of the linux kernels (in case they change in the future). Here we reverted to fixed known start addresses, to fix a bug in 1.2.

getkernels v1.1

Spoiler:

getkernels v1.1

PHP Code:

#!/bin/sh
#===================================================================================
# getkernels v1.1 - get kernel images from mmcblk0.img or /dev/mmcblk0
# Copyright (C) 2012 by geekmaster
# Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php
#-----------------------------------------------------------------------------------
[[ "$1" == "" ]]&&echo "Usage: getkernels mmbclk0.img, or getkernels /dev/mmcblk0"&&exit
IN="$1"; BS=1024; BLKSIN=$((32*1024*1024/BS)); DN=/dev/null
echo "Creating main_kernel.img and diags_kernel.img files..."
HD="$(dd if=$IN bs=$BS count=$BLKSIN 2>$DN|hexdump -C|grep -B2 Linux-|cut -b-60)"
set $(echo $HD); SKIP1=$((0x$1/$BS)); shift 13; BLKSOUT1=$(((0x$1$2$3$4+$BS-1)/$BS))
shift 39; SKIP2=$((0x$1/$BS)); shift 13; BLKSOUT2=$(((0x$1$2$3$4+$BS-1)/$BS))
dd if=$IN of=main_kernel.img bs=$BS skip=$SKIP1 count=$BLKSOUT1
dd if=$IN of=diags_kernel.img bs=$BS skip=$SKIP2 count=$BLKSOUT2 


Now that both commands are contained in a script file, they can share input data (see $HD above), saving many seconds of execution time. Also added "user-friendly" feedback message.

getkernels v1.0

Spoiler:

getkernels v1.0

PHP Code:

#!/bin/sh
#===================================================================================
# getkernels v1.0 - get kernel images from mmcblk0.img or /dev/mmcblk0
# Copyright (C) 2012 by geekmaster
# Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php
#-----------------------------------------------------------------------------------
[[ "$1" == "" ]]&&echo "Usage: getkernels mmbclk0.img, or getkernels /dev/mmcblk0"&&exit
IN=$1; BS=1024; BLKSIN=$((32*1024*1024/BS)); DN=/dev/null
set $(dd if=$IN bs=$BS count=$BLKSIN 2>$DN|hexdump -C|grep -B2 Linux-|head -n1)
SKIP=$((0x$1/$BS));shift 13;BLKSOUT=$(((0x$1$2$3$4+$BS-1)/$BS))
dd if=$IN of=main_kernel.img bs=$BS skip=$SKIP count=$BLKSOUT
set $(dd if=$IN bs=$BS count=$BLKSIN 2>$DN|hexdump -C|grep -B2 Linux-|tail -n3|head -n1)
SKIP=$((0x$1/$BS));shift 13;BLKSOUT=$(((0x$1$2$3$4+$BS-1)/$BS))
dd if=$IN of=diags_kernel.img bs=$BS skip=$SKIP count=$BLKSOUT 


After copying my "linux one-liner" commands (see v0.1 below) into a script file, I added the "shebang" so it can be executed as a standalone command. I converted the vars to UPPERCASE so that they would stand out in my condensed code. This "first release" version contains both complete original commands with no optimization.

getkernels v0.1

Spoiler:

getmain v0.1

PHP Code:

set $(hexdump -C mmcblk0.img|grep -B2 Linux-|head -n1);bs=1024;skip=$((0x$1/$bs));shift 13;count=$(((0x$1$2$3$4+$bs-1)/$bs));echo "skip=$skip count=$count";dd if=mmcblk0.img of=kernel.img bs=$bs skip=$skip count=$count 


getdiags v0.1

PHP Code:

set $(hexdump -C mmcblk0.img|grep -B2 Linux-|tail -n3|head -n1);bs=1024;skip=$((0x$1/$bs));shift 13;count=$(((0x$1$2$3$4+$bs-1)/$bs));echo "skip=$skip count=$count";dd if=mmcblk0.img of=diags_kernel.img bs=$bs skip=$skip count=$count 


These "linux one-liners" show how I usually create and use my scripts the first time. I reuse them from the shell command history with the up-arrow key, and use the back-arrow key to edit them. Later after tested and working, I "copy/paste" them to a file with "cat >> file.sh", then split the lines to fit the window.

Linux one-liners such as these are "fair use" ("few paragraphs" clause), and "don't need no stinkin' license". And just how would you squeeze a license into one line of code anyway?

Enjoy and learn!

(Only tested on Linux host PC, K3, K4 and Touch at this time, but should work on all of the eink kindles.)

[dave2008]

Thanks for the handy script

[geekmaster]

Updated first post to include new "speed-enhanced" version 1.1, and old version 0.1 (to show how I create these things). Also added MIT license. Research shows that License by URL is acceptable, and in some cases preferred (unless you want specific license version lock-in). This makes these scripts a LOT smaller than when I embedded license text in previous scripts (which I may go back and replace).

[NiLuJe]

FWIW, I'm also using the 'License by URL' trick in the credits file of the fonts/ss hacks.

[geekmaster]

getkernels v1.2 now supports K3 (and earlier) kindles, and has significant speed and error-handling enhancements (see first post in this thread).

[geekmaster]

The code and scripts that I post in this forum are formatted to fit an entire "concept" into the mobileread "code" window, without needing to scroll that window.

I would like some feedback on my old-school "condensed coding style". For me, the condensed code is actually MORE readable. Of course, if you get paid for "lines of code" instead of "getting the job done", you might want to stick with the popular white-space laden "fluffy" coding style.

[seaniko7]

@geekmaster, you might consider including this code in your script. It works as expected :

#!/bin/bash
DEBUG=
TEMP_DIR=/tmp
KERNEL_FILE=kernel
KERNEL_GZIP_FILE=kernel.gz
INITRAMFS_FILE=initramfs.cpio
INITRAMFS_DIR=initramfs_root

# DO NOT MODIFY BELOW THIS LINE
[ -z $1 ] && exit 1 || zImage=$1
[ ! -e $1 ] && exit 1

#GET CURRENT DIR
CURRENT_DIR=`pwd`

function pre_clean()
{
    [ -z $DEBUG ] || echo "-D- Function: pre_clean()"
    [ -e $INITRAMFS_FILE ] && ( [ -z $DEBUG ] || echo "-D- Deleting $INITRAMFS_FILE"; rm -f $INITRAMFS_FILE )
    [ -e $INITRAMFS_DIR ] && ( [ -z $DEBUG ] || echo "-D- Deleting $INITRAMFS_DIR"; rm -rf $INITRAMFS_DIR )
    [ -z $DEBUG ] || echo
}

function ungzip_kernel()
{
    #========================================================
    # find start of gziped kernel object in the zImage file:
    #========================================================
    [ -z $DEBUG ] || echo "-D- Function: ungzip_kernel()"

    pos=`grep -P -a -b -m 1 --only-matching '\x1F\x8B\x08' $zImage | cut -f 1 -d :`
    echo "-I- Extracting gzip'd kernel image from file: $zImage (start = $pos)"

    if [ ! -z $pos ]; then
        dd if=$zImage of=$TEMP_DIR/$KERNEL_GZIP_FILE bs=1 skip=$pos 2>/dev/null >/dev/null
        gunzip -qf $TEMP_DIR/$KERNEL_GZIP_FILE
    else
        echo "-E- Compressed kernel image not found"; exit 1
    fi
    [ -z $DEBUG ] || echo
}

function search_cpio()
{
    #========================================================
    # Determine cpio compression type:
    #========================================================
    [ -z $DEBUG ] || echo "-D- Function: search_cpio()"

    for x in gzip bzip lzma none; do
        case $x in
            bzip)
                csig='\x{31}\x{41}\x{59}\x{26}\x{53}\x{59}'
                ucmd='bunzip2 -q'
                fext='.bz2'
                ;;

            gzip)
                csig='\x1F\x8B\x08'
                ucmd='gunzip -q'
                fext='.gz'
                ;;

            lzma)
                csig='\x{5D}\x{00}\x..\x{FF}\x{FF}\x{FF}\x{FF}\x{FF}\x{FF}'
                ucmd='unlzma -q'
                fext='.lzma'
                ;;

            none)
                csig='070701'
                ucmd=
                fext=
                ;;
        esac

        #========================================================================
        # Search for compressed cpio archive
        #========================================================================
        search=`grep -P -a -b -m 1 --only-matching $csig $TEMP_DIR/$KERNEL_FILE | cut -f 1 -d : | head -1`
        pos=${search:-0}

        if [ ${pos} -gt 0 ]; then
            if [ ${pos} -le ${cpio_compressed_start:-0} ] || [ -z $cpio_compressed_start ];then
                cpio_compressed_start=$pos

                compression_name=$x
                compression_signature=$csig
                uncompress_cmd=$ucmd
                file_ext=$fext

                [ -z $DEBUG ] || echo "-D- Checking for compression type: $compression_name | signature: $compression_signature | in file: $TEMP_DIR/$KERNEL_FILE | offset = $pos"
            fi
        fi
    done 

    [ $compression_name = "bzip" ] && cpio_compressed_start=$((cpio_compressed_start - 4))
    echo "-I- CPIO compression type detected = $compression_name | offset = $cpio_compressed_start"

    [ -z $DEBUG ] || echo
}

function extract_cpio()
{
    [ -z $DEBUG ] || echo "-D- Function: extract_cpio()"

    if [ ! $compression_name = "none" ]; then
        echo "-I- Extracting $compression_name'd compressed CPIO image from kernel image (offset = $cpio_compressed_start)"
        [ -z $DEBUG ] || echo "-D- dd if=$TEMP_DIR/$KERNEL_FILE of=$TEMP_DIR/$INITRAMFS_FILE$file_ext bs=1 skip=$cpio_compressed_start; $uncompress_cmd $TEMP_DIR/$INITRAMFS_FILE$file_ext"
        dd if=$TEMP_DIR/$KERNEL_FILE of=$TEMP_DIR/$INITRAMFS_FILE$file_ext bs=1 skip=$cpio_compressed_start 2>/dev/null >/dev/null
        $uncompress_cmd -q $TEMP_DIR/$INITRAMFS_FILE$file_ext

    else
        echo "-I- Extracting non-compressed CPIO image from kernel image (offset = $cpio_compressed_start)"
        [ -z $DEBUG ] || echo "-D- dd if=$TEMP_DIR/$KERNEL_FILE of=$TEMP_DIR/${INITRAMFS_FILE}${file_ext} bs=1 skip=$cpio_compressed_start 2>/dev/null >/dev/null"
        dd if=$TEMP_DIR/$KERNEL_FILE of=$TEMP_DIR/${INITRAMFS_FILE}${file_ext} bs=1 skip=$cpio_compressed_start 2>/dev/null >/dev/null
    fi

    [ -z $DEBUG ] || echo
}

function uncompress_cpio()
{
    #==========================================================================
    # find start and end of the "cpio" initramfs image inside the kernel object:
    # ASCII cpio header starts with '070701'
    # The end of the cpio archive is marked with an empty file named TRAILER!!!
    #==========================================================================
    [ -z $DEBUG ] || echo "-D- Function: uncompress_cpio()"

    if [ ! $compress_type = "none" ]; then
        start=`grep -a -b -m 1 --only-matching '070701' $TEMP_DIR/$INITRAMFS_FILE | head -1 | cut -f 1 -d :`
        end=`grep -a -b -m 1 --only-matching 'TRAILER!!!' $TEMP_DIR/$INITRAMFS_FILE | head -1 | cut -f 1 -d :`

        if [ ! -z $start ] || [ ! -z $end ]; then
            #11 bytes = length of TRAILER!!! zero terminated string, fixes premature end of file warning in CPIO
            end=$((end + 14))
            [ -z $DEBUG ] || echo "-D- Kernel start = $start"
            [ -z $DEBUG ] || echo "-D- Kernel end = $end"
            count=$((end - start))

            if (($count < 0)); then
                echo "-E- Couldn't match start/end of the initramfs image."
                exit 1
            fi

            echo "-I- Extracting initramfs image from file: $inputfile (start = $start, end = $end)"
            dd if=$TEMP_DIR/$INITRAMFS_FILE of=$CURRENT_DIR/$INITRAMFS_FILE bs=1 skip=$start count=$count 2>/dev/null >/dev/null
            INITRAMFS_FILE=$CURRENT_DIR/$INITRAMFS_FILE
        else
        
            echo "-E- No CPIO image found in $inputfile."
        fi
    else
        echo "-I- CPIO already uncompressed."
    fi
    [ -z $DEBUG ] || echo
}

function expand_cpio_archive()
{
    [ -z $DEBUG ] || echo "-D- Function: expand_cpio_archive()"
    echo "-I- Expanding CPIO archive: $INITRAMFS_FILE to $INITRAMFS_DIR."

    if [ -e $TEMP_DIR/$INITRAMFS_FILE ]; then
        mkdir $INITRAMFS_DIR
        cd $INITRAMFS_DIR
        cpio --quiet -i --make-directories --preserve-modification-time --no-absolute-filenames -F $TEMP_DIR/$INITRAMFS_FILE 2>/dev/null
    fi
    [ -z $DEBUG ] || echo
}

function clean_up()
{
    [ -z $DEBUG ] || echo "-D- Deleting $TEMP_DIR/$KERNEL_FILE"; rm -f $TEMP_DIR/$KERNEL_FILE
    [ -z $DEBUG ] || echo "-D- Deleting $TEMP_DIR/$INITRAMFS_FILE"; rm -f $TEMP_DIR/$INITRAMFS_FILE
    [ -z $DEBUG ] || echo "-D- Deleting $TEMP_DIR/$INITRAMFS_FILE$file_ext"; rm -f $TEMP_DIR/$INITRAMFS_FILE$file_ext
}





reset
pre_clean
ungzip_kernel
search_cpio
extract_cpio
uncompress_cpio
expand_cpio_archive
clean_up

Oh, to make things clear. I'm not the author of this script.

[geekmaster]

Originally Posted by seaniko7

@geekmaster, you might consider including this code in your script. It works as expected :
...
Oh, to make things clear. I'm not the author of this script.

I see that this handles compressed kernels and cpio. Nice.

Do any of the kindles use gzipped kernels? Of course, if not now, they may later. It is nice to be prepared for the future.

Of course, this script is loaded with white-space "pollution" that needs a lot of vertical scrolling just to view it, and those long variable names require plenty of horizontal scrolling too, but after it is on the search path, who cares really? And a delete key can fix a lot of that.

Where did you find this little gem?

[seaniko7]

I found it at xda forums, while I was preparing custom recovery kernel for 3.3.

[352478140]

in "simple kindle touch (and k4nt) debricking method",there was only main_kernel of k4 nt there.so someone plz share the diags_kernel one.

[geekmaster]

Originally Posted by 352478140

in "simple kindle touch (and k4nt) debricking method",there was only main_kernel of k4 nt there.so someone plz share the diags_kernel one.

New Pastebin link to urls with k4 diags kernel now in first "simple debricking" post.

[geekmaster]

The getkernels script has been updated to version 1.4 to fix an intermittent bug that could cause up to 64 bytes to be truncated from the end of a kernel image in some cases (such as the 4.0.1 kernels).

It now adds 64 bytes (the length of the flash header) to the image length extracted from the flash header, before rounding up to the next 1024-byte block size.

Some comments were also added to the script to make it less "obfuscated".

EDIT: Sorry about the delay getting this updated. I thought I updated it a long time ago. That can happen when you have too many projects "in progress".

[NiLuJe]

@geekmaster: Thanks! I can provide 5.1.0 diags/main kernels if the current ones are borked.

[geekmaster]

Originally Posted by NiLuJe

@geekmaster: Thanks! I can provide 5.1.0 diags/main kernels if the current ones are borked.

They should only be "borked" if their length was a multiple of 1024 plus 1 to 64 extra bytes, and that extra was not just padding. I think most kernels are okay, but they can be checked by extracting with the new getkernels and see if the new ones are larger than the old ones. If not larger, the old ones are fine.

[kasumi]

Sorry for my ignorance, but I don't know how to use this script.

I placed getkernels-1.0.tar.gz in the kindle root and tried the following in ssh:

./getkernels /dev/mmcblk0
cd /mnt/us && ./getkernels /dev/mmcblk0 && cd -
getkernels /dev/mmcblk0

they all reply
-sh: getkernels: not found

Also, the outputs main_kernel.img and diags_kernel.img actually corresponds to mmcblk0p1.img and mmcblk0p2.img right?

Thanks!