How to update Kindle 2/DX/DXG with YifanLu's firmware from 3.2.1 to 3.4.2

[knc1]

Originally Posted by Kaan88

I can't upgrade to 3.3 because of a different "powerd" file. (checksum error whilte patching) Any idea how I can restore it?

Edit: And I deleted my rootfs backup accidentally. I'm stuck with 3.2.1.
Does anyone have rootfs image for Kindle 2 US? PM me.

Ah, not very likely.
Suspect local error(s) before you suspect updates that have been posted for years by Amazon and used by millions of K3 owners.

I.E: If you can "accidentally" delete something as important as your rootfs backup image, suspect that you also fat fingered something else.

[donB006]

This is probably too late so maybe this will help others.

When upgrading the OS why not make a backup along the way: SEE BELOW FIRST

dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.3-dx.img conv=sync
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.4-dx.img conv=sync
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.4.2-dx.img conv=sync

Somebody else also suggested turning these services off before beginning:

/etc/init.d/powerd stop
/etc/init.d/netwatchd stop

When finished these services should be started again or just reboot.

EDIT: This should be a better way than the above:

dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.3-dx.img bs=4096
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.4-dx.img bs=4096
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.4.2-dx.img bs=4096

[knc1]

Originally Posted by donB006

This is probably too late so maybe this will help others.

When upgrading the OS why not make a backup along the way:

dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.3-dx.img conv=sync
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.4-dx.img conv=sync
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs-3.4.2-dx.img conv=sync

Somebody else also suggested turning these services off before beginning:

/etc/init.d/powerd stop
/etc/init.d/netwatchd stop

When finished these services should be started again or just reboot.

A horrible example.
You are erasing and re-programming every erase block 4096 times for each block written.

The 'conv' option to dd does text file conversions, these are binary files, so 'conv=' is a sure way to trash the image.

Originally Posted by man dd

sync pad every input block with NULs to ibs-size; when used with block or unblock, pad with spaces rather than NULs

And since the ibs default size is one byte ...

Translation:
Remove the 'conv=sync' option -
Add the bs=4096 option

[Kaan88]

Did I say anything other than an error on my end? You come of as quite a prick. I didn't put any blame on anyone but myself.

I do have a backup, I made one from sda. I thought it included root partition too. That was my mistake.

Thankfully, there are helpful and understanding people too on this site.

Edit: And just so you know, powerd file really was different. I never did anything to it. I managed to restore it with someone else's backup.

[knc1]

Originally Posted by Kaan88

- - - -
You come of[f] as quite a prick.
- - - -

It took decades of practice - about 7 of them.

[donB006]

Originally Posted by knc1

A horrible example.
You are erasing and re-programming every erase block 4096 times for each block written.

The 'conv' option to dd does text file conversions, these are binary files, so 'conv=' is a sure way to trash the image.

And since the ibs default size is one byte ...

Translation:
Remove the 'conv=sync' option -
Add the bs=4096 option

Thanks for the observation knc1. The proper use of dd is very important and my errors are inexcusable in this case as dd is very unforgiving. I really should know better.

In reality, I was just trying to flush the write buffer in case that was left unfinished before copying the rootfs.img across the USB. That is probably unneccesary for the kindle dx because that is not a hard disk. If I knew more about the emmc I would probably want to match the giant block erase size except I don't really know what that is. I have had no problem before using the defaults even though probably not the most efficient.

According to the dd man page:

       ibs=BYTES
              read up to BYTES bytes at a time (default: 512)

       obs=BYTES
              write BYTES bytes at a time (default: 512)

    sync   pad every input block with NULs to ibs-size; when used with block or unblock, pad with spaces rather than NULs

I didn't really study the man page before being so sloppy but doesn't the sync just fill the very last block out to 512 bytes if that is left unfilled?

What I really meant was this:
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs.img; sync

I have frequently used this without problems:
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs.img

I have no problem recommending this:
dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs.img bs=4096

Hey, Kaan88, glad you have things working so far. Let us know how things turn out. And give knc1 a little space. We all know his way after so many posts and is still good for a smile most of the time. Of course, we probably aren't so frustrated with the kindle at the moment unlike too many here and might fail to understand sometimes. Have fun with the project.

[knc1]

Originally Posted by donB006

- - - -
I didn't really study the man page before being so sloppy but doesn't the sync just fill the very last block out to 512 bytes if that is left unfilled?
- - - -

"Up to" - it should be making a system call to determine the optimum read size for ibs and the optimum write size for obs.
**should**
But on the Kindle's eMMC it is using a block size of 1 byte.
Right or wrong, that is what history shows that it uses.
Also, keep in mind this is the 'dd' provided by (a usually very old) Busybox.

dd if= of= bs=4096 works (the erase block size of both the 2Gbyte and 4Gbyte eMMC devices seen so far is 4096 bytes).

Since 512 is an integer divisor of 4096 (8), there will be nothing to pad (and the eMMC driver would fake it, even if it wasn't PS: it isn't, its something like 526 * 8 IIRC ).

= = = =

Just to add confusion to unusual behaviors observed:

core2quad ~ $ ssh kpw

Welcome to Kindle!

#################################################
#  N O T I C E  *  N O T I C E  *  N O T I C E  # 
#################################################
Rootfs is mounted read-only. Invoke mntroot rw to
switch back to a writable rootfs.
#################################################
[root@kindle root]# stat /dev/mmcblk0p1
  File: /dev/mmcblk0p1
  Size: 0         	Blocks: 0          IO Block: 4096   block special file
Device: ch/12d	Inode: 491         Links: 1     Device type: b3,1
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 1971-04-06 21:23:09.000000000
Modify: 1971-04-06 21:23:09.000000000
Change: 1971-04-06 21:23:09.000000000

[root@kindle root]# stat /mnt/base-us
  File: /mnt/base-us
  Size: 8192      	Blocks: 16         IO Block: 8192   directory
Device: 700h/1792d	Inode: 1           Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-12-24 19:59:12.000000000
Modify: 2012-12-24 19:59:12.000000000
Change: 2012-12-24 19:59:12.000000000

[root@kindle root]# stat /mnt/us
  File: /mnt/us
  Size: 8192      	Blocks: 16         IO Block: 4096   directory
Device: fh/15d	Inode: 1           Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-12-24 19:59:12.000000000
Modify: 2012-12-24 19:59:12.000000000
Change: 2012-12-24 19:59:12.000000000

All I can say is:
Duh
No wonder dd throws up its hands and transfers 1 byte at a time.

[donB006]

Originally Posted by knc1

But on the Kindle's eMMC it is using a block size of 1 byte.
Right or wrong, that is what history shows that it uses.
Also, keep in mind this is the 'dd' provided by (a usually very old) Busybox.

That is very good information and I am very surprised.

I have been reading many books with my DXG for the past few years instead of working on the OS so can easily plead guilty by reason of ignorance. I can certainly appreciate the level of expertise and experience needed to know so much about all of these kindles.

Thanks knc1.

[knc1]

The last time (a long time ago), twobob and I worked on his (wife's) DXG -
I don't think we thought to check (the DXG uses multiple storage devices) but just used 4096 in our scripting.

[donB006]

Originally Posted by knc1

The last time (a long time ago), twobob and I worked on his (wife's) DXG -
I don't think we thought to check (the DXG uses multiple storage devices) but just used 4096 in our scripting.

Actually, I have only made my own backups and have never used those backups to restore the partition. I just use the image to keep as a record and use to explore the past setups. I typically use:
mount -t ext3 -o loop rootfs.img /mnt/kindle-image

The requirements for using the rootfs.img to flash the kindle might be totally different for all I know and might cause a brick if not done properly so I think this is an important topic.

I have been looking at Yifan Lu's original kindleupdater.zip and the way he made the backup of the rootfs and found he used bs=1024 for some reason. I wonder if that was selected as a divisor of the partition size. I worry that somebody might try to restore the partition and possibly overwrite data by accident.

The problem is even worse when writing the two kernel images to the mtdblocks. I expect those could easily overwrite an adjacent block if not done properly. There Yifan uses bs=131072 to calculate a block count. Fortunately, few people should be changing their DX kernels without Yifan's techniques.

When writing the rootfs to the mmcblk0p1 with the wrong size image, the excess might just result in an 'Out of Disk Space' error with the unwritten data and just being nulls anyway. That would be a best case so I hope anybody trying to restore from the rootfs images would check out these details first. I am now unwilling to experiment with my trusty DXG.

[knc1]

The Linux flash storage stack will not allow writes outside of the partition area addressed.
That would be just like allowing excessive writes to one partition to overwrite the next partition. The emulation of a device handles that.

Writing to the raw device, outside of a partition (outside of an emulated device) is left to the programmer (person, not device).
Things like the primary and secondary kernels in the case of a series 2 device.

There is no magic in these apparent magic numbers.
Although some factors of purely historic interest creeps in.

Kindles that use a single eMMC device (K3 and more recent). . . .
(I think, but have not confirmed, that this same logic was used in the series 2 design - K2/DX/DXG).

The raw storage uses a "DOS disk label" - which implies an MBR followed by a partition table.
The layout format used is the "hard disk" format for a DOS disk label.

The label occupies the first track in CISC machines (Intel) and the first cylinder in RISC machines (the 64bit Alpha).
Tracks where 16 sectors.
Cylinders where 2 tracks (hard disks only had two platter sides externally accessible for storage).

ARM is RISC.
The RISC code for the MBR will not (would not) fit in one track of 16 sectors, which is why the allowance for the first cylinder (32 sectors).

In flash storage, multiples of an erase block are ok, fractions of an erase block are a big no-no.
Erase blocks are 4,096 bytes. I.E: the sector size of flash is 4096 (8*512).
16 sectors per track, 2 tracks per cylinder ==
(8*512)*16*2 = 131072 bytes (a hard disk layout mapped to a 4096 byte sector) == space occupied by the disk label.

That first disk label defines four primary partitions.
Each of the first three contain **only** a file system (ext3 in our case).
The fourth primary partition contains another "DOS disk label" (this is legal per the Microsoft specifications - any partition may have its own (additional) disk label - nested to any depth)

Which means you will find another MBR area prior to the partition table.
A handy place to put custom code, since it is exported along with the USB storage area and accessible with Linux (and Windows, with custom code).
We don't (yet) use that area (neither does Amazon).

This partition table defines only a single, 'hard disk layout' partition. A.K.A: USB mass storage.

It could define more than one and some people here have posted about doing just that - - the Kindle firmware survives the experience.
Or - at least it used to, I don't think anyone has checked recently.

The file system on this single partition is FAT32.
Which has an internal structure defined in terms of sectors (not bytes).

To keep all of the internal structure erase block aligned, that FAT32 area is formated as if it had a sector size of 8,192 bytes (2 erase blocks).

See? Nothing magic about the numbers used.

= = = =

Note:
People who get the bright idea to re-format their Kindle's USB mass storage area using a disk format program that does not allow specifing a non-default (512 byte) sector size (translation: Microsoft CICS disk formatters) are doing themselves a great dis-service.
It'll work, but not very well, since the erase block boundraries are not observed.

Much better to just invalidate the USB mass storage area (trash its MBR and partition table area) and re-boot the Kindle.
The Kindle's start-up routines will discover the trashed USB mass storage and re-format it to fit the flash structure.
You may have to look at the boy under the tree an unusually long time, but it will do the job.

[Kaan88]

Do you guys have 3.2.1 K3 fb.modes file? I can't properly patch fb.modes for my kindle 2. KF8 still works but it would be better with a proper file.

Thanks for this great guide by the way.


Edit: Solved through PM

[donB006]

Originally Posted by knc1

In flash storage, multiples of an erase block are ok, fractions of an erase block are a big no-no.
Erase blocks are 4,096 bytes. I.E: the sector size of flash is 4096 (8*512).
16 sectors per track, 2 tracks per cylinder ==
(8*512)*16*2 = 131072 bytes (a hard disk layout mapped to a 4096 byte sector) == space occupied by the disk label.

That first disk label defines four primary partitions.
Each of the first three contain **only** a file system (ext3 in our case).
The fourth primary partition contains another "DOS disk label" (this is legal per the Microsoft specifications - any partition may have its own (additional) disk label - nested to any depth)

That was some very interesting information. I just looked in my DXG to double check.

mmcblk0p1 = 12801 x 32768 = 419463168 bytes

For comparison I get these sizes:

419454976 when dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs.img bs=512
419454976 when dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs.img bs=1024
419454976 when dd if=/dev/mmcblk0p1 of=/mnt/us/rootfs.img bs=4096

I found a KindleDXGRepair.rar on the internet containing the same rootfs.img size so this seems to be confirmation of the technique. Unfortunatly, I no longer have the initial Yifan 2.5.8 rootfs.img or the 3.1 rootfs.img originally used when updating my DXG to 3.1.

delta = 419463168 - 419454976 = 8192 bytes

This might not be a problem as long as the rootfs.img being flashed into the mmcblk0p1 partition is smaller and not larger than the partition. Maybe that has something to do with the vfat mmcblk0p4 containing an ext3 image. I still don't know why Yifan Lu chose bs=1024 except that worked when I flashed the 3.1 rootfs.img to my DXG.

[knc1]

do you have command line access to your DXG?

if so, please post the result of:

cat /proc/partitions

[donB006]

Originally Posted by knc1

do you have command line access to your DXG?

if so, please post the result of:

cat /proc/partitions

Check the kindle-disks.txt above.