How to update Kindle 2/DX/DXG with YifanLu's firmware from 3.2.1 to 3.4.2

[donB006]

Originally Posted by newman

I'd suggest to add following to the respective upgrade script which is in the OTA update bin, and see what happens:

set -x
exec > /mnt/us/my-upgrade.out
exec 2> /mnt/us/my-upgrade.err

(For more see the guide in comment #0.)

Yes, I am still at 3.1 in my DXG. Your suggestion worked and now I know the reason for getting the U007 errors and now U006 errors. I had trouble getting the update files to give more info until I finally deleted the .dat file, made the .sh script executable and generated a FD03 version bin. I then had a verbose messages dump and the 2 files in the /mnt/us directory.

I have found the problem is my own from editing my syslog-ng.conf file so the Amazon md5 check failed. I now know what I need to know. In the future I will save the original config file and edit a copy for my own purposes so I can easily restore the original when needed. Attached is part of the messages file showing the mismatched md5 and is shown in more depth in the two upgrade files in the /mnt/us directory. Thanks Newman for the answer.

[knc1]

Originally Posted by donB006

- - - -

The kernel from Yifan Lu and the OS 3.1 from Yifan Lu should be a matched set because I haven't flashed any different kernels or rootfs partitions since I first used my K3G to upgrade the DXG to 3.1 so I think that much is normal.

Whooaa - - -
Please clarify.
Is (was) the DXG running the kernel and the "OS 3.1" from Yifan Lu?
OR
Is (was) the DXG running the kernel and rootfs from your K3G?

Originally Posted by donB006

I would be surprised if everybody didn't have those errors with the DXG. Of course, the 3.1 OS has some 2.5.8 features with the custom 2.5.8 kernel.

Once again, please be specific.
What device(s) is/was running the "custom 2.5.8 kernel"?
Are they (where they) also running the matching "custom 2.5.8 module build" (a component of the rootfs image)?

Originally Posted by donB006

I now suspect the ERROR might be because there is no WIFI or the 3G WAN was turned off and the Lab126 didn't test for that before trying to remove the modules. Maybe they have something to do with the recovery kernel. Whatever is running the rmmod in preparation for the update still allows continuing into the update so I think there could be another reason. I tried to get more information with ;debugOn and then Update with the same result. I think this is the relavent part:

151102:162349 system: I S50updater:def:wand shutdown complete
151102:162350 system: I S50updater:def:read bundle line [/mnt/us/update-06-558700031-611680021.bin]
151102:162350 system: I S50updater:def:Applying update /mnt/us/update-06-558700031-611680021.bin
151102:162350 system: I S50updater:def:Size of update: 1926999 bytes
151102:162351 system: I S50updater:def:Version before update: 031-S1-055870
151102:162351 system: I _otaupexec:def:processing update /mnt/us/update-06-558700031-611680021.bin
151102:162351 system: I _otaupexec:def:version is "FC02"
151102:162353 system: I _otaupexec:def:update image checksum OK
151102:162357 system: I S50updater:def:Deleting OTA bundle file "/mnt/us/update-06-558700031-611680021.bin"
151102:162357 init: Switching to runlevel: 5

That doesn't help at all. The update lasts for 7 seconds until the upgrade is unsuccessful and issues the U007 reason. I still suspect the jailbreaks and am now going to investigate. I remember reading about a leftover certificate that can cause the U007.
Some say once the Kindle is jailbroken there is no reason to uninstall or reinstall the jailbreak and some say the jailbreak should be kept up-to-date.

Please do not quote anonomous sorces of information.
Who says?
In connection with what device / firmware version series?

I have read both myself, but they have always been about different devices.
So please either quote full context or links to where you are getting that conflicting information.

Originally Posted by donB006

At least I have a backup now in case I spoil my current installation.

There is a possibility the U007 is caused by my history with the KDX and maybe with the K3G before the DXG upgrade.

Now your getting closer to the cause.
Since you say you fixed the problem on both devices with the same fix - the devices must have somehow been sharing the same problem.

This is beginning to sound like they where sharing the same case of:
PEBKAC

The only thing clear so far is that both devices share the same owner.

Originally Posted by donB006

I have used various hacks like usbNetworking, launchpad, readabilitymod, updatesprovider, screensaver and readers like Duokan, Librerator, kpdfviewer and coolreader3. Those are now gone except for usbNetworking and launchpad. I have tweaked the DXG on my own only in the area of the logs where I turned off the logging until just decided to try the 3.4.2 upgrade. When changing the logging I might have modified a couple of files that Amazon checks for changes. I don't know. This is a confusing mystery.

Maybe I should read all of the Kindle Developers Corner again because I know others have had the U007 problem. I need to look into the jailbreaks. Thanks for the attempt knc1.

[donB006]

OK, KNC1-

Let me try to clarify my situation. I first bought a Kindle3 with 3G with OS 3.1 and applied a jailbreak and tried various hacks. I applied the update-providers hack because I was happy with 3.1 and didn't want the additional social reading features with the 3.2 updates. I then bought a Kindle DXG with OS 2.5.8 and wanted the better PDF reader and browser features of the 3.1 OS. I used the Yifan Lu kindleupdater to update the DXG to OS 3.1 using my K3G. That results in the DXG using Yifan's custom recovery kernel based on the 2.5.8 kernel while borrowing most of the OS 3.1 from the K3G. For example my DXG now has modules from both kernels in the /lib/modules/2.6.22.19-lab126 and /lib/modules/2.6.26-rt-lab126. So my 3.1 DXG now runs the 2.6.22.19 kernel while having most of the OS files from the 3.1 K3G built for the 2.6.26-rt kernel. The DXG kernel is actually a custom kernel built by Yifan Lu based on the 2.5.8 kernel and not the stock kernel from the DXG 2.5.8. I expect the same 2.6.22 Yifan custom kernel will be used even if I upgrade my DXG to the 3.4.2 OS.

Reference: https://www.mobileread.com/forums/sho...d.php?t=127969

I now know my problem trying to upgrade my DXG from 3.1 to 3.3 is caused by my editing the syslog-ng.conf file and not the jailbreak. Concerning jailbreaks- NiJule says in his first jailbreak post:
*Note for Kindle >= 3.1 Users:*
If you haven't done so already (either with >= 0.5.N or yifanlu's
jailbreak), please update your jailbreak hack.

Yifan Lu at his website and various places says:

(If you are already jailbroken, regardless of what version you're
running, you don't need to download this. The actual jailbreak hasn't
been updated, just the injection method.)

I figured some of the people who use NiJule's jailbreak would keep their jailbreak up-to-date while some of the people who use Yifan Lu's jailbreak would leave well enough alone. I probably didn't find the best quotes except that is what I meant and that has always caused me to be confused as to what to do. Maybe I misunderstood. Now I think my jailbreak isn't the cause of the failed update.

Yes, I am the cause of my problems. I had edited my syslog-ng.conf file to only report a high level of "notice" and so there was never anything put in my /var/log/messages. I think that improved the speed and saved the eMMC from wear and tear. Coincidently, Amazon looks in the /var/log/messages for various reasons like sending OTA updates so those can be affected. That has never caused any problems as far as I am concered. I never expected Amazon to check the file for the MD5 and that was my mistake. Amazon has a habit of patching numerous files instead of replacing them during an update and they refuse to patch if the file has been changed causing the update to fail. I am now looking for the unmodified syslog-ng.conf to make the Amazon updater happy. I must blame myself for this problem.

Thank you for your interest and detailed questions knc1. I now know what I need to do.

[knc1]

Originally Posted by donB006

OK, KNC1-
- - - -
For example my DXG now has modules from both kernels in the /lib/modules/2.6.22.19-lab126 and /lib/modules/2.6.26-rt-lab126.

So you have support installed for both build versions of the kernel.
That should take care of any mis-match problems.

Originally Posted by donB006

- - - -
I am now looking for the unmodified syslog-ng.conf to make the Amazon updater happy. I must blame myself for this problem.

Thank you for your interest and detailed questions knc1. I now know what I need to do.

Check the recovery images for that file.

= = = =

Note:
The "which directions" question about the jail break -

Note that the suggestions where written years apart.
Consider only the most recently dated suggestion.

[donB006]

Originally Posted by knc1

Check the recovery images for that file.

I found the file with MD5 desired by Amazon updates.

I am still stuck on 3.1 for now and want to spend some time learning about the updates and the kindle_update_tool.py. I am also going to try some experiments because the updates are very interesting.

Attached is a very wierd log extract and can be ignored for now while I learn some more.

EDIT: I suspect the update is using the logs to check for errors and finding my old problems so I am going to wait for the logs to rotate and settle. Also, maybe I didn't make a good update file.

EDIT2: Now I see the syslog-ng.conf has a checksum error while the MD5 matches. Darn.

[newman]

donB006: I suggest you read the original guide, if you haven't done so already, I know it's quite verbose but it contains valuable information, e.g. regarding your former problem with .dat files.

re syslog-ng.conf: Feel free to extract the file from your Kindle and bspatch it on your PC with a the binary patch from the OTA update. Then verify the MD5s (the original, the patched and the expected-new). See how the script does it, it's fairly simple shell string match. (I somehow guess that you still don't have the proper/expected/vanilla syslog-ng.conf file.)

[knc1]

Translation:

+ echo ota_install: I def:patch:patchinfo="/etc/syslog-ng/syslog-ng.conf":patching file
+ mkdir -p patch/etc/syslog-ng/
+ bspatch /etc/syslog-ng/syslog-ng.conf patch/etc/syslog-ng/syslog-ng.conf 006.syslog-ng.conf.patch
+ [ 0 -ne 0 ]
+ md5sum patch/etc/syslog-ng/syslog-ng.conf
+ awk { print $1; }
+ _PATCH_MD5=fdd6871cff4a3d2499cb6d8d61a2ccf8
+ _EXPECTED_MD5=e4e92ce6bd23329049b09f72a29689af
+ [ ! fdd6871cff4a3d2499cb6d8d61a2ccf8 = e4e92ce6bd23329049b09f72a29689af ]
+ logmsg C patch patchinfo="/etc/syslog-ng/syslog-ng.conf" checksum failure

bspatch failed (without a returned error code, which they detected in the logic).
something is seriously wrong with one or more of the files involved in that step.

Note:
You can find bsdiff and bspatch both in source and in binary if you search the web a bit.
It is a utility that hasn't changed for years (decades?).

Edit:
That still doesn't make a lot of sense.

In *nix, config files are plain text.
Why not just view the before and after in a text editor and see if they are as expected.
Be certain to retain the line ending type.

[donB006]

Originally Posted by newman

donB006: I suggest you read the original guide, if you haven't done so already, I know it's quite verbose but it contains valuable information, e.g. regarding your former problem with .dat files.

re syslog-ng.conf: Feel free to extract the file from your Kindle and bspatch it on your PC with a the binary patch from the OTA update. Then verify the MD5s (the original, the patched and the expected-new). See how the script does it, it's fairly simple shell string match. (I somehow guess that you still don't have the proper/expected/vanilla syslog-ng.conf file.)

Thank you for the information. I went back and tried carefully to follow your excellent guide. In my case to simply edit the Amazon 3.3 update for my 3.1 installation would not work. I was able to extract the files and make an update following your directions for the 3.4 update.

I have discovered the file I thought had the desired md5sum was actually the file to result after the patch. The CURRENT means the old file, the EXPECTED is the md5 required of the patched file and the PATCH is the md5 of the patched file. I was trying to make the CURRENT = EXPECTED and I should have tried to make the PATCH(ed) = EXPECTED.

Because I already have the file expected after the patching no changes were needed and I commented out the section trying to update the syslog-ng.conf. I then found I must have also edited the tinyrot.conf file so I also commented that out. The update then almost finished before finding a mismatch with the md5 for the libdtp_module.so.0.1 after patching. So I have my own problems here now although your updating technique works for me almost to completion.

Originally Posted by knc1

Note:
You can find bsdiff and bspatch both in source and in binary if you search the web a bit.
It is a utility that hasn't changed for years (decades?).

I found the bspatch for my debian in the bsdiff package. I was able to get the same result on my desktop as inside the DXG. I had the expected result and the Amazon binary patch for a file I did not have. I tried to patch a few different versions of the syslog-ng.conf and compare the result with the expected result without a match. I found the 'patch -R' not the 'bspatch -R' option so I finally edited the shell script to leave the syslog-ng.conf file I now have because that is what Amazon wanted in the end anyway.

I now see I need to decide how much of the K3G updates I want to allow for my DXG. Yifan Lu originally decided for the 2.5.8 upgrade to 3.1 installation and now I have a mixture of old DXG versions and K3G versions. I am going to follow this thread and study the guide in the original post to see if I can learn enough to do the upgrade properly. I feel I don't understand enough now to know whether I should upgrade certain packages or not to go from 3.1 to 3.3. I have seen some more threads addressing these matters like the 3.2.1 update so maybe I can look there too.

I recently upgraded my K3G from 3.1 to 3.4.2 so my occasional use of the 3G for checking email when away from the internet is already possible even without the DXG. I really like to read with my DXG now so I am not too unhappy without the TLS for my DXG. I enjoy learning and want my DXG to work as good as possible so I will follow this thread.

[newman]

donB006: I think it's quite clear you are perfectly technically capable of doing the upgrade from 3.1 to 3.4.2 it will just take some time to figure out how to overcome Amazon update hassles. So, keep trying. Personally it took me several week to do the upgrade and I knew nothing about Kindle internals at the start.

At a times when the patching fails you may feel you have 'a' correct file already but not a bit-by-bit exact match, then feel free to edit the script and alter respective MD5 sums to match your case. It should be pretty safe with text files, but beware of executables (e.g. dll's).

About the mix of 2.5.8 FW and 3.x FW. I believe that the right/expected mix on this hybrid system is kernel - the YifanLu's kernel built from 2.5.8 FW sources - (+ uboot?) and init system + few fixes from 2.5.8 and pretty much everything else from 3.x FW.

Several OTA updates contain a kernel update, the kernel is updated but your 2.6.22 kernel from 2.5.8 FW is not affected due to, I believe, HW differences or because the 2.5.8 FW uboot kicks in from MTD device at a first code after boot and it knows where to look for the old 2.5.8 FW kernel.

[knc1]

Originally Posted by newman

- - - - -
It should be pretty safe with text files, but beware of executables (e.g. dll's).
- - - - -

For future readers:

Substititute:
"It should be pretty safe with text files, but beware of binary files."

Kindles run Linux, a *nix based system, which means:

Shared library names end with: .so not with: .dll
Executables do not have a specific extension (or any extension at all), not .exe
All configuration files are plain text not binary. There is no binary system-wide 'registry'.

In point of fact, *nix based systems do not have any 'reserved' name extensions (the ones commonly seen, like .so, is just convention), because file and path names do not have an 'extension' field.

[donB006]

Originally Posted by newman

About the mix of 2.5.8 FW and 3.x FW. I believe that the right/expected mix on this hybrid system is kernel - the YifanLu's kernel built from 2.5.8 FW sources - (+ uboot?) and init system + few fixes from 2.5.8 and pretty much everything else from 3.x FW.

The correct mix of 2.5.8 and 4.3.2 files is my real problem. I really don't know what files I should upgrade. The files originally added and replaced from the 3.1 should be safe to upgrade while leaving the remaining 2.5.8 files in place. There might still be something I don't know about and that is why I hesitate.

I do have the Kindle Keyboard with 3G already updated to 3.4.2 and I could always just take the desired files from there and put them in the DXG. I should be able to recover using Yifan Lu's recovery kernel and my saved rootfs.img from before the experiment. The kindles are just so unfamiliar to me especially with the Java that I don't feel confident to know the files I should upgrade.

I would really just like to update the webkit part for the TLS and browser improvements. I don't want to disturb the actual operation of the DXG just the application for browsing the internet. The webkit is open source and included in the Amazon k3g source code so that would be nice to be able to do separately.

I feel I could use your techniques to finally upgrade my DXG to the 3.4.2 applications if I only wouldn't disturb the software needed by the DXG and that is now beyond my understanding.

[newman]

Understood. Reasonable approach.

Originally Posted by donB006

I would really just like to update the webkit part for the TLS and browser improvements. I don't want to disturb the actual operation of the DXG just the application for browsing the internet. The webkit is open source and included in the Amazon k3g source code so that would be nice to be able to do separately.

When it comes to TLS the TLS-enabling part happened in the gnutls library. Do a diff of gnutls sources from a 3.4 to 3.4.2 FW releases of Amazon-released sources. There you'll find a one-liner change. If you're able to build the newer gnutls, it may be sufficient to replace the stock gnutls libs with the custom ones. Or you could extract patches all the way to 3.4.2 and apply them to the gnutls libs (if there are problems with linking apply patches to more components... you could even apply all the patches locally to your rootfs image, keep a backup though , and add the results selectively...).

[knc1]

Originally Posted by newman

Understood. Reasonable approach.



When it comes to TLS the TLS-enabling part happened in the gnutls library. Do a diff of gnutls sources from a 3.4 to 3.4.2 FW releases of Amazon-released sources. There you'll find a one-liner change. If you're able to build the newer gnutls, it may be sufficient to replace the stock gnutls libs with the custom ones. Or you could extract patches all the way to 3.4.2 and apply them to the gnutls libs (if there are problems with linking apply patches to more components... you could even apply all the patches locally to your rootfs image, keep a backup though , and add the results selectively...).

???
That says you have the binary you are starting with and the binary you want to reach -
So why not just bsdiff the two (skipping any incrementals) and apply (bspatch) that?

[newman]

Originally Posted by knc1

???
That says you have the binary you are starting with and the binary you want to reach -
So why not just bsdiff the two (skipping any incrementals) and apply (bspatch) that?

Yes, there are several ways how to do it. Thanks for pointing that out. Or... since donB006 has K3 he could copy (or perhaps mount --bind first) those files directly: K3→DXG.

[knc1]

Originally Posted by newman

Yes, there are several ways how to do it. Thanks for pointing that out. Or... since donB006 has K3 he could copy (or perhaps mount --bind first) those files directly: K3→DXG.

Which brings to mind a question I have had for awhile now . . . .

Couldn't we just build 2.5.8 -> 3.4.2 set of bspatches?
Well, technically we could, but that raises another question . . . .
Could we distribute them, ready to use?

There is no question about the open source parts.
But those parts that are Amazon proprietary . . . .

Would a bzpatch be considered a "Fair Use" quote?

"Fair Use" is a complicated subject, even outside of computer binaries.
https://janefriedman.com/the-fair-use-doctrine/

= = = =

If some lawyer showed up at my door, I would be willing to argue that it is within the intent of "Fair Use" (non-profit research).