calibre and the new kindle layout engine

[jackie_w]

Originally Posted by JSWolf

I'm still thinking it's a toggle in the header of the KF8 book.

I guess the mists will part and all will be revealed in the next few weeks

[JSWolf]

Originally Posted by jackie_w

I guess the mists will part and all will be revealed in the next few weeks

What might work would be to get a copy of the an eBook that has typesetting not enabled and an updated copy that does have typesetting enabled to compare the headers. It would help if we had a few different eBooks to compare. If it is a header toggle and the place in the header for the toggle is found, than this can be put into Calibre.

[shalym]

Originally Posted by JSWolf

What might work would be to get a copy of the an eBook that has typesetting not enabled and an updated copy that does have typesetting enabled to compare the headers. It would help if we had a few different eBooks to compare. If it is a header toggle and the place in the header for the toggle is found, than this can be put into Calibre.

Ok--I just downloaded an updated version of Revival by Stephen King. I have the original already downloaded as well. If you can tell me how to look at the header of both books, I will do that.

Shari

[jhowell]

Amazon knows for which device a copy is being downloaded and can deliver different versions of books depending on the capabilities of that device. Once someone has both a PW3 and a slightly older device it would be interesting to download the same book to both and see what, if anything, is different between them.

[Doitsu]

Originally Posted by shalym

Ok--I just downloaded an updated version of Revival by Stephen King. I have the original already downloaded as well. If you can tell me how to look at the header of both books, I will do that.

You can extract the header information with DumpMobiHeader_v019.py, which is included in KindleUnpack. Simply unzip the archive, copy DumpMobiHeader_v019.py to the folder that contains both books, open a command prompt window and enter the following commands (replace User with your Windows user name):

cd "C:\Users\User\Downloads"
python DumpMobiHeader_v019.py book_1.azw3 > book_1.log

The above commands assume that you've Python installed and copied both books and the python file to your downloads folder. You can, of course, use any folder.

If all went well you should end up with a book_1.log file with the header information for the first book.

[odamizu]

Originally Posted by jackie_w

... the presence or absence of hyphenation css in the azw3 itself may not be the controlling factor. To use the Kobo as an example ... overriding css file built into the firmware. It's how the customisation of font, size, line-spacing, margins, font weight is handled. Maybe the hyphenation css will be placed in a similar Kindle css override file.

At this point I'm kind of thinking CSS inside the book is not the controlling factor. I suppose there might be an overriding CSS file in the firmware, but if the controlling factor is device-side, why would Amazon have to re-process all their ebooks? Unless the overriding CSS file is a totally new thing and they need to add something to the Kindlebooks so they'll recognize it? This really is quite mystifying.

Originally Posted by jhowell

Amazon knows for which device a copy is being downloaded and can deliver different versions of books depending on the capabilities of that device. Once someone has both a PW3 and a slightly older device it would be interesting to download the same book to both and see what, if anything, is different between them.

Agreed. This is my concern with all my tests -- that the AZW3s I'm downloading via K4Mac don't actually have the enhanced code because K4Mac doesn't support enhanced typesetting yet.

Originally Posted by Doitsu

You can extract the header information with DumpMobiHeader_v019.py ...

Thanks! I just tried it, then ran "diff" to compare the two log files. I then ran diff to compare everything in the OEBPS folder, including the CSS and OPF files.

Sadly, I don't think it shed any light on this puzzle. The only difference between the files was in the "Watermark (hex)" content: the numbers were different. Everything else was the same. However, I think (but am not positive) that the watermark changes with every download, in which case it would be expected that they would be different, and this finding is not significant.

Also, I'm not sure what the "Watermark (hex)" content is -- is it DRM-related?

I'm now thinking the AZW3s I'm getting via K4Mac are not in fact enhanced, and as jhowell points out, we need an enhanced AZW3 being downloaded to a PW3 to really figure this out.

[blaenk]

Originally Posted by odamizu

At this point I'm kind of thinking CSS inside the book is not the controlling factor. I suppose there might be an overriding CSS file in the firmware, but if the controlling factor is device-side, why would Amazon have to re-process all their ebooks? Unless the overriding CSS file is a totally new thing and they need to add something to the Kindlebooks so they'll recognize it? This really is quite mystifying.

That's exactly what I was saying is confusing, glad to see it's not just me

Originally Posted by odamizu

I'm now thinking the AZW3s I'm getting via K4Mac are not in fact enhanced, and as jhowell points out, we need an enhanced AZW3 being downloaded to a PW3 to really figure this out.

I agree. I got my PW3 earlier but I didn't have a chance to play around with it. If someone can point me to a typesetting-enabled book that is free I'll help in any way I can.

I have a suspicion though, that none of this will work until the new typesetting firmware is released. From what I've read and seen, the PW3 doesn't ship with it. Sure I see Bookerly the font, but not any of the type setting features (nor do I see any of the new UI stuff that I've seen in the review PW3s, I think they got beta versions of that firmware). Perhaps like was said earlier, since it's not the new typesetting supporting firmware, it won't download typesetting enabled books.

[PoP]

Originally Posted by blaenk

...
If someone can point me to a typesetting-enabled book that is free I'll help in any way I can.
...

PandathePanda posted two links here perhaps can you throw your PW3 at it.

[odamizu]

Originally Posted by blaenk

That's exactly what I was saying is confusing, glad to see it's not just me

Nope, not just you. I'm definitely with you on this

Curious to see if you discover anything interesting when downloading directly to your PW3.

[eteb]

Originally Posted by odamizu

Curious to see if you discover anything interesting when downloading directly to your PW3.

I downloaded the hot romance book on both my PW1 and my PW3. They render slightly differently (PW3 on the right): http://i.imgur.com/5bBdAMM.jpg. Can't see any hyphenation going on though.

md5sum says:

3f04f6d5df5f44e21381186263b7b183  pw1-Deadly Desires (ACRO)_B00CRJOLNO.azw
b0927336639fd624b5adbd674f612a76  pw3-Deadly Desires (ACRO)_B00CRJOLNO.azw

A header dump only shows a difference in the watermark key.

Edit: doing a binary diff on the files using cmp in Linux only shows one difference (w differ: char 1228, line 4), which I guess is the header.

[odamizu]

Originally Posted by eteb

I downloaded the hot romance book on both my PW1 and my PW3 .... doing a binary diff on the files using cmp in Linux only shows one difference (w differ: char 1228, line 4), which I guess is the header.

Thanks for this info! Would you mind showing the two different versions of line 4?

[blaenk]

This is why I didn't bother checking this with my PW3. I'm pretty sure that Amazon either hasn't started serving typesetting enabled books for download yet, or they are but only to devices with the newer firmware, which the PW3 still doesn't have.

[eteb]

Originally Posted by odamizu

Thanks for this info! Would you mind showing the two different versions of line 4?

The dump file diff:

Spoiler:

$ diff pw1.log pw3.log 
2c2
< pw1-Deadly Desires (ACRO)_B00CRJOLNO.azw .AZW
---
> pw3-Deadly Desires (ACRO)_B00CRJOLNO.azw .AZW
177c177
<         Value: "atv:kin:2:33vi3S71c9ungjhb+iT7w49Dw1ePELT/4XlyGIhmJZnho3swwTCc4Dgs5/RSrzewnWzOOS5GmYEfqS5OkT1BQVg8f3ABPpJvAwGFdK6cUsvw6fQ7B2M5i5HNwMTg3kEQwdGoTlDUJqm4db8/SijByDAX/yut1AVbaoHGQzgNV5k=:++wJ2HKo0FgHdaC8+IRQ2kOEC68="
---
>         Value: "atv:kin:2:gp4+WjkJUuTyAlGqJC70XY9Dw1ePELT/4XlyGIhmJZnho3swwTCc4Dgs5/RSrzewnWzOOS5GmYEfqS5OkT1BQVg8f3ABPpJvAwGFdK6cUsuB6ERN2nhoUj1sO3E3nHqiwdGoTlDUJqm4db8/SijByL3qVDYtMF96AWCeiVPtyDM=:9HkjS1UBPyMNkClCpFBFfV73aPY="

And the binary diff:

Spoiler:

$ cmp -l pw1-Deadly\ Desires\ \(ACRO\)_B00CRJOLNO.azw pw3-Deadly\ Desires\ \(ACRO\)_B00CRJOLNO.azw | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}'
000004CC 33 67
000004CD 33 70
000004CE 76 34
000004CF 69 2B
000004D0 33 57
000004D1 53 6A
000004D2 37 6B
000004D3 31 4A
000004D4 63 55
000004D5 39 75
000004D6 75 54
000004D7 6E 79
000004D8 67 41
000004D9 6A 6C
000004DA 68 47
000004DB 62 71
000004DC 2B 4A
000004DD 69 43
000004DE 54 37
000004DF 37 30
000004E0 77 58
000004E1 34 59
00000536 76 75
00000537 77 42
00000539 66 45
0000053A 51 52
0000053B 37 4E
0000053C 42 32
0000053D 32 6E
0000053E 4D 68
0000053F 35 6F
00000540 69 55
00000541 35 6A
00000542 48 31
00000543 4E 73
00000544 77 4F
00000545 4D 33
00000546 54 45
00000547 67 33
00000548 33 6E
00000549 6B 48
0000054A 45 71
0000054B 51 69
00000561 44 4C
00000562 41 33
00000563 58 71
00000564 2F 56
00000565 79 44
00000566 75 59
00000568 31 4D
00000569 41 46
0000056A 56 39
0000056B 62 36
0000056C 61 41
0000056D 6F 57
0000056E 48 43
0000056F 47 65
00000570 51 69
00000571 7A 56
00000572 67 50
00000573 4E 74
00000574 56 79
00000575 35 44
00000576 6B 4D
00000579 2B 39
0000057A 2B 48
0000057B 77 6B
0000057C 4A 6A
0000057D 32 53
0000057E 48 31
0000057F 4B 55
00000580 6F 42
00000581 30 50
00000582 46 79
00000583 67 4D
00000584 48 4E
00000585 64 6B
00000586 61 43
00000587 43 6C
00000588 38 43
00000589 2B 70
0000058A 49 46
0000058B 52 42
0000058C 51 46
0000058D 32 66
0000058E 6B 56
0000058F 4F 37
00000590 45 33
00000591 43 61
00000592 36 50
00000593 38 59

[odamizu]

@eteb: Thanks again!

I downloaded the same book to Kindle for Mac and was surprised to see it unpack as a mobi

[odamizu]

I've become obsessed with this puzzle. Thinking out loud here:

1. At this point, the only devices with the enhanced typsetting engine are iOS devices and Kindle Fire tablets.
   
2. Per the Wiki, iOS devices receive AZK format and Kindle Fire receives primarily AZW3.
   
3. If the enhanced typesetting were device-side only, there would be no need to rejigger every Kindlebook. And yet, Amazon seems to be doing just that. Once a book has been rejiggered, it receives an "enhanced typesetting" label on the web site.
   
4. Since both iOS devices and Kindle Fires are able to take advantage of the enhanced typesetting, it suggests that both AZK and AZW3 must be rejiggered.
   
5. I find it hard to believe that Amazon has 2 versions of AZW3 they're distributing, one for Kindle Fires and one for eink Kindles. And yet, the lack of any sign of "enhanced typesetting" inside the AZW3s we've peeked in suggest just that.

What am I missing? I feel like I'm missing something big here. Thoughts?