calibre and the new kindle layout engine

[JSWolf]

Originally Posted by mbovenka

I don't think that was his point.

So what was the point?

[jackie_w]

Originally Posted by JSWolf

So what was the point?

Only PoP can answer that, but FWIW, when I read that post I took it to mean that the enhanced typesetting had no apparent impact on Alf's tools, Calibre conversions and KindleUnpack.

[PandathePanda]

Originally Posted by jackie_w

Only PoP can answer that, but FWIW, when I read that post I took it to mean that the enhanced typesetting had no apparent impact on Alf's tools, Calibre conversions and KindleUnpack.

Noope it has no impact on those tools

[odamizu]

I downloaded a dozen of my purchased Kindle books to Kindle for Mac.

Of those, half were labelled as having enhanced typesetting on Amazon's website -- though it's possible the versions I received don't include enhanced typesetting code and only devices that can handle enhanced typesetting (PW3, K4iOS, etc.) will receive "enhanced" versions.

Curiously, 2 books showed hyphenation when I opened them in K4Mac. One supposedly had enhanced typesetting, the other did not. Five other books supposedly having enhanced typesetting did not have hyphens. Repeat: hyphenation doesn't seem to be reliant on this enhanced typesetting when using K4Mac. After KindleUnpack-ing the books, I noticed that books with hyphenation had CSS for hyphens (i.e., -epub-hyphens:auto; -webkit-hyphens:auto) whereas books without hyphens lacked this CSS. Whether or not the books had enhanced typesetting seemed irrelevant.

I also did a quick comparison of opf files and found no toggle, like there is with TTS. This was disappointing. I was hoping JSWolf was right about a toggle. Though there may still be something in the opf that enables enhanced typesetting that I missed on my cursory scan. There was also nothing in the CSS (other than the hyphen code) that shouted "enhanced typesetting" to me.

I think it's too early to tell what impact this enhanced typesetting will have on Alf, Calibre, KindleUnpack, etc. Hopefully nothing, and I don't expect it to. But the fact that developers attempting to reverse engineer the enhanced typesetting have been thwarted (so far) worries me.

[PoP]

Originally Posted by jackie_w

Only PoP can answer that, but FWIW, when I read that post I took it to mean that the enhanced typesetting had no apparent impact on Alf's tools, Calibre conversions and KindleUnpack.

This is what I meant.

Originally Posted by PandathePanda

Noope it has no impact on those tools

Exactly.

[blaenk]

Originally Posted by odamizu

After KindleUnpack-ing the books, I noticed that books with hyphenation had CSS for hyphens (i.e., -epub-hyphens:auto; -webkit-hyphens:auto) whereas books without hyphens lacked this CSS. Whether or not the books had enhanced typesetting seemed irrelevant.

Well we at least have that, that's great to hear. Does anyone know if Calibre already does this? CSS hyphenation?

Originally Posted by odamizu

I also did a quick comparison of opf files and found no toggle, like there is with TTS. This was disappointing. I was hoping JSWolf was right about a toggle. Though there may still be something in the opf that enables enhanced typesetting that I missed on my cursory scan. There was also nothing in the CSS (other than the hyphen code) that shouted "enhanced typesetting" to me.

That's weird, since Amazon seemingly is indeed processing the books to have the new typesetting. I'm no expert in how ebook files are constructed, but is it possible that Amazon is actually processing the book content itself? Perhaps they added hyphenation markers to the actual content, for example. Perhaps they also did something similar (perhaps proprietary) for the rest of the typesetting? I imagine it would be a matter of inspecting the actual book content. My guess would be that they added some sort of hinting markers, such as soft hyphens, to facilitate the new typesetting. It's unfortunate that it's not as simple as some metadata toggle or CSS directive.

[Doitsu]

Originally Posted by odamizu

Whether or not the books had enhanced typesetting seemed irrelevant.

IMHO, the whole "enhanced typesetting" business is nothing but marketing hype. All they did was add a new font with better kerning and hyphenation support. Both features combined will automatically make for a more pleasant reading experience.

Originally Posted by odamizu

After KindleUnpack-ing the books, I noticed that books with hyphenation had CSS for hyphens (i.e., -epub-hyphens:auto; -webkit-hyphens:auto) whereas books without hyphens lacked this CSS. Whether or not the books had enhanced typesetting seemed irrelevant.

I did some test with the 4.9.1 Kindle for iOS app and found out that sideloaded mobi6 and mobi7 files are automatically hyphenated even if the original CSS was missing -epub-hyphens:auto; -webkit-hyphens:auto entries.
OTOH, AZK (KF8/Mobi8) files required either -epub-hyphens:auto; or -webkit-hyphens:auto; entries for hyphenation.
(The new Bookerly font can be selected for both file types.)

Originally Posted by odamizu

Though there may still be something in the opf that enables enhanced typesetting that I missed on my cursory scan. There was also nothing in the CSS (other than the hyphen code) that shouted "enhanced typesetting" to me..

Since Amazon has neither updated KindlePreviewer/KindleGen nor the Kindle Publishing Guidelines, it's highly unlikely that there are new "enhanced typesetting" .opf entries.
IMHO, all that Amazon needs to do enable "enhanced typesetting" for existing books is append -epub-hyphens:auto; or -webkit-hyphens:auto; entries to the main stylesheet and maybe remove/override all existing body font definitions so that books will automatically be displayed using the default Bookerly font.

[JSWolf]

Originally Posted by odamizu

I downloaded a dozen of my purchased Kindle books to Kindle for Mac.

Of those, half were labelled as having enhanced typesetting on Amazon's website -- though it's possible the versions I received don't include enhanced typesetting code and only devices that can handle enhanced typesetting (PW3, K4iOS, etc.) will receive "enhanced" versions.

Curiously, only 2 books showed hyphenation. One supposedly had enhanced typesetting, the other did not. Five other books supposedly having enhanced typesetting did not have hyphens. Repeat: hyphenation doesn't seem to be reliant on this enhanced typesetting. After KindleUnpack-ing the books, I noticed that books with hyphenation had CSS for hyphens (i.e., -epub-hyphens:auto; -webkit-hyphens:auto) whereas books without hyphens lacked this CSS. Whether or not the books had enhanced typesetting seemed irrelevant.

I also did a quick comparison of opf files and found no toggle, like there is with TTS. This was disappointing. I was hoping JSWolf was right about a toggle. Though there may still be something in the opf that enables enhanced typesetting that I missed on my cursory scan. There was also nothing in the CSS (other than the hyphen code) that shouted "enhanced typesetting" to me.

I think it's too early to tell what impact this enhanced typesetting will have on Alf, Calibre, KindleUnpack, etc. Hopefully nothing, and I don't expect it to. But the fact that developers attempting to reverse engineer the enhanced typesetting have been thwarted (so far) worries me.

The OPF won't say if the advanced typesetting is a toggle or not as KindleUnpack doesn't know about this (if indeed it is a toggle like TTS). I did look at one of the Kindle books I have that says it has typesetting enabled and I cannot see anything in the CSS and OPF that turns this on. There is no option in CSS for hyphens. But what I can do is try the hyphen option you've specified and see if that works. I hope it's something that simple to make it work.

I was thinking of getting a PW3 for someone as a gift to replace her Kindle Touch. But if I cannot get side loaded eBooks to use the new typesetting, I might not do that.

[blaenk]

Originally Posted by Doitsu

IMHO, all that Amazon needs to do enable "enhanced typesetting" for existing books is append -epub-hyphens:auto; or -webkit-hyphens:auto; entries to the main stylesheet and maybe remove/override all existing body font definitions so that books will automatically be displayed using the default Bookerly font.

Originally Posted by JSWolf

There is no option in CSS for hyphens. But what I can do is try the hyphen option you've specified and see if that works. I hope it's something that simple to make it work.

That would be really awesome.

Originally Posted by Doitsu

IMHO, the whole "enhanced typesetting" business is nothing but marketing hype. All they did was add a new font with better kerning and hyphenation support. Both features combined will automatically make for a more pleasant reading experience.

That actually sounds plausible to me. At least we wouldn't have to worry about proprietary changes that must be reverse engineered if this is true.

[blaenk]

One curiosity I have though is, if it's really that simple of a change that it requires to enable the new typesetting, then why would Amazon have to go through each book to apply this? Couldn't they just set that at the firmware level, for the system to force-apply these changes (e.g. the CSS hyphenation setting) automatically? As for the body font definitions they could simply ignore those, no? I'm not sure why they have to go through each book and make it "new typesetting enabled."

[odamizu]

Originally Posted by blaenk

Does anyone know if Calibre already does this? CSS hyphenation?

You can instruct Calibre to add the CSS when you convert: Look & Feel > Extra CSS > body {-epub-hyphens:auto; -webkit-hyphens:auto;}

I just tested this. When you add the above CSS to an AZW3 and open it in K4Mac, it will have hyphenation. But it does not work on a Kindle Voyage (yet?). It will be interesting to see whether it will work on a PW3 or other eink Kindles once they get updated with the new enhanced typesetting firmware.

Originally Posted by blaenk

That's weird, since Amazon seemingly is indeed processing the books to have the new typesetting. I'm no expert in how ebook files are constructed, but is it possible that Amazon is actually processing the book content itself? Perhaps they added hyphenation markers to the actual content, for example.

I was wondering the same thing -- whether they are adding soft hyphens or ligatures -- but when I looked inside the KindleUnpack-ed ebooks, I saw nothing like that, just straight clean text. (Though as I said, perhaps ebooks downloaded via K4Mac don't include the enhanced typesetting code?)

Originally Posted by Doitsu

IMHO, the whole "enhanced typesetting" business is nothing but marketing hype. All they did was add a new font with better kerning and hyphenation support. Both features combined will automatically make for a more pleasant reading experience.

To be honest, I don't care about having the enhanced typesetting.

What I do care about is whether "enhanced" Kindlebooks can still be liberated, tweaked and converted (so far so good), and whether any updated firmware forced on my Kindle Voyage will result in problems with ebooks that I liberate, tweak, convert and sideload.

Also, I'm interested in hyphenation not because I want it, but because I want to be sure I can turn it off

Originally Posted by Doitsu

IMHO, all that Amazon needs to do enable "enhanced typesetting" for existing books is append -epub-hyphens:auto; or -webkit-hyphens:auto; entries to the main stylesheet and maybe remove/override all existing body font definitions so that books will automatically be displayed using the default Bookerly font.

That's what I originally thought, but this exchange with Nate the Great worries me:

Originally Posted by odamizu

Ah, but will we be able to add it [enhanced typesetting] to ebooks we convert with Calibre? That is the question.

Originally Posted by Nate the great

Not at this time, no.

I know a couple developers who have tried to force the issue by reverse engineering the changes and then adding the new css to their ebooks before uploading or sideloading them. It didn't work. [emphasis added]

Also, emailing an ebook to your Kindle account won't do it either.

If the enhanced typesetting is as simple as tweaking CSS, the developers wouldn't have had problems getting it to work, no?

[JSWolf]

Originally Posted by blaenk

One curiosity I have though is, if it's really that simple of a change that it requires to enable the new typesetting, then why would Amazon have to go through each book to apply this? Couldn't they just set that at the firmware level, for the system to force-apply these changes (e.g. the CSS hyphenation setting) automatically? As for the body font definitions they could simply ignore those, no? I'm not sure why they have to go through each book and make it "new typesetting enabled."

One of my Kindle eBooks that says it is typesetting enabled does not have any hyphenation commands in the CSS. That's why I think it's a toggle in the header like the TTS toggle Amazon added to the header.

[odamizu]

Originally Posted by JSWolf

One of my Kindle eBooks that says it is typesetting enabled does not have any hyphenation commands in the CSS. That's why I think it's a toggle in the header like the TTS toggel Amazon added to the header.

I think you're right about the toggle. Just wish we could find it.

I have 13 AZW3s downloaded from Amazon into K4Mac. 7 supposedly have enhanced typesetting, 6 do not. (I say supposedly because I'm only like 95% sure K4Mac receives the same version as an eink Kindle. Does anyone know for sure?)

Of the 7 enhanced AZW3s, only 1 has CSS for hyphens.

K4Mac: When opening the enhanced AZW3s in K4Mac, the only one that shows hyphens is the one with hyphen-CSS.

K4iOS: When downloading the same titles into K4iOS (and receiving AZK? or MOBI? I'm not sure what K4iOS receives, but it's not AZW3), 6 out of 7 show hyphens, including 5 that do not have hyphen-CSS in the AZW3.

Of the 6 non-enhanced AZW3s, 1 has CSS for hyphens.

K4Mac: When opening the non-enhanced AZW3s in K4Mac, the one with hyphen-CSS has hyphens, even though it is non-enhanced. The rest have no hyphens.

K4iOS: When downloading the same titles into K4iOS, the same thing happens: if it has hyphen-CSS in the AZW3, it shows hyphens in K4iOS, even if it is non-enhanced.

Not surprisingly, my Kindle Voyage does not show hyphens for anything, regardless of CSS or enhanced typesetting. But once the updated software is released, it will be interesting to see what happens -- and until then, the above tests are somewhat flawed as it's like working with half a blindfold.

[jackie_w]

To add some more (completely unfounded) speculation ... the presence or absence of hyphenation css in the azw3 itself may not be the controlling factor. To use the Kobo as an example, it has 2 main book rendering apps one for standard epub and one for kepubs. Each app has its own, but different, overriding css file built into the firmware. It's how the customisation of font, size, line-spacing, margins, font weight is handled. Maybe the hyphenation css will be placed in a similar Kindle css override file.

In my own experience these override css files can be a boon or a curse depending on your point of view and how aggressively they attempt to change each book's own css file.

[JSWolf]

Originally Posted by jackie_w

To add some more (completely unfounded) speculation ... the presence or absence of hyphenation css in the azw3 itself may not be the controlling factor.

Because some books labeled as typesetting enabled and do not have anything in CSS that might control hyphenation, I'm still thinking it's a toggle in the header of the KF8 book. It's how Amazon controls TTS. I see no reason why Amazon would not do it that way. It also means if you want the typesetting, you would have to buy from Amazon for books they list as enabled. One thing I do not know is if removing DRM would cause typesetting to not work for books that have typesetting enabled.