Joshua Tallent on the Single Epub Challenge

[DiapDealer]

Originally Posted by JSWolf

I have loaded one of these new publisher ePub 3 into Sigil and it caused errors where none existed. This was just loading with no editing.

There is no such thing as"just loading" into Sigil. There are certain structural considerations and well-formed checks that are GOING to happen. Since these things are predicated on EPUB2 (And there not being any support for HTML5 in its well-formed checks), it can certainly make a dog's lunch of an EPUB3.

Being able to open an EPUB3 without breaking it is certainly one of the short-term goals of current Sigil development, but it's not ready.

[JSWolf]

Originally Posted by DiapDealer

There is no such thing as"just loading" into Sigil. There are certain structural considerations and well-formed checks that are GOING to happen. Since these things are predicated on EPUB2 (And there not being any support for HTML5 in its well-formed checks), it can certainly make a dog's lunch of an EPUB3.

Being able to open an EPUB3 without breaking it is certainly one of the short-term goals of current Sigil development, but it's not ready.

I would love a feature that would allow a conversion of ePub 3 to ePub 2 on load. Most ePub 3 are nothing special and should have been ePub 2 from the start.

[Turtle91]

Originally Posted by DiapDealer

Sigil currently has no such function. There exists a third-party Sigil plugin which if properly installed, can export the current epub as an EPUB3. And yes, it's true that you cannot turn around and open that exported EPUB3 for editing with Sigil. Because Sigil is currently only an EPUB2 editor.

ahhh. That makes sense. Thanks!

[Hitch]

Originally Posted by JSWolf

I would love a feature that would allow a conversion of ePub 3 to ePub 2 on load. Most ePub 3 are nothing special and should have been ePub 2 from the start.

IMHO, most of them are bloody pretentious, and do NOT do anything that a plain old, plain old ePUB2 would do, that's the part that slays me. And we're STILL at the "most devices won't read ePUB3, anyway" point, no matter what we all think about it.

And Joshua, of all people, should bloody well know that. Saying "start with the best and work your way down..." (or words to that effect, not a perfect quote) is just...to me, frustrating. ePUB3 isn't necessarily "the best." I for one preferred the NCX over ePUB3 nav, and some other aspects. I think parts of ePUB3 are more-complex than they need be, just for the sake of it.

I'll say it again, and you've all heard it from me before: wish the Committee had spent more time focusing on conformance and straightening out those things that could have used clarification, and a lot bloody less on "moving forward for forward's sake," (and media!). A lot of ePUB3 feels foofy, rather than genuine progress in the eBooks themselves.

Offered FWIW--just my opinion.

Hitch

[Notjohn]

Originally Posted by Hitch

As our resident Curmudgeon has pointed out, usually, it's when Apple is involved. They have some display glitches, usually for images, that might need extra tweakage. In fact, I have one right now, that has come back, 6 months later...{sigh}, and the client is claiming that now it doesn't work right. It works a treat on ADE and Readium and SonyReader and Nook for PC and Nooks of all sizes...but it's doing something funky on bloody Apple. When I see more, I'll let you know.

Hitch

I have noticed that the iPad (or anyhow the iPad emulation at KDP online) seems to ignore an inline width=n instruction, just like the early Kindles, blowing up my colophons to an awkward size. Oddly this doesn't happen on my iPhone 4.

[dgatwood]

Originally Posted by Notjohn

As a practical matter, Hitch, when is a single epub not sufficient? That's all I have done since 2012 when you pointed out that I could upload epubs to the KDP.

I've found it difficult to use a single EPUB for both Amazon and everything else, because kindlegen does a really, really bad job of translating CSS into HTML tags, requiring lots of gross HTML, often including deprecated HTML tags and attributes to work around its limitations so that the content looks decent on older Kindles.

I do, however, single-source my EPUB content intended for actual EPUB delivery.

[Hitch]

Originally Posted by dgatwood

I've found it difficult to use a single EPUB for both Amazon and everything else, because kindlegen does a really, really bad job of translating CSS into HTML tags, requiring lots of gross HTML, often including deprecated HTML tags and attributes to work around its limitations so that the content looks decent on older Kindles.

I do, however, single-source my EPUB content intended for actual EPUB delivery.

Dag:

Aren't you just using KF7 fallback, rather than trying to use old HTML to work around that?

Hitch

[dgatwood]

Originally Posted by Hitch

Dag:

Aren't you just using KF7 fallback, rather than trying to use old HTML to work around that?

Where possible, yes, but I don't blow all that KF7 fallback crap into the master EPUB because I was having problems with some readers rendering the fallback content.

Here's a list of differences between my Kindle EPUB and my production EPUB:

1. ASIN instead of ISBN.
2. Larger art files because I don't have to deal with Apple's restrictions, and kindlegen scales them down anyway, so it is better to start from a higher quality, larger image.
3. Extra paragraph after the chapter title for KF7 purposes.
4. Very different CSS markup because kindlegen misinterprets CSS selectors too badly to be usable as soon as you do anything more than 1:1 paragraph to style correspondence in either direction. I end up using more tags on Kindle to allow more opportunities for blowing in styles, and use tables for certain formatting that would normally be handled by left margins on elements, because otherwise KF7 won't indent. And it doesn't make sense to include two copies of a chapter heading for KF7 purposes, so I just made it work for KF8 with the KF7 markup and different styles, but the markup is too ugly for me to ship it for EPUB readers.
5. The use of the <nobr> tag in various places because Kindle doesn't respect the white-space: nowrap CSS property. (That might only affect KF7, but because the tag is not actually in the HTML specification at all, I can't ship that workaround in a proper EPUB or it will fail validation.)
6. Use of &nbsp; entities to force left indentation in certain spots (shown only on KF7).
7. Use of <center> tags to force centering on KF7. That tag is deprecated, and is considered unsupported in HTML 5, and hence EPUB 3, so I'd get verifier errors if I shipped an EPUB 3 book with those workarounds.
8. Nested <blockquote> tags to force sufficient indentation of certain blocks on KF7. To avoid repeating those blocks of content with fewer tags for KF8, I use different CSS on KF8 than EPUB so that the nested blockquotes won't cause KF8 readers to indent too much.
9. An added &nbsp; at the end of every paragraph to ensure that KF7 readers don't fail to display empty paragraphs.
10. Fewer workarounds for old iBooks bugs.
11. Extra <span class="kf8only">...</span> around zero-width joiner entities to avoid KF7 devices rendering them as a "missing glyph" rectangle.
12. Title pages, part pages, and other SVG-based special pages contain an extra HTML version of the content, with the SVG content wrapped to be visible only on KF8-compatible devices and the non-SVG content wrapped to be visible only on KF7-compatible devices. (I don't think KF8 readers support the <epub:switch> tag, so the non-SVG fallback markup will continue to be different even if I start shipping non-SVG fallback markup in my EPUB content.)
13. Soft hyphen entities (& #173;) replaced with <wbr/>, because KF7 readers support the tag, but don't support the equivalent entity. I don't do this in the general EPUB because it wasn't an officially supported tag until HTML 5/EPUB 3, so it risks breaking older readers. Amusingly, older ADE versions would probably do the right thing entirely by accident because of their infamous "wrap at tag boundaries" bug, but....
14. Font sizes in SVG are translated from CSS properties into attributes on the elements to work around a bug in KDP ingest.
15. Fewer -ibooks-* styles and other platform-specific bits in the Kindle CSS.

It is not a small list. For KF8 readers, the EPUB content would probably be survivable, if a little quirky in spots for lack of testing. For KF7 readers, it would basically be a fireball. And as previously noted, several of the KF7 workarounds require tags that would be illegal to ship in an EPUB book, so I can't make the Kindle version be the base version, either.

Bear in mind, of course, that my formatting is pretty complex, coming in just shy of 30k in the main CSS file alone. Not everyone will need or want to go quite that nuts.

Fortunately, that is all automated. I start out with DocBook XML (with a few custom extensions). So there is a single source for both books (and for my print editions); it is just one level of abstraction farther away from the final content.

[DaleDe]

Interesting list. KindleGen would have fixed some of these issues anyway and some could be done with device specific CSS that KindleGen would have used. However, if you have it all automated there is no reason for you to change. Of course KF7 has no support for CSS at all so some conversions are already happening or are you only supporting KF8?

I do know that some users automatically use a different CSS for Kindle than for regular ePub since it is a simple substitution prior to using KindleGen.

Dale

[dgatwood]

Originally Posted by DaleDe

Interesting list. KindleGen would have fixed some of these issues anyway and some could be done with device specific CSS that KindleGen would have used. However, if you have it all automated there is no reason for you to change. Of course KF7 has no support for CSS at all so some conversions are already happening or are you only supporting KF8?

I do know that some users automatically use a different CSS for Kindle than for regular ePub since it is a simple substitution prior to using KindleGen.

I'm not producing the KF7 content myself. That list is what I had to do to my EPUB source just to make the latest (*) version of Kindlegen convert it properly to KF7/KF8. For KF7, Kindlegen converts the CSS into HTML markup, but does it very badly, with a CSS parser that doesn't properly handle selectors containing multiple elements, multiple class names on single elements, lists of selectors applied to a single rule set, etc., resulting in all sorts of joy for those of us who routinely use nontrivial CSS.

Honestly, if Kindlegen offered a way to provide it separate content for KF7 and KF8, I'd do that in a heartbeat, because I'm pretty sure that a few lines of well-written Perl code would do a better job than Kindlegen. After all, I'm emitting the styles myself in Perl code; it doesn't take much longer to have an "if" statement that causes it to emit tags instead.

Of course, if I were producing a general-purpose solution, I'd use a different approach, using a WebKit WebView to render the content, walk the DOM tree, and blow in tags based on the computed styles for each node. It would probably take only double-digit lines of code in total, and it would put Kindlegen to shame by being 100% correct in its interpretation of the CSS every freaking time.

Basically, Amazon felt the need to reinvent the wheel, both by creating their own format (KF8) for no good reason and by creating their own tools (kindlegen) for no good reason. And as usual, when reinventing the wheel, they had to make it different, so they made it square.... I just don't get it.

But I digress.

(*) As of a few months ago

[Notjohn]

Trivial CSS -- yes! If only there were a book or a website devoted to Trivial CSS, the way we have the Children's Table at Thanksgiving! I would be happy there.

[mattmc]

I happened to find this thread on Google when looking for something else...

Originally Posted by dgatwood

Here's a list of differences between my Kindle EPUB and my production EPUB:

...

It is not a small list. For KF8 readers, the EPUB content would probably be survivable, if a little quirky in spots for lack of testing. For KF7 readers, it would basically be a fireball. And as previously noted, several of the KF7 workarounds require tags that would be illegal to ship in an EPUB book, so I can't make the Kindle version be the base version, either.

Wow. That's an impressive list! And scary.

Originally Posted by dgatwood

For KF7, Kindlegen converts the CSS into HTML markup, but does it very badly, with a CSS parser that doesn't properly handle selectors containing multiple elements, multiple class names on single elements, lists of selectors applied to a single rule set, etc., resulting in all sorts of joy for those of us who routinely use nontrivial CSS.

I wasn't aware of that. I'm dealing with CSS from InDesign, and I did see rules with multiple selectors--I guess I need to blow those apart with a script.

Originally Posted by dgatwood

Of course, if I were producing a general-purpose solution, I'd use a different approach, using a WebKit WebView to render the content, walk the DOM tree, and blow in tags based on the computed styles for each node. It would probably take only double-digit lines of code in total, and it would put Kindlegen to shame by being 100% correct in its interpretation of the CSS every freaking time.

One thing I did do, when I was working with iBooks and doing some fancy stuff with popovers, was I had to inline a bunch of styles onto the contents of the <aside> elements. Basically, iBooks was throwing out all the CSS that should have affected the contents of those elements.

I'm already doing all of my processing with NodeJS, and the NPM ecosystem is vast, so I easily came across Juice, which is a library originally designed for inlining CSS styles for emails (which can have just as archaic rendering systems as the original Kindles). It worked great (uses another library called Cheerio, which is a browser-less DOM parser and jQuery re-implementation). Anyway, that sounds like more or less what you were talking about with your WebKit WebView and DOM walking.

...Would you recommend doing style inlining for all of my Kindle files? I didn't want to Juice the whole file, because it's going to bloat it with style="..." attributes on practically every tag, but if it's necessary for KF7, I could make it part of my Kindle workflow...

[dgatwood]

Originally Posted by mattmc

One thing I did do, when I was working with iBooks and doing some fancy stuff with popovers, was I had to inline a bunch of styles onto the contents of the <aside> elements. Basically, iBooks was throwing out all the CSS that should have affected the contents of those elements.

You sure it wasn't just a specificity issue? The iBooks stylesheets throw a lot of stuff into universal selectors like this:

:root[__ibooks_internal_theme*="Night"] * { ... }

Which has the same precedence as

element.class element

or (0,0,1,2). If you define

element.class { ... }

then IIRC, it has lower specificity because it has the same number of classes, but fewer elements in the selector. Your inline style has specificity (1,0,0,0), which beats everything except !important (unless you used !important, in which case it beats everything previously declared).

Try this:

body element.class, element.class * { ... }

which gives you the same specificity as the built-in rules, and by virtue of being later, wins. (The universal selector in the second part is so that it applies to all elements inside. This overrides the universal selector for those. Do not include that for styles that set relative font sizes, obviously, or for box model stuff.)

[mattmc]

Originally Posted by dgatwood

You sure it wasn't just a specificity issue?

Wow. You just opened my eyes to the concept of CSS specificity. Before today, I'd just had vague ideas about certain rules, that were more specific than others, trumping. Now I actually know what's going on. Thanks VM

It actually totally could have been a specificity issue. I already have Juice working full-on, but it may be worth revisiting that.

But then, how about my Kindle question? Worth inlining styles for that purpose, if not for iBooks?

[Turtle91]

Is there an idiot's guide to the specificity rules?? I knew they were out there as well, but don't totally understand them...