Cleaning ePubs: automatically, fast and with as many generic rules as possible

[Hitch]

Originally Posted by Elfwreck

Because they got started with stoneage rendering tools when those were the most modern and efficient ones available, and they work tolerably well (at least as far as "produce ebooks that the general public is willing to pay for"); they have no incentive to follow the cutting edge of the industry and re-tool their production process with new software every six months.

Not to mention, when they do re-tool, and revamp, it makes a boatload more work for the bookmakers, who then have to change direction, yet again. There's absolutely nothing to be gained by always being "cutting edge" on eBook production, I have to say it. We try to do it, and I'm always getting metaphysical paper cuts. For example (I use this one a lot), when Nook decided "hyphenation is good," and wreaked havoc with headers. Or when iBooks decided that centering was BAD, and you had to span everything to get it to center (just FYI, for those of you who think all spans are "bad"). Or when Kindle upgraded its KDP 3 weeks ago, and wreaked havoc with some subset of fonts yet to be defined.

I mean, as Elfwreck says, what the hell is to be gained, in telling a story, by deciding whether markdown or html or XML or the Big Blue Weenie markup language works best behind the scenes?

Standardization would meant total market dominance by whoever creates the standard. All manufacturers--of ebooks and devices--are invested in that creator *not* being their competitor. That means deliberately not doing things the way the competitors do it, regardless of how well that works.

Yes. This = Apple.

If one method had a sharp, noticeable, commercial advantage over the others, it would catch on. (See: dominance of epub; decline of LIT and LRF.) However, the general public does not care whether chapter headers are "properly" formatted H1, or <span> sections with specific formatting details. They care that chapter headers start on a new page and are bold. (Sometimes. Sometimes they don't even care about that.)

"Clean" ebook files are for formatting fanatics; readers, especially paying readers, aren't seeing the code. Readers care about whether the book "looks nice," but that's a subjective standard; there's no universal agreement on the best basic look. (I prefer indented paragraphs with no space between. Many readers prefer flush-left with spaces between. I like slab serifs; some people like serifs; some prefer sans. And so on.)

There is no universal method because there is no way that works "best," especially with many specific requirements in play for niche markets that don't matter for the larger crowd. And there's no cooperation to create a universal system, because the major developers are competitors.

Exactly. I honestly don't get this discussion, the more I think about it. I could almost understand a discussion, in print books, if printers started using newsprint, so that your hands were filthy after ten pages...or using such thin paper that pages tore when you turned them...but this is storytelling, or conveying useful information. They're BOOKS. They're not intended to be technological miracles, or cutting-edge. I really and truly don't understand why, ibu, you give two tosses about how the underlying bookmaker made the books. If you care that much, take it, crack it open and re-code it. {shrug}. It seems really...I don't know. I cannot imagine this sort of discussion 15 years ago about a print book ("OMG! Random House used an archaic printing machine, found only in the deep jungles of the Phillipines, to print this book! We have to change that, because it's wrong!"), and I genuinely don't understand the issue here.

I think I'm going with Diap's interpretation here...there's obviously some deep philosophical discussion going on that I'm simply too shallow to understand.

It's a BOOK.

Hitch

[Elfwreck]

Originally Posted by Hitch

It seems really...I don't know. I cannot imagine this sort of discussion 15 years ago about a print book ("OMG! Random House used an archaic printing machine, found only in the deep jungles of the Phillipines, to print this book! We have to change that, because it's wrong!"), and I genuinely don't understand the issue here.

If one is in the habit of cracking them open to fix things (like em-based margins that grow as you increase the font size), or you want to combine several into one ebook to make an omnibus, the lack of standardization is horribly frustrating.

Learning how to make ebooks is difficult--people make badly formatted ebooks (by which I mean: noticeable to readers; no bold in chapter headers, or random italicized paragraphs, etc.) because if they stray from their tutorial's exact system and borrow advice from something else (perhaps because their tutorial didn't cover the details that the book needs), they run the risk of creating an unreadable mess.

And of course, making ebooks according to one tutorial doesn't mean the ebook will work on the device you want, or be sellable in the store you want.

So, yeah, lack of standardization is a problem. It's just not a problem that matters to many people--and for them, it matters in different ways--so there aren't going to be any simple, over-arching solutions anytime soon.

[mrmikel]

Maybe there are secret messages encoded in the HTML. If it is all ems instead of all bolds, it means there will be an attack at midnight! (I have been reading too many WWII books lately.)

[Hitch]

Originally Posted by Elfwreck

If one is in the habit of cracking them open to fix things (like em-based margins that grow as you increase the font size), or you want to combine several into one ebook to make an omnibus, the lack of standardization is horribly frustrating.

Learning how to make ebooks is difficult--people make badly formatted ebooks (by which I mean: noticeable to readers; no bold in chapter headers, or random italicized paragraphs, etc.) because if they stray from their tutorial's exact system and borrow advice from something else (perhaps because their tutorial didn't cover the details that the book needs), they run the risk of creating an unreadable mess.

And of course, making ebooks according to one tutorial doesn't mean the ebook will work on the device you want, or be sellable in the store you want.

So, yeah, lack of standardization is a problem. It's just not a problem that matters to many people--and for them, it matters in different ways--so there aren't going to be any simple, over-arching solutions anytime soon.

Elfwreck:

I am absolutely the last person you'll ever get an argument from on that score (that lack of standardization is a problem). You've all seen me bitch about it here, on too many occasions.

I guess my viewpoint, though, coming from very jaded eyes, is that given Apple's behavior and Amazon's, it's not going to change. Apple drove the last ePUB3 bus; we all know that. IMHO, the "NCX is dying" aspect of ePUB3 was a stupid move, and the whole push for multimedia, over standardization of content is...I won't type here the word that describes how I think about that. If Apple hadn't utterly (again, IMHO) screwed the standardization issue by pushing ePUB3 in that direction (oooooh, flashy pretty thingies!), instead of focusing on simplifying and conforming ePUB2++, I would have retained some hope, but once I saw the ePUB3 final specs, I gave up on the idea.

I don't think Amazon will adapt to ePUB3's direction, not really...I don't see either side as being willing to give ground, and I think that leaves folks like me stuck in the "make two books" rut (now, the make 3 books rut, if you count Smashwords' idiotic requirements, and maybe 4, if you throw in a Kobo version). That's all I'm saying.

And I don't really think that standardization was the original point of the OP, per se; but...I guess we all got here from there. My original take was that he had certain impressions about what constituted "good" coding versus "bad," and while I think most of us can agree that embedding 4 spans to do one thing is silliness, and not "neat," y'know, as I said earlier, ebooks are sausage, and there's a lot of ways to get to a bolded chapter header.

And, of course, most of us here would likely agree on a "badly-made ebook" when we saw it; probably you, me, mrmikel, Diap, even our resident curmudgeon (excepting TOC discussions, LOL), etc. We all know crap when we see it. And if you get a lot of crap books that you have to rip apart to make your reading experience better, I can understand that (margins, etc.) without hesitation. But I do think that a lot of the OTHER--use em versus a spanned class, etc., is a bit of tail-wagging-the-dog. But, hey, that's why we have Vanilla, Chocolate, and, of course, Coffee Heath-Bar Crunch ice cream, right?

@mrmikel: One em if by land, one span if by sea....


Hitch

[mrmikel]

I think most people who work on epubs by hand dislike what calibre produces, mostly because it is hard to edit. BUT, the calibre output works, it doesn't cause the reader to respond slowly, so from the reader's point of view, what's wrong with it. It is not like something mechanical which is going to require further work, so if the epub put-put is powered by rubber bands, it really doesn't make any difference. Still it is a bit like Rodin's The Thinker being made of papier-mache.

Auntie em, she's melting! Toto, I don't think we're in Kansas any more. Bizzaroworld is more like it!

[Agama]

Originally Posted by PeterT

@Agama: Where can we find your plugin?

It's not online anywhere because it's probably too specific to my own workflow with Markdown sources.

As well as the features I've already alluded to it does a few other things: (1) All the _split_nnn.html files get renamed; (2) an externally prepared cover image and logo are copied into the book; (3) page_styles.css is removed from the xhtml files and an externally prepared stylesheet.css is copied into the book; (4) the long string IDs in toc.ncx are replaced with simple IDs of "P" followed by the playorder number; (5) in the xhtml files it replaces "Unknown" in <title>Unknown</title> with the actual book title.

Attached image shows list of files in a typical ePub after processing.

[Agama]

Originally Posted by ibu

I gave an example how it could be markupped without an "extra class", just with em. And I explained, why I think, that it's the better markup for that purpose.

When it's true, that many devices cannot interprete a simple descendant selector like "h1 em" than I would suggest an
<h1>Chapter <em class="chapter-number">3</h1>

But may be I missunderstood you.

I meant that by introducing <em> you are implying that the text is emphasised which may not be the case; you simply do not know without reading the book. Using <span class="number"> it is possible that the style is { font-szie:0.75em; font-style:normal; font-weight:normal; color:#cccccc } which is not emphasis.

So whilst introducing extra tags could allow contextual styling, it would need to be done with care.

Imagine an ePub reader which had the option to read the book out loud - introducing your <em> tag in place of <span> would cause the reading voice to emphasise the word.

You would therefore need to understand the book's stylesheet/s before making structural changes to the xhtml files and this would be quite difficult for a simple plugin.

[ibu]

@Agama
FULLACK. Structural changes has to be done with care.

I argued carefully in this specific case, why I think it is reasonable to markup the number with em.
The number is, relative to the generic classifier "Chapter", more important.
IMHO.

But of course, an editor of such an ebook could decide different.
He may just add a span (which has by definition of the W3C no specific semantic), with a "speaking class", in this case "chapter-number" seems a good one.

[Jellby]

Originally Posted by ibu

I argued carefully in this specific case, why I think it is reasonable to markup the number with em.
The number is, relative to the generic classifier "Chapter", more important.
IMHO.

Well... then something like:

<h1>Chapter 4
A man called The Oak and his fiancée</h1>

Where you might want (or maybe not) to give a special style to the number, you might want (or maybe not) to force the title to start at a new line, you might want (or maybe not) to have "The Oak" and "fiancée" in italics (for different reasons), and you might want to hide the word "Chapter" completely.

Of course, you can get imaginative with selector rules, but I believe this is clearer, and more flexible:

<h1><span "name">Chapter</span> <span class="number>4</span>
<span class="title">A man called <em>The Oak</em> and his <span lang="fr">fiancée</span></span></h1>

span.name { display: none; } /* If you don't want the word "Chapter" */
span.number { font-size: 200%; } /* Extra large chapter number */
span.title { display: block; } /* Title in new line (no hard-coded <br/>) */
:lang(fr) { font-style: italic; } /* Foreign words in italic, normally */
/* (I'm not 100% sure of this syntax now, but it doesn't work in ADE anyway) */
h1 em, h1 :lang(fr) { font-style: normal; } /* No italics in chapter titles, please */

Sometimes spans, divs and classes galore are just a sign of sloppy conversion, but sometimes it's simpler to use them, rather than trying to find convoluted selection rules that might bite you back later.

[ibu]

@Jellby
I fully agree with your thoughts about your new example.

If all the ereaders would support to generate content via CSS I would suggest to generate the signifier "chapter" and the number via CSS in your example.

Result:

<h1>A man called <em>The Oak</em> and his <span lang="fr-FR">fiancée</span</h1>

But of course, they do not support.

My own suggestion for real life ereaders would vary slightly from yours:

<h1><span class="chapter-prefix><span class="chapter-signifier">Chapter </span><span class="chapter-number>4 </span></span>A man called <em>The Oak</em> and his <span lang="fr-FR">fiancée</span></h1>

I would omit the span around "A man called The Oak and his fiancée".
And I would enclose the spaces behind "chapter-prefix" and "chapter-number" in the span, to beware the option to fade it out completely with CSS, if the layouter wishes that.
And a very sophisticated tiny thing:
May be it's good to declare the language more detailed, because there are several regions, where they speak French.

Thinking about the best markup is plain fun

[Jellby]

Originally Posted by ibu

If the ereader would be able to generate content via CSS I would suggest to generate the term "chapter" and the number via CSS in your example.

Indeed, but counters are out of the ePub 2.0.1 spec. And if you want to use the full "Chapter # <Title>" in the TOC, having that explicitly written makes it easier to generate the TOC with Sigil, for instance. Anyway, this was only an example of how things could be, not optimized by any means, don't take it to the letter

I would omit the span around "A man called The Oak and his fiancée".
And I would enclose the spaces behind "chapter-prefix" and "chapter-number" in the span, to beware the option to fade it out completely with CSS, if the layouter wishes that.

Fair enough, different approaches with the <span>ning, and agree about the spaces, I intended to mention that you should be careful with spaces if you are planning on hiding content.

And a very sophisticated tiny thing:
May be it's good to declare the language more detailed, because there are several regions, where they speak French.

But I don't think it makes a difference in this case, especially since it's a standard French word imported into English (some might even argue that it should not be italicized at all). If we don't have any further hint of the specific variant the word belongs to, I think it's better to keep it generic with "fr", even if we know the author is from Montréal, "The Oak" from Brussels and the girl from Geneva

[ibu]

@Jellby

About the TOC:
ePubs contain a lot redundant information.

Life of epub creators could be so much easier, when ereader would be able to create TOCs based on Hn on their own.
And title pages based on the metadata.
And displaying cover based simply on the existance of cover.jpg.
And ...



Something else:
Do you fill the title element or do you just use a H1?

[Jellby]

Originally Posted by ibu

Do you fill the title element or do you just use a H1?

I use just <h1> (or <h2>), but I don't use Sigil, and create the TOC by hand

[ibu]

@Jellby
May I ask, which tool do you prefer to Sigil (and why in a short sentence)?

And why do you create the TOC by hand?

[Jellby]

Originally Posted by ibu

@Jellby
May I ask, which tool do you prefer to Sigil (and why in a short sentence)?

Just good ol' vim. I like full control on what is going into the ePub, so most WYSIWYGgy tools are out of question. There was no Sigil when I started, and I guess it has come to a point where it would really be a help for my ebook creation...

I'm still thinking that some day I should compile or grab some binary build of Sigil and use it, but laziness and inertia are a dangerous combination

And why do you create the TOC by hand?

Same as above. Plus creating it by hand allows me to control exactly what is going where, without having to change the main text (except for adding ids).