Cleaning ePubs: automatically, fast and with as many generic rules as possible

[ibu]

Originally Posted by Agama

I'm not so sure. If NCX entries could only point to Hn tags then some flexibility would be lost. There are times when it can be appropriate to link to a paragraph which does not start with a header.

I have no real life example in my mind in which such a link - in a TOC - makes sense.
But may be I overlook situations.
Do you have an example please?

Originally Posted by Agama

Have you done any web searches for HTML tidying tools. Maybe there is something out there with some flexible user configuration that could help you.

I'm still searching for a tool which lists all used HTML elements of all HTML files inside a directory.

Like this:

p
p class="foo"
p id="f1"
span
span class="bar"
span style="ugly:yes"
h1
h1 id="z1"
...

For an analysation and overview of the complete text files of an given epub.

But until now I didn't have success.

[Toxaris]

Did you check the reports in sigil? Those can give more insight, but not all I think.

[ibu]

Originally Posted by Toxaris

Did you check the reports in sigil? Those can give more insight, but not all I think.

Yes, I checked them.

But they do not list enough.
No id attributes.
No style attributes.
No naked elements.

Not enough for an analysis.

[Jellby]

Originally Posted by ibu

I have no real life example in my mind in which such a link - in a TOC - makes sense.
But may be I overlook situations.
Do you have an example please?

A link to a map or a letter that appears in the flow of a chapter. You might want to include that in the TOC, but it makes no sense to add a sectioning command (read <h#>) where the map or letter appears.

An example: Thror's map in "The Hobbit". Although I guess pretty much all editions show the map (if at all) at the beginning of the book, I believe it could be better to show it only when it appears in the story, and yet have a link to it in the TOC.

[ibu]

@Jellby
Thanks for the example.
I'm not sure, if it fits my own taste of a TOC, but I respect of course, that such wishes for TOCs exist.
I've never saw it in book.

May be an utopian spec could react to such special cases with a special treatment of the specific HTML element, which should appear inside the TOC.

It could get a specific class. And the content of the TOC entry could be placed e.g. in a title attribute.

<p id="thorsmap" class="toc" title="Thror's map">...</p>

In my view it highly valuable, when standard cases do not need any extra work/files from an editor to offer access to a nice TOC.

Did I mention, that I dream of an utopian web standard, where the complete navigation is generated by the user agent (browser) and not anymore manually in thousands of styles by the webauthor.

Navigation has to be under full control of the user - IMHO.
The web author should just provide structure.

In the past, Firefox offered a toolbar which interpretes some meta elements for navigation. But Mozilla kicked it.

There's a wonderful tiny project
http://www.standard-sitemap.org/fire...tension.en.php
which allows to feel how nice and consistent web navigation could be - in a better world

But the most user are used to bad information architecture. Resign, ignorance, silent suffering ...
There are many causes that better usability will never have a chance.

[Hitch]

Originally Posted by Jellby

A link to a map or a letter that appears in the flow of a chapter. You might want to include that in the TOC, but it makes no sense to add a sectioning command (read <h#>) where the map or letter appears.

An example: Thror's map in "The Hobbit". Although I guess pretty much all editions show the map (if at all) at the beginning of the book, I believe it could be better to show it only when it appears in the story, and yet have a link to it in the TOC.

Again, Jellby beat me to it, but: a list of illustrations. A list of images (we did a book that was about a famous modern painter; a large part of the navigational elements were links with captioned comments about the individual works); other such elements. As Jellby said, maps. Sub-items which are not structural (for example, charts, lists, diagrams).

Lots of "stuff" that could go in an NCX, but isn't necessarily structural. I am big on "all headings should be structural, not stylistic," but, not all NCX or TOC entries need be structural. (Ah, I feel myself sliding into Wonderland here. The Cheshire Cat will be here soon.)

Hitch

[ibu]

@Hitch
You convinced me with your arguments about a list of illustrations, charts, diagrams.
Thanks.

I cut the suggestion "no NCX" out of the utopian spec.

And I agree fully, that Hns should not be missused for stylistic purposes. That was never my intention.

May be I didn't had the lists of diagrams, charts, pictures in my mind, because such lists are typical for reference books and not for fiction.

And because the software of my Kobo Glo is completly unqualified for an efficient navigation, I forgot, that there are books with more complex structure.

Hopefully, future versions of eReaders will offer highly efficient (multi finger) gestures and shortcuts to browse throught huge information spaces.
With the same ease as a thumb in a paper book.
May be in 10 years

[Toxaris]

Originally Posted by ibu

Yes, I checked them.

But they do not list enough.
No id attributes.
No style attributes.
No naked elements.

Not enough for an analysis.

When I am back from vacation, I might look into it. No promise though!

[ibu]

Originally Posted by Toxaris

When I am back from vacation, I might look into it.

Thanks
Very nice from you.

[stevelitt]

Originally Posted by ibu

Typical ePub XHTML has an extremly poor quality.

Inline-Styles.
DIVs.
SPANs.
Redundant class or id attributes.
Lack of semantic markup.
...

In other words: an horror for a friends of high quality semantic html.

Thanks.

What's wrong with divs and spans? I have span.code and span.emph and span.definition. I have div.titlepage, which sets everything bold and center, and then inside that I have p.title, p.author, etc, each with its own size. You can't replace a div with a p , because p can't contain other paragraphs.

SteveT

[Hitch]

Originally Posted by stevelitt

What's wrong with divs and spans? I have span.code and span.emph and span.definition. I have div.titlepage, which sets everything bold and center, and then inside that I have p.title, p.author, etc, each with its own size. You can't replace a div with a p , because p can't contain other paragraphs.

SteveT

Steve:

There's nothing "wrong" with divs or spans used where appropriate. What we were all discussing here is the sloppy use of them in lieu of more appropriate coding, for example, an inline style div instead of using em or strong, or bold or italic. Most epubs ought to have as much of the styling defined in the CSS as possible, as opposed to inline, on the fly styles.

For example, I stupidly (I've told this story here before) once agreed to take on and "fix" an ePUB that was purportedly completed, to rebuild into a MOBI. Some 66 spans later, named sequentially, "char-override style 1" through "char-override-style-66," I regretted it enormously. Or spans to achieve a font size (e.g., font-size 80%, or some such) instead of a named class. Once someone gets in the habit of "on the fly" styling spans, it's really no different than the ad hoc styling that we all get to see far too often from ill-created Word files, or worse, the output from something like Adobe Acrobat Pro to an "exported Word file," which is usually pretty catastrophic. ;-)

And yes, divs are for dividing up sections of screen/pages and to contain other elements. But spans tend to be the output children of programs that are intended for print, and thus, make for sloppier, less discernible coding. That's all.

Hitch

[stevelitt]

Originally Posted by Hitch

Steve:

There's nothing "wrong" with divs or spans used where appropriate. What we were all discussing here is the sloppy use of them in lieu of more appropriate coding, for example, an inline style div instead of using em or strong, or bold or italic. Most epubs ought to have as much of the styling defined in the CSS as possible, as opposed to inline, on the fly styles.

For example, I stupidly (I've told this story here before) once agreed to take on and "fix" an ePUB that was purportedly completed, to rebuild into a MOBI. Some 66 spans later, named sequentially, "char-override style 1" through "char-override-style-66," I regretted it enormously. Or spans to achieve a font size (e.g., font-size 80%, or some such) instead of a named class. Once someone gets in the habit of "on the fly" styling spans, it's really no different than the ad hoc styling that we all get to see far too often from ill-created Word files, or worse, the output from something like Adobe Acrobat Pro to an "exported Word file," which is usually pretty catastrophic. ;-)

And yes, divs are for dividing up sections of screen/pages and to contain other elements. But spans tend to be the output children of programs that are intended for print, and thus, make for sloppier, less discernible coding. That's all.

Hitch

Thanks Hitch,

First, I meant to interleave-post, discussing each of your points below where you made it in your quoted text, but I didn't know how. If someone can tell me how, I'll do that next time. And now back to the topic...

Now I understand. I use span and div to define styles that assign an appearance to text and paragraphs of a specific purpose. Ibu's point was that some people, and some brain-dead converters, use them to fingerpaint appearances directly to text, with no context as to the text's purpose.

As an aside, I think that what Ibu called "high quality semantic html" is similar to what I call "Styles Based Authoring", meaning that no appearance is ever applied directly to text: The only thing applied to text is a style with a one to one correspondence to the purpose of the text (p.author, div.appendix, but not p.rightjust). Personally, I use h1, h2, h3 for Part, Chapter, Section. I think that was always the intent of the h? tags.

I have several pro-styles-based arguments in my new ePub construction tutorial at http://www.troubleshooters.com/ebook..._demystify.htm .

I feel your pain on that fingerpainted Xhtml with 66 spans. I'm a big fan of LyX, but by the time I massage LyX's Xhtml output to make it sane for ePub, I'm ready to spend some time in a mental hospitial. And the blame isn't entirely that of the converters: If the document's author fingerpainted his source doc, whether it was MS Word, LibreOffice, LyX, Bluefish authored Xhtml, Docbook, or something else, then it's going to mess up the ePub, or else your appearances will disappear.

In my new tutorial, several places I urge the reader never to apply an appearance directly to text, and never to make a style whose sole purpose is an appearance, and never to name a style after an appearance. From my perspective, if the source doc is styles-based, all you need is to write a program to page-break appropriately while ring-tossing pages into the Manifest, Spine, Guide, and NCX file, and then make sure every style has appropriate an CSS style. Yeah, that's an oversimplification, but you know what I mean.

Thanks for the clarification.

SteveT

[stevelitt]

Originally Posted by Hitch

Steve:

For example, I stupidly (I've told this story here before) once agreed to take on and "fix" an ePUB that was purportedly completed, to rebuild into a MOBI. Some 66 spans later, named sequentially, "char-override style 1" through "char-override-style-66," I regretted it enormously. Or spans to achieve a font size (e.g., font-size 80%, or some such) instead of a named class. Once someone gets in the habit of "on the fly" styling spans, it's really no different than the ad hoc styling that we all get to see far too often from ill-created Word files, or worse, the output from something like Adobe Acrobat Pro to an "exported Word file," which is usually pretty catastrophic. ;-)
Hitch

Hi Hitch,

As long as I have your ear, I need your opinion. I'm writing my ePubs in Xhtml, with Bluefish. I write the whole book as one Xhtml file, and then my converter (which isn't yet complete) splits it into files, making Manifest, Spine, NCX, and Guide as necessary entries for each page. The converter has a YAML config file that defines relationship between <h?> and Part, Chapter, Section, etc, and which levels get their own page. I typically page break on Part and Chapter, but it's configurable.

Here's my question: Some non-hierarchical entities, such as Acknowledgements, Apendix, etc, need their own pages, so there needs to be a way, in the full book Xhtml source, to indicate a new page. I *could* make things like the appendix title an <h?> tag of a special class indicating it's not in the hierarchy and therefore don't call it "Chapter" and don't increment the Chapter number, but for various reasons, that would be messy.

What I was thinking of was having a <div class=pagebreak/>, which would tell the converter to break here, without implying this is a Part, chapter, etc. In the multichapter source file it would look like a bunch of dashes, but in the individual ePub Xhtml file it would look like nothing at all, with margins of 0, and would gain an id value of "toppage". Yeah, it's a kludge, but it's a darned convenient kludge, both for the book author, and for the guy who programs the converter (me in both cases).

What do you think?

Thanks,

SteveT

[Hitch]

Originally Posted by stevelitt

Hi Hitch,

As long as I have your ear, I need your opinion. I'm writing my ePubs in Xhtml, with Bluefish. I write the whole book as one Xhtml file, and then my converter (which isn't yet complete) splits it into files, making Manifest, Spine, NCX, and Guide as necessary entries for each page. The converter has a YAML config file that defines relationship between <h?> and Part, Chapter, Section, etc, and which levels get their own page. I typically page break on Part and Chapter, but it's configurable.

Here's my question: Some non-hierarchical entities, such as Acknowledgements, Apendix, etc, need their own pages, so there needs to be a way, in the full book Xhtml source, to indicate a new page. I *could* make things like the appendix title an <h?> tag of a special class indicating it's not in the hierarchy and therefore don't call it "Chapter" and don't increment the Chapter number, but for various reasons, that would be messy.

What I was thinking of was having a <div class=pagebreak/>, which would tell the converter to break here, without implying this is a Part, chapter, etc. In the multichapter source file it would look like a bunch of dashes, but in the individual ePub Xhtml file it would look like nothing at all, with margins of 0, and would gain an id value of "toppage". Yeah, it's a kludge, but it's a darned convenient kludge, both for the book author, and for the guy who programs the converter (me in both cases).

What do you think?

Thanks,

SteveT

Hi, Steve:

I'd recommend that you look at Sigil, which works remarkably like this, and at Kovid's new Calibre add-in/add-on, etc. Sigil uses a pagebreak tag which subsequently separates the file.

I think you'll find that the frontmatter-type pages work best when they are in their own XHTML file, just like the chapters. This is also fairly crucial if you intend to use this same ePUB file for MOBI conversion, as Mobi won't give you any top-margin for chapter heads, etc., without it being a new file.

Check those out; those are better sources of information than I, in this particular matter, I'd say.

Hitch

[stevelitt]

Originally Posted by Hitch

Hi, Steve:

I'd recommend that you look at Sigil, which works remarkably like this, and at Kovid's new Calibre add-in/add-on, etc. Sigil uses a pagebreak tag which subsequently separates the file.

I think you'll find that the frontmatter-type pages work best when they are in their own XHTML file, just like the chapters. This is also fairly crucial if you intend to use this same ePUB file for MOBI conversion, as Mobi won't give you any top-margin for chapter heads, etc., without it being a new file.

Check those out; those are better sources of information than I, in this particular matter, I'd say.

Hitch

Thanks Hitch,

You were more help than you think: The fact that you didn't get grossed out and say "Your top of page sentinels are a horrible idea" gives me a lot of information.

Five minutes ago, I used Sigil to parse apart a multi-h1 document, breaking it apart manually with "break at cursor". If Sigil uses a pagebreak tag, it doesn't survive into the Xhtml of the chapters, so I can't deduce what Sigil does in order to do the same thing in my program.

My current intent is to have my <div class="toppage"/> pagebreak tags get put into the actual Xhtml pages, with my parser inserting ids so everything can find them. That way the parser needn't mess with the IDs of my H1 or other tags.

One beauty of my current intent is that, although my parser will automatically insert these tags before every h1 or h1 and h2 (depending on config), if the author manually inserts one of these tags before his dedication, for instance, it will page break there, so the author has complete control. This addresses your admonition that frontmatter-type pages work best if they're in their own Xhtml file.

I think the time has come for me to quit asking questions and just try it and see if it works. Thanks for all your help, and I'll report back with any results.

SteveT