Cleaning ePubs: automatically, fast and with as many generic rules as possible

[ibu]

@Jellby
Thanks for telling your motives.

Until now I like Sigil (But I'm a newbie in epub at all and working at it since a week

I miss some control about the code generation.

E.g. the pattern of the IDs:
It's id="FILENAME.FILEXTENSION"
instead of just
id="FILENAME"

Or the UUID:
It is automatically added even if there's an ISBN in the metadata.

Or the pattern of the ISBN:
I'm looking for an automatical transformation from an (unformated) ISBN10 into a formated ISBN13

[Agama]

Originally Posted by ibu

About the TOC: ePubs contain a lot redundant information.
Life of epub creators could be so much easier, when ereader would be able to create TOCs based on Hn on their own.

Although this may be considered redundant information it does enable a reader to access the toc quickly. If a toc had to be built on-the-fly when a book is opened then all the xhtml files would have to be parsed - and an assumption made about which tags to include. For my ePubs this would be H2 and H3 only but another producer may use H1..H4 or another producer may use <span class="toc1"> etc., etc.

Calibre allows you to define expressions which determine those tags which should contribute to the toc and their level in the toc, so this gives quite a lot of control without resorting to hand-coding.

Not sure what level of control Sigil gives you as I don't use it.

[ibu]

Originally Posted by Agama

Although this may be considered redundant information it does enable a reader to access the toc quickly.

High speed is only necessary while using a book. When a reader needs some seconds to charge his internal cache files: no problem - in my rating.

Originally Posted by Agama

If a toc had to be built on-the-fly when a book is opened then all the xhtml files would have to be parsed - and an assumption made about which tags to include. For my ePubs this would be H2 and H3 only but another producer may use H1..H4 or another producer may use <span class="toc1"> etc., etc.

We are talking about an utopy of an epub standard.
In my utopy using H1 for the highest level is obligatory. It is not anymore a decision of the editor.
And "span class="toc1" is simply not a valid markup for a heading.

But well, I don't think there are many readers in this forum, who like to talk about "how a better epub spec could be".
That's no offense at all. But in my experience in forums in general, people prefer to talk about the reality

[DiapDealer]

Originally Posted by ibu

But well, I don't think there are many readers in this forum, who like to talk about "how a better epub spec could be".
That's no offense at all. But in my experience in forums in general, people prefer to talk about the reality

But only because you can't really read, edit, sell or share imaginary epubs built to a "better" spec.

Face it: the current ePub spec is largely ignored whenever it suits specific vendors of devices/apps that must read the ePubs. That wouldn't change with your new utopian ePub spec. So what's the point of building (or even imagining) a better--more concise-- imaginary ePub spec that would be adhered to (or supported/implemented in toto) by almost no one?

Don't get me wrong ... idealism can be fun. There's nothing wrong with dreaming about what could be. It's just that most of us here (with intimate knowledge of how ePub works--practically--on a range of devices/apps) have already gone through that idealistic phase long ago ... and had it crushed out of us by reality.

[Rizla]

It doesn't sound that hard to do. A parser using lots of string manipulation and outputing a new and stripped version of the file would do it. Aspects of the decision-making about what to strip would probably require user-intervention. But no, it doesn't sound that hard.

[Hitch]

Originally Posted by DiapDealer

But only because you can't really read, edit, sell or share imaginary epubs built to a "better" spec.

Face it: the current ePub spec is largely ignored whenever it suits specific vendors of devices/apps that must read the ePubs. That wouldn't change with your new utopian ePub spec. So what's the point of building (or even imagining) a better--more concise-- imaginary ePub spec that would be adhered to (or supported/implemented in toto) by almost no one?

Don't get me wrong ... idealism can be fun. There's nothing wrong with dreaming about what could be. It's just that most of us here (with intimate knowledge of how ePub works--practically--on a range of devices/apps) have already gone through that idealistic phase long ago ... and had it crushed out of us by reality.

I think that the people who care about quality ePUBs are already making them. Those that don't--those that simply want to churn as many ePUBs and MOBIs as possible through, in the shortest amount of time possible, for the highest profit margin, don't and won't. 99% of all readers certainly don't care if a TOC has a spanned paragraph entry versus a header class; they want the content, not the code behind the content.

There already are ePUB specifications, and there already are "good" HTML coding versus "bad" HTML coding specifications. Anyone who wants to make a higher-quality ePUB can do so--certainly, no one is stopping them. Anyone who wants to can code "perfect" ePUBs or ePUBs that meet their own "should" criteria...{shrug}. When it comes to many of the original "shoulds," as I said some posts ago, everyone's going to have their own view on that, as there are, again, multiple roads to Damascus. While structural issues have more basis in "should," (so as not to confuse what something is versus what it looks like), all the rest is driven by how someone learns; what they like, versus what they don't; what program they're using (someone using INDD with an export plug-in is going to be far, far likelier to use spans for stylistic choices than someone using Sigil, for example), and the like.

{shrug}. And what's the point of a set of "rules" or "shoulds" when it's likely that the books we make will have to be altered to work on retail devices, anyway? As we discussed some posts back, as merely one example, all those extra spans in an ePUB for iBooks are to make centering and other aspects work, for the first-gen iPad. For a 1st-Gen iPad, you still can't float images and text at the top of a screen without the text overwriting the images, and Apple can't be bothered to issue a fix, because they have a positive animus about supporting older tech. Like I said, don't mean to sound like a buzz-kill...all these woulda's and shoulda's...but it's a bit difficult when nobody else is playing along, particularly the Elephants.

Just my $.02. Again.

Hitch

[DiapDealer]

Originally Posted by Hitch

{shrug}. And what's the point of a set of "rules" or "shoulds" when it's likely that the books we make will have to be altered to work on retail devices, anyway?

Ed-zachary.

[Agama]

Originally Posted by ibu

And "span class="toc1" is simply not a valid markup for a heading.

Agreed, but it is valid markup for a TOC entry.

TOC entries do not have to be restricted to heading tags. Although Hn tags probably make most sense for the TOC this is not an ePub requirement, except perhaps in the utopian ePub.

[ibu]

Originally Posted by Agama

TOC entries do not have to be restricted to heading tags. Although Hn tags probably make most sense for the TOC this is not an ePub requirement, except perhaps in the utopian ePub.

Exactly.
I think it's a mistake of the actual spec, kind of cowardice, that it's not more strict.

[Elfwreck]

Originally Posted by ibu

Exactly.
I think it's a mistake of the actual spec, kind of cowardice, that it's not more strict.

How would "TOC entry" be defined in a way that would make one type of markup for it mandatory? "In any list of links to other places within the same epub, Hn headings must be used?"

What's the objectively-identifiable difference between "linked list of chapter subsections" and "ToC links?" Who decides what makes a "real" table of contents instead of an internal list of links, which could be in standard <p> format?

And before you get to "the TOC comes at the beginning," how would that apply to omnibus documents containing several books?

(For that matter... how would you identify "Things That Must Be Marked As Headers?" Some people throw plain text into epub with no markup. Would you declare that any text that's bolded and centered must use an <H> tag?)

[Hitch]

Originally Posted by ibu

Exactly.
I think it's a mistake of the actual spec, kind of cowardice, that it's not more strict.

Y'know...

With all due respect, using header tags for the actual TOC entries makes not one iota of sense at all. Using header tags to generate NCX entries--that makes sense. Using header tags in the actual ePUB, to define struture--this makes sense. Using header tags, for nothing other than what is a link? Why does this make sense?

An html toc is nothing but a page full of text and links. Using header tags for it, in fact, should be contraindicated.

I mean, if we're going all Utopian here. There's not one damn thing "structural" about a page full of text links. Just sayin'.

Ooops...I see that Elfwreck beat me to it. ;-)

Hitch

[ibu]

Originally Posted by Hitch

Using header tags to generate NCX entries--that makes sense. Using header tags in the actual ePUB, to define struture--this makes sense.

That's what I meant of course.

Please look into my previous postings. I wrote "...based on headers".

[ibu]

Originally Posted by Elfwreck

(For that matter... how would you identify "Things That Must Be Marked As Headers?" Some people throw plain text into epub with no markup. Would you declare that any text that's bolded and centered must use an <H> tag?)

I think you missunderstand my intention of the utopian spec regarding "offering a TOC for a reader".

I just plead for obligatory Hns in the text files for headings.
No *.ncx at all.

When an editor wants, that his readers get a TOC, he has to markup the headings in his textfiles.
When the editors thinks "I don't care about a TOC", than he markups how it gives him private joy. But than his readers will get no TOC.
It's very simple with strict specs.

[Toxaris]

It is in the specs, iirc, that on the devices the ncx could/should be used as a TOC.

[Agama]

Originally Posted by ibu

Exactly.
I think it's a mistake of the actual spec, kind of cowardice, that it's not more strict.

I'm not so sure. If NCX entries could only point to Hn tags then some flexibility would be lost. There are times when it can be appropriate to link to a paragraph which does not start with a header. If the spec forced one as a requirement then this could introduce sematic structure in places where there shouldn't be any - just to allow the toc to link to it.

Most of us recognise and dislike messy markup but there are many ways of using different, but well-formed, markup to achieve the same end result.

Have you done any web searches for HTML tidying tools. Maybe there is something out there with some flexible user configuration that could help you.