EPUB2 and the DOCTYPEgate

[roger64]

Hi

There is currently a difference betwen calibre's Editor and Sigil. The Editor gets systematically rid of the DOCTYPE and Kovid Goyal writes that the DOCTYPE is required only when there are named entities (like nbsp).

I am personnally quite happy with this but I hear people arguing against it nearly for "moral" reasons. Illegal, non respect of such norm, and so on.

I would like to find the reference file where this is an allowed use. The problem for me is that any document I find gives a reference to another and so on.

I am sure this is old news, because some years ago already, writer2xhtml offered an option to produce an EPUB2 (EPUB3 was non-existent at that time) using only UTF8 or UTF16 characters - like the Editor today. (see screenshot)

I speak about the common EPUB2 not about EPUB3, html 5 or anything else. I found in the xhtml 1.1 specs telling that the DOCTYPE was required (screenshot 2). But there is another one about xhtml 2 which says that it MAY only be required (screenshot 3) and is only necessary "when the character encoding of the document is other than the default UTF-8 and UTF-16" but it's a draft.

Which one applies to EPUB 2? How old is the xhtml 2 specification?

IDPF specifies for the EPUB 2.01 (May 2010), that the required MIME media type for application/xhtml + xml must be XHTML 1.1. That would mean that a DOCTYPE is required.
http://www.idpf.org/epub/20/spec/OPS...m#Section1.3.4

[Jellby]

As far as I could understand, the DOCTYPE is indeed not needed.

[Doitsu]

Originally Posted by Jellby

As far as I could understand, the DOCTYPE is indeed not needed.

But isn't the ePub 2.01 standard based on the second edition of the older xhtml 1.1. standard, which does require a doctype?

[Toxaris]

It is a bit technical, but for the XHTML spec the DOCTYPE is required. However, an ePUB must contain XML files, not per se XHTML. However, pure XML files will not work as far as I know...

[Jellby]

It says it must be XML, with all elements (not in islands) from XHTML.

And then of course there's the specifications and there's the implementations

[Arios]

Roger,

I did a quick test with a file created and saved in Sigil (I'm new with the Calibre eBook-Edit).

Then the file was imported into the Calibre eBook-Edit module. I applied "Beautify all files", save the file and loaded the epub on my Kobo and Sony: both were displayed the files perfectly.

So it seems, from a practical point of view at least for my Kobo and Sony, that both specs work as expected.

And, by the way, the files passed epubcheck without a hitch.

[DaleDe]

Take a look at our wiki https://wiki.mobileread.com/wiki/DOCTYPE

Dale

[roger64]

Thanks all for your inputs on this. The expression "some believe" in the wiki is a bit unclear...

Now lets come back to the full original statement from Kovid Goyal

Originally Posted by kovidgoyal

Unicode characters work in every single place that entities work. Both unicode characters and entities require a declaration in the header. The DOCTYPE in the case of named entities and the character encoding in the case of unicode characters.

The DOCTYPE is an absolute requirement in order to use named entities in XHTML, while the character encoding is not, since the default encoding for XHTML is UTF-8 when undeclared, which is the encoding the calibre editor uses.

Therefore, named entities are actually *less* likely to work than unicode characters.

From what I can see, the second paragraph of this statement is ambiguous because it's both right and wrong.

- wrong, because formally, EPUB 2.0.1 which is the latest iteration of the official norm before EPUB 3, still relies on xhtml 1.1 specs, for which the DOCTYPE is an absolute requirement. Point.

- right, because technically, this DOCTYPE is only useful for named entities. So even if the DOCTYPE is required formally, it is not needed technically if all the code is made out of (UTF-16) Unicode characters.

I agree with Arios: there seems to be absolutely no compatibility problems arising from this suppression of the DOCTYPE as long as the EPUB 2 contains only (UTF-16) Unicode characters.

[Doitsu]

Originally Posted by roger64

So even if the DOCTYPE is required formally, it is not needed technically if all the code is made out of UTF-16 characters.
I agree with Arios: there seems to be absolutely no compatibility problems arising from this suppression of the DOCTYPE as long as the EPUB 2 contains only UTF-16 characters.

According to the ePub 2.01 specs, files can be encoded either as utf-8 or utf-16 files. I.e., there's no need to use utf-16 files just because you want to avoid including a DOCTYPE. Also, by definition, utf-16 files are at least twice the size of utf-8 files and offer no advantages for languages that use the Latin alphabet.

[roger64]

Originally Posted by Doitsu

According to the ePub 2.01 specs, files can be encoded either as utf-8 or utf-16 files. I.e., there's no need to use utf-16 files just because you want to avoid including a DOCTYPE. Also, by definition, utf-16 files are at least twice the size of utf-8 files and offer no advantages for languages that use the Latin alphabet.

Thank you for this interesting info. I just spoke about UTF-16 because I had read (where?) that calibre editor used hexadecimal unicode characters. I knew no more about this. It would have been better to write only: Unicode characters.

I correct it above.

[Jellby]

Originally Posted by roger64

- wrong, because formally, EPUB 2.0.1 which is the latest iteration of the official norm before EPUB 3, still relies on xhtml 1.1 specs, for which the DOCTYPE is an absolute requirement. Point.

I don't see that. Where is it stated that ePub documents must be valid XHTML 1.1? As I said above, they must be vaild XML, with XHTML.

Originally Posted by roger64

I just spoke about UTF-16 because I had read (where?) that calibre editor used hexadecimal unicode characters. I knew no more about this. It would have been better to write only: Unicode characters.

You are mixing several things:

Named entity: é (needs DOCTYPE)
Decimal entity: é
Hexadecimal entity: é ("x" means it's hexadecimal, where "e" means fourteen, so "e9" = 15*16^1+9*16^0 = 233)
Unicode character: é

The Unicode character may be actually stored as utf-8 or utf-16, but that's mostly invisible to the user:
In utf-8 it will be saved as: C3 A9 (two bytes)
In utf-16 it will be saved as: 00 E9 (two bytes)

[Doitsu]

Originally Posted by Jellby

The Unicode character may be actually stored as utf-8 or utf-16, but that's mostly invisible to the user:
In utf-8 it will be saved as: C3 A9 (two bytes)
In utf-16 it will be saved as: 00 E9 (two bytes)

However, since in utf-16, all characters need be encoded with at least two bytes, utf-16 files can be up to twice the size as utf-8 files (if they only contain characters from the Basic Latin block).

IMHO, there's no real benefit in using utf-16 for ePubs or anything else for that matter.

[SBT]

Originally Posted by Doitsu

However, since in utf-16, all characters need be encoded with at least two bytes, utf-16 files can be up to twice the size as utf-8 files (if they only contain characters from the Basic Latin block).

Surely that overhead more or less disappears when the files are zipped into the epub file?

[Doitsu]

Originally Posted by SBT

Surely that overhead more or less disappears when the files are zipped into the epub file?

It does indeed more or less disappear when the files are zipped, but since the reading software has to unzip the .html files in order to render the text and some older readers have problems with .html files significantly larger than 280 KB, IMHO, there's no point in using utf-16 in the first place.

AFAIK, utf-16 only needs to be used with certain APIs that expect utf-16 formatted strings and for the proper handling of some CJK characters.

[roger64]

Thank you for your information regarding characters and entities. It's a little like the ruins of Troja, you find still another layer when you start digging...

Originally Posted by Jellby

I don't see that. Where is it stated that ePub documents must be valid XHTML 1.1? As I said above, they must be vaild XML, with XHTML.

This statement makes me uneasy. Look how the Editor behaves with valid XHTML 1.1. I produce EPUB2 from a converter with valid xhtml 1.1. files. As soon as I open this EPUB2 with the Editor, the DOCTYPEs are beheaded (and named or numbered entities transformed). It's not XML but pure XHTML though...