epubveri – standalone EPUB validator

[j.p.s]

Originally Posted by Karellen

I haven't read back far enough to understand the "why" or how this differs from epubcheck.

Post #1 covers that.

[Doitsu]

Originally Posted by JSWolf

Can we have a setting in epubveri to say if an ePub3 is backwards compatible enough with ePub2?

@Kayadelenium EPUBCheck will return a isBackwardCompatible value that is true if EPUBCheck considers a book epub2-compatible.

However, EPUBCheck has rather low standards. AFAIK, it'll consider an epub3 book backwards-compatible als long as it's not a fixed layout book, contains no Javascript and has an .ncx file.
EPUBCheck also used to check for HTML5-only tags but that check was apparently dropped some time ago.

If you decide to create your own version, you might want to add a check for HTML5-only tags in documents other than the NAV doc.

IMHO, at a bare minimum, the following features also should also disqualify a book from being considered epub2-compatible.

• Multiple renditions
• Media overlays
• MathML, unless a fallback was defined.

[JSWolf]

Originally Posted by Doitsu

@Kayadelenium EPUBCheck will return a isBackwardCompatible value that is true if EPUBCheck considers a book epub2-compatible.

However, EPUBCheck has rather low standards. AFAIK, it'll consider an epub3 book backwards-compatible als long as it's not a fixed layout book, contains no Javascript and has an .ncx file.
EPUBCheck also used to check for HTML5-only tags but that check was apparently dropped some time ago.

If you decide to create your own version, you might want to add a check for HTML5-only tags in documents other than the NAV doc.

IMHO, at a bare minimum, the following features also should also disqualify a book from being considered epub2-compatible.

• Multiple renditions
• Media overlays
• MathML, unless a fallback was defined.

Don't forget rem instead of em in the CSS, <figure> instead of <div>, classes in things ePub2 doesn't support such as <section>, media queries that older RMSDK thinks is an error, and lack of xlink: in SVG code.

[Turtle91]

Originally Posted by JSWolf

Don't forget rem instead of em in the CSS, <figure> instead of <div>, classes in things ePub2 doesn't support such as <section>, media queries that older RMSDK thinks is an error, and lack of xlink: in SVG code.

While I have nothing against checking for these things just so I know (I wouldn't mind a message saying it is an ePub3 thing only).... those don't really break anything in a compliant ePub2...compliant renderer's are supposed to be graceful and ignore anything they don't understand. At worst, you just won't get the styling you expect if you code for ePub3 and read on an ePub2 device. As far as checking for things "that older RMSDK thinks is an error".... ummm I think this should just check that the ePub is valid, not compare it against the litany of things that older renderer's just can't grok.

[Kayadelenium]

Originally Posted by Doitsu

EPUBCheck will return a isBackwardCompatible value that is true if EPUBCheck considers a book epub2-compatible.

Thank you, I did not know that field existed, and I checked it in the 5.4.0 source before replying. You are right on every point. It starts as true and only three things turn it false: scripting, fixed layout, and no NCX (meaning the spine has no toc attribute, or it points at something that is not in the manifest). It appears only in the JSON and XML reports, never as a message, and it never touches the verdict. I found no HTML5-element test in the current code, so the check you remember is gone, whenever that happened.

Which means that a book using rem, figure, section and media queries, with a correct NCX, is reported by EPUBCheck today as backwards compatible. By JSWolf's list at #333 that is simply not true on the devices he cares about. So the reference tool already answers this question, and by the standards of the people in this thread it answers it wrongly. I find that more interesting than anything I could build, because it means the question is not "should a validator say this" but "what would it take for a validator to say it truthfully".

JSWolf, this is my answer to #330 as well, and it is a set of questions, because the honest answer depends on yours.

Part one: where does a validator's trust begin?

This thread is titled "Possible replacement for EPUBCHECK". I take that seriously in both directions. A replacement that says less than EPUBCheck misses real problems. One that says more is, to anyone comparing the two, indistinguishable from one that is wrong. That is why everything epubveri reports beyond EPUBCheck lives behind --advisory and can never change the verdict or the exit code: a book that passes EPUBCheck passes epubveri. I hold that line on purpose, and I would like to know whether you think it is the right one.

1. What is a "valid" from epubveri actually worth to you? If it meant "valid, except where the author of the tool decided the specification was wrong", would you still trust it? Would a publisher? Would a retailer's ingestion pipeline?

2. If epubveri drifts from EPUBCheck and the specification, why would anyone use it? Speed, no Java, running inside Sigil and calibre, clearer messages: those are conveniences. None of them is a reason to trust a verdict. So what is? And what is the first thing that would make you stop using it?

3. Where is the line between checking a book and predicting a device? Doitsu's list (scripting, fixed layout, no NCX, multiple renditions, media overlays, MathML without fallback, HTML5 elements outside the nav) can all be decided from the book alone; each is either there or not. JSWolf's list (rem, figure, section, media queries, SVG without xlink) is about what one particular engine does with those things, and none of us can test every version of that engine. Do you see those as the same kind of statement? Should one tool make both?

4. Who is entitled to bend a standard? If a tool softens the specification for one group's devices, it hardens it for somebody else's. Is that ever a tool's call to make, or only the Working Group's? And if EPUBCheck itself already bends here, as above, should a second tool copy the bend or refuse it?

Part two: fifteen years

EPUB 3.0 became a specification in October 2011. It will be fifteen years next month, and in this thread we are still working out how to keep EPUB 3 books readable by EPUB 2 engines. I do not think that is anyone's personal failing, but I do think it is a fact that needs an explanation, and I would genuinely like to know where each of you puts it.

5. Is it the reading devices? Engines that stopped being developed, firmware that is never updated, hardware people rightly keep using for ten years. Should the devices have moved on, or is a book that outlives its reader exactly what the format is for?

6. Is it the publishers? Doitsu said at #315 that all his big-five fiction still carries an NCX. Is that deliberate care for old readers, or production pipelines built for EPUB 2 that nobody has had a reason to rebuild? Would publishers move faster if the tools pushed them, or would they just stop running the tools?

7. Is it the specification? EPUB 3.4 keeps the NCX and the OPF 2 guide only for "a measure of backwards compatibility", but it never says how much, or for which reading systems. There is no compatibility profile, nothing a producer can target and a checker can check. Is that a gap the Working Group left, or a question it was right to refuse?

8. Is it the retailers and distributors? Ingestion requirements, conversion steps, the formats a store actually delivers to the device. Do they hold the old floor in place more than anyone in this thread does?

9. Or is it the tools, us included? Validators and editors that never told anyone to stop, because nobody wanted to be the one breaking old readers. Would you want a tool to be more opinionated here, or is the silence what keeps books working?

10. Or is none of this a problem at all? EPUB 2 works, the books open, the readers are content. If you think the fifteen years are simply the format doing its job, I would like to hear that argument too.

I am not proposing anything, and I will not build anything off the back of this until the replies are in. If they point at something that belongs to epubveri, I will say so here before doing it. If they point somewhere else, that is worth knowing just as much.

[KevinH]

FYI - the first epub3 spec was basically ignored or received little traction for the first 5 to 8 years after it was first published. It was international publishing, mainly the Japanese marketplace, that first began pushing epub3 adoption for language reasons. Remember Kindle was the big publisher for NA and they only began to accept epub3 input when the KF8 format became available. Sigil itself did not even begin to support epub3 spec until January of 2016 and that was still just partial support. There are places in europe where epub2 publishing is still dominant.

So 15 years sounds like a long time but the publishing industry is very slow moving unlike web browsers. And most people hated the fact that epub3 broke almost all backwards compatibility with epub2 at first till people figured out how to add backwards compatibility for simple fiction books.

[JSWolf]

Originally Posted by Turtle91

While I have nothing against checking for these things just so I know (I wouldn't mind a message saying it is an ePub3 thing only).... those don't really break anything in a compliant ePub2...compliant renderer's are supposed to be graceful and ignore anything they don't understand. At worst, you just won't get the styling you expect if you code for ePub3 and read on an ePub2 device. As far as checking for things "that older RMSDK thinks is an error".... ummm I think this should just check that the ePub is valid, not compare it against the litany of things that older renderer's just can't grok.

If you don't support rem, then your spacing is all wrong. It needs to be em to work. <figure> doesn't work so you need <div>. <section> is ignored. But if there's a class, that class is not used. So any epub3 only attributes that have a class have that class ignored. If there is no xtlink: in the SVG code, no image will display. And if there is a media query that an old version of RMSDK doesn't like, then the entire CSS is ignored.

These are important things to be ePub2 compatible enough because they do break things that should not be broken.

[Turtle91]

Originally Posted by JSWolf

If you don't support rem, then your spacing is all wrong. It needs to be em to work. <figure> doesn't work so you need <div>. <section> is ignored. But if there's a class, that class is not used. So any epub3 only attributes that have a class have that class ignored. If there is no xtlink: in the SVG code, no image will display. And if there is a media query that an old version of RMSDK doesn't like, then the entire CSS is ignored.

These are important things to be ePub2 compatible enough because they do break things that should not be broken.

Yup, like I said. They will just not have the expected styling someone coding for ePub3 only expects. The compliant ePub2 renderer will still open the ePub3...it just won't look the same. That does not make the ePub non-compliant...which is the point of checking with ePubveri... that is the fault of the coder for not making it backwards compatible and including fall-back coding.

Like I also said - I wouldn't mind getting an advisory message that the coder used ePub3 only coding (especially if it is designated as an ePub2 book) - but it should not fail the check.

[JSWolf]

Originally Posted by Turtle91

Yup, like I said. They will just not have the expected styling someone coding for ePub3 only expects. The compliant ePub2 renderer will still open the ePub3...it just won't look the same. That does not make the ePub non-compliant...which is the point of checking with ePubveri... that is the fault of the coder for not making it backwards compatible and including fall-back coding.

Like I also said - I wouldn't mind getting an advisory message that the coder used ePub3 only coding (especially if it is designated as an ePub2 book) - but it should not fail the check.

It's not a matter of looking different. Some of that will actually break the ePub. Yes it will load but not everything will display well enough. Would you like to read an ePub with no CSS?

[Turtle91]

Originally Posted by JSWolf

It's not a matter of looking different. Some of that will actually break the ePub. Yes it will load but not everything will display well enough. Would you like to read an ePub with no CSS?

Like? No.
Is it a valid epub? Yes.

This is a tool to determine if the ePub is valid, not to fix all the things a coder can screw up… or to go over all the things all old renderers are non-compliant for…

Not to say that a tool for checking how things would display on old non-compliant renderers, such as RMSDK, is not a good idea… that actually has merit, but that is a different tool, and shouldn’t be joined together with a tool that checks if an ePub is valid.

[DNSB]

Originally Posted by JSWolf

It's not a matter of looking different. Some of that will actually break the ePub. Yes it will load but not everything will display well enough. Would you like to read an ePub with no CSS?

I've done back when early ereaders were not happy with some CSS to the point that the CSS was tossed into the bitbucket.

These days when reformatting/editing/whatever an ePub for others, I test my hopefully final ePub on multiple applications and hardware ereaders. There is no way to have a validator spot all the oddities such as the Nook ereaders being unhappy with margin shortcuts. I don't even want to think about the Wild West that is most of the Android renderers.

[Kayadelenium]

epubveri 0.17.1: a security release: one crash fixed, one false error fixed, and a way to exhaust memory closed. Release notes and downloads

I ran a security review of epubveri this week, the kind that assumes the book in front of the validator was made by someone who wants it to fail. A validator reads files other people made, so that is the only honest assumption. It found three things worth fixing, and none of them came from a report. Here they are, most likely to reach you first.

A crash. A malformed tag with a letter such as à or Å right next to an equals sign stopped epubveri with a crash instead of a report. One example is a p tag written as title=Voilà=oui, with the value unquoted. That kind of document is already broken and should get a clear fatal error saying so, which is what it gets now. The cause was a scanner that read single bytes as if they were characters, and the second byte of à and Å happens to count as whitespace when read that way. It was found by feeding epubveri thousands of deliberately corrupted books. That test now runs before every release.

A false error. A book containing a single image larger than 64 MiB was reported INVALID with LIM-001, while EPUBCheck calls the same book valid. The image check only needs the first few bytes of a file to tell what it is, but it was unpacking the whole image to get them, and that ran into a limit meant for something else. It now reads just those bytes, and such a book validates normally, using less than half the memory it did.

A small file could use gigabytes of memory. XML lets a document declare a piece of text once and repeat it by name. A 1.6 KB book that did this enough times made epubveri use 5 GB of memory for several seconds and then call the book valid. A larger one would have exhausted the machine, or the browser tab running the online demo. epubveri now works out how large a document would become before reading it, and refuses one that would grow past 64 MiB. No real book comes anywhere near that, and the limit is set above the largest such expansion EPUBCheck itself still validates normally.

Nothing else changes. Every result on the 474 books I test against is identical to 0.17.0, and so is the speed. If you use the Sigil or calibre plugin, it fetches 0.17.1 by itself on its next run, unless you have switched off its automatic update.

If you find a security problem yourself, please report it privately rather than here, where anyone could use it before it is fixed. GitHub now takes private reports for all the veripublica projects, and SECURITY.md explains what counts and how to check that a download is the one I published. A wrong validation result is not a security problem, and this thread remains the best place for those.

[JSWolf]

Originally Posted by Turtle91

Like? No.
Is it a valid epub? Yes.

This is a tool to determine if the ePub is valid, not to fix all the things a coder can screw up… or to go over all the things all old renderers are non-compliant for…

Not to say that a tool for checking how things would display on old non-compliant renderers, such as RMSDK, is not a good idea… that actually has merit, but that is a different tool, and shouldn’t be joined together with a tool that checks if an ePub is valid.

This is not the program to fix code. This is the program to tell us if there are problems with the code. And an ePub3 that's not properly backward compatible with ePub2 when that's what you want means the ePub is broken.

[JSWolf]

Would it be possible to have a warning when an HTML file is too large for the mobile limit?

[Kayadelenium]

epubveri 0.17.4 is out — and with it 0.17.2 and 0.17.3 from earlier today. Two things to say about them: it is a lot faster, and it is harder to knock over.

Faster

I spent a good part of today on speed, and it paid off. Most of epubveri's time went into checking content documents against the XHTML grammar, and the engine doing it was rebuilding the whole attribute model of an element for every single attribute it read, and the parent's content model again for every child. It now remembers what it has already worked out: elements of one kind lead to the same state, so after the first paragraph the rest are looked up rather than recomputed.

What that means on real books, measured today on the same 474 EPUBs and the same machine, one run per book the way a plugin calls it:

• A typical book: 0.05 s for epubveri, 1.96 s for epubcheck 5.4.0 — about forty times faster.
• All 474 books: 55 seconds against about 16 minutes — about eighteen times faster.
• One CPU core against nearly four, and 8 MiB of memory for a typical book against 421 MiB.
• A heavy chapter full of attributes: 4 MiB used to take 17 seconds and now takes 1.

None of the answers changed. Every book on that shelf gets byte-for-byte the same report it got before; it just arrives sooner. The full comparison, including how to repeat it on your own books, is here: docs/BENCHMARK.md. As always, those are one machine's and one library's numbers, so measure your own before relying on them.

Harder to knock over

epubveri reads files other people made, so a book built to hurt it should get a report, not a crash. epubveri has guards against two such shapes: nesting deep enough to overflow the stack, and entities that blow a small file up into gigabytes (that one new in 0.17.1). While preparing those guards for use by epubsana, I tested them properly and found they could be walked past: nine different constructions got through, six of them crashing the validator and three driving it to 4.8 GB of memory. 0.17.2 closes all nine. The check now reads a document the way the XML parser does, and each of the nine is kept as a test, so none of them can come back unnoticed.

I also added three limits that no real book comes near:

• at most 256 attributes on one element (the most on my 474 books is 11), because the parser's cost grows with the square of that number;
• at most a million elements in one document (the most on my books is about 20,000), because memory is spent per element — a 60 KB file could previously make epubveri use 1.3 GB;
• at most 256 MiB read from one publication in total (the most any of our books reads is about 25 MB). Past that, epubveri tells you which files it did not check, as the new LIM-002.

A book that crosses one of these is reported, never silently skipped, and none of the books we test against is affected. If you ever do find a file that crashes epubveri or eats memory out of proportion to its size, please report it privately through GitHub's security advisories rather than here; SECURITY.md in the repository has the link.

Also

The README now says that building epubveri from source on Windows needs Microsoft's C++ build tools, after a user's cargo install stopped at "link.exe not found". If you just want to run it, the ready-made Windows download needs none of that.

Updating

If you use one of our Sigil or calibre plugins, there is nothing to do: they fetch the newest epubveri themselves. Otherwise the downloads are on the releases page, and the full list of changes is in CHANGELOG.md.