Possible replacement for EPUBCHECK

[Kayadelenium]

epubveri 0.10.0 is out — crates.io, npm, and the eight pre-built binaries.

Why 0.10.0 and not 0.9.30, since that will look like a bigger jump than it is. The number is about the Rust library API, not about the validator: a struct gained a field and an enum variant changed shape, which for a pre-1.0 crate means a minor bump. Nothing moved for anyone using the command line, the JSON output or the browser build, and the findings themselves are unchanged — across my 385-book test shelf the machine output is byte-identical to yesterday's build. If you use the plugin or the CLI, treat this as an ordinary release.

What you will actually notice

- `USAGE` findings are hidden by default now, as in epubcheck. Pass `-u` (or `--usage`) to see them. This is JSWolf's original complaint and Doitsu's proposal — thank you both. Of my 385 books, 60 produce nothing but usage findings, so 60 perfectly good books were printing a median of six lines a reader could reasonably read as problems. `info` still shows, which I took off epubcheck's own `--help` rather than assuming.
- Findings are grouped by severity, most serious first, with `--sort document` for the previous order. Reasoning in my last post.
- A column now points at the offending attribute, not at the element carrying it (JSWolf, #220). On a long start tag that is the difference between a usable column and a decorative one.
- `encryption.xml` — an empty one, and one still pointing at a font you deleted, are both errors now, matching epubcheck (JSWolf, #221 and #223).
- A schema message that contradicted itself — `element "html" is not allowed here; expected "html"` — now says what actually differs, which is the namespace (BeckyDTP, issue #84).
- Two `media:*` metadata rules were firing once per occurrence instead of once per constraint, and were wrong in both directions.

For plugin and tool authors

There is a new page, `docs/INTEGRATING.md`, and it exists because I had left you to guess. The JSON envelope has always been documented, but nothing user-facing linked to it and `USAGE.md` gave `--format json` four lines without ever showing its shape — so the only way to learn the format was to read doc comments in the Rust source. Reaching for the human text instead was the reasonable thing to do with what I had published.

Short version: parse `--format json`, not the human report. The human output is written for a person and is free to change — this release alone changed which severities it shows, its order, and a column. The JSON is versioned, documented, never filtered and never reordered, and it carries things the human report has no room for: a stable sub-code for each finding (`RSC-005` alone means dozens of unrelated conditions), an XPath-style path to the offending node with its namespace bindings, and, new here, a machine token saying which kind of schema violation you are looking at.

The browser build now returns all of that too. Through 0.9.x it carried only part of it, so a web tool got strictly less than the command line — an oversight rather than a decision, and one I found by writing a documentation page that claimed otherwise. The demo page also gained an Order control mirroring `--sort`.

One caveat for anyone pinning flags, since it fails unhelpfully: `--sort` and `-u` do not exist before this release, and an unknown option is an exit-2 usage error, not a warning.

As always: if something here reports a problem on a book that is actually fine, that is the report I most want to see.

[JSWolf]

Since the code already exists to show unused resources for ePub3, why not access the same code for ePub2?

[Doitsu]

Originally Posted by Kayadelenium

- `USAGE` findings are hidden by default now, as in epubcheck. Pass `-u` (or `--usage`) to see them.

Doesn't seem to work with epubveri 0.10.0 in JSON mode when I tested it with this book.
Even though I didn't use the -u/--usage switch one USAGE message was still reported.

epubveri reported:

      "path": "scrambled.epub",
      "status": "problems",
      "summary": {
        "errors": 13,
        "warnings": 0

but it also outputs

{
  "type": "finding",
  "code": "CSS-028",
  "rule": "css.font_face.declared",
  "severity": "usage",
  "location": "page_styles.css",
  "position": {
	"line": 5,
	"column": 1
  },
  "message": "@font-face declaration"
},

The command line was:

epubveri -i scrambled.epub --format json --advisory

In text mode, no USAGE message is reported.

[Kayadelenium]

Originally Posted by JSWolf

Since the code already exists to show unused resources for ePub3, why not access the same code for ePub2?

The version gate is one line, so I removed it locally and measured what would come out, rather than reasoning about it.

On my 385-book shelf, 312 of them EPUB 2: 75 books (24%) would gain findings, 171 in total — overwhelmingly images, 112 jpg and 28 png, which makes this the same thing as the unused-images request you made earlier.

Then I looked at which files, and two of the commonest answers are wrong:

- `OEBPS/cover.jpg`, repeatedly. In EPUB 2 the cover is declared with `<meta name="cover" content="…"/>` in the metadata, not linked from a content document. The image is used. Calling it unused would be a false alarm on nearly every book that has a cover.
- `Text/toc.xhtml`, likewise. An EPUB 2 HTML table of contents is usually reached through the `<guide>`, not from the spine or from any content document.

Neither mechanism is in the rule's exemption list, and neither feeds its set of "things that got referenced" — because in EPUB 3 neither needs to be. EPUB 3 declares a cover with `properties="cover-image"` in the manifest and its navigation document is an ordinary content document. The same code is right there and would be wrong here.

Where epubcheck stands on this, since it is the more useful half of the answer. The rule lives in epubcheck's EPUB 3 checker and has no EPUB 2 counterpart — that is a decision on their side, not an oversight, and EPUB 2's cover and guide mechanisms are a large part of why. Worth noting too that even in EPUB 3, where epubcheck does ask the question, it reports the answer at `usage` severity: a note about the book, not a defect in it. So nothing here is being missed in the sense of a book being wrong and going unreported.

That matters for how I work: where epubcheck has an opinion, I match it, and where it does not, anything I add goes behind `--advisory` so a book that passesepubcheck still passes epubveri. An EPUB 2 version of this would be in that second category — opt-in, and never affecting the verdict.

So I am not promising it. It would mean teaching the check two EPUB 2-only reference mechanisms first, and the measurement above says the naive version would put a false alarm on a quarter of my test shelf — which is the one thing I work hardest to avoid. It is a reasonable idea and I would like to look at it properly when there is time for it; I would rather say that than commit to a date and have it sit undone.

Leaving the numbers here so the question does not have to be re-argued from scratch next time it comes up.

[Kayadelenium]

Originally Posted by Doitsu

Doesn't seem to work with epubveri 0.10.0 in JSON mode when I tested it with this book.
Even though I didn't use the -u/--usage switch one USAGE message was still reported.

epubveri reported:

      "path": "scrambled.epub",
      "status": "problems",
      "summary": {
        "errors": 13,
        "warnings": 0

but it also outputs

{
  "type": "finding",
  "code": "CSS-028",
  "rule": "css.font_face.declared",
  "severity": "usage",
  "location": "page_styles.css",
  "position": {
	"line": 5,
	"column": 1
  },
  "message": "@font-face declaration"
},

The command line was:

epubveri -i scrambled.epub --format json --advisory

In text mode, no USAGE message is reported.

Doitsu — your observation is exactly right, and it is deliberate rather than a bug, which means the failure here is mine for not making it obvious enough.

`-u` filters the human report only. `--format json` and `--format ids` always carry every finding, at every severity. It is in `--help` under `-u` ("the machine formats … are never filtered") and on the new `docs/INTEGRATING.md` page, but you went straight to it within an hour of the release, so clearly that was not where anyone would look.

And you are right to be surprised, because epubcheck does not behave this way. I checked rather than assumed: `epubcheck --json` without `-u` omits the usage messages, and with `-u` includes them. So this is a real divergence, and one I chose.

The reasoning is asymmetry of harm. A machine consumer that receives more findings than it wants can ignore them in three lines; one that receives fewer than the validator produced cannot recover what it never got, and has no way to know something was withheld. Given a choice between those two failures I would rather the tooling always see everything and decide for itself.

It also happens to serve the plan you described earlier in this thread — adding a plugin option for these. With the JSON complete you can run once and toggle in your own UI; if the JSON were filtered you would either have to pass `-u` permanently and filter client-side anyway, or re-run the validator every time somebody flips the switch.

Same for `--format ids`, and the same for the library API, where nothing is ever filtered by severity: epubsana dispatches three of its repair rules on findings below error severity, so hiding those would silently take working repairs offline with nothing on either side reporting it.

If the divergence actively costs you something, say so and I will reconsider. This is a CLI behaviour, not a verdict — nothing about it changes whether a book is valid, so I am not defending it on parity grounds and I do not need to be right about it. What I would not do is filter quietly in one place and not the other, which is why `-u` stops at the human report.

Thank you for testing the release the hour it landed.

[Doitsu]

Originally Posted by Kayadelenium

If the divergence actively costs you something, say so and I will reconsider. This is a CLI behaviour, not a verdict — nothing about it changes whether a book is valid, so I am not defending it on parity grounds and I do not need to be right about it. What I would not do is filter quietly in one place and not the other, which is why `-u` stops at the human report.

The plugins parse the JSON output and I'd like to give users the possibility to suppress Usage messages, which doesn't work because the -u/--usage parameter is ignored.
Also, IMHO, it doesn't make sense in terms of consistency to apply command line parameters only to the human readable output.

Speaking of JSON, IMHO, it'd helpful if you added fatal, usage and info under summary:

        "fatal": 1,
        "error": 13,
        "warning": 0,
        "usage": 0,
        "info": 0,

I'd also recommend switching to singular nouns, since information has no plural and usages doesn't make sense in this context.

[Kayadelenium]

0.11.0 is out with all three of your requests, Doitsu — crates.io, npm and the binaries, so the plugin will pick it up.

`-u` now decides what every format contains. You were right that one flag meaning two things depending on `--format` is not something anyone should have to remember, and measuring settled the rest: epubcheck's own `-u` gates its JSON too, `nUsage` included. Same book, both tools, before and after:

```
epubcheck epubveri 0.11.0
no -u nUsage=0, 1 message usage=0, 1 item
-u nUsage=1, 2 messages usage=1, 2 items
```

`--format ids` follows the same rule. My advice for the plugin is to pass `-u` and filter in your own UI — one run, and your toggle does not have to re-invoke the validator.

One thing to expect when you re-test, since it is what produced your original report: `--advisory` findings still print without `-u`. They are emitted at usage severity, so a filter written as "hide usage" would make that flag silently do nothing. `--advisory` is their switch, `-u` is everyone else's. So `--format json --advisory` with no `-u` now shows `ADV-`/`NEXT-` and nothing else at usage severity.

The summary has all five counts, and this is a breaking change for you — the keys are singular now, so `errors`/`warnings` are `error`/`warning`, plus `fatal`, `info` and `usage`. Your note about the grammar is why: "information has no plural and usages doesn't make sense" is correct, and doing it while the counts were being added meant one break instead of two. `fatal`, `info` and `usage` are omitted when zero, as `fatal` already was; `error` and `warning` are always present. They describe what the output contains, so with no `-u` the usage count is 0 rather than a number you cannot see the findings for — again, what epubcheck does.

```json
"summary": { "error": 1, "warning": 0, "usage": 1 }
```

The browser build mirrors the same shape, but has no flag to filter on and so always returns everything.

What has not changed, deliberately: the library API still never filters. `validate_bytes` returns every finding at every severity whatever the CLI was given. That is the boundary that matters — a repair tool dispatching on findings below error severity would otherwise go quietly dark — and it is the one thing I was actually protecting when I got the CLI half wrong.

Thank you for pushing on it. I had written the reasoning down as a decision before you replied, including what would make me reverse it, and you produced exactly that, so the record now says superseded rather than defended.

[Doitsu]

Originally Posted by Kayadelenium

0.11.0 is out with all three of your requests, Doitsu — crates.io, npm and the binaries, so the plugin will pick it up.

Thanks for implementing the changes.

For completeness's sake you also might want to add an encryption.xml validator.

If an epub contains an encryption.xml file that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<encryption xmlns="urn:oasis:names:tc:opendocument:xmlns:container" xmlns:enc="http://www.w3.org/2001/04/xmlenc#">
  <enc:EncryptedData>

  </enc:EncryptedData>
</encryption>

EPUBCheck will report:

ERROR RSC-005: Error while parsing file: element "enc:EncryptedData" incomplete; missing required element "enc:CipherData"

epubveri doesn't report this as an error.

[JSWolf]

The ePub has tours in the OPF just in front of the guide. epubcheck reports this as an error and epubveri does not.

<tours>
</tours>

epubcheck:
content.opf Line:159 Col:9 ERROR(RSC-005): Error while parsing file: element "tours" incomplete; missing required element "tour"

[JSWolf]

Another error found. The ePub has an ecrypted font. epubveri is reporting the wrong file.

ePubcheck reports...
font00207.otf Line:-1 Col:-1 INFO(RSC-004): File "OEBPS/Fonts/font00207.otf" is encrypted, its content will not be checked.

epubveri reports...
META-INF/encryption.xml 5 7 INFO RSC-004: File "OEBPS/Fonts/font00207.otf" is encrypted; its content will not be checked

[JSWolf]

Here's another error in epubveri.

Here is the code in the OPF.

<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="e9780759525795" version="2.0" xml:lang="en-US">

epubveri reports no error. epubcheck reports: package.opf Line:2 Col:113 ERROR(RSC-005): Error while parsing file: attribute "xml:lang" not allowed here; expected attribute "id"

Changing an ePub3 to ePub2 without changing the rest of the code is actually pretty good at finding errors in epubveri.

[KevinH]

Technically xml:lang is always part of any xml namespace without having to add anything. So although epubcheck flags that as an problem, I think they do so in error as there are child elements of the package tag that do accept and use xml:lang. Setting it once in the parent tag should be 100% correct.

Now epubcheck has long called this an error in epub2 but I think this is one of those cases where I think epubcheck oversteps.

[JSWolf]

Originally Posted by KevinH

Technically xml:lang is always part of any xml namespace without having to add anything. So although epubcheck flags that as an problem, I think they do so in error as there are child elements of the package tag that do accept and use xml:lang. Setting it once in the parent tag should be 100% correct.

Now epubcheck has long called this an error in epub2 but I think this is one of those cases where I think epubcheck oversteps.

I found this.

EPUB 2 does not allow a global language declaration using the xml:lang attribute on the package element. You must declare an xml:lang attribute on every metadata tag.

https://kb.daisy.org/publishing/docs/epub/language.html

[KevinH]

But that is a epubcheck choice, their org, the idpf wrote the spec and designed the schema for an OPF, and a wrong one in my opinion since it violates the xml spec which came first and says that xml:lang is an always valid optional tag.

And there is no good reason for it. Most books are published in one language at a time and metadata in that same language is typical. So stating it once on the package tag makes the most sense. Assuming it is the same as the dc:language value is simply not a great idea. The OPF metadata could be in one language while the epub is actually in another, and knowing the xml:lang of the OPF itself is still needed imho.

[JSWolf]

Originally Posted by KevinH

But that is a epubcheck choice, their org, the idpf wrote the spec and designed the schema for an OPF, and a wrong one in my opinion since it violates the xml spec which came first and says that xml:lang is an always valid optional tag.

And there is no good reason for it. Most books are published in one language at a time and metadata in that same language is typical. So stating it once on the package tag makes the most sense. Assuming it is the same as the dc:language value is simply not a great idea. The OPF metadata could be in one language while the epub is actually in another, and knowing the xml:lang of the OPF itself is still needed imho.

The thing is, with xml:lang not being allowed in ePub 2, then it would be best to do away with it. With xml:lang, we have dc:language and xml:lang.

But it would have been good to have just one language declaration.