epubsana — automatic repair for what epubcheck reports (not an editor)

[Kayadelenium]

Some of you have been putting epubveri through its paces in PeterT's thread, and nearly every false positive fixed since 0.5.15 came from that — real books finding what fixtures can't. Thank you. I'd like to try the same thing for the other half.

epubsana repairs the defects epubveri reports. epubveri says what's wrong; epubsana turns the safely-fixable findings into edits you approve one at a time, applies them, and prints a report of exactly what changed.

What it is not: it is not an editor and it is not trying to become one. It has no UI for writing markup, no preview, no book view. Sigil and calibre are where you edit a book; epubsana is meant to clear the mechanical, boring defects before you open it there, so the errors you're left staring at are the ones that actually need a human. If it ever competes with an editor, I've made a mistake.

Three ways to try it

• In your browser, nothing installed: https://veripublica.github.io/epubsana/ — the file never leaves the page; it's WebAssembly running locally, not an upload.
• CLI:

  cargo install epubsana

  , or prebuilt binaries for Windows/macOS/Linux on the releases page.
• Library: Rust crate

  epubsana

  , or npm

  @veripublica/epubsana-wasm

  .

Why it's safe to point at a real book

• It never writes in place. Output goes to

  <name>_fixed.epub

  , and it refuses outright if you aim the output at the input.

• --dry-run

  shows you every proposed fix and changes nothing.
• Every fix is confirmed one at a time by default, with a plain-English rationale for each, and the run ends with a report of what was changed and what was left alone.
• It re-validates the result, so the report's before/after numbers are measured, not claimed.

What it refuses, and why that's the interesting part

A repairer that guesses is worse than no repairer. Some examples of things it will not do:

• An

  <img>

  with no

  alt

  .

  alt=""

  is correct for a decorative image and wrong for a meaningful one, and nothing in the file says which. Inventing alt text is an accessibility bug that looks like a fix.
• A malformed date. It removes an empty

  <dc:date>

  , but leaves

  2022-09-08)

  alone — that's a date the author wrote, and deciding which characters are stray is a guess.
• An empty title. It fills

  <title></title>

  from the book's own TOC label or first heading, and declines when the book names that document nowhere. It never falls back to

  dc:title

  .
• Choosing between identifiers. If a package's declared unique identifier resolves to nothing and the book carries both a UUID and an ISBN, which one is canonical is your decision, not the tool's.
• HTML5 markup in an EPUB 2 book. Renaming

  <figure>

  /

  <section>

  to

  <div>

  would clear thousands of errors and destroy meaning that assistive technology uses. Not doing it.

Known limitations, so you don't have to find them the hard way

• Fixes are planned once, from the initial validation. A defect that only becomes visible after an earlier fix is not proposed — run it twice if you want those.
• There's no per-fix undo yet: it's approve-then-apply, and the safety net is that the input is untouched.

• --goal openable

  changes the verdict, not yet which fixers run.

What would actually help

Two things, and the second is the one I can't get anywhere else.

1. Tell me when a repair is wrong. Not "it didn't fix X" — "it changed X and it shouldn't have". epubsana checks its own work by re-validating with epubveri, which means it is structurally blind to any damage epubveri doesn't know how to see. A person who knows their own book is the only check that isn't circular. If a repaired file opens differently, renders differently, or loses something, that's the report I most want.

2. Which defects do you fix by hand, over and over? I rank new fixers by what shows up in real books, and my own shelf has run out of candidates that can be repaired without guessing. Everything still on it is either genuinely ambiguous or needs information the file doesn't contain. So the ranking now has to come from outside — if you find yourself making the same tedious correction across books, that's exactly what should be automated next.

Repo, with a per-defect catalogue of what each fix changes and when it declines: https://github.com/veripublica/epubsana

It's AGPL, or a commercial licence for closed embedders. Written in Rust, no Java, no C dependencies.

[rantanplan]

Great idea!

[Doitsu]

Originally Posted by Kayadelenium

epubsana repairs the defects epubveri reports. epubveri says what's wrong; epubsana turns the safely-fixable findings into edits you approve one at a time, applies them, and prints a report of exactly what changed.

It'd help a lot if the JSON output included a standard unified diff (the diff -u/git diff format) for each fix's edit(s).
This would greatly simplify creating plugins for Sigil, Calibre and other epub editors.

[Kayadelenium]

Thanks, both of you.

Doitsu — you're right, and the current output is a worse fit for your purpose than the docs make it look. Here is a real fix item from --format json as it stands today:

{
  "type": "fix",
  "outcome": "proposed",
  "code": "RSC-005",
  "rule": "opf.content_document.schema_violation",
  "severity": "error",
  "location": "Chapter-003.html",
  "message": "Wrap 3 runs of non-block content in <div> in Chapter-003.html",
  "data": {
    "fix_id": "fix.bare_text_in_body",
    "tier": "confirm_needed",
    "changes": [
      "wrap in <div>: \"<br />\"",
      "wrap in <div>: \"<br />\"",
      "wrap in <div>: \"<br /><br /><br /><br />\""
    ]
  }
}

Those changes entries are descriptions written for a person to read. Nothing downstream can render them or act on them — so for the consumer the JSON format exists to serve, they are not enough, exactly as you say.

I checked where a diff would have to live before replying. Our JSON contract makes items[].data tool-owned and requires consumers to ignore fields they don't recognise, so epubsana can add this on its own: no change to the shared spec, and no existing consumer breaks. The contract also already has the mode this belongs in — --dry-run emits the same envelope with every item marked "proposed", which is precisely the run a plugin would make to show someone a change before applying it.

Two issues are now open:

- #8 — Emit a unified diff per fix under items[].data: https://github.com/veripublica/epubsana/issues/8
- #7 — Workspace: a speculative apply: https://github.com/veripublica/epubsana/issues/7

The dependency between them is real rather than bureaucratic. Fixers currently write their edits straight into the in-memory container, so there is no way to ask what a fix would produce without producing it. That same missing piece is what per-fix rollback needs, so I would rather build it properly than bolt a diff onto the end.

Three questions where your answer is worth more than my guess:

1. One diff per fix, or one per file? Some fixes span files — renaming an invalid id rewrites every reference to it, across content documents and the NCX.
2. Diffs on applied fixes as well, or only under --dry-run?
3. Large diffs — cap them, and if so, how would you want a cap signalled? A fix over a big document can produce a lot of output, and I would rather settle that before someone's plugin runs into it.

One case has no text diff at all: the PKG-006 fix re-packages the container so the mimetype entry comes first and stored. Whatever gets emitted there has to say so, rather than showing an empty diff that reads as "nothing changed".

No timeline promised — I would rather land it correctly than quickly.

[Doitsu]

Originally Posted by Kayadelenium

1. One diff per fix, or one per file?

For plugin integration one diff per file would be ideal. If that's too much work to implement, one diff per change is also fine.
It should only be output if --dry-run is selected.
The idea is that a plugin could run epubsana in --dry-run mode, parse the json output and update only the files where problems were found instead of replacing all files.
It doesn't have to be unified diff. Any standard diff format that will allow Python based Calibre or Sigil plugins to update a text file is fine.

Originally Posted by Kayadelenium

One case has no text diff at all: the PKG-006 fix re-packages the container

All packaging problems can be ignored because both Sigil and Calibre Editor will repair epubs with packaging problems when the file is saved again.

Originally Posted by Kayadelenium

3. Large diffs — cap them

I personally would prefer all diffs, unless the diff output is close to the original file size.

Speaking of the diff output. It might even be easier to process the fixes that epubsana found if you embedded the complete fixed file as a base64 encoded zip file in the JSON output. (I realize that this is not very efficient. Maybe some other kind of compressed file format that is compatible with json would work better.)

Then a plugin could simply retrieve all updated files from the json file and replace the original text with the changed text.

[Kayadelenium]

Doitsu — that's three decisions, thank you. All of it is now recorded on #8.

One diff per file. That happens to be the shape the code already has: a fix's changes are recorded per container entry, so a fix that spans files — renaming an invalid id rewrites the document, the other documents linking to it, and the NCX — already produces one record per file. The diff attaches to that.

--dry-run only. This halves the work, and it sharpens the dependency rather than removing it: a dry run applies nothing, so the diff can only come from asking what a fix would produce. That was already the harder half; now it is the only path.

It also surfaced a requirement neither of us listed. If diffs are dry-run-only, your workflow is two invocations — one to get the diffs and show them, one to apply — and that is only sound if the second run plans identically to the first. So planning determinism has to become a documented guarantee rather than something that happens to be true today. Worth saying out loud before anyone builds on it.

A proportional cap. Agreed, and your fourth idea answers the question the cap leaves open — what happens past the threshold. The natural answer is: emit the complete new content of that file instead. So a diff and a full file are not two competing features; the full file is the diff's overflow mode. One field, either a diff or a replacement, explicitly tagged, so you never have to guess which you received. It won't ever silently truncate.

On embedding whole fixed books as base64 — I want to make sure I understood yon reads two ways and they are quite different. Did you mean the completerepaired EPUB as one blob, or the new content of each changed file?

If it's the second, we already agree — that's the overflow mode above.

If it's the first, my hesitation isn't size (though an EPUB is already a deflated ZIP, so compressing it again gains little while base64 adds about a third). It's that epubsana asks about each fix separately, so the repaired book is a function of which fixes you accepted — and under --dry-run you haven't accepted any yet. An embedded "fixed book" would have tomean "the book if you accept everything", which is one arbitrary choice out of many and wrong for anyone who approves two fixes out of three. Diffs compose; a finished artifact doesn't.

There's also -o, which writes the repaired file already. The one place that doesn't help is a browser, where there's no filesystem — and there the WASM binding hands the repairedbytes straight back to JavaScript, so that case is covered without going through the JSON at all.

But if you're after a single atomic response for a reason I'm not seeing, say so and I'll cost it properly rather than talk you out of it.

[j.p.s]

I think python can work with gzip compressed JSON objects.

[Doitsu]

Originally Posted by Kayadelenium

It also surfaced a requirement neither of us listed. If diffs are dry-run-only, your workflow is two invocations — one to get the diffs and show them, one to apply.

I looked at this with plugin development in mind. Currently, the only way to use epubsana with a plugin is to run it, open the fixed epub file, compare all files against the original files and then only replace the files that have changed.

There are several ways, in which epubsana could simplify this task.
a) output file based diffs for each fixed file.
b) embed fixed files with in the json output as lzma, gzip or zip archives.
c) simply output an additional compressed file that contains only the files that were changed and maybe some kind of file list.

I'm not a programmer, but I'm pretty sure that someone else has already come up with a more elegant solution.

[Kayadelenium]

Doitsu — that description of the current workflow is the most useful thing in this thread so far, because it says what you actually have to do rather than what you want emitted. Two answers: one that makes today less bad, and one that removes the step.

The comparison is cheaper than it looks. epubsana never re-zips a book. It keeps the original archive and walks it in its own order on the way out; an entry no fix touched is raw-copied, still compressed, so its bytes, compression method, timestamps and directory entries survive bit-for-bit. An entry a fix did rewrite is re-encoded with whatever method the original used, so a Stored entry never silently becomes Deflated. So you can compare raw compressed bytes without inflating anything, and an untouched file will compare equal every time. There is a test that holds this. That is worth knowing anyway, because it is also the reason the tool can claim it preserves what it doesn't touch.

But you shouldn't have to compare at all, and this is the part I got wrong. The information you're reconstructing is already in the tool and I throw it away at the JSON boundary. Internally a change is recorded as a container path plus a description; the emitter keeps the description and drops the path. So:

"data": {
  "fix_id": "fix.bare_text_in_body",
  "tier": "auto_safe",
  "changes": ["wrap 3 stray text runs in <p>"]
}                                                                                                                                                                               
becomes                                                                                                                                                                         
"data": {                                                                                                                                                                         "fix_id": "fix.bare_text_in_body",
  "tier": "auto_safe",                                                                                                                                                            "changes": [
    { "path": "OEBPS/Chapter-003.html", "note": "wrap 3 stray text runs in <p>" }                                                                                                 ]
}

That is your (c), except you get more than a file list: you get which fix touched which file, so a plugin that applies a subset knows exactly which subset of files that subset accounts for. I'd rather ship that than (b) — embedding the bytes duplicates ad compression choices then become my problem forever. (j.p.s — gzipped JSONwould work fine, I'd just rather not have to make it necessary.)

Two things I checked before saying this rather than assuming it. There is no way for a fix to delete a container entry — the only mutator adds or replaces — so the changed set is exactly "rewritten plus added", with no third category to catch you out. And et records a change does supply a real path, including the packaging one.
Which brings up the one case this does not cover, and it's the one I flagged ekaging fix (PKG-006) leaves the file's content byte-identical and changes onlywhere it sits in the archive and whether it's stored. A plugin that writes changed files back cannot reproduce it, because in Sigil terms it isn't a file edit at all. It will apin changes with a path, and it would be wrong to read that as "this file's congin surfaces it, it has to surface it as "re-save the container", or defer toepubsana's own output for that one.

And a correction to what I said this morning. I agreed diffs should be --dry-run only, and for diffs that's still right. But the changed-path list is not a diff, and your write-problem lives on the applied run, where a dry run by definition tells you nothn both. Your three answers were about showing a user what would happen; this is about writing the result back, and I filed them under one heading when they're two.

One question back, since data is tool-owned and you're the only consumer I know of: changes going from an array of strings to an array of objects is a shape change, not an addedfield. Pre-1.0 I'd rather change it than carry two fields forever, but if you ding those strings, say so and I'll add the paths alongside instead.
One thing that holds across all three tools, and won't change. None of them wrveri only reads. epubsana writes <name>_fixed.epub beside the input, orwherever -o points, and the input file is never opened for writing — the repair happens in memory and the original bytes are still what the untouched entries get copied out of. epublift is the same. If a plugin ever wants in-place replacement that has to made on its own backup, not something the tool does on your behalf. Repair isirreversible, and a tool that is confident about that on someone else's library has misunderstood the job.

Recorded on #8. No timeline — as before, I'd rather land it correctly than quickly.

[Doitsu]

Originally Posted by Kayadelenium

But you shouldn't have to compare at all, and this is the part I got wrong. The information you're reconstructing is already in the tool and I throw it away at the JSON boundary.

I missed that part. And you're right. With that information it's relatively easy to update an epub in Sigil with the changes. Besides, for smaller files, it's probably faster to simply replace the whole file from the fixed epub than applying diffs.

[KevinH]

And if you really want diffs to see exactly what changes, you can make a Sigil Checkpoint then run a diff compare after the plugin to see all file changes.

Alternatively, you could design a plugin that makes a github based Checkpoint both before and after and the user could easily use the Checkpoint features to see what changes and revert if needed,

[Doitsu]

I've created a quick-and-dirty Sigl epubsana edit plugin with the help of Claude AI.

It's definitely not ready for production. It also won't create a backup. If you want to test it, make sure to create a Checkpoint in Sigil before running the plugin.

The plugin will create a copy of the current epub and have epubsana apply all fixes. It'll then replace the files in the current epub with the files fixed by epbsana.

You can use the attached sample file for first tests. epubsana should be able to fix all problems in that file.

[KevinH]

Which then leads to a License question. Sigil is GPL3 not AGpl. It does not have the network clause. Sigil itself is not a commercial project, but some users of Sigil do use Sigil in their own commercial businesses. We the developers of Sigil do NOT make one dime from it. We do it fully on a volunteer basis to give back to the community.

Is Sigil or Calibre's Gpl licensed software (without the network clause) compatible with your AGpl license? According to what you wrote in your license I am not sure.

Can a Sigil plugin using your software be freely used by ALL Sigil users? Even the ones who use Sigil to create commercial epubs? Or a book author who uses Sigil to create his/her own epub for sale as a self-published author?

I just want to clarify this upfront.

Same questions for epubveri.

Of course the plugin itself is not shipped with Sigil and can have whatever license the plugin developer wants that is consistent with your AGpl license? Does that make any difference?

The plugins do not typically even include the source, so there us that partof the issue as well.

[Kayadelenium]

Kevin — thank you for asking this out loud instead of quietly deciding it wasn't worth the risk. It is the right question and I would rather answer it in public, once, than have people guess.

Taking your questions in the order that matters to your users rather than the order you asked them.

Can a Sigil plugin using epubsana be freely used by ALL Sigil users, including people producing commercial EPUBs, including a self-published author selling their own book?

Yes. All of them, no distinction, no fee, no permission needed, and no asterisk I am saving for later.

The reason is not generosity, it's what the license actually covers: the AGPL — like the GPL — governs copying, modifying and distributing the software. It says nothing about the data the software processes. Repairing an EPUB with epubsana no more licenses that EPUB than gcc licenses the binary it produces, or than Sigil licenses the book someone edits in it. The output is the author's, unconditionally. Sell it, ship it to a retailer, give it away — you owe me nothing and I have no claim to make.

Is GPL-3.0 compatible with AGPL-3.0?

Yes, and by an explicit clause rather than anyone's charitable reading — but let me quote it properly, because you are right to be careful and a soft summary would not survive you reading the text yourself.

GPLv3 section 13 permits linking or combining a GPLv3 work with an AGPLv3 work into a single combined work and conveying it; the GPL continues to apply to the covered part, "but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such." AGPLv3 section 13 permits the same combination from the other direction, and there the GPL work "will remain governed by version 3 of the GNU General Public License."

So the two clauses are not symmetrical, and I would rather say that plainly than tell you the network requirement politely confines itself to my half. It does not, in the GPLv3 direction.

What makes this moot for Sigil is that no combined work arises at all:

• A Sigil plugin invokes epubsana as a separate process — the only shape available to a Python plugin calling a compiled binary, and what Doitsu's does. Section 13 is about linking or combining into a single work and conveying it; it never engages.
• Sigil does not ship the plugin, so Sigil conveys nothing.
• And AGPL section 13's requirement triggers only on modifying the program and offering the modified version to users interacting with it remotely over a network. A desktop editor does neither.

What about the plugin's own license?

You asked whether it can be "whatever license the plugin developer wants that is consistent with your AGpl license". I'd push back gently on the premise: it does not have to be consistent with mine, because nothing of mine is in it. Two programs talking at arm's length stay two programs. The plugin author picks any licence at all, including a proprietary one, and neither the plugin nor Sigil inherits a thing.

Someone who instead linked the Rust crate directly into a GPLv3 program would be on the section 13 route above — but a Sigil plugin is Python calling a binary, so that shape does not arise here even in principle.

Same answers for epubveri, and in one respect a cleaner answer: identical license structure, but epubveri never writes to your book. It reads and reports. There is no path by which any part of it can end up inside an EPUB, so for the validator the "output is not covered" argument is not even needed.

epubsana does write — that is its job — so I checked what of mine ends up in your files before promising anything unconditionally. No branding or generator string, no generated identifiers or invented values, and no new files added to the container: every edit goes back into a document that was already there. The only text it authors is the structural markup a defect requires — a

<p>

wrapper around stray text, a

<!DOCTYPE html>

, a

<meta charset="utf-8"/>

, an identifier mechanically derived from one already in your book.

So what is the commercial license for?

Two situations, both about distributing or serving my code rather than using it: shipping epubsana inside a closed-source product (an e-reader, a retailer's ingestion pipeline), or running a modified version as a network service without publishing the modifications. That is the whole target. The AGPL is there because work of mine was once closed and commercialised by someone else and I got nothing back — it is aimed at that, and at nothing that happens in this thread.

On "we do not make one dime from it."

I know, and I'd have been uneasy in your position too. Sigil and calibre are the reason a lot of these books are repairable at all, and measuring my test shelf this week made that concrete in a way I did not expect. The clean part I can state as a fact: all 49 Project Gutenberg books on it produce zero errors — not one error or fatal between them. (They do draw usage-level advisories, mostly epub:type values outside the default vocabulary; nothing that makes a book invalid.)

I had a ratio here comparing calibre-produced books against the rest, and I cut it, because it does not survive being checked: the comparison group is ten books and one of them carries 93% of that group's errors, so the figure moves — and changes sign — depending on which perfectly reasonable statistic you pick. There is no stable number there yet, and I would rather say that than quote whichever version flatters my point.

What I will say without a number: the damage is upstream of your tools, not caused by them, and I am not going to charge the volunteers cleaning it up.

Rather than leave this as a forum post nobody can find in six months, I am adding a LICENSING-FAQ to the repo saying all of the above in writing, so a plugin author can link to it instead of taking my word from a thread. I will post the link when it's up. If any wording in it still leaves you unsure, tell me and I'll fix the wording — an answer you have to interpret is not an answer.

Usual disclaimer, meant plainly rather than defensively: I am not a lawyer, and the FAQ is a statement of intent and of how I read my own license, not legal advice. What is not hedged is the commitment itself — I will not come after users, plugin authors, or your project.

[Kayadelenium]

Kevin — the FAQ is up, so you can hold me to it rather than to a forum post:

https://github.com/veripublica/epubs...CENSING-FAQ.md

Same answers as above, in writing and linkable: the books you repair are yours unconditionally, commercial use of the tool needs no commercial license, an arm's-length CLI call forms no combined work so a plugin's licence is its author's choice, and the commercial licence covers only closed-source embedding or serving a modified version over a network. It quotes both section 13 clauses exactly, including the asymmetry, rather than paraphrasing them in my favour.

It covers epubveri as well as epubsana — same licence structure, and epubveri never writes to your book at all. It says explicitly that it does not modify LICENSE, and that LICENSE governs anywhere the two could be read differently.

If a plugin author ever needs a sentence in there sharpened, or you spot wording that still leaves room to worry, say so and I will change it. It is meant to be something you can point people at without having to explain it afterwards.