epubsana — automatic repair for what epubcheck reports (not an editor)

[Kayadelenium]

Thanks, DNSB — that's the distinction exactly.

Jon, it is the same reasoning: the file-size check stayed out of epubveri not because another tool has it, but because it isn't an EPUB spec rule. epubsana follows the same line — it only repairs what the spec says is wrong. It doesn't remove the Adept.resource line; it renames value to content so the element becomes valid and the book keeps its information. If you want the line gone altogether, Modify ePub is the right tool for that, and the two don't conflict.

[JSWolf]

Originally Posted by Kayadelenium

Thanks, DNSB — that's the distinction exactly.

Jon, it is the same reasoning: the file-size check stayed out of epubveri not because another tool has it, but because it isn't an EPUB spec rule. epubsana follows the same line — it only repairs what the spec says is wrong. It doesn't remove the Adept.resource line; it renames value to content so the element becomes valid and the book keeps its information. If you want the line gone altogether, Modify ePub is the right tool for that, and the two don't conflict.

Renaming the Adept.resource line changes it and it then no longer is what it is. It should either be removed or left alone. I see no reason to change it.
It causes no error. No program (that I know of) has an issue with it. So why not just leave it as is like it should be?

[Kayadelenium]

epubsana 0.25.0

https://github.com/veripublica/epubs...es/tag/v0.25.0

Some things don't change between releases. Your original file is never modified: the repaired copy is written beside it. Every fix is shown to you before it is made, and the run ends with a report of exactly what changed. The browser version needs no install and uploads nothing: https://veripublica.github.io/epubsana/

My last post was for 0.20.0. Five releases have gone out since, and this post covers all of them. There are three new repairs. There are also two problems I need to tell you about plainly, and one of them may affect books you have already repaired.

New repairs

• The mimetype entry's header. If the mimetype file at the start of the ZIP carries an optional "extra field" in its header, the book is invalid. epubsana now writes that entry again without the field, using the same fix that already moves mimetype to the front. The content of the entry does not change.
• Empty folders inside the ZIP. A folder entry left behind with nothing in it, such as OEBPS/fonts/, is an error. epubsana now leaves it out of the archive. It does not do this when removing the folder would leave another folder empty.
• Empty extra identifiers. An empty dc:identifier with no id is removed, as long as the book keeps a real identifier. The identifier the package names as the book's own is never touched, even when it is empty, because choosing a book's identity is the author's decision.

A book was called valid when it was not (fixed in 0.22.0)

Before it repairs anything, epubsana checks your book. Until 0.22.0 it did that check on the book as it would write it out, not on the file you gave it. Its ZIP writer drops that optional extra field from every entry's header. So the mimetype error above could never show up in the report. On my library, eight books had it and nobody saw it, and three of them were reported valid, with exit status 0, while the file was not. The check before repair now reads the book exactly as it arrived. epublift found this by comparing epubsana's results with its own.

A bad repair: no-break spaces (fixed in 0.24.0)

In EPUB 2 books, epubsana wraps loose text sitting directly in the body in a div. Every version before 0.24.0 treated a no-break space at the end of that text as ordinary whitespace and left it outside the div. epubcheck counts a no-break space as text, so each one left behind was a new error in the repaired book. epubsana did not catch this, because the validator it checks itself with, epubveri, only began reading a no-break space as text at 0.22. One book on my library ended up with about 2,100 of these errors. The fix now keeps those spaces inside the wrapper.

If you repaired EPUB 2 books with an earlier version, check the repaired copies with epubcheck or a current epubveri. If they show stray text errors, run 0.25.0 on the repaired copy and it will wrap the leftover spaces.

A crash (fixed in 0.24.0)

A link with a % followed by a non-ASCII letter, such as %Bölüm.xhtml in a table of contents, crashed the run. When that happened the book was not repaired and no file was written. Now it is left as written, like any other invalid escape.

Reading names the way epubcheck does (0.25.0)

• A file name and a link to it can be written in different Unicode forms. macOS writes ö as an o followed by a combining mark, while a table of contents may write it as one character. epubcheck treats the two as the same file, and the repairs that look up a linked document now do too.
• A link with escapes in it, such as my%20pic.jpg, is now recognised as pointing at its file. Before, a manifest entry could be dropped as unused while a link like that still pointed at it.
• When an invalid id is renamed, the new id keeps every character an XML name allows and nothing the validator rejects, so a rename can no longer produce an id that is itself invalid.
• The book's identifier is compared the way epubcheck compares it. In particular, a no-break space counts as part of the value.

None of these changed a single repair on my own library. They matter for books that have these shapes.

For plugin authors and library users

epubsana now requires epubveri 0.24.0. The JSON format has not changed. The new repairs report fix_id fix.empty_directory and fix.empty_extra_identifier, and the mimetype header repair reports under the existing fix.mimetype_packaging.

The ask

Same as always. epubsana checks its own repairs with epubveri, so it cannot see damage epubveri cannot see. The no-break space problem is exactly that kind of damage. If a repair leaves one of your books worse, please tell me here. The same goes for a book that is still reported valid when epubcheck disagrees.

crates.io: https://crates.io/crates/epubsana. Command-line binaries for Windows, macOS and Linux are on the releases page.

[fogice]

How are people actually using this tool? What's a good step-by-step procedure for taking a book tagged by epubveri, running it through epubsani, and replacing the file in Calibre while keeping the reading progress recorded by Kobo Utilities?

[Kayadelenium]

Originally Posted by fogice

How are people actually using this tool? What's a good step-by-step procedure for taking a book tagged by epubveri, running it through epubsani, and replacing the file in Calibre while keeping the reading progress recorded by Kobo Utilities?

There is no calibre plugin that runs epubsana yet; the current calibre plugins only validate. One is close, but I will not put a date on it. Until then it is a manual round trip, and this is the one I would use. It does not depend on which reader you use.

1. Find the books. The epubveri library plugin lists defects with the number of books carrying each. Clicking a row filters calibre's book list to those books.

2. Repair a copy outside the library. Right-click the book, then Open containing folder, to get the path of its EPUB. epubsana never modifies its input. By default it writes Book_fixed.epub beside it, which would leave a stray file inside the calibre library folder, so give it an output path elsewhere:

epubsana -i "/path/to/library/Author/Title (123)/Title - Author.epub" --dry-run
epubsana -i "/path/to/library/Author/Title (123)/Title - Author.epub" -o ~/Desktop/fixed.epub

The first command lists every proposed fix and changes nothing. The second asks you about each fix, one at a time. Add --auto-safe to apply the provably safe ones without asking. Exit status 0 means the result is valid. If you would rather not install anything, the browser version does the same in the page: https://veripublica.github.io/epubsana/

3. Check the result with epubveri or epubcheck before it goes anywhere near the library.

4. Before replacing, save the reading position, if a plugin for your device can store it in calibre, so you have a copy whatever happens next.

5. Replace the format, not the book. Keep the same book selected, then Add books, then Add files to selected book records, and pick the fixed file. calibre offers to replace the existing EPUB. The book record stays the same, so its id, metadata and custom columns stay too, including any a device plugin writes to. From the command line: calibredb add_format 123 ~/Desktop/fixed.epub, which replaces an existing EPUB unless you pass --dont-replace.

6. Send it to the device again. As long as title and author have not changed, calibre writes it to the same path on the device. Open the book. If the position is gone or wrong, restore the copy from step 4.

One thing to know about the position. epubsana never renames or removes a file inside the book: every repair rewrites an existing file under its own name. So the chapter your reader remembers is still there. The exact spot inside a chapter can still shift if a repair changed that chapter's markup, and the dry run in step 2 shows which files a repair touches. That is why step 4 comes before step 5.

If you try this on a batch, I would be glad to hear where it was awkward. That is exactly what the plugin should take away.

Until the plugin is out, the most helpful thing anyone can do is try books in the browser version, https://veripublica.github.io/epubsana/, and check the repaired copy with both epubcheck and epubveri. A book where either of them disagrees with the result, or where the repair made something worse, is exactly what I need to hear about. It would make my job very much easier.

[fogice]

Ah, Step 5 was what I was missing. Thanks.