epubsana — automatic repair for what epubcheck reports (not an editor)

[Kayadelenium]

Thanks, DNSB — that's the distinction exactly.

Jon, it is the same reasoning: the file-size check stayed out of epubveri not because another tool has it, but because it isn't an EPUB spec rule. epubsana follows the same line — it only repairs what the spec says is wrong. It doesn't remove the Adept.resource line; it renames value to content so the element becomes valid and the book keeps its information. If you want the line gone altogether, Modify ePub is the right tool for that, and the two don't conflict.

[JSWolf]

Originally Posted by Kayadelenium

Thanks, DNSB — that's the distinction exactly.

Jon, it is the same reasoning: the file-size check stayed out of epubveri not because another tool has it, but because it isn't an EPUB spec rule. epubsana follows the same line — it only repairs what the spec says is wrong. It doesn't remove the Adept.resource line; it renames value to content so the element becomes valid and the book keeps its information. If you want the line gone altogether, Modify ePub is the right tool for that, and the two don't conflict.

Renaming the Adept.resource line changes it and it then no longer is what it is. It should either be removed or left alone. I see no reason to change it.
It causes no error. No program (that I know of) has an issue with it. So why not just leave it as is like it should be?

[Kayadelenium]

epubsana 0.25.0

https://github.com/veripublica/epubs...es/tag/v0.25.0

Some things don't change between releases. Your original file is never modified: the repaired copy is written beside it. Every fix is shown to you before it is made, and the run ends with a report of exactly what changed. The browser version needs no install and uploads nothing: https://veripublica.github.io/epubsana/

My last post was for 0.20.0. Five releases have gone out since, and this post covers all of them. There are three new repairs. There are also two problems I need to tell you about plainly, and one of them may affect books you have already repaired.

New repairs

• The mimetype entry's header. If the mimetype file at the start of the ZIP carries an optional "extra field" in its header, the book is invalid. epubsana now writes that entry again without the field, using the same fix that already moves mimetype to the front. The content of the entry does not change.
• Empty folders inside the ZIP. A folder entry left behind with nothing in it, such as OEBPS/fonts/, is an error. epubsana now leaves it out of the archive. It does not do this when removing the folder would leave another folder empty.
• Empty extra identifiers. An empty dc:identifier with no id is removed, as long as the book keeps a real identifier. The identifier the package names as the book's own is never touched, even when it is empty, because choosing a book's identity is the author's decision.

A book was called valid when it was not (fixed in 0.22.0)

Before it repairs anything, epubsana checks your book. Until 0.22.0 it did that check on the book as it would write it out, not on the file you gave it. Its ZIP writer drops that optional extra field from every entry's header. So the mimetype error above could never show up in the report. On my library, eight books had it and nobody saw it, and three of them were reported valid, with exit status 0, while the file was not. The check before repair now reads the book exactly as it arrived. epublift found this by comparing epubsana's results with its own.

A bad repair: no-break spaces (fixed in 0.24.0)

In EPUB 2 books, epubsana wraps loose text sitting directly in the body in a div. Every version before 0.24.0 treated a no-break space at the end of that text as ordinary whitespace and left it outside the div. epubcheck counts a no-break space as text, so each one left behind was a new error in the repaired book. epubsana did not catch this, because the validator it checks itself with, epubveri, only began reading a no-break space as text at 0.22. One book on my library ended up with about 2,100 of these errors. The fix now keeps those spaces inside the wrapper.

If you repaired EPUB 2 books with an earlier version, check the repaired copies with epubcheck or a current epubveri. If they show stray text errors, run 0.25.0 on the repaired copy and it will wrap the leftover spaces.

A crash (fixed in 0.24.0)

A link with a % followed by a non-ASCII letter, such as %Bölüm.xhtml in a table of contents, crashed the run. When that happened the book was not repaired and no file was written. Now it is left as written, like any other invalid escape.

Reading names the way epubcheck does (0.25.0)

• A file name and a link to it can be written in different Unicode forms. macOS writes ö as an o followed by a combining mark, while a table of contents may write it as one character. epubcheck treats the two as the same file, and the repairs that look up a linked document now do too.
• A link with escapes in it, such as my%20pic.jpg, is now recognised as pointing at its file. Before, a manifest entry could be dropped as unused while a link like that still pointed at it.
• When an invalid id is renamed, the new id keeps every character an XML name allows and nothing the validator rejects, so a rename can no longer produce an id that is itself invalid.
• The book's identifier is compared the way epubcheck compares it. In particular, a no-break space counts as part of the value.

None of these changed a single repair on my own library. They matter for books that have these shapes.

For plugin authors and library users

epubsana now requires epubveri 0.24.0. The JSON format has not changed. The new repairs report fix_id fix.empty_directory and fix.empty_extra_identifier, and the mimetype header repair reports under the existing fix.mimetype_packaging.

The ask

Same as always. epubsana checks its own repairs with epubveri, so it cannot see damage epubveri cannot see. The no-break space problem is exactly that kind of damage. If a repair leaves one of your books worse, please tell me here. The same goes for a book that is still reported valid when epubcheck disagrees.

crates.io: https://crates.io/crates/epubsana. Command-line binaries for Windows, macOS and Linux are on the releases page.

[fogice]

How are people actually using this tool? What's a good step-by-step procedure for taking a book tagged by epubveri, running it through epubsani, and replacing the file in Calibre while keeping the reading progress recorded by Kobo Utilities?

[Kayadelenium]

Originally Posted by fogice

How are people actually using this tool? What's a good step-by-step procedure for taking a book tagged by epubveri, running it through epubsani, and replacing the file in Calibre while keeping the reading progress recorded by Kobo Utilities?

There is no calibre plugin that runs epubsana yet; the current calibre plugins only validate. One is close, but I will not put a date on it. Until then it is a manual round trip, and this is the one I would use. It does not depend on which reader you use.

1. Find the books. The epubveri library plugin lists defects with the number of books carrying each. Clicking a row filters calibre's book list to those books.

2. Repair a copy outside the library. Right-click the book, then Open containing folder, to get the path of its EPUB. epubsana never modifies its input. By default it writes Book_fixed.epub beside it, which would leave a stray file inside the calibre library folder, so give it an output path elsewhere:

epubsana -i "/path/to/library/Author/Title (123)/Title - Author.epub" --dry-run
epubsana -i "/path/to/library/Author/Title (123)/Title - Author.epub" -o ~/Desktop/fixed.epub

The first command lists every proposed fix and changes nothing. The second asks you about each fix, one at a time. Add --auto-safe to apply the provably safe ones without asking. Exit status 0 means the result is valid. If you would rather not install anything, the browser version does the same in the page: https://veripublica.github.io/epubsana/

3. Check the result with epubveri or epubcheck before it goes anywhere near the library.

4. Before replacing, save the reading position, if a plugin for your device can store it in calibre, so you have a copy whatever happens next.

5. Replace the format, not the book. Keep the same book selected, then Add books, then Add files to selected book records, and pick the fixed file. calibre offers to replace the existing EPUB. The book record stays the same, so its id, metadata and custom columns stay too, including any a device plugin writes to. From the command line: calibredb add_format 123 ~/Desktop/fixed.epub, which replaces an existing EPUB unless you pass --dont-replace.

6. Send it to the device again. As long as title and author have not changed, calibre writes it to the same path on the device. Open the book. If the position is gone or wrong, restore the copy from step 4.

One thing to know about the position. epubsana never renames or removes a file inside the book: every repair rewrites an existing file under its own name. So the chapter your reader remembers is still there. The exact spot inside a chapter can still shift if a repair changed that chapter's markup, and the dry run in step 2 shows which files a repair touches. That is why step 4 comes before step 5.

If you try this on a batch, I would be glad to hear where it was awkward. That is exactly what the plugin should take away.

Until the plugin is out, the most helpful thing anyone can do is try books in the browser version, https://veripublica.github.io/epubsana/, and check the repaired copy with both epubcheck and epubveri. A book where either of them disagrees with the result, or where the repair made something worse, is exactly what I need to hear about. It would make my job very much easier.

[fogice]

Ah, Step 5 was what I was missing. Thanks.

[Kayadelenium]

epubsana 0.26.0

https://github.com/veripublica/epubs...es/tag/v0.26.0

As before: your original file is never modified, and the repaired copy is written beside it. Every fix is shown to you before it is made, and the run ends with a report of exactly what changed. The browser version needs no install and uploads nothing: https://veripublica.github.io/epubsana/

This release has two new repairs and one stricter check. The stricter check is the part you may need to act on.

New repairs

• SVG attribute names that lost their capitals. Some tools treat a book's XHTML as HTML and write viewbox instead of viewBox, or preserveaspectratio instead of preserveAspectRatio. A browser still understands the lowercase names, but a reading system that reads the file as XML ignores them, so a cover image can lose its scaling. epubsana now gives each name back its capitals, taken from the HTML standard's own table of these names. Only the name changes. Before it offers the fix, it checks the result with the validator, and it leaves out any rename the validator would still reject.
• A text direction written in capitals. calibre sometimes writes dir="LTR" on the body. A browser reads that as ltr, but an EPUB 2 book has to say ltr in lowercase. epubsana now lowercases LTR and RTL and changes nothing else. Any other value is left alone.

Both are offered as safe fixes, the kind --auto-safe applies without asking.

A stricter check on EPUB 2 books

epubsana checks books with epubveri, and it now requires epubveri 0.26.0. Since its 0.25.0 release, epubveri checks the style, script, meta and base elements of an EPUB 2 book the way epubcheck does. For example, a style element with a class attribute, or with no type attribute, is now an error. kepubverto found this, because its converter wrote such a style element and epubveri did not flag it.

epubsana does not repair these errors yet. So a book that has them was reported valid after repair before, and it is not now: the run ends with exit status 1 and the errors stay in the report. None of the books in my own library have these errors, so this changed nothing for me. If you repaired EPUB 2 books with an earlier version, a current epubveri or epubcheck will show you whether any of them are affected.

Not built

Some calibre-split chapters (the ..._split_001.html files) have no title element at all. epubsana fills an empty title from the book's table of contents or the chapter's first heading, but these files have neither: the table of contents names only the first part, and the file opens with a paragraph. Copying a title from a neighbouring file would be a guess, so these stay reported.

For plugin authors and library users

The JSON envelope now says "convention": "0.7", because epubsana follows version 0.7 of the veripublica conventions. Nothing else in the output changed. The new repairs report fix_id fix.svg_attribute_case and fix.dir_case.

The ask

Same as always. epubsana checks its own repairs with epubveri, so it cannot see damage epubveri cannot see. If a repair leaves one of your books worse, please tell me here. The same goes for a book that is still reported valid when epubcheck disagrees.

crates.io: https://crates.io/crates/epubsana. Command-line binaries for Windows, macOS and Linux are on the releases page.

[JSWolf]

If there is no title element, just use the book's title. That works. Using the TOC entry does not work because there may or may not be a ToC entry. Also, in ePub2, the title should be the name of the book. In ePub3, I change the title garbage to be the name of the book.

[Turtle91]

Actually, I’m pretty sure accessibility standards require chapter titles - not just book title - to be in the <title>. That would cover both ePub 2 and 3.

Sooo, I would retrieve the title in the following priority:
1) from the ToC if it’s available
2) then header tag at the top of the page
3) then book title
4) leave empty if the others don’t work and notify

[Kayadelenium]

Originally Posted by JSWolf

If there is no title element, just use the book's title. That works. Using the TOC entry does not work because there may or may not be a ToC entry.

Thank you, that was a good argument, and it is in epubsana 0.27.0, out today.

https://github.com/veripublica/epubs...es/tag/v0.27.0

As before: your original file is never modified, and the repaired copy is written beside it. Every fix is shown to you before it is made, and the run ends with a report of exactly what changed. The browser version needs no install and uploads nothing: https://veripublica.github.io/epubsana/

Titles

In my last post I said that a title copied from somewhere else would be a guess. The book's own title is not a guess, though: it is the book's own words, and a title element is metadata a reader never sees in the chapter. So epubsana now does this:

• An empty title element is filled from the table of contents entry for that file, or the file's first heading, or, when the book has neither, the book's title from its metadata. Before, that last case was left alone.
• A head with no title element at all now gets one, from the same three places in the same order. This covers the calibre split files from my last post and many word-processor files. On EPUB 3 the same thing is only a warning, and it is fixed the same way.

I kept the table of contents first. You are right that many files have no entry, and those now get the book's title. But where there is an entry, it names that file and not the whole book, so I think it is the better title.

Turtle91's order in #54 is exactly the one 0.27.0 uses: the table of contents, then the file's first heading, then the book's title. His last step is there too. When the book has none of the three, the title is left as it is and the finding stays in the report, so you are told. Both repairs are shown to you with the exact text before anything is written. epubsana does not replace a title that already has text in it. It cannot tell a generated title from one the author meant, so that stays a choice for the editor.

Also new

• An NCX navPoint with no id gets a new one that is unique in the NCX. Nothing can point to an id that did not exist, so nothing else changes.

A stricter check you may need to act on

epubsana now requires epubveri 0.27.0, which checks the character encoding of every XML file in the book, as epubcheck does. Before, it checked only the package file. A chapter saved as Latin-1 or Windows-1252, or an SVG saved as UTF-16, is now an error. epubsana does not repair encodings yet, so such a book now ends with exit status 1 and the error stays in the report. Converting a file to UTF-8 is only safe when its declared encoding is right, and a file labelled Latin-1 that is really UTF-8 is common. None of the books in my own library have a file that is not UTF-8, so I have nothing to test a repair on yet. If you have such a book, I would like to hear about it.

For plugin authors

With --format json, a file that cannot be read, or is not an EPUB, now gives a JSON envelope with "status": "error" and exit status 2. Before, stdout was empty. When epubsana cannot run at all (a usage error, an output file that already exists without -f), stdout is still empty, as before. The new repairs report fix_id fix.missing_title and fix.ncx_navpoint_id.

The ask

Same as always. epubsana checks its own repairs with epubveri, so it cannot see damage epubveri cannot see. If a repair leaves one of your books worse, please tell me here. The same goes for a book that is still reported valid when epubcheck disagrees.

Command-line binaries for Windows, macOS and Linux are on the releases page.

https://crates.io/crates/epubsana
https://www.npmjs.com/package/@verip.../epubsana-wasm

[JSWolf]

One thing I've seen a number of times for an ePub3 eBook is <title>Chapter 1: The name of the book</title>.

In epub2, it is supposed to be the name of the book in <title>. The list of how you are doing the <title> is incorrect for epub2.

[Kayadelenium]

@JSWolf, I looked into this before answering, because if EPUB 2 required it, the order would be wrong. I could not find that rule, and here is what I found instead.

The EPUB 2 specification. OPS 2.0.1 says nothing about what the title element in a content document should hold. One of its own examples shows an HTML head holding <title>Chapter 2</title>. The phrase "the title of the publication" is in the OPF specification, and it describes dc:title in the package file. That is where EPUB 2 puts the book's name.

What books do. I counted the title elements in the books in my own library. Among the EPUB 2 books, 75 mostly use the book's title, and 233 mostly use something else. Another 128 leave them mostly empty. So there are two habits, and neither is the rule. Your "Chapter 1: The name of the book" pattern is there too, and it is more common in my EPUB 3 books than in my EPUB 2 ones.

Why the table of contents comes first. epubsana only fills a title that is empty or missing. It never replaces one that has text in it. When it fills one, the table of contents entry names that file, while the book's title is the same in every file and already sits in dc:title. So the file's own name tells a reader more, and the book's title is the fallback when nothing names the file.

Where I could be wrong is reading systems. If a reader shows the title element where it should show the book's name, for example in a header, then a chapter name there would look wrong. Which reading systems do that? If you can name one, I will test it, and then the order for EPUB 2 is worth another look.

[Kayadelenium]

epubsana 0.28.0 is out today.

https://github.com/veripublica/epubs...es/tag/v0.28.0

As before: your original file is never modified, and the repaired copy is written beside it. Every fix is shown to you before it is made, and the run ends with a report of exactly what changed. The browser version needs no install and uploads nothing: https://veripublica.github.io/epubsana/

SVG element names

A tool that treats XHTML as HTML lowercases the camelCase names in SVG. epubsana 0.26.0 already repaired the attributes (viewbox for viewBox). The same tools also lowercase element names: lineargradient, clippath, textpath, foreignobject. A reading system that reads the file as XML does not know those elements, so a cover can lose its gradient or its clipping path.

epubsana now renames each of them, in the start tag and the end tag, to the spelling in the HTML standard's own table of these names. That table is what a browser uses to read them back, so the repair makes the file say what it already means in a browser. Only the name changes. Its attributes and contents stay as they are.

Before it proposes anything, epubsana tries the renames on a copy of the book and asks epubveri about the result. It leaves a rename out in two cases:

• The restored name is still not allowed there. feDropShadow, for example, does not exist in the SVG version EPUB 2 uses.
• The rename makes the file worse in another way. A textPath has to sit inside a text element, so a lowercased one sitting anywhere else would become a new error once its name is fixed. epubsana then leaves every rename in that file alone.

A name that is not in the table, such as a plain typo, is left alone too, because it has no single correct spelling.

Also fixed

The SVG attribute repair from 0.26.0 skipped an SVG written with a prefix (s:svg instead of svg), without saying so. It now repairs those too.

Tested on test books only

None of the books in my own library has a lowercased SVG element name, so this repair has been tested only on books I built for it. If you have a book where it applies, I would like to hear how it went.

A stricter check you may need to act on

epubsana now requires epubveri 0.28.0, which checks the names of MathML attributes, as epubcheck does. Before, any attribute on a MathML element passed. A title attribute on mi, for example, or a lowercased definitionurl, is now an error. epubsana does not repair these, so such a book now ends with exit status 1 and the error stays in the report. None of the books in my own library changed under the new check.

For plugin authors

The new repair reports fix_id fix.svg_element_case. It is AutoSafe, like the attribute repair, so --auto-safe applies it without asking.

The ask

Same as always. epubsana checks its own repairs with epubveri, so it cannot see damage epubveri cannot see. If a repair leaves one of your books worse, please tell me here. The same goes for a book that is still reported valid when epubcheck disagrees.

Command-line binaries for Windows, macOS and Linux are on the releases page.

https://crates.io/crates/epubsana
https://www.npmjs.com/package/@verip.../epubsana-wasm

[JSWolf]

I've been reading up on <title> and it should be something descriptive. It should not be <title>c01</title>. It should be <title>Chapter 01</title>. Some programs that read the ePub aloud read the title before that HTML file.

[Kayadelenium]

Originally Posted by JSWolf

I've been reading up on <title> and it should be something descriptive. It should not be <title>c01</title>. It should be <title>Chapter 01</title>. Some programs that read the ePub aloud read the title before that HTML file.

Agreed, and that is the order epubsana fills an empty title in. It takes the file's entry in the book's table of contents first, then the file's own first heading, and only then the book's title, which you argued for earlier. So a chapter gets the most specific name the book itself gives it. The book's title is the last resort, for a file nothing else names.

What it does not do is replace a title that is already there, such as c01. No validator reports that title as a defect, and picking a better name is a choice for whoever edits the book. epubsana only repairs what epubveri reports.

Which read-aloud programs read the title? If you can name one, I will test how it handles each of those three sources.