kepubverto — EPUB to KEPUB for Kobo, without breaking the book

[Kayadelenium]

Some of you know epubveri, epubsana and epublift from their threads here. This is the fourth member of the family.

While I was building those three, I converted a lot of books to KEPUB, with kepubify and with epublift's own Kobo conversion, and I checked every result with epubveri. Again and again, a book that was valid before the conversion had errors after it. kepubify is a good program, and it is the reason most of us know what a KEPUB is; I learned the format from it. But I wanted a converter I could trust with any book, so I wrote kepubverto. It is built around one rule: it never makes a book worse.

What it does

It gives a book what a Kobo needs for reading statistics, faster page turns and the proper progress bar, and nothing else. Your text and the rest of the book are left as they were. The span ids are the same as kepubify's, so if you switch, your reading position and highlights should carry over.

kepubverto converts; it does not check or repair. That is the job of epubveri and epubsana, and the planned Sigil and calibre plugins will run them together. kepubverto is built so that a valid book stays valid, and I test that on my whole shelf before every release. On my Kobo, reading position and the progress bar work, both on a converted book alone and at the end of my epublift chain (validate, metadata, optimize, convert).

Why it is safe to point at a real book

• It never writes over your file. book.epub becomes book.kepub.epub next to it.
• An existing output file is replaced only if you ask for it (-f).
• A chapter it cannot convert safely is left exactly as it was, and it tells you which one.
• The web page converts inside your browser. The book is not uploaded anywhere.

How to use it today

Until the plugins exist:

• In your browser, nothing to install: https://veripublica.github.io/kepubverto/ Drop an EPUB on the page and download the KEPUB.
• Command line: binaries for Windows, macOS and Linux on the releases page. The macOS and Windows binaries are not code-signed, so the first run shows the usual warning. Or, with Rust installed: cargo install --locked kepubverto

kepubverto -i book.epub

What it does not do (yet)

kepubify's optional extras (smarten punctuation, hyphenation, extra CSS and the rest) are not in this first version, and I have not decided yet which of them will come. It does not add a dummy title page, and it does not change the book's metadata or cover.

What would help

1. Your device. I have tested on my own Kobo. If yours is a different model or firmware, convert a book and tell me whether position, progress and page turns behave. If you switch a book you already read with kepubify, I would especially like to know whether your highlights and last position come through.

2. A book that came out worse. Anything that opens, looks or behaves differently after conversion. Your original is untouched, so you can compare the two, and epubveri or epubcheck will check the result.

3. Which kepubify option you miss most. That decides which one comes first.

Repo: https://github.com/veripublica/kepubverto

It is AGPL, written in Rust, no Java, no C dependencies.

[JSWolf]

Hyphenation is the most feature important to add next.

What does kepubverto do that the KePub conversion in the KoboTouch driver doesn't do?

[DNSB]

For my uses, I only used kepubify to add the kobo spans, etc. For hyphenation, smartening punctuation, CSS and find/replace are items that I use other tools for.

Since my preferred layout for reading is left justified, hyphenation is more or less a matter of theoretical interest to me.

Currently, I am not using the calibre option to send kepubs to my Kobo Libra Colour. Unless you change settings, the RMSDK renderer is now used for DRMmed ePubs only with Kobo's TENG renderer being used for kepub, epub and LCP DRMmed ePubs.

As for what TENG stands for? I haven't seen an answer from Kobo but I've seen Tolino ENGine renderer and The Epub Next Generation renderer as possible answers.

[Kayadelenium]

kepubverto 0.1.1 is out. It fixes a mistake in the markup kepubverto adds to a book. If you converted EPUB 2 books with 0.1.0, I suggest converting them again. Your original EPUBs were never touched, so this takes a minute: run 0.1.1 or the web page on the original and replace the .kepub.epub.

What was wrong

kepubverto adds one small style element to the head of each chapter. It removes the top and bottom margins of the wrapper the Kobo lays pages out in. In 0.1.0 that element had an attribute that EPUB 2's XHTML does not allow there. The books read normally on a Kobo, so this is about validity, not about what you see. But epubcheck reported one error for every converted chapter of an EPUB 2 book, and that breaks the one rule this tool is built around. In EPUB 3 books the same element had an attribute that HTML calls obsolete. epubveri counts that as a usage note, not an error, but it still raised the book's total.

0.1.1 writes the element the way each EPUB version allows. Nothing else changed: the spans and their ids are the same as before. On my shelf, a conversion now adds no findings at any level, usage notes included. An EPUB 2 book converted with 0.1.1 reads fine on my Kobo.

The same mistake elsewhere

kepubverto is not the only converter with this mistake. I tried two other KEPUB converters on a valid EPUB 2 book, and both still write a style element that EPUB 2 does not allow, in every chapter. My guess is that nobody has noticed, because the books read fine on a Kobo and the error only shows up when you validate the result. This is the reason kepubverto exists: other converters leave errors in books, and kepubverto tries not to. When it fails at that, as it did here, I fix it.

Why my tests missed it

I check every conversion with epubveri, and epubveri did not check that attribute in EPUB 2. epubcheck does. epubveri 0.25.0 checks it now. kepubverto's tests and my shelf run also count usage notes now, not only errors, because the total is the number a reader looks at.

Updating

• Web page: https://veripublica.github.io/kepubverto/ already runs 0.1.1. Reload it if you have it open.
• Command line: binaries on the release page, or cargo install --locked kepubverto

To check a book yourself, run epubveri (0.25.0 or newer) or epubcheck on it, before and after converting.

JSWolf: thank you for the questions about hyphenation and calibre's KEPUB conversion. Right now I am fixing bugs. When that is done, I will go through requests like these one at a time and answer here.

[Kayadelenium]

kepubverto 0.2.0 is out. It adds hyphenation, the first of kepubify's optional extras.

https://github.com/veripublica/kepub...ses/tag/v0.2.0

If you do not use the new option, nothing changes: a book converted with 0.2.0 comes out exactly as it did with 0.1.1, so there is no reason to convert your books again. As before, your original file is never modified, and the web page converts inside your browser without uploading anything.

Hyphenation

On the command line, --hyphens auto turns hyphenation on and --hyphens none turns it off. On the web page, the same choice is a menu above the drop area: "as the book says", "on" or "off". Without the option, the book's own CSS decides, as it always did.

• auto hyphenates the text but leaves headings and table cells unbroken.
• none turns hyphenation off everywhere, even where the book's CSS turns it on.

The Kobo breaks the words itself, with its dictionary for the book's language. kepubverto only adds the CSS that asks for it, written the way each EPUB version allows, so a valid book stays valid with either setting. I tested both on my Kobo Forma, with a Turkish book: auto hyphenates, and none stops a book that hyphenates on its own.

JSWolf: you asked for hyphenation first, so it came first. You also asked what kepubverto does that the KEPUB conversion in calibre's KoboTouch driver does not. To answer that properly, I ran calibre 9.16's conversion with the driver's default settings on every book on my shelf, and checked each result with epubveri. I checked a few valid books with epubcheck as well. Here is what I found:

• Validity. calibre's conversion added errors to almost every book. In EPUB 2 books it puts an id on the style elements it adds and a properties attribute in the package file, and EPUB 2 allows neither. In EPUB 3 books it adds a script to every chapter without declaring it in the package file. Every valid book I checked with epubcheck was invalid after calibre's conversion and still valid after kepubverto's. kepubverto did not add a single finding to any book, usage notes included.
• What changes. calibre writes every chapter and the package file out again, and it adds a script file to the book. kepubverto only adds the spans, the two wrapper elements, one style rule and, if you ask for it, the hyphenation CSS to the chapters. Every other file in the book, the package file included, stays as it was.
• Span ids. calibre numbers its spans differently from kepubify, and kepubverto numbers them the way kepubify does. So reading positions and highlights in a book that calibre converted may not carry over to the same book converted by kepubverto or kepubify, and the other way round.
• Chapters it cannot convert. kepubverto leaves such a chapter untouched and tells you which one and why.
• No calibre needed. It works in the browser, on the command line, or as a library inside other programs.

As far as I know, these books read fine on a Kobo either way. The difference is in what a validator sees and in what the book carries afterwards. If you find a book that reads differently on your Kobo depending on which of the two converted it, I would like to see it.

For plugin and script authors

--format json prints the family's shared machine-readable report: where the output went, how many chapters were converted, and one entry for each chapter left as it was, with a code. The codes are listed in the README.

What would help

Hyphenation on other Kobo models and firmware, and in languages other than Turkish. And, as before, which kepubify option you miss most now that hyphenation is in.

• Web page: https://veripublica.github.io/kepubverto/ already runs 0.2.0. Reload it if you have it open.
• Command line: binaries for Windows, macOS and Linux on the release page, or cargo install --locked kepubverto

https://crates.io/crates/kepubverto
https://www.npmjs.com/package/@verip...epubverto-wasm

[patrik]

It would be helpful if --help started with showing version number.

[JSWolf]

@Kayadelenium Thanks. I'll give it a go.

[Kayadelenium]

Originally Posted by patrik

It would be helpful if --help started with showing version number.

Thanks for the suggestion. The version has its own option: kepubverto -V, or --version, prints it along with the build it came from.

kepubverto -V
kepubverto 0.2.0+99140b0

The part after the plus is the commit the binary was built from, so it pins down exactly which code you are running. --help lists -V among its options, so that is where it points you.

I am keeping the two apart on purpose: --help says how to use the tool, -V says which version it is. Many of the command-line tools I use every day split them the same way. cargo, rustc, git and curl do not show a version in their help; cargo -V, rustc -V, git --version and curl -V print it. epubveri and epubsana, kepubverto's sister tools, work the same way, so the version is found the same way in all three.

JSWolf: thanks, I would like to hear how it goes on your Kobo.

[JSWolf]

I think its not a good idea to have --help not displaying the version. It should not be difficult to display the version with --help. Just because other programs get it wring is no reason for kepubverto to also get it wrong.

[JSWolf]

It's working so far.

[Kayadelenium]

Originally Posted by JSWolf

It's working so far.

Thanks for trying it. If anything looks wrong on your Kobo, a page that reads oddly or a highlight that does not land where it should, I would like to hear about it.

Originally Posted by JSWolf

I think its not a good idea to have --help not displaying the version.

I understand, and some tools do print it there. I am still keeping it in one place: kepubverto -V. Its sister tools, epubveri and epubsana, do the same, so whatever reads the version, a person or a plugin, finds it the same way in all three. --help lists -V, so it is one step away.

A word on why kepubverto exists at all. There are several tools that turn an EPUB into a KEPUB. When I validated what they produced, books that passed validation before conversion failed it afterwards. I doubt many people check: the book opens on the Kobo, so it looks fine. I wrote kepubverto because I wanted a conversion that leaves a valid book valid.

That is the rule for the whole family, epubveri, epubsana and kepubverto: your original file is never touched, the output is written to a new file, and the output has to follow the EPUB spec. The goal is not just to produce something, it is to produce something without errors. I hold even the reference tool to that: this week I found places where epubcheck does not do what the EPUB spec asks, and reported them upstream (w3c/epubcheck issues 1742 and 1743).

So here is a challenge for anyone with a Kobo. Take a few books you know well. Convert each one with kepubverto and with the tool you use now. You do not need the command line for any of it: both steps run in the browser, and your book never leaves the page.

• Convert with kepubverto: https://veripublica.github.io/kepubverto/
• Validate the original and both outputs with epubveri: https://veripublica.github.io/epubveri/ A finding that is in an output but not in the original was added by the conversion.
• Read on the device. Do pages break in the same places? Does the reading position hold when you close and reopen the book? Do highlights stay where you put them?
• Watch the battery. Does one version drain it faster over the same chapters?

On the command line, the validation is one line:

epubveri -i book.epub -i book.kepub.epub -i other.kepub.epub

epubcheck gives you a second opinion on the same files.

I have measured validation on my own test shelf, where kepubverto's conversion added no findings in 562 books. I have not measured battery or page layout. If kepubverto turns out worse at any of this, tell me here, with the book or just what you saw, and I will fix it straight away. If it is the other tool that breaks your books, now you will know.

A small release is out, 0.2.3. It moves kepubverto to the newest version of epubveri, the validator it shares code with. Conversion is unchanged: on my test shelf every converted book comes out byte for byte the same as with 0.2.2, so there is no need to update if 0.2.2 works for you.

https://github.com/veripublica/kepub...ses/tag/v0.2.3
https://crates.io/crates/kepubverto
https://www.npmjs.com/package/@verip...epubverto-wasm

[JSWolf]

The problem could be that I may not find something wrong if there is because I also have NickelTypeFix installed.

[JSWolf]

How does kepubverto compare to kepubify? That too is an external standalone ePub > KePub converter that does not use calibre.

[Kayadelenium]

Originally Posted by JSWolf

The problem could be that I may not find something wrong if there is because I also have NickelTypeFix installed.

That is fair for what you see on the screen: a patch that changes how the reader draws text can hide a difference, or cause one. Two things are not affected by it. Validation looks at the file, not the device, so a finding the conversion added shows up the same whatever is installed on your Kobo. And if you read both conversions on the same Kobo, with the same patch, a difference between them still comes from the conversion.

Originally Posted by JSWolf

How does kepubverto compare to kepubify? That too is an external standalone ePub > KePub converter that does not use calibre.

kepubify is the converter I measure against, and I learned the format from it. Both add the same things a Kobo needs: the koboSpan markup, the two wrapper divs and one style rule. Where they differ:

• Same span ids. kepubverto numbers its spans the way kepubify does, so a book you already read in a kepubify conversion should keep its reading position and highlights. On my test shelf the spans are identical in 543 of 562 books. The rest are documents kepubify misreads, or places where a span would make the book invalid and kepubverto leaves it out, keeping kepubify's ids everywhere else.
• How the chapter is read. kepubify reads each chapter with an HTML parser and writes it out again. EPUB chapters are XHTML, and the two do not always agree: an element written as <div/> is complete in XHTML, but an HTML parser takes it as left open, and the rest of the chapter ends up inside it. kepubverto reads the chapter as XML, as the EPUB spec requires, and only inserts its markup. Every other byte stays as it was: DOCTYPE, entities, line endings, whitespace.
• Validation. The re-writing is where kepubify's output picks up errors that the original did not have, for example a span inside an element that does not allow one, or a style element that is not valid in an EPUB 2 book. On my shelf, kepubverto's conversion added no findings in 562 books, graded with epubveri before and after.
• What kepubify has that kepubverto does not, yet. Several books or a whole folder in one run (kepubverto takes one book per run), smart punctuation, find and replace, extra CSS, the fullscreen fixes, the dummy title page and the cover change in the OPF. Hyphenation is in, as --hyphens auto or none. The other options will come one at a time, each with its own release.

If you already use kepubify, the README has a table that maps each kepubify option to kepubverto's: https://github.com/veripublica/kepub...-from-kepubify The details of every difference, book by book, are in docs/SPANS.md.

The quickest way to see it on your own books is still the one from my last post: convert a book with both, then drop the original and the two outputs on https://veripublica.github.io/epubveri/

https://crates.io/crates/kepubverto
https://www.npmjs.com/package/@verip...epubverto-wasm

[JSWolf]

I did check the code from kepubverto and looking at the hyphenation code, how does this code work to enable hyphens?

</style>
<style type="text/css">
* {
  -webkit-hyphens: auto;
  hyphens: auto;
}
h1, h2, h3, h4, h5, h6, td {
  -webkit-hyphens: none !important;
  hyphens: none !important;
}
</style>