epublift — EPUB toolbox: modernise, shrink, kepub, validate, repair, archive

[Kayadelenium]

Some of you have been testing epubveri and epubsana in their threads here for the last three months. I never said where they came from, so here it is.

epublift came first. I started it before either of them, as a way to shrink and modernise my own EPUBs. While building it I kept running into broken books, and I saw a gap: checking them meant epubcheck and a Java runtime, and fixing them meant opening each one in an editor and correcting the same mechanical defects by hand. That gap is why I decided to write epubveri, and then epubsana. Both grew into projects of their own, and epublift is where they come back together for an ordinary reader: one program, or one web page, that does the boring jobs on a book you already have.

What it does

• Optimize: upgrades a book to EPUB 3.3 (a nav document built from the NCX, the guide turned into landmarks, HTML5 doctypes) and re-encodes JPEG and PNG images to WebP. The NCX stays in, so EPUB 2 reading systems still find the table of contents.
• Kobo: --kepub writes a .kepub.epub with Kobo's koboSpan markup, the same idea as kepubify, which I learned it from.
• Validate and Repair: epubveri and epubsana, the same versions as in their own threads. Nothing new for those of you who already use them, just in the same place as the rest.
• Metadata: show, edit, or fill missing fields by ISBN from Open Library or Google Books. It only fills a field in the book's own language.
• Archive: packs books into a lossless .eparc to save disk (Zstandard over text and fonts, images stored as they are) and restores them content-exact. It is my own format and no reader opens it; it is for storage only.
• Import (experimental): PDF to reflowable EPUB for PDFs that have a text layer; the command-line version can also OCR scanned PDFs. A Markdown file, folder or zip works too.

Why it is safe to point at a real book

• It never writes over your file. The result is a new file next to it: book_v3.3.epub, book_repaired.epub, and so on.
• An image is only replaced if the WebP is actually smaller. Otherwise the original stays, so a book never grows.
• For a JPEG, the WebP quality is capped at the source's own estimated quality, so a poor scan is not re-encoded into more bytes with no more detail. Grayscale stays grayscale.
• Things I tried and dropped because they change what you see: deconvolution sharpening, vectorising, AI upscaling.

The first thing you will ask: WebP on e-ink

Stock Kobo shows WebP images as blank pages. I confirmed it on a Forma and a Sage. The cause is that Kobo's reading software is built on Qt 5.2.1, and Qt only gained a WebP image plugin in 5.3.0. Kindle, PocketBook and the Adobe-engine readers (ADE, Tolino, Nook) do not list WebP in their own documentation either, though I have not tested those myself.

So if you read on e-ink, use --keep-images (on the web page: Keep original). You still get the structural upgrade and your images are left exactly as they were. On the command line, --kepub keeps the original images by default.

For the tinkerers: I backported Qt 5.3.0's WebP plugin to Kobo's Qt 5.2.1. With it, WebP renders inside .kepub.epub books, but not in plain .epub library covers, which Kobo draws through a different engine. It is in the repo, unofficial and at your own risk.

Three ways to try it

• In your browser: https://epublift.itpax.net. Validate and Repair run inside the page as WebAssembly, so the book never leaves your computer. The other modes (Optimize, Metadata, Archive, Import) upload the book to my server, where it is processed in memory and deleted straight away; nothing is stored or logged. If that is not acceptable to you, the next two ways upload nothing.
• Command line: binaries for Windows, macOS and Linux (Raspberry Pi included), plus .deb and .rpm, on the releases page. The macOS and Windows binaries are not code-signed yet, so the first run shows the usual warning.
• Your own web page: the same web page runs locally with one Docker command, shown below.

docker run -d -p 127.0.0.1:8080:8080 ghcr.io/epublift/epublift-web:latest

What it does not do

• It is not an editor. Sigil and calibre are where you edit a book.
• No DRM removal.
• No AI on the public page. There is an AI OCR import (Mistral) for people who self-host and bring their own key, because with it the book goes to a third party. The public page has no key on purpose.

What would help

1. Which of your readers show WebP? The table of devices I have verified myself has two rows: Kobo no, Apple Books yes. If you have a PocketBook, a Boox, a Tolino, KOReader, or anything else, convert one book with images and tell me whether they show. That is the data I need to decide whether WebP should stay the default at all.

2. A book that came out worse. Anything that renders differently, or loses something, after Optimize. Your original is untouched, so you can open both side by side, and epubveri or epubcheck can check the result.

Repo, with a guide for each command: https://github.com/ePubLift/epublift

It is AGPL, written in Rust, no Java, no C dependencies.

[KevinH]

A few things:

1. Why change png and jpeg which are universally accepted to webp whose only advantage is web partial rendering that has no real impact on locally loaded images?

2. Why create yet another proprietary archive format? The world already has too many archive formsts. Especially when epubs are already zip archives. Why not just enable better zip compression --9 instead? Each epub should compress very little additionally and still nicely open as is on any reader.

[Kayadelenium]

Thanks Kevin, both fair questions.

1. WebP

The reason is size, not partial rendering. WebP keeps photos and PNG artwork at the same visual quality in noticeably fewer bytes, and it is not a web-only format any more: EPUB 3.3 lists it as a core media type, next to JPEG and PNG. EPUB 3.4 goes further and adds AVIF and JPEG XL, and JPEG XL can repack an existing JPEG losslessly into a smaller file that gives back the exact original JPEG. With disk prices up two or three times lately, I think every saving counts.

The gap is reading systems that claim EPUB 3 but do not render a 3.3 core media type. Kobo is the one I checked myself. Until they catch up, --keep-images keeps the originals, and the first post asks which readers do show WebP, so the advice can be precise.

2. .eparc

It is not meant to be read, only to save disk on a personal library, and it is not closed. It is an ordinary stored ZIP holding a JSON manifest, one standard Zstandard stream, and the images as they were. unzip and zstd take it apart without epublift, and the manifest gives each file's size, so a few lines of script split the stream back into files. The layout is documented here: eparc-format.md.

Why not zip -9: ZIP is only the container, and the compression inside an EPUB is deflate, which is about thirty years old. It compresses each file on its own with a 32 KB window, so it cannot use what the chapters of a book share, and -9 gains little over the default, as you say.

I measured it so you don't have to take my word for it. The text of 16 public-domain Project Gutenberg books (Alice, Frankenstein, Pride and Prejudice, Moby Dick, War and Peace, Don Quixote, the Shakespeare collection and others; 1,198 files, 24.1 MB), compressed with Python 3.14's own zlib and zstd modules, which are the standard C libraries, on an M2 Pro:

method                         size     vs deflate-6   compress   decompress
deflate -6, per file (EPUB)    9.18 MB      --          43 MB/s     701 MB/s
deflate -9, per file (zip -9)  9.15 MB     -0.3%        34 MB/s     703 MB/s
zstd -3,  per file             9.49 MB     +3.4%       280 MB/s     940 MB/s
zstd -19, per file             8.80 MB     -4.1%         6 MB/s     840 MB/s
zstd -3,  one stream per book  7.64 MB    -16.8%       282 MB/s    1177 MB/s
zstd -19, one stream per book  6.19 MB    -32.6%       4.5 MB/s    1362 MB/s

Two things stand out. Swapping deflate for Zstandard file by file gains almost nothing in size; at -3 it is even slightly larger, though it compresses six times faster. The size win comes from compressing the whole book as one stream, which is what .eparc does at -19: a third smaller than deflate, and it decompresses about twice as fast. Its cost is a slow encode, paid once when you archive.

That is also why I opened a discussion with the EPUB working group about allowing Zstandard inside the EPUB container itself, as a future option next to deflate: epub-specs discussion #3025. Until something like that exists, a book has to stay deflate to open everywhere, and .eparc is where I can use Zstandard today without breaking anything, because restore gives back an ordinary EPUB.

In most books the images are the bulk, and the archive stores them as they are, so the whole book shrinks much less than the text. Today I also found a book that comes out larger: its JPEGs carry several megabytes of embedded metadata, its EPUB had deflated them, and the archive does not. So it is marked experimental, and I am still working out what it should promise.

[JSWolf]

Don't convert graphics to webp. ADE/RMSDK does not support it. Anyone using RMSDK on a Kobo or nook or Sony Reader or any other older Reader will have trouble. Make --keep-images the default instead of having to remember to use it every time. Oh and not every firmware for a Kobo can handle webp. So you'll end up breaking things that don't need to be broken. Stupid choices for the ePub standard do not mean you have to make the same stupid choices.

.eparc is also another bad idea. Sure it might compress better, but if we actually want access to the eBook, we'd have to uncompress and there is no native Windows tool to do this. Plus. I store my eBooks in calibre libraries which doesn't know .eparc. There's nothing at all wrong with ZIP for ePub and whatever Amazon uses for their eBooks.

Why would you want a non-standard compression format made part of ePub when most people won't be able to use it. It's just going to cause all kinds of problems. Please don't. Lease the compress and graphic images alone.

Go tell the EPUB working group that you are wrong and you want to leave compression as it is. .eparc will break things.

DON'T DO THESE THINGS AS THEY WILL CAUSE ALL KINDS OF PROBLEMS THAT DON'T EXIST!

Don't convert to any image formats that cannot be opened by RMSDK on a Kobo. All you're doing it causing trouble where trouble doesn't exist.

A compression format that makes the file smaller but cannot be used is worse then a ZIP file with no compression.

Before building in these mistakes, you should have asked first if they were a good idea and we could have told you they are a very bad idea so you could leave them out.

[Kayadelenium]

To sum up where I stand, since Kevin and you raised the same worries from different sides:

• Nothing that exists is changed. epublift never touches the original: it writes a new file next to it, and your EPUB stays exactly as it was.
• Nobody is forced into anything, and that is not my intention. Nobody has to convert a book, and nobody has to use .eparc. If your books live in calibre libraries, .eparc is not for you, and that is fine.
• It is a compatibility tool for newer devices. It gives readers that support the current spec a smaller book. WebP is not a break from the standard, it is part of it: EPUB 3.3 lists it as a core media type. For readers that do not render it, RMSDK ones included, --keep-images keeps every JPEG and PNG exactly as it was. Your RMSDK and Kobo firmware reports are useful, thank you; they are the kind of thing the first post asked for.

What your post shows, though, is a problem bigger than any tool. A reader who has a WebP book come up blank did nothing wrong: the file says EPUB, the device says EPUB, and neither tells the truth about what the other needs. EPUB 2 and EPUB 3 come from two very different worlds, and they share one name and one extension. As long as the file is called .epub, nobody can tell which one it is without opening it up. Had EPUB 3 had its own extension, device makers, spec writers and readers would all have had an easier time, and a publisher could simply have released both.

I think the one thing we can fairly ask of someone who just wants to read is to know whether their device reads EPUB 2 or EPUB 3. Everything past that, image formats, font formats, 3.0 versus 3.3, is a burden for editors, publishers, device makers and people who write software. We can and should carry it. We cannot hand it to the reader.

Where I do disagree with you is that everything should stay as it is. New technology arrives all the time, and I think modern devices need a more efficient format, one that takes it in over time while the old way stays as the baseline and no existing book stops working. That has to be argued with data, including data against my own ideas: the table in my previous post shows that swapping deflate for Zstandard file by file gains almost nothing, and the gain only comes with a different structure. That is exactly what a working group should know before deciding anything.

All of this deserves its own discussion rather than a side thread about one tool, so I am opening a separate thread on what such a format should look like. I hope you will both join it.

[JSWolf]

There's no need to convert to webp just because a bunch of people said it's OK. They don't get what the real world is like. A lot of people are not using devices with software that handles webp. So that will break for a lot of people.

Also, nobody is using any software that can handle .eparc. I know I don't want my ePub written in .eparc format. I don't want to have to specify options to keep things working. The defaults should keep things working, And these things that WILL BREAK things should go away.

Also, you are trying to break thing by doing this...That is also why I opened a discussion with the EPUB working group about allowing Zstandard inside the EPUB container itself...

Why would you do this? If you can convince them, then you seriously break things. DON'T CHANGE WHAT WILL BREAK THINGS AND DOESN'T NEED TO BE CHANGED!

[Comfy.n]

I played around with epublift a while ago, given its impressive compression ratio. It would reduce some EPUB files to a tenth of their original size, IIRC. Then I found that several e-reading programs don't support WebP, like SumatraPDF. There is a long-standing MuPDF bug where maintainers reportedly stated "no intention of adding this support".

https://github.com/sumatrapdfreader/...scussions/5087

https://bugs.ghostscript.com/show_bug.cgi?id=697749

Maybe epublift can finally give them a reason to update their position.

edit to add: back in 2018, they said "WebP is not common enough to warrant adding yet another third party library dependency and code size."

[JSWolf]

Originally Posted by Comfy.n

I played around with epublift a while ago, given its impressive compression ratio. It would reduce some EPUB files to a tenth of their original size, IIRC. Then I found that several e-reading programs don't support WebP, like SumatraPDF. There is a long-standing MuPDF bug where maintainers stated "no intention of adding this support".

https://github.com/sumatrapdfreader/...scussions/5087

https://bugs.ghostscript.com/show_bug.cgi?id=697749

Maybe epublift can finally give them a reason to update their position.

But you'll never get Kobo older Readers with older versions of RMSDK to change to the latest version. So that WILL break!

Webp will break things. So it's best not to use it.

[Kayadelenium]

Comfy.n, thank you, that is exactly the kind of report I was hoping for. SumatraPDF and MuPDF go on my list of readers that do not show WebP, and since MuPDF sits underneath several other programs, it matters beyond one app.

JSWolf, I think we are closer than it sounds. Nobody deletes an old book, and nobody removes the old way of making them. Your EPUB 2 books stay EPUB 2, on your devices, for as long as you want. epublift writes a new file next to the original, and names it book_v3.3.epub precisely so nobody mixes the two up.

What I do not accept is the other half: that because some devices will never update, every new device has to stay on the old format too. New readers come out every year, and they support the current standard. Holding them to what a ten-year-old RMSDK can show, when they could have smaller books that are still standard EPUB, seems unfair to their owners.

So my answer is both: keep the old format for the devices that need it, and offer the modern one for the devices that can use it, and let each reader pick the file that fits their device. A publisher could do the same with a book.

You are right about one thing, and I will not pretend otherwise: a book made with a newer feature will not open on an old reader. I know it first-hand, because my own Kobo Forma does not show WebP either. That is true of WebP today and would be true of any new compression in the container. It is the reason old and new have to be offered side by side, and never one replacing the other.

The bigger question of what the format itself should look like has its own thread now, and I would like your view there too: If you were designing EPUB today, what would you change?

[Lukusaukko]

Originally Posted by Kayadelenium

So my answer is both: keep the old format for the devices that need it, and offer the modern one for the devices that can use it, and let each reader pick the file that fits their device. A publisher could do the same with a book.

Publishers won't support multiple formats of the same book beyond what is commercially viable. Support EPUB and AZW3 - fine, makes commercial sense. Support two different EPUB formats - no way; it's added expense, added complexity in production, and a customer support nightmare when you get complaints from a person that bought a book they can't read on their old device/software because they couldn't tell them apart, for practically no gain at all.

They also won't switch their tooling to something that supports that - they hardly support making native ebooks beyond exporting a print book from InDesign (which doesn't support WEBP), so they absolutely won't do a thing to adopt new formats when the support is fledgling and the old ones are good enough for 99.999% of the clientele.

This does feel a bit like trying to reinvent the wheel when the support still isn't wide enough to bother with, for a solution to a largely non-existant problem. Yeah, it would be nice to have lossless images instead of JPG or PNG that don't take up more space than the text content 10 times over, but the publishers largely couldn't care less, nor most users (besides, AVIF is even better than WEBP - though support is even worse).

Still, an interesting personal project no doubt, but I wouldn't expect any widespread adoption (the overwhelming majority of ebooks hardly make any use out of EPUB3-specific features, and it's over a decade-old spec).

[JSWolf]

Originally Posted by Kayadelenium

Comfy.n, thank you, that is exactly the kind of report I was hoping for. SumatraPDF and MuPDF go on my list of readers that do not show WebP, and since MuPDF sits underneath several other programs, it matters beyond one app.

JSWolf, I think we are closer than it sounds. Nobody deletes an old book, and nobody removes the old way of making them. Your EPUB 2 books stay EPUB 2, on your devices, for as long as you want. epublift writes a new file next to the original, and names it book_v3.3.epub precisely so nobody mixes the two up.

What I do not accept is the other half: that because some devices will never update, every new device has to stay on the old format too. New readers come out every year, and they support the current standard. Holding them to what a ten-year-old RMSDK can show, when they could have smaller books that are still standard EPUB, seems unfair to their owners.

So my answer is both: keep the old format for the devices that need it, and offer the modern one for the devices that can use it, and let each reader pick the file that fits their device. A publisher could do the same with a book.

You are right about one thing, and I will not pretend otherwise: a book made with a newer feature will not open on an old reader. I know it first-hand, because my own Kobo Forma does not show WebP either. That is true of WebP today and would be true of any new compression in the container. It is the reason old and new have to be offered side by side, and never one replacing the other.

The bigger question of what the format itself should look like has its own thread now, and I would like your view there too: If you were designing EPUB today, what would you change?

But the book created if broken. It doesn't work. It's compressed so it cannot be uncompressed. The images may not be displayed. And as for backwards comparability, that's gone for good. What we end up with is a seriously broken mess. Converting ePub2 to ePub3 ends up as a disaster. I get a modified eBook I cannot use at all. It's 100% useless. So why bother when what's created is not going to work for anyone?

[DiapDealer]

It would seem to me that keeping the NCX to appease epub2 rendering systems is going to be rendered effectively moot by the decision to convert images to webp.

[Kayadelenium]

Thank you all for your replies, KevinH, JSWolf, Comfy.n, Lukusaukko and DiapDealer. You disagree with me on a lot, and that is exactly what makes them valuable to me: they come from people who have made, converted and read ebooks for many years. I have read every one of them carefully, some of what you raised I am already looking into, and I will come back here with what I find.

Lukusaukko's point that publishers support EPUB and AZW3, but would never support two kinds of EPUB, left me with a question I would like to ask you, since many of you have lived through this history:

Why are there so many ebook formats?

Here is a list of formats people have used, or still use, from memory and surely incomplete:

• EPUB 2 and EPUB 3, and Kobo's KePub on top of EPUB
• Mobipocket (MOBI), AZW, AZW3 (KF8) and KFX for Kindle
• Topaz and Print Replica, also Kindle
• PDF, and DjVu for scans
• FB2 (FictionBook)
• LIT (Microsoft Reader)
• LRF and LRX (Sony BBeB)
• PDB (Palm eReader), Plucker, TCR
• iBooks Author's .ibooks
• CBZ and CBR for comics
• DAISY for accessible books
• and plain TXT, RTF, HTML and CHM

Some of these are older than EPUB, but many came after it. Each was made by someone who decided that what already existed was not enough, or was not theirs. So:

1. Why did people keep creating new formats when EPUB already existed? Was something missing technically, or was it about control: DRM, a store, a device?
2. Which of these did something better than EPUB, and what was it?
3. Which ones died, and why: the format itself, or the company behind it?
4. If you still convert to something other than EPUB today, what does it give you that EPUB does not?

I ask because I think the answer says a lot about where EPUB should go next. The wider discussion about that is in its own thread, and every view is welcome there too: If you were designing EPUB today, what would you change?

[JSWolf]

Originally Posted by DiapDealer

It would seem to me that keeping the NCX to appease epub2 rendering systems is going to be rendered effectively moot by the decision to convert images to webp.

The thing is, there are a lot of people who are using Readers that cannot handle webp. My Kobo cannot handle webp. So if you convert the images to webp, the ePub will be broken. There is no need to change what works in all cases to one that only works in some cases.

Also, the new compression works in no cases.

[JSWolf]

KF8, KFX, Topaz, and Print Replica are Amazon because they didn't want to use an ePub. KF8 is Amazon's answer to ePub. KFX is Amazon's answer to their DRM having been broken. iBooks is ePub. All the others came before ePub.

The thing is, it took a long time after ePub3 came out to adopt it. The existing software did not support it. So until there was enough software to support ePub3, it was not adopted.

So what needs to be done is the software needs to come out and be widely available before your changes can be implemented. You need to software first. If you move ahead before the software is read, nobody can read what you've created.