epublift — EPUB toolbox: modernise, shrink, kepub, validate, repair, archive

[DiapDealer]

Originally Posted by JSWolf

The thing is, there are a lot of people who are using Readers that cannot handle webp. My Kobo cannot handle webp. So if you convert the images to webp, the ePub will be broken. There is no need to change what works in all cases to one that only works in some cases.

Also, the new compression works in no cases.

I'm fairly certain you missed my point entirely.

[DiapDealer]

Originally Posted by JSWolf

The thing is, it took a long time after ePub3 came out to adopt it. The existing software did not support it. So until there was enough software to support ePub3, it was not adopted.

Epub3 was being widely used for much longer than you give it credit for. So long as it was text-based fiction (that included an ncx), epub2 systems were happy to render it. There was no need for software updates for the initial support. It was only for the advanced features of epub3 (features that most commercial text-based fiction is still not incorporating anyway, by the way) that updates were needed. People were buying and reading epub3s on their old, un-updated devices before they knew that epub3s were available. It was widely adopted once people figured out that embracing epub3 didn't mean their preferred reading material was going to be all video-ed up and jammed full of interactive JavaScript content.

For instance: back when you were still fighting the notion of all things epub3 tooth and nail (We don't need it! We don't need it!), the vast majority of new releases on Google Books were already being sold as epub3s.

[JSWolf]

But this new compression and webp are not backwards compatible with anything.

[DiapDealer]

Originally Posted by JSWolf

But this new compression and webp are not backwards compatible with anything.

I know that, Jon. YOU know I know that. I'm not certain what has given you the impression that I'm embracing all of the decisions in this new tool. I have no need of this tool, so I'm not going to get worked up over decisions that won't really effect me. I was merely inquiring as to why make the point of accommodating epub2 reading systems (RE including the NCX), only to then break that backward compatibility when converting all images to webp. Seems counter-productive to me.

[Lukusaukko]

Originally Posted by Kayadelenium

Why are there so many ebook formats?

As a gross simplification, the first formats were born of device limitations, general disagreement on what would constitute a good ebook format (DRM and other concerns), and everyone trying to lock in their corner of the business and keep others out.

Then Amazon comes along and blows everyone else out of the water with (at the time) superior device and ecosystem - at this point some companies either folded or got out of the ebook market. Now the remaining players start to think that if they can't agree on something, no one else will be making any money except Amazon, so ePub comes along. And now you have two markets, Amazon and everyone else - and Amazon can get away with having a proprietary format due to its vendor lock-in and captive market share, the others not so much.

[JSWolf]

Sony was the first to add ePub to the Sony Reader PRS-505.

[Sarmat89]

Originally Posted by Kayadelenium

Why are there so many ebook formats?

Some of those are machine-readable conversions of text/HTML for use with low-power devices and primitive storage: PDB, MOBI, AZW, LRF.
DJVU is supposed to be more compact than PDF for bicolor images.
FB2 adds metadata that can be used to catalog and manage the books automatically, and allows advanced features not found in EPUB like chapter headings, epigraphs, poetry and footnotes.
RTF allows for more adequate formatting of the book matter.

EPUB is just a comittee bastard child, which does everything poorly. The decision to made it around the obsolescent HTML format basically closed all the possible roads to improve it.

[DNSB]

Originally Posted by Sarmat89

FB2 adds metadata that can be used to catalog and manage the books automatically, and allows advanced features not found in EPUB like chapter headings, epigraphs, poetry and footnotes.

You can easily do exactly the same text display in an ePub by making use of CSS and with more flexibility than in a FB2 ebook. Chapter headings using the h(1-6) header tags plus CSS, footnotes using epub:type="noteref", role="doc-noteref", <aside> & epub:type="footnote" work rather well, epigraphs and poetry simply require the CSS for formatting the text. A quick check of my complete CSS stylesheet shows I have 4 epigraph and 7 poetry entries.

As for the metadata, IMNSHO, ePub metadata is more capable and flexible that FB2 metadata.

[Sarmat89]

Originally Posted by DNSB

Chapter headings using the h(1-6) header tags plus CSS, footnotes using epub:type="noteref", role="doc-noteref", <aside> & epub:type="footnote" work rather well

...or don't.

epigraphs and poetry simply require the CSS for formatting the text.

A lot of CSS for such basic things. Note that poetry requires setting that cannot be reproduced with CSS.

ePub metadata is more capable and flexible that FB2 metadata.

Well, both FB2 and EPUB have insufficient metadata. The issue is the EPUB tries to use some obsolete formats like MARC and DublinCore which have almost no bearing to the books. There is no concept of compilation editor, cover artist, or poetry author/translator.
The greater issue is that EPUB metadata cannot handle even such trivial things as translated books. At all.

[DNSB]

Have you even looked at the capabilities of ePub3 metadata? Heck, ePub2 allowed opf:role="trl" while ePub3 uses refines for a contributor to specify a translator or original book information

<dc:contributor id="translator01">Ann Onymous</dc:contributor>
<meta refines="#translator01" property="role" scheme="marc:relators">trl</meta>

and optionally:

<meta refines="#translator01" property="file-as">Onymous, Ann</meta>

For the original title, use a title element with a refines to indicate the original title:

<dc:title id="t1">The Metamorphosis</dc:title>
<dc:title id="t-original">Die Verwandlung</dc:title>
  <meta refines="#t-original" property="title-type">main</meta>

Yes, the src-title-info may be a bit simpler but ePub2 nevermind ePub3 are fully capable of storing metadata for a translated book.

The next question is whether any reading software can make much use of either format's metadata. I do know that a while back, people were complaining that while FB2 had the sequence metadata for series/series index, the reading programs did not seem to make use of the info.

[Kayadelenium]

Thank you all for your replies, the critical ones included. I asked because I wanted to hear your views, and I got them.

One thing I would like to make clear: none of the software I have written has ever forced anything on anyone. epublift never touches the original file; it writes a new one next to it. On the web page the image format is a choice, WebP or Keep original. Readers whose devices support WebP can upgrade. Readers whose devices do not, or who simply do not want to, keep their images as they are. I am one of them: my own Kobo Forma does not show WebP.

What stood out to me in this thread is how cautious everyone is about WebP, even though it has been a core media type since EPUB 3.3. EPUB 3.4, which the W3C is finishing now (a Candidate Recommendation Draft since 2 October) and which will probably be published next year, adds AVIF and JPEG XL. I am curious who will support them, and when. If you have a guess for your own reader or app, I would like to hear it.

I have been quiet here for a few days because I found new bugs, and I am fixing those before anything else. I have also been planning a new sub-project. I will write about both when there is something to try.

[PeterT]

Any plans on publishing epublift to cargo.io? My approach to all of you tools is doing a cargo build rather than a download of a precompiled binary; it just simplifies my update process

Sent from my SM-X238U using Tapatalk

[JSWolf]

Originally Posted by Kayadelenium

Thank you all for your replies, the critical ones included. I asked because I wanted to hear your views, and I got them.

One thing I would like to make clear: none of the software I have written has ever forced anything on anyone. epublift never touches the original file; it writes a new one next to it. On the web page the image format is a choice, WebP or Keep original. Readers whose devices support WebP can upgrade. Readers whose devices do not, or who simply do not want to, keep their images as they are. I am one of them: my own Kobo Forma does not show WebP.

What stood out to me in this thread is how cautious everyone is about WebP, even though it has been a core media type since EPUB 3.3. EPUB 3.4, which the W3C is finishing now (a Candidate Recommendation Draft since 2 October) and which will probably be published next year, adds AVIF and JPEG XL. I am curious who will support them, and when. If you have a guess for your own reader or app, I would like to hear it.

I have been quiet here for a few days because I found new bugs, and I am fixing those before anything else. I have also been planning a new sub-project. I will write about both when there is something to try.

W3C are idiots. They don't get what works and what doesn't work. If they did, they'd not be saying to use Webp. Webp does not work in more situations then it works. I've not seen a single new eBook that has any webp images. The publishers know they'd have many complaints if the images and/or the ePub didn't work. That's why they don't use Webp.

What would really help to stop the complaints is to make the defaults to keep the existing images and ZIP format for the container and make it so you need to set the options to get Webp and/or this new compression.

[KevinH]

Originally Posted by Kayadelenium

What stood out to me in this thread is how cautious everyone is about WebP, even though it has been a core media type since EPUB 3.3. EPUB 3.4, which the W3C is finishing now (a Candidate Recommendation Draft since 2 October) and which will probably be published next year, adds AVIF and JPEG XL. I am curious who will support them, and when. If you have a guess for your own reader or app, I would like to hear it.

I think you might misunderstand what recent EPub 3.x changes mean. They have been trying to merge it with common html usage (including trying and failing to remove xhtml in favour of html) to effectively "future proof" the spec itself. They unfortunately have little to no understanding of the publishing industry or end consumers and what they want, so they are guessing. They are pushing webp and webm, avif, and jxl because they guess they represent the future. Unfortunately, they completely misunderstand that the hardware base and stability required in publishing take a much longer term view.

Why would any epub publisher ever use avif or jxl, or webp in an epub when that will limit its adoption and cause customer complaints. One format, jxl, is not yet even supported out of the box on actual browsers such as Chrome. Even the very latest Chrome builds require users to enable it. That includes all the Electron and QtWebEngine apps that are based on Chrome as well.

So just because the Epub spec supports a format, do not expect the publishing community to start throwing away gif, jpeg and png any time soon. Especially for so little gain (if any).

Unfortunately like most specs done by a committee, they always end up broken by design.

[Kayadelenium]

epublift 3.3.0 (command line) and 1.24.0 (web page) are out, and https://epublift.itpax.net is already running them. Three changes are worth knowing about.

Metadata edits on EPUB 2 books are fixed

The metadata editor writes its result to a new file (book_meta.epub, or a download on the web page), so the book you started from was never touched. The new file, though, was worse than it should have been when the book was EPUB 2. The editor wrote EPUB 3 markup into it (refines and property metas, dcterms:modified), which EPUB 2 does not allow, so every edited EPUB 2 book failed validation with RSC-005. The web form sends every field when you save, so there it happened on every save. It has been like this since the metadata editor first shipped.

EPUB 2 books are now written the EPUB 2 way: opf:role and opf:file-as on the author, opf:scheme="ISBN" on the ISBN, and calibre:series for the series. If you kept a copy made with the editor, run it through Validate or epubcheck. Making the edit again from your original gives a clean file.

Saving the form also no longer loses what the form does not show: a second language, or an author's sort name and role (a translator listed as a creator now stays a translator). It no longer adds a second copy of an ISBN the book already has. EPUB 2 has no subtitle field, so for those books the subtitle box is switched off instead of writing something that would not be valid.

You can change the cover

Someone asked for this on GitHub. The Metadata tab has a new Cover section with three choices: keep the current cover, use the one an ISBN lookup found on Open Library, or upload your own JPEG or PNG. The Open Library cover is shown next to the book's own with both sizes, and you get a warning when it is smaller, which it often is. On the command line it is epublift meta set --cover cover.jpg book.epub.

The image goes in exactly as you give it. It is not converted to WebP, for the reason this thread already knows: Kobo shows no WebP cover in a plain .epub. If you want it smaller, run Optimize afterwards. Every page that shows the old cover follows the new one, including calibre's SVG cover pages, which take the new image's size so it is not stretched. A book with no cover gets one, with a cover page at the start. If the result would have more validation errors than the book you started from, nothing is written.

On a Kobo, delete the old copy from the device before copying the new one, or the library keeps showing the old cover.

Subjects from Open Library are now your choice

Open Library collects subject headings from every library that catalogued a book, in that library's language, without saying which language they are in. An English programming book came back with German and Dutch subjects next to the English ones, and I cannot sort short headings by language reliably ("Hardware" is German too). So the web page now shows the subjects it found as tick boxes, none ticked. On the command line, meta enrich lists them and only writes them with --include-subjects. If you have a script that relied on enrich writing subjects, add that flag.

Validate and Repair now run epubveri 0.24.0 and epubsana 0.25.0, the same versions as in their own threads.

The conversion problems I mentioned last time are not part of this release. I am still working on those.

Full release notes: cli-v3.3.0 and web-v1.24.0. If an edit or a new cover comes out wrong on one of your books, please tell me here. Your original is untouched, and Validate will show what changed.