epubveri – standalone EPUB validator

[Kayadelenium]

Originally Posted by JSWolf

Would it be possible to have a warning when an HTML file is too large for the mobile limit?

Before I answer, I want to be sure I understand what you are after, because "the mobile limit" could mean more than one thing. Which limit do you have in mind, and where does it bite: which reading app, device or retailer, and what actually happens when a file is over it? And is this for books you are making yourself, or for checking books that come to you from elsewhere?

[JSWolf]

Originally Posted by Kayadelenium

Before I answer, I want to be sure I understand what you are after, because "the mobile limit" could mean more than one thing. Which limit do you have in mind, and where does it bite: which reading app, device or retailer, and what actually happens when a file is over it? And is this for books you are making yourself, or for checking books that come to you from elsewhere?

Most Readers cannot handle an ePub that is one HTML file. The Sony Reader PRS-500 which was the first Reader to have ePub could not. So it was decided to split the ePub into sections (i.e., chapters). Most chapters are small enough.

If you go too large, it may slow down your Reader or not work properly in some way. So it would be good to know if your HTML file is too large.

If I am correct, the uncompressed size is 300kb and the compressed size is 100kb.

[DiapDealer]

Originally Posted by JSWolf

If I am correct, the uncompressed size is 300kb and the compressed size is 100kb.

260Kb uncompressed was the number I remember.

[JSWolf]

Originally Posted by DiapDealer

260Kb uncompressed was the number I remember.

That's the size calibre's checker uses to say if an HTML is too large or not. So I think that is most likely the correct figure.

[Kayadelenium]

Thank you both, that settles which limit it is. 260 KB uncompressed is calibre's number in two places: its EPUB output splits HTML files at that size by default, citing Adobe Digital Editions, and Check Book in the editor flags anything larger as "File too large", at info level, with the advice to split the file.

That also answers my own question, though. It is a limit of some reading systems, not of EPUB: a 400 KB chapter is a perfectly valid EPUB. Everything epubveri reports, including the optional --advisory notes, is meant to be true of the book whichever reader opens it. A size warning is true only of some readers, which makes it a different kind of statement, and I would rather not mix the two.

The good news is that, for anyone making books in calibre's editor, the warning you are asking for already exists: Check Book gives exactly this one, at exactly 260 KB. Sigil has no such warning, but Tools / Reports (Ctrl+Shift+R) lists every HTML file with its size in a column you can sort, so the large ones are one click away.

[JSWolf]

if you are checking your ePub in your library, you'll get the message about the file size and can then deal with it if you want.

[JSWolf]

Can epubsana have the ability to splt HTML that are larger then 260KB?

[Kayadelenium]

epubveri 0.17.5 is out: the same reports, and much faster now.

On the 474 books I test against, a full run went from 48 seconds to 19, and every book gets exactly the report it got before. The biggest single gain was a book that went from 3.6 seconds to 0.23. Very little of it came from faster code. Most of the time was going on repeated work: one book's files were being decompressed 4,777 times, every chapter re-read the shared stylesheet, and a notes file that every chapter links to was parsed once per chapter, 232 times in one book. Each of those is now done once.

The trail started with a careful profile from a user who runs epubveri over a 2,798-book library (GitHub issue #137). Thank you for it.

One small fix rides along: a book whose manifest lists the same file twice could get a slightly different report from one run to the next. It now gets the same one every time.

If you use one of our Sigil or calibre plugins, there is nothing to do; they pick up the new version themselves. Everything else is on the releases page.

[Kayadelenium]

Originally Posted by JSWolf

if you are checking your ePub in your library, you'll get the message about the file size and can then deal with it if you want.

I see: the point is to find the large files across the whole library without opening each book in the editor, which Check Book can't do. That is a request for the calibre library plugin rather than for epubveri itself, and I have noted it there. No promise yet on whether or when, but it is written down.

Originally Posted by JSWolf

Can epubsana have the ability to splt HTML that are larger then 260KB?

That one is a question about what epubsana is for, so I'll think it through on that side and come back to you here.

[DNSB]

Originally Posted by JSWolf

Can epubsana have the ability to splt HTML that are larger then 260KB?

Originally Posted by Kayadelenium

That one is a question about what epubsana is for, so I'll think it through on that side and come back to you here.

I would prefer not to have automatic splitting. A report that a large file exists and splitting is recommended if fine by me but from experience trying calibre's autosplitting, I prefer a human eyeball in the loop.

Also as mentioned in the calibre epubveri library thread, the time for a 17703 book library went from 1hr 29 min in my first run (single core) to 10 min 48 seconds in the second run (14 core) to 3 minutes 5 seconds (14 core).

[JSWolf]

Originally Posted by DNSB

I would prefer not to have automatic splitting. A report that a large file exists and splitting is recommended if fine by me but from experience trying calibre's autosplitting, I prefer a human eyeball in the loop.

Also as mentioned in the calibre epubveri library thread, the time for a 17703 book library went from 1hr 29 min in my first run (single core) to 10 min 48 seconds in the second run (14 core) to 3 minutes 5 seconds (14 core).

I would have the auto-splitting to be optional.

[Kayadelenium]

@JSWolf, @DNSB: I promised to come back on splitting. epubsana will not split files, not even as an option. That is by design: epubsana only repairs defects that epubveri reports, and only when there is exactly one correct fix that keeps the content intact. A chapter over 260 KB is valid EPUB, because the limit belongs to some reading systems and not to the format. Where to cut is a decision about the book, as DNSB says, and a split has to rewrite every link, TOC entry and manifest item around the cut. That is a conversion, not a repair. calibre already does it on output, and Sigil's Split at Cursor and Split at Markers are the right tools when you want to choose the cut yourself.

DNSB, thank you for the library timings: 17,703 books in about three minutes on 14 cores.

[Kayadelenium]

epubveri 0.18.0 is out: GitHub release, crates.io and npm. The Sigil and calibre plugins pick it up on their own.

This release is about matching epubcheck on the parts of EPUB that most books never use. I went through epubcheck 5.4.0's rules one by one and built a small test book for each, and ran both tools on it. That took over two hundred books: EPUB Indexes, EDUPUB, Dictionaries, Previews, collections, region-based navigation, ARIA roles, and the package document's own rules. Where epubveri answered differently, it now gives the same message IDs on the same lines as epubcheck.

What this means for your books:

• If your books do not use those features, nothing changes. My own shelf of 474 books gives exactly the same reports as with 0.17.5.
• If a book does use them, it may now get errors it did not get before. They are the errors epubcheck already reports. That is why this is 0.18 and not 0.17.6.
• noscript is now rejected in EPUB 3, and an audio or video element may have a src attribute or source children, not both. Both follow epubcheck.
• RSC-020 now reports every malformed URL that epubcheck rejects, not only URLs with spaces.

Four wrong findings are gone. I found these myself while comparing; none came from a user. A refines pointing into a content document (ch1.xhtml#p) drew a warning. The teacher-edition rule fired on books that are not EDUPUB. Two dictionary rules ran in books that are not dictionaries. And a package document broken before its root element drew a wrong OPF-030. epubcheck reports none of these, and now neither does epubveri.

After this release, epubveri and epubcheck give the same message IDs on all 687 expected messages in epubcheck's own test suite, and on all 209 of W3C's conformance publications.

I found one case where epubcheck itself skips a document: when the spine's toc attribute points at something that is not an NCX. I reported it upstream as w3c/epubcheck#1735. epubveri reports OPF-050 there and checks the document as usual.

[Kayadelenium]

@JSWolf, you brought splitting up again in the Sigil plugin thread (#57, #59). I am answering here, because what epubveri does and does not do belongs in this thread. The answer is the same as in #357: neither epubveri nor epubsana will split files. Not now, and not as an option.

The warning you are after already exists. calibre's Edit Book has it built in: Check Book reports every HTML file larger than 260 KB in an EPUB as "File too large" and suggests splitting it. The Quality Check plugin reports large files too, as DNSB showed in the Sigil thread (#53). In Sigil, Tools / Reports lists every HTML file with its size.

The products are kept apart on purpose. styloria checks CSS, epubveri validates, and epubsana repairs what epubveri reports. Each one has one job and stays inside it. That is not something peculiar to my tools; it is how most of this ecosystem works. epubcheck validates and does not repair. Ace by DAISY checks accessibility and nothing else. Sigil edits, calibre converts and manages libraries. A tool with one job can be trusted with that job, and you can combine it with the others.

Splitting is not validation, and the EPUB specification sets no size limit on content documents. The 260 KB figure comes from older reading systems such as the Adobe RMSDK you mentioned, not from the format. The page break you tracked down on the Kobo is a real problem, but it is that reader's problem, and a validator reports what the format says.

If I ever decide to build a splitting tool, it will be a separate product with its own name. It will not be part of epubveri or epubsana.

[JSWolf]

When checking the eBooks in calibre (not the editor), it would be good to know if any of the eBooks have HTML files that are too large. calibre's check book does not work outside the editor. The thing is, do you really want to make us have to run Quality Check before or after running epubvari? That's adding more work to check eBooks. It's best to have everything in one app to run once.