Veripublica epub-test-books: small EPUBs for testing EPUB tools

[Kayadelenium]

I've started a library of small EPUB files for testing EPUB tools: validators, converters, fixers and reading apps. Each book shows one thing, either a rule from the EPUB or CSS specifications or a defect I have met in real books, and nothing else. When a tool gets a book wrong, you can see exactly what it got wrong.

I made it because I was testing my own tools against shelves of commercial books. They can't be shared, they take a long time to scan, and the one thing you are looking for hides among a thousand others. W3C's EPUB tests already cover reading systems well. These books are mostly for the tools that check or change books, and many of them are broken on purpose.

The rule I care about most is where the answer comes from. For every book, I write the expected result from the specification, quoting the sentence, before any tool runs. Then I run epubcheck and epubveri, my own validator, on it and keep their output next to the book. When a tool disagrees with the specification, the book says so, and that includes epubveri.

Every book starts as a public issue on GitHub with the question, the clause and the expected result. The discussion happens there, and the issue closes when the book is in. If you think a result is wrong, say so under its issue, or here, and I will carry it over. When I change a result, the release notes say what changed and why, and the old outputs stay as they were, so anyone can see where I was wrong and how it was fixed.

Three releases are out. The latest, v0.2.0, has the two templates, one for EPUB 3 and one for EPUB 2, and 22 sets on the EPUB container: damaged ZIP files, the mimetype file, file names, container.xml and its rootfile. Each release is a tag you can pin. The books are ready-made .epub files, listed on one index page with what each should give and what each validator says, and you can download them one by one or all together in a single zip. They are CC BY 4.0, so you can copy them into your own test suite with one line of credit.

The container section is not finished, and that is partly why I'm posting now. I have left out every case where the specification and the validators disagree, and listed them in one issue: open questions. Most of them come down to one question: should a book say invalid when the specification says so and no validator reports it? I would rather settle that with people who read the specification closely than on my own.

If you have a book that broke your tool, a rule you think nobody checks, or a reading of the specification you disagree with, post it here or open an issue. You don't need to build anything. I'll make the book.

https://github.com/veripublica/epub-test-books

[Kayadelenium]

v0.5.0 is out. When I opened this thread there were 22 sets, and now there are 224, plus the two templates.

The container section has grown to 33 sets, with encryption.xml and font obfuscation added. The new sections are the EPUB 3.3 package document (56 sets), navigation document (37), publication resources (31) and content documents (31), and the EPUB 2.0.1 package and NCX (36). Most of the books are broken on purpose, but 36 sets are expected to be valid, so they also catch a tool that reports something the specification allows.

Every book is on one index page, with what it should give and what each validator says. The release has them all as downloads, one by one or in a single zip, so there is nothing to build.

Nine books now disagree with epubcheck 5.4.0. In five it reports nothing where the specification says invalid, and in four it reports an error where the specification says valid. Each of those books quotes the clause and links the issue I opened on epubcheck. These are my readings of the specification, and they are open for anyone to dispute, including epubcheck's maintainers.

No book is marked against epubveri, my own validator, but don't read much into that. Building these sets turned up a long list of gaps in it, and I fixed them before this release. epubcheck hasn't had that chance.

So far nobody but me has checked the expected results of the 191 new books, and that is where I could use the most help. If a result looks wrong to you, say so here or under the book's issue. If I change it, the release notes will say what changed and why.

The open questions from my first post are still open. The cases the specification calls invalid and no validator reports are still left out. I'm holding them until 15 November, so anyone who reads them differently has time to say so. The cases with no clear rule stay out until W3C answers.

Not built yet: fixed layout, EPUB 2 content documents, CSS beyond what EPUB 3.3 itself restricts, and books encrypted with AES. If you would like one of those sooner, or you have a case of your own, tell me and I'll make the book.

https://github.com/veripublica/epub-...ses/tag/v0.5.0