epubcheck, any software can pass its validiation check?

[kovidgoyal]

Allow me to construct a little gendanken to illustrate things for people that need examples to understand abstract reasoning:

Say we want to create an EPUB e-book. Say it has 50 chapters each of which is in a nice separate XHTML file. Now suppose we've edited all the files and seen they look really pretty in a nice WYSWYG HTML editor. Now it's time to test our EPUB. Imagine two parallel realities.

In reality A our EPUB creator (lets call him Mr. X) was dropped on his head when he was a baby and so has an unholy reverence for XML schemas and thinks they are the cure for all evils.

In reality B our EPUB creator was dropped on his head twice so he's forgotten all his reverence, holy or unholy.

Finally, let's stipulate that Mr. X has a finite amount of time, say two hours to check his EPUB file before reaching his publishing deadline.

Reality A

Mr. X imediately fires up epubcheck, because he believes, with all the fervor of a true believer that if epubcheck passes his EPUB, he's golden.

epub check spits out lots and lots of lines, say about a 1000 that look like this:

ERROR: mybook.epub(8): could not parse content/index_cr_2.html: duplicate id: top
and this message is repeated 50 times.

Then there are messages like

ERROR: mybook.epub(12): attribute "name" not allowed at this point; ignored

repeated say a 1000 times

So our conscientious Mr. X is horrified. My God, my EPUB will never work he says. So he spends an hour googling to figure out what those error messages actually mean. He realises he has to, horror-of-horrors actually use a text editor to fix those problems. So he painstakingly edits each file by hand (he's never learned to use regexps) and by dint of sheer determination manages to make all these error messages go away. He now has 30 seconds left to meet his deadline. But epubcheck says all is well, so he quickly fires up firefox and submits his book to an online distributor. The distributor in turn runs epubcheck, which says nothing and so the book is released to an unsuspecting public.

The next day Mr. X finds his INBOX filling up with emails from disappointed readers. One gy says he tried to open the epub in stanza and all the links in the epub didn't work. Another guys says he tried to use the table of contents on his sony reader and it took half an hour to load. A third guys says that on his PDA the text in some chapters runs of the screen. A fourth guy complains that the text on the dedication page seems to start at the middle of the page and run off the right.

epubcheck really saved the day for Mr. X

Switch to reality B

Mr. X saves his epub file then immediately opens it in say the calibre viewer, where everything loos good. But being wise, he then opens it up in the desktop ADE where he sees the dedication page is wrong and the text runs of the screen in a few chapters where he has used a <table> to layout text. So he quickly fires up his editor and fixes the table, replacing it with a simple linear layout. Then he googles a bit and learns that ADE doesn't support text-anchor="middle" for SVG text (even though it is perfectly valid as per the SVG spec) so he changes his dedication page to use simple text.

Then he opens his epub on his SONY reader and finds the Table of Contents take forever to load. He googles some more and learns that for some weird reason, if he uses anchors in the toc.ncx, then the sony reader preloads all the files before rendering the table of contents. So he quickly goes and remove the anchor, which is rather useless anyway (though perfectly valid) because all his chapters are in separate files anyway.

Phew, now his book is looking like it might work. But then he remembers Stanza, so he loads his file onto an iPod and discovers none of the hyperlinks work, they all take his to incorrect locations. Horrified he googles some more and figures out that stanza can't handle files in different sub-direcories that have the same file name. Oh boy! Now he has to rename all his files and all the links that point to them. A panicky half an hour later, that's done.

Finally, Mr. X remembers that his best friend (who was dropped on his head only once) recommended he use epubcheck. But he looks at his clock and sees he doesn't have any time to do that anymore. So he fires up firefox and submits his file to the distributor. In this reality, the distributor's tech guy is really lazy so he doesn't have the system setup to run epubcheck. The EPUB is therefore released without any further ado.

The next day Mr. X's inbox fills up with emails from happy readers telling him how his book has changed their lives and how they were able to read it on their iPods, and sony readers and PDAs all with no problems what so ever.


I hope that clears up just why I object to epubcheck. And let me say that if unlike Mr. X, you have infinite time to proof your epubs, by all means use epubcheck.

[Jellby]

Fortunately, I was dropped on my head three times, so I'm in reality C. I run epubcheck, learn that attribute "name" is not supported in XHTML and should be changed to "id", and make my ids unique, and learn it for my next books. After my book passes epubcheck I'm fairly confident my book is valid ePUB, and now start checking for software glitches. I have a quick look in a web browser, and in calibre, then upload it to my Cybook and read the book. After some days or weeks of swearing over ADE's glitches, stupid Cybook's margins, incomplete Unicode font support, etc. (this is the time it takes me to "proofread" the book), I have a book that I find satisfying enough and that, to the best of my knowledge, should work in sufficiently-standards-conformant ePUB readers. Even if I know it's not perfect (no smallcaps in ADE, slow TOC in Sony, no SVG cover in you-name-it...), I decide to publish it as is, and if someone thinks it's crap, they can blame their reading software and write them hate mail... after all, my book is published for free, with no DRM, and everyone is free to change it

Seriously, though, I think both viewpoints are important, we should care about validation and following specs, and we should also care about real-world performance. Mr. X should have woken up two hours earlier, so he would have had time to make both kinds of check.

[kovidgoyal]

Originally Posted by Jellby

Seriously, though, I think both viewpoints are important, we should care about validation and following specs, and we should also care about real-world performance. Mr. X should have woken up two hours earlier, so he would have had time to make both kinds of check.

I agree, he should have. But in the real world he won't. And so in a resource constrained environment, it's important that he not be deluded into thinking that running epubcheck should be his top priority. Which is all I'm trying to convey. Run epubcheck by all means, but please, please realize there's a lot more to producing a book that will render as you want it to, than doing an XML schema check. Indeed, running an XML schema check is just about the least important.

[charleski]

Anyone who can't craft a simple regexp to fix the problems you describe has no business taking money from a publisher to create ebooks in the first place. I'd say that in reality A epubcheck has done us all a favour by exposing his incompetence and getting him fired .

No-one is saying that epubcheck is enough on its own. If you only have a limited amount of time, then sure, there are other ways of testing that should take precedence. But I have to take you back to my first reply to this thread - epubcheck is about future-proofing. What happens in 5 years' time when some xhtml standards committee decides that the 'name' attribute now specifies the number of sparkly lights to surround a character? Publishers need to take the long-view. They need to minimise the chance that they'll have to spend a ton of money in the future getting someone to correct a ton of errors that shouldn't have been made in the first place.

[kovidgoyal]

If at any time in the next 50 years it so happens that someone produces a widely used EPUB reader that fails to render a document because of a name attribute that is where it shouldn't be, I will print out this entire thread, and eat it.

[Jellby]

Originally Posted by kovidgoyal

Run epubcheck by all means, but please, please realize there's a lot more to producing a book that will render as you want it to, than doing an XML schema check.

On the other hand, flooding the net with valid ePUB files that display deficiently in current reading systems could be a way to force software developers into improving the reading systems

[kovidgoyal]

Originally Posted by Jellby

On the other hand, flooding the net with valid ePUB files that display deficiently in current reading systems could be a way to force software developers into improving the reading systems

It could also be a way of ensuring EPUB dies an early death.

[Peter Sorotokin]

Originally Posted by kovidgoyal

If at any time in the next 50 years it so happens that someone produces a widely used EPUB reader that fails to render a document because of a name attribute that is where it shouldn't be, I will print out this entire thread, and eat it.

Kovid, don't make statements like that, please. I can hear devil wispering in my ear: "only a small Easter egg... and such fun for everyone!"

[kovidgoyal]

Originally Posted by Peter Sorotokin

Kovid, don't make statements like that, please. I can hear devil wispering in my ear: "only a small Easter egg... and such fun for everyone!"

Ah an evil developer, now there's a prospect to give my stomach the cramps.

[ATimson]

Originally Posted by kovidgoyal

Ah an evil developer, now there's a prospect to give my stomach the cramps.

There are non-evil developers (asks a developer)?

[Chang]

Maybe this goes under this thread..

I was wondering why my ePubs pass the epubcheck 1.0.5 validation but with some books ADE gives a warning "The document appears to have minor errors that might cause it to be displayed incorrectly". This doesn't even happen every time with same ePub, only sometimes.

Does someone know what's the logic behind ADE's warning because afaik it uses epubcheck validator also or am I totally wrong?

[pdurrant]

Originally Posted by Chang

Maybe this goes under this thread..

I was wondering why my ePubs pass the epubcheck 1.0.5 validation but with some books ADE gives a warning "The document appears to have minor errors that might cause it to be displayed incorrectly". This doesn't even happen every time with same ePub, only sometimes.

Does someone know what's the logic behind ADE's warning because afaik it uses epubcheck validator also or am I totally wrong?

ADE does it's own validation in addition or possibly instead of using epubcheck.

Originally Posted by kovidgoyal

If at any time in the next 50 years it so happens that someone produces a widely used EPUB reader that fails to render a document because of a name attribute that is where it shouldn't be, I will print out this entire thread, and eat it.

Reminds me of the days when I worked for a company producing online courses and was doing all the html/JavaScript/ASP/DB coding and always went through hell trying to produce a product that would work with multiple versions of NN and IE.

Anyway I do try and run my files through epubcheck before uploading here and this seems to be a good place to ask a couple of questions that have been annoying me.

First I am running under Windows XP and would like to have the epubcheck output write to a file so that I have the written output and also can use a BAT file to run it instead of typing the whole command at a command prompt each time. The trouble is that a BAT file of:

java -jar epubcheck-1.0.5.jar colomba.epub > verifyout.txt

produces the file verifyout.txt with content consisting only of:

Epubcheck Version 1.0.5

Whereas the actual output should continue:

ERROR: colomba.epub: length of first filename in archive must be 8, but was 9

Check finished with warnings or errors!

I always get this error, but have yet to figure out what it means.

[pdurrant]

Originally Posted by Hamlet53

First I am running under Windows XP and would like to have the epubcheck output write to a file so that I have the written output and also can use a BAT file to run it instead of typing the whole command at a command prompt each time. The trouble is that a BAT file of:

java -jar epubcheck-1.0.5.jar colomba.epub > verifyout.txt

produces the file verifyout.txt with content consisting only of:

Epubcheck Version 1.0.5

Whereas the actual output should continue:

ERROR: colomba.epub: length of first filename in archive must be 8, but was 9

Check finished with warnings or errors!

I always get this error, but have yet to figure out what it means.

I think that epubcheck writes error reports out to stderr rather than stdout, which would explain why you're only seeing the version statement.

The error means just what it says. The first file in the epub must be called mimetype, must not be compressed and must contain just the characters application/epub+zip.

Either you've mis-named the mimetype file, or you're not zipping your epub up correctly.

[DaleDe]

Originally Posted by pdurrant

I think that epubcheck writes error reports out to stderr rather than stdout, which would explain why you're only seeing the version statement.

The error means just what it says. The first file in the epub must be called mimetype, must not be compressed and must contain just the characters application/epub+zip.

Either you've mis-named the mimetype file, or you're not zipping your epub up correctly.

Actually the error is cryptic since it would have been much more instructive to have said the first file must be named mimetype. If it is only checking the file name length then it is doing the user a disservice in passing a possibly bad archive that happens to have a filename that is 8 characters long but not mimetype.

Dale