[Plugin] ePub3-itizer - epub3 output plugin for Sigil

[JonathanMagus]

Just tried this with interesting results. Very impressive though with some minor errors:

Starting with a completely valid ePUB2, the ePUB3 that was generated from it had the following issues:

The original contained two different contributor roles in addition to a creator role in the .opf file. In the ePUB3 these were replicated with the same ID (refines="#contrib0"), so came up as an error in ePUBcheck.

Something odd about the generated nav.xhtml document. Unzipping the ePUB3 file there it is called nav.xhtml, however, it doesn't open showing content in an editor (Bluefish). Accessing the file without unzipping shows jumbled errors in the ol tags of the nav.xhtml file and the same is reported by ePUBcheck. Opening in ViewPorter (a Sigil clone?) shows the mark-up, however, says it is called toc.xhtml, although that may be a quirk of that particular software.

Anyway, the point here is that the ordered list in the TOC doesn't come out right after the conversion.

Also, the toc.ncx was retained in the ePUB3, however, registered an error with ePUBcheck because of the DOCTYPE. Not sure if that is correct or not.

Do I understand correctly that Sigil will become ePUB3 compatible in a future release? It would be fantastic if it does. I have been discussing making Bluefish, which toggles well with Sigil, more ePUB3 friendly for detailed editing of component files recently and perhaps it would make sense to coordinate this?

Thanks for the work on this so far.

[KevinH]

Hi Jonathon,

Originally Posted by JonathanMagus

J

The original contained two different contributor roles in addition to a creator role in the .opf file. In the ePUB3 these were replicated with the same ID (refines="#contrib0"), so came up as an error in ePUBcheck.

Yep, that's a bug I will fix that in the next release.

Something odd about the generated nav.xhtml document. Unzipping the ePUB3 file there it is called nav.xhtml, however, it doesn't open showing content in an editor (Bluefish). Accessing the file without unzipping shows jumbled errors in the ol tags of the nav.xhtml file and the same is reported by ePUBcheck. Opening in ViewPorter (a Sigil clone?) shows the mark-up, however, says it is called toc.xhtml, although that may be a quirk of that particular software.

Anyway, the point here is that the ordered list in the TOC doesn't come out right after the conversion.

I can not recreate your issue with a valid toc.ncx file. So can you please post (or pm me) with the toc.ncx (from Sigil) for an ebook that ends up with a messed up nav.xhtml. The created nav.xhtml is called is "nav.xhtml" not "toc.xhtml" and will be found right beside the content.opf file, not in the Text folder.

Also, the toc.ncx was retained in the ePUB3, however, registered an error with ePUBcheck because of the DOCTYPE. Not sure if that is correct or not.

That is actually a bug in epubcheck3. You need that valid doctype for fallback on epub2 readers. There is a bug fix for this in the epubcheck4 (alpha or beta version?). You can find the bug report that discusses why that DOCTYPE can not be removed on the official idpf website.

Do I understand correctly that Sigil will become ePUB3 compatible in a future release? It would be fantastic if it does. I have been discussing making Bluefish, which toggles well with Sigil, more ePUB3 friendly for detailed editing of component files recently and perhaps it would make sense to coordinate this?

That is the plan. We are experimenting with ways to replace Tidy (and its limitations) with an embedded python version of html5lib and beautiful soup4), to create a much more robust environment for poor html, then breaking off flightcrew into a plugin, and starting on support for epub3 metadata, opf, etc. This is a major undertaking that will take some time, which is why I threw this plugin together to create a stopgap mechanism for epub3 authors while we make all of these changes.

Take care,

KevinH

[KevinH]

Hi Jonathon,

I just posted ePub3-itizer_v017.zip with a bug fix (hopefully!) for using unique ids when multiple creators and/or contributors are used.

Please let me know if this doesn't fix your issue. And if you can send me something to recreate your nav issue, that would be grand.

Thanks,

KevinH

[JonathanMagus]

Hi Kevin

I attach a .zip file containing the original toc.ncx and the nav.xhtml generated from it by ePub3-itizer. A text file with the ePUBcheck output from the generated ePUB3 file is also included.

The nav file will not open in any external editor that I have tried it with. The contents of the file shows in Notepad and will display in Sigil if the converted ePUB3 is reopened in it. (Re-opening in Sigil is of course confusing and unreliable in this context though, because it reverts the DOCTYPE to its ePUB2 default and introduces other errors that probably really aren't there, such as showing the nav.xhtml file to be in the Text section. These errors are not in the converted ePUB3 file of course.)

Yes, I thought the retained toc.ncx was probably valid.

I will download ePub3-itizer_v017.zip next and try it.

Thanks

Jonathan

[JonathanMagus]

Hi Kevin

Yes, ePub3-itizer_v017.zip eliminates the ID error from the contributor entries in the ePUB3 output when tried with the same ePUB2 input file.

For some reason this time the generated nav.xhtml file may be successfully opened in an external editor, however, it still records the same 'ol' errors when the ePUB3 file is validated with ePUBcheck. At a quick glance I am guessing that an opening or closing tag is missing in the nested elements.

Regards

Jonathan

[KevinH]

Hi Jonathan,

Yes, there were a couple of bugs. It worked well for one and two levels of nesting but not when the nesting level changed by more than one at one time.

I have now fixed that I hope. I will post a v018 version shortly. I just want to run it through some more tests.

Thanks so much for your bug report and test case!

KevinH

Originally Posted by JonathanMagus

Hi Kevin

Yes, ePub3-itizer_v017.zip eliminates the ID error from the contributor entries in the ePUB3 output when tried with the same ePUB2 input file.

For some reason this time the generated nav.xhtml file may be successfully opened in an external editor, however, it still records the same 'ol' errors when the ePUB3 file is validated with ePUBcheck. At a quick glance I am guessing that an opening or closing tag is missing in the nested elements.

Regards

Jonathan

[KevinH]

Hi Jonathan,

I have updated the plugin to ePub3-itizer_v018.zip. When you get a free moment, please give it a try and verify that it fixes your nav.xhtml generation bug.

Thanks again for the test case!

Take care,

KevinH

[JonathanMagus]

Hi Kevin

I have tried the same file again with version 0.1.8 of ePub3-itizer, also updating the quickparser.py and have run the output through ePUBcheck once more. This still produces similar errors. However, nav.xhtml now opens cleanly in an external editor and I can see what the issue is, which is that the nested lists should each be contained in their own list item of the parent list (or preceding list in the hierarchy), whereas they appear as ordered lists in between list items instead of inside of one.

Interestingly if I open the page as a standalone item in any of the main web browsers all show the nested list as it should appear and do not report any errors. Technically though it is wrong, which must be what ePUBcheck doesn't like about it.

I am also getting and error for epub:type="title-page" in the Guide saying it is an undefined property, which when I look is possibly because the same href item has been put for every landmark entry. The last page in the eBook has been repeated for each of them.

I have tried it again with a slightly more complex ePUB2 file, which produces similar errors in the nav.xhtml file (and obviously still complains about the DOCTYPE in the toc.ncx). This also gives some errors in the .opf file for the use of dc:type, which is in the original and doesn't raise an error when validated as ePUB2. Without checking the specifications I am not sure if this is actually an error in ePUB3 or not.

This is still a brilliant outcome though. Being able to convert Sigil's output to ePUB3 with just a few minor details to resolve that could easily be fixed manually is a great starting point for developing high quality ePUB3.

If it might help to have some sample files to test with, either for ePUB3-itizer or for Sigil's further development, contact me directly. They would have to be provided on a non-disclosure basis though.

Thanks again for your work on this.

Regards

Jonathan

[JonathanMagus]

An update: I now wonder if epub:type="title-page" is in fact wrong. There is some ambiguity about whether it should in fact be "titlepage".

The test files I am using are valid ePUB2, though may be a couple of years' old and some aspects of best practice could have formalised since then.

Everything I am reporting is otherwise correct, however.

Regards

Jonathan

[KevinH]

Hi Jonathan,

I have tried the same file again with version 0.1.8 of ePub3-itizer, also updating the quickparser.py and have run the output through ePUBcheck once more. This still produces similar errors. However, nav.xhtml now opens cleanly in an external editor and I can see what the issue is, which is that the nested lists should each be contained in their own list item of the parent list (or preceding list in the hierarchy), whereas they appear as ordered lists in between list items instead of inside of one.

Yes I finally found the spec that showed it nested inside <li> tags. To me that is a very silly way to create the nesting for the nav.xhtml but it is the spec ...
so I think I have finally figured this one out (but I am not 100% sure). At least this new version should not be any worse than the older one!

Interestingly if I open the page as a standalone item in any of the main web browsers all show the nested list as it should appear and do not report any errors. Technically though it is wrong, which must be what ePUBcheck doesn't like about it.

Yes, and I like my version better ;-) but ... the spec it must be.

I am also getting and error for epub:type="title-page" in the Guide saying it is an undefined property, which when I look is possibly because the same href item has been put for every landmark entry. The last page in the eBook has been repeated for each of them.

Yes, the href was a bug and I fixed that. But I have found multiple lists of allowed vocabulary for the epub:type element.

I have gone with this list:

http://www.idpf.org/accessibility/gu...l/sections.php

And it lists titlepage instead of title-page.

There is also this much much longer list:

http://www.idpf.org/epub/vocab/structure

but that list is getting so long it is absurd. It also lists "titlepage" not "title-page"

So I have translated the "title-page" which is legal opf2 guide to "titlepage".
I used the first link to form the basis for the list I am supporting and have reworkeing the guide types to epub:type mapping. BTW: since the opf2 guide elements allow guide types of "other-*", I have set things up so that "other-*" prefixes will be mapped to the "*" part if it exists in the legal vocabulary of epub:types given by the first list I found.

I have tried it again with a slightly more complex ePUB2 file, which produces similar errors in the nav.xhtml file (and obviously still complains about the DOCTYPE in the toc.ncx). This also gives some errors in the .opf file for the use of dc:type, which is in the original and doesn't raise an error when validated as ePUB2. Without checking the specifications I am not sure if this is actually an error in ePUB3 or not.

I checked this and dc:type has gone from a pretty much free form field in epub2 to be quite restricted when used in epub3.

See: http://www.idpf.org/epub/vocab/package/types/

So I have changed the code to look for one of the approved types and if not, then simply skipping that dc:type metadata

If it might help to have some sample files to test with, either for ePUB3-itizer or for Sigil's further development, contact me directly. They would have to be provided on a non-disclosure basis though.

Understood and I will probably take you up on that especially if my nav.xhtml is still messed up in your testing.

Thank you for all of your testcases and bug reports!

Please see the first post for the latest ePub3-itizer_v019.zip which will hopefully (finally) get the nav correct.

Take care,

Kevin

[KevinH]

Hi All,

ePub3-itizer_v019.zip is now available at the first post with bug fixes.

Thanks,

KevinH

[JonathanMagus]

Hi Kevin

I have tried version 0.1.9 with the same two ePUB2 files and this time the conversion process failed, producing an error message, which at a glance looks to me to be the same in both cases. I attach a text file with these in.

Just to be sure there were no other factors involved I reinstated ePUB3-itizer_0.1.8.zip and that works in Sigil as it did before.

Coincidentally I have just been looking at some of the same specifications for a different reason that I may follow up on separately with you sometime.

I am guessing that the error messages will mean more to you than they do to me. Let me know if you need anything more on this.

Regards

Jonathan

[KevinH]

Hi,

It means I can't type worth a damn. When fixing the dc:type error, I typed "tcontext" instead of "tcontent" and created this error myself. None of the 15 test ebooks I use have dc:type set, so I never caught my typo.

If you have a free moment, could you test the v020 version and see if the new nav code actually works? That is my biggest worry and none of my test cases have a multi-level toc.ncx and so none of the nesting issues were ever apparent.

Thanks,

Kevin

ps: There is now an 0.2.0 version with the typo fix

Originally Posted by JonathanMagus

Hi Kevin

I have tried version 0.1.9 with the same two ePUB2 files and this time the conversion process failed, producing an error message, which at a glance looks to me to be the same in both cases. I attach a text file with these in.

Just to be sure there were no other factors involved I reinstated ePUB3-itizer_0.1.8.zip and that works in Sigil as it did before.

Coincidentally I have just been looking at some of the same specifications for a different reason that I may follow up on separately with you sometime.

I am guessing that the error messages will mean more to you than they do to me. Let me know if you need anything more on this.

Regards

Jonathan

[JonathanMagus]

Hi Kevin

Brilliant! The same two files now produce ePUB3 versions that go through ePUBcheck cleanly. The only theoretical error reported by it remains the .ncx DOCTYPE.

I will probably drop the .ncx file anyway, because the rest of the ePUB3 document would most likely end up too far away from compatibility with an ePUB2 eReader after further development following the conversion, so having one wouldn't be essential. It does make sense to keep it in the conversion process output though because sometimes it will be good to have.

However, as a point of interest this documentation suggests that Sigil's original DOCTYPE in the .ncx file isn't absolutely necessary, although obviously it is valid as you said previously:

http://www.idpf.org/epub/20/spec/OPF...Section2.4.1.2

This is a big step forward in the practicalities of developing for ePUB3.

Thanks

Regards

Jonathan

[KevinH]

Hi Jonathan,
Here is the official open bug report for epubcheck 3 which is targeted to be closed in epubcheck 4.

https://github.com/IDPF/epubcheck/issues/305

There are times the doctype is needed for the ncx to actually work. That is why pre-release versions of epubcheck4 now allow the doctype on the toc.ncx.

Take carw,

KevinH