[Plugin] ePub3-itizer - epub3 output plugin for Sigil

[KevinH]

Hi All,

Updated: January 19, 2023
Current Version: "0.5.8"

See the attached: ePub3-itizer_v058.zip

Official Github site: https://github.com/kevinhendricks/ePub3-itizer

License/Copying: GNU LGPL Version 2 or Version 3 your choice. Any other license terms are only available directly from the author in writing.

Change Log

Spoiler:

Changes since v056
1) removed basic xml named entities from named to numeric conversion
2) allow named entity conversion of text inside pre tags

Changes since v055
1) add "ePub Nav" as title to newly created nav.xhtml
2) make sure NCX uid always exactly matches OPF uniqueid meta value

Changes since v051
1) make start folder for output much smarter
2) set spine nav attribute to linear="no" if html toc exists in guide

Changes since v050
1) fix missing title tag in head of nav

Changes since v041
1) update it to use the new plugin interface in Sigil 1.0 and
to work with epubs with non-standard layouts
2) should continue to work with Sigil versions before Sigil 0.9.991
3) removes support for SMIL updating as Sigil properly handles that in epub3

Changes since v040
1) fix typo in nav labeling code
2) convert "big" tags that are not in html5 to "span" tags with inline style equivalent

Changes since v039
1) make new opf id generation more robust to prevent id clashes
2) update to my new icon until the vote is complete
3) recognize calibre:title_sort and convert to its epub3 equivalent

Changes since v038
1) remove any charset attributes from link elements
2) recognize calibre:series and calibre:series_index metadata and convert to epub3 equivalent
3) add a plugin.png icon to work with the next version of Sigil

Changes since v037:
1) restrict fix that forced focus to only Mac systems to prevent problems on other platforms
2) remove display:none css from parts of the nav since they are redudant given the hidden attribute
3) remove empty dc: metadata since for epub3 epubcheck barfs on it
4) Add LGPL V2 or LGPL V3 license info to plugin.py

Changes since v036:
1) Force the FileDialog to the front with focus so people know plugin is running

Changes since v035:
1) bug fix to prevent xml headers from interfering with smil processing

Changes since v034:
1) added autostart tag to plugin.xml

2) modified code to allow only epub2 as input to the plugin

3) Removed code needlessly duplicated from plugin launcher codebase


Changes since v033:
1) bug fix to prevent empty guide element from being passed to epub3 opf

2) bug fix to actually use the fixed media types for fonts


Changes since v032:
1) integrated in support for SMIL Media Overalay metadata (Thank you Alpe!)

2) allowed guide to pass through to epub3 opf to maintain backwards compatibility

3) added hidden attribute to nav page-list and landmarks

4) properly added inline styles display:none for landmarks and page-lists (Thank you Doitsu)

5) map epub2 font x-font-ttf and x-font-opentype media-types to vnd.ms-opentype
(will be "font-sfnt when EPUB 3.1 is final)

6) create a github repo for ePub3-itizer code at:
https://github.com/kevinhendricks/ePub3-itizer


Changes since v031:
1) Fixed typo bug in opf_converter that caused problem with spine properties
(Thanks DiapDealer for the fix! And thanks Olivier_g for the bug report!)

2) added lang= and xml:lang= to specify language from opf dc:language for nav.xhtml

Changes since v030:
1) Fixed nasty bug I introduced in v030 that downcased all ncx tags and removed navPoints
Thanks for Dotisu for catching this error

Changes since v022:
1) nav changes to add style in head to prevent display of landmarks
and to use ol list-style-none

2) remove "linear = "yes" from nav entry in spine

3) remove DOCTYPE from toc.ncx (not sure if this is correct or not)

4) remove guide from content.opf since now optional

5) should now properly identify and add manifest properties "scripted"

Very Important Note

Support for this plugin is only provided for Sigil 0.8.6 or later. This plugin requires Sigil 0.8.2 or later to even operate.

Before Conversion

1. Make sure your epub2 validates under FlightCrew and/or EpubCheck
2. If your epub uses ".htm" or ".html" file extensions, you should first change them to use ".xhtml" file extensions as follows:

• In Sigil's BookBrowser window select all of your .htm or .html files
• Right click on your selection and select Rename from the context menu
• in the "Rename Files Starting At" dialog box, remove everything there and replace it with ".xhtml" (no quotes), then hit "OK"

How it Converts from valid epub2 to epub3

This program walks all xhtml files doing the following:

- converting DOCTYPE to <!DOCTYPE html>
- adds epub: namespace to html tag
- converts meta charset info to be: <meta charset="utf-8">
- converts all html named character entities to numeric entities
- converts "big" tag to "span" with inline style equivalent
- collects any fixed layout metadata in the head tag for opf3 spine page properties
- notes any use of svg, epub:switch, mathml, and script for opf3 manifest properties
- collects any epub:type attributes to help extend nav (landmarks) in the future

Then it reads the current opf and converts it on the fly to meet package 3 requirements

- converts package tag and adds rendition prefix information
- converting metadata it can , adding refines if need be
- passes any unknown meta tags with name/content pairs through unchanged
- adds the required dcterms modified metadata information
- adds manifest page properties where needed including use of mathml, svg, switch, scripted and cover
- adds spine page properties where needed
- extracts the guide for use in creating the nav landmarks and removes it from opf
- adds an entry in the manifest for the new nav document
- adds the new nav document to the end of spine

Then it parses the current toc.ncx extracting doctitle, toc, and any pagelist information and removes its DOCTYPE

It then merges this with the original guide information from the opf2 to create a new nav.xhtml file.
It will now nicely handle multi-level tocs.

Finally it adds the mimetype file and zips it all up and then launches a gui to ask the user what to name the file and where to save it.

---

Thanks to Doitsu, DiapDealer, and JonathanMagus and others for reporting back bugs. I think this code is now stable and usable. Bug reports and feature requests welcomed.

If you are at all interested ... load up your favorite epub2 ebook, then go to output plugins and run the ePub3-itizer.

Take care,

KevinH

[Toxaris]

You might want to take a look at this. It contains a stylesheet called ncx2end.xsl which can be used to transform a NCX to a nav TOC. It does support multiple levels of TOC.

I basically looked at the same thing yesterday to enable ePUB3 support in my add-in. However, I tried one document with MathML with fallback for ePUB2 and the results were dramatic. Even most ePUB3 readers made a mess of it. Only Chromium handled it correctly. I decided it was not worth the effort for me at the time.

[KevinH]

Hi Toxaris,

Thanks for that link. I am not a fan of using xslt, and to tell you the truth getting the nav to support multiple levels is quite easy. Also, that version does not parse the guide elements in the opf to create the landmarks so I'll stick with what I have now and just improve on it.

Yes, support for inline mathml (even if it is just the presentation subset of mathml) is hard. Assuming you added all of the correct namespaces, it should have worked. I will try playing round with that next just to see how it appears in preview mode and bookview.

Take care,

KevinH

[Toxaris]

Originally Posted by KevinH

Hi Toxaris,

Thanks for that link. I am not a fan of using xslt, and to tell you the truth getting the nav to support multiple levels is quite easy. Also, that version does not parse the guide elements in the opf to create the landmarks so I'll stick with what I have now and just improve on it.

Yes, support for inline mathml (even if it is just the presentation subset of mathml) is hard. Assuming you added all of the correct namespaces, it should have worked. I will try playing round with that next just to see how it appears in preview mode and bookview.

Take care,

KevinH

No problem, it was an option. I agree that it is not that complex (at least that part...). The issue with MathML was not so much the support (although it is abysmal), but the failure in honoring the epub:switch. You need that to be able to handle a fallback, but the reading systems either did both the MathML and the image or just the image. It is a bit ironic that the systems that do support MathML do not seem to support epub:switch...
Only Chromium did it correctly IIRC, but there the MathML does not resize together with the font resize.

[KevinH]

Hi Toxaris,

Yes, I have never actually seen an example of epub:switch working. Technically, my plugin should look for all instances of epub:switch and add "switch" to the manifest properties for that file since "switch" is one of those things that must be noted in the opf3.

I think some readers are using some javascript to actually implement the epub:switch functionality by effectively only presenting one case. Those readers that do not support any background scripting will have to wait for their display engine to learn how to handle "switch" which in turns make it useless for doing what it was designed for: fallback!

Another part of the problem is that epub3 has become so very very bloated and is constantly changing. The epub3 specs seem to be driven by textbook publishers and much of what they add is truly not needed for normal ebooks. And I personally think pdf would be a better choice for textbooks anyway.

For example: I tried to more fully add support for the epub:type vocabulary and it was changed yet again about a month or so ago (September). The recognized vocabulary is now huge and the documentation of where each word is allowed to be used has been written by someone who must have been a bad lawyer in an earlier life!

The epub3 creators seem to have forgotten the very meaning of KISS in engineering their spec.

BTW: If you have a simple example of using epub:switch that you think should work, I would love to have it for testing and improving Sigil down the road with it. I looked at some of the official epub3 samples and many of them at not consistent with their own specs!

Take care,

KevinH

[Toxaris]

True, and then to think that epub:switch was already in the specs for ePUB2. They would have been better of fixing ePUB2 instead of the bloated thing that is ePUB3. It should have been independent and they should not have listened to Apple specifically.
On one hand I like the fact that there are more possiblities for semantics and structure, on the other hand that is what Latex is for and it makes the ePUB much more complex. ePUB is a result, not an intermediate format.
I think they can better take their loss and create an ePUB4 standard without all the hassle and that just works for books.

[user_none]

I should mention that this plugin is a experimental test best. Eventually epub3 will be integrated directly into Sigil.

[KevinH]

Hi All,

Originally Posted by user_none

I should mention that this plugin is a experimental test best. Eventually epub3 will be integrated directly into Sigil.

Yes, as user-none said, this is just a stopgap measure until we get Sigil to support epub3.

I should have a much improved version with gui for save-as ready by the end of the week.

Take care,

KevinH

[KevinH]

Hi All,

An ePub3-itizer_v010beta.zip is now available in the first post. Changes include, a gui file dialog for selecting where to save it and what to name it, support for conversion of multi-level tocs to multi-level nav, a refactoring to make the code easier to support, and lots of bug fixes.

Please let me know if you run into any problems with it.

Hope this helps,

KevinH

[KevinH]

Hi All,

A bug fix release ePub3-itizer_v015beta.zip with an updated plugin for Sigil and an even newer bug fixed version of quickparser.py.

See the first post for all of the links.

Special thanks to Doitsu for all of his hard work testing and finding bugs!

Thanks.

KevinH

[David Kudler]

Originally Posted by KevinH

Hi All,

A bug fix release ePub3-itizer_v015beta.zip with an updated plugin for Sigil and an even newer bug fixed version of quickparser.py.

See the first post for all of the links.

Special thanks to Doitsu for all of his hard work testing and finding bugs!

Thanks.

KevinH

Bless you all. To be honest, this has been my biggest grumble with Sigil for a while.

[Doitsu]

If you're a Windows user and would like to give this plugin a try, here's how you update quickparser.py:

1. Press +R to display the Run dialog.

2. Copy and paste %ProgramFiles%\Sigil\plugin_launchers\python into the input box and press Enter. This will open the Plugin Launchers Python folder in Windows Explorer.

3. Copy the new version of quickparser.py into this folder.

[DiapDealer]

Originally Posted by KevinH

- collects any epub:type attributes to help extend nav (landmarks) in the future

if len(etypes) > 0:
            epub_types[mid] = " ".join(etypes)

That join is failing (line 99-ish: plugin.py) whenever epub:types are encountered because etypes is a list of tuples that is created on line 229 of the same file.

Suggest just storing the list of tuples in the epub_types dict until future implementation.

[KevinH]

Hi DiapDealer,

Thanks for the bug report. I will take a look at it and also fix your other bug report.

Thanks!

KevinH

[KevinH]

Hi,

Just released ePub3-itizer_v016.zip. This is a bug-fix release (Thank you DiapDealer for your bug reports!) . See the links in the first post for the downloads.

Take care,

KevinH