[Plugin] ePub3-itizer - epub3 output plugin for Sigil

[KevinH]

Originally Posted by AlPe

(Where do you prefer discussing about it? Here or on GitHub?)

Here is fine.

1. the user already has an EPUB 2 file (Open...)
2. the user creates the EPUB (2) file from scratch (New...)
3. the user has an EPUB 3 file already (Open...)

only cases 1 and 2 are doable now. Only simple edits with no tools used are possible with epub 3 now. We can modify the plugin when Sigil can handle more epub3 editing.

A. inserting suitable id's (actually <span id="...">) into the XHTML markup to denote the MO fragments (I plan to code a separate edit plugin for doing this automatically);
B. adding audio files into the container (Sigil can do this already);
C. adding the SMIL files into the cointainer (for now, assume they are "dummy" SMIL files, but they can be computed using my aeneas tool: http://www.readbeyond.it/aeneas/ );
D. outputting to EPUB3: this can be done with your plugin plus some modifications I can easily code.

Is your aeneas opensource? Is it cross-platform? If not, I am not sure this project is doable as my code is gnu gpl and is meant to support all users of Sigil.

My questions are:

a) since step D is almost already covered by your current plugin, does it make sense/are you fine with me adding functionalities to handle Media Overlays (SMIL) files to it, at least for cases 1. and 2.?

Happy to as long as your tool is opensource and crossplatform (builds,runs on Linux, MacOSX, and Windows).

b) What would you suggest for case 3.? Is it better to wait for Sigil to become fully EPUB3-capable? Shall a specific version of the plugin be made for outputting EPUB3 if the user opens an EPUB3?

Hold off on case 3 since Sigil is not there yet.

Thanks,

KevinH

[AlPe]

Thank you for the quick reply.

Good, I will fork the repo, code the part for handling MO under cases 1. and 2., and open a pull request.

(BTW, aeneas is released under the AGPLv3, and it works on Linux, Windows and OS X, albeit it currently requires Python 2.7.x. In any case, it will be used outside Sigil. My idea is to create a Sigil plugin that will create an "aeneas Job" from the current EPUB --- basically, a ZIP file with the appropriate resources to compute the SMIL files --- so that the user could run aeneas on it outside Sigil, and import back the computed SMIL files.)

[KevinH]

Hi AlPe,

Have you thought of making aeneas code work on both Python2.7 and Python 3.4 from the same code base. It is simply not that hard to do, and some Linux distributions are moving away from installing python 2.7 at all. There are a number of packages that can help make the transition easier, including six, and own own compatibility_utils libraries. I would also be happy to help convert it to run on both if you run into any issues. We have done that with almost all of our plugins since Sigil's support for Python 2.7 may go away at some point as it is redundant given the embedded Python 3.4 interpreter.

If you do make that conversion in your code, you could then invoke it via a python subprocess or even directly import the code from the zip.

Please note, although epub3-itizer does not use much of them, there is a settings module and tk gui elements module that can be used to make the plugin more interactive if that is something you need.

KevinH

Originally Posted by AlPe

Thank you for the quick reply.

Good, I will fork the repo, code the part for handling MO under cases 1. and 2., and open a pull request.

(BTW, aeneas is released under the AGPLv3, and it works on Linux, Windows and OS X, albeit it currently requires Python 2.7.x. In any case, it will be used outside Sigil. My idea is to create a Sigil plugin that will create an "aeneas Job" from the current EPUB --- basically, a ZIP file with the appropriate resources to compute the SMIL files --- so that the user could run aeneas on it outside Sigil, and import back the computed SMIL files.)

[AlPe]

Work on aeneas started when lxml for Python 3 was broken, that's why it is 2.7.x. I am already planning to port it to Python 3.x, as the aeneas users seem happier with the "radical port" solution rather than having "bilingual" Py2+Py3 transitional code.

Moreover, there is another reason for executing aeneas externally: it might take some time to align text and audio, from few seconds to several minutes, depending on the length of the input audio files. Hence, I think the "async" execution flow I suggested is the best option for now: the "SMILify" plugin can create "dummy" SMIL files (i.e., syntactically correct SMIL files, but with dummy timing values), later to be replaced by the final SMIL files, with correct timing values.

[KevinH]

Hi Alpe,

If the changes become too invasive or if it is simply easier for you, why don't you just copy in everything I have from epub3-itizer as call it your own (permanently fork it) and create your own epub3-itizerSMIL plugin but ... perhaps choose a better name for it!

I am also okay with that approach, if that would be easier. As long as the resulting project can be used with all platforms and is open source, I am happy one way or the other.

KevinH

[AlPe]

I am actually glad to contribute the code for handling MO into epub3-itizer.

I need to produce an edit plugin (the "MOfier") to insert the id's or the <span>s for MO fragments into the XHTML files anyway, so I do not feel constrained at all.

[KevinH]

Okay,
Let me know if you have any plugin related questions.

Sigil's embedded python has access to a variety of parsers built right in including lxml, our own bs4 variation, html5lib, google's gumbo parser, and an easy to use python only serial parser called QuickParser, so there are lots of tools to use. Also we have cssselect that works with lxml, and cssutils if you need to parse css stylesheets. Plus access to opf_parser.py to extract info from the opf as needed.

BTW: as Sigil's epub3 handling improves, we will have need of an SMIL dtd for parsing SMIL files. If you have access to one with a usable license, I would love to know where to find it. Alternatively, we could develop specialized python for doing just that not using lxml at all.

Thanks,

KevinH

[AlPe]

Thank you.

Before starting modifying the .py code, I created a new branch on my clone and pushed a couple of EPUB2 trees demoing some possible inputs. You can find them here:

https://github.com/pettarin/ePub3-it...ee/mo/tests/mo

When you have 5 minutes, would you please confirm that handling those epub2_* publications is in scope for this plugin?

(The epub3_* is just the "base" from which I "backported" the EPUB2 trees, and it is not meant to be handled, as we discussed above, just ignore it.)

=== === ===

If the above is fine with you, I will proceed coding the SMIL parser/patcher and include it into the current plugin code. Basically, there will be a check for .smil files into Misc/ . If so, they will be parsed, to determine which XHTML files are associated to, and then the suitable metas and properties will be added into the OPF. The SMIL files might be patched too, to adjust the src attributes of text and audio elements.

=== === ===

SMIL DTD: the specs link to a RelaxNG: http://www.idpf.org/epub/301/spec/ep...verlays-schema which I think can be converted to DTD using jing and trang. I will check with the DAISY dev responsible for MO in EPUB3 if there is a "free" DTD available.

[KevinH]

Hi Alpe,

Thanks, I will take a look at it. Misc is used for things that Sigil would rather not deal with. So perhaps we should make a change in Sigil to create a Smil folder and put any smil files there? This is an easy enough change for Sigil and it better prepares things for epub3 going forward.

Is this something you would like?

KevinH

[AlPe]

Take your time, I do not think I will have time to work on this before next week.

=== === ===

SMIL files position: I have no preference, and it makes sense for Sigil to move the .smil files into OEBPS/Smil/ for consistency with the other type of assets.

(Currently, I put the .smil files of my EPUB3 in the OEBPS/Text/ directory, along with the .xhtml files. But the specs does not mandate a specific position for them. With respect to reading systems, there is no drawback choosing one internal location over another. Those supporting Media Overlays do it in a "spec compliant" way, NOT relying on naming conventions like expecting the smil files in say OEBPS/Text/ or similar.)

[AlPe]

Update on SMIL DTD: I asked around and it seems no DTD is available. I tried converting the RelaxNG linked in the specs into a DTD using trang, but it fails with errors. Moreover, I believe the conversion cannot be lossless anyway, since RelaxNG allows for external namespaces, which EPUB 3 Media Overlays use indeed (e.g. epub:textref or epub:type).

Is having a DTD an hard requirement?

[KevinH]

Hi Alpe,
No, it is not required.
I spent some time looking at the full smil spec and the subset? used in epub3. If we had a dtd we could load it into lxml and then serializing the lxml tree back to xml would be easier (void tags would remain void, etc), we could validate the input, and any changes made, etc.

Alternatively, we can write a smil specific parser and serializer if needed. I am thinking more long-term here for the best way to support smil in epub3 under Sigil.

Thanks,

KevinH

[AlPe]

I put into https://github.com/pettarin/ePub3-itizer/tree/mo the changes to output the required meta elements and media-overlays attributes into the OPF to support Media Overlays.

I tested it against the epub2 trees I prepared last week, and it works well. If the ebook has no SMIL files, the output is the same as before.

If it looks fine with you, I can open a pull request.

[KevinH]

Hi,
I looked at your commit. Do you really need to change single quotes to double quotes, and add unnecessary parentheses in if statements, and rename things that do not need to be renamed. You have made so many unnecessary changes it actually was hard to find exactly what you really do need to change.

I realize everyone programs by their own rules and many like to use the equivalent of python lint to meet some pep8 "standard", but those types of changes really shouldn't be mixed into the same commits with new features.

So please either back out those unneeded changes. Or, alternatively separate your commit into two pieces. One that simply cleans up what you didn't like, and then a second that actually adds the code you want to add to support SMIL.

Thanks,

KevinH

Originally Posted by AlPe

I put into https://github.com/pettarin/ePub3-itizer/tree/mo the changes to output the required meta elements and media-overlays attributes into the OPF to support Media Overlays.

I tested it against the epub2 trees I prepared last week, and it works well. If the ebook has no SMIL files, the output is the same as before.

If it looks fine with you, I can open a pull request.

[AlPe]

You are right. I forgot to disable the auto-cleaner in Vim.

(Not going to argue about your code style, however there are some inconsistent usage, e.g. double space after "=" in assignments, or module constants in uppercase and lowercase, etc. Anyway, your house, your rules. I'm ok with it.)

I pushed a "minimal" set of changes on the "momin" branch: https://github.com/pettarin/ePub3-itizer/tree/momin

The changes are:

1. tests/mo/ : directory containing the epub trees to test MO support;

2. README.md : changed date

3. src/plugin.xml : bumped to v0.3.3

4. src/plugin.py : added code to detect and patch SMIL files

5. src/opf_converter.py : added code to generate the <meta...> elements and media-overlay="..." attributes required when MO are defined.

Other notable changes:

A. I believe this was a bug:

https://github.com/kevinhendricks/eP...e12a271c48L259

B. refactored several open(...).write(...) calls into:

https://github.com/kevinhendricks/eP...406fd63772R114

C. there was a potential bug in the creation of the guide landmark in nav.xhtml:

https://github.com/kevinhendricks/eP...406fd63772R209