Sigil Development News

[Doitsu]

Originally Posted by KevinH

I may also change the build to use a shared library version of the google gumbo parser since a ctypes interface to it already exists. That would allow the user to use a full fledged, validated html5 parser inside a plugin without needing to add much. In fact, with some planning we may be able to offer a few types of other Sigil services via a bi-directionaly interface.

IMHO, that's a good idea. The more bi-directional interfaces, the better.
Do you have an ETA on the first beta release?

[KevinH]

Hi Doitsu,

We (DiapDealer and I) have run into a snag about how best to embed python3 into our build/source tree and the final installer. Once we have a handle on that, We will be releasing an alpha to get feedback on it.

Almost the entire internals of Sigil will have been gutted. We removed Xerces, Tidy, Flightcrew (it will become a plugin) added in an embedded interface to python3, and created our own version of gumbo to stop anything from interfering with epub3 html5 code while still keeping epub2 support. To add insult to injury, John (user_none) spent a bunch of time removing the use of null references all through the source code (which Valloric seemed to be a fan of) which are not valid C++ and which are now broken with the latest version of the clang compiler. This necessitated big changes as well. These changes only exist in master since master has had so very many changes since the 0.8.X tree. They simply are not and worth backporting.

The alpha release should allow editing of epub3 epubs at a codeview level but lots of work will still need to done in the future to auto generate a nav in the create TOC menus, handle the new epub3 metadata editing (and its associated gui changes), some large changes to support viewing of mathml (which Qt does not support).

So a lot of work remains but this intermediate release will hopefully be useful for epub3 people (and still work the same for epub2 people) and for feedback on the stability of the massive changes we have made.

So I am hoping for an alpha release by the end of August or September at the latest.

Hope this helps,

KevinH

[Doitsu]

Hi KevinH,

Originally Posted by KevinH

We (DiapDealer and I) have run into a snag about how best to embed python3 into our build/source tree and the final installer. Once we have a handle on that, We will be releasing an alpha to get feedback on it.

Again, this might be another stupid question, but can't you create a test build that'll autodetect and use an already existing Python 3 installation or will the next version be largely Python-based like Calibre?

I'm also wondering whether the next version will still support Python 2 plugins, or will all existing plugins need to be updated for Python 3?

Anyway, I really appreciate the hard work of you, user_none and DiapDealer, and I'm sure that the next version will be awesome.



P.S. Maybe a mod could move this post and the two posts before it to a new thread for Sigil development updates.

[eschwartz]

The test build presumably has to include a bundled python anyway, in case there is no pre-existing python3 installation. Unless you mean to have Sigil distribute multiple versions, with and without bundled python interpreters?

A major goal seems to be to avoid requiring users to install additional program dependencies. Would be nice to test that too.

[KevinH]

Hi All,

Python 3.4.X will be embedded directly into the upcoming Sigil binary and install. The vast majority of the Sigil code is C++ and will remain that way. That said, the use of Xerces for *pure* XML parsing (not xhtml as gumbo handles that) has been replaced with a python3/lxml/BeautifulSoup4 combination which is invoked via a dedicated C++/Python interface module.

So yes on Windows and Mac, Python 3 (with lxml/bs4 added) will be installed as part of Sigil to guarantee a workable Python3 version exists. It will be required for Sigil to properly function on those platforms.

We also plan to modify the Plugin Interface to see and use our own pre-installed Python3 automatically. Plugin users will still be able to select their own external Python 2 interpreter if they so desire. Plugins will continue to support both python 2 and python 3. The only advantage will be that Python3 will be guaranteed to be there and work (so no separate install required) for your plugin to work. This is why we spent time redesigning the plugin interface code and all support code to work on both Python 2 and Python 3 since we knew Python 3 was coming as part of a future install.

On Linux platforms, this is all moot anyway. Linux Packagers can decide to use their own Python 3.4 version with python 3 LXML and python 3 BS4 modules as required packages for Sigil just like they do now by requiring HunSpell, PCRE, mini-zip and other libraries as Sigil required packages. In fact, by removing Xerces and the associated (possibly non-free) files that were used for epub2 validation/flightcrew, we have made the Sigil code more "free" so that Debian and others who care about the exact license can package Sigil and include it with less worries.

So Sigil will always be a C++ program but python 3.X will be required to make everything work as expected and for some new feature development work. It just made no sense to re-invent the wheel and develop lots of C++ code to use libxml to replace Xerces for pure XML parsing when a Python 3.4 with lxml and BS4 would do everything we need and allow plugin developers to stop worrying about Windows users who were afraid to install any form of python (and we have all seen posts from them).

Hopefully, when the alpha is released, a user will be able to take a valid epub2 and use the epub3-itizer plugin to convert it to epub3 and then proceed to load the resulting epub3 and make codeview level changes with Sigil.

Eventually, special "internal" python3 plugins to autoconvert from epub2 to 3 (and possibly epub3 back to epub 2 if there is interest) will become part of a future Sigil 1.0 stable release alongside an epub3 metadata editing gui, auto creation of nav, and mathml appearing in Preview and BookView among other changes. That's the plan anyway.

Hope this explains things,

KevinH

[KevinH]

As suggested a thread for those interested what's on tap for future Sigil releases

[Notjohn]

You guys frighten me. If I don't understand a word of what you're talking about, will Sigil still be as easy to use as now?

(I LOVE Flight Crew! I use it regularly, often two or three times in a working session, if I'm making serious changes to the text.)

[KevinH]

Hi,
The main GUI will be basically unchanged. The only new user interface will be something to allow editing of metadata that is specific to epub3.

Flightcrew will be a plugin for epub2 users much like Doitsu's epubcheck plugin is now, but something like the SanityCheck plugin will eventually become part of Sigil and allow and report back well-formed errors quite quickly.

So no worries, almost all of these changes impact internals not the user interface.
Take care,

KevinH

[Notjohn]

Thanks. My heartrate has slowed a bit.

[DiapDealer]

And remember ... no one will be forcing you to update. If the current version(s) work fine for you, stick with them.

[Toxaris]

It sounds very ambitious, but also very exciting. Improving a program under the hood is not so 'sexy', but usually much more fun.