Plugin Development
 

I have started this thread to help answer any plugin developer's questions, and to provide links to some common code examples to help speed development of Sigil plugins.

Attached below are the following:


libgui.zip
a python module compatible with both python 2.7 and 3.4 that hides the details
of using Tk for generating simple modal dialogs such as fileChooser(), folderChooser(),
fileSaveAs(), and selectFromList(). Also included are routines to convert ebook titles
to safe file names, and to get the path to the User's home directory.
The main routine illustrates simple examples on how to use these routines in your plugin.

sampleOutput.zip
a sample epub output plugin that supports font obfuscation and removes the Sigil specific metadata. It requires Sigil-0.9.3 to work due to bugs in earlier launcher code.

testme3_v031.zip
a sample edit plugin that provides many simple examples of how to use the plugin interface

Pre-Release Versions of Sigil's Python Plugin Launcher Code
 

Pre-Release Versions for Plugin Developers

Because storing preferences can be important for a Plugin, DiapDealer has designed a new python launcher preferences class and integrated it right into the *container classes, so persistent preferences storage is easily available for all plugins.

These changes are new since the Sigil 0.8.2 release and they will appear in the next future version of Sigil.

To illustrate how to create and use these settings, DiapDealer has designed two simple plugins that clearly demonstrate how the interface should be used.

See the attached "PrefsExampleGroups_v0.0.2.zip" and "PrefsExampleSimple_v0.0.2.zip".

Important
In order to use this preferences interface, you will need to be using Sigil v0.8.3 or greater.

It's possible to update only the python plugin launcher files if you can't wait for Sigil 0.8.3. To facilitate that we have thrown together a launcher_updater python program that will try to automatically find your Sigil installation and properly copy in the python files. *NOTE: this is unnecessary if you have Sigil 0.8.3 or higher.* The instructions for updating the plugin launcher files are being left in the event that that future preview releases appear.

To run the launcher_updater: do the following:

1. download the zip file attached to this post, do not unzip it

2. In a terminal, cd to where this zip file is (again, do not unzip it) and run the following command: (and yes you are telling python to run a zip file!).

Note that on Linux you will probably need to run this command with "sudo" and on Windows you should open a command-prompt as Administrator

Before using this updater, please verify that it matches the following md5sum:

/sbin/md5 launcher_updater_20141204.zip
MD5 (launcher_updater_20141204.zip) = 9a4a2fbc79aacc736f95d74cc4b9eea9

If you know how to manually install the new python launcher files, you can simply unzip the launcher_updater and look in the payload folder.

Note, these launcher files will only work with stock Sigil 0.8.2, NOT Sigil 0.8.1.

Note, this pre-release returns a launcher_version value of 20141204 versus the earlier version in Sigil 0.8.2

And as always the very latest version of the python launcher code used by Sigil can always be be found here:

https://github.com/Sigil-Ebook/Sigil...unchers/python

Mapping of Sigil Versions to Plugin Launcher Versions
 

Here is a mapping of Sigil version with the version information set by the plugin launcher code:

bk.launcher_version()

We will keep this table up to date. And we will continue to increment the launcher_version with each new release so that it can be used as a proxy for Sigil version in all versions of Sigil in the Sigil-0.9.X series.

Information as to when significant new plugin features were added or bugs fixed is provided below:

20150909 (Sigil-0.8.900 pre-release)
------------------------------------------
- lxml, sigil_bs4, PIL, regex, six, html5lib first added to Bundled Python
- spellcheck support first added
- sigil_gumbo parser support first added
- FlightCrew is now a Sigil Plugin


20151001 (Sigil-0.8.901 pre-release)
------------------------------------------
- major bug fixes for sigil_bs4 prettyprint_xhtml and serialize_xhtml
- epub3 interface features for pugins
- cssutils, cssselect, chardet first added to Bundled Python
- validation plugins now autoclose


20151024 (Sigil-0.9.0 release)
----------------------------------
- Use Bundled Python option first introduced
- fix bug in PluginRunner that try to well-form check xml files as xhtml
- fix launcher epub3 interface to convert null properties to None
- fix for pluginhunspell.py to try and find hunspell on Linux systems


20151120 (Sigil-0.9.1 release)
----------------------------------
- fix bad bug in PluginRunner that coerced xhtml to xml when new files added
- fix to prevent missing hunspell from stopping all plugins
- fix for bundled PIL and cssutils on Mac OS X
- creation of testplugin_v010.zip to allow builders to test their plugins
- made launcher success and error messages robust to non-utf8 strings


20151215 (Sigil-0.9.2 release)
----------------------------------
- Simplified the UseBundled Interpreter Logic
- revamped sigil_bs4 prettyprint_xhtml and serialzie_xhtml to be more robust
- update sigil_bs4 to use numeric entities for raw n0n-breaking spaces
- better handle void tags in sigil_bs4/prettyprint_xhtml and serialize_xhtml


20160130 (Sigil-0.9.3 release)
----------------------------------
- Improved plugin epub3 handling
- Added epub_version interface
- Many epub_utils bug fixes for font support
- sigil_gumbo_bs4_adapter bug fixes for namespaced attributes


20160313 (Sigil-0.9.4 Release)
-----------------------------------
- Fixed epub_utils IDPF font obfuscating routine to better match the spec.
- Other minor fixes and cleanups


20160325 (Sigil-0.9.5 Release)
-----------------------------------
- Changed sigil-bs4 to treat both ruby and rt tags as inline when prettyprinting xhtml


20160605 (Sigil-0.9.6 Release)
-----------------------------------
- added support for epub3 bindings element in opf


20160909 (Sigil-0.9.7 Release)
-----------------------------------
- extend validation plugin interface with add_extended_result() method to allow better cursor positioning
- remove support for python2.7 only plugins


20170227 (Sigil-0.9.8 Release)
-----------------------------------
- make Sigil's UI and spellchecker language settings available to plugins (bk.sigil_ui_lang, and bk.sigil_spellcheck_lang)
- fix bug in prettyprint_xhtml in sigil_bs4 (put back inadvertently dropped is_void_tag routine)
- harden plugin interface code to properly unquote/quote hrefs


Please note, currently the only supported versions of Sigil are the very latest release version (at this time Sigil-0.9.8), and Sigil master. If users are using any earlier versions of Sigil, they are strongly recommended to update to the current Sigil Release.

Hope this helps,

KevinH

Sigil Plugin Framework Documentation
 

Hi,

Please note there currently exists 4 types of Sigil plugins:

1. edit - used to create modify and delete files in the ebook currently being edited by Sigil

2. input - used to load a newly created epub into Sigil

3. output - used to read the ebook currently in Sigil and convert or generate a new ebook file you store externally. No changes to the current ebook inside Sigil are made.

4. validation - used to return error messages from epub validators run on the current ebook in Sigil. It inherits from the output plugin.


For specific information on interface calls available for each type of plugin, check out:

bookcontainer.py for the interface for an "edit" type plugin

outputcontainer.py for the interface for an "output" type plugin

inputcontainer.py for the interface for an "input" container

validationcontainer.py for the interface for a "validation" type plugin

The current Sigil Plugin Framework Documentation is up-to-date with the upcoming Sigil 0.9.991 (pre Sigil-1.0) release. Here it is as an epub for reference by Sigil Plugin Developers.

Hope this helps,

KevinH

Hi,

I just tested this and the metadata in the Sigil ebook was in fact properly copied over by that function in all of my tests.

So please post a sample epub that displays this problem as I can not recreate this at all.
I will track it down and get it fixed.

Thanks!

Kevin

Will try again tomorrow and do some additional testing first. From the looks of it, the opf is recreated. The manifest has a different sequence it seems.

Ok, I found the culprit. The difference is in the OPF, specifically the metadata tag. In the my test-book I have:
If I remove the red part, it does work. The tag is valid though as far as I know.

So, it appears that the OPF is not copied, but recreated.

I noticed something similar to this when using the epub3 output plugin, but hadn't tracked down the culprit before now. Having the opf prefix in the metadata tag (which seems to be valid according to Flightcrew) causes the resulting epub3 to be created without a metadata section in the opf. Might need to accommodate unnecessary, but apparently not invalid, prefixes in the opf.

Hi DiapDealer and Toxaris,

Thanks for figuring that out. The actual content.opf text itself is never used in the launcher. It literally parses the content.opf from inside Sigil and then provides an interface to change its pieces and to rebuild it on the fly.

The code to parse the Sigil content.opf simply did not properly handle the case of using explicit opf: prefixes on the tags. The content.opf that is rebuilt on the fly then was missing the metadata. The rebuilt-on-the-fly version does not use explicit opf: prefixing (as it is redundant give how it creates the metadata tag), the new version of the opf_parser.py simply removes any redundant opf: prefixes from tag names.

This in turn allows all of the metadata to be properly parsed and captured, and therefore the rebuilt on the fly version to now have the metadata.

The upcoming new release is expected shortly and it will have that bug fixed.

I have attached an updated opf_parser.py, that can be used until then.

Thanks so much for your bug report!

Kevin

Kevin, this version does handle the metadata perfect.

Another question, in the bookcontainer class (that is for edit plugins, right?), there is a function to replace files in the ePUB after adaptation, writefile.
I need the id from the OPF and I need to input data. For text based files that would be a string I pressume. However, what if I want to replace (or add for that matter) a non-text based file like a font, audio or video? Must I use some kind of binary stream?
Can you shed some light on that?

Hi Toxaris,

Yes that is the container class for "edit" plugins - it was not named "editcontainer.py" because it existed before we thought of the separate plugin types, and as such it kept its original name "bookcontainer.py"

If you are editing or changing an already existing file then "writefile" is the way to go (assuming the file is part of the manifest). The media-type field from the manifiest is used to determine what should be done with the data you want to write.

Based on the media-type if it is text (css or xhtml/xml) the data is tested to see if it is unicode or a bytestring. If a bytestring it is assumed to be utf-8, if unicode, it is converted to utf-8 and written to the output file.

Based on the media-type,if it is not text, the data is assumed to be a string of bytes and it is written (as binary) unchanged to the output file.


Now if you want to create an entirely new file that should be part of the manifest, you would use the "addfile". Here you can explicitly pass in a media-type or a media-type can be deduced from the filename extension. And again, the media-type determines how the data will be treated.

Hope this helps,

KevinH

@KevinH: Does bk.addfile() support bytearrays?

I did a quick test and came up with the following proof-of-concept code, which will add a blackletter ttf font located in the plugin's directory to the epub:

This code appears to be working, but I'm wondering if there is a better and/or more desirable method to add binary files?

Hi Doitsu,

Technically it is either a "bytestring" ie. a standard non-unicode string in python2.7 or it has type "bytes" in python3.4. But yes, as long as you specify the media-type properly, the binary data will not be touched and will be written properly.

A "bytearray" is a closely related but different type under both python2.7 and python3.4

I would simplify this only slightly and not use the bytearray class at all. Also the basename should not have the directory at all, the correct place inside of Sigil should again be determined by the mime-type/file extension automatically.

Using DiapDealer's KindleImport as a model:


FWIW: I personally do not like creating uuid based manifest ids. I have my own unique id creator routine I invoke. Something along the lines of:

id = create_unique_id("fnt", 0)

Others will simply pick a name and preface it with "x" until unique. There is of course no one "right" way.


Hope something here helps.

Kevin

Hi Kevin,

I must have mixed it up with other functions that require an href.

Thanks that is very helpful. I merely used uuid(), because it's so easy to use.

Thanks!
D.

Hi Doitsu,

Yes the "addotherfile" needs the book root relative href since it is for files outside the manifest. Anything in the manifest should be automatically put in the proper place by "addfile" unless I messed up someplace.

Take care,

KevinH

Another day, another question. I need to delete a file (a font) in the ePUB, so I use:The value of 'element[1]' is the manifest id, as it should. The strange thing is, that the manifest is correctly removed from the opf, but not the file itself. It still resides in the font folder. The internal check also does not report that there is an referenced file btw.

Is it a bug or am I doing something wrong?

@Toxaris: I was able to reproduce this bug. I also tried bk.deleteotherfile(), which removed the font from the Fonts folder in the Book View window, but not the actual epub file.

@KevinH: deleteotherfile() is defined as:

IMHO, book_href should be replaced with id.

Hi,

No, book href is correct for deleteotherfile. It would be incorrect for deletefile.

Kevin

Hi,
No actual deletes are done until the plugin completes. Only the fact you want to delete it should be recorded.

Perhaps there is a bug in the Sigil side not properly handling the delete or not properly updating the list of files in the viewer. Given it passes the check, we probably just need to update the viewer window inside Sigil.

I will look into this.

Thanks for the bug report!

Kevin

That was my hunch as well, that the viewer was not updated. However, if I save the ePUB and reopen, the file is still there.

You may want to double-check this one. If I use your improved code,

a font file with the href Fonts/WallauRundgotisch-Heavy.ttf will be added.

However, if I subsequently, comment out the last line of the code, replace it with a bk.deleteotherfile() statement and use the above href as the parameter:

I'm getting the following error message:

However, if I use the id instead of the href, I get no error message and the file is deleted both from the fonts folder and the manifest (but not from the actual epub file.)

D.

Hi Doitsu,

Please do not try and delete files that exist in the manifest with the "other" family of interface calls.

deletefile
addfile
readfile
writefile

are all designed to work only on manifest files. The new version of the launcher code will enforce that.

Almost all files should be in the manifest. To access files not in the manifest such as container.xml, mimetype, OEBPS/content.opf etc, you would use the following interface routines:

readotherfile
writeotherfile
addotherfile
deleteotherfile

There is a serious bug in the deletefile interface in both the wrapper.py and in the launcher.py that I now have a fix for. I will post them in one minute so please hang on.

Thanks!

KevinH

deletefile bug fix
 

Hi Toxaris and Doitsu,

There was a serious "thinko" in how I implemented how the plugin deletes files in the manifest that needed to be fixed. The previous version worked on files I added via plugin, but not on files that only existed in the original ebook.

I have now fixed that. Attached are two new versions of wrapper.py and launcher.py.

I have now committed them to my tree so hopefully they will be in time to make the upcoming release for Sigil 0.8.2

Also, since I can see that the launcher code needs to be updated faster that the Sigil main codebase (especially considering how many bugs you guys are finding!), I have developed a launcher_updater program, so that I can release launcher updates more frequently.

I have a rough working version of the launcher_updater but need to polish it before making my first release.

So until then, please grab and unzip the two files attached and add them to your plugin launcher code to replace their namesakes.

When Sigil 0.8.2 does come out, all of these changes will be included.

Please let me know if this does the trick.

KevinH

ps. DiapDealer just found and fixed another bug in deletefile so I have updated the wrapper.py yet again. When it rains it pours!!!

Great, will test it out tonight. With some luck I can polish it enough to be able to release my plugin tonight as well. It will be a font subsetter and image optimizer.

I get two errors in wrapper.py:
First a popup that says: Error parsing result XML: Premature end of document

Second the following error:
Traceback (most recent call last):
File "C:\Program Files\Sigil\plugin_launchers\\python\launcher.py", line 62, in <module>
from validationcontainer import ValidationContainer
ImportError: No module named validationcontainer

Of course I replaced both files you posted.

Hi Toxaris,

Unfortunately there are changes inside Sigil that are needed to make this version work. I will modify something for you to test with that does not need Sigil to be changed and post it for you this evening.

Sorry about that, I am using and testing with a pre-release version of Sigil 0.8.2 and not the version you are using.

Kevin

Hi Toxaris,

I stole a free moment at work to try and turn off the feature that allows plugin prints to appear immediately in the Sigil window. This feature requires changes in Sigil as well.

So please try the following version of launcher.py that should disable this new feature and hopefully work for you.

Sorry about that. I keep forgetting that Sigil 0.8.2 is not out yet!

KevinH

ps. To get around your second error you will need to add the following files to the plugin_launcher as well (after unzipping).

Unfortunately, the latest launcher version causes the following error message:

Do you have the compatibility_utils installed? The latest one?

At this point, your best bet is to grab all the latest launcher files from Kevin's github and then replace launcher.py with the most recent one he posted in this thread. That's what I did with Sigil 0.8.1 and everything worked as it should.

I've only installed the updates posted in this thread. If there was one for compatibility_utils.py I must have missed it.

BTW, given the many updates, it'd be helpful if the plugin launcher files had version numbers in the comments section.

I assumed that the current Github laucher files have been updated for the upcoming 0.8.2 version. For example, there are validator plugin python files, but, AFAIK, Sigil 0.8.1 doesn't have validator plugin support.
Anyway after grabbing the latest compatibility_utils.py version from Github the error disappeared.

No problem at all. I replaced all the files and I do get further. Now I get a popup saying: Error parsing result XML: Start tag expected

Then I see the results, but they are not pushed to Sigil.

Hi Toxaris,

The second modified launcher.py should have handled that error but I guess it did not. DiapDealer has said he needs other files from the new launcher as well to get anything to work with the Sigil 0.8.1 build.

So I will put together a full set when I get home from work tonight and modify them to work with Sigil 0.8.1 so that you have something to continue development with while we wait for Sigil 0.8.2 to officially come out.

Take care,

KevinH

Hi KevinH,

I installed all new files and didn't get any error messages. However, my test case still doesn't fully work.
When I add and remove a font with bk.addfile() and bk.deletefile (using the code that I've already posted), the font is added and subsequently removed from the .opf file and the Book View Fonts folder, but when I unzip the saved ePub file, it's still in the Fonts folder.

D.

Hi Doitsu,

I am confused. If you use bk.addfile to add a new file to the ebook and then use bk.deletefile to remove that same file, you should net to no change in the underlying epub? Or are you saying that a deletefile on some other file will not work after any addfile?

Please post your full testcase so that I can recreate what you are seeing exactly. The only code you had posted previously was mixing addfile and deleteotherfile which is always a mistake. If the file has been added to the manifest with addfile, you should delete it with deletefile NOT deleteotherfile.

Thanks,


Kevin

Hi Kevin,

To test bk.deletefile() I opened Sigil without a book ran the original plugin, which added the font file to the epub. So far so good. I then saved the file, reopened it, changed the the last file in the plugin code from:

to:

I then ran the modified plugin again, which removed the font that I added during the first run from the .opf file and the Book View Fonts folder but not the actual Fonts folder in the epub.

D.

Hi Doitsu!

Yes that is a bug! I can now recreate this when trying to delete any file in the original ebook. It just appears to go away but is still there somehow.

Unfortunately this bug is in the Sigil code base and not the launcher code so I can't fix this until the upcoming release, which should be soon but ...

I have just now committed a fix for this to my tree so that it is available when user-none does the official Sigil 0.8.2 builds. With this change Sigil will actually do the "physical" removal as well as the "logical" removal of the ebook.

There is no workaround I can do from my end. If you work on a Mac, I would be happy to build you a pre-release version of Sigil 0.8.2 so that you can continue to work. If Linux, I could probably get a Linux build done as well. Unfortunately, I do not own a Windows box and so could not make a build for you but perhaps DiapDealer would be willing to share his once he updates from my github tree to get this Sigil fix.

Thanks ever so much for catching this bug before the release!!!!!!

Take care,

Kevin

Hi Kevin,

Actually, all the credit goes to Toxaris who found the bug in the first place.

Thanks for the kind offer, but I'm going to wait for the official 0.8.2 release.

D.

No problem, I am glad that it was on time! I only found out about it since I actually wanted an edit plugin instead of a output plugin.

I will hold up the release of the plugin now and wait for Sigil 0.8.2 for testing. I might build Sigil myself, as I have a windows development machine with VS2010, but I haven't gotten around to reading the build instructions yet.

Hi Doitsu and Toxaris,
user-none has just released Sigil 0.8.2 which should now do what you need to get deletefile and your plugins going.

Please let me know if you still run into any issues.

Thanks,

KevinH