Sigil script manager

[varlog]

Introducing DGSM.

Spoiler:

This assumes that some of you will be able and willing to play with it.

I'm writing some code for Sigil - to scratch my itch and to learn some C++, which I wanted to do for a considerable time now. As C was many many years ago my (unreturned ) love at first sight, I had thought embracing C++ would be the next step to digital nirvana, but I chickened out up to now. Well - now, almost half a year later digital nirvana is nowhere to be seen but I think I'll code from now on in C++. And probably Qt too - in a way of user documentation Qt is something of developer wet dream.

The itch comes from this discussion.
It is learn-by-doing endeavor - I've just started to do something I have no idea about and my plan evolves as I go. The general plan was to equip Sigil with plugin functionality, which was thought of but never emerged.

So as I go my knowledge of C++/Qt/Sigil/XML/Epub grows, so the project mutates. Actual code is something like second full rewrite and many many more partial rewrites. Actually I'm continuously rewriting.

I'm mostly only adding to original code - the Sigil original functionality is at the moment off limits for me.

I do it using Qt Creator in Linux (Ubuntu 12.04 64b) and I have no idea if it works on other platforms. I do try to use what I hold for generic C++/Qt so there is a good chance that that it almost not platform dependent? This is one of the secondary reasons for this expose - I'm curious how much Qt really abstracts.

Disclaimer: I am not willing to set up c++ development environment in Windows just for this project... and I'll never ever posses an Apple device: doesn't agree with my ambiance .

This is not! plugin interface for Sigil. It is just DUPA or ASS: Dumb Unified Processing Addition or Additional Scripting Supplement... actually it is DGSM: Dumb Generic Script Manager. Let's stick with it.

DGSM allows to execute from within Sigil a sort of call "[interpreter][parameters] scriptName[.scriptSuffix] [parameters]". Similar to Sigil context menu "Open with" function, just trying to be more general.
Theoretically interpreter could be anything: bash, cmd, python, Java, JavaScript, Basic... whatever. I concentrated in my tests on Python - as it seems to be language of choice in this environment.
It exposes to invoked scrip some internal data structures of Sigil, temporary Sigil folder mostly and actually selected items, and tries to react to the eventual changes the script has made.
The communication between script and Sigil is at the moment very unsophisticated one: script has to have XML manifest: scriptName.xml where script exposes itself and all its necessities.

The actual Sigil script XML manifest definition is:

tag "sigilScript": root tag
|has three attributes:
|interpreter=name of interpreter program ("python", "java"... etc)
|internal="true"/"false" -at the moment ignored! to be used for real plugins! if ever...
|resident="true"/"false" - should it appear in some Sigil menu?
|includes tags:
|tag "name": name of the script, necessary, no spaces, letters/numbers only!
|tag "author": author
|tag "description": short descriptions
|tag "files": script file tree
||includes:
||tag "file" : name of file
|||has one optional attribute "type":
|||type="entry"/"icon":
|||entry: entry script, necessary for one file!
|||icon: icon for menu, not implemented yet.
|||one file has to have attribute "entry"
||tag "dir":
|||has one no-optional attribute:
|||name="name of the folder"
|||to be on the safe side don't use spaces, letters/numbers only!
|||tag "dir" could include tag "file" and tag "dir" (scary... recursion!).
|tag "requires":read as "script requires"
||includes tags:
||tag "sigilMenu": menu text, should be human readable script name.
|||has one optional attribute "menu":
|||menu="menuBar"/"context"/"none" - what menu should script appear in
||tag "sigilResource": should be one of:
|||noResource/generic/text/xml/html/css/image/svg
|||/font/opf/ncx/miscText/audio/video/book/selected/all -
|||what resource script needs from Sigil
||tag "sigilAction": noAction/reloadResource/markBookChanged/reloadBook/:
|||what should Sigil do after the script is done

Not all is implemented yet of course. Perhaps not all should be, perhaps I've missed something essential, perhaps it is all beside the point: answer to this questions is one of primary reasons for this expose.

I've ported all known to me Sigil scripts to DGSM. Just to see what is possible I did minimalistic port of Mobi_Unpack_v053 too: it is supposed to read .mobi (and other formats?) directly into Sigil. As I don't know Python (Python sucks: I miss my lovely curly braces, indent rule is major PITA!, use PyDev in Eclipse, better ideas?) I've done a poor job of it. I've tested it on some .mobi .azw .prc files and it works on my test files in my enviroment, but there be sure dragons there.
I've put all those scripts in file pluginsDGSM-0.0.0.zip. You will find there also the actual XML schema of the script manifest: sigilScript.xsd - subject to alternations!

Global things:
I've changed C++ standard: C++x0 is now set up in cmake, C++98 won't compile now (initializations, lambda functions I wanted to explore but haven't)! Because of that - narrowing conversions C++0x error - I had to modify (scary,scary!) Sigil/src/Xerces/xercesc/util/regx/ParserForXMLSchema.cpp and Sigil/src/Xerces/xercesc/util/regx/RegxParser.cpp.

Qt version: I use 5.2.1 - but it should not matter.
The number of scripts handled is theoretically 999 (settings thing, easy to change).

Eventual memory leaks could've escaped me. Threads and concurrency I'll be glad to learn of. I'm trying to use existing Sigil functionality as I'm unraveling it - sometimes it's revelation, sometimes PITA.

I've added three new classes: ScriptWidget, ScriptManager and ScriptInterface. I've modified MainWindow, BookBrowser and Preferences to accommodate them.
MainWindow and BookBrowser include now script functionality elements. Depending on sigilScript resident attribute script will appear - depending on sigilMenu menu attribute under Scripts menu on menu bar or in right mouse click context menu.

Preferences accommodates ScriptWidged.
ScriptWidget is responsible for panel Edit->Preferences->Scripts which is front end for ScriptManager class. You can add/remove interpreter, set path to it, set path to script folder, install/remove/execute scripts with it. There are default values for paths which function perfectly for my Linux and should hopefully do something reasonable in other environments?.

ScriptInterface is responsible for executing script. This is the primary reason for this expose. What is the minimalistic useful functionality? My temporar goal was to be able to execute all known to me Sigil scripts. It is surly not enough.

This code is not even pre-alpha: it is just feasibility study/learning project. Don't use it in production environment! The sanity checks are sparse. After I managed to erase my home directory I've build in some - so read carefully any pop up messages before clicking OK.

After starting your successfully(?) compiled DGSM-Sigil you should import included (unzip first!) scripts through Edit->Preferences->Scripts->install button: the interpreter will be created automatically, you must check the "Interpreter Path" value and eventually per "Browse" or typing in the path point to the interpreter executable and set your preferred interpreter parameters. Alternatively you can set up interpreter per "New interpreter" button and than point "Scripts Path" per "Browse" or typing in the path to folder containing unzipped scripts. I included only python scripts, so it's python you'll want to configure. And than you can start to play with it.

The included patch is based on at the moment latest Sigil git commit.

Depending on response I'll consider future actions.

In case somebody wonders: YES, I'm aware that in the meantime Calibre has an editor with functionality approaching Sigil level, growing fast. It doesn't concern me, not my itch.

Last but not least: If you have not understood it - just forget it!
If you have doubt(s) or are curious - well, feel free to ask...

The pictures:

About informs you now that DGSM has proudly reached version 0.0.0
Next you see Script Manager Cockpit.
Script menu in menu bar.
Context script menu.

[DiapDealer]

Cool! Can't wait to play.

[varlog]

Hmm...
depending on your conclusions... would you consider making .deb package for others willing but not able?

[signum]

How exciting! It sounds like you are well on the way and have learned a lot already.

This project reminds me of a feature I wish more editors had. My model is the vi editor's shell escape. For those who are unfamiliar with vi, the shell escape will create an anonymous stream of lines that are piped into the program of your choice. The output of your program is automatically piped back in to vi.

The program can be anything, even a shell script on its own. The major rule is that all input and output are pipes. A consequence of this is that it is allowable to read a line from the input pipe, process it somehow, and write the edited output to the output pipe. When the input reaches end of file, everything is closed and control returns to vi. This is in contrast to the "open with" feature, where all input is delivered at once to the editor, all changes are made, and all output is written at once back to sigil. I would call this a simplex pipe. Either input all at once or output all at once. The shell escape is a pair of pipes that allow partial reads and partial writes to be interleaved. Much more in line with the Unix stream concept.

Were this available to sigil, you could pipe your file to stream-oriented programs such as sed, awk, wc, sort, etc. All non-interactive common utilities, but they each do one thing, and do it well.

Just a thought for your consideration.

[varlog]

Originally Posted by signum

Were this available to sigil, you could pipe your file to stream-oriented programs such as sed, awk, wc, sort, etc. All non-interactive common utilities, but they each do one thing, and do it well.
.

I personally think that what Sigil really needs now is:
1. Interface attractive for users who want the bloody thing just to work.
2. Internals attractive for people who like to code.

which would be my imaginary goal if I had one .

[DiapDealer]

I haven't had much time to play, but I did want to let you know that I got Sigil patched and built OK from your patch file. DGSM is there and the one script I tried seemed to work. That's about it for now.

[DiapDealer]

Been using this quite a bit lately.

I even made a few changes and got it to compile on Windows. I've chosen to use Sigil v0.7.3 to patch (I'm just not confident in v0.7.4 or the latest git). It "feels" a little precarious on Windows, but it works! I've had a few random crashes, but that could be my fault--with regard to the changes I made to get it to compile.

One thing I'd be interested in would be the ability to determine whether or not the script actually changed anything. It'd be nice to be able to skip the whole reloadResource or markBookChanged if the script in question didn't find anything to do. For example; if the smarten punctuation script didn't find anything to smarten, it'd be nice if Sigil didn't change the project to an "unsaved" state.

Cool stuff!

I do have a 64-bit Linux version available (with Qt5.2.1) if anyone is interested in playing. It should install alongside the standard version of Sigil (although both versions will still share the same config directory).

I can make the Windows version available as well--if anyone is feeling adventurous. You're on you're own though if something blows up!

[varlog]

Originally Posted by DiapDealer

Been using this quite a bit lately.

I even made a few changes and got it to compile on Windows. I've chosen to use Sigil v0.7.3 to patch (I'm just not confident in v0.7.4 or the latest git). It "feels" a little precarious on Windows, but it works! I've had a few random crashes, but that could be my fault--with regard to the changes I made to get it to compile.

Is your windows environment perhaps 32b? If yes I'd be interested in executable (have 32b XP bitrotting somewhere around), in any case could I have the changes you've made, please?

One thing I'd be interested in would be the ability to determine whether or not the script actually changed anything. It'd be nice to be able to skip the whole reloadResource or markBookChanged if the script in question didn't find anything to do. For example; if the smarten punctuation script didn't find anything to smarten, it'd be nice if Sigil didn't change the project to an "unsaved" state.

Well, yes... I wanted it too, but the work involved - save Sigil state, let script run (amok?), compare new Sigil state to old one - would mean some design decisions which I was not prepared to take yet. But it is on my wish list too .
EDIT: it would be relatively easy to implement it as a reaction to signal from script itself - but, as I don't write the scripts myself only sigil-malform existing ones, this solution seems not general enough to me.

There is one uninitialized int i in ScriptInterface::removeNotAccountedFor (ScriptInterface.cpp, line 401)
is: for(int i;i<htmlFiles.count();++i){
should be!: for(int i=0;i<htmlFiles.count();++i){
in case your compiler hasn't noticed!

In the meantime I've noticed some more things which must be absolutely taken care of...

Thank you for your input .

[DiapDealer]

Originally Posted by varlog

Is your windows environment perhaps 32b? If yes I'd be interested in executable (have 32b XP bitrotting somewhere around), in any case could I have the changes you've made, please?

I don't mess around with trying to build the 64-bit version of Sigil on Windows anymore. I built the Windows installer package. You're welcome to it. I'll upload it somewhere and post a link.

The only real changes were to the ScriptManager.h file. The QStringList constants at the top were giving the Windows compiler fits. It sees curly braces and automatically thinks "Aha! A function!" and then immediately barfs because it has no parameters. I just constructed the QStringLists a little differently. I have no idea if those changes will consequently work on Linux. I just kept playing 'til it compiled, and then quit.

I also needed a few tweaks to the main src/Sigil/CMakeLists.txt file to build the installer package because of changes to Qt5. More things have been moved to plugins and new libraries added (and I think Qt5V8 went away). Qt5Positioning needed to be included with the Windows installer for the program to run. I also included the position plugin just to be on the safe side.

There is one uninitialized int i in ScriptInterface::removeNotAccountedFor (ScriptInterface.cpp, line 401)
is: for(int i;i<htmlFiles.count();++i){
should be!: for(int i=0;i<htmlFiles.count();++i){
in case your compiler hasn't noticed!

It barked, but it didn't barf! Should I recompile with that change before uploading the Windows installer? Or won't it matter?

[varlog]

sorry for late answer: on the road again with no internet .

Originally Posted by DiapDealer

It barked, but it didn't barf! Should I recompile with that change before uploading the Windows installer? Or won't it matter?

it is loop counter, it MUST be initialised to zero.

[DiapDealer]

Gotcha.

Here's the link (google drive) to the Windows installer package.
https://drive.google.com/file/d/0B5C...it?usp=sharing

[varlog]

This is a fix for some things I wanted to fix:
- compiles and functions similar in Linux and Windows XP32b (yes, I went so far as to set up compiling environment in my 32b long since forgotten XP abomination, perhaps it is enough for all other vista,7,8 sucking derivatives, this is ridiculous, for all of this is DiapDealer (thanks!) to blame! )
- added icons to menus and scripts (see snapshot, paid for, EUR 17!) just for fun of it!
- replaced Mobi2Sigil with Kindle2Sigil based on KindleUnpack_v65: it is a slightly better port as the Mobil2Sigil but still - less dragons doesn't mean no dragons, anybody out there who knows better? I ignore KindleUnpack creators original intent and try to make .epub direct from old mobi7 files: some info/comments on it?
- added WhatsMyMobi because it was easy and could help by Kindle2Sigil.

DGMS-0.0.1.zip includes full patch to Sigil git and patch to DGSM-0.0.0.

could somebody check it on mac, please?

[DiapDealer]

Originally Posted by varlog

I ignore KindleUnpack creators original intent and try to make .epub direct from old mobi7 files: some info/comments on it?

You definitely have your work cut out for you there. The mobi markup in the older, "standard" mobi files is similar to HTML 3.2. Getting it cleaned up and into some semblance of valid XHTML (appropriate for ePub) can be a trying experience.

[varlog]

Originally Posted by DiapDealer

You definitely have your work cut out for you there.

What I wanted to have was the shortest way from .mobi to .epub in Sigil. For this I malform KindleUnpack output into .epub structure, never minding what input was. Sigil has to have "Clean Source" on by "Open" for Kindle2Sigil to work - shoud've done it programmatically but havn't - so it cleans a little by reading files in. If "Clean Source" is on by "Save" it cleans some other things missed when saving... What remains are straightforward regex cases... I think.
Actually I'm not in the business of transforming .mobi to .epub, sometimes I just want to take a look inside.

[DiapDealer]

I'll be honest; Sigil's "Clean Source" upon opening may very well close the tags on mobi6|7 markup and make it valid xhtml, but it might not look anything like the book did when it was a mobi.

But if you just want to "let 'er fly." Most mobis unpack into everything you need to slap together an ePub (html, opf, ncx). There's some python code in KindelUnpack that should help you zip up all the files in the correct manner--and create the mimetype/container.xml files.