Sigil script manager

[varlog]

Originally Posted by DiapDealer

... but it might no look close anything like the book did when it was a mobi. ...

Yes, all HTML3.2 formatting is gone - if that's what you mean. Are there some better options out there?

[DiapDealer]

Not that I'm aware of. Unless you want to take a look at calibre's source. It unpacks mobis into an oebps-type structure for viewing with it's internal viewer. I think it uses html5lib to serialize the markup unto xhtml. But I'm also pretty sure the very latest version of html5lib has removed its xhtml serializer... for what that's worth. The lxml module's serializer doesn't do a bad job of whipping mobi markup into shape (as long as you fix up the page-break tags and a few other things before-hand).

[varlog]

Actually I did took a look at Calibre, attracted by see picture... but I'm not on (sucking )python trip at the moment: I'm on c++/qt thing now.

[DiapDealer]

Originally Posted by varlog

Actually I did took a look at Calibre, attracted by see picture... but I'm not on (sucking )python trip at the moment: I'm on c++/qt thing now.

Quite understandable.

[varlog]

I did not have much time to do something nice lately... but...
dealing with Skin Games I realized my approach to porting existing scripts to DGSM is totally wrong! I should have been writing wrappers only! with no changes to original.... understandable - I don't know python, so changing is easier es creating (where are you, all you python zombies?).
My real somehow cryptic question is: can I post on this forum Alf2Sigil if/when I have it? Only as an Alf2Sigil.xml and wrapper.py?

[varlog]

walking through the net one day I've found this- so I wrote toc2nav as my contribution to Sigil epub3 functionality. It converts toc.ncx into epub3 nav toc and quide from opf into nav landmarks and adds it as nav.xhtml to Text files.

In the meantime I've modified DGSM a little.
DGSM 0.0.2 changes:
I've added loadNewBook to sigilActions as a preparation to write really clean importing alien formats. The idea is: when DGSM encounters noResource as required resource it gives the script the path to temporary data folder. The script could create this folder and leave something in it. On script departure Sigil checks the folder, do something with its contents -if any - and deletes it. At the moment by loadNewBook action Sigil opens the first epub book found there.
Got rid of some bugs, I think, I hope. It compiles on Windows but I have not done any tests.

Always use the latest DGSM with latest scripts package because I could have changed something.

The files are featured at the beginning of this thread.

[KevinH]

Hi,

Just found this thread. I have a python script that converts old mobi markup language to normal xhtml (python based) and I would be happy to incorporate it into KindleUnpack future versions.

Also, have you looked at how calibre handles plugins and by that I mean their philosophy. They include the various plugins by having them inherit from a generic plugin class specific to their type, and overload the abstract methods defined therein. They allow different types of plugins, such as plugins the run on input, output, etc. There could also be clean-up plugins as well.

I will download your latest version and get it to build on a Mac and send you build fix patches if need be. I would very much like to add something along these lines to my own copy of Sigil specific only to python scripting, that will invoke the python interpreter as a library, map a C++ plugin class to a python plugin class, run things in their own worker threads etc,

Take care,

KevinH

[varlog]

Originally Posted by KevinH

Hi,
....

Also, have you looked at how calibre handles plugins and by that I mean their philosophy. They include the various plugins by having them inherit from a generic plugin class specific to their type, and overload the abstract methods defined therein. They allow different types of plugins, such as plugins the run on input, output, etc. There could also be clean-up plugins as well.

...

My problem is: I don't speak python. And this is my frst C++/Qt.

Originally Posted by KevinH

Hi,
....
I will download your latest version and get it to build on a Mac and send you build fix patches if need be.
...

I appreciate this very much.

Originally Posted by KevinH

Hi,
....
I would very much like to add something along these lines to my own copy of Sigil specific only to python scripting, that will invoke the python interpreter as a library, map a C++ plugin class to a python plugin class, run things in their own worker threads etc,
...

Well, that would be proper python interface: I hoped I'll do it when I'm finished with DGSM (if ever). Now, I hope you'll do it!

[KevinH]

Hi Varlog,
I exchanged e-mails with Sigil's owner user_none, and he is willing to accept patches along the lines of your script DGSM approach for extending Sigil. He felt that would be a lighter, faster, more flexible way of extending Sigil than a more formal plugin structure that could be developed later.

So I am going to try and build what you have done, update it to current trunk, and see if I can help you work on it.

user_none has given me some tips about the process of flushing everything to disk before a script is run. He wrote ...

The issue you're going to run into is every file is loaded as a
resource. So you need to do something like this

a) Save all tab data.
b) Save resources to disk.
c) Run plugin.
d) Add and Remove resources from the FolderKeeper as needed.
e) Reload all resources from data on disk.

After eyeballing your patch, I wasn't sure you had saved all of the current data in the tabs. If not, we should probably add that. Another idea he had was to invoke the script using a modal Dialog with an infinite progress bar that waits and blocks editing until the script has completed.

Anyway, please post the very very latest version you have, and I will try to build it on trunk with a Mac and work with you to improve on it so that we can offer it to user_none and hopefully get him to add it to the official tree.

Take care,

KevinH

[varlog]

I've uploaded DGSM-0.0.3, which is basically just DGSM-0.0.2 rebased on your latest work (there were conflicts). There are some other small changes. The source is raw, I haven't cleaned it lately.

Originally Posted by KevinH

...
I exchanged e-mails with Sigil's owner user_none, and he is willing to accept patches along the lines of your script DGSM approach for extending Sigil. He felt that would be a lighter, faster, more flexible way of extending Sigil than a more formal plugin structure that could be developed later.

So I am going to try and build what you have done, update it to current trunk, and see if I can help you work on it.
...

It is a very nice development. It is too early to include it in main Sigil tree of course. Perhaps a branch, if you'll find it worthy?
Then again, judging from this thread's traffic, the Sigil community here (DD being an exception) is not very much interested in this functionality?
Anyway, my itch is as strong as ever and I'll be scratching it. Any help/comments are really really welcomed.

Originally Posted by KevinH

...

user_none has given me some tips about the process of flushing everything to disk before a script is run. He wrote ...

The issue you're going to run into is every file is loaded as a
resource. So you need to do something like this

a) Save all tab data.
b) Save resources to disk.
c) Run plugin.
d) Add and Remove resources from the FolderKeeper as needed.
e) Reload all resources from data on disk.
...

a) I have done almost nothing with tabs.
b) only selected resources are saved at the moment
d) I do this, I think
e) depending on sigilAction I reload selected or all

Thank you for your time.

[KevinH]

Hi Varlog,

Yes, I think we can get the script manager into main branch just fine if we make the following additions:

1. When scripts are registered the user must select a type from the following list:

1. Input Script
2. Output Script
3. Editor Script
4. Metadata Script

That is stored in Sigil SettingsStore.

This "type" will determine the interface and arguments to be passed to the script

2. Add the modal Dialog with infinite progress bar and cancel button. Upon completion it will show success or failure and if failure a error widget with text gathered from the QProcess standard error pipe.

3. In order to pass results and information back to Sigil we have only the exit status, and, stdout and stderr pipes/channels from the QProcess to work with. Therefore we will serialize the required output into an xml stream to pass back in the stdout channel.

4. Inside Sigil we can read this file after the script finishes and parse it easily using a simple xml parser to get the results, files modified, etc. We can write wrapper scripts in python and other languages to illustrate how output stream is done. The user would simply copy this wrapper, and use it to call their script. The exact language of the serialized xml result file will of course vary by script type.


My main weakness is Qt widgets and ui, whereas I understand forking processes and working with and serializing process pipes/channels quite well. So I propose we divide and conquer.

If you could focus on items 1 and 2 above (extending the script registration to include it script type, and creating a modal dialog with infinite progress indicator, a cancel button and a popup text window to show error results), I can focus on items 3 and 4 and writing a python wrapper script with functionality similar to what calibre plugins can do.

How does that sound?

Take care,

KevinH

Originally Posted by varlog

I've uploaded DGSM-0.0.3, which is basically just DGSM-0.0.2 rebased on your latest work (there were conflicts). There are some other small changes. The source is raw, I haven't cleaned it lately.


It is a very nice development. It is too early to include it in main Sigil tree of course. Perhaps a branch, if you'll find it worthy?
Then again, judging from this thread's traffic, the Sigil community here (DD being an exception) is not very much interested in this functionality?
Anyway, my itch is as strong as ever and I'll be scratching it. Any help/comments are really really welcomed.



a) I have done almost nothing with tabs.
b) only selected resources are saved at the moment
d) I do this, I think
e) depending on sigilAction I reload selected or all

Thank you for your time.

[varlog]

Sounds fantastic.

Originally Posted by KevinH

...
1. When scripts are registered the user must select a type from the following list:

1. Input Script
2. Output Script
3. Editor Script
4. Metadata Script
....

Not quite understanding this new architecture yet but: I can add new attribute "type" to script manifest with appropriate values:

 
<attribute name="type" use="required">
    <simpleType>
      <restriction base="string">
       <enumeration value="input" />
       <enumeration value="output" />
       <enumeration value="edit" />
       <enumeration value="meta" />
       <enumeration value="other" />
      </restriction>
     </simpleType>
  </attribute>

and remember it in Settings and elsewhere, so that ScriptInterface class (or other classes?) could react to this. This will make my sigilResource tag redundant. Do you propose to remove it? The whole architecture depends on it at the moment.
Just in case I added "other".

Originally Posted by KevinH

...
2. Add the modal Dialog with infinite progress bar and cancel button. Upon completion it will show success or failure and if failure a error widget with text gathered from the QProcess standard error pipe.
...

Ok, I'll look into it... just be easy on me, I'm newbie...

Regards

[KevinH]

Hi Varlog,

Not quite understanding this new architecture yet but:

The general idea is to register a script and it interpreter path as you have done along with a type attribute.

When a user launches the script, a Modal Dialog Pops up with infinite progress widget and a cancel button, and it gets passed all of the info needed for the script when it invokes ScriptInterface.

ScriptInterface invokes QProcess so that it exists as an object even though the method code that started it has closed. This QProcess will be passed a specific set of arguments depending on its registered type:

For an editor type it will be passed:
- path to the root of the book (read only)
- path to the opf of the book (read only)
- path to a temporary directory where all results will reside

The wrapper python code (or any code for that matter) will read these values, copy into the output space any files it needs to modify, and proceed to modify those files.

Depending on the wrapper language, the wrapper can add xml parsing, xpaths, spine decoding, opf parsing (to get back a list of id by mimetype for example). The wrapper can also use its own gui widgets at this point to ask for paths, user input, etc.

Then the wrapper script will import the actual target script and run it collecting the output and any error messages.

Once the target script has returned, the wrapper script will build a simple xml file that it will print to stdout before exiting. It would look something like the following in its simplest form (this is just off the top of my head, we can adjust this as we see fit)

<wrapper charset="utf-8">
    <result>success</result>
    <deleted>
        <manifestid>id</manifestid>
        <manifestid>id</manifestid>
        <manifestid>id</manifestid>
        <manifestid>id</manifestid>
    </deleted>
    <modified>
        <manifestid>id</manifestid>
        <manifestid>id</manifestid>
        <manifestid>id</manifestid>
        <manifestid>id</manifestid>
    </modified>
    <new>
        <file href="url" mimetype="newfilemimetype" uniqueid="id" />
        <file href="url" mimetype="newfilemimetype" uniqueid="id" />
        <file href="url" mimetype="newfilemimetype" uniqueid="id" />
        <file href="url" mimetype="newfilemimetype" uniqueid="id" />
    </new> 
    <errorlog>
          "escaped text from target script stderr output only if failed"
    </errorlog>
    <successmsg>
          "escaped text from target script stdout output only if success"
    </successmsg>
</wrapper>

For an input script, we would only pass in the temporary output folder and then its wrapper would basically pass back xml that represents all of the new files (or simply a path to the new root and new opf),

For an output script, we would pass in the read-only path to the root of the book and path to its opf file.

The output wrapper would return a a similar xml success script along with a successmsg of where the user can find the file.

The ScriptInterface code would then grab the wrapper result xml and use it to perform all of the required manipulation inside Sigil to accomplish what needs to be done, show either the error message or success message and wait for the user to click <done> ending the model dialog.

The key idea is that I can write the python "wrapper" codes for all of these types that users can simply use to copy and paste in importing their own "target" code and getting it all to work. We can also develop wrappers for other crossplatform scripting languages such as perl , Lua, Javascript, etc.

I can add new attribute "type" to script manifest with appropriate values:

 
<attribute name="type" use="required">
    <simpleType>
      <restriction base="string">
       <enumeration value="input" />
       <enumeration value="output" />
       <enumeration value="edit" />
       <enumeration value="meta" />
       <enumeration value="other" />
      </restriction>
     </simpleType>
  </attribute>

and remember it in Settings and elsewhere, so that ScriptInterface class (or other classes?) could react to this. This will make my sigilResource tag redundant. Do you propose to remove it? The whole architecture depends on it at the moment.

See the above which should hopefully make the idea more understandable.

Just in case I added "other".

Please don't. I have no idea what a wrapper class would be for other and how it would interface. We can always include other concrete script types in the future when and if we need more script types.

Ok, I'll look into it... just be easy on me, I'm newbie...

No worries! Once you see how the interface wrapper works, all of this should make more sense. Just remember, we will not be just integrating in any old script, instead script writers will use our premade wrappers scripts to add their code to it to create a proper script plugin.

I can create one for KindleUnpack, so that Sigil will gain full mobi support, one to invoke kindlegen to allow for Mobi output support, scripts to strip koboisms, scripts to create nav (like yours), scripts to add convert current meta data to epub 3 versions, etc.

We can also actually add public methods to the friended classes so that we no longer need the friend stuff nor the pre C++11 switch either.

Hope this all sounds good to you. If not, let me know.

Thanks,

KevinH

[varlog]

I will.

[KevinH]

Hi varlog,

I have the first python launcher script for edit scripts almost ready to go. Hopefully, I can post a zip for you to play with both within and outside of Sigil tomorrow. I will build a simple Qt program, you can test with outside of Sigil, until we get the widget and modal dialog parts going.

KevinH