Sigil script manager

[varlog]

Hi Kevin,
I have implemented type attribute for scripts, now I'm playing with modal dialog: to do something useful I'll need precise definition of "input" "output" "edit" and "meta" in Sigil context. Hopefully your script will define "edit" for me at least.
This working on copies of files to be changed is something I should have been doing from the begining...

[KevinH]

Hi,
I will post the edit type python 2 script launcher/wrapper with a test script hopefully tonight or tomorrow. I'm still ironing out the bugs. I have added all the opf parsing and iterators you had (implemented in python) and only copy files on write to the output directory.

I would also like to add a simple stream xhtml parser to the wrapper script so that it is easy for people to edit xhtml files without knowing a full dom or xpath.

Take care,

KevinH

Originally Posted by varlog

Hi Kevin,
I have implemented type attribute for scripts, now I'm playing with modal dialog: to do something useful I'll need precise definition of "input" "output" "edit" and "meta" in Sigil context. Hopefully your script will define "edit" for me at least.
This working on copies of files to be changed is something I should have been doing from the begining...

[KevinH]

Hi Varlog,

As promised:

This is a Python 2.7 or 2.6 script but not a Python 3 script

The edit_launcher.py needs the following 5 inputs:

1. path to ebook saved as epub by Sigil and unpacked into this directory

2. path to the opf file in that ebook

3. path to the output directory

4. the name of the target python script module you want to run
(do not add the .py)

5. the directory which holds the target pythoin script you want to run


If you unpack the zip I uploaded into your home directory it will generate:

- an empty ebook directory (ebook) into which you should unzip an epub previously save by Sigil

- a launcher directory which will become part of Sigil - Sigil will actually always run QProcess on the edit_launcher.py script which in turn spawns a thread and runs the target script

- a target_script directory where I have placed a test target python script

- and an empty testout directory (which is a placeholder for the output directory Sigil will create

Once you have unpacked an epub Saved by Sigil in ebook/ you can basically test the edit_launcher.py by invoking it from the command line just by doing the following (change to match where you unpack things)

python edit_launcher.py ~/ebook ~/ebook/OEBPS/content.opf ~/testout test_target ~/target_script

You should see the result XML file with all of the stdout and stderr logs from the target script. I do that so I can validate the output, that can be turned off when in production

python target script programmers can check out the edit_launch.py script and
the bookcontainer.py script which represents the real interface.

The test_target.py script simply tries to exercise that interface.

Please give this a try and see if this helps explain the idea.

Just let me know if you have any questions at all.

[KevinH]

Hi,
After thinking about it last night, I probably need a way to access all of the non-manifested files for reading and writing through the wrapper in bookcontainer.py . You would need something like this to deal with encrypted fonts for example. Perhaps an iterator for non-manifested files returning their unique path relative to the ebook root. Alternatively we could just provide a full book relative path iterator.

I would like to add an xml.etree parser as well for dom access and xpath. Since this is an external python interpreter, we don't know if they will have external python packages like lxml, Qt, html5lib, etc, so we are limited in what we can do.

For output scripts, I would like Sigil to generate an ask folder style gui element and pass the result to the output script to use.

Similarly, for input scripts, I would like Sigil to generate a file dialog and pass the path to the selected item to the script. Somehow, we will need to figure out a way to set file type filters for it, but I am not sure how yet.

Instead, we could assume that the installed version of python has some basic GUI widgets kit like Tk and let the target script deal with all of that.

Please let me know what you think about those ideas.

KevinH

[varlog]

Wow.

I'm trying to warp my mind around your wrapper... It will take some time.
Just a few first thoughts.

1. I differentiate between menu bar script and context menu script and am somehow fond of it. Context script operates "only" on selected items. Either tell me why not or edit_launcher should be able to take selected items as arguments.

2. I have assumed that all script input data will be referenced to/copied from Sigil temporary work folder. When you know Sigil(!) epub root directory, you know the path to opf (./OEBPS/content.opf). Do we need to pass the path to it nevertheless?

3. Doing tests with test_target I got Manifest ID error: I think it's because Sigil doesn't write mimetype file in its tmp book folder - which I used for test at first. I think we don't really want to make epub from Sigil temporary files and then unpack it somewhere else?

Just to clarify:
"input" means Sigil will be reading in epub produced by script,
"output" means script will transform data from Sigil and write it somewhere outside Sigils jurisdiction.
"edit" means script will change Sigil data and Sigil will have to accommodate it.
"meta" is actually "edit" but only for metadata.

is this correct? Why do we need "meta"?

Since this is an external python interpreter, we don't know if they will have external python packages like lxml, Qt, html5lib, etc, so we are limited in what we can do.

The script should be able to use all generally available libraries or provide its own libraries. I would be unscrupulously self-indulgent here. Some information about it will be due, i guess.

thx

[KevinH]

Hi varlog,

1. I differentiate between menu bar script and context menu script and am somehow fond of it. Context script operates "only" on selected items. Either tell me why not or edit_launcher should be able to take selected items as arguments.

Should not be an issue, we can simply pass along information via command line to modified edit_launcher.py and it can add info to bookcontainer.py passed to target script

2. I have assumed that all script input data will be referenced to/copied from Sigil temporary work folder. When you know Sigil(!) epub root directory, you know the path to opf (./OEBPS/content.opf). Do we need to pass the path to it nevertheless?

And yes we could build opf_path if we know Sigil will always name the opf as content.opf and not target.opf or allow the base name to be language specific.

The wrapper knows the output dir and ebook root, but it is hidden from bookcontainer.py which is the object passed to target scripts run() method.

All file access is done thru bookcontainer.py through the wrapper and it copies nothing until you actually want to write out a modified version. Reading from the true Sigil ebook files is also wrappered but allowed. The actual files inside Sigil are never modified and only the modified or added ones ever get written to the output directory

3. Doing tests with test_target I got Manifest ID error: I think it's because Sigil doesn't write mimetype file in its tmp book folder - which I used for test at first. I think we don't really want to make epub from Sigil temporary files and then unpack it somewhere else?

The manifest id error came from where? If you can capture the console log to a file and zip it and post it, I can try to figure out what is happening. If it is the test_target.py code, then the provided content.opf file manifest is incomplete or messed up (or a bug exists in my launcher code that parses the opf to create the initial state of the wrapper! The actual mimetype file is never looked for or used, just the mime media-type from the manifest section of the opf.

Or, alternatively, perhaps my post was unclear ... the instructions I sent were to test with from a Linux/MacOSX terminal bash prompt command line, not to interface into Sigil yet. So to *simulate* what a Sigil ebook directory might be like, I asked you to save an epub from Sigil and then unpack it. So please just test from command line (not part of Sigil yet) to get a feeling for how the process would work.

Just to clarify:
"input" means Sigil will be reading in epub produced by script,
"output" means script will transform data from Sigil and write it somewhere outside Sigils jurisdiction.
"edit" means script will change Sigil data and Sigil will have to accommodate it.
"meta" is actually "edit" but only for metadata.

Yes. All correct, and you are right we don't need the meta type script as the edit type can do the job.

Hope this explains things a bit more.

KevinH

[KevinH]

Hi varlog,
user_none added a few good ideas
1. each target script gets its own directory
2. all the targets python code is stored there along with a script info.xml file
3. each scriptinfo.xml file could have the following info in it
- Script Type (edit, input, output)
- Name
- Version
- Description
- Programming Language/Interpreter
- Programming Language/Interpreter version needed
- List of dependencies (We won't check the list but it's informational
so a user knows what they need installed).
- Name of target module to invoke file to run from launcher

If we did that, it may make the registration process gui part easier as the user provides a zip for each "script" that is stored in the official scripts home that either Sigil or the launcher could search.

All of this can be implemented later in stages, I would like to focus on getting what we have merged together and working properly and then add the bells and whistles.

When I get free time I will focus on adding a few support routines to Resources, FolderKeeper and MainWindow (I think that is where tab flushing is done), to make flushing to disk, and integrating results easier.


Take care,

Kevin

[varlog]

Originally Posted by KevinH

...
1. each target script gets its own directory
2. all the targets python code is stored there along with a script info.xml file
3. each scriptinfo.xml file could have the following info in it
- Script Type (edit, input, output)
- Name
- Version
- Description
- Programming Language/Interpreter
- Programming Language/Interpreter version needed
- List of dependencies (We won't check the list but it's informational
so a user knows what they need installed).
- Name of target module to invoke file to run from launcher
...

Have you actually tried to run DGSM? All those things are already there - well, not the red ones, the green one I've added now.

Manifest Id error was my mistake. I do test outside of Sigil but grabbed the files from it - and did it too early in this case.

I tried to groak your wrapper but alas: not enough python between my ears. Which brings me to one of my second thoughts. I was able, using guesses and stackoverflow, to write some things in python which did something I set out to do. I'm not able to do it with your wrapper in between. It is probably for the best - one should really learn the language first and than write scripts.
But what remains is:
1. your wrapper needs proper documentation.
2. the call to execute script will depend on interpreter (or on: do we have wrapper for this interpreter?).

Before writing proper documentation could you implement New UUID/Smarten it just for me to have something to lean on?

[KevinH]

Hi varlog,

Originally Posted by varlog

Have you actually tried to run DGSM? All those things are already there - well, not the red ones, the green one I've added now.

I am jut passing along ideas from user_none that I thought might be good and that were not part of what we talked about yet.

[Manifest Id error was my mistake. I do test outside of Sigil but grabbed the files from it - and did it too early in this case.

Glad to hear it

I tried to groak your wrapper but alas: not enough python between my ears. Which brings me to one of my second thoughts. I was able, using guesses and stackoverflow, to write some things in python which did something I set out to do. I'm not able to do it with your wrapper in between. It is probably for the best - one should really learn the language first and than write scripts.

That is exactly how I feel about gui widgets in Qt C++!

But what remains is:
1. your wrapper needs proper documentation.
2. the call to execute script will depend on interpreter (or on: do we have wrapper for this interpreter?).

The entire interface is laid out in bookcontainer.py These are the iterators and functions used by any script to safely access files from within Sigil. The OPF is currently not modifiable directly, but only in pieces. That is easily changed but it is safest as the opf has to maintain consistency across the addition and removal of files, changing spine order, etcc.

Everything else such as checking to see which files are modified, and building the xml result file is handled in the wrapper and the user should be protected from all of that. We of course have yet to implement handling the response XML and taking the modifications and changes listed there and having Sigil apply all of those changes and things on its own.

Before writing proper documentation could you implement New UUID/Smarten
it just for me to have something to lean on?

If I need to change bookcontainer.py to make it easier, I will send you that and the new one.

I should have something for you by this evening.

Kevin

[varlog]

Originally Posted by KevinH

I should have something for you by this evening.

It is deep night for me now. Started on modal dialog but must go to sleep, then to work and than there is this 800 km journey... take your time.

ciao

[KevinH]

Hi varlog,

Okay I didn't want to change new_uuid.py much, but I did have to add a return 0 at the end of the main routine.

I also added a short uuid_runner.py that interfaces between what bookcontainer.py provides and what new_uuid.py wants for input.

#!/usr/bin/env python                                                                                                           
# -*- coding: utf-8 -*-                                                                                                         
# vim:ts=4:sw=4:softtabstop=4:smarttab:expandtab                                                                                

import sys, os
import tempfile, shutil

import new_uuid

def run(bk):
    # get the opf data from the bookcontainer interace                                                                          
    opfdata = bk.get_opf()

    # create a temporary directory store it there as new_uuid wants a path                                                      
    mydir = tempfile.mkdtemp()
    opfpath = os.path.join(mydir, 'content.opf')
    file(opfpath,'wb').write(opfdata)

    # now invoke new_uuid's main routine and get its return value                                                               
    argv = ['new_uuid.py', opfpath]
    rv = new_uuid.main(argv)

    # if it failed raise failed exception                                                                                       
    if rv is not None and rv != 0:
        shutil.rmtree(mydir)
        raise Exception("new_uuid failed")

    # success so grab revised metadata from temp content.opf                                                                    
    # and set it via the interface                                                                                              
    if rv == 0:
        # extract just the new metadata and set it                                                                              
        opfdata = file(opfpath,'rb').read()
        ms = opfdata.find('<metadata')
        me = opfdata.find('</metadata>')
        if ms > -1 and me > -1:
            newmetadata = opfdata[ms:me+11] + '\n'
            bk.setmetadataxml(newmetadata)
        else:
            shutil.rmtree(mydir)
            raise Exception("no metadata found")

    # clean up our own tempspace                                                                                                
    shutil.rmtree(mydir)
    return 0

I have zipped everything up along with a number of other changes and fixes to all of the scripts. So please grab the attached zip archive.

To test it, I unpacked this zip on my Desktop, unpacked a previously saved epub from Sigil into ebook/ and cd'd into launcher and ran the following:

python edit_launcher.py ~/Desktop/ebook ~/Desktop/ebook/OEBPS/content.opf ~/Desktop/testout uuid_runner ~/Desktop/NewUUID


Here is the xml result file it produced along with a new content.pdf in the output directory

<?xml version="1.0" encoding="UTF-8"?>
<wrapper>
<result>success</result>
<deleted>
</deleted>
<added>
</added>
<modified>
<metadataxml>metadataxml</metadataxml>
</modified>
<successmsg>
bookid
urn:uuid:aaddb689-572a-4ef0-819d-630091072ea6
urn:uuid:9067c7b2-0534-4d34-952a-eb214c3a5724
Script Success
</successmsg>
</wrapper>

So it all appears to work on the command line. Sigil would still have to parse this result xml and know to copy the content.opf resource from the Sigil temporary directory and replace the one inside.

Hope this helps you understand the interface a bit better.

KevinH


Ps. I think the smarten one would be easy as well so I'll write a quick runner interface to it and post it here as well.

So here is the smarten_runner.py code I added to the Smarten to interface it with the code. The interesting thing is the bookcontainer.py has access to a built in QuickXHTMLParser, which separates out the Text from the tags on the fly (it is a stream parser) and so creating Smarten using that approach would have been even simpler.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# vim:ts=4:sw=4:softtabstop=4:smarttab:expandtab

import sys, os
import tempfile, shutil

import smarten

def run(bk):

    # iterate through all text files smartening their punctuation
    for id, href in bk.text_iter():
        data = bk.readfile(id)
        sdata = smarten.smarten(data)
        bk.writefile(id, sdata)

    return 0

 
def main():
    return -1
    
if __name__ == "__main__":
    sys.exit(main())

The command line to invoke it from inside of the launcher folder was ...
python edit_launcher.py ~/Desktop/ebook ~/Desktop/ebook/OEBPS/content.opf ~/Desktop/testout smarten_runner ~/Desktop/Smarten

I have zipped up Smarten with the runner inside and have posted it here.

[KevinH]

Hi varlog,

Okay, I spent some time reworking the python script support end for Sigil. I have modified the launcher.py to work for all 3 main types of scripts (input, edit, and output) and have modified the wrapper.py file to allow editing and changing of non-manifests files (i.e. all ebook files). I have also written containers for output and input to use the wrapper.

Included are numerous bug fixes and things.

To test with this, simply unpack the attached zip and cd into launcher. Then unzip an epub previously saved from Sigl into the ebook directory.

The launcher now takes slightly different parameters:
1.) path to ebook root
2.) path to output directory
3.) type of script (edit, input, output)
4.) path to target script to import and invoke run

So for example when I unpack all of these onto my Desktop I would use the following command line"

python ./launcher.py ~/Desktop/ebook/ ~/Desktop/testout/ edit ~/Desktop/target_script/test_target.py

The Smarten and NewUUID versions I posted earlier should all still function with this new version.

The python end of things is getting close to completion. It still needs a lot more testing and a way to pass in a single resource to the launcher which in turn needs to pass that along to the target script in some fashion. Other than that, this version should be functionally complete.

I have attached the latest version.

As for the Sigil side of things, I have spent some time looking at that too. It seems we will need to set the dialog modal but still use show() so that the event loop keeps working underneath it all. We also have to do our own wait loop, that allows gui events to be processed while waiting for the QProcess to complete (and not call waitforfinished()). Let me know when you have something rough working from the Sigil end and I will try building it and can then help out more. Lets focus on just launching the script and passing the parameters in first. We can then work together on parsing the wrapper xml and properly updating the Sigil internals.

Hope this helps,

Take care,

KevinH

[varlog]

I had not much time till Tuesday for this. I hope this weekend I'll have something working. I use exec() and end it with finished() signal (don't use waitforfinished() anymore). Cancel signal (from QProgressDialog) just kills the process at the moment. This seems to work properly.
Now I'm getting to invoking scripts - but had to change/add some general things in ScriptManager/ScriptInterface first. And build some rudimentary ScriptExecWidget for evaluation of script output.

The launcher now takes slightly different parameters:
1.) path to ebook root
2.) path to output directory
3.) type of script (edit, input, output)
4.) path to target script to import and invoke run

and selected resources ?

regards

[KevinH]

Pre copied into temp directory by Sigil and passed in as ebook root

[varlog]

You have to elaborate... but I have to sleep now...
actually I wanted the script to create tmp folder if needed, Sigil would only pass the path... to avoid creating unnecessary folders.