Plugin Development

[DiapDealer]

I'll let Kevin speak for himself (as he already has while I was typing this!), but I think as long as you're clear up front that your plugin is downloading updates from GitHub (and you provide a mechanism for the user to disable the updates), it should be fine.

Maybe default the auto-update feature to disabled, and then explain how those who want the dependency to be automatically kept up to date can enable it for themselves?

[Doitsu]

Originally Posted by KevinH

Are the fixes in fonttools really needed?

Not really. The plugin works fine with the bundled version, which is pretty stable, however, each new release might make the subsetting more efficient.

Originally Posted by KevinH

Is there some way to have your plugin check if there are important updates to fonttools, and if so ask the user to update it manually.

I haven't created the actual code yet. I thought about checking the fonttools github release notes for "subsetting."

Originally Posted by DiapDealer

Maybe default the auto-update feature to disabled, and then explain how those who want the dependency to be automatically kept up to date can enable it for themselves?

Thanks for your helpful comments!

[KevinH]

Perhaps we should use Python's built-in user specific module location and allow pip3 to be used to add external packages/modules in the user specific library location so that Sigil's embedded python package can find those packages.

That way Sigil Embedded Python can access additional 3rd party python modules when installed in a specific location via pip3?

[Doitsu]

Originally Posted by KevinH

Perhaps we should use Python's built-in user specific module location and allow pip3 to be used to add external packages/modules in the user specific library location so that Sigil's embedded python package can find those packages.

That way Sigil Embedded Python can access additional 3rd party python modules when installed in a specific location via pip3?

That's an excellent idea!

[DiapDealer]

I'd have to to do a lot of testing on Windows. Pip used to have a lot of hardcoded paths that made running it from alternative locations rather difficult (without using virtual python environments) on Windows. Not sure if that's still the case. But a totally isolated, embedded python environment that would capable of installing modules to its own predetermined location easily/safely is no small undertaking.

Wouldn't it be easier to have users just switch to an external Python interpreter for plugins that require additional python modules? Those who are already competent with pip would have no difficulty installing what they need to their own Python installation; and those who are technically challenged are never going to be comfortable running pip (the correct pip) from a command line to install modules to a custom location (and run the risk of breaking something).

I'm not opposed to the idea in theory, I just think there's lots of potential for problems/glitches and messed up Python installations.

[Doitsu]

Originally Posted by DiapDealer

I'd have to to do a lot of testing on Windows. [...] I'm not opposed to the idea in theory, I just think there's lots of potential for problems/glitches and messed up Python installations.

In that case, forget about it. It's simply not worth the effort. I thought that all you'd have to do is bundle the pip3 Windows/macOS binaries and maybe change some configuration files.

For the time being, I'll simply add instructions on how to manually update the embedded library.

[DiapDealer]

I don't want to entirely forget about it--it's an idea that's been flown before. And it would solve a lot of problems. But ...

The Mac version of Sigil already includes a fully-functioning, portable version of Python (with pip) included. Windows doesn't. Windows Sigil is creating the correct python environment each time a plugin is run by setting environment variables for the python process that is being launched.

So even if we could make sure that environment is being properly set for pip on Windows (by creating a wrapper for it, or something), there could still be issues with things like upgrades, dependencies and uninstallations.

But if someone ran Sigil's pip (if it was included with Windows) without the proper environment being set before-hand, it would still likely use the system's Python location for determining if dependencies were met, or upgrades were required, before installing a new module (even if the new module were being installed to a different location).

The only way I could, in good conscience, roll something like this out with Windows Sigil is if I could totally ensure that Sigil's pip would only use Sigil's bundled Python to determine if dependencies are installed (or need upgraded), and only if the new modules could be installed in a way that wouldn't affect a user's System Python.

It's a bit of a tall order. Especially to accomplish something that could be achieved as easily as switching to a different external Python interpreter to run certain plugins.

[Doitsu]

Originally Posted by DiapDealer

It's a bit of a tall order. Especially to accomplish something that could be achieved as easily as switching to a different external Python interpreter to run certain plugins.

Of course switching to an different external Python interpreter would be the easiest solution, however, many Sigil plugin users apparently have limited technical skills and some might have difficulties installing Python.
Since my plugins don't require the latest version of all embedded libraries this is more of a "first world problem" anyway and nothing you should waste your time on.

[DiapDealer]

Originally Posted by Doitsu

Of course switching to an different external Python interpreter would be the easiest solution, however, many Sigil plugin users apparently have limited technical skills and some might have difficulties installing Python.

I agree. But I also think those same users with limited technical skills would struggle just as mightily using pip from a command-line to install a new module.

Those who know ... likely already can; and those who don't know, will probably be just as lost with even a Sigil-centric command-line pip anyway

For what it's worth, something like this has always been on my back-burner (because I know Kevin could implement it pretty easily on Macs). But I'm thinking that for it to be useful to the most people, it almost needs to be a function that's kicked off from within Sigil itself (Install Additional Modules to Sigil's Bundled Python). Something that can be used to entirely control how the bundled pip would be launched.

Something that could conceivably be implemented now would be to have a path in a user's Sigil Preferences directory be added to Sigil's bundled Python's sys.path. Knowledgeable users could copy pure python modules there to be available to the bundled interpreter. But that doesn't seem any more advantageous to knowledgeable users to me than simply switching python interpreters.

[KevinH]

FWIW, pip already groks a user specific install location at least on Linux and Mac.

pip3 install --user PACKAGENAME

And should happily install it in: $HOME/.local/lib/pythonx.y/site-packages/

or at least it used to. We could easily include pip3 in the embedded binary and users could use it to install a package locally. I believe that this path is even auto-added to the python path for each user but I would have to check it.

That might be easier than a full external python just to add a package that the embedded python is missing.

[DiapDealer]

Couldn't installing modules to $HOME/.local/lib/pythonx.y/site-packages/ also affect any system installed Pythons? the latest versions of pip allow for a --target parameter that might be better for our needs (or --install-option="--prefix='/blah'"). Not sure.

Another thing to consider is that "pip3 install --user PACKAGENAME" will only install the module to the user's home location. But on Windows, it won't use Sigil's bundled Python environment to determine what dependencies are needed (including any necessary updates). As I said, the Windows bundled Python included with Sigil is not a true standalone version of Python. It is only through the diddling of the process's environment that Sigil's bundled Python is isolated when run on Windows.

There's just more to it on Windows than adding the pip3 binary and utilizing the -u switch.

[KevinH]

I did not know that about Windows. We could of course hard code an extra path into the embedded python search path in Sigil so that it could find a user specific python module in some place normal external python does not use, just to be safe.

It is an idea.

[DiapDealer]

I'll have to do some more experimenting. But including pip3 on Windows seems to be a non-starter right now. I can't even get it to isolate itself from any system installed pip3.

I've renamed the system Scripts directory (where pip lives on Windows), and pip3 still insists on defaulting to the system python's site-directory/pip module. Paths are hardcoded in the binary when it's installed. You can see it by opening pip3.exe in a text editor. I've tried patching the binary, but it only results in errors.

Using pip3 -u or pip3 -t on Windows could result in all dependencies (including one that are already part of Sigil's bundled Python) being installed in the target directory (unless --ignore-dependencies is used). And if there's a compatible system Python already installed, a sigil-supplied pip3 is going to use that to determine which modules are already installed, which dependencies are required, and which dependencies might need upgraded. It's a bit of a mess.

[slowsmile]

I've currently have a problem that's a bit baffling.

Basically, if you have any svg images in your epub then your svg code should look something like this in the code:

<div>
    <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" height="100%" preserveAspectRatio="none" version="1.1" viewBox="0 0 571 910" width="100%">
<image height="910" width="571" xlink:href="../Images/00005.jpeg"/>
    </svg>
</div>

Please note the attribute order within the svg tag. The above svg code will also pass Epubcheck.

However, if you have a bs4 method in your plugin -- any bs4 method -- then using bs4 always changes the svg code in your epub to this:

<div>
   <svg height="100%" preserveaspectratio="none" version="1.1" viewbox="0 0 571 910" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
  <image height="910" width="571" xlink:href="../Images/00005.jpeg"></image>
   </svg>
</div

Note that the attribute order of this svg code is different from the first version and this svg code always causes Epubcheck errors.

I eventually found bs4 to be the culprit causing this problem by writing a test plugin with this code:

Spoiler:

from __future__ import unicode_literals, division, absolute_import, print_function

import os, os.path, sys
from tempfile import mkdtemp                  

try:
    from sigil_bs4 import BeautifulSoup, Comment
except:
    from bs4 import BeautifulSoup, Comment    


def run(bk):  
    print('Python version: ', sys.version, '\n')
    print('Running Test SVG Plugin...Please wait\n') 
    
    WDIR= mkdtemp()
    files = []
    
    # copies all epub xhtml files to the wprk dir
    for (id, href) in bk.text_iter():
        file = bk.href_to_basename(href)
        file = os.path.join(WDIR, file)        
        with open(file, 'wt', encoding='utf-8') as outfp:
            data = bk.readfile(id)
            outfp.write(data)
            files.append(file)

    print('\n>>> Files copied to WORK_DIR...')
    for _, f in enumerate(files):
        print(f)        
            
    # a bs4 routine that just removes <br> tags
    # so we can test bs4's disruptive effects on svg image
    # formatting in xhtml files.  
    for fname in files:       
        output = os.path.join(WDIR, 'remove_br_tags.html')
        outfp = open(output, 'wt', encoding='utf-8')
        html = open(file, 'rt', encoding='utf-8').read()
        
        soup = BeautifulSoup(html, 'html.parser')    
        orig_soup = str(soup)
        
        for tag in soup.find_all(True):
            if tag.name == 'br':
                tag.attrs = {}
                tag.decompose() 
                
        # write the modified work files back to the epub.            
        if str(soup) != orig_soup:
            bk.writefile(id, str(soup))
    print('\n\n>>> Completed successfully...')    
    return(0)        

    
def main():
    print('I reached main when I should not have\n')
    return -1
    
if __name__ == "__main__":
    sys.exit(main())

Just run this test plugin on any epub that contains svg images. Then run Epubcheck and you will see the problem for yourself.

If you run this test plugin on any epub in Sigil that contains svg images and stop it after it writes files to the work directory then any svg code in the xhtml files will be fine and it will pass Epubcheck. But if you run the plugin to include the trivial bs4 module then the svg attribute order will be changed by bs4 and this always causes errors when you run Epubcheck.

Is there anything I can do to prevent bs4 altering the svg code like this?

I'm using 'html.parser' whenever I initialize BeautifulSoup4. Could using the wrong parser be causing this svg problem ?

Does this problem have anything to do with xmlns declarations involving svg ?

[DiapDealer]

It's not the attribute order that's causing your problems. Attribute order doesn't matter. It's the fact that the legitimate camel-case svg attribute names have been changed to invalid all lower-case ones.