Python Libraries included in Plugin?

[JimmXinu]

Hi,

cryzed has started helping me with the FFDL plugin and would really like to use current versions of BeautifulSoup4 and html5lib.

What is the appropriate way to include libraries in a plugin?

While including the packages in the plugin zip and manipulating sys.path directly to include the dir inside the plugin zip appears to work, it seems to me that is likely to conflict with calibre and/or other plugins at some point.

I know calibre already has versions of html5lib and BeautifulSoup, but neither is current. FFDL has always included its own BeautifulSoup, but it wasn't a packaged library then.

Any help you can offer would be appreciated.

Thanks.

[cryzed]

The problem is that the BeautifulSoup 4 module is now essentially a Python package, i.e. a folder with an "__init__.py" file in it. The files within make certain assumptions about the environment, in this case specifically that "bs4" is directly importable and that the Python path points to the package in some way.

For example an import like this would fail:

import packages.bs4

While I could try and modify the code to use relative imports, I doubt that this is something that I will only encounter once and it strikes me as a bit unclean. So I decided to modify the Python path by inserting values into "sys.path" at runtime, which also allows for nicer code in the project's "adapter" modules which take care of the site-specific scraping.

I read up a bit and saw that Calibre's Plugin class implements "__enter__" and "__exit__" magic methods, where Python path modifications are made and removed respectively. If the module is "zipsafe" the path is simply modified to point directly to the *.zip plugin file, if not the plugin is unpacked into a temporary directory and the path pointing to it added to the Python path. Using the "sys_insertion_path" is unfortunately not an option, since the module that contains the Plugin subclass imports modules that might already expect the Python path to be modified by that point and raise ImportErrors. So I make this modification myself in the InterfaceActionBase subclass:

def initialize(self):
    packages_path = os.path.join(self.plugin_path, 'packages')
    sys.path.insert(0, packages_path)

This allows the Plugin to work when used via Calibre's Command-line interface, and via the GUI since it's the main entry point of the Calibre plugin. The concern now is, since I am modifying the path directly and using packages that Calibre itself uses in its own code, if Calibre might accidentally import unexpected versions of the packages. And I'd really prefer to use the "sys_insertion_path" attribute, since it already takes care of an important bit of logic, but I'm not sure how that should be possible.

Basically I am looking for advice on how to best include third-party Python modules and packages and making them directly importable in subsequently called code, without having to use horrible Python path hacks and possibly changing Calibre's internal state unexpectedly.

[kovidgoyal]

I suggest you stick to using the html5lib that ships with calibre. It has many improvements over the vanilla html5lib. See https://github.com/html5lib/html5lib-python/issues/119

As for the general issue of using python packages in plugins, you have two approaches:

1) manipulate sys.path, simply unmanipulate it after you are done importing to ensure that it does not cause problems with the rest of calibre.

2) Modify the python files in the package you are bundling to import its submodules from the calibre_plugins namespace

-----

1) is quick and dirty and will cause problems if the modules you are importing shadow modules that are part of calibre

2) Is much more robust, but requires a little work.

[cryzed]

Using 2) is unfortunately not an option since the plugin needs to be able to run without Calibre and 1) is less than ideal.

I suppose 2) could be easier achieved with branches in the repository and constant merging back and forth...

I think I'll go with option 3) and modify the modules to use relative imports, and import from within the packages like this:

from packages import bs4

Thank you for the answer though Kovid!

[kovidgoyal]

I'm not sure why being able to run outside calibre makes it a problem:

try:
    from calibre_plugins.myplugin.whatever import x
except ImportError:
    from whatever import x

[cryzed]

I suppose. It would mostly be a problem because it adds tons of overhead to existing adapters and the stand-alone versions of the script (command-line and Google App Engine), simply don't have anything to do with Calibre. Using Calibre specific code in every adapter file (100+ since recently) doesn't feel very good.

EDIT: Nevermind, I misunderstood you. You meant putting that try-except block into the packages submodule and then re-importing the names from within. That's a good idea.