strange interaction with .so files and __file__ in plugins

[bugstomper]

I encountered a strange problem while working on a plugin and I'm not sure if it indicates a bug in Calibre or just something to look out for or do differently.

I had two library packages embedded in the plugin zip. One of them uses os.path.dirname(__file__) to get the path of a resource file in the package's directory that it reads at runtime. Debugging logging showed me that in normal operation __file__ has as its prefix the path to the plugin's zip file, and the package has no problem with the I/O routines reading from it that way.

When I added the second library package to the plugin's tree, I found that in the first package __file__ now contained a path in a temporary directory on disk, one which did not exist at runtime, As a result the attempt to load the resource file failed.

Experimenting, I found that for some reason the second package contains a file with extension .so. Even without that library, if I simply create any file with a .so extension in its name, anywhere in the plugin zip directory tree, with any contents even zero length, then Calibre loads the plugin in such a way that __file__ is a temporary directory that does not exist at runtime instead of being a path to the zip file.

I saw this on Calibre 5.8.1 and still see it on 5.9, on macOS Catalina. I haven't tried Windows or Linux. Does Calibre have a heuristic in which it unzips some plugins if it sees it has certain file types such as *.so which maybe can't be used from a zip? Are there some uses of __file__ that a python library might use but could break when the library is used in a plugin?

[kovidgoyal]

Yes native code cant be loaded from zip files so they have to be unzipped. And to load resources from plugin zip files, use the calibre provided APIs https://manual.calibre-ebook.com/cre...lugin-zip-file

[bugstomper]

I can see why Calibre would unzip such a plugin, and it might not be worth having it recognize that a .so file is only a binary resource on a linux platform, .dylib for macOS, dll for Windows, etc., but why would the directory that __file__ points to not exist at runtime if that is the directory that Calibre unzips the plugin to?

The problem that results from this is that these are two third party libraries. Perhaps I would be stuck patching one or the other of them to use the Calibre plugin API, but is there a reason why the use of __file__ should not work if Calibre does unzip to a directory and __file__ does point to that directory? If the .so file does need to be on the disk, how does that work if the directory no longer exists?

[kovidgoyal]

Actually, I take it back, I was mis-remembering, calibre doesnt auto-unzip plugin zip files. The plugin has to choose to do that for itself, if it wants to.

And the third party libraries should really be fixed to use the importlib.resources stdlib module to load its resources. Indeed, I dont see how your third party library could possibly be using __file__ from a zip file successfully.

This discussion would be a lot easier with some actual code.

[bugstomper]

The easiest way to reproduce this before I write some artificial test case is using the plugin where I saw it more in the wild. The plugin FanFicFare currently includes the open source version of the Python library cloudscraper. You would need to get a copy of FanFicFare now, because for reasons about to become obvious it is likely about to remove cloudscraper. If you install FanFicFare in Calibre, it will add a UI element to the menu toolbar. Use its menu "Download from URLs", giving it the URL of any story you wish that is from fanfiction.net. That will probably fail with an error that mentions seeing a version 2 challenge, which means the attempt was blocked by an advanced level of protection from Cloudflare. That error comes from cloudscraper, is now the expected result, and part of why cloudscraper and access to fanfiction.net will likely be removed from FanFicFare shortly. You can see the error in the job's details or by running calibre-debug for it. If it doesn't fail on that attempt, that is ok too, just that lately the error is the more common result.

The current release version of FanFicFare.zip is always in a link in the first post of the thread https://www.mobileread.com/forums/sh...d.php?t=259221

(The next part is something I only tried on my machine which runs macOS Catalina. I have not seen if it reproduces on Windows or Linux)

Unzip FanFicFare.zip, create a foo.so file in its root directory. Contents don't matter, even length 0 using touch, or even in a subdirectories you create, none of that matter, make a zip file with it, and reinstall that in Calibre.

This time when you try to download a story from fanfiction.net it should fail with a different error

No such file or directory: '/var/folders/j_/l6_c445j0v7gy0dw7y35jq240000gn/C/calibre_5.9.0_tmp_dsx7opbm/i0zmj1_1plugin_unzip/cloudscraper/user_agent/browsers.json'

This error comes from cloudscraper/user_agent/__init__.py where it gets the path for its included file by using os.path.dirname(__file__)

I tried patching the code to replace cloudscraper/user_agent/browsers.json with a browsers.py I wrote that has a def browsers() that returns a string and calling json.loads on it instead of json.load on the file. When I do that, trying to get a story from fanfiction.net gets past there to a new error, which is in the included requests library when it tries to read a root certificates file trying to access an https URL

Could not find a suitable TLS CA certificate bundle, invalid path: /var/folders/j_/l6_c445j0v7gy0dw7y35jq240000gn/C/calibre_5.9.0_tmp_727zsud6/w19l8s4xplugin_unzip/certifi/cacert.pem

I added debugging logging to print the value of __file__ in cloudscraper/user_agents/__init__.py where it makes use of os.path.dirname(__file__)

I found that without adding the foo.so it shows FanFicFare.zip as the root of the path, and with foo.so existing it shows that /var/folders/... path on the disk. Debugging logging also showed that at the time of the error the /var/folders/... path specified by __file__ does not exist.

It might be easy to dismiss this as something that cloudscraper/user_agent should not be doing with __file__ but the fact that a more common package like requests fails when accessing https concerns me.

[kovidgoyal]

Ah, that gives me some more information. The plugin is using

with self:
pass

to unzip itself. Relevant code is here: https://github.com/kovidgoyal/calibr...init__.py#L279


As for requests, it is a pretty old and poorly designed library, so it doesnt surprise me that it does stupid things like using __file__ instead of the correct stdlib facilities for loading resources. That said this should indeed not break in calibre and was probably broken by the changes to the plugin loader system to support python 3.

And just FYI, if you want browser user agents, a good way to get them is

from calibre import random_user_agent

and if you want https certificates, the way to get a path to them is

P('mozilla-ca-certs.pem')

No idea if these libraries allow you to pass such things into them, however.

[kovidgoyal]

And, this error is likely caused by fanficfare doing

with self:
    import whatever

use whatever outside the with statement.

Outside the with statement the unzipped directory will not exist. Instead fanficfare should do self.__enter__() and remember to call self.__exit__() for cleanup when appropriate.

[bugstomper]

The two places that FanFicFare uses with self: are in the def load_actual_plugin and def cli_main in its __init__.py. Is it wrong for load_actual_plugin to return its result calculated inside with self: where it will be used outside of the scope of the with self:? If it should use self.__enter__() and self:__exit__() where would they go to make sure it works?

Also, if that is wrong, what is going on with the problem only showing up when a .so file exists?

I looked for all uses of __file__ in files in the plugin and found that the one for certificates is in file certifi/core.py. That is very short and simple. It looks like it first tries to use importlib.resources and falls back to __file__ if it gets an ImportError exception. Is there something wrong with the call to get_path or __enter__ which would interact with whatever happens when there is a .so file in the plugin zip?

[bugstomper]

Oh, I see how certifi/core.py could interact with this problem. If its where() method is called for the first time in a state where get_path returns the temporary directory, and then the temporary directory is deleted, subsequent calls to where() will fail in just this way. It does not have to have anything to do with the code in it that uses __file__ as long as it is called for the first time in a context in which get_path really is supposed to return the temporary directory.

[kovidgoyal]

I suggest you simply monkey patch either certifi or requests to use the certs file bundled with calibre, the path to which is simply P('mozilla-ca-certs.pem')

[bugstomper]

Patching certifi or requests like that makes sense. Sorry if this is a newbie question, but Googling didn't help... What does P('mozilla-ca-certs.pem') mean? It looks like a function named P but I can't find documentation for it and it is hard to search for.

Also, do you have an answer for my question about load_actual_plugin? The entire definition for it in FanFicFare's __init__.py is simply

    def load_actual_plugin(self, gui):
        with self: # so the sys.path was modified while loading the
                   # plug impl.
            return InterfaceActionBase.load_actual_plugin(self,gui)

Is this the root cause of the problem, that it is returning a value that is evaluated within the scope of the with self: that ends up being used outside of the scope? If that is the case, where should the calls to self.__enter__() and self:__exit__() be?

[kovidgoyal]

It's a calibre specific function defined globally. Just use it as is, it will always return the path to the specified file. If you want to see its implementation look in utils/resources.py in the calibre code.

Yes, if you return from inside with, __exit__ is called before the function returns. As for where to call __enter__ call it in load_actual_plugin, and there's no need to call __exit__, since it will be cleaned up automatically on process exit. However, I recommend monkey-patching over this, since this approach affects global state, and therefore all other code running in the same process.

[bugstomper]

Thank you so much for being so responsive to questions from a newbie to the Calibre code! I passed along the information to the actual dev of FanFicFare and they have fixed it in certifi and requests.

[JimmXinu]

I did not really think about plugins containing binaries libraries. If it's reasonably do-able while still being cross platform, I might want to.

Are there any plugins that would be good examples that contain bundled compiled python modules?

The Alf plugin is the only one I could find with binary libs but I don't think it's representative.

[kovidgoyal]

it's do-able, bu you have to bundle versions of the libraries for each platform. I dont think anyone other than alf has ever bothered.