MobileRead Forums - View Single Post - Plugins with large, native library dependencies

fake-name · 09-06-2021, 04:04 AM

I'm interested in extending the duplicate finding plugin to do aproximate text/cover matches. I've done both outside of calibre in the past in python successfully.

However, to do so, there are substantial library dependencies:

For text similarity (in a reasonable timeframe), I need a MinHashLSH implementation. I've used datasketch (https://github.com/ekzhu/datasketch) previously, which has a hard dependency on numpy (and I'd really like scipy, as MinHashLSH can use scipy for speeding up the initialization there).

For Image similarity, a DCT based p-hash works very well. I can use a pure python DCT implementation, but scipy provides convenent DCT functions, and I need it anyways for MinHashLSH.

What's the correct way for working on plugins like this? From what I've read there's no way to specify that your plugin has external dependencies (requirements.txt, etc...). Vendoring (and packaging) versions of all the packages for every platform is prohibitive.

09-06-2021, 04:04 AM	#1
fake-name Member Posts: 13 Karma: 10 Join Date: Jun 2008 Device: PRS-505	Plugins with large, native library dependencies I'm interested in extending the duplicate finding plugin to do aproximate text/cover matches. I've done both outside of calibre in the past in python successfully. However, to do so, there are substantial library dependencies: For text similarity (in a reasonable timeframe), I need a MinHashLSH implementation. I've used datasketch (https://github.com/ekzhu/datasketch) previously, which has a hard dependency on numpy (and I'd really like scipy, as MinHashLSH can use scipy for speeding up the initialization there). For Image similarity, a DCT based p-hash works very well. I can use a pure python DCT implementation, but scipy provides convenent DCT functions, and I need it anyways for MinHashLSH. What's the correct way for working on plugins like this? From what I've read there's no way to specify that your plugin has external dependencies (requirements.txt, etc...). Vendoring (and packaging) versions of all the packages for every platform is prohibitive. Last edited by fake-name; 09-06-2021 at 04:07 AM.