How to import numpy?

[xxyzz]

I'd like to use numpy for my plugin(for nltk to find named entity) by append 'site-packages' folder to sys.path but I get this error:

Traceback (most recent call last):
  File "/Users/x/Library/Preferences/calibre/plugins/WordDumb.zip/.venv/lib/python3.9/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "zipimport.py", line 259, in load_module
  File "/Users/x/Library/Preferences/calibre/plugins/WordDumb.zip/.venv/lib/python3.9/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "zipimport.py", line 259, in load_module
  File "/Users/x/Library/Preferences/calibre/plugins/WordDumb.zip/.venv/lib/python3.9/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/gui2/threaded_jobs.py", line 83, in start_work
  File "calibre_plugins.worddumb.parse_job", line 42, in do_job
    for node in ne_chunk(pos_tag(word_tokenize(text.decode('utf-8')))):
  File "/Users/x/Library/Preferences/calibre/plugins/WordDumb.zip/.venv/lib/python3.9/site-packages/nltk/chunk/__init__.py", line 185, in ne_chunk
    chunker = load(chunker_pickle)
  File "/Users/x/Library/Preferences/calibre/plugins/WordDumb.zip/.venv/lib/python3.9/site-packages/nltk/data.py", line 757, in load
    resource_val = pickle.load(opened_resource)
  File "zipimport.py", line 259, in load_module
  File "/Users/x/Library/Preferences/calibre/plugins/WordDumb.zip/.venv/lib/python3.9/site-packages/numpy/__init__.py", line 145, in <module>
    from . import core
  File "zipimport.py", line 259, in load_module
  File "/Users/x/Library/Preferences/calibre/plugins/WordDumb.zip/.venv/lib/python3.9/site-packages/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.8 from "/Applications/calibre.app/Contents/MacOS/calibre-debug"
  * The NumPy version is: "1.20.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

then I look up this file, it's called '_multiarray_umath.cpython-39-darwin.so'. How can I import numpy properly for calibre? I guess even if it's possible I also need to install different versions of numpy for different platform.

[kovidgoyal]

See https://www.mobileread.com/forums/sh...d.php?t=336389

[xxyzz]

I unzipped the site-packages folder but still get the same error:

Traceback (most recent call last):
  File "/Users/x/Library/Preferences/calibre/plugins/worddumb-libs1.1/.venv/lib/python3.9/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/Users/x/Library/Preferences/calibre/plugins/worddumb-libs1.1/.venv/lib/python3.9/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/Users/x/Library/Preferences/calibre/plugins/worddumb-libs1.1/.venv/lib/python3.9/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

[kovidgoyal]

You need to include the .so file, and that means a .so file built for the version of python calibre ships with or otherwise one that does not contain version tags like cpython-39-darwin in its file name (which is dangerous since in theory python can break its ABI between versions).

[xxyzz]

Oh... I have to match the Python version that calibre uses(3.8) and the platform. Thanks!

[xxyzz]

I download the whl file via pip, unzip it then add the folder to sys.path but get this code sign error on mac 11.2.1:

Traceback (most recent call last):
  File "/Users/x/Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/Users/x/Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/Users/x/Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/Users/x/Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/_multiarray_umath.cpython-38-darwin.so, 2): no suitable image found.  Did find:
        /Users/x/Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/_multiarray_umath.cpython-38-darwin.so: code signature in (/Users/x/Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/_multiarray_umath.cpython-38-darwin.so) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.

But if I do the same thing in python(installed from homebrew python@3.8), it works.

[kovidgoyal]

Yeah apple doesnt like you loading .so files from just anywhere, I doubt there is a way to get this to work on macOS

[xxyzz]

I tried to sign all the .so files with my signing identity but get another import error(ad-hoc signing doesn't work):

ImportError: dlopen(Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/_multiarray_umath.cpython-38-darwin.so, 2): no suitable image found.  Did find:
        Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/_multiarray_umath.cpython-38-darwin.so: code signature in (Library/Preferences/calibre/plugins/worddumb-numpy1.20.1/numpy/core/_multiarray_umath.cpython-38-darwin.so) not valid for use in process using Library Validation: mapping process and mapped file (non-platform) have different Team IDs

There is another option, disable library validation. I find it at https://github.com/opencv/opencv/iss...ment-554495051. I'm not sure how it's applied in the building process of calibre but it seems the only way. Damn apple.

[kovidgoyal]

Yes, once can disable library validation at build time, but this is not suitable for general calibre builds. You could possibly workaround it by either:

a) disabling gatekeeper on your mac
b) building calibre from source yourself
c) re-doing the code-signing of the entire calibre bundle using an adhoc signing key and removing the library validation restriction when doing so

And yeah, this whole thing is utterly ridiculous on Apple's part. It's so, umm moronic, that they allow processes to load code in the form of text files, but not pre-compiled code, as if one is more or less capable than the other of doing harm.

[xxyzz]

I will skip mac for now, all three methods are very cumbersome just for a plugin to work. I totally agree with you, apple has lots of restrictions in the name of security but not all of them are necessary.