Attn plugin devs: porting to python 3

[JimmXinu]

I'm getting an error when trying to open a file dialog. It happens with both Add Book and Create Library. Both with beta and running from source. Pretty sure it's unrelated to plugins.

Debug log attached.

(Sorry if there's a better place to report this sort of thing.)

[jhowell]

I have a plugin that uses the following import:

from Crypto.Cipher import AES

This loads fine under calibre 4.6.0 portable, but under 4.99.0 portable it produces this error:

Spoiler:

Traceback (most recent call last):
  File "ui.py", line 157, in __init__
  File "ui.py", line 171, in init_iaction
  File "__init__.py", line 612, in load_actual_plugin
  File "__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "zipplugin.py", line 184, in load_module
  File "calibre_plugins.overdrive_link.action", line 7, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "__init__.py", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "_mode_ecb.py", line 35, in <module>
  File "_raw_api.py", line 299, in load_pycryptodome_raw_lib
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Trying '_raw_ecb.cp38-win32.pyd': Could not find module 'C:\calibre-beta\Calibre Portable\Calibre\app\pylib.zip\Crypto\Cipher\_raw_ecb.cp38-win32.pyd'. Try using the full path with constructor syntax., Trying '_raw_ecb.pyd': Could not find module 'C:\calibre-beta\Calibre Portable\Calibre\app\pylib.zip\Crypto\Cipher\_raw_ecb.pyd'. Try using the full path with constructor syntax.

[JimmXinu]

Another issue I've found. (Win10, portable calibre 4.99 w/ py3 branch source.)

In Python3.7, zipfile.py wants the file-like objects it is handed to have attribute seekable.

SpooledTemporaryFile doesn't always have that attribute.

Calibre's db.format(..., as_file=True) hands out a SpooledTemporaryFile (child class). Which I can't read as a zip file, rather defeating the purpose of asking for an epub, for example.

There is an Explanation about why SpooledTemporaryFile isn't seekable.

So the question is, should Calibre return something else for db.format(..., as_file=True), or should plugins not call that when they want a usable zip file?

Method db.cover(..., as_file=True) behaves the same. I haven't looked for other uses of SpooledTemporaryFile in Calibre.

[thiago.eec]

Originally Posted by eschwartz

@thiago.eec

You can get a debug log by using:

Preferences -> Restart in debug mode

It will create a text file containing the debug log, IIRC.

I did the porting using modernize now. Got better results:

1) ACE ported almost flawlessly. It works fine but for one error in the config dialog:

a) OPEN FOLDER. This gives the same error reported by @JimmXinu.

Spoiler:

calibre, version 4.99.0
ERRO: Exceção sem tratamento: <b>TypeError</b>:%b requires a bytes-like object, or an object that implements __bytes__, not 'int'

calibre 4.99 Portable embedded-python: True is64bit: False
Windows-10-10.0.18362-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.18362')
Python 3.8.0
Windows: ('10', '10.0.18362', 'SP0', '')
Interface language: pt_BR
Successfully initialized third party plugins: ACE (1, 1, 0) && Skoob Sync (0, 3, 0)
Traceback (most recent call last):
File "calibre_plugins.ACE.config", line 158, in get_directory
File "win_file_dialogs.py", line 238, in choose_dir
File "win_file_dialogs.py", line 161, in run_file_dialog
File "win_file_dialogs.py", line 59, in serialize_binary
TypeError: %b requires a bytes-like object, or an object that implements __bytes__, not 'int'

2) Skoob Sync had more problems. First I had a problem with encoding a password with base64 (which I ended up giving up). There were a lot of other errors about encode/decode of strings/bytes with urllib requests.
Right now, it is working partially. Some functions work, others make calibre hang. Trying to debug now...

______________________________
EDIT: Looks like the problem is on my multi-threading code. Until now (4.6), sometimes I got an error like this:

QObject::setParent: Cannot set parent, new parent is in a different thread

But it did not prevent the plugin to run. Now, every time this error shows up, calibre hangs. Apparently, python 3 is more error intolerant ....

This is my code for multi-threading:

Spoiler:

    def run_parallel_in_threads(self, target, args_list):
        ''' Utility - spawn a thread to execute target for each args '''

        result = six.moves.queue.Queue()

        # Wrapper to collect return value in a Queue
        def task_wrapper(*args):
            result.put(target(*args))

        threads = [threading.Thread(target=task_wrapper, args=args) for args in args_list]

        count = 0

        for t in threads:
            count += 1
            t.start()
            # Don't send all requests at once
            list_size = len(args_list)
            if (list_size - 20) > 10:
                if count == 20:
                    time.sleep(1)
                elif count == 30:
                    time.sleep(1)
                elif count == 40:
                    time.sleep(2)
            elif (list_size - 20) > 0:
                if count == 20:
                    time.sleep(1)
            time.sleep(0.1)
            QApplication.processEvents(QtCore.QEventLoop.ExcludeUserInputEvents)
        for t in threads:
            t.join()
        return result

Anyway... I will keep digging.

[eschwartz]

@JimmXinu, @thiago.eec

Can you try out my fork here? https://github.com/eli-schwartz/calibre/commits/py3

I think this commit should fix the file dialogs error: https://github.com/eli-schwartz/cali...381ecdb581d274

[eschwartz]

@jhowell,

Nothing in calibre depends on pycrypto, so I'm surprised you have it available in the prebuilt binaries at all.

[JimmXinu]

Originally Posted by eschwartz

Can you try out my fork here? https://github.com/eli-schwartz/calibre/commits/py3

I think this commit should fix the file dialogs error: https://github.com/eli-schwartz/cali...381ecdb581d274

Working for me.

[eschwartz]

PR submitted as https://github.com/kovidgoyal/calibre/pull/1078

[eschwartz]

Originally Posted by thiago.eec

2) Skoob Sync had more problems. First I had a problem with encoding a password with base64 (which I ended up giving up). There were a lot of other errors about encode/decode of strings/bytes with urllib requests.

Indeed, the majority of issues you're likely to have are related to encoding/decoding strings, and tools like modernize/futurize usually cannot help with that. You'll just need to resolve them via (usually) simple trial and error.

[kovidgoyal]

@jhowell: pycrypto is no longer maintained, so it has been replaced by pycryptodome, which is mostly a drop in replacement. https://github.com/Legrandin/pycryptodome

[eschwartz]

Originally Posted by kovidgoyal

@jhowell: pycrypto is no longer maintained, so it has been replaced by pycryptodome, which is mostly a drop in replacement. https://github.com/Legrandin/pycryptodome

Yeah, but the imports are really the same, and it's the import that fails.

https://github.com/kovidgoyal/calibr...rces.json#L694

Why does calibre even bundle this -- I don't have it as a dependency for calibre, and it's always worked just fine.

If I do extensive grepping of the sources, I see old versions of calibre bundled ezPyCrypto.py, but did not AFAICT actually use it.

[kovidgoyal]

Originally Posted by eschwartz

Yeah, but the imports are really the same, and it's the import that fails.

Likely something in the windows freezing process, will tke a look at it whenI have a moment.

Why does calibre even bundle this -- I don't have it as a dependency for calibre, and it's always worked just fine.

If I do extensive grepping of the sources, I see old versions of calibre bundled ezPyCrypto.py, but did not AFAICT actually use it.

Because there was a time when it did bundle it and various plugins started to depend on it. calibre itself no longer uses it.

[jhowell]

Originally Posted by eschwartz

Yeah, but the imports are really the same, and it's the import that fails.

From the error message it appears that there is a missing file causing the failure:
_raw_ecb.cp38-win32.pyd

Originally Posted by eschwartz

Why does calibre even bundle this -- I don't have it as a dependency for calibre, and it's always worked just fine.

Cyptographic functions are useful. I use them in some of my personal plugins and they are also used by a popular plugin that is not distributed via MobileRead.

[kovidgoyal]

Originally Posted by JimmXinu

SpooledTemporaryFile doesn't always have that attribute.

Ugh this is just python devs being lazy: https://bugs.python.org/issue26175

Guess I will have to do their work for them.

[kovidgoyal]

I uploaded 4.99.1 with fixes for all reported issues.