Attn plugin devs: porting to python 3

[jhowell]

Originally Posted by kovidgoyal

I uploaded 4.99.1 with fixes for all reported issues.

That fixed my import error. Thank you!

[thiago.eec]

Originally Posted by eschwartz

Indeed, the majority of issues you're likely to have are related to encoding/decoding strings, and tools like modernize/futurize usually cannot help with that. You'll just need to resolve them via (usually) simple trial and error.

All good now! No problems with the multi-threading code. Just some other missing encoding/decoding issues.
Both plugins are working 100% on python 3 now. The open dialog and translation are fixed too.

Thanks for the help.

[DaltonST]

Kovid,

Given that Calibre 4.99 will silently fail at startup with no indication as to the cause if a UI plugin that is not compatible with Python 3 is initialized, and given that many users have many plugins, IMHO it might be prudent to add a new attribute to InterfaceActionBase that defaults to False and means that the respective UI plugin is "safe" to use with Python 3 Calibre only when the plugin returns True for that new attribute. This should help to avoid needless user mass confusion and panic in the future.


DaltonST

[kovidgoyal]

I dont see how calibre 4.99 will fail at startup regardless of what UI plugins do, unless those UI plugins mess with sys.path, or other global interpreter state, which is the case with all versions of calibre.

[thiago.eec]

Originally Posted by DaltonST

...Calibre 4.99 will silently fail at startup with no indication as to the cause if a UI plugin that is not compatible with Python 3 is initialized

Calibre starts normally here. In the debug log I can see all the plugins that failed to load. Also, some plugins load, but fail when I click on then.

[DaltonST]

Originally Posted by kovidgoyal

I dont see how calibre 4.99 will fail at startup regardless of what UI plugins do, unless those UI plugins mess with sys.path, or other global interpreter state, which is the case with all versions of calibre.

See your answer to this post:https://www.mobileread.com/forums/sh...82&postcount=9



which you provided here: https://www.mobileread.com/forums/showpost.php?p=3929798&postcount=15

@DaltonST: Likely one of those plugins is causing the issue them, run calibre as

calibre --ignore-plugins

You were correct. calibre --ignore-plugins allowed 4.99 to start up. Otherwise, back to my original post at https://www.mobileread.com/forums/sh...82&postcount=9


Remember that virtually ALL plugins will fail in 4.99 unless they are ported to Python 3 first. Users will have no idea why Calibre silently dies on them. Hence, user mass confusion.



DaltonST

[thiago.eec]

Hi @kovidgoyal

Just to be sure: is the metadata download fully functional on 4.99.1? I was trying to port a plugin for it, but I'm not sure it is working, since even the bundled plugins are failing.
The log points out to some problem with Qt geometry:

Spoiler:

calibre 4.99.1 embedded-python: True is64bit: False
Windows-10-10.0.18362-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.18362')
Python 3.8.0
Windows: ('10', '10.0.18362', 'SP0', '')
Interface language: pt_BR
Failed to initialize plugin: 'C:\\Users\\go_th\\AppData\\Roaming\\calibre\\plug ins\\DeDRM.zip'
Traceback (most recent call last):
File "ui.py", line 703, in initialize_plugins
File "ui.py", line 61, in load_plugin
File "zipplugin.py", line 202, in load
File "__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "zipplugin.py", line 179, in load_module
File "calibre_plugins.dedrm.__init__", line 164
print u"{0} v{1}: Copying needed library files from plugin's zip".format(PLUGIN_NAME, PLUGIN_VERSION)
^
SyntaxError: invalid syntax
Failed to initialize plugin: 'C:\\Users\\go_th\\AppData\\Roaming\\calibre\\plug ins\\Goodreads.zip'
Traceback (most recent call last):
File "ui.py", line 703, in initialize_plugins
File "ui.py", line 61, in load_plugin
File "zipplugin.py", line 202, in load
File "__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "zipplugin.py", line 184, in load_module
File "calibre_plugins.goodreads.__init__", line 11, in <module>
ImportError: cannot import name 'quote' from 'urllib' (C:\Users\go_th\Documents\Calibre Portable\Calibre\\app\pylib.zip\urllib\__init__.py c)
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 25, 0) && Package KFX (from KFX Input) (1, 25, 0) && ACE (1, 1, 0) && Count Pages (1, 9, 0) && Diaps Editing Toolbag (0, 3, 5) && EpubCheck (0, 2, 1) && EpubMerge (2, 7, 0) && EpubSplit (2, 8, 0) && Find Duplicates (1, 6, 3) && Goodreads Sync (1, 12, 1) && KFX metadata reader (from KFX Input) (1, 25, 0) && KFX Input (1, 25, 0) && Set KFX metadata (from KFX Output) (1, 35, 0) && KFX Output (1, 35, 0) && KindleUnpack - The Plugin (0, 81, 5) && LanguageTool (0, 1, 0) && Open With (1, 5, 10) && ScrambleEbook (0, 4, 1) && Skoob Books (1, 3, 1) && Skoob Sync (0, 3, 0) && TTS to MP3 (0, 1, 3)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 120.0 x 120.0
physicalDpi: 141.7674418604651 x 141.4020618556701
Using calibre Qt style: True
[0.00] Starting up...
[0.02] Showing splash screen...
[0.55] splash screen shown
[0.55] Initializing db...
[0.61] db initialized
[0.61] Constructing main UI...
Traceback (most recent call last):
File "ui.py", line 157, in __init__
File "ui.py", line 171, in init_iaction
File "__init__.py", line 612, in load_actual_plugin
File "__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "zipplugin.py", line 184, in load_module
File "calibre_plugins.epubmerge.epubmerge_plugin", line 14, in <module>
ModuleNotFoundError: No module named 'StringIO'
Traceback (most recent call last):
File "ui.py", line 157, in __init__
File "ui.py", line 171, in init_iaction
File "__init__.py", line 612, in load_actual_plugin
File "__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "zipplugin.py", line 184, in load_module
File "calibre_plugins.epubsplit.epubsplit_plugin", line 14, in <module>
ModuleNotFoundError: No module named 'ConfigParser'
Traceback (most recent call last):
File "ui.py", line 157, in __init__
File "ui.py", line 171, in init_iaction
File "__init__.py", line 612, in load_actual_plugin
File "__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "zipplugin.py", line 184, in load_module
File "calibre_plugins.goodreads_sync.action", line 25, in <module>
File "zipplugin.py", line 184, in load_module
File "calibre_plugins.goodreads_sync.core", line 5, in <module>
ImportError: cannot import name 'tostring' from '_elementtree' (C:\Users\go_th\Documents\Calibre Portable\Calibre\app\bin\_elementtree.pyd)
Traceback (most recent call last):
File "ui.py", line 157, in __init__
File "ui.py", line 171, in init_iaction
File "__init__.py", line 612, in load_actual_plugin
File "__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "zipplugin.py", line 179, in load_module
File "calibre_plugins.kindleunpack_plugin.action", line 268
except Exception, e:
^
SyntaxError: invalid syntax
PyQt5.QtWebKitWidgets.QWebView failed
PyQt5.QtWebEngineWidgets.QWebEngineView failed
Traceback (most recent call last):
File "ui.py", line 157, in __init__
File "ui.py", line 171, in init_iaction
File "__init__.py", line 612, in load_actual_plugin
File "__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "zipplugin.py", line 179, in load_module
File "calibre_plugins.tts_to_mp3_plugin.uiaction", line 129
raise SelectedBookError, errmsg
^
SyntaxError: invalid syntax
Traceback (most recent call last):
File "ui.py", line 412, in initialize
File "calibre_plugins.find_duplicates.action", line 54, in initialization_complete
File "calibre_plugins.find_duplicates.duplicates", line 69, in __init__
File "calibre_plugins.find_duplicates.config", line 111, in get_exemption_lists
NameError: name 'xrange' is not defined
[2.03] main UI initialized...
[2.03] Hiding splash screen
[2.53] splash screen hidden
[2.53] Started up in 2.53 seconds with 981 books
Metadata sources cache was recently updated not updating again
QWindowsWindow::setGeometry: Unable to set geometry 604x218+676+354 on QWidgetWindow/'DialogWindow'. Resulting geometry: 604x246+676+354 (frame: 9, 38, 9, 9, custom margin: 0, 0, 0, 0, minimum size: 530x218, maximum size: 16777215x16777215).

Also, I just noticed an error with QWebView that is showing both on calibre 4.6 and 4.99.1:

Spoiler:

PyQt5.QtWebKitWidgets.QWebView failed
PyQt5.QtWebEngineWidgets.QWebEngineView failed

[DaltonST]

Originally Posted by thiago.eec

Calibre starts normally here. In the debug log I can see all the plugins that failed to load. Also, some plugins load, but fail when I click on then.

See my original debug log. Calibre silently died AFTER initialization of all 20 of my installed plugins, with nothing in the debug log. Hence, my first post in this thread.




DaltonST

[JimmXinu]

(Tangentially off topic?)

So while working on this, I'm wondering: How common are pre-3.48 Calibre users now?

I'm thinking I might make the min Calibre version for all my plugins 3.48 and start removing some old cruft and the PyQt4 code...

[DiapDealer]

Looks like I have a little bit of playing to do.

[eschwartz]

Originally Posted by DaltonST

See my original debug log. Calibre silently died AFTER initialization of all 20 of my installed plugins, with nothing in the debug log. Hence, my first post in this thread.

I've been running python3 in various stages for the better part of a year with lots of plugins installed, and no plugin-related issues.

Whatever *your* plugin-related issue is, it somehow caused a function in Calibre itself to go missing so that Calibre itself could not find it and failed at startup.

I'm baffled what your plugins could do that caused such an issue. It's certainly not something that most plugins should be concerned about.

If you can test your profile with a single plugin enabled at a time, and figure out what actually caused the crash to happen (or even a way for someone else to repeat it) that would be great and we could investigate the cause, and determine whether it's a sufficiently painful issue to require skipping plugins.

But at the moment, we are working on vague theories and insinuations, so I think we should go with Kovid's gut instinct and not restrict anything here.

[eschwartz]

Originally Posted by JimmXinu

(Tangentially off topic?)

So while working on this, I'm wondering: How common are pre-3.48 Calibre users now?

I'm thinking I might make the min Calibre version for all my plugins 3.48 and start removing some old cruft and the PyQt4 code...

The pyqt4 code should only be needed on 1.48, which basically means Windows XP users I think.

I'm not sure there's any advantage in requiring anything stricter than >=2.0.0

It's possible there are people still using 2.x because they like the old content server or don't like the fact that running calibre-server in the background prevents starting the GUI.

[JimmXinu]

Another minor issue: When running debug from Preferences -> Restart in Debug, after restart, when presented with the 'Running in debug, log saved to this temp file', if I clicked the 'Copy to Clipboard' button, then okay, there's an error in the log apparently from the 'Copy to Clipboard'. But the text of the dialog is copied to clipboard.

See log:

Spoiler:

calibre Debug log
calibre 4.99.1 Portable embedded-python: True is64bit: False
Windows-10-10.0.18362-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.18362')
Python 3.8.0
Windows: ('10', '10.0.18362', 'SP0', '')
Interface language: None
Successfully initialized third party plugins: 
calibre 4.99.1 Portable embedded-python: True is64bit: False
Windows-10-10.0.18362-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.18362')
Python 3.8.0
Windows: ('10', '10.0.18362', 'SP0', '')
Interface language: None
Successfully initialized third party plugins: 
Turning on automatic hidpi scaling
devicePixelRatio: 2.0
logicalDpi: 96.0 x 96.0
physicalDpi: 141.35652173913041 x 141.4020618556701
Using calibre Qt style: True
[0.00] Starting up...
[0.02] Showing splash screen...
[0.44] splash screen shown
[0.44] Initializing db...
[0.44] db initialized
[0.44] Constructing main UI...
[1.02] main UI initialized...
[1.02] Hiding splash screen
Traceback (most recent call last):
  File "__init__.py", line 997, in flush_clipboard
  File "__init__.py", line 359, in __init__
TypeError: a bytes-like object is required, not 'str'
[7.36] splash screen hidden
[7.36] Started up in 7.36 seconds with 23 books

[DiapDealer]

What version did UI plugins first get introduced to calibre?

[DiapDealer]

Originally Posted by DiapDealer

What version did UI plugins first get introduced to calibre?

Never mind. Found it my own big self.
>= 0.8.60