Attn plugin devs: porting to python 3

kovidgoyal · 12-17-2019, 12:10 AM

calibre will be migrating from Python 2 to Python 3 sometime next year. Migration of the calibre codebase itself is more or less complete, for instance, I personally use the Python 3 version as my daily driver.

calibre plugins will need to be ported to python 3 for that transition. To make that easier I have uploaded beta builds of calibre with python 3.8 that you can use to test your plugins. On windows, you should probably use the portable version so as not to disturb your main calibre install. Betas are available at:

https://download.calibre-ebook.com/betas/

The best way forward is to make your plugins be single source python 2 and 3 compatible. This way, your plugins will continue to work with older as well as future calibre releases. Of course, depending on the details of what your plugins do, this may or may not be possible.

The official guide for making python code 2 and 3 compatible is here: https://docs.python.org/3/howto/pyporting.html (Please use Modernize rather than Futurize to port your plugin code if you are porting it automatically).

calibre itself has done this and there exists the polyglot module in calibre to aid with making code work on both python versions (think of it as a lightweight version of six). six itself is also available, so you can use that if you prefer.

Note that if you install the betas and also use CALIBRE_DEVELOP_FROM, change your calibre source code checkout to the py3 branch, not master, with

Code:

git pull && git checkout py3

If you have questions, I will be happy to help as much as possible.

And a big thank you to @eschwartz for doing a lot of the python 3 porting work.

eschwartz · 12-17-2019, 01:02 AM

Yay!

On the porting front, you'll need to decide how many old versions of calibre you're trying to support. The popular six.py library is pretty available in calibre, since it's a dependency of dateutil and thus calibre has always (indirectly) used it:

calibre 1.48.0 -- 2.15.0: six 1.4.1
calibre 2.16.0 -- 2.85.0: six 1.7.2
calibre 3.0.0 -- current: six 1.10.0

A Kovid said, the polyglot module, internal to calibre, provides some compat layers as well, starting with calibre 3.32 you have:

Code:

from polyglot.builtins import is_py3, reraise, zip, map, filter, iteritems, iterkeys, itervalues
from polyglot.urllib import urlopen, urlencode

And lots more with later versions (primarily 3.41 and on), we can help you figure out what versions have what (if you need it).

Some previous discussion has occurred on the topic:

Python 2 to Python 3, Polyglot & Plugins
Python 3, Polyglot & @six.wraps

eschwartz · 12-17-2019, 01:21 AM

Note that on Linux you may have existing options for running on python3:

If you are using Arch Linux, I'm the distro maintainer there, and for a couple of releases I've provided split packages for "calibre" and "calibre-python3". You can toggle between the two, if you have both installed, using the helper command

Code:

calibre-alternatives

Fedora exclusively provides a python3 build of calibre due to the deprecation of python2 itself.

thiago.eec · 12-17-2019, 06:23 AM

I tried changing one of my plugins.

1) Got an error about translations when trying to add the plugin. Is this working as usual?

Spoiler:

Removing 'load_translations' function let me install the plugin. But... I couldn't add it to the toolbar. So, I went to Preferences > Plugins and tried to configure it. Got a message saying I need to restart calibre after trying to configure it. I restarted calibre and even the computer, but got no luck.

Any hints?

kovidgoyal · 12-17-2019, 07:44 AM

The error in load_translations is indeed a bug. As for its not being added even when you remove the call to load_translations(), run calibre in debug ode and you should get a more detailed error.

thiago.eec · 12-17-2019, 10:05 AM

Actually, the plugin is added fine. No error.
But it won't load. It keeps giving me this message to restart calibre, when I try to configure it via Preferences > Plugins > Customize Plugin.

I've converted my code to be py2/py3 compatible using futurize, as suggested by the link you posted. But, it won't load in calibre 4.6 either. In this case, it gives me errors about non existent modules (future, builtins, etc).

Another plugin was added fine, and I can even configure it via Preferences > Plugins > Customize Plugin. But I can add it to the toolbar (it is just not there for me to select).

To be completely honest, I'm a novice in python and code in general. I guess I'll wait for the pros to convert their plugins and I will check how it was done later.

eschwartz · 12-17-2019, 10:08 AM

Again, can you run calibre using the terminal command line with the command

Code:

calibre-debug -g

and post the error log?

DaltonST · 12-17-2019, 10:43 AM

Py3 32-bit Win10 4.99 (a separate install from Py2 64-bit Win10 4.6, which still works perfectly).

First time failed with: ImportError: cannot import name 'comments_to_html' from 'calibre.library.comments' (C:\Program Files (x86)\Calibre2\\app\pylib.zip\calibre\library\comm ents.pyc)

Code:

 calibre 4.99  embedded-python: True is64bit: False
Windows-10-10.0.18362-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.18362')
Python 3.8.0
Windows: ('10', '10.0.18362', 'SP0', '')
Interface language: None
Successfully initialized third party plugins: Audit Log (1, 0, 13) && Author Book Count (2, 1, 8) && Author Book Count Hierarchy (1, 2, 1) && CalibreSpy (1, 0, 76) && Consolidate All Library Metadata (2, 0, 33) && Count Pages (1, 9, 0) && Drop Search Results (1, 0, 9) && English Noun Frequency (1, 0, 10) && Entities Manager (1, 0, 11) && Extract ISBN (1, 4, 4) && Favourites Menu (1, 0, 4) && Find Duplicates (1, 6, 3) && Job Spy (1, 0, 169) && Library Codes (1, 0, 41) && Library Splitter (1, 0, 1) && Media File Importer (1, 0, 21) && Modify ePub (1, 4, 0) && MultiColumnSearch (1, 0, 77) && Quality Check (1, 9, 11) && QuarantineAndScrub (3, 6, 105) && Reading List (1, 6, 6) && Resize Cover (1, 0, 2) && User Category (1, 5, 3) && View Manager (1, 4, 3) && Zotero Metadata Importer (1, 0, 61)
Registering with default programs...
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 95.6235294117647 x 95.58188153310104
Using calibre Qt style: True
Registered with default programs in 2.0 seconds
Traceback (most recent call last):
  File "runpy.py", line 192, in _run_module_as_main
  File "runpy.py", line 85, in _run_code
  File "site.py", line 89, in <module>
  File "site.py", line 84, in main
  File "site.py", line 55, in run_entry_point
  File "debug.py", line 277, in main
  File "gui_launch.py", line 73, in calibre
  File "main.py", line 557, in main
  File "main.py", line 384, in run_gui
  File "<frozen zipimport>", line 259, in load_module
  File "ui.py", line 42, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "init.py", line 20, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "book_details.py", line 19, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "render.py", line 16, in <module>
ImportError: cannot import name 'comments_to_html' from 'calibre.library.comments' (C:\Program Files (x86)\Calibre2\\app\pylib.zip\calibre\library\comments.pyc)

kovidgoyal · 12-17-2019, 10:49 AM

Works fine for me:

Code:

calibre-debug -c "from calibre.library.comments import comments_to_html; print(comments_to_html.__module__); import sys; print(sys.version_info)"
calibre.library.comments
sys.version_info(major=3, minor=8, micro=0, releaselevel='final', serial=0)

Remember if you are using CALIBRE_DEVELOP_FROM you cannot use the same one for both python 2 and python 3

thiago.eec · 12-17-2019, 11:00 AM

Quote:

Originally Posted by eschwartz

Again, can you run calibre using the terminal command line with the command calibre-debug -g and post the error log?

Ok. So to be more explicit about what I did:

1) I used

Code:

futurize --stage1 -w script.py

and

Code:

futurize --stage 2 -w script.py

on all my .py files. Although this was what I could get out the link pointed, I suspect there is a lot more to do than this.

2) Then, I tried to add the modified plugins ACE and SKOOB SYNC.
P.S.: I've attached the plugins created like this.

For calibre 4.6, here is the log:

Spoiler:

The ACE plugin works fine on 4.6

.
As for the SKOOB SYNC, it gives the error shown on the log above. If I go to Preferences > Plugins, it is there, but when I try to configure it, it keep telling me to restart, even if I've already done it:

Spoiler:

As for the log for calibre 4.99, how can do it with calibre-portable? There is no calibre-debug in calibre-portable folder.

When I try to add plugins manually on 4.99, this happens:

1) SKOOB SYNC: It is successfully added. But I can't add it to the toolbar. And when try to configure, it tells to restart, as also happens with 4.6.

2) ACE: It is successfully added. I can configure it via Preferences > Plugins > Customize Plugins. But I can't add it to the toolbar, and so can't use.

DaltonST · 12-17-2019, 11:22 AM

I use no Calibre environment variables whatsoever.

I executed: "C:\Program Files (x86)\Calibre2\calibre-debug" -g

eschwartz · 12-17-2019, 11:23 AM

WARNING: DO NOT USE 'python-futurize'.

Personally, I don't like futurize because it embeds tons of goop which makes everything depend on the futurize module at runtime. This is problematic for calibre, because your plugin cannot depend on futurize being installed in calibre!

See for details: https://python-future.org/faq.html#w...thon-modernize

tl;dr Use python-modernize, it can help you update your code to use pythonic idioms which work well on both python2 and python3. The only dependency it will add is to the six module, which is okay, because calibre guarantees that six is available.

eschwartz · 12-17-2019, 11:26 AM

@thiago.eec

You can get a debug log by using:

Preferences -> Restart in debug mode

It will create a text file containing the debug log, IIRC.

thiago.eec · 12-17-2019, 11:29 AM

Quote:

Originally Posted by eschwartz

Use python-modernize, it can help you update your code to use pythonic idioms which work well on both python2 and python3. The only dependency it will add is to the six module, which is okay, because calibre guarantees that six is available.

Ok. I'll try that, then.

Quote:

Originally Posted by eschwartz

@thiago.eec

You can get a debug log by using:

Preferences -> Restart in debug mode

It will create a text file containing the debug log, IIRC.

After using your advice above, I'll try again and post the log. Thanks!

kovidgoyal · 12-17-2019, 12:06 PM

@DaltonST: Likely one of those plugins is causing the issue them, run calibre as

calibre --ignore-plugins

12-17-2019, 12:10 AM	#1
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Attn plugin devs: porting to python 3 calibre will be migrating from Python 2 to Python 3 sometime next year. Migration of the calibre codebase itself is more or less complete, for instance, I personally use the Python 3 version as my daily driver. calibre plugins will need to be ported to python 3 for that transition. To make that easier I have uploaded beta builds of calibre with python 3.8 that you can use to test your plugins. On windows, you should probably use the portable version so as not to disturb your main calibre install. Betas are available at: https://download.calibre-ebook.com/betas/ The best way forward is to make your plugins be single source python 2 and 3 compatible. This way, your plugins will continue to work with older as well as future calibre releases. Of course, depending on the details of what your plugins do, this may or may not be possible. The official guide for making python code 2 and 3 compatible is here: https://docs.python.org/3/howto/pyporting.html (Please use Modernize rather than Futurize to port your plugin code if you are porting it automatically). calibre itself has done this and there exists the polyglot module in calibre to aid with making code work on both python versions (think of it as a lightweight version of six). six itself is also available, so you can use that if you prefer. Note that if you install the betas and also use CALIBRE_DEVELOP_FROM, change your calibre source code checkout to the py3 branch, not master, with Code: git pull && git checkout py3 If you have questions, I will be happy to help as much as possible. And a big thank you to @eschwartz for doing a lot of the python 3 porting work. Last edited by kovidgoyal; 12-17-2019 at 11:17 PM.

12-17-2019, 01:02 AM	#2
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Yay! On the porting front, you'll need to decide how many old versions of calibre you're trying to support. The popular six.py library is pretty available in calibre, since it's a dependency of dateutil and thus calibre has always (indirectly) used it: calibre 1.48.0 -- 2.15.0: six 1.4.1 calibre 2.16.0 -- 2.85.0: six 1.7.2 calibre 3.0.0 -- current: six 1.10.0 A Kovid said, the polyglot module, internal to calibre, provides some compat layers as well, starting with calibre 3.32 you have: Code: from polyglot.builtins import is_py3, reraise, zip, map, filter, iteritems, iterkeys, itervalues from polyglot.urllib import urlopen, urlencode And lots more with later versions (primarily 3.41 and on), we can help you figure out what versions have what (if you need it). Some previous discussion has occurred on the topic: Python 2 to Python 3, Polyglot & Plugins Python 3, Polyglot & @six.wraps Last edited by eschwartz; 12-17-2019 at 01:07 AM.

12-17-2019, 01:21 AM	#3
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Note that on Linux you may have existing options for running on python3: If you are using Arch Linux, I'm the distro maintainer there, and for a couple of releases I've provided split packages for "calibre" and "calibre-python3". You can toggle between the two, if you have both installed, using the helper command Code: calibre-alternatives Fedora exclusively provides a python3 build of calibre due to the deprecation of python2 itself.

12-17-2019, 06:23 AM	#4
thiago.eec Guru Posts: 927 Karma: 1177583 Join Date: Dec 2016 Location: Goiânia - Brazil Device: iPad, Kindle Paperwhite	I tried changing one of my plugins. 1) Got an error about translations when trying to add the plugin. Is this working as usual? Spoiler: calibre, version 4.99.0 ERRO: Exceção sem tratamento: <b>AttributeError</b>:'GNUTranslations' object has no attribute 'ungettext' calibre 4.99 Portable embedded-python: True is64bit: False Windows-10-10.0.18362-SP0 Windows ('32bit', 'WindowsPE') 32bit process running on 64bit windows ('Windows', '10', '10.0.18362') Python 3.8.0 Windows: ('10', '10.0.18362', 'SP0', '') Interface language: pt_BR Traceback (most recent call last): File "plugins.py", line 319, in add_plugin File "ui.py", line 472, in add_plugin File "ui.py", line 61, in load_plugin File "zipplugin.py", line 202, in load File "__init__.py", line 127, in import_module File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "zipplugin.py", line 184, in load_module File "calibre_plugins.Skoob_Sync.__init__", line 10, in <module> File "zipplugin.py", line 114, in load_translations AttributeError: 'GNUTranslations' object has no attribute 'ungettext' Removing 'load_translations' function let me install the plugin. But... I couldn't add it to the toolbar. So, I went to Preferences > Plugins and tried to configure it. Got a message saying I need to restart calibre after trying to configure it. I restarted calibre and even the computer, but got no luck. Any hints? Last edited by thiago.eec; 12-17-2019 at 07:04 AM.

12-17-2019, 10:08 AM	#7
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Again, can you run calibre using the terminal command line with the command Code: calibre-debug -g and post the error log?

12-17-2019, 07:44 AM	#5
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The error in load_translations is indeed a bug. As for its not being added even when you remove the call to load_translations(), run calibre in debug ode and you should get a more detailed error.

12-17-2019, 10:05 AM	#6
thiago.eec Guru Posts: 927 Karma: 1177583 Join Date: Dec 2016 Location: Goiânia - Brazil Device: iPad, Kindle Paperwhite	Actually, the plugin is added fine. No error. But it won't load. It keeps giving me this message to restart calibre, when I try to configure it via Preferences > Plugins > Customize Plugin. I've converted my code to be py2/py3 compatible using futurize, as suggested by the link you posted. But, it won't load in calibre 4.6 either. In this case, it gives me errors about non existent modules (future, builtins, etc). Another plugin was added fine, and I can even configure it via Preferences > Plugins > Customize Plugin. But I can add it to the toolbar (it is just not there for me to select). To be completely honest, I'm a novice in python and code in general. I guess I'll wait for the pros to convert their plugins and I will check how it was done later.

12-17-2019, 10:49 AM	#9
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Works fine for me: Code: calibre-debug -c "from calibre.library.comments import comments_to_html; print(comments_to_html.__module__); import sys; print(sys.version_info)" calibre.library.comments sys.version_info(major=3, minor=8, micro=0, releaselevel='final', serial=0) Remember if you are using CALIBRE_DEVELOP_FROM you cannot use the same one for both python 2 and python 3

12-17-2019, 11:22 AM	#11
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	I use no Calibre environment variables whatsoever. I executed: "C:\Program Files (x86)\Calibre2\calibre-debug" -g

12-17-2019, 11:23 AM	#12
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	WARNING: DO NOT USE 'python-futurize'. Personally, I don't like futurize because it embeds tons of goop which makes everything depend on the futurize module at runtime. This is problematic for calibre, because your plugin cannot depend on futurize being installed in calibre! See for details: https://python-future.org/faq.html#w...thon-modernize tl;dr Use python-modernize, it can help you update your code to use pythonic idioms which work well on both python2 and python3. The only dependency it will add is to the six module, which is okay, because calibre guarantees that six is available.

12-17-2019, 11:26 AM	#13
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	@thiago.eec You can get a debug log by using: Preferences -> Restart in debug mode It will create a text file containing the debug log, IIRC.

12-17-2019, 12:06 PM	#15
kovidgoyal creator of calibre Posts: 43,859 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@DaltonST: Likely one of those plugins is causing the issue them, run calibre as calibre --ignore-plugins

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Plugin Devs: Should we include a License or COPYING file?	KevinH	Plugins	3	08-15-2017 01:09 PM
Attn plugin developers: calibre moving to Qt 5	kovidgoyal	Plugins	157	01-01-2015 11:22 PM
Can I develop a plugin in a launguage other than Python?	jamawg	Development	2	06-13-2014 12:38 AM
Calibre in a Python 3.2 world, attn: Kovid	Kevin McAleavey	Calibre	8	01-09-2012 05:49 PM
How do I Create a Python Plugin?	Sydney's Mom	Plugins	25	01-27-2010 06:26 AM

Advert

Advert