Fixing python zipimport issues on Windows

kovidgoyal · 10-03-2020, 02:18 AM

There have been several bugs since 5 was released, of the form of imports failing. Either at calibre startup, or during metadata download or with some plugins. I have traced them all down to the zipimport module in python 3 being buggy when importing from multiple threads. On Windows all python code in calibre is placed in a single zip file for performance reasons.

So to fix it, I ended up writing my own replacement for zipimport, which doesn't use a zip file, but instead a single memory-mapped file containing all the python modules, from which individual modules can be imported using a single hash lookup via a perfect hash function generated at build time. This is pretty much the fastest possible implementation for python importing, faster than zipimport and faster than normal filesystem based importing. And it does not rely on the broken zipimport.

It was quite a journey into Python's internals to get this working, but as of now I cannot replicate any of the reported import bugs on Windows, or any other platform. This is quite a lot of new code, in a deep, dark place, so I would appreciate a bit more testing from you. If you are using CALIBRE_DEVELOP_FROM please try with it unset as well, as when it is set the importing happens from the source code rather than the new frozen importer.

The builds using the new importing framework are available as calibre 5.1.1 at:

https://download.calibre-ebook.com/betas/

chaley · 10-03-2020, 04:08 AM

It works on Windows 10/64. Things I did, chosen to try to stress importing:

Template search with & without using stored templates
Template search using Python template functions
Add & modify a template function and a stored template
Run the Modify Epub plugin
Run several tests using the template tester

And just for completeness:

Edit metadata single w/download metadata
"Standard" searches
Connect using the wireless device.
Switch between cover & list view.
Bunch of stuff in the tag browser
Switch between virtual libraries
Define a column icon rule

I am running in debug mode but not from source. There are no untoward messages in the debug log.

Are there other tests I can do that will stress it?

davidfor · 10-03-2020, 04:16 AM

As a start, have tried the portable version. The Kobo/Kepub plugins are all loading with this version. Though I haven't loaded anything else yet.

kovidgoyal · 10-03-2020, 04:39 AM

Quote:

Originally Posted by chaley

Are there other tests I can do that will stress it?

Leave it running for a while, especially if you use the content server. I am a bit leery about the mmaped I/O stability.

davidfor · 10-03-2020, 05:07 AM

Using a copy of my configuration directory, three plugins didn't load. They were Databazeknih, Overdrive Link and Embed Comic Metadata. When I tried to install Overdrive Link, I got this error:

Code:

calibre, version 5.1.1
ERROR: Install plugin failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart calibre.

Traceback (most recent call last):
  File "calibre\gui2\dialogs\plugin_updater.py", line 731, in _install_clicked
  File "calibre\customize\ui.py", line 472, in add_plugin
  File "calibre\customize\ui.py", line 61, in load_plugin
  File "calibre\customize\zipplugin.py", line 279, in load
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "calibre\customize\zipplugin.py", line 171, in exec_module
  File "calibre\customize\zipplugin.py", line 167, in get_code
  File "calibre_plugins.overdrive_link.__init__", line 1
    #!/usr/bin/env python
    ^
SyntaxError: invalid character in identifier

When I tried using calibre-debug, the console show a square in front of the "#" in the output. This was there for all three failing plugins. I pasted the above into the editor, and it tells me that the character is a zero-width no-break space. All three have that character at the start of the file that causes the error. That should be fixed by the plugin authors, but, it is a difference in how they are handled. But, is it actually a sign of a bug in the zipimport module?

kovidgoyal · 10-03-2020, 05:45 AM

That's the UTF-8 BOM. I should have the plugin loader strip that. https://github.com/kovidgoyal/calibr...51d0aa2c8622b5

Quote:

Originally Posted by davidfor

Using a copy of my configuration directory, three plugins didn't load. They were Databazeknih, Overdrive Link and Embed Comic Metadata. When I tried to install Overdrive Link, I got this error:

Code:

calibre, version 5.1.1
ERROR: Install plugin failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart calibre.

Traceback (most recent call last):
  File "calibre\gui2\dialogs\plugin_updater.py", line 731, in _install_clicked
  File "calibre\customize\ui.py", line 472, in add_plugin
  File "calibre\customize\ui.py", line 61, in load_plugin
  File "calibre\customize\zipplugin.py", line 279, in load
  File "importlib\__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "calibre\customize\zipplugin.py", line 171, in exec_module
  File "calibre\customize\zipplugin.py", line 167, in get_code
  File "calibre_plugins.overdrive_link.__init__", line 1
    #!/usr/bin/env python
    ^
SyntaxError: invalid character in identifier

When I tried using calibre-debug, the console show a square in front of the "#" in the output. This was there for all three failing plugins. I pasted the above into the editor, and it tells me that the character is a zero-width no-break space. All three have that character at the start of the file that causes the error. That should be fixed by the plugin authors, but, it is a difference in how they are handled. But, is it actually a sign of a bug in the zipimport module?

kovidgoyal · 10-03-2020, 05:48 AM

In fact, this is better: https://github.com/kovidgoyal/calibr...e7fe6ea3bf7d42

kovidgoyal · 10-03-2020, 07:34 AM

Released 5.1.2 with fixes for the issue with UTF-8 BOM and some plugins reported by davidfor and another issue I discovered where using CALIBRE_DEVELOP_FROM was not working with the content server

jackie_w · 10-03-2020, 04:07 PM

Quote:

Originally Posted by kovidgoyal

Released 5.1.2 with fixes for the issue with UTF-8 BOM and some plugins reported by davidfor and another issue I discovered where using CALIBRE_DEVELOP_FROM was not working with the content server

I'm a little confused.

When you say "Released 5.1.2", do you mean "Released a beta 5.1.2" or "Released a full 5.1.2"?

If it's meant to be the latter then my currently installed calibre v5.0.1 64-bit is only offering "Update found:5.1.0" in the statusbar.

Terisa de morgan · 10-03-2020, 04:19 PM

Quote:

Originally Posted by jackie_w

I'm a little confused.

When you say "Released 5.1.2", do you mean "Released a beta 5.1.2" or "Released a full 5.1.2"?

If it's meant to be the latter then my currently installed calibre v5.0.1 64-bit is only offering "Update found:5.1.0" in the statusbar.

Deltas never appear in the statusbar.

Uncle Robin · 10-03-2020, 04:35 PM

Quote:

Originally Posted by Terisa de morgan

Deltas never appear in the statusbar.

The msi on the download page is also still showing 5.1.0, not 5.1.2

DNSB · 10-03-2020, 05:53 PM

Quote:

Originally Posted by Uncle Robin

The msi on the download page is also still showing 5.1.0, not 5.1.2

You need to check the beta download page at https://download.calibre-ebook.com/betas/ for the beta releases.

kovidgoyal · 10-03-2020, 11:24 PM

Apologies fo rthe confusion, by released I meant released a beta: https://download.calibre-ebook.com/betas/

I want some people running it for a few days before I make a general release as the changes are quite extensive internally.

10-03-2020, 02:18 AM	#1
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Fixing python zipimport issues on Windows There have been several bugs since 5 was released, of the form of imports failing. Either at calibre startup, or during metadata download or with some plugins. I have traced them all down to the zipimport module in python 3 being buggy when importing from multiple threads. On Windows all python code in calibre is placed in a single zip file for performance reasons. So to fix it, I ended up writing my own replacement for zipimport, which doesn't use a zip file, but instead a single memory-mapped file containing all the python modules, from which individual modules can be imported using a single hash lookup via a perfect hash function generated at build time. This is pretty much the fastest possible implementation for python importing, faster than zipimport and faster than normal filesystem based importing. And it does not rely on the broken zipimport. It was quite a journey into Python's internals to get this working, but as of now I cannot replicate any of the reported import bugs on Windows, or any other platform. This is quite a lot of new code, in a deep, dark place, so I would appreciate a bit more testing from you. If you are using CALIBRE_DEVELOP_FROM please try with it unset as well, as when it is set the importing happens from the source code rather than the new frozen importer. The builds using the new importing framework are available as calibre 5.1.1 at: https://download.calibre-ebook.com/betas/

10-03-2020, 04:08 AM	#2
chaley Grand Sorcerer Posts: 12,525 Karma: 8065948 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	It works on Windows 10/64. Things I did, chosen to try to stress importing: Template search with & without using stored templates Template search using Python template functions Add & modify a template function and a stored template Run the Modify Epub plugin Run several tests using the template tester And just for completeness: Edit metadata single w/download metadata "Standard" searches Connect using the wireless device. Switch between cover & list view. Bunch of stuff in the tag browser Switch between virtual libraries Define a column icon rule I am running in debug mode but not from source. There are no untoward messages in the debug log. Are there other tests I can do that will stress it?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
KTerm & Python Issues	aBP	Kindle Developer's Corner	4	02-10-2016 06:23 PM
how to use python script with windows xp	tuufbiz1	Other formats	12	01-08-2011 09:22 AM
Python 261 In Windows Problems	jwsheets	Kindle Formats	3	08-14-2009 10:08 AM
using python with windows xp	tuufbiz1	Kindle Formats	10	05-06-2009 12:53 AM
GUI Frontend for python scripts [windows] - seeking mod input first	sherman	Workshop	7	04-27-2009 02:04 AM

10-03-2020, 04:16 AM	#3
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	As a start, have tried the portable version. The Kobo/Kepub plugins are all loading with this version. Though I haven't loaded anything else yet.

10-03-2020, 05:48 AM	#7
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	In fact, this is better: https://github.com/kovidgoyal/calibr...e7fe6ea3bf7d42

10-03-2020, 07:34 AM	#8
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Released 5.1.2 with fixes for the issue with UTF-8 BOM and some plugins reported by davidfor and another issue I discovered where using CALIBRE_DEVELOP_FROM was not working with the content server

10-03-2020, 11:24 PM	#13
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Apologies fo rthe confusion, by released I meant released a beta: https://download.calibre-ebook.com/betas/ I want some people running it for a few days before I make a general release as the changes are quite extensive internally.