View Single Post
Old 10-03-2020, 01:18 AM   #1
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,418
Karma: 27757236
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Fixing python zipimport issues on Windows

There have been several bugs since 5 was released, of the form of imports failing. Either at calibre startup, or during metadata download or with some plugins. I have traced them all down to the zipimport module in python 3 being buggy when importing from multiple threads. On Windows all python code in calibre is placed in a single zip file for performance reasons.

So to fix it, I ended up writing my own replacement for zipimport, which doesn't use a zip file, but instead a single memory-mapped file containing all the python modules, from which individual modules can be imported using a single hash lookup via a perfect hash function generated at build time. This is pretty much the fastest possible implementation for python importing, faster than zipimport and faster than normal filesystem based importing. And it does not rely on the broken zipimport.

It was quite a journey into Python's internals to get this working, but as of now I cannot replicate any of the reported import bugs on Windows, or any other platform. This is quite a lot of new code, in a deep, dark place, so I would appreciate a bit more testing from you. If you are using CALIBRE_DEVELOP_FROM please try with it unset as well, as when it is set the importing happens from the source code rather than the new frozen importer.

The builds using the new importing framework are available as calibre 5.1.1 at:

https://download.calibre-ebook.com/betas/
kovidgoyal is offline   Reply With Quote