Plugins cross-influencing??

jindroush · 09-26-2020, 01:26 PM

Hi,
because of change to 5.0.1, one of the metadata plugins I use (Databazeknih) stopped working. I made some fixes, it works.

BUT: I use two metadata sources - one for Databazeknih with priority 1, the other one is built-in Google Images with priority 2.

When there is only Google Images enabled, it works.
As soon as I enable Databazeknih, Databazeknih works and Google Images stops working (in log are missing imports, for example: ImportError: cannot import name 'Browser' from 'calibre.utils.browser')

How could these two influence each other? I know the solution (Add some import statements to Databazeknih __init__, but I can't understand why this happens, and I still think this is somewhat rotten behavior and wrong thing to fix).

kovidgoyal · 09-26-2020, 09:04 PM

All plugins are loaded into the same process, so they can in theory do anything to each other. calibre does not treat plugins as potentially hostile code, it assumes plugin developers "do the right thing".

jindroush · 09-27-2020, 04:20 AM

I'm not criticizing this, as I also don't think there is a real need for separation.

I just can't understand how can Google Images work on its own, but when I load some code BEFORE it, it stops working because some imports are suddenly missing.

I attached the plugin. To test it, just configure DatabazeKnih with priority 1, and GoogleImages with priority 2 and try to download some covers.

With "FIXED" GoogleImages don't work, with FIXED2 it works. But I still don't know where the actual problem lies, as I fixed only the symptoms, not the root of the problem.

kovidgoyal · 09-27-2020, 04:24 AM

Are you on windows? If so, then the problem will be because python's zipimport is not threadsafe, and metadata download happens in multiple threads. So unless importing is done in __init__ or at module level things will break. See for example: I just finished dealing with this problem in another part of the code: https://github.com/kovidgoyal/calibr...632fc76534bd53

kovidgoyal · 09-27-2020, 04:30 AM

Apparently its even worse. importing in python 3 is generally not thread safe, on all OSes and even when not using zipimport. Sigh.

https://bugs.python.org/issue38884

jindroush · 09-27-2020, 05:21 AM

Yes, forgot to state - running latest 5.0.1 64bit on Windows 10 x64.

I downloaded latest tarball, looked up the missing import - importing is done deep there on line 311 of global __init__.py, in function browser
from calibre.utils.browser import Browser
ImportError: cannot import name 'Browser' from 'calibre.utils.browser' (C:\Program Files\Calibre2\\app\pylib.zip\calibre\utils\browse r.pyc)
so this fails when there is DatabaseKnih plugin loaded. If it's because loading order in multi-threaded environment, I can't think of different solution than to put all import statements somewhere in global init.

BTW: Is it possible for the debugging purposes to unpack pylib.zip to see if the problem vanishes?

jindroush · 09-27-2020, 05:39 AM

Regarding my last question: I unpacked pylib.zip to bin directory, and deleted pylib.zip
Started to work.
So your theory about zipimport seems to be valid.
I can see two solutions:
1) Drop pylib.zip idea and simply unpack gazillions of small files to bin
2) Somehow move all imports to one central place. This would be helluva work and will be prone to errors.
3) [Some other solution I don't understand/know about because I'm not the python guy]

kovidgoyal · 09-27-2020, 05:48 AM

The files are in a zip file for performance reasons. Python is much faster at importing from a single file on windows.

The solution is pretty simple, if the issue is only with this plugin, simply move the import causing the problem to the top level/__init__

If it turns out more metadata plugins are impacted by this, then I will have to re-architext calibre to run the metadata plugins in their own processes, shouldnt be very hard.

jindroush · 09-27-2020, 06:18 AM

The plugin does this:
from calibre import browser
that function internally (I assume much later) does this:
from calibre.utils.browser import Browser
This creates conflict with Google Images - preventing it from loading Browser.
When I put both of these in plugin's __init__, it works. But how do I know which such functions I should put in _my_ init, not to break other plugins?

kovidgoyal · 09-27-2020, 06:19 AM

There arent that many, most plugins will be importing the same things. So for the moment, just make the change, if it causes any more issues I will revisit running te plugins per process

kovidgoyal · 10-03-2020, 01:19 AM

See https://www.mobileread.com/forums/sh...d.php?t=333686

jindroush · 10-03-2020, 02:56 AM

Thx for info. Will test it later.

In the meantime I found out the possible error cause in this plugin - the importing of browser in that init was superfluous, as there were two similar calls:
br = browser()
br = self.browser
In Worker __init__, there's self.browser = browser.clone_browser()
So simple replacing of first line with self.browser and removing import caused all problems to vanish.
Yes, this plugin would need some code cleanup

09-26-2020, 01:26 PM	#1
jindroush Connoisseur Posts: 94 Karma: 104 Join Date: Nov 2014 Device: Kindle	Plugins cross-influencing?? Hi, because of change to 5.0.1, one of the metadata plugins I use (Databazeknih) stopped working. I made some fixes, it works. BUT: I use two metadata sources - one for Databazeknih with priority 1, the other one is built-in Google Images with priority 2. When there is only Google Images enabled, it works. As soon as I enable Databazeknih, Databazeknih works and Google Images stops working (in log are missing imports, for example: ImportError: cannot import name 'Browser' from 'calibre.utils.browser') How could these two influence each other? I know the solution (Add some import statements to Databazeknih __init__, but I can't understand why this happens, and I still think this is somewhat rotten behavior and wrong thing to fix).

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Plugins Help	grumbleskin	Plugins	7	11-25-2023 02:10 PM
Plugins	BetterRed	Sigil	20	09-24-2019 07:25 AM
What PlugIns Might I want/need?	ShelaghDB	Calibre	4	11-07-2015 07:16 PM
Plugins	junkml	Plugins	32	06-19-2009 06:43 AM
Plugins?	Mitchll	Plugins	0	12-27-2008 02:36 PM

09-26-2020, 09:04 PM	#2
kovidgoyal creator of calibre Posts: 45,300 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	All plugins are loaded into the same process, so they can in theory do anything to each other. calibre does not treat plugins as potentially hostile code, it assumes plugin developers "do the right thing".

09-27-2020, 04:24 AM	#4
kovidgoyal creator of calibre Posts: 45,300 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Are you on windows? If so, then the problem will be because python's zipimport is not threadsafe, and metadata download happens in multiple threads. So unless importing is done in __init__ or at module level things will break. See for example: I just finished dealing with this problem in another part of the code: https://github.com/kovidgoyal/calibr...632fc76534bd53

09-27-2020, 04:30 AM	#5
kovidgoyal creator of calibre Posts: 45,300 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Apparently its even worse. importing in python 3 is generally not thread safe, on all OSes and even when not using zipimport. Sigh. https://bugs.python.org/issue38884

09-27-2020, 05:21 AM	#6
jindroush Connoisseur Posts: 94 Karma: 104 Join Date: Nov 2014 Device: Kindle	Yes, forgot to state - running latest 5.0.1 64bit on Windows 10 x64. I downloaded latest tarball, looked up the missing import - importing is done deep there on line 311 of global __init__.py, in function browser from calibre.utils.browser import Browser ImportError: cannot import name 'Browser' from 'calibre.utils.browser' (C:\Program Files\Calibre2\\app\pylib.zip\calibre\utils\browse r.pyc) so this fails when there is DatabaseKnih plugin loaded. If it's because loading order in multi-threaded environment, I can't think of different solution than to put all import statements somewhere in global init. BTW: Is it possible for the debugging purposes to unpack pylib.zip to see if the problem vanishes?

09-27-2020, 05:39 AM	#7
jindroush Connoisseur Posts: 94 Karma: 104 Join Date: Nov 2014 Device: Kindle	Regarding my last question: I unpacked pylib.zip to bin directory, and deleted pylib.zip Started to work. So your theory about zipimport seems to be valid. I can see two solutions: 1) Drop pylib.zip idea and simply unpack gazillions of small files to bin 2) Somehow move all imports to one central place. This would be helluva work and will be prone to errors. 3) [Some other solution I don't understand/know about because I'm not the python guy]

09-27-2020, 05:48 AM	#8
kovidgoyal creator of calibre Posts: 45,300 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The files are in a zip file for performance reasons. Python is much faster at importing from a single file on windows. The solution is pretty simple, if the issue is only with this plugin, simply move the import causing the problem to the top level/__init__ If it turns out more metadata plugins are impacted by this, then I will have to re-architext calibre to run the metadata plugins in their own processes, shouldnt be very hard.

09-27-2020, 06:18 AM	#9
jindroush Connoisseur Posts: 94 Karma: 104 Join Date: Nov 2014 Device: Kindle	The plugin does this: from calibre import browser that function internally (I assume much later) does this: from calibre.utils.browser import Browser This creates conflict with Google Images - preventing it from loading Browser. When I put both of these in plugin's __init__, it works. But how do I know which such functions I should put in _my_ init, not to break other plugins?

09-27-2020, 06:19 AM	#10
kovidgoyal creator of calibre Posts: 45,300 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	There arent that many, most plugins will be importing the same things. So for the moment, just make the change, if it causes any more issues I will revisit running te plugins per process

10-03-2020, 01:19 AM	#11
kovidgoyal creator of calibre Posts: 45,300 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	See https://www.mobileread.com/forums/sh...d.php?t=333686

10-03-2020, 02:56 AM	#12
jindroush Connoisseur Posts: 94 Karma: 104 Join Date: Nov 2014 Device: Kindle	Thx for info. Will test it later. In the meantime I found out the possible error cause in this plugin - the importing of browser in that init was superfluous, as there were two similar calls: br = browser() br = self.browser In Worker __init__, there's self.browser = browser.clone_browser() So simple replacing of first line with self.browser and removing import caused all problems to vanish. Yes, this plugin would need some code cleanup

Advert

Advert