ScrambleBook and Calibre 4

jackie_w · 09-09-2019, 04:39 PM

I'm trying to get my ScrambleEbook user interface plugin working in the beta but I'm having a problem importing QWebEngineView.

This is the import code at the top of the main .py file:

Code:

try:
    from PyQt5.QtWebKitWidgets import QWebView as Webview
    print('PyQt5.QtWebKitWidgets.QWebView OK')
except:
    print('PyQt5.QtWebKitWidgets.QWebView failed')
    try:
        from PyQt5.QtWebEngineWidgets import QWebEngineView as Webview 
        print('PyQt5.QtWebEngineWidgets.QWebEngineView OK')
    except:
        print('PyQt5.QtWebEngineWidgets.QWebEngineView failed')

Under 3.47.1 the first try is successful and everything seems to be working OK.

In beta 3.99.1 I'd expect the first try to fail and the second to succeed. However, both of them fail. You can see the error message prints (in red) during calibre startup in debug mode. Despite the error messages, the plugin does get installed.

Spoiler:

Code:

D:\sysfiles\Port_Progs\npp>echo off 
D:\sysfiles\calibre\calcache
calibre 3.99.1  embedded-python: True is64bit: False
Windows-10-10.0.18362-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: DeDRM (6, 6, 2) && Search The Internet (1, 7, 6) && EpubModBuild (0, 1, 10) && Goodreads (1, 4, 0) && EpubCheck (0, 1, 9) && Copy Cover To Device (2, 0, 0) && TTS to MP3 (0, 1, 5) && KePub Metadata Reader (3, 0, 2) && KePub Metadata Writer (3, 0, 3) && JS_Modify_Epub (0, 0, 8) && FlightCrew (0, 0, 3) && Barnes & Noble (1, 2, 15) && Kobo Utilities (2, 8, 0) && KoboTouchExtended (3, 1, 5) && Device Old Books (0, 0, 1) && ScrambleEbook (0, 0, 9) && EpubModBuildJobs (0, 0, 10) && Epub2Kepub (0, 0, 34) && Quality Check (1, 9, 11) && Fantastic Fiction (1, 3, 0) && DOC Input (1, 0, 1) && Kobo Books (1, 7, 0) && KePub Input (3, 0, 0) && Extract ISBN (2, 0, 0) && KindleUnpack - The Plugin (0, 81, 5) && JS_Editor_Tools (0, 2, 40) && Count Pages (1, 8, 2) && Kindle hi-res covers (0, 4, 0) && Obok DeDRM (6, 5, 4)
Turning on automatic hidpi scaling
Registering with default programs...
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 127.664921466 x 127.590697674
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Initializing db...
Registered with default programs in 1.0 seconds
[0.61] db initialized
[0.61] Constructing main UI...
calibre css_parser 1.0.4
lxml etree: 4.3.3
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 KoboUtilites::dialogs.py - loading translations
DEBUG:    0.0 KoboUtilites::action.py - loading translations
PyQt5.QtWebKitWidgets.QWebView failed
PyQt5.QtWebEngineWidgets.QWebEngineView failed
DEBUG:    0.0 obok::utilities.py - loading translations
DEBUG:    0.0 obok::dialogs.py - loading translations
DEBUG:    0.0 obok::config.py - loading translations
DEBUG:    0.0 obok::action_err.py - loading translations
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
DEBUG:    1.9 No Kobo Touch, Glo or Mini appears to be connected
DEBUG:    1.9 rebuild_menus - self.supports_ratings=None, self.supports_tiles=None
DEBUG:    1.9 KoboUtilities:set_toolbar_button_tooltip - start: text='None'
DEBUG:    1.9 KoboUtilities:set_toolbar_button_tooltip - setting to text='Utilities to use with Kobo ereaders

Driver: KoboTouchExtended'
[3.70] main UI initialized...
[3.70] Started up in 3.70 seconds with 181 books
SMART_DEV (   0.00:  0.002) _attach_to_port try ip address "" on port 9090
SMART_DEV (   0.96:  0.830) _startup_on_demand listening on port 9090
SMART_DEV (   0.99:  0.000) _attach_to_port try ip address "" on port 54982
SMART_DEV (   1.03:  0.000) _startup_on_demand broadcast socket listening on port 54982
SMART_DEV (   2.12:  1.055) run All IP addresses {u'{B5C0D296-CA26-11E7-AD69-806E6F6E6963}': [{'broadcast': u'127.255.255.255', 'addr': u'127.0.0.1', 'netmask': u'255.0.0.0'}], u'{B707974E-9488-42B1-BAAD-254627B9DE69}': [{'broadcast': u'192.168.0.255', 'addr': u'192.168.0.7', 'netmask': u'255.255.255.0'}]}

The more obvious error message happens when running the plugin and pressing the 'Preview content' button - which is the only place the QWebView/QWebEngineView widget is used.

Spoiler:

I'm confused because in one of my personal Editor Tool plugins I've used the same import approach and both 3.47.1 and 3.99.1 seem to be working fine.

I've also attached a copy of the plugin in it's current state if you have time to look at it. If not I'll ask again when the initial beta bug reporting dies down a bit.

kovidgoyal · 09-09-2019, 10:17 PM

@jackie_w: That is because webengine is not useable in the main calibre program only the editor/viewer. This is because it is very resource intensive and Qt requires that it be imported before starting the appication. Since the main GUI does not use it, I do not import it, which also means it cant be imported in any plugins. What does scrambleebook use it for? There are two possible workarounds, it is is not something that requires sophisticated HTML support then QTextBrowser can be used, otherwise you have to run it in a separate process.

@eschwartz: Given that all plugins will need to be ported to python3, it will be a long time before calibre is ready for py3. But yes, my next painful job after v4 is done is to work on getting betas ready with py3 so the plugin developers can start work on porting their plugins.

jackie_w · 09-09-2019, 11:28 PM

Quote:

Originally Posted by kovidgoyal

@jackie_w: That is because webengine is not useable in the main calibre program only the editor/viewer. This is because it is very resource intensive and Qt requires that it be imported before starting the appication. Since the main GUI does not use it, I do not import it, which also means it cant be imported in any plugins. What does scrambleebook use it for? There are two possible workarounds, it is is not something that requires sophisticated HTML support then QTextBrowser can be used, otherwise you have to run it in a separate process.

I don't know what counts as 'sophisticated'. Doesn't sound much like me.

It's a utility to preview a page from an original container object alongside the same page from the scrambled container object, so that a user can confirm nothing is readily identifiable before posting to MR. See the 2nd screenshot at the bottom of this post.

Cols 1 & 3 (beige), were originally QWebView widgets displaying pre- & post-scramble versions of the container name user-highlighted in col 2 (white). The webview widgets are styled using the container CSS file(s) referenced in the page <head> <link>s.

I think these would be the relevant bits of code:

Code:

from calibre.ebooks.oeb.display.webview import load_html

    def webview_refresh(self, name):
        name_orig = self.revfmap.get(name, name)
        abspath_orig = self.orig.name_to_abspath(name_orig)
        abspath = self.ebook.name_to_abspath(name)
        
        load_html(abspath, self.webview_scram)
        load_html(abspath_orig, self.webview_orig)

I suspect my ability to give them a beige background (so that it's more obvious where images start/end) will also be lost as I was using the webview.settings().setUserStyleSheetUrl() technique you used in the old calibre Viewer prefs. Even if I had access to QWebEngineView it doesn't seem to have a setUserStyleSheetUrl option.

kovidgoyal · 09-10-2019, 02:08 AM

@jackie_w f all you are doing is previewing some text so users can check it, probably QTextBrower will be good enough for you, that is the simplest option.

jackie_w · 09-10-2019, 08:57 AM

Quote:

Originally Posted by kovidgoyal

@jackie_w f all you are doing is previewing some text so users can check it, probably QTextBrower will be good enough for you, that is the simplest option.

If I can make QTextBrower work it will be the solution I ought to have used in the first place.

In the past I've only ever used those for displaying simple text, like large log results, which are far too big for a calibre info_dialog.

jackie_w · 09-11-2019, 10:20 AM

Quote:

Originally Posted by kovidgoyal

@jackie_w: So the situation with fullscreen images was interesting.

I'm glad you had fun.

Now back to not-so-much fun ... ScrambleEbook and QWebView vs QWebEngineView vs QTextBrowser

TL;DNR: Unless I'm missing something, for previewing the contents of an ebook, QTextBrowser is a poor substitute for QWebView/QWebEngineView due to its limited style support.

I realise this minor plugin is not going to be the tail that wags the calibre dog so I'm open to ideas if there's anything I can do to get the beta plugin QTextBrowser closer to how it looks in the QWebView of 3.47.1. Bear in mind I have limited Python skills (and none at all for Java, C++).

If what I've already done in the beta plugin is as good as it's going to get then I need to decide whether I'm going to discontinue the calibre plugin leaving only its standalone equivalent. It's fair to say there should be no detriment to the actual scrambled ebook output by the plugin, current vs. beta. However, whilst the look&feel of text in a widget may be cosmetic to some extent, I don't feel right releasing an update which is visually a fairly clunky downgrade.

----------------------------------------------------------------
More detail ...

I've attached some screenshots from ScrambleEbook current version vs. beta version. It has a calibre plugin and a standalone (kind of) version which uses a .bat file to run the .py file via calibre-debug.

Image 1: from current calibre plugin. Current standalone looks the same because both use QWebView.
Image 2: beta standalone using QWebEngineView
Image 3: beta calibre plugin using QTextBrowser

Standalone. See image 1 vs. image 2
I found that my beta standalone can access QWebEngineView OK. The only minor difference between current and beta is I can't reproduce the beige background of the widget. I don't know why its .setStyleSheet() method doesn't produce the result I wanted, but I'll live with it.
Calibre plugin. See image 1 vs. image 3
It was easy enough to change the QWebEngineView widget to QTextBrowser and to load a container HTML page into it using .setSource(QUrl.fromLocalFile(abs_path)).

I'm not sure whether the widget accesses the container CSS files referenced in the HTML <head> links or not. It may do because using the .setHtml() method to load content shows no styles/images at all. Whatever it is doing, the lack of style support is a real backward step compared to the webview widgets:
- defaults to sans-serif body text and no paragraph indents
- no control of image size
- can't see images in <svg> tags at all, which often means the first thing you see, when requesting a Preview, is 2 blank widgets where the cover pages should be.
- embedded fonts not displayed.

kovidgoyal · 09-11-2019, 10:35 AM

@jackie_w: You should be able to use qwebengine in your plugin by running it in a separate process using calibre-debug so your previewer becomes a simple standalone program inside your plugin. Something like

Code:

from calibre.debug import run_calibre-debug
run_calibre_debug(['-c', 'from calibre_plugins.your_plugin.viewer import main; import sys; main(sys.argv[-2], sys.argv[-1]);', 'path to first html file', 'path to second html file'])

And then implement your viewer using qwebengine.

jackie_w · 09-11-2019, 11:06 AM

Quote:

Originally Posted by kovidgoyal

@jackie_w: You should be able to use qwebengine in your plugin by running it in a separate process using calibre-debug so your previewer becomes a simple standalone program inside your plugin. Something like

Code:

from calibre.debug import run_calibre-debug
run_calibre_debug(['-c', 'from calibre_plugins.your_plugin.viewer import main; import sys; main(sys.argv[-2], sys.argv[-1]);', 'path to first html file', 'path to second html file'])

And then implement your viewer using qwebengine.

This is hopeful but I think I need a bit more meat on the bone. Do you have a working example of this technique in action so that I can follow exactly what needs to go where?

kovidgoyal · 09-11-2019, 11:46 AM

No cant say I have an example, but here is a barebones viewer.py to add to your plugin. It only shows a single passed in html file, for breviry

Code:

def main(path_to_html_file, path_to_ther_html_file):
     from calibre.gui2 import Application
    app = Application([])
    w = QWebEngineView()
    w.setUrl(QUril.fromlocalFile(path_to_html_file))
    w.show()
    w.raise_()
    app.exec_()

add it to your plugin as viewer.py and call it using the run_calibre_debug() function I showed you earlier, and there you have a web engine based HTML viewer.

jackie_w · 09-11-2019, 12:01 PM

Thanks for this. I'm still having problems 'joining the dots' in how this viewer links to what's happening in the central listwidget column where when the user moves the cursor up and down the list of html names the contents of the webviews on either side change accordingly. Rather than exasperate you further let me think about it some more and try to sort it out - or maybe come up with some better questions.

kovidgoyal · 09-11-2019, 12:09 PM

basically rather than having the column view you have now, you instead have a preview button. the user selects the file he wants to preview the changes to and clicks the button. A new window opos up with the preview. Behind the scenes the new window is implemented using the pattern i showed you. Think of it like Sigils PageEdit.

jackie_w · 09-11-2019, 12:27 PM

Well, I haven't used Sigil's PageEdit, I'm strictly calibre when it comes to editing, but have I got this right? What you're saying is that instead of a single dialog with 3 widgets layed out in 3 columns there will be an initial dialog with just the filename listwidget which will then launch a child dialog with the 2 before/after webviews for the selected file in it?

eschwartz · 09-11-2019, 02:26 PM

Quote:

Originally Posted by jackie_w

Well, I haven't used Sigil's PageEdit, I'm strictly calibre when it comes to editing, but have I got this right? What you're saying is that instead of a single dialog with 3 widgets layed out in 3 columns there will be an initial dialog with just the filename listwidget which will then launch a child dialog with the 2 before/after webviews for the selected file in it?

What about making the plugin launch a separate process that does *everything* in the preview?

I don't recall offhand if it needs to connect back to the DB API. But that would be the only thing, honestly, which actually needs to be done in-process for locking reasons, right? So the plugin could just be there to provide, essentially, a GUI button to launch the standalone utility.

jackie_w · 09-11-2019, 03:10 PM

Quote:

Originally Posted by eschwartz

What about making the plugin launch a separate process that does *everything* in the preview?

If I can get Kovid's idea working at all I can experiment with what goes on each side of the 'divide'. At the moment I'm at the 'are-you-up-to-this-jackie' stage and it will be closely followed by the stage I call 'why-are-you-sharing-plugins-with-others-when-you-know-you're-at-the-limit-of-your-abilities'. I hate large calibre upgrades. Not really, but it's always way outside my comfort zone

Quote:

Originally Posted by eschwartz

I don't recall offhand if it needs to connect back to the DB API. But that would be the only thing, honestly, which actually needs to be done in-process for locking reasons, right? So the plugin could just be there to provide, essentially, a GUI button to launch the standalone utility.

I believe the only link to the library is upfront to get hold of the right epub/azw3 file. Once there's a copy of it in calibre's temp area the library isn't involved as the scrambled book file is saved to the OS outside the library.

eschwartz · 09-11-2019, 03:25 PM

Quote:

Originally Posted by jackie_w

I believe the only link to the library is upfront to get hold of the right epub/azw3 file. Once there's a copy of it in calibre's temp area the library isn't involved as the scrambled book file is saved to the OS outside the library.

Perfect! So I'd advise that, then. e.g. let the plugin be able to run with calibre-debug --run-plugin entry points, accepting the path of the ebook to scramble. Make the GUI element simply figure out the path to the book to be scrambled, and use run_calibre_debug to spawn a new instance of the plugin which is set up to do webengine stuff.

09-11-2019, 10:35 AM	#7
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@jackie_w: You should be able to use qwebengine in your plugin by running it in a separate process using calibre-debug so your previewer becomes a simple standalone program inside your plugin. Something like Code: from calibre.debug import run_calibre-debug run_calibre_debug(['-c', 'from calibre_plugins.your_plugin.viewer import main; import sys; main(sys.argv[-2], sys.argv[-1]);', 'path to first html file', 'path to second html file']) And then implement your viewer using qwebengine. Last edited by kovidgoyal; 09-11-2019 at 11:42 AM.

09-11-2019, 11:46 AM	#9
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No cant say I have an example, but here is a barebones viewer.py to add to your plugin. It only shows a single passed in html file, for breviry Code: def main(path_to_html_file, path_to_ther_html_file): from calibre.gui2 import Application app = Application([]) w = QWebEngineView() w.setUrl(QUril.fromlocalFile(path_to_html_file)) w.show() w.raise_() app.exec_() add it to your plugin as viewer.py and call it using the run_calibre_debug() function I showed you earlier, and there you have a web engine based HTML viewer.

09-11-2019, 12:27 PM	#12
jackie_w Grand Sorcerer Posts: 6,212 Karma: 16534894 Join Date: Sep 2009 Location: UK Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3	Well, I haven't used Sigil's PageEdit, I'm strictly calibre when it comes to editing, but have I got this right? What you're saying is that instead of a single dialog with 3 widgets layed out in 3 columns there will be an initial dialog with just the filename listwidget which will then launch a child dialog with the 2 before/after webviews for the selected file in it? Last edited by jackie_w; 09-11-2019 at 12:31 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
calibre-sync: Automatically download books from your Calibre Content Server onto Kobo	mishagale	Related Tools	14	05-13-2019 05:18 PM
Calibre plus Moon + Reader Pro - How to Make Calibre Library Sync to My Shelf?	bookiebabe	Related Tools	5	02-09-2017 02:47 AM
Calibre Sync on Android stopped working properly with recent version of Calibre?	khahoon	Calibre	5	07-06-2013 08:14 AM
How do I import Calibre settings from regular Win Calibre to Calibre Portable?	ABW	Calibre	9	05-20-2013 02:34 PM
Can not start recoll from calibre /opt/calibre/lib/libz.so.1: no version information	Satas	Development	5	04-19-2013 11:22 PM

09-09-2019, 10:17 PM	#2
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@jackie_w: That is because webengine is not useable in the main calibre program only the editor/viewer. This is because it is very resource intensive and Qt requires that it be imported before starting the appication. Since the main GUI does not use it, I do not import it, which also means it cant be imported in any plugins. What does scrambleebook use it for? There are two possible workarounds, it is is not something that requires sophisticated HTML support then QTextBrowser can be used, otherwise you have to run it in a separate process. @eschwartz: Given that all plugins will need to be ported to python3, it will be a long time before calibre is ready for py3. But yes, my next painful job after v4 is done is to work on getting betas ready with py3 so the plugin developers can start work on porting their plugins.

09-10-2019, 02:08 AM	#4
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@jackie_w f all you are doing is previewing some text so users can check it, probably QTextBrower will be good enough for you, that is the simplest option.

09-11-2019, 12:01 PM	#10
jackie_w Grand Sorcerer Posts: 6,212 Karma: 16534894 Join Date: Sep 2009 Location: UK Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3	Thanks for this. I'm still having problems 'joining the dots' in how this viewer links to what's happening in the central listwidget column where when the user moves the cursor up and down the list of html names the contents of the webviews on either side change accordingly. Rather than exasperate you further let me think about it some more and try to sort it out - or maybe come up with some better questions.

09-11-2019, 12:09 PM	#11
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	basically rather than having the column view you have now, you instead have a preview button. the user selects the file he wants to preview the changes to and clicks the button. A new window opos up with the preview. Behind the scenes the new window is implemented using the pattern i showed you. Think of it like Sigils PageEdit.

Advert

Advert