Attn plugin devs: porting to python 3

[kovidgoyal]

Originally Posted by jackie_w

What about if I install a new Win32 4.6 production version and then a Win64 4.99.2 beta version over the top of my Win64 4.6 production? They would be sharing the same config directory. Will that trash my config?

No you can use the same config with both.

[eschwartz]

Originally Posted by jackie_w

What about if I install a new Win32 4.6 production version and then a Win64 4.99.2 beta version over the top of my Win64 4.6 production? They would be sharing the same config directory. Will that trash my config?

The config directory should support both python2 and python3 without modifications. And in fact, it has done so for me throughout my testing.

[jackie_w]

Originally Posted by kovidgoyal

No you can use the same config with both.

Originally Posted by eschwartz

The config directory should support both python2 and python3 without modifications. And in fact, it has done so for me throughout my testing.

Thanks. I think I'll try this then as the least inconvenient option. I can live with a production version running in Win32 mode for a short time.

When you said

calibre will be migrating from Python 2 to Python 3 sometime next year.

were you anticipating January or December?

[JSWolf]

However, you could use the 4.6 portable and copy the config for use with the portable version. That way you won't have to worry about your config getting messed up with the 4.99+ beta. Plus, all your existing plugins will still work.

[BetterRed]

@jackie_w - FWIW - another way,

1. install the 4.99 portable in c:\_calibre4.99\portable;
2. install the 64 bit 4.99 beta in c:\_calibre4.99\64 bit;
3. copy c:\_calibre4.99\64 bit\calibre to c:\_calibre4.99\portable\calibre;
4. uninstall the 64 bit 4.99 beta in c:\_calibre4.99\64 bit.

NB: the c:\_calibre4.99 locations are notional.

If/when a new beta is released repeat steps 2-4.

This will leave your 'everyday' calibre as is.

BR

[jhowell]

One thing I noticed running portable version of the 4.99.2 beta is a difference in error reporting for an EPUB containing DRM. Trying to convert or edit pops up the expected "This book has DRM" dialog. But invoking the viewer instead produces a different error dialog:

Spoiler:

calibre, version 4.99.2
ERROR: Loading book failed: Failed to open the book at C:\calibre-beta\Calibre Portable\Calibre Library\Harry Harrison\50 in 50 (10457)\50 in 50 - Harry Harrison.epub. Click "Show details" for more info.

Failed to convert book: C:\calibre-beta\Calibre Portable\Calibre Library\Harry Harrison\50 in 50 (10457)\50 in 50 - Harry Harrison.epub with error:
InputFormatPlugin: EPUB Input running
on C:\calibre-beta\Calibre Portable\Calibre Library\Harry Harrison\50 in 50 (10457)\50 in 50 - Harry Harrison.epub
Traceback (most recent call last):
File "runpy.py", line 192, in _run_module_as_main
File "runpy.py", line 85, in _run_code
File "site.py", line 89, in <module>
File "site.py", line 84, in main
File "site.py", line 55, in run_entry_point
File "worker.py", line 195, in main
File "<string>", line 1, in <module>
File "render_book.py", line 829, in viewer_main
File "render_book.py", line 820, in render_for_viewer
File "render_book.py", line 797, in render
File "book.py", line 63, in extract_book
File "conversion.py", line 245, in __call__
File "epub_input.py", line 290, in convert
calibre.ebooks.DRMError: 50 in 50 - Harry Harrison.epub
Failed to run pipe worker with command: from calibre.srv.render_book import viewer_main; viewer_main()

[kovidgoyal]

Originally Posted by jackie_w

were you anticipating January or December?

Mid-year most likely. Want to give plugin developers time to port their plugins and there will likely be a beta for general users as well.

[kovidgoyal]

Originally Posted by jhowell

One thing I noticed running portable version of the 4.99.2 beta is a difference in error reporting for an EPUB containing DRM. Trying to convert or edit pops up the expected "This book has DRM" dialog. But invoking the viewer instead produces a different error dialog:

https://github.com/kovidgoyal/calibr...6bcbb6e8c9b9a5

[jhowell]

Originally Posted by kovidgoyal

https://github.com/kovidgoyal/calibr...6bcbb6e8c9b9a5

Thank you!

[jackie_w]

Win10 running in debug mode from source with a fully up-to-date 4.99.2

I think I've found a bug in the Editor when editing an image. These steps will reproduce it:

1. Open any epub and open an HTML page containing either a JPG or a PNG, so that the image is visible in the Preview window.
2. Right-click the image name and select 'Open'.
3. Rotate the image (or use the Trim tool).
4. The image disappears from Preview and debug shows this error message:
   

   Turning on automatic hidpi scaling
   devicePixelRatio: 1.0
   logicalDpi: 96.0 x 96.0
   physicalDpi: 127.66492146596858 x 127.59069767441859
   Using calibre Qt style: True
   calibre css_parser 1.0.4
   lxml etree: 4.3.3
   Traceback (most recent call last):
     File "preview.py", line 192, in requestStarted
     File "preview.py", line 50, in get_data
     File "image.py", line 155, in get_raw_data
     File "canvas.py", line 337, in get_image_data
   ValueError: Cannot save jpeg format images

GIFs don't appear to have the same problem as JPG/PNG.

[kovidgoyal]

@jackiew: this takes care of it: https://github.com/kovidgoyal/calibr...56f741a8048232

[jackie_w]

Originally Posted by kovidgoyal

@jackiew: this takes care of it: https://github.com/kovidgoyal/calibr...56f741a8048232

Thanks That seems OK, now.

[jackie_w]

@Kovid and @davidfor,

I've been working my way through my Editor plugins and I think I've found something a bit odd with the count_words function in calibre.spell.break_iterator in calibre 4.99.2.

I sometimes use count_words in my Editor plugins before & after a plugin's mass cleanups to sound an immediate alarm if text content accidentally got removed. It seems to run very much slower in 4.99.2 than in 4.8 and I wonder if you can shed any light on why that might be?

@davidfor, I know count_words is also an integral part of the Count Pages plugin.

I ran some tests to illustrate the problem. I selected 4 books (2 long, 1 medium, 1 short) in EPUB2 format. They all validate clean in both calibre CheckBook and EpubCheck.

For each EPUB I counted the words (python script attached in spoiler below) using 3 different versions of calibre in debug mode:
- Win 64bit 4.99.2
- Win 32bit 4.8
- Win 64bit 4.99.2 run from source (fully up-to-date)

These are the results. As you can see v4.99.2 runs 10 to 35 times slower than v4.8:

1. Alexandre Dumas - The Count of Monte Cristo
    DEBUG:    0.0 Start: calibre: 4.99.2 [64bit]; ispy3: True
    DEBUG:    7.4   End: Wordcount: 496791

    DEBUG:    0.0 Start: calibre: 4.8; ispy3: False
    DEBUG:    0.7   End: Wordcount: 496791

    DEBUG:    0.0 Start: calibre: 4.99.2* [64bit]; ispy3: True
    DEBUG:    7.4   End: Wordcount: 496791

2. Peter F Hamilton - The Naked God
    DEBUG:    0.0 Start: calibre: 4.99.2 [64bit]; ispy3: True
    DEBUG:   21.9   End: Wordcount: 455220

    DEBUG:    0.0 Start: calibre: 4.8; ispy3: False
    DEBUG:    0.6   End: Wordcount: 455220

    DEBUG:    0.0 Start: calibre: 4.99.2* [64bit]; ispy3: True
    DEBUG:   22.0   End: Wordcount: 455220

3. EF Benson - Mapp and Lucia
    DEBUG:    0.0 Start: calibre: 4.99.2 [64bit]; ispy3: True
    DEBUG:    2.8   End: Wordcount: 114695

    DEBUG:    0.0 Start: calibre: 4.8; ispy3: False
    DEBUG:    0.2   End: Wordcount: 114695

    DEBUG:    0.0 Start: calibre: 4.99.2* [64bit]; ispy3: True
    DEBUG:    2.9   End: Wordcount: 114695

4. Evelyn Waugh - Vile Bodies
    DEBUG:    0.0 Start: calibre: 4.99.2 [64bit]; ispy3: True
    DEBUG:    1.3   End: Wordcount: 76836

    DEBUG:    0.0 Start: calibre: 4.8; ispy3: False
    DEBUG:    0.1   End: Wordcount: 76836

    DEBUG:    0.0 Start: calibre: 4.99.2* [64bit]; ispy3: True
    DEBUG:    1.3   End: Wordcount: 76836

This was the simple script I used:

Spoiler:

from __future__ import (unicode_literals, division, absolute_import, print_function)
import sys
import os

from calibre.constants import ispy3, get_version
from calibre.devices.usbms.driver import debug_print
from calibre.ebooks.oeb.base import xml2text
from calibre.ebooks.oeb.polish.container import get_container    
from calibre.spell.break_iterator import count_words

if __name__ == "__main__":   
    pathtoepub = sys.argv[1]
    container = get_container(pathtoepub)
    
    debug_print('Start: calibre: {0}; ispy3: {1}'.format(get_version(), ispy3))
    wordcount = 0
    for name, lin in container.spine_names:
        root = container.parsed(name)
        text = xml2text(root)
        wordcount += count_words(text)

    debug_print('  End: Wordcount: {0}'.format(wordcount))

[eschwartz]

Using linux from source on git master I don't see any difference between python2 and python3; both are averaging times of "0.0".

EDIT: for a 3224 word short story. The variance is no different though, for a 313426 word story, averaging out 3.8 for both (with the time taking 3.6 being one of the python3 ones, as if that were statistically relevant).

[kovidgoyal]

Originally Posted by eschwartz

Using linux from source on git master I don't see any difference between python2 and python3; both are averaging times of "0.0".

I haven't looked into it yet but almost certainly this i because in windows strings are natively utf-16 which is what they are in python 2 as well and in ICU, so there is no conversion required. With python3 one has to copy/convert strings to and fro