Attn plugin devs: porting to python 3

[davidfor]

Originally Posted by kovidgoyal

I haven't looked into it yet but almost certainly this i because in windows strings are natively utf-16 which is what they are in python 2 as well and in ICU, so there is no conversion required. With python3 one has to copy/convert strings to and fro

I have just updated most of the Count Pages plugin for Python 3, and I can see this happening. The non-ICU count for a 1.1 million word book took 11 seconds. Using the ICU library it was running for 10 minutes before I killed it. With 4.8, it was less than 20 seconds.

There were similar timings reported on some Linux boxes late last year. I couldn't dig into it, but, I assumed it was something like this as well.

[kovidgoyal]

Well I looked at the code and there was an O(n^2) algorithm in there, that is now gone. However, that algorithm was always there under linux, so I am surprised that linux performance is fine. After my changes, counting 450K words from the gutenberg count of monte cristo text file takes 0.1 seconds with python 3 and 0.2 seconds with 4.8 on windows and similar on linux. Tested with

 calibre-debug -c "from calibre.spell.break_iterator import *; import sys; raw = open(sys.argv[-1], 'rb').read().decode('utf-8'); from calibre.utils.monotonic import *; st = monotonic(); print(count_words(raw)); print(monotonic() - st);" cmc.txt

[eschwartz]

Hrm.

Now on linux with the latest changes, for an EPUB with 313426 words, I still consistently get times of around 3.8 on python3, but it dropped to 0.5 on python2. (Using jackie_w's test script.)

[kovidgoyal]

Use the calibre-debug code to check count_words() it will eliminate other issues.

[eschwartz]

Sorry, I'm stupid. I used /usr/bin/python3 with "import init_calibre" (easier than following absolute paths to where I installed each version of calibre-debug), but it turns out I was missing /home/eschwartz/.local/lib/python3.8/site-packages/init_calibre.py, so it ended up using the stable tagged version installed in /usr instead.

I guess that's the cost of having multiple copies of calibre installed I have a python2 build and a python3 build to test for distribution packaging, and another set of python2 and python3 builds which I usually use actively.

...

To be clear, after fixing my install, my test EPUB now averages 0.3 on python3, and 0.5 on python2, and your calibre-debug line (on a downloaded project gutenberg /tmp/1184-0.txt) now averages 0.093600690001 on python2 and 0.08859269299864536 on python3, so that's consistently a bit better on python3.

Linux performance doesn't seem to care significantly whether python2 or python3 is used, but the current git master makes both equally faster (your calibre-debug line took 409.37419593899904 when running it on python3 with the old O(n^2) code! and 0.08 with the new code).

[kovidgoyal]

And I uploaded 4.99.3 with the fix.

[jackie_w]

Originally Posted by kovidgoyal

And I uploaded 4.99.3 with the fix.

Yes, that's much better, thanks

[DaltonST]

See the attached images of a Calibre 4.99.3 bytes/string type error.


DaltonST

[eschwartz]

The module has a test which you can run like this:

$ calibre-debug /home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py /tmp/foo.txt
Traceback (most recent call last):
  File "/usr/local/calibre-3/bin/calibre-debug", line 20, in <module>
    sys.exit(main())
  File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 349, in main
    run_script(args[1], args[2:])
  File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 261, in run_script
    exec_path(ef, g)
  File "/home/eschwartz/git/calibre/src/polyglot/builtins.py", line 103, in exec_path
    exec(code, ctx)
  File "/home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py", line 126, in <module>
    create_book(mi, path, fmt=ext)
  File "/home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py", line 45, in create_book
    f.write(mi.title)
TypeError: a bytes-like object is required, not 'str'

https://github.com/kovidgoyal/calibre/pull/1085

[DaltonST]

TypeError: can only concatenate str (not "list") to str

See the attached image.


DaltonST

[DaltonST]

See the attached.



DaltonST

[kovidgoyal]

@DaltonST: that TypeError is not python 3 related as fr as I can see. And the font subsetting error has been fixed by https://github.com/kovidgoyal/calibr...ed799a8d110ae2

[davidfor]

I have just posted a py3 version of the Goodreads Sync plugin. All that swearing you heard from downunder was me, not people fighting the fires. Of course, if I'd checked for updates to one of the libraries it includes, it would have been a lot easier.

[kovidgoyal]

Originally Posted by davidfor

I have just posted a py3 version of the Goodreads Sync plugin. All that swearing you heard from downunder was me, not people fighting the fires. Of course, if I'd checked for updates to one of the libraries it includes, it would have been a lot easier.

just be glad you didn't have to port all half a million lines of calibre to python 3.

[DaltonST]

V4.99.4 look_feel.py Error in Preferences > Look & Feel

See the attached image.


DaltonST