01-06-2020, 07:22 AM | #91 | |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
There were similar timings reported on some Linux boxes late last year. I couldn't dig into it, but, I assumed it was something like this as well. |
|
01-06-2020, 07:31 AM | #92 |
creator of calibre
Posts: 43,850
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Well I looked at the code and there was an O(n^2) algorithm in there, that is now gone. However, that algorithm was always there under linux, so I am surprised that linux performance is fine. After my changes, counting 450K words from the gutenberg count of monte cristo text file takes 0.1 seconds with python 3 and 0.2 seconds with 4.8 on windows and similar on linux. Tested with
Code:
calibre-debug -c "from calibre.spell.break_iterator import *; import sys; raw = open(sys.argv[-1], 'rb').read().decode('utf-8'); from calibre.utils.monotonic import *; st = monotonic(); print(count_words(raw)); print(monotonic() - st);" cmc.txt |
01-06-2020, 07:52 AM | #93 |
Ex-Helpdesk Junkie
Posts: 19,422
Karma: 85397180
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
|
Hrm.
Now on linux with the latest changes, for an EPUB with 313426 words, I still consistently get times of around 3.8 on python3, but it dropped to 0.5 on python2. (Using jackie_w's test script.) |
01-06-2020, 09:19 AM | #94 |
creator of calibre
Posts: 43,850
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Use the calibre-debug code to check count_words() it will eliminate other issues.
|
01-06-2020, 09:33 AM | #95 |
Ex-Helpdesk Junkie
Posts: 19,422
Karma: 85397180
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
|
Sorry, I'm stupid. I used /usr/bin/python3 with "import init_calibre" (easier than following absolute paths to where I installed each version of calibre-debug), but it turns out I was missing /home/eschwartz/.local/lib/python3.8/site-packages/init_calibre.py, so it ended up using the stable tagged version installed in /usr instead.
I guess that's the cost of having multiple copies of calibre installed I have a python2 build and a python3 build to test for distribution packaging, and another set of python2 and python3 builds which I usually use actively. ... To be clear, after fixing my install, my test EPUB now averages 0.3 on python3, and 0.5 on python2, and your calibre-debug line (on a downloaded project gutenberg /tmp/1184-0.txt) now averages 0.093600690001 on python2 and 0.08859269299864536 on python3, so that's consistently a bit better on python3. Linux performance doesn't seem to care significantly whether python2 or python3 is used, but the current git master makes both equally faster (your calibre-debug line took 409.37419593899904 when running it on python3 with the old O(n^2) code! and 0.08 with the new code). Last edited by eschwartz; 01-06-2020 at 09:37 AM. |
01-06-2020, 09:43 AM | #96 |
creator of calibre
Posts: 43,850
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
And I uploaded 4.99.3 with the fix.
|
01-06-2020, 11:11 AM | #97 |
Grand Sorcerer
Posts: 6,212
Karma: 16534894
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
|
|
01-07-2020, 10:25 AM | #98 |
Deviser
Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
|
File "create.py", line 45, in create_book TypeError: a bytes-like object is required
See the attached images of a Calibre 4.99.3 bytes/string type error.
DaltonST |
01-07-2020, 10:54 AM | #99 |
Ex-Helpdesk Junkie
Posts: 19,422
Karma: 85397180
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
|
The module has a test which you can run like this:
Code:
$ calibre-debug /home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py /tmp/foo.txt Traceback (most recent call last): File "/usr/local/calibre-3/bin/calibre-debug", line 20, in <module> sys.exit(main()) File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 349, in main run_script(args[1], args[2:]) File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 261, in run_script exec_path(ef, g) File "/home/eschwartz/git/calibre/src/polyglot/builtins.py", line 103, in exec_path exec(code, ctx) File "/home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py", line 126, in <module> create_book(mi, path, fmt=ext) File "/home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py", line 45, in create_book f.write(mi.title) TypeError: a bytes-like object is required, not 'str' |
01-07-2020, 11:53 AM | #100 |
Deviser
Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
|
Polishing Job: TypeError
TypeError: can only concatenate str (not "list") to str
See the attached image. DaltonST |
01-07-2020, 01:42 PM | #101 |
Deviser
Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
|
Polishing Job: Not Infrequent Error in 4.99.3 in "Real" Books
See the attached.
DaltonST |
01-07-2020, 10:19 PM | #102 |
creator of calibre
Posts: 43,850
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
@DaltonST: that TypeError is not python 3 related as fr as I can see. And the font subsetting error has been fixed by https://github.com/kovidgoyal/calibr...ed799a8d110ae2
|
01-12-2020, 05:42 AM | #103 |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
I have just posted a py3 version of the Goodreads Sync plugin. All that swearing you heard from downunder was me, not people fighting the fires. Of course, if I'd checked for updates to one of the libraries it includes, it would have been a lot easier.
|
01-12-2020, 07:19 AM | #104 |
creator of calibre
Posts: 43,850
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
just be glad you didn't have to port all half a million lines of calibre to python 3.
|
04-28-2020, 10:52 AM | #105 |
Deviser
Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
|
V4.99.4 look_feel.py Error in Preferences > Look & Feel
V4.99.4 look_feel.py Error in Preferences > Look & Feel
See the attached image. DaltonST |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Plugin Devs: Should we include a License or COPYING file? | KevinH | Plugins | 3 | 08-15-2017 01:09 PM |
Attn plugin developers: calibre moving to Qt 5 | kovidgoyal | Plugins | 157 | 01-01-2015 11:22 PM |
Can I develop a plugin in a launguage other than Python? | jamawg | Development | 2 | 06-13-2014 12:38 AM |
Calibre in a Python 3.2 world, attn: Kovid | Kevin McAleavey | Calibre | 8 | 01-09-2012 05:49 PM |
How do I Create a Python Plugin? | Sydney's Mom | Plugins | 25 | 01-27-2010 06:26 AM |