Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 01-06-2020, 08:22 AM   #91
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 18,379
Karma: 30662156
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by kovidgoyal View Post
I haven't looked into it yet but almost certainly this i because in windows strings are natively utf-16 which is what they are in python 2 as well and in ICU, so there is no conversion required. With python3 one has to copy/convert strings to and fro
I have just updated most of the Count Pages plugin for Python 3, and I can see this happening. The non-ICU count for a 1.1 million word book took 11 seconds. Using the ICU library it was running for 10 minutes before I killed it. With 4.8, it was less than 20 seconds.

There were similar timings reported on some Linux boxes late last year. I couldn't dig into it, but, I assumed it was something like this as well.
davidfor is offline   Reply With Quote
Old 01-06-2020, 08:31 AM   #92
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,046
Karma: 15014823
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Well I looked at the code and there was an O(n^2) algorithm in there, that is now gone. However, that algorithm was always there under linux, so I am surprised that linux performance is fine. After my changes, counting 450K words from the gutenberg count of monte cristo text file takes 0.1 seconds with python 3 and 0.2 seconds with 4.8 on windows and similar on linux. Tested with

Code:
 calibre-debug -c "from calibre.spell.break_iterator import *; import sys; raw = open(sys.argv[-1], 'rb').read().decode('utf-8'); from calibre.utils.monotonic import *; st = monotonic(); print(count_words(raw)); print(monotonic() - st);" cmc.txt
kovidgoyal is online now   Reply With Quote
Old 01-06-2020, 08:52 AM   #93
eschwartz
Ex-Helpdesk Junkie
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 19,358
Karma: 83238367
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Hrm.

Now on linux with the latest changes, for an EPUB with 313426 words, I still consistently get times of around 3.8 on python3, but it dropped to 0.5 on python2. (Using jackie_w's test script.)
eschwartz is offline   Reply With Quote
Old 01-06-2020, 10:19 AM   #94
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,046
Karma: 15014823
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Use the calibre-debug code to check count_words() it will eliminate other issues.
kovidgoyal is online now   Reply With Quote
Old 01-06-2020, 10:33 AM   #95
eschwartz
Ex-Helpdesk Junkie
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 19,358
Karma: 83238367
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Sorry, I'm stupid. I used /usr/bin/python3 with "import init_calibre" (easier than following absolute paths to where I installed each version of calibre-debug), but it turns out I was missing /home/eschwartz/.local/lib/python3.8/site-packages/init_calibre.py, so it ended up using the stable tagged version installed in /usr instead.

I guess that's the cost of having multiple copies of calibre installed I have a python2 build and a python3 build to test for distribution packaging, and another set of python2 and python3 builds which I usually use actively.

...

To be clear, after fixing my install, my test EPUB now averages 0.3 on python3, and 0.5 on python2, and your calibre-debug line (on a downloaded project gutenberg /tmp/1184-0.txt) now averages 0.093600690001 on python2 and 0.08859269299864536 on python3, so that's consistently a bit better on python3.

Linux performance doesn't seem to care significantly whether python2 or python3 is used, but the current git master makes both equally faster (your calibre-debug line took 409.37419593899904 when running it on python3 with the old O(n^2) code! and 0.08 with the new code).

Last edited by eschwartz; 01-06-2020 at 10:37 AM.
eschwartz is offline   Reply With Quote
Old 01-06-2020, 10:43 AM   #96
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,046
Karma: 15014823
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
And I uploaded 4.99.3 with the fix.
kovidgoyal is online now   Reply With Quote
Old 01-06-2020, 12:11 PM   #97
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 5,270
Karma: 12221060
Join Date: Sep 2009
Location: UK
Device: PRS-350, Kobo: Aura6", H2O, GloHD, KA1, ClaraHD, Forma
Quote:
Originally Posted by kovidgoyal View Post
And I uploaded 4.99.3 with the fix.
Yes, that's much better, thanks
jackie_w is offline   Reply With Quote
Old 01-07-2020, 11:25 AM   #98
DaltonST
Deviser
DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.
 
DaltonST's Avatar
 
Posts: 1,606
Karma: 255700
Join Date: Aug 2013
Location: Texas
Device: 10" Win10 Tablet w/Calibre64, CalibreSpy & Freda+
File "create.py", line 45, in create_book TypeError: a bytes-like object is required

See the attached images of a Calibre 4.99.3 bytes/string type error.


DaltonST
Attached Thumbnails
Click image for larger version

Name:	2020-01-07 09_13_20-Window.jpg
Views:	36
Size:	86.7 KB
ID:	176221   Click image for larger version

Name:	2020-01-07 09_15_08-Window.jpg
Views:	35
Size:	201.9 KB
ID:	176222   Click image for larger version

Name:	2020-01-07 09_18_08-Window.jpg
Views:	31
Size:	263.4 KB
ID:	176223  
DaltonST is offline   Reply With Quote
Old 01-07-2020, 11:54 AM   #99
eschwartz
Ex-Helpdesk Junkie
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 19,358
Karma: 83238367
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
The module has a test which you can run like this:

Code:
$ calibre-debug /home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py /tmp/foo.txt
Traceback (most recent call last):
  File "/usr/local/calibre-3/bin/calibre-debug", line 20, in <module>
    sys.exit(main())
  File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 349, in main
    run_script(args[1], args[2:])
  File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 261, in run_script
    exec_path(ef, g)
  File "/home/eschwartz/git/calibre/src/polyglot/builtins.py", line 103, in exec_path
    exec(code, ctx)
  File "/home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py", line 126, in <module>
    create_book(mi, path, fmt=ext)
  File "/home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/create.py", line 45, in create_book
    f.write(mi.title)
TypeError: a bytes-like object is required, not 'str'
https://github.com/kovidgoyal/calibre/pull/1085
eschwartz is offline   Reply With Quote
Old 01-07-2020, 12:53 PM   #100
DaltonST
Deviser
DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.
 
DaltonST's Avatar
 
Posts: 1,606
Karma: 255700
Join Date: Aug 2013
Location: Texas
Device: 10" Win10 Tablet w/Calibre64, CalibreSpy & Freda+
Polishing Job: TypeError

TypeError: can only concatenate str (not "list") to str

See the attached image.


DaltonST
Attached Thumbnails
Click image for larger version

Name:	2020-01-07 10_50_03-Window.jpg
Views:	35
Size:	127.6 KB
ID:	176225  
DaltonST is offline   Reply With Quote
Old 01-07-2020, 02:42 PM   #101
DaltonST
Deviser
DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.
 
DaltonST's Avatar
 
Posts: 1,606
Karma: 255700
Join Date: Aug 2013
Location: Texas
Device: 10" Win10 Tablet w/Calibre64, CalibreSpy & Freda+
Polishing Job: Not Infrequent Error in 4.99.3 in "Real" Books

See the attached.



DaltonST
Attached Thumbnails
Click image for larger version

Name:	2020-01-07 12_34_51-Window.jpg
Views:	36
Size:	140.8 KB
ID:	176230  
DaltonST is offline   Reply With Quote
Old 01-07-2020, 11:19 PM   #102
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,046
Karma: 15014823
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@DaltonST: that TypeError is not python 3 related as fr as I can see. And the font subsetting error has been fixed by https://github.com/kovidgoyal/calibr...ed799a8d110ae2
kovidgoyal is online now   Reply With Quote
Old 01-12-2020, 06:42 AM   #103
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 18,379
Karma: 30662156
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
I have just posted a py3 version of the Goodreads Sync plugin. All that swearing you heard from downunder was me, not people fighting the fires. Of course, if I'd checked for updates to one of the libraries it includes, it would have been a lot easier.
davidfor is offline   Reply With Quote
Old 01-12-2020, 08:19 AM   #104
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,046
Karma: 15014823
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by davidfor View Post
I have just posted a py3 version of the Goodreads Sync plugin. All that swearing you heard from downunder was me, not people fighting the fires. Of course, if I'd checked for updates to one of the libraries it includes, it would have been a lot easier.
just be glad you didn't have to port all half a million lines of calibre to python 3.
kovidgoyal is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Plugin Devs: Should we include a License or COPYING file? KevinH Plugins 3 08-15-2017 02:09 PM
Attn plugin developers: calibre moving to Qt 5 kovidgoyal Plugins 157 01-02-2015 12:22 AM
Can I develop a plugin in a launguage other than Python? jamawg Development 2 06-13-2014 01:38 AM
Calibre in a Python 3.2 world, attn: Kovid Kevin McAleavey Calibre 8 01-09-2012 06:49 PM
How do I Create a Python Plugin? Sydney's Mom Plugins 25 01-27-2010 07:26 AM


All times are GMT -4. The time now is 11:47 PM.


MobileRead.com is a privately owned, operated and funded community.