[GUI Plugin] Count Pages - Page 88

theducks · 12-05-2019, 09:51 AM

Quote:

Originally Posted by davidfor

I tried this at home tonight, and the results were completely different. For the same book, it took less than 20 seconds. And was a little faster than the non-ICU word count.

I don't think it is the hardware. Home is Windows 10 laptop, with an I7 and 16GB of memory. At work it was a desktop running Ubuntu, which I think has 16GB of memory, I don't know the CPU, but, it isn't slow otherwise.

David, It may be the hardware architecture. Many years ago, my boss bought an Hot i7 16G system to run a 8 channel video streamer. Fail- random streams locked.
The stream board Mfgr was puzzled. 'We use a Dell 386 server with 3 of those boards (12 streams)'
The boss bought a used Dell 1U, rack server. No more problems.
Game systems are Graphics intensive. Servers tend to be Data intensive.

davidfor · 12-05-2019, 11:11 PM

Quote:

Originally Posted by theducks

David, It may be the hardware architecture. Many years ago, my boss bought an Hot i7 16G system to run a 8 channel video streamer. Fail- random streams locked.
The stream board Mfgr was puzzled. 'We use a Dell 386 server with 3 of those boards (12 streams)'
The boss bought a used Dell 1U, rack server. No more problems.
Game systems are Graphics intensive. Servers tend to be Data intensive.

It could be. Checking the work machine, it to is an i7, probably a newer version than in the laptop. 16GB memory, graphics card is "NVIDIA Corporation GM107GL [Quadro K2200]". It's an ordinary desktop, built mainly to be able to access the production servers and develop code for them. I wouldn't expect it to have anything unusual in it. Or in my Windows laptop.

Just retested it on the work machine. It runs one core at 100% for the duration of the count. I'll try one of the other machines at home, but, I'll probably have to setup a Linux machine to get another test case.

snarkophilus · 12-07-2019, 05:09 AM

No idea if this info will help or not, but speed of counting words with ICU algo enabled using a simple "words per minute" metric varies a lot with the largest (by word count) books in my library. Calibre is running on an Intel NUC with an i3-5010 at 2.1GHz, 16GB RAM and Windows 10 Home. The following are all with a single job executing at a time:

IOO Classic Books I (Amazon)
3722105 words 2h30m49s = 24814 wpm

The Complete Works of C. S. Lewis (Google Play)
1549384 words 32m32s = 48418 wpm

Complete Works Oscar Wilde (Mobileread)
1155998 words 26m34s = 44461 wpm

Mark Twain. The Complete Novels (Amazon)
927648 words 7m01s = 132521 wpm

Galaxia: Thirteen Stories Spanning the Unknown Multiverse (Amazon)
857373 words 9m35s = 95263 wpm

A Jules Verne Omnibus (Mobileread)
766952 words 10m31s = 76695 wpm

Sherlock Holmes Complete (eReader)
665358 words 5m11s = 133071 wpm

Sherlock Holmes Omnibus (Mobileread)
659356 words 6m20s = 109892 wpm

Complete Sherlock Holmes (Amazon)
627972 words 5m05s = 125594 wpm

Sherlock Holmes Ultimate Collection (Amazon)
577244 words 3m47s = 192414 wpm

The books from Amazon and eReader have been converted by Calibre, the others are unconverted ePubs.

davidfor · 12-07-2019, 06:09 AM

I just grabbed "Complete Works Oscar Wilde". It took 9 seconds on my laptop (Win10, 16GB, i7, NVIDIA graphics).

Could you post the log from a count? It doesn't matter which, I just want to check in case it shows anything strange.

JSWolf · 12-07-2019, 04:58 PM

Quote:

Originally Posted by davidfor

I just grabbed "Complete Works Oscar Wilde". It took 9 seconds on my laptop (Win10, 16GB, i7, NVIDIA graphics).

Could you post the log from a count? It doesn't matter which, I just want to check in case it shows anything strange.

8.5 seconds on my laptop running Win10 64-bit, 8GB, i7 5700HQ running about 2.7GHz, and some flavor Intel HD graphics.

NiLuJe · 12-07-2019, 05:34 PM

Assuming that's the copy found here on MR, you can chuck the "it's a Linux thing" theory out the window (pun intended), it takes less than 2s over here.

Granted, that is *NOT* with the official Calibre binaries, so, eh, YMMV. (Desktop souped up Skylake, x64, AMD). Also, /tmp lives in RAM (tmpfs).

Code:

Count Page/Word Statistics
        do_count_statistics - book_path=/tmp/calibre_4.5.0_tmp_PROEuY/GUM_VG_count_pages/6376.epub, pages_algorithm=0, page_count_mode=Estimate, statistics_to_run=[u'PageCount'], custom_chars_per_page=1500, icu_wordcount=True
        do_count_statistics - job started for file book_path=/tmp/calibre_4.5.0_tmp_PROEuY/GUM_VG_count_pages/6376.epub
        -------------------------------
        Logfile for book ID 6376 (Complete Works)
                Method of counting _page_count_mode=Estimate _download_sources=[]
                results= {u'PageCount': 4259}
                Found 4259 pages
        6376
        do_statistics_for_book:  /tmp/calibre_4.5.0_tmp_PROEuY/GUM_VG_count_pages/6376.epub 0 Estimate [] [u'PageCount'] 1500 True
                Estimated accurate page count
                  Lines: 132049  Divs: 773  Paras: 42789
                  Accurate count: 4259  Fast count: 3122
                Page count: 4259

BetterRed · 12-07-2019, 05:35 PM

Soft hyphens perhaps

BR

davidfor · 12-07-2019, 06:27 PM

Quote:

Originally Posted by NiLuJe

Assuming that's the copy found here on MR, you can chuck the "it's a Linux thing" theory out the window (pun intended), it takes less than 2s over here.

Granted, that is *NOT* with the official Calibre binaries, so, eh, YMMV. (Desktop souped up Skylake, x64, AMD). Also, /tmp lives in RAM (tmpfs).

Code:

Count Page/Word Statistics
        do_count_statistics - book_path=/tmp/calibre_4.5.0_tmp_PROEuY/GUM_VG_count_pages/6376.epub, pages_algorithm=0, page_count_mode=Estimate, statistics_to_run=[u'PageCount'], custom_chars_per_page=1500, icu_wordcount=True
        do_count_statistics - job started for file book_path=/tmp/calibre_4.5.0_tmp_PROEuY/GUM_VG_count_pages/6376.epub
        -------------------------------
        Logfile for book ID 6376 (Complete Works)
                Method of counting _page_count_mode=Estimate _download_sources=[]
                results= {u'PageCount': 4259}
                Found 4259 pages
        6376
        do_statistics_for_book:  /tmp/calibre_4.5.0_tmp_PROEuY/GUM_VG_count_pages/6376.epub 0 Estimate [] [u'PageCount'] 1500 True
                Estimated accurate page count
                  Lines: 132049  Divs: 773  Paras: 42789
                  Accurate count: 4259  Fast count: 3122
                Page count: 4259

That only shows an APNX page count. The problem seem to be in the ICU library included in calibre and the plugin uses for the ICU count.

NiLuJe · 12-07-2019, 06:38 PM

@davidfor: Ah, good point. Remnant from the Kindle days

.

What should I be testing, then? Switching to ADE yields the very same results (except for a different count, of course ;p), FWIW.

EDIT: Same thing for every algorithm in that combobox (in fact, APNX might possibly be the slowest of the bunch, by a very thin margin). Use ICU is checked at all times.

As I said, that's a source build on a source distribution, so things might very well vary on a binary install

.

davidfor · 12-07-2019, 07:55 PM

Quote:

Originally Posted by NiLuJe

@davidfor: Ah, good point. Remnant from the Kindle days

.

What should I be testing, then? Switching to ADE yields the very same results (except for a different count, of course ;p), FWIW.

EDIT: Same thing for every algorithm in that combobox (in fact, APNX might possibly be the slowest of the bunch, by a very thin margin). Use ICU is checked at all times.

As I said, that's a source build on a source distribution, so things might very well vary on a binary install

.

The problem is the word count, not the page count. You are not doing a word count. Set a column for this and it will be done.

The problem looks to be in the ICU library. Unless you are setting different options for that, or the build takes into account your hardware, then it shouldn't matter. But, it will add a data point.

And from a UI point of view, the ICU check box should be disabled if no column is selected for the word count.

snarkophilus · 12-07-2019, 07:57 PM

Quote:

Originally Posted by davidfor

Could you post the log from a count? It doesn't matter which, I just want to check in case it shows anything strange.

Nothing hugely useful at a glance:

Spoiler:

It was faster this time - 26m06s today vs 26m36s yesterday

I'm using the current Calibre Windows 64-bit build. I update everytime Calibre reminds me there's a new version.

Are there any Calibre debug options that might give a more verbose log file?

EDIT: I'm using the ADE page count algorithm if that's relevant.

EDIT2: Attached screenshot of my Count Pages configuration

davidfor · 12-07-2019, 08:27 PM

Quote:

Originally Posted by snarkophilus

Nothing hugely useful at a glance:

Spoiler:

It was faster this time - 26m06s today vs 26m36s yesterday

I'm using the current Calibre Windows 64-bit build. I update everytime Calibre reminds me there's a new version.

Are there any Calibre debug options that might give a more verbose log file?

EDIT: I'm using the ADE page count algorithm if that's relevant.

EDIT2: Attached screenshot of my Count Pages configuration

It does tell me what I wanted to know. The count is done in a couple of ways. I wanted to check it was happening correctly. The two lines I highlighted tell me it is.

And the configuration is what I was expecting.

The problem seem to be in a library calibre has. The only other place I can see that it is used is in the editor for the spelling check. Can you open the book and then open one of the internal files? If the option to show mispelt words is on, it will probably take a long time. It is different to the plugin. The spelling check is done at the tag level (basically per paragraph) whereas the plugin does it for the full book in one go.

snarkophilus · 12-07-2019, 09:03 PM

Quote:

Originally Posted by davidfor

The problem seem to be in a library calibre has. The only other place I can see that it is used is in the editor for the spelling check. Can you open the book and then open one of the internal files? If the option to show mispelt words is on, it will probably take a long time. It is different to the plugin. The spelling check is done at the tag level (basically per paragraph) whereas the plugin does it for the full book in one go.

Do you mean open the book with Calibre's in built epub editor? I've got the "Show misspelled words underlined in the code view" option enabled (must be the default, I only use Sigil and haven't used the Calibre editor). DorianGray.htm is the biggest text file in the book and it opens pretty much instantly.

NiLuJe · 12-07-2019, 09:34 PM

@davidfor:

Duh. That is indeed currently burning a hole in a CPU core ;p.

EDIT: Nothing untoward happening in the editor, and spellchecking works as expected.

davidfor · 12-07-2019, 10:07 PM

Quote:

Originally Posted by snarkophilus

Do you mean open the book with Calibre's in built epub editor? I've got the "Show misspelled words underlined in the code view" option enabled (must be the default, I only use Sigil and haven't used the Calibre editor). DorianGray.htm is the biggest text file in the book and it opens pretty much instantly.

My statement was badly phrased. It's about how long it takes to do the spelling check and mark the words. I'd expect this to take a long time for a decent size for you. What you could try is to turn off the spelling option, reopen the file and go to the bottom. Add a misspelled word and turn the option on again. For me, the word gets marked in a few seconds. If what I think is going on, it might take minutes for that to happen.

12-07-2019, 06:38 PM	#1314
NiLuJe BLAM! Posts: 13,506 Karma: 26047202 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	@davidfor: Ah, good point. Remnant from the Kindle days . What should I be testing, then? Switching to ADE yields the very same results (except for a different count, of course ;p), FWIW. EDIT: Same thing for every algorithm in that combobox (in fact, APNX might possibly be the slowest of the bunch, by a very thin margin). Use ICU is checked at all times. As I said, that's a source build on a source distribution, so things might very well vary on a binary install . Last edited by NiLuJe; 12-07-2019 at 06:42 PM.

12-07-2019, 09:34 PM	#1319
NiLuJe BLAM! Posts: 13,506 Karma: 26047202 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	@davidfor: Duh. That is indeed currently burning a hole in a CPU core ;p. EDIT: Nothing untoward happening in the editor, and spellchecking works as expected. Last edited by NiLuJe; 12-07-2019 at 09:38 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Quality Check	kiwidude	Plugins	1252	08-02-2025 09:53 AM
[GUI Plugin] Open With	kiwidude	Plugins	404	02-21-2025 05:42 AM
[GUI Plugin] Quick Preferences	kiwidude	Plugins	62	03-16-2024 11:47 PM
[GUI Plugin] Kindle Collections (old)	meme	Plugins	2070	08-11-2014 12:02 AM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

12-07-2019, 05:09 AM	#1308
snarkophilus Wannabe Connoisseur Posts: 426 Karma: 2516674 Join Date: Apr 2011 Location: Geelong, Australia Device: Kobo Libra 2, Kobo Aura 2, Sony PRS-T1, Sony PRS-350, Palm TX	No idea if this info will help or not, but speed of counting words with ICU algo enabled using a simple "words per minute" metric varies a lot with the largest (by word count) books in my library. Calibre is running on an Intel NUC with an i3-5010 at 2.1GHz, 16GB RAM and Windows 10 Home. The following are all with a single job executing at a time: IOO Classic Books I (Amazon) 3722105 words 2h30m49s = 24814 wpm The Complete Works of C. S. Lewis (Google Play) 1549384 words 32m32s = 48418 wpm Complete Works Oscar Wilde (Mobileread) 1155998 words 26m34s = 44461 wpm Mark Twain. The Complete Novels (Amazon) 927648 words 7m01s = 132521 wpm Galaxia: Thirteen Stories Spanning the Unknown Multiverse (Amazon) 857373 words 9m35s = 95263 wpm A Jules Verne Omnibus (Mobileread) 766952 words 10m31s = 76695 wpm Sherlock Holmes Complete (eReader) 665358 words 5m11s = 133071 wpm Sherlock Holmes Omnibus (Mobileread) 659356 words 6m20s = 109892 wpm Complete Sherlock Holmes (Amazon) 627972 words 5m05s = 125594 wpm Sherlock Holmes Ultimate Collection (Amazon) 577244 words 3m47s = 192414 wpm The books from Amazon and eReader have been converted by Calibre, the others are unconverted ePubs.

12-07-2019, 06:09 AM	#1309
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	I just grabbed "Complete Works Oscar Wilde". It took 9 seconds on my laptop (Win10, 16GB, i7, NVIDIA graphics). Could you post the log from a count? It doesn't matter which, I just want to check in case it shows anything strange.

12-07-2019, 05:35 PM	#1312
BetterRed null operator (he/him) Posts: 21,794 Karma: 30237628 Join Date: Mar 2012 Location: Sydney Australia Device: none	Soft hyphens perhaps BR