[GUI Plugin] Count Pages

[kiwidude]

@vargaviktor - I have no plans to change anything. I call two different calibre utility functions which handle zips and rars, both of which are in my understanding modified versions of some standard Python libraries.

As I said on the quality check thread averaging size/dpi is not an appropriate function for that or indeed this plugin - I suggest if you want such a feature that you write your own plugin for it. Undoubtedly there would be code in plugins like this one which could assist you.

[vargaviktor]

Originally Posted by kiwidude

@vargaviktor - I have no plans to change anything. I call two different calibre utility functions which handle zips and rars, both of which are in my understanding modified versions of some standard Python libraries.

As I said on the quality check thread averaging size/dpi is not an appropriate function for that or indeed this plugin - I suggest if you want such a feature that you write your own plugin for it. Undoubtedly there would be code in plugins like this one which could assist you.

OK, I Understand. The unzipping behaviour was just asked me, to mention you, maybe it can be a bug. Now, I know, this is the correct way of working.

I would like to give a try to make a plugin, but my python is not the best. Your plugin will help me, to concentrate on the calculations. Thank you.

regards. Viktor

[ElMiko]

I'm getting a wordcount that is way off the expected value... by about 90%, would be my guess. Is there any way to determine what's causing hiccup?

[kiwidude]

Remember that word count is for every single file in the ePub that is in the manifest. Frequently books include chapters of upcoming releases and all sorts of other cruft. Use Tweak ePub, open each HTML page in a browser, paste into word and get it to do a word count if you don't believe it. Just don't cheat and skip any pages...

[ElMiko]

Originally Posted by kiwidude

Remember that word count is for every single file in the ePub that is in the manifest. Frequently books include chapters of upcoming releases and all sorts of other cruft. Use Tweak ePub, open each HTML page in a browser, paste into word and get it to do a word count if you don't believe it. Just don't cheat and skip any pages...

Oops, sorry, I wasn't clear. The issue is that it is UNDERcounting. It's giving a wordcount of around 9k for a book that's probably closer to 100k...

[kiwidude]

@ElMiko - the same response applies regardless. I'm not sure what you want expect anyone to tell you. I can't do anything without seeing the book for myself, and who "says" the number is supposed to be 100K? The plugin logic is as I described above, and I've never seen any counts which are widely anomalous to the number of pages (from Goodreads) for the book. Perhaps your copy of the book is corrupted.

[ElMiko]

@kiwidude - I did as you suggested and copy/pasted the text into a word document. In so doing, it seems I may discovered what the problem is, albeit not the solution. There are four html files in the MOBI file, and Count Pages isn't counting the first file.

[kiwidude]

That's because it is invalid html. Look at the closing body tag immediately after the opening body tag and all the content is after the body.

[ElMiko]

Thanks! Ran page count on an EPUB conversion and that did the trick. Really appreciate your looking into it!

[BetterRed]

I'll make this as detailed as I can, I'm using remote control over a 64Kb/s satellite link with long latencies so I want to avoid to-ing & fro-ing with questions and answers.

I use Windows 7 64 bit with all updates installed, Calibre 0.8.69 and Count Pages 1.6.3 - My settings for the plug-in are attached.

I use Process Explorer to observe what programs are doing.

Some of my books are taking a long time to word count (15 minutes and more), although word counts of similar books of similar size, by the same author, in the same format, from the same source take a few seconds

First what happens when I count words in a single 'slow' book.

Whilst the plugin is counting a 'slow' book's words, one of the two instances of calibre-parallel uses 24.n% of my quad core I5 - i.e almost an entire core. My observation is that the main calibre process spawns a calibre-parallel process which spawns another calibre-parallel process. It's the second calibre-parallel that chews up processor resources, so I assume its the one doing the 'work'. This situation doesn't bother me too much - it gets there eventually

Now what happens when I select a group of books.

I can run into the situation where the group includes several of these 'slow' books. The plug-in spawns up to 5 instances of calibre-parallel. When this happens my computer can become unusable for an unacceptable period (my patience ran out after 33 minutes); because the 4 calibre-parallel secondary processes are EACH consuming 24.n% of a CPU core processing 'slow' books, totaling ~98% of the entire CPU

If I fight with the sloth like mouse to stop the job, that doesn't give me back the CPU resources. Because whilst the primary instance of calibre-parallel dies, the four that were doing the work get detached from the main calibre process, and continue working independently and continue to hog the CPU, so I have to kill them individually with Process Explorer.:

So rather than killing the job from calibre, its faster & easier to kill the process tree of the primary calibre-parallel process with Process Explorer. Calibre complains, but it doesn't crash and there's no apparent harm done; i.e. the books and database are OK because the plug-in is not accessing them when its doing the counting.

What have I done to try to 'fix' it

Based on reading this thread I progressively disabled DEP (at the command line so it was disabled for everything including Windows), disabled my AV, disabled the firewall, disconnected the router, closed all other programs (including disabling the ones that start in the tray), and restarted Windows between each, the final one into Safe Mode - all to no avail.

I haven't tried Closing the Tag Browser, which I saw suggested in this thread and I've seen suggested elsewhere as a possible solution to various problems - because I can't figure out how to do that.

There's no sign of any memory leaks . And the 'slow books' are in the minority, estimate < 10%.

Changing the algo between 'ADE' & 'calibre E-book' makes no obvious difference to the speed. Didn't try the 'APNX ' algo because it seems to be about page counting.

Is there some way to have this plug-in work in a serial manner rather than multi-tasking via spawning multiple calibre-parallel processes, i.e. limit the number of secondary instances of calibre-parallel to ONE.

It would 'help' if I could determine the identity (author-title) of book(s) currently being processed in a job, or books that have completed. Then I could kill the job, put the 'slow books' aside to be done one at a time, and redo the group without the slow books.

My limited tests show that a 'slow to count' book does not appear to take significantly longer to convert to & from EPUB and RTF than similar 'fast to count' books. In fact 'slow book' conversion is usually (always ?) faster 'slow book' word counting, but the opposite is true of a 'fast book'

I have no doubt that its something to do with the book content, but I've no idea what - given that similar books are OK, and other plugs-ins are not slow in their processing of the same books

I'll try starting calibre via the start command with the affinity switch set at 2, theoretically that should limit calibre to 2 cores only

BR

[DoctorOhh]

Originally Posted by BetterRed

I can run into the situation where the group includes several of these 'slow' books. The plug-in spawns up to 5 instances of calibre-parallel. When this happens my computer becomes practically unusable for an unacceptable period (my patience ran out after 33 minutes); because 4 of the calibre-parallel processes are EACH consuming 24.n% of a CPU core, totalling ~ 98% of the entire CPU

Why they are taking up so much cpu time is immaterial if you want to fix it go to Preferences - Behavior and change Job priority from Normal to Low then restart calibre. This will ensure your computer is responsive to you and the calibre-parallel processes will cede to you when you want to do something other than stare at the computer.

Originally Posted by BetterRed

I'll try starting calibre via the start command with the affinity switch set at 2, theoretically that should limit calibre to 2 cores only

You can also control cpu cores via Preferences - Miscellaneous, but once you change Job priority you should be fine.

[Dopedangel]

Originally Posted by DoctorOhh

Why they are taking up so much cpu time is immaterial if you want to fix it go to Preferences - Behavior and change Job priority from Normal to Low then restart calibre. This will ensure your computer is responsive to you and the calibre-parallel processes will cede to you when you want to do something other than stare at the computer.



You can also control cpu cores via Preferences - Miscellaneous, but once you change Job priority you should be fine.

I could be wrong but it could be because the original source of the epub is word html. Word html leaves a lot of crap behind in the html coding and converting those also takes a lot longer than normal epubs. The count pages might be having a similar problem.

[DoctorOhh]

Originally Posted by Dopedangel

I could be wrong but it could be because the original source of the epub is word html. Word html leaves a lot of crap behind in the html coding and converting those also takes a lot longer than normal epubs. The count pages might be having a similar problem.

Seems like a reasonable assumption. I would have guessed the slowness was in the conversion before counting the words, but she said those books converted as fast, maybe faster than the ones where the count finished quickly. Still your idea makes the most sense to a layman such as myself.

The only thing I know for sure, changing Job priority from Normal to Low allows me the freedom to do whatever I want no matter how much work those calibre-parallel processes are doing.

[BetterRed]

Originally Posted by DoctorOhh

Why they are taking up so much cpu time is immaterial if you want to fix it go to Preferences - Behavior and change Job priority from Normal to Low then restart calibre. This will ensure your computer is responsive to you and the calibre-parallel processes will cede to you when you want to do something other than stare at the computer.



You can also control cpu cores via Preferences - Miscellaneous, but once you change Job priority you should be fine.

Pref->Behavior->Job Priority made no difference - a single secondary calibre-parallel process still ran the CPU at 22-25%, if I set the priority to Below Normal with Windows its also makes no difference. Will it yield if all four cores are at 24.n% usage - dunno, don't care - see next para.

I think I've found the root cause - I had Behaviour->Preferred Output set to EPUB and Preferred Input to RTF, because for my main workflow that's the stage I'm at i.e. I'm doing mainly doing RTF conversions to EPUB.

There's a lot of earlier discussion in this thread around this issue, i.e. which format is used to do the counting - I was left with the impression that the pecking order was EPUB, MOBI, otherwise ?? Given I have an EPUB and an RTF for everything, I thought it was using the EPUBs.

On a whim I pushed EPUB to the top of the Preferred Input order and the slow books become fast books and the fast books became even faster books.

Kiwidude mentioned putting a preferred source format list in the plug-in Configuration similar to the one in Behaviour. I'd settle for a being able to specify a single default format as the one to use to do the count - if its not available then... whatever is easiest for kiwidude.

Have to go, I'm in a bricks & mortar Library using their WiFi, its 18:30, and the real Librarian is rattling keys, and wagging her finger...

BR

[DoctorOhh]

Originally Posted by BetterRed

Pref->Behavior->Job Priority made no difference - a single secondary calibre-parallel process still ran the CPU at 22-25%, if I set the priority to Below Normal with Windows its also makes no difference. Will it yield if all four cores are at 24.n% usage - dunno, don't care - see next para.

I didn't mean to imply that the CPU wouldn't still run high. Calibre will use all the power you allow it. By changing from Normal to Low priority your computer will remain responsive and the calibre-parallel processes will ease back and let you work. You have to restart calibre for this to take effect.

Just for fun I kicked off 1000 books counting pages. It spawned 8 callibre-parallel processes and while they're running they use their max 12.5% per core. But since I have calibre set to Low priority it never gets in the way of me responding to this post.

Did you adjust the CPU items at Preferences - Miscellaneous? I don't mess with this but you can throttle things way back here.

Originally Posted by BetterRed

On a whim I pushed EPUB to the top of the Preferred Input order and the slow books become fast books and the fast books became even faster books.

Kiwidude mentioned putting a preferred source format list in the plug-in Configuration similar to the one in Behaviour. I'd settle for a being able to specify a single default format as the one to use to do the count - if its not available then... whatever is easiest for kiwidude.

I guess you found the universal place for selecting the preferred format.

Good Luck