Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 09-04-2012, 07:17 PM   #436
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@vargaviktor - I have no plans to change anything. I call two different calibre utility functions which handle zips and rars, both of which are in my understanding modified versions of some standard Python libraries.

As I said on the quality check thread averaging size/dpi is not an appropriate function for that or indeed this plugin - I suggest if you want such a feature that you write your own plugin for it. Undoubtedly there would be code in plugins like this one which could assist you.
kiwidude is offline   Reply With Quote
Old 09-07-2012, 03:12 AM   #437
vargaviktor
Member
vargaviktor began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Sep 2012
Device: Alcatel T60 Tablet (Android 2.x)
Quote:
Originally Posted by kiwidude View Post
@vargaviktor - I have no plans to change anything. I call two different calibre utility functions which handle zips and rars, both of which are in my understanding modified versions of some standard Python libraries.

As I said on the quality check thread averaging size/dpi is not an appropriate function for that or indeed this plugin - I suggest if you want such a feature that you write your own plugin for it. Undoubtedly there would be code in plugins like this one which could assist you.
OK, I Understand. The unzipping behaviour was just asked me, to mention you, maybe it can be a bug. Now, I know, this is the correct way of working.

I would like to give a try to make a plugin, but my python is not the best. Your plugin will help me, to concentrate on the calculations. Thank you.

regards. Viktor
vargaviktor is offline   Reply With Quote
Old 09-09-2012, 01:04 AM   #438
ElMiko
Addict
ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.
 
ElMiko's Avatar
 
Posts: 320
Karma: 56788
Join Date: Jun 2011
Device: Kindle
I'm getting a wordcount that is way off the expected value... by about 90%, would be my guess. Is there any way to determine what's causing hiccup?
ElMiko is offline   Reply With Quote
Old 09-09-2012, 04:42 AM   #439
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Remember that word count is for every single file in the ePub that is in the manifest. Frequently books include chapters of upcoming releases and all sorts of other cruft. Use Tweak ePub, open each HTML page in a browser, paste into word and get it to do a word count if you don't believe it. Just don't cheat and skip any pages...
kiwidude is offline   Reply With Quote
Old 09-10-2012, 10:32 AM   #440
ElMiko
Addict
ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.
 
ElMiko's Avatar
 
Posts: 320
Karma: 56788
Join Date: Jun 2011
Device: Kindle
Quote:
Originally Posted by kiwidude View Post
Remember that word count is for every single file in the ePub that is in the manifest. Frequently books include chapters of upcoming releases and all sorts of other cruft. Use Tweak ePub, open each HTML page in a browser, paste into word and get it to do a word count if you don't believe it. Just don't cheat and skip any pages...
Oops, sorry, I wasn't clear. The issue is that it is UNDERcounting. It's giving a wordcount of around 9k for a book that's probably closer to 100k...
ElMiko is offline   Reply With Quote
Old 09-10-2012, 11:55 AM   #441
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@ElMiko - the same response applies regardless. I'm not sure what you want expect anyone to tell you. I can't do anything without seeing the book for myself, and who "says" the number is supposed to be 100K? The plugin logic is as I described above, and I've never seen any counts which are widely anomalous to the number of pages (from Goodreads) for the book. Perhaps your copy of the book is corrupted.
kiwidude is offline   Reply With Quote
Old 09-10-2012, 02:44 PM   #442
ElMiko
Addict
ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.
 
ElMiko's Avatar
 
Posts: 320
Karma: 56788
Join Date: Jun 2011
Device: Kindle
@kiwidude - I did as you suggested and copy/pasted the text into a word document. In so doing, it seems I may discovered what the problem is, albeit not the solution. There are four html files in the MOBI file, and Count Pages isn't counting the first file.

Last edited by ElMiko; 09-10-2012 at 08:24 PM.
ElMiko is offline   Reply With Quote
Old 09-10-2012, 05:42 PM   #443
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
That's because it is invalid html. Look at the closing body tag immediately after the opening body tag and all the content is after the body.
kiwidude is offline   Reply With Quote
Old 09-10-2012, 08:25 PM   #444
ElMiko
Addict
ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.ElMiko actually enjoys Vogon poetry.
 
ElMiko's Avatar
 
Posts: 320
Karma: 56788
Join Date: Jun 2011
Device: Kindle
Thanks! Ran page count on an EPUB conversion and that did the trick. Really appreciate your looking into it!
ElMiko is offline   Reply With Quote
Old 09-19-2012, 02:03 AM   #445
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,477
Karma: 26944418
Join Date: Mar 2012
Location: Sydney Australia
Device: none
I'll make this as detailed as I can, I'm using remote control over a 64Kb/s satellite link with long latencies so I want to avoid to-ing & fro-ing with questions and answers.

I use Windows 7 64 bit with all updates installed, Calibre 0.8.69 and Count Pages 1.6.3 - My settings for the plug-in are attached.

I use Process Explorer to observe what programs are doing.

Some of my books are taking a long time to word count (15 minutes and more), although word counts of similar books of similar size, by the same author, in the same format, from the same source take a few seconds

First what happens when I count words in a single 'slow' book.

Whilst the plugin is counting a 'slow' book's words, one of the two instances of calibre-parallel uses 24.n% of my quad core I5 - i.e almost an entire core. My observation is that the main calibre process spawns a calibre-parallel process which spawns another calibre-parallel process. It's the second calibre-parallel that chews up processor resources, so I assume its the one doing the 'work'. This situation doesn't bother me too much - it gets there eventually

Now what happens when I select a group of books.

I can run into the situation where the group includes several of these 'slow' books. The plug-in spawns up to 5 instances of calibre-parallel. When this happens my computer can become unusable for an unacceptable period (my patience ran out after 33 minutes); because the 4 calibre-parallel secondary processes are EACH consuming 24.n% of a CPU core processing 'slow' books, totaling ~98% of the entire CPU

If I fight with the sloth like mouse to stop the job, that doesn't give me back the CPU resources. Because whilst the primary instance of calibre-parallel dies, the four that were doing the work get detached from the main calibre process, and continue working independently and continue to hog the CPU, so I have to kill them individually with Process Explorer.:

So rather than killing the job from calibre, its faster & easier to kill the process tree of the primary calibre-parallel process with Process Explorer. Calibre complains, but it doesn't crash and there's no apparent harm done; i.e. the books and database are OK because the plug-in is not accessing them when its doing the counting.

What have I done to try to 'fix' it

Based on reading this thread I progressively disabled DEP (at the command line so it was disabled for everything including Windows), disabled my AV, disabled the firewall, disconnected the router, closed all other programs (including disabling the ones that start in the tray), and restarted Windows between each, the final one into Safe Mode - all to no avail.

I haven't tried Closing the Tag Browser, which I saw suggested in this thread and I've seen suggested elsewhere as a possible solution to various problems - because I can't figure out how to do that.

There's no sign of any memory leaks . And the 'slow books' are in the minority, estimate < 10%.

Changing the algo between 'ADE' & 'calibre E-book' makes no obvious difference to the speed. Didn't try the 'APNX ' algo because it seems to be about page counting.

Is there some way to have this plug-in work in a serial manner rather than multi-tasking via spawning multiple calibre-parallel processes, i.e. limit the number of secondary instances of calibre-parallel to ONE.

It would 'help' if I could determine the identity (author-title) of book(s) currently being processed in a job, or books that have completed. Then I could kill the job, put the 'slow books' aside to be done one at a time, and redo the group without the slow books.

My limited tests show that a 'slow to count' book does not appear to take significantly longer to convert to & from EPUB and RTF than similar 'fast to count' books. In fact 'slow book' conversion is usually (always ?) faster 'slow book' word counting, but the opposite is true of a 'fast book'

I have no doubt that its something to do with the book content, but I've no idea what - given that similar books are OK, and other plugs-ins are not slow in their processing of the same books

I'll try starting calibre via the start command with the affinity switch set at 2, theoretically that should limit calibre to 2 cores only

BR
Attached Thumbnails
Click image for larger version

Name:	Capture.JPG
Views:	291
Size:	72.4 KB
ID:	92576  

Last edited by BetterRed; 09-19-2012 at 03:06 AM. Reason: forgot the attachment & para re algo's
BetterRed is offline   Reply With Quote
Old 09-19-2012, 02:59 AM   #446
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by BetterRed View Post
I can run into the situation where the group includes several of these 'slow' books. The plug-in spawns up to 5 instances of calibre-parallel. When this happens my computer becomes practically unusable for an unacceptable period (my patience ran out after 33 minutes); because 4 of the calibre-parallel processes are EACH consuming 24.n% of a CPU core, totalling ~ 98% of the entire CPU
Why they are taking up so much cpu time is immaterial if you want to fix it go to Preferences - Behavior and change Job priority from Normal to Low then restart calibre. This will ensure your computer is responsive to you and the calibre-parallel processes will cede to you when you want to do something other than stare at the computer.

Quote:
Originally Posted by BetterRed View Post
I'll try starting calibre via the start command with the affinity switch set at 2, theoretically that should limit calibre to 2 cores only
You can also control cpu cores via Preferences - Miscellaneous, but once you change Job priority you should be fine.
DoctorOhh is offline   Reply With Quote
Old 09-19-2012, 03:09 AM   #447
Dopedangel
Wizard
Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.
 
Dopedangel's Avatar
 
Posts: 1,759
Karma: 30063305
Join Date: Dec 2006
Location: Singapore
Device: Boyue
Quote:
Originally Posted by DoctorOhh View Post
Why they are taking up so much cpu time is immaterial if you want to fix it go to Preferences - Behavior and change Job priority from Normal to Low then restart calibre. This will ensure your computer is responsive to you and the calibre-parallel processes will cede to you when you want to do something other than stare at the computer.



You can also control cpu cores via Preferences - Miscellaneous, but once you change Job priority you should be fine.
I could be wrong but it could be because the original source of the epub is word html. Word html leaves a lot of crap behind in the html coding and converting those also takes a lot longer than normal epubs. The count pages might be having a similar problem.
Dopedangel is offline   Reply With Quote
Old 09-19-2012, 03:20 AM   #448
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by Dopedangel View Post
I could be wrong but it could be because the original source of the epub is word html. Word html leaves a lot of crap behind in the html coding and converting those also takes a lot longer than normal epubs. The count pages might be having a similar problem.
Seems like a reasonable assumption. I would have guessed the slowness was in the conversion before counting the words, but she said those books converted as fast, maybe faster than the ones where the count finished quickly. Still your idea makes the most sense to a layman such as myself.

The only thing I know for sure, changing Job priority from Normal to Low allows me the freedom to do whatever I want no matter how much work those calibre-parallel processes are doing.
DoctorOhh is offline   Reply With Quote
Old 09-19-2012, 04:31 AM   #449
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,477
Karma: 26944418
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by DoctorOhh View Post
Why they are taking up so much cpu time is immaterial if you want to fix it go to Preferences - Behavior and change Job priority from Normal to Low then restart calibre. This will ensure your computer is responsive to you and the calibre-parallel processes will cede to you when you want to do something other than stare at the computer.



You can also control cpu cores via Preferences - Miscellaneous, but once you change Job priority you should be fine.
Pref->Behavior->Job Priority made no difference - a single secondary calibre-parallel process still ran the CPU at 22-25%, if I set the priority to Below Normal with Windows its also makes no difference. Will it yield if all four cores are at 24.n% usage - dunno, don't care - see next para.

I think I've found the root cause - I had Behaviour->Preferred Output set to EPUB and Preferred Input to RTF, because for my main workflow that's the stage I'm at i.e. I'm doing mainly doing RTF conversions to EPUB.

There's a lot of earlier discussion in this thread around this issue, i.e. which format is used to do the counting - I was left with the impression that the pecking order was EPUB, MOBI, otherwise ?? Given I have an EPUB and an RTF for everything, I thought it was using the EPUBs.

On a whim I pushed EPUB to the top of the Preferred Input order and the slow books become fast books and the fast books became even faster books.

Kiwidude mentioned putting a preferred source format list in the plug-in Configuration similar to the one in Behaviour. I'd settle for a being able to specify a single default format as the one to use to do the count - if its not available then... whatever is easiest for kiwidude.

Have to go, I'm in a bricks & mortar Library using their WiFi, its 18:30, and the real Librarian is rattling keys, and wagging her finger...

BR

Last edited by BetterRed; 09-19-2012 at 04:34 AM.
BetterRed is offline   Reply With Quote
Old 09-19-2012, 04:55 AM   #450
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by BetterRed View Post
Pref->Behavior->Job Priority made no difference - a single secondary calibre-parallel process still ran the CPU at 22-25%, if I set the priority to Below Normal with Windows its also makes no difference. Will it yield if all four cores are at 24.n% usage - dunno, don't care - see next para.
I didn't mean to imply that the CPU wouldn't still run high. Calibre will use all the power you allow it. By changing from Normal to Low priority your computer will remain responsive and the calibre-parallel processes will ease back and let you work. You have to restart calibre for this to take effect.

Just for fun I kicked off 1000 books counting pages. It spawned 8 callibre-parallel processes and while they're running they use their max 12.5% per core. But since I have calibre set to Low priority it never gets in the way of me responding to this post.

Did you adjust the CPU items at Preferences - Miscellaneous? I don't mess with this but you can throttle things way back here.

Quote:
Originally Posted by BetterRed View Post
On a whim I pushed EPUB to the top of the Preferred Input order and the slow books become fast books and the fast books became even faster books.

Kiwidude mentioned putting a preferred source format list in the plug-in Configuration similar to the one in Behaviour. I'd settle for a being able to specify a single default format as the one to use to do the count - if its not available then... whatever is easiest for kiwidude.
I guess you found the universal place for selecting the preferred format.

Good Luck

Last edited by DoctorOhh; 09-19-2012 at 04:59 AM.
DoctorOhh is offline   Reply With Quote
Reply

Tags
count, count pages, page count, pages, plugin

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Open With kiwidude Plugins 403 Yesterday 08:39 AM
[GUI Plugin] Quality Check kiwidude Plugins 1171 03-23-2024 05:18 AM
[GUI Plugin] Quick Preferences kiwidude Plugins 62 03-16-2024 11:47 PM
[GUI Plugin] Kindle Collections (old) meme Plugins 2070 08-11-2014 12:02 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 06:07 AM.


MobileRead.com is a privately owned, operated and funded community.