[GUI Plugin] Count Pages

[davidfor]

Originally Posted by theducks

Use Current Preferred output format: (<show the value as a hint only>)

Standard Calibre usage locks in the then current settings
eg Convert


BTW my Library used ADE page counts historically (Astak PEz).

I now use my K4NT
CP won't count using ADE when only a AZW3 exists
I really don't want to recalc all those (besides, changes the Last Modified flag)

A count method sb just an algorithm to use (assumes the filoe can be processed) If I want to count a Kindle Book using ADE rules, why not?

The problem is the algorithm. The ADE page count uses the compressed size of the internal text files. It doesn't matter what is in the files, it just divides the compresses size by 1024 to get the number of pages for each file and then adds them up. There are all sorts of things wrong with that as an algorithm, but that's the way it works. A simple one is that if you use a different compression level, you get a different page count.

I suppose the same could be done for AZW3, but, I have no idea of the internal format of the format, so I don't know if it is constructed in a suitable way. Plus, whatever compression is used will affect it.

I'll put it on the list to look at, but I can't promise anything.

[JSWolf]

Originally Posted by davidfor

That doesn't make sense. There is no ICU method for counting ADE pages and the only overlapping code between the word count and the ADE page count is opening the epub for counting.

But, assuming you meant there was a big difference between the non-ICU word count with the beta and the word count with the released version, I need examples. And a definition of "really incorrect". When I compared the two, the difference is the number of files in the book. That is expected with the fix I put into the code for the word count.

If the difference is bigger than that, then I need actual examples I can look at including the counts you are seeing with the two versions.

Under Word count options there is a check for Use ICU algorithm for counting words.

ICU on

Page count: 255.0
Word count using icu_wordcount - trying to count_words
Word count - used count_words: 94052
Word count: 94052

ICU off

Page count: 255.0
Word count using older method - trying to count_words
Word count: 96049

I think that's a pretty big difference. I've scrambled the ePub so I can attach it.

[BetterRed]

@JSWolf - maybe hyphenations are counted as one word by ICU and multiple words by non ICU.

Select Tools->Reports->Words in the calibre book editor, filter with '-', save to csv, and get your spreadsheet to accumulate the Times Used column; note bene: if my suggestion is true then forget-me-not would be counted as three words by non-ICU

BR

[davidfor]

Originally Posted by JSWolf

Under Word count options there is a check for Use ICU algorithm for counting words.

ICU on

Page count: 255.0
Word count using icu_wordcount - trying to count_words
Word count - used count_words: 94052
Word count: 94052

ICU off

Page count: 255.0
Word count using older method - trying to count_words
Word count: 96049

I think that's a pretty big difference. I can use the scramble plugin to scramble it and then attach it if you'd like.

Jon: I never would have worked out from your original post that you were talking about the difference between the ICU based word count and the original algorithm.

In any case, go back to the discussion in this thread at this time last year. You were the one that started that "discussion" by pointing out a possible bug. And that was the point of adding the ICU method as it is seemed to handle some things in a better way and was language aware. Back then, I did post explanations of some of the differences if you want to look.

Also, both methods rely on code in calibre. If that is updated, then it might change the count the plugin produces.

Personally, I expect both numbers to be wrong. I tend to think the ICU method is the more accurate, but that is based on me counting very small samples. I take all the statistics produced by the plugin as approximations. And during last year's discussion I was very tempted to introduce a "nearest 1000" option. Of course, that would raise the argument of rounding vs truncating.

[JSWolf]

The ICU count is more accurate then the non-ICU method. The ICU gets closer to the correct word count. The difference between the word count produced with Word 2016 when the ePub is converted to RTF is 36 difference. Not a lot and not enough to be bothered with. I've scrambled and posted the ePub in the message with the counts if you are interested in seeing it.

[davidfor]

Originally Posted by BetterRed

@JSWolf - maybe hyphenations are counted as one word by ICU and multiple words by non ICU.

Select Tools->Reports->Words in the calibre book editor, filter with '-', save to csv, and get your spreadsheet to accumulate the Times Used column; note bene: if my suggestion is true then forget-me-not would be counted as three words by non-ICU

Yes, that's one of the differences. The also handle the other characters that could be used as hyphens differently as well.

And for reference, the editor is using the ICU algorithm. There is a difference in the counts between the plugin and the editor. I haven't gotten around to looking at what the difference is yet.

[JSWolf]

I have it... Convert to text, count the words, delete the text version and done.

[davidfor]

Originally Posted by JSWolf

I have it... Convert to text, count the words, delete the text version and done.

Go ahead. Then show me the differences so I can decide. And don't forget to tell me what you consider a word. The delimiters chosen are part of the differences in the two algorithms we are talking about. As I said, I mentioned some of this last year during the discussion.

As I said, I take either count as an approximation. Until someone demonstrates that one or the other is wrong, and how, I am going to accept that they work.

[BetterRed]

Originally Posted by davidfor

Yes, that's one of the differences. The also handle the other characters that could be used as hyphens differently as well.

And for reference, the editor is using the ICU algorithm. There is a difference in the counts between the plugin and the editor. I haven't gotten around to looking at what the difference is yet.

The editor includes the content.opf in it's counts and in its spell checking. Try typing 'rubbbbish' into a tag, and 'aaardvaaark' into a comment, then Polish the book with update metadata set and open it in the editor and have a look.

I often remove description, subjects etc from the content.opf (with Sigil) before I use calibre's spell checker, why calibre's spell-checker and not Sigil's - because it's multi-lingual.

I would prefer that the content.opf file not be included in the PI's calculations. I'd also quite like some front and back matter to be excluded, but that's a much bigger ask.

BR

[theducks]

Originally Posted by davidfor

I'll put it on the list to look at, but I can't promise anything.

(If it had been simple, Kiwidude would have probably done it )

[davidfor]

Originally Posted by BetterRed

The editor includes the content.opf in it's counts and in its spell checking. Try typing 'rubbbbish' into a tag, and 'aaardvaaark' into a comment, then Polish the book with update metadata set and open it in the editor and have a look.

Yes your right. I happen to have a book with a single word in the text. The plugin counts one (both methods!) but the editor shows 141. It is also counting the words in the NCX and title tags in each of the files. I suppose it makes sense for the editor as it is running the spelling check on all of these. The plugin doesn't touch these, only the words with the body tags.

I often remove description, subjects etc from the content.opf (with Sigil) before I use calibre's spell checker, why calibre's spell-checker and not Sigil's - because it's multi-lingual.

Yes, I like the multilingual spelling. I will sometimes add the lang attribute to a tag to reduce the spelling errors shown when there is non-English dialog.

I would prefer that the content.opf file not be included in the PI's calculations. I'd also quite like some front and back matter to be excluded, but that's a much bigger ask.

No, the plugin is looking at the text of the book, not the OPF or any of the metadata objects. Excluding the front and back matter would be good, but trying to decide what is back and front matter is not something I am very interested in doing.

[BetterRed]

Originally Posted by davidfor

Excluding the front and back matter would be good, but trying to decide what is back and front matter is not something I am very interested in doing.

As I said "a much bigger ask"

IIRC someone wrote in the earlier discussion, "The only time word count accuracy matters is if someone is paying for or being paid for the words."

BR

[davidfor]

OK, here's another beta with the option that BR asked for. The text for the label and tooltip is somewhere between what I had and BR had.

Other than fine-tuning the labels and tooltips or if someone finds a bug, I'm planning for this to be the last beta.

[BetterRed]

@davidfor - works for me, and as I hoped (knew) it would be, it's a heck of lot faster now it's not redoing the conversion.

Thanks again.

BR

I don't think I'll ever understand why it didn't work this way from the getGo.

[JSWolf]

Originally Posted by davidfor

Go ahead. Then show me the differences so I can decide. And don't forget to tell me what you consider a word. The delimiters chosen are part of the differences in the two algorithms we are talking about. As I said, I mentioned some of this last year during the discussion.

As I said, I take either count as an approximation. Until someone demonstrates that one or the other is wrong, and how, I am going to accept that they work.

I used Calibre to convert to text. I loaded the text file into Notepad++ and used the Word Count function of the TextFX plugin and I get a count of 94750.

I will leave it up to you to say if this difference is enough to warrant any more changes.