[GUI Plugin] Count Pages

[ownedbycats]

Originally Posted by kiwidude

Well I would hope so

Previously the plugin had a crude approach of extracting the html body content and stripping html tags using the raw text and regular expressions. Which for a rough estimate is usually perfectly fine.

I suspect this did not work as expected on "HTML for Dummies"

[DNSB]

Originally Posted by ownedbycats

I suspect this did not work as expected on "HTML for Dummies"

Wait till you do word and page counts on a ePub with Base64 encoded images.

[JSWolf]

Originally Posted by kiwidude

Well I would hope so

Previously the plugin had a crude approach of extracting the html body content and stripping html tags using the raw text and regular expressions. Which for a rough estimate is usually perfectly fine.

However as with any shortcuts a crude approach creates edge cases where you can get outlier results. In this case the problem is the encoding of html entities - one book (or even two formats of the same book) might have text like "it's" and the other has "it's". Previously I was not "decoding" the first case, so if you were counting characters in the body obviously the number is larger, and if counting words then the way word boundaries were defined it might count as two words rather than one based on the semi-colon being in there.

Now however I am using the BeautifulSoup parser to process the html and then asking it to give me the body text. Because it can be told to decode all those entities (and strip out all the other meaningless html tags like spans etc) so I have a consistent starting point of the second example text to calculate page and word counts for.

Provided BeautifulSoup is given fairly well formed html it should indeed all work fine. Give it bad html books and it will probably give terrible results - but then if the html is that awful then many ereaders might struggle with it too...

Wouldn't converting the eBook to text and then counting the words give a fairly accurate word count and in most cases an exact word count?

[ownedbycats]

Originally Posted by JSWolf

Wouldn't converting the eBook to text and then counting the words give a fairly accurate word count and in most cases an exact word count?

eBooks are made up of... HTML files...

[JSWolf]

Originally Posted by ownedbycats

eBooks are made up of... HTML files...

And when you convert them to text, you get rid of all of the HTML code. Plus, you convert the entities to symbols. Then you count the words and there you go.

[DNSB]

I've attached an epub with what the Adobe SPN algorithm says contains 27 words and 3,115 pages.

A pathological case.

Personally, I find Count Pages to be good enough if not perfect.

[kiwidude]

Originally Posted by JSWolf

And when you convert them to text, you get rid of all of the HTML code. Plus, you convert the entities to symbols. Then you count the words and there you go.

The plugin would be orders of magnitude slower, with all the additional disk activity, temp files etc. It won't be changing to doing that

Originally Posted by DNSB

I've attached an epub with what the Adobe SPN algorithm says contains 27 words and 3,115 pages.

A pathological case.

Personally, I find Count Pages to be good enough if not perfect.

Therein lies the weakness in the simplistic approach of the ADE algorithm - inlined Base64 images result in enormous html file sizes which is what ADE calculates page estimates based on. And also why I don't use it personally myself - because unless you are in the habit of opening every file to inspect whether it has been edited this way (rather than external image files via links) you would never know if it is a genuinely big book or from this flaw in approach.

[JSWolf]

Originally Posted by DNSB

I've attached an epub with what the Adobe SPN algorithm says contains 27 words and 3,115 pages.

A pathological case.

Personally, I find Count Pages to be good enough if not perfect.

I only count 15 words. Where are the other 12 words? I converted this to text and it is only 15 words.

I'll have to try the older version of Count Pages on my Surface and see what result I get. I'd be very interested if the older version gives a more accurate number of words.

[ownedbycats]

Originally Posted by JSWolf

I only count 15 words. Where are the other 12 words? I converted this to text and it is only 15 words..

If you open in ebook-editor, there's a few h3s that are invisible due to CSS. 12 words there.

[DNSB]

Originally Posted by kiwidude

Therein lies the weakness in the simplistic approach of the ADE algorithm - inlined Base64 images result in enormous html file sizes which is what ADE calculates page estimates based on. And also why I don't use it personally myself - because unless you are in the habit of opening every file to inspect whether it has been edited this way (rather than external image files via links) you would never know if it is a genuinely big book or from this flaw in approach.

I've only found Base64 encoded images in 2 ePubs in the wild. For both, I converted them back to image files. In both cases, the issues became very noticeable just from the number of pages reported. The sample ePub I attached was generated during a discussion @davidfor and I had quite a while back about some claims that Base64 encoded images rendered faster though I did modify the posted file to leave only the Base64 encoded image pages.

[JSWolf]

Originally Posted by ownedbycats

If you open in ebook-editor, there's a few h3s that are invisible due to CSS. 12 words there.

But they don't count as they are not displayed. So my idea of converting to text and then counting is most accurate.

[DNSB]

Originally Posted by JSWolf

But they don't count as they are not displayed. So my idea of converting to text and then counting is most accurate.

If you convert to text, they would no longer be hidden and would be counted.

I somehow doubt that anyone is going to start looking for display: none, height:0 and whatever other methods of hiding text can be used, excluding words in the ePub 3 navigation document even when it is visible, etc. since that way lies extreme slowness.

[JSWolf]

Originally Posted by DNSB

If you convert to text, they would no longer be hidden and would be counted.

I somehow doubt that anyone is going to start looking for display: none, height:0 and whatever other methods of hiding text can be used, excluding words in the ePub 3 navigation document even when it is visible, etc. since that way lies extreme slowness.

calibre did not convert the hidden text. Thus, only 15 words are ever displayed.

[DNSB]

Originally Posted by JSWolf

calibre did not convert the hidden text. Thus, only 15 words are ever displayed.

Now we are trying to add calibre conversion into the mix? Might have been nice of you to mention that earlier. Have you never noticed that calibre's conversion to txt seems to lose text in headers that are display: none?

OTOH, did you count the words in my original epub with 15 words in the nav.xhtml and the other 12 words in the hidden headers? Again, please note that when you remove the html tags, all 27 words are visible.

Perhaps knocking off the what-if-isms would make it easier for you to communicate with others.

Anyhow, enough said on this off-topic topic. If you wish to continue this, take it to a vent and rant thread.

[ownedbycats]

Originally Posted by DNSB

Now we are trying to add calibre conversion into the mix? Might have been nice of you to mention that earlier.

He keeps claiming that converting to text gets a more accurate wordcount. Nevermind the impracticality of it.