Word Count

[ldolse]

There is a word count function embedded inside Calibre, currently used internally for heuristics. With the new plugin framework it should be simple to create a user interface plugin for this. What would people want done with the information? I'm thinking populate a user configurable column might be the way to go.

[chaley]

Originally Posted by ldolse

There is a word count function embedded inside Calibre, currently used internally for heuristics. With the new plugin framework it should be simple to create a user interface plugin for this. What would people want done with the information? I'm thinking populate a user configurable column might be the way to go.

Unless the value is precomputed for each book (book, not format), a custom column would be a performance catastrophe. Reason: touching any custom column for a book record causes the computation of them all. Sorting on something would compute the word count for all books in the library. Doing that for 10,000 books could take many minutes.

[CCarrot]

Originally Posted by chaley

Unless the value is precomputed for each book (book, not format), a custom column would be a performance catastrophe. Reason: touching any custom column for a book record causes the computation of them all. Sorting on something would compute the word count for all books in the library. Doing that for 10,000 books could take many minutes.

Yeeps, that's not good. Good brainstorming, though, thanks Idolse and chaley!

Is there any way to add a metadata field to hold the page or word count, then have it populate on either an add or a database check? It would extend the time for a db check, but that takes some time for 10,000 books anyways...and I wouldn't think it would add too much overhead to the add function, since people aren't likely to try adding that many books at once.

I'm not sure if adding a metadata field to a book is available to the average user, though, or if it would require a plugin. That's outside my expertise, i'm afraid...

[Manichean]

Originally Posted by CCarrot

I'm not sure if adding a metadata field to a book is available to the average user, though, or if it would require a plugin. That's outside my expertise, i'm afraid...

Custom column = custom book metadata field. Thus, the same performance consideration would apply, unless the plugin calculated the number once and stored it in a numerical column. That way, the calculation would be user initiated only.

[ldolse]

Originally Posted by chaley

Unless the value is precomputed for each book (book, not format), a custom column would be a performance catastrophe. Reason: touching any custom column for a book record causes the computation of them all. Sorting on something would compute the word count for all books in the library. Doing that for 10,000 books could take many minutes.

I was thinking the plugin would work this way:

1. User highlights a list of books, selects 'Calculate Word Count'. (or whatever it's called)
2. Grab of 'one' of the books formats (probably use the preferred format list to choose the best one)
3. Get the word count, compute it, and populate the custom field/column (field chosen in plugin config).

In other words the user would need to manually populate the field by using the plugin - this would be the equivalent of what you mean by pre-computed, correct?

If the user did select several thousand books and ask to compute word count for them all this would indeed be a hit of many minutes, but I think it would be one time this way, right?

My problem might have been my use of the term 'custom column', not sure if that means a column that's dynamically populated to some people.

[chaley]

@idolse: yes, that is what I mean by precomputed. If the plugin populates a integer (or float?) custom column, then the performance penalty during search & sort is close to zero.

As for how often things are computed: certainly doing a word count on thousands of books will take a while. You might consider storing the date the format was last changed (database2.format_last_modified) in plugin storage (use the per-book persistent data feature). You could then provide an 'update' function that compares the stored date with the format's current date, and recompute the count only if the format has changed.

You might also consider working through the formats in some order until you find one that is not DRM infested.

[ldolse]

Originally Posted by chaley

@idolse: yes, that is what I mean by precomputed. If the plugin populates a integer (or float?) custom column, then the performance penalty during search & sort is close to zero.

As for how often things are computed: certainly doing a word count on thousands of books will take a while. You might consider storing the date the format was last changed (database2.format_last_modified) in plugin storage (use the per-book persistent data feature). You could then provide an 'update' function that compares the stored date with the format's current date, and recompute the count only if the format has changed.

You might also consider working through the formats in some order until you find one that is not DRM infested.

Cool, sounds like a plan then. I'll start simple and if it seems popular I'll look into those additional niceties.

[theducks]

My 2 cents
What don't you count when you calculate the word count.
Front matter(copyright, books by author...)? Teasers for other books? Pages of 'Rave reviews'? Sample chapters? Glossary? Dramatis Persona?

I am trying to think of all the other stuff I frequently see other than the Story (Forward through Epilogue)

I guess I never worry about word count.
When I get to 'The End', I am done

[ldolse]

Originally Posted by theducks

My 2 cents
What don't you count when you calculate the word count.
Front matter(copyright, books by author...)? Teasers for other books? Pages of 'Rave reviews'? Sample chapters? Glossary? Dramatis Persona?

I am trying to think of all the other stuff I frequently see other than the Story (Forward through Epilogue)

I guess I never worry about word count.
When I get to 'The End', I am done

I wasn't thinking about trying to get fancy and detect where the book actually starts. The way I do it in heuristics is just delete all the html tags with a regex (potentially error prone but faster than proper text conversion). Then I just count everything that's left. The way I'm thinking to do this is basically identical to the extract ISBN plugin's mechanism, so the regex aproach would actually be required, as the function the plugin relies on creates a book preview that's incompatible with the conversion engine.

[CCarrot]

Originally Posted by ldolse

The way I do it in heuristics is just delete all the html tags with a regex (potentially error prone but faster than proper text conversion). Then I just count everything that's left.

That sounds like a sensible way to go about it. I know I'm not looking for an *exact* word count, just an idea of proportionally how 'long' the book is compared to other books (or short stories) in the collection. File size is misleading in this respect, since it includes the graphics which, while nice to have, don't impact the 'reading length' of the book.

If we had access to a stored word count for each book, that could be converted to approximate page count easily by dividing by 200 (or 500 or whatever the typical number of words on a page is, idk).

One quick question though: would the calculated word counts be persistent, or would they have to be re-calculated (at the user's request of course) each time the program starts up?

Thanks!
cc

[ldolse]

It would persist as part of the books metadata.

[CCarrot]

Originally Posted by ldolse

It would persist as part of the books metadata.

:

[kiwidude]

@Idolse - if you are going to write this you might want to check with user_none as per this post since it sounds like both functions could be rolled into one.

[ldolse]

Originally Posted by kiwidude

@Idolse - if you are going to write this you might want to check with user_none as per this post since it sounds like both functions could be rolled into one.

I'll check with user_none before I go to town on it then - I'm sort of thinking this could grow into a general statistics plugin with a number of stats that could be gathered.

[cybmole]

MS word used to compute a readability factor ( maybe still does but I've not looked). based IIRC on stuff like number of words per sentence.
anyways that stats generated by Word ( & how it computes them) is a possible starting point for compiling a wish list of useful? book info