Word use frequency - Is there a programme?

[Nabeel]

Dear All,

A question came up in my Creative Writing group: there must be a computer programme or app that, when given a text, will analyse how frequently different words are used.

Obviously, the really useful thing would be a programme that points out that you have unwittingly used a word like 'vast' five times in the same paragraph, but we're not looking for miracles.

Any advice or information would be very useful,

Nabeel

[arjaybe]

Calibre can do that. Select the book and edit it. Under Tools, select Reports.

Maybe there's an easier way, but ...

[theducks]

Spell check in Sigil or Calibre editor: Show all words, also gives a count for words of more than 1 letter.

[gmw]

I expect most editing software will do this, I know that Editor from Serenity Software showed both word and phrase frequency counts (the latter can be rather interesting). Please note that I only ever trialled v4 of that software; I don't know what has happened in v5, so this is not a recommendation, just a suggestion of where to look.

A few posts on page 2 of this thread talk about editing software (with links).

[skb]

I think Scrivener has a word frequency feature.

[Nabeel]

Much thanks for this useful advice. Calibre is amazing!

But...

When I run a spell check, it's obvious that Calibre is only looking at a part of the text: in this case, 8,000 words of a 73,000 word text. How do I get the edit function to look at the whole text?

Nabeel

[Nabeel]

Spoke too soon. The answer, obviously, is to untick 'show only mispelled words'.

Nabeel

[Araucaria]

8,000 is a lot to be wrong. Perhaps a case of Jasper Fforde's mispeling vyrus?

[arjaybe]

Originally Posted by Araucaria

8,000 is a lot to be wrong. Perhaps a case of Jasper Fforde's mispeling vyrus?

It might just be the wrong dictionary.

[Gregg Bell]

This is what you want. https://www.online-utility.org/text/analyzer.jsp

[Nabeel]

Thank you Greg! Calibre doesn't do the job badly, but the site you recommend is more sophisticated.

Nabeel

[evanhson357]

Wow! Thanks! I didn't know Calibre could do that.

[BetterRed]

And, there's a English Noun Frequency plugin for calibre.

BR

[evanhson357]

Thanks again.
There's a bunch of good info here. Anything to make the job easier.

[Tex2002ans]

Originally Posted by Nabeel

A question came up in my Creative Writing group: there must be a computer programme or app that, when given a text, will analyse how frequently different words are used.

As others have stated, a user-friendly way for single words is to use Sigil or Calibre's Spellcheck lists. If you use Word, Toxaris's EPUB Tools also has this built in.

If you want to get "repeated phrases", that's called n-grams. Gregg Bell linked to one such tool, but there are plenty others.

Side Note: I personally use a commandline tool for n-grams. This gives me full control over the variables. Then I import it into a spreadsheet so I can sort by frequency.

This is an example of an n-gram example with an n-gram example.

2-grams would be all 2 words in a row:

1 This is
2 an n-gram
1 is an
1 an example
1 example of
1 of an
2 n-gram example
1 example with
1 with an

3-grams:

1 This is an
1 is an example
[...]
2 an n-gram example
[...]

Typically smaller n-grams are so full of cruft, they aren't really helpful ("he said" + "she said"). But I find the helpful patterns start to pop out at 4-grams and higher.

When you run this on a book-length text, you tend to see the author's own writing patterns.

I recently ran this on a ~70k word novel, and there were 26 "XYZ took a deep breath and" and 34 "XYZ shook her head". That's 292 words of characters taking a deep breath and shaking their heads.

Or a different author had the tendency to write "she said with an evil smirk on her face", "she said with a smile". So that author would probably want to go through and focus on chopping down "she said with".

A different book had 15 "What the f*** do you think you are doing?" That's 9 * 15 = 135 words.

These are typically a sign that you have to go through your book again and spice it up with variations.

Nobody wants to read hundreds of the same exact words again and again and again. Or slight variations of the words again and again... and again.

Originally Posted by Nabeel

Obviously, the really useful thing would be a programme that points out that you have unwittingly used a word like 'vast' five times in the same paragraph, but we're not looking for miracles.

The only tool I've come across that does this is TeXStudio:

https://www.texstudio.org/

It is a LaTeX editor, but you could use it for plain text if you wanted to.

It has a function called "Word Repetition":

https://tex.stackexchange.com/questi...from-texstudio

What it does is gives you a little green squiggly for the same word repeated within X number of words (you can set the min/max variables).

It tends to gives a lot of false positives though. One of the ways it could be made better would be if you could have some sort of whitelist, so you could ignore very common words ("the" + "if" + "and" + "but" + [...]).