Indexing all words in an ebook

[DNSB]

I have one acquaintance who wants to have a mass of words in his magnum opus to be in an index with the page location of each word and the page location linked back to the word. He also wants this to show in an azw3/kfx Kindle format conversion.

You would see a word index (I can't bring myself to call it a concordance) where clicking on the page number e.g. 76 would move to that page in ebook preferably with the cursor on the word aardvark.

aardvark pg. 1, pg. 14, pg. 49, pg. 76
syzygy pg. 21, pg 48, pg. 103

My suggestion was that he forget the idea since a reflowable ebook does not have fixed page numbers and creating the index he wants would take an incredible amount of work for no real gain.

His argument is that there must be an automated solution to doing this task pointing at AntConc 3.5.8 as an example, the other example he pointed out was written in Fortran and has not been maintained since the 80's.

I'm asking for any feedback from anyone who has managed a similar task.

Edit: could an moderator change the title to all words instead of all works? Please?

Edit2: Thanks!

[Doitsu]

Originally Posted by DNSB

You would see a word index (I can't bring myself to call it a concordance) where clicking on the page number e.g. 76 would move to that page in ebook preferably with the cursor on the word aardvark.

aardvark pg. 1, pg. 14, pg. 49, pg. 76
syzygy pg. 21, pg 48, pg. 103

IMHO, page numbers would be of little use to the user.

What you could try is:

• Export the whole epub as a text or markdown file.
• Generate a list of all unique words (excluding stop words).
• Copy the list to the system clipboard.
• Add the list in Sigil via Tools > Index Editor > Add Entry > Paste.

(Add Entry supports multiline lists.)

[Quoth]

If it's NOT a concordance, then it's absolutely pointless as the built in search will work better.

[DNSB]

Originally Posted by Doitsu

IMHO, page numbers would be of little use to the user.

What you could try is:

• Export the whole epub as a text or markdown file.
• Generate a list of all unique words (excluding stop words).
• Copy the list to the system clipboard.
• Add the list in Sigil via Tools > Index Editor > Add Entry > Paste.

(Add Entry supports multiline lists.)

That still requires a fair amount of manual labour including generating the list of stop words. What he wants is an automated (i.e. someone else does the work) solution. Adding the forward/backward links alone is a nightmare never mind his insistence that it MUST have page numbers.

Originally Posted by Quoth

If it's NOT a concordance, then it's absolutely pointless as the built in search will work better.

That's what I've argued. From his email this AM, he has decided to to find a "professional" who will give him what he wants and at the rather low price he wants to pay.

[Doitsu]

Originally Posted by DNSB

His argument is that there must be an automated solution to doing this task pointing at AntConc 3.5.8 as an example

I haven't used AntConc in a while, but I'm pretty sure that it doesn't return page numbers.

Originally Posted by DNSB

[...] the other example he pointed out was written in Fortran and has not been maintained since the 80's.

I seriously doubt that an 80's program will return page numbers. What's the name of the Fortran program?

Originally Posted by DNSB

That still requires a fair amount of manual labour including generating the list of stop words.

There are ready-made stop word lists. For example:

MySQL

NLTK

It's also relatively easy to create a list of all unique words with NLTK and other tools. But insisting on page numbers doesn't make sense.

Originally Posted by DNSB

From his email this AM, he has decided to to find a "professional" who will give him what he wants and at the rather low price he wants to pay.

Good riddance.

[JSWolf]

Originally Posted by DNSB

I have one acquaintance who wants to have a mass of words in his magnum opus to be in an index with the page location of each word and the page location linked back to the word. He also wants this to show in an azw3/kfx Kindle format conversion.

You would see a word index (I can't bring myself to call it a concordance) where clicking on the page number e.g. 76 would move to that page in ebook preferably with the cursor on the word aardvark.

aardvark pg. 1, pg. 14, pg. 49, pg. 76
syzygy pg. 21, pg 48, pg. 103

My suggestion was that he forget the idea since a reflowable ebook does not have fixed page numbers and creating the index he wants would take an incredible amount of work for no real gain.

His argument is that there must be an automated solution to doing this task pointing at AntConc 3.5.8 as an example, the other example he pointed out was written in Fortran and has not been maintained since the 80's.

I'm asking for any feedback from anyone who has managed a similar task.

Edit: could an moderator change the title to all words instead of all works? Please?

To do page numbers that work, you would have to have an Amazon page number file without the index and another that is the same but adds in he index.

[DNSB]

Originally Posted by Doitsu

I haven't used AntConc in a while, but I'm pretty sure that it doesn't return page numbers.

From what I read in the docs after he mentioned AntConc, it doesn't return page numbers.

Originally Posted by Doitsu

I seriously doubt that an 80's program will return page numbers. What's the name of the Fortran program?

I think he may have gotten the name wrong since he referred to Cambridge but I think he meant the Oxford Concordance program.

Originally Posted by Doitsu

There are ready-made stop word lists. For example:

It's also relatively easy to create a list of all unique words with NLTK and other tools. But insisting on page numbers doesn't make sense.

Good riddance.

For what he was willing to pay and my estimate of the time required, I'd be better off walking around town looking for refundable bottles. Plus think of the healthy exercise.

[Tex2002ans]

Originally Posted by DNSB

I have one acquaintance who wants to have a mass of words in his magnum opus to be in an index with the page location of each word and the page location linked back to the word.

Why? What use would this be?

Indexes serve a purpose because they are human-curated. A random hodgepodge of every occurrence of a word isn't very helpful.

Originally Posted by DNSB

He also wants this to show in an azw3/kfx Kindle format conversion.

Does he have a final typeset "source" PDF that he is going to use as his be-all-end-all for all things page numbers?

Originally Posted by DNSB

where clicking on the page number e.g. 76 would move to that page in ebook

aardvark pg. 1, pg. 14, pg. 49, pg. 76
syzygy pg. 21, pg 48, pg. 103

If dealing with PDF, and you had a list of words, you could generate such a linked Index using LaTeX + hyperref:

Original Text:

This is an example sentence with aardvark and syzygy.

with normal index markup:

This is an example sentence with aardvark\index{aardvark} and syzygy\index{syzygy}.

or create a LaTeX macro:

\newcommand{\indexthis}[1]{#1\index{#1}}

[...]

This is an example sentence with \indexthis{aardvark} and \indexthis{syzygy}.

(More info can be found here: https://en.wikibooks.org/wiki/LaTeX/Indexing )

Now... getting that back into EPUB would be a different problem.

Originally Posted by DNSB

preferably with the cursor on the word aardvark.

lol.

Again, what is the point? At this level, you might as well be using Search, which does exactly what you want.

Trying to replicate "jumping to the exact point" would require disgusting word-level markup like Sigil's Index Tool, and even then, in 99.9% of the ereaders, it wouldn't happen like he imagines.

Side Note: And another thing, would he want all variants of "aardvark" ("aardvarks", "aardvarking") under an entry "aardvark" too? Or does he consider those all unique words/entries?

Originally Posted by DNSB

My suggestion was that he forget the idea since a reflowable ebook does not have fixed page numbers and creating the index he wants would take an incredible amount of work for no real gain.

Linked Indexes in ebooks are already a hell of a lot of extra work for minimal gain (as was just discussed a few days ago in " Create index on epub from printed book").

What he wants is off-the-deep-end absurd.

Originally Posted by DNSB

His argument is that there must be an automated solution to doing this task pointing at AntConc 3.5.8 as an example,

Thanks for this. I never heard of this program before. Looks like it might be of some use.

Originally Posted by DNSB

I'm asking for any feedback from anyone who has managed a similar task.

Run away now.

Maybe question this "friend's" sanity. Perhaps recommend them to a mental health specialist for wanting to unleash such horrors onto their potential readers.

Originally Posted by DNSB

That's what I've argued. From his email this AM, he has decided to to find a "professional" who will give him what he wants and at the rather low price he wants to pay.

Linked Indexes are exponentially more expensive to markup/create, and as you can read in the many Index topics... 99% of the time, even publishers don't want to spend the extra money to get them produced.

[DNSB]

Originally Posted by Tex2002ans

Run away now.
Maybe question this "friend's" sanity. Perhaps recommend them to a mental health specialist for wanting to unleash such horrors onto their potential readers.

I've turned down the "opportunity" and I have questioned his sanity. Unfortunately, he knows better than I what can be economically done in an ebook.

Originally Posted by Tex2002ans

Linked Indexes are exponentially more expensive to markup/create, and as you can read in the many Index topics... 99% of the time, even publishers don't want to spend the extra money to get them produced.

Perhaps I should point him in your direction or perhaps at Hitch. After all, he is now looking for a "professional". Preferably one who will not attempt a near criminal level of overcharging.

[Quoth]

I don't think Tex2002ans or Hitch have done anything to deserve it.
I sometimes think that some people have only used paper and PDFs and think ebooks are PDFs. PDFs are print replicas, proofing tools, or even a file for publishing to paper. I wonder have people asking for these things read real ebooks on real ereaders. Not PDFs on 19" screens or 12" tablets.
Oh, and my name is Manuel, I'm from Barcelona and I know NOTHING about making ebooks. Also that is not a dead rat in the kitchen. It's the cat's toy.

[DNSB]

Originally Posted by Quoth

I don't think Tex2002ans or Hitch have done anything to deserve it.

I was more thinking about watching either of them chop him off at the ankles. Either that or giving him a quote that will cause him severe distress.

Originally Posted by Quoth

I sometimes think that some people have only used paper and PDFs and think ebooks are PDFs. PDFs are print replicas, proofing tools, or even a file for publishing to paper. I wonder have people asking for these things read real ebooks on real ereaders. Not PDFs on 19" screens or 12" tablets.
Oh, and my name is Manuel, I'm from Barcelona and I know NOTHING about making ebooks. Also that is not a dead rat in the kitchen. It's the cat's toy.

Just convincing some people not to hit enter for every line in Word is painful.

As for PDFs? They are great for their intended purposes. Sadly as you mentioned, too many people tend to think of PDF as being the only ebook format. Reflow? Unreal page number? What's that?

[Hitch]

Originally Posted by DNSB

I've turned down the "opportunity" and I have questioned his sanity. Unfortunately, he knows better than I what can be economically done in an ebook.



Perhaps I should point him in your direction or perhaps at Hitch. After all, he is now looking for a "professional". Preferably one who will not attempt a near criminal level of overcharging.

You're welcome to send him in my direction. By the time I'm done with him, he'll have had a proper rethunk. ;-) Seriously, send him over, but I'm taking all 3 bloody days of this upcoming weekend off, b/c I and my crew of miscreants have been working our assess off for months now. Who knew that Covid would spawn an entire library's worth of scribbles? All of which urgently need to be made into books?

I do want to say that I would ABSOLUTELY criminally overcharge him for this. The brain-damage alone is worth thousands, IMHO.

I figure when people see the sign on the website, saying that we're closed, they'll roll their eyes, thinking "but, but, everybody's been 'off' for months!" Not us, man.

Hitch

[Jellby]

I don't understand... Sure, the sole proposal of such an index is nuts and ludicrous, but it doesn't seem to be difficult at all to do (except if one wants the page numbers to be the reflowable page numbers, and the "cursor on the word aardvark" part).

As for the stop words, just ignore them (i.e. don't exclude them), let him remove the corresponding entries once the index is done.

[Quoth]

Originally Posted by Jellby

I don't understand... (except if one wants the page numbers to be the reflowable page numbers, and the "cursor on the word aardvark" part).

Stop words

[Hitch]

Originally Posted by Jellby

I don't understand... Sure, the sole proposal of such an index is nuts and ludicrous, but it doesn't seem to be difficult at all to do (except if one wants the page numbers to be the reflowable page numbers, and the "cursor on the word aardvark" part).

As for the stop words, just ignore them (i.e. don't exclude them), let him remove the corresponding entries once the index is done.

Which is what he's requested, Jellby mon sweet.

Hitch