Indexing all words in an ebook

[DNSB]

Originally Posted by JSWolf

With RMSDK, this index could be made as it uses page numbers that do not change when any settings change.

Jon, how are you planning on generating those page numbers since they will have to be generated from the final epub? Remember that making changes to a file in the epub can cause the page numbers to change.

To quote from Adobe's documentation:

1. Determine a compressed byte length of each resource which is referenced in the spine, subtracting any known encryption overhead (IV size)
2. Assume that there is a page for each 1024 bytes in each resource, rounding up to the nearest whole number of pages for each resource
3. To map page breaks into a resource, use the number of pages for the resource as determined in step 2, count the number of Unicode characters in the resource; distribute synthetic page breaks in the resource evenly between the characters by dividing the number of characters by the number of pages; if the number of characters don’t divide evenly among the pages, round the number of characters per page up and let the last “page” contain less characters than the rest.

[DNSB]

Originally Posted by DNSB

Why do I have the horrible desire to toss page-map and page list into the discussion?

Originally Posted by Turtle91

Because you're an evil genius that wants world domination??

I don't want world domination. That makes you too much of a target. Now a nice easily manipulated puppet I could hide behind... Tempting...

[Jellby]

Originally Posted by DNSB

Why do I have the horrible desire to toss page-map and page list into the discussion?

That's why I said "reflowable" page numbers. Those you mention are not reflowable, they are fixed, and they are easily included in an index.

[Doitsu]

Originally Posted by Jellby

Those you mention are not reflowable, they are fixed, and they are easily included in an index.

I don't think that this could be easily done without creating a custom script that does the following:

• Locate page target tags in all XHMTL files.
• Locate index marker target tags between two page marker tags and save the page number for each index marker target tag.
• Update the index file href tags with page number references.

IMHO, that'd be a lot of work for a feature with little benefit to most readers.

[Jellby]

Oh, yes, some script-fu would be needed, but it should be relatively straightforward with some decent (x)(ht)ml parser

[Tex2002ans]

Originally Posted by Doitsu

I don't think that this could be easily done without creating a custom script that does the following:

[...]

IMHO, that'd be a lot of work for a feature with little benefit to most readers.

Usually I merge the entire book into one huge XHTML file, add the links, then split all files. This lets Sigil/Calibre take care of all the link locations/renaming.

I sometimes do a similar workflow when dealing with Endnotes too.

[JSWolf]

Originally Posted by Hitch

Oh, sugah, that horse turned into a pink pony a loooong time ago. Thus the wagon. ;-)

Hitch

As long as that pink pony doesn't turn into a pink elephant, you'll be excellent.

[rkomar]

Originally Posted by JSWolf

Well, I did give a solution that would work. But I never said it would be worth the effort. I think it's a terrible idea. I think the index idea should be scrapped. If you are going to fall off your wagon, make sure the horse has stopped first.

Just linking every instance of a word into an index would be a bad idea, but a properly constructed index showing places where terms are defined or demonstrated is a useful thing. It requires the author to mark those spots when writing though, like with the latex \index{} command noted previously. I don't think you could create a good index automatically.

A text search would find the same useful places, but you would have to filter out the unhelpful matches yourself. That can be a chore if the term is used extensively in the document. I don't know how many times I've passed right by the stuff I was looking for while hitting "Find Next" repetitively.

So, I think a proper index is good thing, even in a searchable document. Whether it can be implemented well in an EPUB is another question for those more expert than I am.

[Hitch]

Originally Posted by rkomar

Just linking every instance of a word into an index would be a bad idea, but a properly constructed index showing places where terms are defined or demonstrated is a useful thing. It requires the author to mark those spots when writing though, like with the latex \index{} command noted previously. I don't think you could create a good index automatically.

A text search would find the same useful places, but you would have to filter out the unhelpful matches yourself. That can be a chore if the term is used extensively in the document. I don't know how many times I've passed right by the stuff I was looking for while hitting "Find Next" repetitively.

So, I think a proper index is good thing, even in a searchable document. Whether it can be implemented well in an EPUB is another question for those more expert than I am.

The implementation, however, in an ebook is not the same as it is in Word-->PDF or Word--INDD. The way that the instances are tagged and marked, and utilized, are not the same.

When you tag a word in Word or LaTEX or INDD, a "marker" is placed for that word effectively at the left-most, top-most position of the page, basically saying "here lies Setters, Irish." When the index is built, the page-tag is picked up and displayed, dynamically changing until the final layout is settled. When the reader reads the book, they flip to the page in question and visually skim the page. Ta-da!

But, in an eBook, the tag would have to travel with the word. As it's an ebook, you're talking about a link. The word or phrase itself would be the target (and thus, on many systems, BLUE and underlined).

There would be no backlinking; there wouldn't be any quickie reference, going forth to the index, choosing to read about Setters, Irish and then flipping quickly to all the "pages" or locations or targets and then back. It would be rather laborious. You'd be constantly going back to the index, back to the word/phrase, back to the annotations (page numbers, sequence numbers, whatever the hell you'd put there to indicate "Fred said something here") and then tediously going to the next.

The only way you could possibly have any backlinking, from the index to the targeted item, would be if each paragraph or item or phrase or whatever, was only ever linked once. That's not that common, actually.

I should also point out that not all eReaders (yes, I'm lookin' at you, ADE engine!), scroll to the spot in question. So...what's the reader get them? A screen full of text that with any luck, is near the item in question.

It's a vexed question, indices in eBooks and there's no great solution, imho. Tex2002ans and I have wrangled over this innumerable times, trying to come up with "better ways" and the reality is, without investing far more time and effort (than any client will pay), the utility of the result, even an "ideal" result, is not that great.

"Just" having the author or indexer tag the text doesn't really solve the problems. The only real functionality comes if some poor schmuck sits there and creates a landing at the beginning of each paragraph, (or mid-paragraph or wherever) where a given term is used or discussed and of course, heaven help you if the original index meant "pages 223,24, 25." I mean, just fuhgeddaboudit.

We've all beaten this particular horse to death here on MR. I have yet to see a "good" solution to it.

Hitch

[JSWolf]

And don't forget that if you are reading and want to go to the index, you'd have to remember where you left off reading or set a bookmark or you'd be a rather long time trying to find where you were in the book. Yet another reason against this index.

[Hitch]

Originally Posted by JSWolf

And don't forget that if you are reading and want to go to the index, you'd have to remember where you left off reading or set a bookmark or you'd be a rather long time trying to find where you were in the book. Yet another reason against this index.

That's the point I'm trying to make about the backlinks.

Hitch

[rkomar]

It sounds like the best you could do in the index is link the chapter/section/subsection/... that the term is found in, and hope it isn't so big that the term gets lost in there. It would probably be better than nothing, but not really that good either. Oh well. It's funny that something that feels like it should be easy in an ereader turns out to be nigh impossible.

[Tex2002ans]

Originally Posted by rkomar

It sounds like the best you could do in the index is link the chapter/section/subsection/... that the term is found in, and hope it isn't so big that the term gets lost in there. It would probably be better than nothing, but not really that good either. Oh well. It's funny that something that feels like it should be easy in an ereader turns out to be nigh impossible.

See my Post #129 + Post #166 in the famous "Sick of Amazon Kindle books without Page Numbers" thread.

I came up with "format-specific" and "format-neutral" page numbering schemes + ways of citing.

Page numbers would be more "format-specific" (Print), while byte- and location-based methods are more ebook-friendly, but make absolutely zero sense in Print.

Chapters/Sections are much more "format-neutral", and work across all formats.

I believe I laid them all out in #166. If anyone thinks of any more categories, I'd be interested in hearing about them.