Snipped from Proposal: Extending Epub

[Nate the great]

Originally Posted by Peter Sorotokin

That's a legitimate point of view, of course. The problem is that building index on the device would be slow and drain the battery and building it elsewhere would mean that special software needs to be used to transfer the book to the device. I think that support for indexing is just too central for a dictionary to leave it out.

I only wrote that because I misunderstood how Mobipocket handled indexes. I was also hoping this thread would die, but oh well.

I think it's safe to assume that the indexes will be made during the ebook creation process. I would suggest that each index in an ebook be in a separate file(s). Given that Epub is basically zipped HTML, an index will likely consist of 1 or more links (that lead to other places in the ebook).

I think we should consider copying the behavior of Mobipocket indexes. A link in the title index doesn't lead to the respective title. Instead, it leads to the beginning of the entry containing the title. (I also agree with Igorsk about the need for multiple head words.) In the keyword index, each keyword will listed once, and link to a separate file consisting of links to each of the entries containing the keyword. The links won't lead to the keyword, but to the beginning of the entry.

[Peter Sorotokin]

Originally Posted by Nate the great

I think it's safe to assume that the indexes will be made during the ebook creation process. I would suggest that each index in an ebook be in a separate file(s). Given that Epub is basically zipped HTML, an index will likely consist of 1 or more links (that lead to other places in the ebook).

I think we should consider copying the behavior of Mobipocket indexes. A link in the title index doesn't lead to the respective title. Instead, it leads to the beginning of the entry containing the title. (I also agree with Igorsk about the need for multiple head words.) In the keyword index, each keyword will listed once, and link to a separate file consisting of links to each of the entries containing the keyword. The links won't lead to the keyword, but to the beginning of the entry.

So what you saying is that an entry in the index will look like this (assuming specialized XML mark-up for index).

<entry href="QU.html#queen">queen</entry>

And corresponding dictionary article like that (assuming XHTML for the content):

<dl id="queen">
<dt>queen<dt>
<dd>a female sovereign or monarch</dd>
</dl>

That would work on the syntax level, but I don't think a flat index file containing all words is going to cut it: it is still going to be too big.

Also, can we used any existing mark-up (e.g. XHTML or perhaps NCX) for index file? Should we just use XHTML-based index with some metadata marking it as such?

I'll think a bit more about it.

[Nate the great]

Actually, I was thinking of straightforward HTML for the index entry:

<a href="dictionary.html#d_somenumberX">dictionary</a><br />

It would have a corresponding link in the body of the ebook, of course.

[HarryT]

It's certainly no problem having multiple headwords for a single entry in Mobi dictionaries. The Chambers dictionary I have on my Gen3 will, for example, find words with variant spellings - eg "center" or "centre".

[Peter Sorotokin]

Originally Posted by Nate the great

Actually, I was thinking of straightforward HTML for the index entry:

<a href="dictionary.html#d_somenumberX">dictionary</a><br />

It would have a corresponding link in the body of the ebook, of course.

I see. So here is a minimalistic proposal based on the discussion so far:

1. Add metadata tags (exact tags TBD) indicating that the EPUB is a dictionary, optional "input" language (the langauage that the dictionary articles are in is indicated by dc:language element), optional reference to the index file and optional collation declaration that describes the order of terms in the dictionary.

2. Dictionary should be split in multiple sections. In addition, an index file can optionally be provided. Index file should have linear="no" attribute in the spine. If an index is provideed, it should be referenced by the metadata.

3. Each entry in the dictionary must be formatted using XHTML dl tag. The first dt tag inside dl is considered to be a primary term. Dictionary entries must go in the order specified by collation - both inside a single section and across all sections as they are referenced in the spine.

4. Index is an XHTML file (exact structure TBD) that lists the sections of the dictionary itself (as opposed to supplementary material) and only the first term for each section. That both allows for efficient search and does not bloat the index.

Peter

[Nate the great]

Originally Posted by Peter Sorotokin

I see. So here is a minimalistic proposal based on the discussion so far:

1. Add metadata tags (exact tags TBD) indicating that the EPUB is a dictionary, optional "input" language (the langauage that the dictionary articles are in is indicated by dc:language element), optional reference to the index file and optional collation declaration that describes the order of terms in the dictionary.

2. Dictionary should be split in multiple sections. In addition, an index file can optionally be provided. Index file should have linear="no" attribute in the spine. If an index is provideed, it should be referenced by the metadata.

3. Each entry in the dictionary must be formatted using XHTML dl tag. The first dt tag inside dl is considered to be a primary term. Dictionary entries must go in the order specified by collation - both inside a single section and across all sections as they are referenced in the spine.

4. Index is an XHTML file (exact structure TBD) that lists the sections of the dictionary itself (as opposed to supplementary material) and only the first term for each section. That both allows for efficient search and does not bloat the index.

Peter

Why don't we try to limit this thread to just the discussion of the index?

1, yes.

2, I think an index should be required due to the need for a speedy lookup.

3, The dl tags seem to be duplicating what we are trying to do with the XML tags, and can't get achieve the specificity desired . Why use both?

4, Let me expand on what I wrote before.

A dictionary, for example, will have at a minimum title index(or something that will serve that purpose). It might also have one or more keyword indexes.

The title index will be in its own file that is separate from the the rest of the book as well as being separate from the other indexes. Each index will be in a separate file (or files) from the other indexes. If there is more than one type of keyword (example: "famous people" & "famous places"), each type of keyword will have its own index with its own files.

Here is where my explanation wasn't clear before. A keyword index, "famous people" for example, would be in the file "famous people_x.html". The entries would look like this:

<a href="johnny appleseed.html">Johnny Appleseed</a><br />
<a href="Kevin Costner.html">Kevin Costner</a><br />
etc.

The file "johnny appleseed.html" would contain entries something like this:

<a href="dictionary.html#d_somenumberX">an entry</a><br />
<a href="dictionary.html#d_somenumberY">another entry</a><br />

So a keyword index would actually consist of a group of files.

[Peter Sorotokin]

Originally Posted by Nate the great

Why don't we try to limit this thread to just the discussion of the index?

OK, but we'll need to discuss XHTML vs. TEI or XDFX. I am leaning towards using XHTML.

A dictionary, for example, will have at a minimum title index(or something that will serve that purpose). It might also have one or more keyword indexes.

While I see similarity between title index and keyword index, for practical purposes they may need to be treated somewhat differently (like in p-word). For foreign language dictionaries, title index is going to bloat to the same size as dictionary itself (since definitions a lot of times are as long as the link would be). On the other hand, each individual piece of the dictionary body is already self-indexing, since words go in alphabetical order.

On the other hand, keyword indices have to list every word (since they cannot rely on the document structure), but typically won't be as large (judging by the printed books). Also, in many cases, keyword index includes a short definition for each term, in addition to the link(s) to the book body. From that perspective, keyword indices are more similar to small dictionaries than to the title index.

Finally, my instincts are to avoid br tag. Wrap it in p, li, dt - whatever - instead.

Peter

[Nate the great]

Originally Posted by Peter Sorotokin

OK, but we'll need to discuss XHTML vs. TEI or XDFX. I am leaning towards using XHTML.

I would prefer XHTML because simpler is usually better.

While I see similarity between title index and keyword index, for practical purposes they may need to be treated somewhat differently (like in p-word). For foreign language dictionaries, title index is going to bloat to the same size as dictionary itself (since definitions a lot of times are as long as the link would be). On the other hand, each individual piece of the dictionary body is already self-indexing, since words go in alphabetical order.

I disagree about the entry length. I looked at the WordNet Mobi dictionary. The average length was at least twice as long as the link.

Also, while the entries of a dictionary are alphabetical, having a list of just headwords without the entries means you can look at and discard more entries at a time. This will make finding a word (with uncertain spelling) faster.

Question: would it be possible to build the headword index into the toc.ncx file? If so, could it behave like an index?

On the other hand, keyword indices have to list every word (since they cannot rely on the document structure), but typically won't be as large (judging by the printed books). Also, in many cases, keyword index includes a short definition for each term, in addition to the link(s) to the book body. From that perspective, keyword indices are more similar to small dictionaries than to the title index.

Peter

A definition would be in the body of the text, not the keyword index. I've never seen a reference title that had an index with definitions. I've seen books with both glossaries and indices, but they were separate entities.

[Peter Sorotokin]

Originally Posted by Nate the great

I disagree about the entry length. I looked at the WordNet Mobi dictionary. The average length was at least twice as long as the link.

Oh, but 2 is approximately 1 ;-). I bet a full index for 100M Russian-English dictionary is going to be at least 10M and my gut feeling tells me that's about 10 times more than practical.

Also, while the entries of a dictionary are alphabetical, having a list of just headwords without the entries means you can look at and discard more entries at a time. This will make finding a word (with uncertain spelling) faster.

You can think of my proposal as search tree (althouh very shallow). I think it is better for searches than flat array in almost all cases.

Question: would it be possible to build the headword index into the toc.ncx file? If so, could it behave like an index?

Per spec, I do not see how, but I'd rather someone else confirm it.

[Peter Sorotokin]

BTW, judging by the level of interest that this thread generated, people care about dictionaries even less than I thought ;-)

[DaleDe]

Originally Posted by Peter Sorotokin

BTW, judging by the level of interest that this thread generated, people care about dictionaries even less than I thought ;-)

Do to the technical nature of this thread I don't see how you can jump to that conclusion. Most people don't care how a dictionary is implemented, they just want one.

Dale

[jgray]

Originally Posted by DaleDe

Do to the technical nature of this thread I don't see how you can jump to that conclusion. Most people don't care how a dictionary is implemented, they just want one.

Dale

I agree with Dale on this. Peter, perhaps you are thinking too much like an engineer and not enough like an average reader? As for interest in this thread, I have been following it closely, as I am sure others have.

[Sabardeyn]

I've been following this discussion since the beginning, although I have not commented. The discussion is about a format that I don't use (mobi/mobipocket), have no personal knowledge of, and provides detailed discussion of the programming of same.

In other words, I don't see that I can contribute to the discussion in any constructive manner. So...

[zelda_pinwheel]

Originally Posted by DaleDe

Do to the technical nature of this thread I don't see how you can jump to that conclusion. Most people don't care how a dictionary is implemented, they just want one.

Dale

Originally Posted by jgray

I agree with Dale on this. Peter, perhaps you are thinking too much like an engineer and not enough like an average reader? As for interest in this thread, I have been following it closely, as I am sure others have.

Originally Posted by Sabardeyn

I've been following this discussion since the beginning, although I have not commented. The discussion is about a format that I don't use (mobi/mobipocket), have no personal knowledge of, and provides detailed discussion of the programming of same.

In other words, I don't see that I can contribute to the discussion in any constructive manner. So...

add my vote and agreement to that. i CARE about dictionaries, a LOT. and i desperately want dictionary support for epub. but i don't know anything about mobipocket to contribute. i have been following a bit though and the discussion interests me. and i'm very glad that nate has decided to tackle the question.

[Valloric]

Originally Posted by Peter Sorotokin

BTW, judging by the level of interest that this thread generated, people care about dictionaries even less than I thought ;-)

I would like to chime in with the rest and say that although I haven't added to the discussion, I'm following it closely. Dictionaries are important.