Quote:
Originally Posted by tshering
This "words" file is used to get the information whether an expression that is looked-up can be found in the respective language dictionary or not, and maybe some other information.
|
All you need to get from a query W to file "words" is: 1) which chunk (file .html(.gz)) contains the word W and its definition; 2) which is the "position" of word W in that chunk.
For 1), usually one assigns an ID to each chunk, like this: 11.html is 0, aa.html is 1, etc. in lexicographical order.
For 2), an easy way is to store, for word W, the offset, in bytes from the beginning of the chunk, where the definition of W starts.
(The dictionary is slit into several chunks to allow faster fetch-decompress-find operations)
See my analysis of the Cybook Odyssey dictionaries at:
http://www.albertopettarin.it/penelope.html
Quote:
Originally Posted by tshering
If we knew what the values of the key-value pairs consist of we could build our own "words" file. This again would enable us to insert new entries into the language dictionaries, or to build up a new dictionary from scrap. How the values look like should be easily ascertained with the marisa tools. However, I failed in my attempts. Therefore, I can only speculate about it.
|
That's the point. I haven't had the chance of playing with the marisa-lib. Understanding the content of the (decompressed) index (file "words") is the key point there.