PRS-650 Access to the Dictionary files

[Papi]

TJ27 : thanks for the info, though I didn't understand everything :

Originally Posted by TJ27

BBeb Dictionary studio worj in 'shiftJS' encoding (tittle, autor and words) - sony prs use unicode. Need big "path" bbebdic.dll

OK, I something was wrong in terms of encoding (I had to use Unicode BE as my CSV test encoding, else BBeB di studio would fail with a 616 error), but didn't know exactly what was going on.

Official selling sony dictionary only japan -

That's what I've concluded, so only a few are available, and most to and from Japanese I guess. That's a bit sad, since I don't think we can achieve the same quality with free dictionaries, if they are even available.

And header working dictionary and "made in studio" - different, not only CDUS <-> FDJP

Interesting, though it doesn't explain why I couldn't replace a built-in dictionary with another built-in dictionary. Maybe the mount --build trick doesn't work, that would explain I get no definition at all.

Only library.mbf work - i remove dictionary and mbf file edited correctly
I attach 'patched' bbedic.dll (only not need correct CDUS <-> FDJP)

This part I didn't understand :
- I know the library.mbf is used to reference the .msd files, but is there anything to edit there ? Does BBeB Studio create this file ?
- The patched bbedic.dll allows to create BBeB dictionary with correct header for the PRS devices ?
- Did you actually succeed implementing your own dictionaries in the PRS 350 ?
- Do you have more information on the MSD /MBF format ?

[TJ27]

No, i dont correct header for full compatible. Only patch for CDUS <-> FDJP. (veryfing header)
Mbf file automatic create after send (modify complection) dictionary from memorystick.
Insert any formatted usb flash disk (or card to cardreader), select destination to this device. Dictionary for studio : x:/ebook/bbeb/dic
For repalce - may be need mount full dict directory (see prs+), and modify .js script file
I bad know
being don't have correct(unicode) version studio - uselessly

[Mark Nord]

Hello,

I understand less then half of your discussion, but could setting bit 2 be of any help?
(Presumable using a dictionary in IM instead of the build in ones?)

Originally Posted by Papi

..
On another topic, in the kconfig.xml, there's the following line :
<variable name="dictionaryApplicationOnly" value="0"/> <!-- bit0:dicApp only, bit1:no Word Log, bit2:/Data/dictionary -->
Anyone knows more about this ?

[TJ27]

Originally Posted by Mark Nord

Hello,

I understand less then half of your discussion, but could setting bit 2 be of any help?
(Presumable using a dictionary in IM instead of the build in ones?)

May be need set 4 and put all dictionary to /Data/dictionary - visible root /dictionary
Tommorow probe this

[Papi]

OK I understand now. I didn't try the "send to memory stick functionality" so I didn't see the MBF file.
As for the replacement, I simply copied a modified version of the original English -> French directory into /opt1/dict, and then did a mount --bind to replace another dictionary (the Nederlands -> English one). I see no errors in the log, but got no successful result. I edited the modified dictionary to put the correct name inside (CDUS125D0000????), I even tried to copy the entire header (anything starting from the beginning to the name) from the target dictionary to the modified one, but that didn't work either. Now I will try to mv the original dictionary file, and copy my modified file as the original one, so I'm sure it's not mount --bind which causes a problem.
But even if that does work, it seems we are stuck with BBeB dic Studio working only with Shift JIS encoding, and that encoding has only "basic" letters, so no é à ç è ã ö, etc. So without a proper software to create a msd file, and without knowledge of the msd format, it seems hard to get a good functional dictionary. Maybe there's a way to implement the functionality using another dictionary format. I hope the PRS+ team succeed whatever the way.

[Papi]

Originally Posted by Mark Nord

Hello,

I understand less then half of your discussion, but could setting bit 2 be of any help?
(Presumable using a dictionary in IM instead of the build in ones?)

Mark : you're right we have to try that too.
I tried yesterday to replace a dictionary with another pre-installed one (instead of mount --build), and no luck either. I tried replacing a good part of the beginning of the header from the source to the target, but I guess there is a sort of checksum in the MBF file.

So next step would be creating a custom dictionary and associated MBF file, and try the dictionaryApplicationOnly trick. There's the problem of Shift JIS / Unicode thing though.

[kartu]

Dictionary is located on VFAT partition I'm not sure if mount --bind works there.
But you can in mount it as read write and replace any file there.

[Papi]

Originally Posted by kartu

Dictionary is located on VFAT partition I'm not sure if mount --bind works there.
But you can in mount it as read write and replace any file there.

Yep, that's what I did : mount as rw, then mv the original file, cp my modified dictionary with the name of the original one. I did a ls -l before and after, and given the names and sizes of files, it seems that this part was executed without errors. But I suspect the library.mbf file contains more informations about the MSD files than just the names, and I don't know what exactly.

[Papi]

In fact I chose to modify the root for dictionaries in kconfig.xml and mount --bind it, so the root of Dictionaries is in /Data for easier access. After many trials, I was able to create a dictionary without errors in BBeB Dictionary Studio, and modifying consequently kconfig.xml, it is selectable and usable (see screenshots), but there are several problems :
- I had to "cheat" BBeB Dict Studio, which takes an CSV as input with the following format : entry,search_pattern1/search_pattern2/../,text_displayed.
For example : car,car,car. However, that would always generate an error in BBeB studio, which could be bypassed by not putting the 3rd field. So car,car, would be OK. The CSV format was given by someone who posted on Yahoo's Librie group, and I found on the same group someone who had the same problem as I had, but his solution didn't work for me. So to obtain the screenshot, I cheated by putting the entry car : test dictionnaire,car,, in the CSV, so it displays "car : test dictionnaire". So it would be nice to understand why BBeB Dict Studio gives an error with the "correct" format : there it would be nice to have someone speaking Japanese because the help file of the software is written in that language.
- I have not tried yet to look for a definition of a word inside the dictionary (given the first point, I'm not sure it works).
- I have not tried yet multiple search patterns.
- The LIBRARY.MBF file references the dictionaries, but I couldn't load the pre-installed dictionaries in BBeB Dict Studio, so I can't create a LIBRARY.MBF which references pre-installed dictionaries AND custom dictionaries at the same time. I have to try a "mix" of two LIBRARY.MBF, maybe it would work (I doubt it though).
- The main problem is : how to get quality dictionaries in CSV format. I was looking for a German -> French one for example but didn't find any. A solution would be to buy one, find a way to convert it to CSV and then to MSD...I don't know what is the main format for dictionaries, but I kind of doubt it's in "readable" format.

[TJ27]

Get dictionary for Dict (txt format, or zd-format and use makezd)
http://free-dict.narod.ru/createzd.html - russian page
Or find utility for conversions/unpack stardick, xdxf dictionary to txt/csv format.
You have error conversion - set encoding for not-unicode program to Japan (in regional setting), reboot - and car,car,car processing OK.

[kartu]

Originally Posted by Papi

The main problem is : how to get quality dictionaries in CSV format. I was looking for a German -> French one for example but didn't find any.

Try this. (well, not "quality", but still dictionary )

[porkupan]

Hey, I see the dictionary discussion has re-activated...

Unfortunately, I am afraid some of you are moving in the wrong direction. We have been though all these exercises last year, and concluded that BBeB Dic Studio cannot by itself be used to produce a working dictionary for Sony Reader. The problem seems to be in the format of the word/search index. The dictionary works with a few words, but only the first few words appear to be indexed correctly.

The BBeb Dic Studio generates a valid LIBRARY.MBF file. The reader's software in fact doesn't care if the dictionary ID starts with FDJP or CDUS. You can either make a binary change in the dictionary header, or hack up the Studio. kconfig.xml points to the dictionary (library) location [bbebDictionaryRootFile], so you can test with dictionaries located in the /Data partition, and you don't need to overmount anything. All you needed to do on PRS-600 was to put the MSD files in the same directory with the MBF file. I suspect it is still so for x50.

It is true that the new BBeB dictionary format supports Unicode, but Shift-JS encoding appears to be sufficient for most applications, so we tested with it.

The bottom line is, the index in the BBeB Dic Studio-generated MSD file must be regenerated. igorsk looked at the structure of the proper index, but he so far hasn't had the time to write code to generate a working index for Sony Reader-type BBeB dictionaries, and replace the Studio-generated index with the proper index. But that seems to be the way to go.

[Papi]

Originally Posted by porkupan

Hey, I see the dictionary discussion has re-activated...

Unfortunately, I am afraid some of you are moving in the wrong direction. We have been though all these exercises last year, and concluded that BBeB Dic Studio cannot by itself be used to produce a working dictionary for Sony Reader. The problem seems to be in the format of the word/search index. The dictionary works with a few words, but only the first few words appear to be indexed correctly.

Well I would have like to know this before I made all those tries/fails, I guess it was discussed on the russian forum, but it's too much pain to try to read the google translation. Well, I learned something in the process, so good for me.

The BBeb Dic Studio generates a valid LIBRARY.MBF file. The reader's software in fact doesn't care if the dictionary ID starts with FDJP or CDUS. You can either make a binary change in the dictionary header, or hack up the Studio. kconfig.xml points to the dictionary (library) location [bbebDictionaryRootFile], so you can test with dictionaries located in the /Data partition, and you don't need to overmount anything. All you needed to do on PRS-600 was to put the MSD files in the same directory with the MBF file. I suspect it is still so for x50.

Yes that's basically how far I came, my "custom" dictionaries + library.mbf are in /Data/mydictionaries and I just overmount kconfig.xml, and assets.xml for the string displayed. I didn't know there was problem with indexing though, my custom dictionary only had 3 entries for testing purpose. I just tried with a more realistic dictionary, it made the reader reboot.

It is true that the new BBeB dictionary format supports Unicode, but Shift-JS encoding appears to be sufficient for most applications, so we tested with it.

I thought Shift JIS only defines "basic" ANSI caracters, so no cyrillic, no é è ù ü õ, a etc...What did I miss here ? I thought in fact that the Shift JIS encoding would be also a big problem, and ideally we would need the new BBeB Dict Studio, but I guess it's not publicly available.

The bottom line is, the index in the BBeB Dic Studio-generated MSD file must be regenerated. igorsk looked at the structure of the proper index, but he so far hasn't had the time to write code to generate a working index for Sony Reader-type BBeB dictionaries, and replace the Studio-generated index with the proper index. But that seems to be the way to go.

Regenerating the index...So you more or less "understand" MSD file format or part of it ? Did it need some reverse engineering there, by generating dictionaries with BBeB dict Studio ?

[porkupan]

Originally Posted by Papi

I thought Shift JIS only defines "basic" ANSI caracters, so no cyrillic, no é è ù ü õ, a etc...What did I miss here ? I thought in fact that the Shift JIS encoding would be also a big problem, and ideally we would need the new BBeB Dict Studio, but I guess it's not publicly available.

Actually, Shift JS does support Russian Cyrillic and some other character sets. I don't think it's enough for full European support though.

Originally Posted by Papi

Regenerating the index...So you more or less "understand" MSD file format or part of it ? Did it need some reverse engineering there, by generating dictionaries with BBeB dict Studio ?

igorsk reverse-engineered the part that interprets the index in the Sony Reader FskBBeBDictionary.so module.

[Papi]

Oh it seems Cyrillic is indeed in Shift JIS. Not much luck for European languages though, except English of course. I don't see any way to get around this problem of accentuated characters. If BBeB Dict Studio uses a translation table from Shift JIS to UTF-16 or UTF-8, maybe there's some way to "hack" this table and replace non used "Shift JIS characters" with all accentuated alphabetic characters.