Dictionary for Sony PRS500/5 (autorun from SD card)

[spooky69]

Okay, I got proactive and created a Dictionary that seems to work really well, though it's awfully large (176,049 lines adding up to ~14mb) and a full search through the whole list takes about 80 seconds. This becomes a problem since if your word isn't found at all or you make a typo, you have to wait though the whole search or pop out the SD card. I really think the search needs to be smarter since it should be relatively easy for the program to find a word within an alphabetical list (I guess this wouldn't apply to any non-dictionary files, but when we're using this type of file it could stand to be a lot faster).
Anyway, haven't had any problems with this dictionary besides the long search times, though it includes suffix definitions and those lines start with the "-" symbol. If anybody knows of a text editor that can go through and find any line starting with that symbol and simply delete it, I think this Dictionary would be great.
By the way, here's where it comes from:
http://www.mso.anu.edu.au/~ralph/OPTED/
I guess it's basically that same 1913 Webster's dictionary from Project Gutenberg.

[Clemenseken]

Hi spooky69!

I appreciate your fast reaction and attitude towards improvements...

I'll be on holiday this coming week and intend to take my iBook with me. But then I don't know if I'll have time for my "hobby" while away with my family...

But end next week there might or (might not) be a new version with some of your ideas integrated. (I am aware that my program is not very apt for really large files, though, as yet.)

[spooky69]

Hey, I'll be keeping an eye out! A few of other things I noticed that need some attention in the future:
-I left the program open on a full charge and when I did something on the screen again it had drained my battery by half! It was only sitting there for 3 to 4 hours, and I assume this means it was actively reading off of the SD card the whole time. If it's possible, it really needs to sit "idle" when somebody's not using it.
-When going to the next/previous definition, it's significantly faster toward the beginning of the file than toward the end. When it's on a word in the "A" section, the flip to the next word is nearly instantaneous. I tried going to the next word after "Zebra" and I think it took almost as long as it would have to look the word up on its own.
-There are some minor graphical glitches present that I don't remember from the last version...here's a screenshot:

I know it's hard to see, but there's a "return back" curved arrow over the title and there are some strange evenly-spaced "dotted lines" running right under the black bar at the top that you can see faintly in the picture. Let me know if you need any more info on these issues.

[Clemenseken]

First things first:
There is a new "add-on" feature for models prs505: it's another alternative keyboard layout (suggested by spooky69).
With this one letters are organized in an A-Z order (abc/def/ghi/...) which may suit 505-owners better, I assume. It wasn't much work, so I just post it here as a "Service Pack 1" for 505ers!
But be aware that this totally different layout needs an (only marginally) changed "autorun.js" file (included); whenever you decide to swap layouts (back) you'll have to swap "autoruns", too...

Originally Posted by spooky69

I left the program open on a full charge and when I did something on the screen again it had drained my battery by half! It was only sitting there for 3 to 4 hours, and I assume this means it was actively reading off of the SD card the whole time.

I noticed very soon when I had bought my prs500 that even READING from SD cards draws hard on battery power.
Functions in my app actually only run when called by some input (keys pressed). BUT it may well be that there are some system routines that now and then look what's happening; so I'd suggest that you switch off (MENU) the app while not needing it.

When going to the next/previous definition, it's significantly faster toward the beginning of the file than toward the end[...]

... as search functions work line-by-line: the sooner found the faster displayed. (I am thinking about some sort of "cacheing", though.)

... there's a "return back" curved arrow over the title and there are some strange evenly-spaced "dotted lines" running right under the black bar [...]

This funny "arrow" is absolutely new to me; hmmm, could be some different display "mode" from my prs500...
I am aware of these dotted lines; these will no longer be visible in 0.8/9...

And one more thing:
I made a little test with a 5MB file. Only reading the lines without actively searching took more than 30 seconds. So 15MB would be nearly 2 minutes!
Comparing a search word to "read" lines actually takes less time than reading them; I am afraid this does not leave much room for more efficient search algorithms in simple txt files...

[Clemenseken]

Originally Posted by Clemenseken

And one more thing:
I made a little test with a 5MB file. Only reading the lines without actively searching took more than 30 seconds. So 15MB would be nearly 2 minutes! [...]

It came to my mind this morning that I use a very old 16MB Multimedia Card (MMC) for development:
Its advantage is that the ereader can scan it very fast. And it's flatter than SD cards: so it's easier to insert.
BUT IT'S VERY SLOW.

I just repeated my test with spooky69's 15MB dictionary on a 256MB card:
Finding the last word (zythum) took about 50-55 secs.
Just reading the file took about 33 secs.
(You can imitate "Only reading" by using the next/previous buttons; that one just counts and reads lines without comparison.)



If anybody reads this post who is experienced with fast file access - you are urgently invited to share your knowledge: how can the fucntion "this.readline()" be quickened or replaced to get faster results ?!? (e.g.: omitting the first half of a file or s.th. else...)
Thanx a lot !!! Clemens

[spooky69]

It seems like you're on track to a solution, and if the universal search speed could get to under 15 seconds or so I'd be perfect happy. It seems that since a "book file" is able to move along incrementally without reading all the way through the file, the same should be possible as part of a search and/or "next word" function.
The obvious people to ask about this on here would be Kovid or igorsk. I'd bet that they would both have some sort of input to offer on any problems and would probably both appreciate simply having the dictionary software available to use on their own Sony Readers.
A couple of other things: is there anyway to keep the software from starting automatically when I turn on my reader or unplug it from a charge? I guess it's simply launching automatically every time the card is read (such as when it starts up from a usb-connected state) but if there's any way to distinguish between the user popping out the SD card on their own during an "idle" state and whatever happens when it reads off the card automatically, then it might make just a tiny bit more usable. This is something I would ask the gurus above about, though it might involve a somewhat complicated solution to a relatively small problem...possibly something for a later version, once all of the program functionalities are ideal!
I'd also humbly suggest that you include the dictionary I uploaded in the main post (unless having those few lines starting with "-" would cause eventual hang-ups). I think that most people on here are looking for a standard English dictionary, and having that available up front will encourage more people to download the software and beta test it.
Once again, I appreciate the effort you're putting into this and I'll do whatever I can to help from the user-end standpoint.

[igorsk]

The dictionary by markscad uses grep and that seems to be working pretty fast.

[Clemenseken]

Originally Posted by igorsk

The dictionary by markscad uses grep and that seems to be working pretty fast.

Hi, igorsk!

While I am downloading the archive for that program (after using google-translate for the first time ever!): will I be able to decompress the extension ".7z" on my normal PC / Mac? I never heard that before...
Thanks! Clemens

[FixB]

Hey, a dictionary for my Sony Reader : Thanks, I think I'll give it a try

As for .7z, just look here : http://www.7-zip.org/ you should not have any problem to decompress your file.

[=X=]

Originally Posted by Clemenseken

It came to my mind this morning that I use a very old 16MB Multimedia Card (MMC) for development:
Its advantage is that the ereader can scan it very fast. And it's flatter than SD cards: so it's easier to insert.
BUT IT'S VERY SLOW.

I just repeated my test with spooky69's 15MB dictionary on a 256MB card:
Finding the last word (zythum) took about 50-55 secs.
Just reading the file took about 33 secs.
(You can imitate "Only reading" by using the next/previous buttons; that one just counts and reads lines without comparison.)



If anybody reads this post who is experienced with fast file access - you are urgently invited to share your knowledge: how can the fucntion "this.readline()" be quickened or replaced to get faster results ?!? (e.g.: omitting the first half of a file or s.th. else...)
Thanx a lot !!! Clemens

Hi Clements,
The slowest possible way to read/access a file is reading a line at a time. However this is the easiest way, and also my preferred way, because if it's simplicity.

One recommendation I have is to change the way you search trough a file. Instead of doing a word comparisons for each word I would implement a binary search until I get a match. (I'd stop at just getting the first letter to match then read one line at a time).


///////////////////// FEEDBACK /////////////
Love the change dictionary feature!
this app gets better and better.

1) I would recommend instead of using 'ita.txt' as the default dictionary. could you maybe use a generic name like 'defdict.txt'
2) Is there a 'clear' button so we don't have to hit backspace to remove the text.
3) On the <Next/Prev> the text description is updated but the word that is being defined is not updated.


=X=

[spooky69]

Originally Posted by =X=

///////////////////// FEEDBACK /////////////
Love the change dictionary feature!
this app gets better and better.

1) I would recommend instead of using 'ita.txt' as the default dictionary. could you maybe use a generic name like 'defdict.txt'
2) Is there a 'clear' button so we don't have to hit backspace to remove the text.
3) On the <Next/Prev> the text description is updated but the word that is being defined is not updated.

These are all good suggestions, but there still has to be a more efficient way to search large files. Can we get Kovid in here to tell us how to jump to a point in a file? If we're searching through a dictionary where all it can progressively look for longer and longer letter combinations and have a "smart" jump around it shouldn't have to read more than 40-50 lines to find any word I would think. It just needs to be able to hone in on the spot (and it should also have a way of ignoring junk lines that are out of place and don't fit in with the lines around it). If there's a way to jump to a line within a text file, I could describe a simple search algorithm for alphabetical lists and you could program it without too much trouble I would hope.

[spooky69]

Also, how about a "search status" icon, even if it's just an indicator that pops up in the corner while the search is active.
I also wouldn't mind a different control scheme...it's awkward to use both the directional pad and the numbers at the same time. Plus, it's really easy to accidentally hit the enter button and have to wait through a long search you didn't want to do (with no indication that it's searching other than it won't let you input). There should be an options menu that lets us switch to a scheme where the page turn buttons move the cursor on the keyboard and the d-pad is used to look up the next definition. With that scheme you can type one-handed with all of the buttons on the right side or have your left hand on the alternate bottom-left page turn button to move the cursor (seems much more handy than using the d-pad since you're going to be using the cursor movement a lot more than the "next word" feature).

[Clemenseken]

@spooky69:

Hi!
As I mentioned in my very first post to this thread, this was meant to be a simple dictionary launcher.
I am not a programmer - and do not plan to buy or sell Yahoo or Microsoft, either.
My aim is not to be better than the Oxford Concise Dictionary or the like.
I just felt the lack of any dictionary for my little Sony-500. That's why I took the trouble to find ways to get a decent translation from other languages on this device.

I fully understand your wishes, demands and suggestions. One or two of these also came to my mind but were not that important at the moment to give them first priority.
There are other activities in my day-to-day life apart from delivering input for this project.

For example I am about to finish a game for my Sony (somewhat similar to Sudoku). That has mainly been on my mind for the last 2 weeks - when doing any programming at all ...

I probably will not ask igorsk for help on the improvements you suggested in your last two posts (I did test functions like "match" or "exec" with "regExp" - but there is no documentation for javascript on Sonys - so it's quite some trouble to get any progress. I am still trying...)
"match" (in a String) actually runs a lot faster that "readLine", I admit. But it does not work with files significally larger than one MB. There may be natural limits to "String" or I missed some detail required.

But anyway I don't mind anybody using my code so far as base for an application of their own.
So there's still hope for you: Just go ahead!

It may take another week or two until I have results for my tests. It might take a little longer, then...
( I will ask my product marketing team next time what they think I should promise or announce...)

[lisak]

Ok, so I finaly found some time to look at the source code and do some tests. I am especialy eager to find the solution for the large dictionary files, but so far it seems that I am stuck. ;(

Basicaly, I wasn't able to find any other method for reading from Stream object than readLine, which is used exclusively in all Sony Reader js applications I've found. stream.read(n) doesn't work, neither does stream.skip(n). Unless we find some way how to traverse through the stream faster, we won't be able to get significant speed boost, I am afraid.

I hoped I would be able to find something in /opt files in my PRS, but unfortunately, igorsk ebook.py dumper doesn't work for me (probably because I have PRS505) - it says that the device is not connected.

Does anybody have the idea how to progress? A dump of scripts from PRS might be usefull, maybe together with tinyhttp and libtinyhttp.so files, if somebody has it at hand...


EDITED:
Ok, I wish I read the previous posts here before I wrote this one - grep and runCommandResult would be probably the way to go.

[=X=]

Originally Posted by spooky69

These are all good suggestions, but there still has to be a more efficient way to search large files.

@spooky69
I realized my post was misleading. The 2nd half which you quoted me on was not my recommendation. More suggestions for a cleaner experience.

My real suggestion for finding a word was using a "binary search" algorithm. I aplogoize for those that are not programmers and do not know what that means. Let me give a quick "Laymans summeary"

The binary search is a very very simple search designed for speed and simplicity. Essentially what it does is search through a file/array and cuts the search by half every time reducing the search time by half every time. This gives a mathematical search time of log n.


Here is the algorithm.
What you need to know before the algorithm starts.
* The array size/file size in line numbers
* Must be sorted (dictionaries usally are)

Say we have a file 26 lines long. Each line is the first letter of the alphabit
>We are looking for the word J
MIN = line 1
MAX = line 26
As we do our algorithm the boundary of the MIN/MAX will shrink as we get
closer.

1) Jump to the half the current size(MAX-MIN). We jump to line 13. That is the letter is "M"
2) Compare "J" to "M". "M" is too large so we search down
3) Cut the max size from 26 to 13.
4) Now look at 1/2 the new MIN/MAX(13-1)/2. We get 6.5 so we round to 7.
5) Compare "J" to the MIN+7. That is the letter "G". "J" is larger than "G" so we set the MIN to 7
6) Now take the 1/2 of the new MIN/MAX = (13-7)/2 = 3
) Compare "J" to the MIN+3 = 10 (remember the new min is 7)
7) BINGO we found a match.

If you notice the sequential search would have taken us 10 reads to find "J". The "binary search" algorithm only took 3 reads.

The real hard part is SONY's API since it is not published I don't know if there is a way to jump to line numbers.


__
Cheers
=X=