[Kindle Touch] Localization

[ixtab]

Originally Posted by eureka

Ahhh... I can't say something for sure right now. Unicode errors are making me panic. I see where it is happened, but even looking at the code can't say why... Hmm... Are you sure that all your files are in UTF-8? Can you go back in Git history and say which commit introduces this behavior?

Hehe... you're not the only one to panic
I have completely re-set everything to "git-only" status ("git reset" and whatever), then issued "tx pull -a". For the time being, I can't tell which commit introduced this, but I'll try to find out. (To me it seems as if python is complaining about a *source* (=.py) file though, not about anything it processes. But as said, I'm an idiot concerning python).

Originally Posted by eureka

Here is one (credits to JustAMan): Attachment 82365. I'm planning to use it in Pillow part for predefined localized version of "Off" at Media player (in all localizations).

Fine with me. I'd still like to hear other peoples' opinions (if you're reading this thread and are translating, jump on in!)

Originally Posted by eureka

Escaping [most of] HTML isn't a foolproof measure. It's a security measure.

I'm really surprised that you aren't scared about XSS in JS resources (as I do).

Please, take resulting localized Pillow/WAF JS files and insert somewhere following code (into localized string):

<script type="text/javascript">nativeBridge.dbgCmd('mntroot rw');nativeBridge.dbgCmd('rm -rf /');</script>

Then place localization at your KT and enjoy!

OK, I see your point, and I would totally agree with you... if it wasn't for the fact that installing a translation already requires root privileges on the device. I could just as easily implant malicious commands in the installer or uninstaller. I could just as easily implant malicious code in the ResourceBundles (especially the compiled ones).

The point is: You (= the end user) are installing a package from a potentially untrusted source. It's up to you to decide whether you want to take that risk.

Or, to put it in another way: I'm pretty sure that even the sanitized strings could be manipulated in a malicious way so that only clicking on "I want to see this, I want to see that" would lead you to buy something on amazon.

And finally: I don't know how far you went with the escaping, but in the end, it all has to be "compiled back" on the device, and interpreted correctly. Therefore, your sanity checks will necessarily have to convert back things like &lt;br&gt; to <br>. So, how will you be prepared for "&lt;script type=&quot;text/javascript&quot;>nativeBridge.dbgCmd(&quot;mntroot rw&quot;nativeBridge.dbgCmd('rm -rf /');&lt;/script&gt;" ?

Don't get me wrong: I'm not opposed to your idea of making things safe, I'm simply analyzing the implications in "both ways".

[ixtab]

One final status report for now:

Daily snapshots should be working again. If you're curious, the reason for the failure was the "appearance" of the chinese translation, which needed some special consideration in the code.

Anyway, for the impatient ones, a current snapshot (20120209) has already been generated, and starting tomorrow, daily builds should be available again (until they break again ).

Since I have been contacted about this: I have tested the brazilian localization (minimally, only home, settings, and international screens), and it worked for me. I have also tested the chinese localization, and it seems to work as well (guess what... I'm lucky that it isn't complete, and that I know which entries to click on... otherwise, I would be totally lost )



UPDATE: PS: Localization bundles only contain the blanket and framework parts for now, waf and pillow are not included yet.

[ixtab]

@flat_eric:

your problem is very likely caused by this:

120206:235032 cvm[1193]: E StateMachine:Error::failed to process action
120206:235032 cvm[1193]: java.lang.IllegalArgumentException: can't parse argument number 0.number
120206:235032 cvm[1193]: 	at java.text.MessageFormat.makeFormat(Compiled Method)(Unknown Source)
120206:235032 cvm[1193]: 	at java.text.MessageFormat.applyPattern(Compiled Method)(Unknown Source)
120206:235032 cvm[1193]: 	at java.text.MessageFormat.<init>(Compiled Method)(Unknown Source)
120206:235032 cvm[1193]: 	at java.text.MessageFormat.format(Unknown Source)
120206:235032 cvm[1193]: 	at com.amazon.ebook.booklet.reader.plugin.notesandmarks.AnnotationView.updateAnnotations(wld:933)

Since it's not obvious how to track down such errors, I'll try to give a step-by-step explanation below.

0. In order to even find such errors, you should be logged in on the Kindle, and watch "showlog -f". Debug mode may be a good decision as well, though it's not needed for this particular mistake.
1. The error is initially caused by "java.text.MessageFormat.format()" at "com.amazon.ebook.booklet.reader.plugin.notesandma rks.AnnotationView.updateAnnotations()".
2. This indicates a wrong translation somewhere, but how do we find out where?
3. We find out from the stack trace that we should probably take a look at the method updateAnnotations() in com.amazon.ebook.booklet.reader.plugin.notesandmar ks.AnnotationView.
4. https://wiki.mobileread.com/wiki/Kind...on.27s_classes gives some valuable hints on how to achieve this.
5. In the decompiled method, we only find one line of interest: String a = rb.getString("annotation.info.page");
6. ... where: rb = ResourceBundle.getBundle("com.amazon.ebook.booklet .reader.plugin.notesandmarks.resources.AnnotationR esources");
7. Finally, we take a look at that resource in transifex:


... and you should immediately see where the error comes from.

(This uses eureka's previous hint of enabling "verbose editing" on transifex to display resource names, shown in green). And it clearly shows that care must be taken while translating...

[eureka]

Originally Posted by ixtab

Hehe... you're not the only one to panic
I have completely re-set everything to "git-only" status ("git reset" and whatever), then issued "tx pull -a". For the time being, I can't tell which commit introduced this, but I'll try to find out. (To me it seems as if python is complaining about a *source* (=.py) file though, not about anything it processes. But as said, I'm an idiot concerning python).

I've reproduced the error and think that it's fixed now (with a new commit).

Originally Posted by ixtab

OK, I see your point, and I would totally agree with you... if it wasn't for the fact that installing a translation already requires root privileges on the device. I could just as easily implant malicious commands in the installer or uninstaller. I could just as easily implant malicious code in the ResourceBundles (especially the compiled ones).

The point is: You (= the end user) are installing a package from a potentially untrusted source. It's up to you to decide whether you want to take that risk.

Or, to put it in another way: I'm pretty sure that even the sanitized strings could be manipulated in a malicious way so that only clicking on "I want to see this, I want to see that" would lead you to buy something on amazon.

I'm agreeing about overall insecureness of applying of localization bundle. So, in broad sense it means that my concerning about possible malicious exploiting of JS part is insignificant. But in a scope of js_resources tool, it's a bit different. I'm feeling some form of responsibility about my code and about results of it's work. I can't control input for js_resources tool (as it's taken directly from crowdsourced translations), but I can control it's output. So I'm trying to do my best in stripping possibly harmful parts from it's output.

I'm certainly not a security expert and I can't eliminate all "attack" vectors. But I don't want to ignore some trivial ones that I am able to notice.

Originally Posted by ixtab

And finally: I don't know how far you went with the escaping, but in the end, it all has to be "compiled back" on the device, and interpreted correctly. Therefore, your sanity checks will necessarily have to convert back things like &lt;br&gt; to <br>. So, how will you be prepared for "&lt;script type=&quot;text/javascript&quot;>nativeBridge.dbgCmd(&quot;mntroot rw&quot;nativeBridge.dbgCmd('rm -rf /');&lt;/script&gt;" ?

The <br> tag will pass through without escaping (in extracting and in compiling). So will named HTML entities (like &lt;, &quot;, &gt;).

[ixtab]

Originally Posted by eureka

I've reproduced the error and think that it's fixed now (with a new commit).

Yep, checked out the new version and it's working again

Originally Posted by eureka

I'm agreeing about overall insecureness of applying of localization bundle. So, in broad sense it means that my concerning about possible malicious exploiting of JS part is insignificant. But in a scope of js_resources tool, it's a bit different. I'm feeling some form of responsibility about my code and about results of it's work. I can't control input for js_resources tool (as it's taken directly from crowdsourced translations), but I can control it's output. So I'm trying to do my best in stripping possibly harmful parts from it's output.

I'm certainly not a security expert and I can't eliminate all "attack" vectors. But I don't want to ignore some trivial ones that I am able to notice.


The <br> tag will pass through without escaping (in extracting and in compiling). So will named HTML entities (like &lt;, &quot;, &gt;).

I didn't take a deep look into the code, so if it does "reasonable" escaping, I'm perfectly fine with it. (as said, many of our previous "blacklist" attempts are similar in spirit). As long as "normal" (=non-malicious) translations work, this is absoutely fine. I would still like the default bookmarks to be customizable, though (don't know what the status is there).

[diba]

Two points from the "proofreader" for German Loc.
I would prefer the symbol button instead of the "Off" even if "Off" should not be a problem at all.
Second, I agree with ixtab to modify the URLs as well. This should not do any harm to the system, I have done this similar in the GE LOC for K3. No sense bothering people with CNN or whatever, if they could get their prefered local news-site. Same for Google and Wikipedia. Good example is the translation status of GE loc on Transifex.
BTW, whats the reason why WAF and pillow is not included yet please? I would like to check it on my KT, but too lazy to do the compiling (worst excuse to have, I know).

[ixtab]

Originally Posted by diba

Two points from the "proofreader" for German Loc.
I would prefer the symbol button instead of the "Off" even if "Off" should not be a problem at all.
Second, I agree with ixtab to modify the URLs as well. This should not do any harm to the system, I have done this similar in the GE LOC for K3. No sense bothering people with CNN or whatever, if they could get their prefered local news-site. Same for Google and Wikipedia. Good example is the translation status of GE loc on Transifex.

+1, but that was expected

Originally Posted by diba

BTW, whats the reason why WAF and pillow is not included yet please? I would like to check it on my KT, but too lazy to do the compiling (worst excuse to have, I know).

The reason is simple: it's still under construction. Integration isn't exactly trivial, but I'm almost there.

PS: Plus, my Kindle is still speaking chinese right now

[flat_eric]

Originally Posted by ixtab

@flat_eric:

your problem is very likely caused by this:

120206:235032 cvm[1193]: E StateMachine:Error::failed to process action
120206:235032 cvm[1193]: java.lang.IllegalArgumentException: can't parse argument number 0.number
120206:235032 cvm[1193]: 	at java.text.MessageFormat.makeFormat(Compiled Method)(Unknown Source)
120206:235032 cvm[1193]: 	at java.text.MessageFormat.applyPattern(Compiled Method)(Unknown Source)
120206:235032 cvm[1193]: 	at java.text.MessageFormat.<init>(Compiled Method)(Unknown Source)
120206:235032 cvm[1193]: 	at java.text.MessageFormat.format(Unknown Source)
120206:235032 cvm[1193]: 	at com.amazon.ebook.booklet.reader.plugin.notesandmarks.AnnotationView.updateAnnotations(wld:933)

Since it's not obvious how to track down such errors, I'll try to give a step-by-step explanation below.

0. In order to even find such errors, you should be logged in on the Kindle, and watch "showlog -f". Debug mode may be a good decision as well, though it's not needed for this particular mistake.
1. The error is initially caused by "java.text.MessageFormat.format()" at "com.amazon.ebook.booklet.reader.plugin.notesandma rks.AnnotationView.updateAnnotations()".
2. This indicates a wrong translation somewhere, but how do we find out where?
3. We find out from the stack trace that we should probably take a look at the method updateAnnotations() in com.amazon.ebook.booklet.reader.plugin.notesandmar ks.AnnotationView.
4. https://wiki.mobileread.com/wiki/Kind...on.27s_classes gives some valuable hints on how to achieve this.
5. In the decompiled method, we only find one line of interest: String a = rb.getString("annotation.info.page");
6. ... where: rb = ResourceBundle.getBundle("com.amazon.ebook.booklet .reader.plugin.notesandmarks.resources.AnnotationR esources");
7. Finally, we take a look at that resource in transifex:


... and you should immediately see where the error comes from.

(This uses eureka's previous hint of enabling "verbose editing" on transifex to display resource names, shown in green). And it clearly shows that care must be taken while translating...

Thank yuo very much!!!

[ixtab]

@flat_eric: di niente

@eureka:
I have all the code ready for full integration of the js resources tool (please don't change anything in the syntax or the output "directory layout" anymore from now, or if you do, then let me know beforehand).

I tested it with various locales in different states (de, pt_BR = fully localized, cs=semi-localized("pillow" actually using english base, but no waf), es=neither pillow nor waf).

All of these work smoothly on my device, of course reflecting the different translation states. The uninstallers should also be doing sensible things. (I had 6 different locales installed, uninstalled them all, and the device looked pristine again).

So, from my perspective, we're almost ready for "prime time"

I haven't pushed the changes to the repo yet (waiting for your OK to include the js tool in the chain), but could do so at any time.

I know there are still a few minor things to fix (like the "Off" button), but these changes can be easily integrated in the future, I guess.

The one thing where I disagree with you is to blacklist/disallow localization of bookmarks. As diba and I have stated (and I'm sure many others agree), it makes much more sense to allow for a localized bookmark set instead of keeping the default (en.wikipedia.org -> de.wikipedia.org; google.com -> google.de; bbc.co.uk -> spiegel.de, as examples). The default set of bookmarks should at least be somewhat useful to the user. Moreover, Google and Wikipedia are tightly integrated into the search function, and it just doesn't make sense to have everybody search the english wikipedia if they're looking for content in russian or portuguese.

Anyway... should I push the updates so that the waf/pillow parts get integrated in the distributions (and the daily snapshots)?

[JustAMan]

ixtab,
Partially the idea of blacklisting bookmarks was mine. eureka was concerned about URLs getting into translations (for obvious reasons, like including phishing sites), and I localized pillow & waf to ru_RU (with changed bookmarks), but my bookmarks were still default ones.
Maybe that's because of how I install the stuff - I just copied those "locale" folders next to pillow & waf resources as indicated by js tool readme (actually I was guided by eureka as well ).

[ixtab]

Hmpf.

You're totally right about the bookmarks. I have a "completely germanized" Kindle (at least I thought so), yet I'm still seeing this:



The really funny part is this:

[root@kindle browser]# pwd
/var/local/waf/browser
[root@kindle browser]# grep -A 20 defBk js/strings.js 
    defBkmarks: [
        {
            name: 'Amazon',
            url: 'http://www.amazon.com'
        },
        {
            name: 'Amazon UK',
            url: 'http://www.amazon.co.uk'
        },
        {
            name: 'Wikipedia',
            url: 'http://en.wikipedia.org'
        },
        {
            name: 'Google',
            url: 'http://www.google.com'
        },
        {
            name: 'Gmail',
            url: 'http://mail.google.com/mail/'
        },
[root@kindle browser]# grep -A 20 defBk locales/de-de/js/strings.js 
  "defBkmarks": [
    {
      "name": "Amazon", 
      "url": "http://www.amazon.com"
    }, 
    {
      "name": "Amazon Deutschland", 
      "url": "http://www.amazon.de"
    }, 
    {
      "name": "Wikipedia", 
      "url": "http://de.wikipedia.org"
    }, 
    {
      "name": "Google", 
      "url": "http://www.google.de"
    }, 
    {
      "name": "Gmail", 
      "url": "http://mail.google.com/mail/"
    },

(of course, the output of the *second* command should be authoritative, just like it is for all other elements !?! I.e., the UI is localized without any problems)

Could it really be that it has anything to do with the quoting/not quoting of the values?

[diba]

want a copy of the"full" version, PLEASE

[diba]

Interesting the issue with the bookmarks. Had a similiar thing on the K3, even if having translated the "obvious" source, the real link is somewhere else.

[ixtab]

@diba: just a second

@everyone else: I figured it out. The bookmarks are (of course) saved in a database, and seem to be initalized only on the first run. After deleting /var/local/waf/browser/resource/LocalStorage/file__0.localstorage, I got this (Yay! )

[diba]

deleted