[Kindle Touch] Localization

[eureka]

Originally Posted by ixtab

Just took a look at it. From the ad, I'm still left slightly confused. Would this allow for a SSH account which can run Java and python (that should do), trigger cronjobs to create bundles, say every day, or even every hour, and then make these bundles available via HTTP? If so, yes, we could think about providing a service which would continuously provide the latest (automated) builds. Then again, every translator can easily create bundles by himself, so I think this is not strictly needed. But it'd certainly be a nice-to-have thing.

Ooops, I'm not so sure about Java and Python at the same time... But all other features are provided judging by their's docs and my small experience (I was able to host static HTML file and SSH into my account. It was account with Python service enabled and I saw Python executable there. But I didn't saw Java executable... Though Java stack is an official option. And I didn't tried to setup cronjobs.) I think with SSH it's possible to download static version of Java...

Originally Posted by ixtab

Why so?

I'm just jokingly deducing the term from the term of working on JS tool.

[ixtab]

I tried it out, but it doesn't seem to be as flexible as we need it. I do have a dedicated root server somewhere, which could probably take care of this, however I don't want to completely reconfigure it. So as a first step, do you think you could make your tools compatible with python 2.6? While installing 2.7 on Ubuntu LTS works, it's a bit adventurous, and I don't know what other side-effects it may have...

[eureka]

Originally Posted by ixtab

I tried it out, but it doesn't seem to be as flexible as we need it. I do have a dedicated root server somewhere, which could probably take care of this, however I don't want to completely reconfigure it. So as a first step, do you think you could make your tools compatible with python 2.6? While installing 2.7 on Ubuntu LTS works, it's a bit adventurous, and I don't know what other side-effects it may have...

OK, I'd try to update the tool right now. Python 2.7 dependency is only for bundled OrderedDict class and argparse library and I think, I saw, both of these features as separate modules for Python < 2.7.

UPD Did it. Will test on Python 2.6 in future.

[JustAMan]

Originally Posted by ixtab

@JustAMan: *missing* entries in a locale file should normally not be an issue (because they will be taken from the default entry then). However, *wrongly translated* ones are probably the cause of all this weirdness. I don't have the time to do it, but try to write a script which finds entries where
- the german "translation" is the *same* as the original en_US one
- and the russian one is *different* from the original.

These are likely to cause problems.

Update: This task will probably be much simpler if you work on the source .properties files ("tx pull -a" first, then look around the files). I'm only referring to the de translation as the "reference implementation" because it seems to be working correctly for everyone, and it's 100% complete.

I wrote the tool to compare localizations.

What it does is:
* takes some "model" language that seems to be working fine (e.g. "de")
* compares all de.properties and en_US.properties to find out what properties inside those are identical
* compares all other languages .properties and en_US.properties
* generates .html reports for all other languages showing folders, property names & values if:
- value for given property name is different for given language and en_US
AND
- value for given property name is identical for en_US and "model" language

That way it should be easier to find out "out-translated" localizations.

I'm attaching the tool and its report: loc-report.zip
To use the tool do "tx pull -a -s" and run "python report.py <model_language>" in tx working directory.
Those reports are generated with German localization as a model.

ixtab,
Can you have a look at those reports? I'm especially interested in Russian one, you know but this tool should generate reports useful for other languages as well.
Maybe by analyzing this we can create a list of properties that shouldn't be translated, rather than say vaguely "don't translate something that is too short".

[ixtab]

Thanks for that!

I can't run the report.py by myself, wherever I try to run it, I keep getting

Traceback (most recent call last):
  File "tool/report.py", line 5, in <module>
    all_lngs.remove(source)
KeyError: 'en_US'

Since I'm a total noob with python (and honestly I hate its syntax), I don't even dare to try to figure out what I'm doing wrong. So, what am I doing wrong?

However, I took a look at the output, and it's pretty much what I expected. Most of the Strings I listed in the announcement are wrongly translated. (There are a few innocent ones, like "OK" or "International" as well). It's probably not a bad idea to generate an informative list of "do-not-translate" properties. However, it will unfortunately be of limited usefulness, because translators do not see the property names on Transifex, only the values. I think the report is very useful and helps to give pointers for manual fixing for "advanced" users. (Update) One more suggestion: leave out empty (=untranslated) strings from the report. As said, they are ok because the base version will be used.

Also, thanks for your offer of the 5.0.0 files, but I'd think that this might lead to even more trouble. I'll just stick with 5.0.1 for the time being. Maybe if I'm feeling adventurous someday, I just try to install 5.0.3. Does a failed update give any information (in the log or so) about *which* files it complained about?

[JustAMan]

Originally Posted by ixtab

Thanks for that!

I can't run the report.py by myself, wherever I try to run it, I keep getting

Traceback (most recent call last):
  File "tool/report.py", line 5, in <module>
    all_lngs.remove(source)
KeyError: 'en_US'

Since I'm a total noob with python (and honestly I hate its syntax), I don't even dare to try to figure out what I'm doing wrong. So, what am I doing wrong?

You didn't download source language it seems
Remember I said do "tx pull -a -s"?
Or you're running it from the wrong place. You have to run it from the root folder where you did "tx init" and "tx pull".
Agreed that I should handled it better but, oh man, I'm lazy!
Maybe in next version...

Originally Posted by ixtab

Thanks for that!
However, I took a look at the output, and it's pretty much what I expected. Most of the Strings I listed in the announcement are wrongly translated. (There are a few innocent ones, like "OK" or "International" as well). It's probably not a bad idea to generate an informative list of "do-not-translate" properties. However, it will unfortunately be of limited usefulness, because translators do not see the property names on Transifex, only the values. I think the report is very useful and helps to give pointers for manual fixing for "advanced" users.

I suppose the list would have some use, like "don't integrate this and that property into locale-*.jar" when compiling it. That would be more error-proof than current situation

Originally Posted by ixtab

(Update) One more suggestion: leave out empty (=untranslated) strings from the report. As said, they are ok because the base version will be used.

I got the idea that missing property is OK. Now you say that empty property is OK... I'm confused. Which one is fine, or they are both fine?

Originally Posted by ixtab

Also, thanks for your offer of the 5.0.0 files, but I'd think that this might lead to even more trouble. I'll just stick with 5.0.1 for the time being. Maybe if I'm feeling adventurous someday, I just try to install 5.0.3.

No problem. Until I update my Kindle stands ready
btw, just thought about dd'ing it just in case...

Originally Posted by ixtab

Does a failed update give any information (in the log or so) about *which* files it complained about?

Nope. I've attached parts of "showlog" on some previous pages - nothing really suspicious except that "Callback thing.
Looks like I'll have to figure it out experimentally.

[ixtab]

Originally Posted by JustAMan

You didn't download source language it seems
Remember I said do "tx pull -a -s"?
Or you're running it from the wrong place. You have to run it from the root folder where you did "tx init" and "tx pull".

Of course I have everything here, including the en_US sources
I also made sure to run it from the root dir. Still, same error.

Originally Posted by JustAMan

I suppose the list would have some use, like "don't integrate this and that property into locale-*.jar" when compiling it. That would be more error-proof than current situation

Agreed. But this will again have to be maintained manually...

If someone can come up with a complete list of properties to ignore (which Bundle(class name), which property), I'm ok to include it in the tool though.

Originally Posted by JustAMan

I got the idea that missing property is OK. Now you say that empty property is OK... I'm confused. Which one is fine, or they are both fine?

See a few pages up. Transifex unfortunately produces empty strings from missing translations, so in this context "missing" and "empty" are equivalent.

[JustAMan]

Originally Posted by ixtab

Of course I have everything here, including the en_US sources
I also made sure to run it from the root dir. Still, same error.

Strange... Try this attachement report.zip, it prints the list of languages it detected prior doing things. Also I think I fixed showing empty strings.

Originally Posted by ixtab

Agreed. But this will again have to be maintained manually...

If someone can come up with a complete list of properties to ignore (which Bundle(class name), which property), I'm ok to include it in the tool though.

That would have to wait 'till the list is composed... Maybe it can be based on what I'll find in ru_RU locale.

Originally Posted by ixtab

See a few pages up. Transifex unfortunately produces empty strings from missing translations, so in this context "missing" and "empty" are equivalent.

Ah, got it. I tried to fix the script to skip those, it works for ru_RU at least....

[ixtab]

Nope. Finds no languages. Well, I did look inside it, and I think your glob is wrong ("*/*.properties")?

Here's what the directory structure actually looks like:

$ find . -name "*.properties"|grep en_US|head -n 1
./src/5.0.0/framework/com/amazon/ebook/util/resources/StringUtilResources_en_US.properties

[JustAMan]

o_O
Here's what I get when I did "tx pull -s -a":

$ find . -name \*.properties|head -n 5
./translations/kindle-touch-framework.5-0-0--c-a-k-k-i-Kindlet/es.properties
./translations/kindle-touch-framework.5-0-0--c-a-k-k-i-Kindlet/pl.properties
./translations/kindle-touch-framework.5-0-0--c-a-k-k-i-Kindlet/eu.properties
./translations/kindle-touch-framework.5-0-0--c-a-k-k-i-Kindlet/th.properties
./translations/kindle-touch-framework.5-0-0--c-a-k-k-i-Kindlet/it.properties

and so on.
I'm running my script in ./translations directory.
Something is really weird here... Maybe my "tx" is different from yours?
I'm using the one that is in Fedora 16 repositories.

I suppose I can adapt the tool to both layouts, however I'd rather understand why there's layout difference in the first place.

[ixtab]

The difference comes from your using the "auto-install" feature of transifex. This is not recommended. The recommended way is to use the .tx/config from the git repository (Or to simply clone the repository in the first place). This gives the correct directory layout and file names.

[JustAMan]

I see... Now I remember reading something about this config in the thread.
I've cloned the repo.

I'll fix my report.py tomorrow, stay tuned

[eureka]

Originally Posted by ixtab

However, I took a look at the output, and it's pretty much what I expected. Most of the Strings I listed in the announcement are wrongly translated. (There are a few innocent ones, like "OK" or "International" as well). It's probably not a bad idea to generate an informative list of "do-not-translate" properties. However, it will unfortunately be of limited usefulness, because translators do not see the property names on Transifex, only the values.

There is a way to see property names on Transifex. In "General Settings" you should choose "Verbose editing".


Or just use this bookmarklet:

javascript:(function(){var cb=$('#toggle_verbose'), p=$('.source_entity');if(cb.attr('checked')){cb.attr('checked',false);p.hide()}else{cb.attr('checked',true);p.show()};})();

Originally Posted by ixtab

Also, thanks for your offer of the 5.0.0 files, but I'd think that this might lead to even more trouble. I'll just stick with 5.0.1 for the time being. Maybe if I'm feeling adventurous someday, I just try to install 5.0.3. Does a failed update give any information (in the log or so) about *which* files it complained about?

Yes. You could see it in 5.0.3 update script:

Spoiler:

source /etc/upstart/functions

logmsg()
{
    f_log $1 ota_install $2 "$3" "$4"
}

perform_patches()
{
  logmsg "I" "patch" "patchinfo=\"/bin/busybox\"" "patching file"

  mkdir -p "patch/bin/"
  bspatch "/bin/busybox" "patch/bin/busybox" "000.busybox.patch"
  if [ $? -ne 0 ]; then
    logmsg "C" "patch" "patchinfo=\"/bin/busybox\"" "patch failure"
    return 1
  fi

  _PATCH_MD5=`md5sum "patch/bin/busybox" | awk '{ print $1; }'`
  _EXPECTED_MD5=26ec0ba302a8066a6efea14d5e19cf1c
  if [ ! "${_PATCH_MD5}" = "${_EXPECTED_MD5}" ]; then
    logmsg "C" "patch" "patchinfo=\"/bin/busybox\"" "checksum failure"

    _CURRENT_MD5=`md5sum "/bin/busybox" | awk '{ print $1; }'`
    logmsg "I" "patch" "current_md5=${_CURRENT_MD5},expected_md5=${_EXPECTED_MD5},patch_md5=${_PATCH_MD5}" "checksum verify"

    return 1
  fi

  # ...skipped...
}

Originally Posted by ixtab

Nope. Finds no languages. Well, I did look inside it, and I think your glob is wrong ("*/*.properties")?

Here's what the directory structure actually looks like:

$ find . -name "*.properties"|grep en_US|head -n 1
./src/5.0.0/framework/com/amazon/ebook/util/resources/StringUtilResources_en_US.properties

I think JustAMan didn't used our .tx/config and just pulled whole project...

[JustAMan]

Here's new version, renamed to checkit.py, with some reports attached: newreport.zip
Try "python checkit.py -h" for some help on usage.

BTW, ixtab, did you fix yesterday some localizations? I don't find much stuff in new report that was found in old, and I don't know whether it's my bug or you fixed the translations.

I also didn't commit my tool to git repo you provided as I don't think I have access. You can either give me access to commit or commit it yourself if you think it's worth being there.

UPD: it turned out that I ran my report tool on outdated source. eureka says transifex is the freshest one, so I'm going to do "tx pull -a" in git-cloned stuff and re-run the report.

[eureka]

Originally Posted by JustAMan

I also didn't commit my tool to git repo you provided as I don't think I have access. You can either give me access to commit or commit it yourself if you think it's worth being there.

Fork repo on Bitbucket, add your tool and make a pull request. I'll then give you commit rights at the original repo.