[Kindle Touch] Localization

[ixtab]

Hehe, I think we scared everybody else away from this thread.

Is there anything I can do while you're at the JS part? Since we're almost there (at least conceptually), I was thinking about starting to plan for a tool which would bundle up all localized resources. A couple of questions come to mind:

• Package type: I'm in favor of providing an update*.bin as installer/uninstaller. Any other thoughts?
• Should packages include or exclude the little hack to enable the "international" settings? Pro inclusion: 1-stop-installation. Pro exclusion: independent update of that file (in case it ever becomes necessary for new firmware) + strictly speaking, it contains (modified) amazon code.
• We will need one final element for each supported language, namely the .properties file that identifies the locale (also described in the previous link). We cannot do that through transifex. But since the file is so small, I guess we can arrange for some sort of "out-of-band" mechanism to create these files.

Concerning the last topic: I suggest to place these files in a 4th directory, along with the framework, locale, and (waf/pillow?) one. Maybe "manifest" would be a good name. That directory would contain only "<locale>.properties" files for each supported locale.

[geekmaster]

Originally Posted by ixtab

Hehe, I think we scared everybody else away from this thread.

Not quite *everybody*... Some of us [lurkers] just have nothing to contribute to this topic (yet).

[eureka]

Originally Posted by ixtab

Is there anything I can do while you're at the JS part?

Yes. Please, be patient while I'm trying to do my best. Parsing is already done. Experiments with XLIFF source files showed that Transifex support for XLIFF is quirky, so I've reverted to idea of outputting to MOZILLAPROPERTIES with metadata in external JSON file. (Metadata is required because JS files structure is not so uniform as Java.)

Originally Posted by ixtab

Since we're almost there (at least conceptually), I was thinking about starting to plan for a tool which would bundle up all localized resources. A couple of questions come to mind:

• Package type: I'm in favor of providing an update*.bin as installer/uninstaller. Any other thoughts?
• Should packages include or exclude the little hack to enable the "international" settings? Pro inclusion: 1-stop-installation. Pro exclusion: independent update of that file (in case it ever becomes necessary for new firmware) + strictly speaking, it contains (modified) amazon code.
• We will need one final element for each supported language, namely the .properties file that identifies the locale (also described in the previous link). We cannot do that through transifex. But since the file is so small, I guess we can arrange for some sort of "out-of-band" mechanism to create these files.

Concerning the last topic: I suggest to place these files in a 4th directory, along with the framework, locale, and (waf/pillow?) one. Maybe "manifest" would be a good name. That directory would contain only "<locale>.properties" files for each supported locale.

+1 for update*.bin. Yifan Lu's KindleTool is great for this purpose.

-1 for including of UI hack into distribution (update bundle or anything).

.properties for locale identification could be generated automatically. Locale code (i.e. locale and locales.supported) could be extracted from localized resource file name and locale's name (i.e. display.name) could be extracted from mapping "locale code -> locale name" which is certainly located somewhere in Internets (and should be small enough for storing it in Git repo). UPD: found it. Common Locale Data Repository (or direct link to right file: http://unicode.org/Public/cldr/2.0.1/core.zip). Whole file content is too big, information could be extracted from 'common/main/en.xml'.

[ixtab]

Originally Posted by eureka

.properties for locale identification could be generated automatically. Locale code (i.e. locale and locales.supported) could be extracted from localized resource file name and locale's name (i.e. display.name) could be extracted from mapping "locale code -> locale name" which is certainly located somewhere in Internets (and should be small enough for storing it in Git repo). UPD: found it. Common Locale Data Repository (or direct link to right file: http://unicode.org/Public/cldr/2.0.1/core.zip). Whole file content is too big, information could be extracted from 'common/main/en.xml'.

It's not as easy as that, because there are some quirks with the Kindle. In fact, the KT is extremely picky about those .properties. Nearly everything that I assumed should work did not actually work... and that was only while editing that one file. Here's the usual output if it's not happy with something:

120117:222533 blanket[1140]: E langpicker:GET_LOCALE_POSIX_ID_FAILED:locale=de:getLocalePosixId() called on a locale ID that isn't installed on the system
120117:222533 blanket[1140]: E langpicker:LOCALE_NOT_INSTALLED:locale=(null):changeLocale to non-installed locale-- locale not changed

... at least, it doesn't crash and burn, but silently stays at en_US

In the last hours, I have thought of a different way, namely of having a special resource on transifex which will allow translators to provide the necessary information. I have temporarily uploaded it (but locked it for translation after making a sample one).

As translators are probably not following this thread closely, and don't need to understand all details, I'll just confront you with my idea: https://www.transifex.net/projects/p.../l/ru_RU/view/

This is essentially what you'd get to see as a translator. A "completed" sample is here: https://www.transifex.net/projects/p...ORE/l/de/view/

Do you think that this makes sense to translators, or is it too confusing? From these files, individual .properties could be extracted.

[eureka]

Originally Posted by ixtab

It's not as easy as that, because there are some quirks with the Kindle. In fact, the KT is extremely picky about those .properties. Nearly everything that I assumed should work did not actually work... and that was only while editing that one file. Here's the usual output if it's not happy with something:

120117:222533 blanket[1140]: E langpicker:GET_LOCALE_POSIX_ID_FAILED:locale=de:getLocalePosixId() called on a locale ID that isn't installed on the system
120117:222533 blanket[1140]: E langpicker:LOCALE_NOT_INSTALLED:locale=(null):changeLocale to non-installed locale-- locale not changed

... at least, it doesn't crash and burn, but silently stays at en_US

In the last hours, I have thought of a different way, namely of having a special resource on transifex which will allow translators to provide the necessary information. I have temporarily uploaded it (but locked it for translation after making a sample one).

As translators are probably not following this thread closely, and don't need to understand all details, I'll just confront you with my idea: https://www.transifex.net/projects/p.../l/ru_RU/view/

This is essentially what you'd get to see as a translator. A "completed" sample is here: https://www.transifex.net/projects/p...ORE/l/de/view/

Do you think that this makes sense to translators, or is it too confusing? From these files, individual .properties could be extracted.

I'm not objecting to your idea to make manifest (I like this name) available for translations. So if you'll decide to go this way, it's OK for me. The exact format of source file is of no matter for me (though these comments looks pretty useful).

But looking at Russian community of KT users (supposing that translators on Transifex.net are ordinary users) I'm doubting that someone will dare to check validness of translated manifest on his/her KT. So we'll end up with unchecked manifest... OK, Russian manifest will be checked by me. But what about other localized versions? Should it be checked manually by us also?

And I'm seeing the way to generate manifest automatically given localized resources pulled from Transifex.net (with locale code suffix or locale code in a name of directory) and information extracted from that Common Locale Data Repository.

locale=de
locales.supported=de-DE
display.name.de-DE=Deutsch
posix.id=en_US.UTF-8

• locale=de -- de is a part of locale code before hyphen
• locales.supported=de-DE -- de-DE is the whole locale code
• display.name.de-DE=Deutsch -- Deutsch could be found in CLDR at file common/main/de.xml as a value of XML element by path <ldml> -> <localeDisplayNames> -> <languages> -> <language type="de">
• posix.id=en_US.UTF-8, as you've said it should stay as-is for all locales. But even if it will be changed, we know all locales existing in KT firmware.

And German could be found at CLDR at file common/main/en.xml as a value of XML element by path <ldml> -> <localeDisplayNames> -> <languages> -> <language type="de">

It could be even possible to pre-generate manifests for all possible locales and check them in a automated manner (by running some script on KT and observing for any errors in log).

So far, I'm still OK with your solution. Just wanted to explain my point of view.

[ixtab]

Wow!... I admit I didn't look at that file as thoroughly as I should have, before assuming it wouldn't work -- simply because until now I had only seen incomplete or mutually conflicting files concerning locales, languages, codes etc.

But you're totally right - that file seems to contain everything that we need! (And what's more, it's internally consistent!)

So I have just found my new task: generate meaningful properties files for the locales that have been registered for translation.

@eureka: thanks a ton for the pointer and the clarifications.

[diba]

ixtab, I will have a look at the translation. Actually the site, I am supporting is changing the layout completely, and so I have to choose, where to spend my spare time.

[Novas]

Hi ixtab,
I have one question and I would like to know your opinion.
What about keyboard for specific country? In my native language(Czech), there are some special characters. You need to use it e.g. for searching, writing notes etc. I know how to do that (add new Czech keyboard). Of course I also need your locale hack to allow user choice keyboard.
Main question is should be keyboard part of localization or separate package?

[eureka]

Originally Posted by Novas

Hi ixtab,
I have one question and I would like to know your opinion.
What about keyboard for specific country? In my native language(Czech), there are some special characters. You need to use it e.g. for searching, writing notes etc. I know how to do that (add new Czech keyboard). Of course I also need your locale hack to allow user choice keyboard.
Main question is should be keyboard part of localization or separate package?

I'm not ixtab, but I have an experience about custom KT keyboard layouts, so I'll answer anyway
Novas, I've made cyrillic keyboard layout for Russian-speaking KT users as separate "hack", distributing it as update bundle. IMHO, creating of additional keyboard layouts should not be included in localization process, it's a separate (major) project.

[victor_2203]

Hello.
I'm very interested in the spanish localization. Will it be available? Can I help with the translation? I'm seeing that you are developing the localization to several languages. Is the spanish language included? Thanks.

[relaxpor]

I come from Thailand. I just recently coming to this forum.
I have basic knowledge about programming.

I would like to contribute to the kindle Society, especially localization things.
But i'm not sure how to start.

If anyone need my hand for anythings, please let me know.
I put all my effort here!

Best Regards,

[ixtab]

@victor_2203, relaxpor:
You are more than welcome to join the translation effort on https://www.transifex.net/projects/p/kindle-touch/ . Feel free to request to be added to an existing team, or to create a new one. Note that there are three subprojects, which all require translation.

In principle, we should be able to create packages for just about any language, given that the actual translations are available.

@Novas: I agree with eureka that keyboards should be separate from the localizations themselves. I have taken a look at the keyboard maps format, and it's not terribly difficult to understand. In fact, I would propose to create only very few layouts, instead of going through the hassle of creating one per language. For example, it would be easy to create a "universal" Latin keyboard, which could include all diacritics present in latin languages. The default english keyboard already includes many, but not all, of them. (Try long-pressing "s" on the english keyboard, you'll see that the " š " is already there. However, for example, " č " is missing). Of course, for non-latin languages, a completely different layout is probably required, but as said, a "universal" latin one should be doable, and is the best idea IMO, because everybody could benefit from it.

@eureka: I have made some progress with the manifest generation. Will check in the current state in a moment. Until then, here is preliminary output for all locales currently registered on transifex:

{locale=ru, posix.id=en_US.UTF-8, display.name.ru-RU=Русский (Россия), display.name=Russian (Russia), locales.supported=ru-RU}
{locale=cs, posix.id=en_US.UTF-8, display.name.cs-CZ=Čeština, display.name=Czech, locales.supported=cs-CZ}
{display.name.hu-HU=Magyar, locale=hu, posix.id=en_US.UTF-8, display.name=Hungarian, locales.supported=hu-HU}
{locale=de, posix.id=en_US.UTF-8, display.name=German, locales.supported=de-DE, display.name.de-DE=Deutsch}
{display.name.eu-ES=Euskara, locale=eu, posix.id=en_US.UTF-8, display.name=Basque, locales.supported=eu-ES}
{locale=hr, display.name.hr-HR=Hrvatski (Hrvatska), posix.id=en_US.UTF-8, display.name=Croatian (Croatia), locales.supported=hr-HR}
{locale=it, display.name.it-IT=Italiano, posix.id=en_US.UTF-8, display.name=Italian, locales.supported=it-IT}
{locale=es, posix.id=en_US.UTF-8, display.name.es-ES=Español, display.name=Spanish, locales.supported=es-ES}
{display.name.uk-UA=Українська (Україна), locale=uk, posix.id=en_US.UTF-8, display.name=Ukrainian (Ukraine), locales.supported=uk-UA}

[relaxpor]

Originally Posted by ixtab

@victor_2203, relaxpor:
You are more than welcome to join the translation effort on https://www.transifex.net/projects/p/kindle-touch/ . Feel free to request to be added to an existing team, or to create a new one. Note that there are three subprojects, which all require translation.

In principle, we should be able to create packages for just about any language, given that the actual translations are available.
[/CODE]

Thanks, ixtab.

I've already register to that website an request for a new team for translate Thai language. Since it's very new to me for this kind of website. I'm not sure how to start the translation. Please give me advise.

Best Regards,

[ixtab]

@relaxpor:

It's very simple actually. If you've never done this before, I suggest to use the Web interface. You can start with the "smallest" project, i.e. the gettext resources.

1. Go here (for example).
2. Click on any of the resources, then click "Translate now".
3. On the left side, you see the english text, on the right side is where you enter your translation.

Note that you will sometimes find special characters (like "choose %s as device language", or MessageFormat strings). Translate them to your language, while keeping the special characters etc.

Also note that not everything is actually meant to be translated. If you are unsure, it is best to either leave strings untranslated and come back later, or to simply copy the original text. See also this post.

As for trying out the translations, we are developing some tools, but they are not ready for "public use". If you're adventurous, you can keep track of the repository. Otherwise, you'd need to deploy things manually. Both of these are not suggested currently, unless you know what you are doing. It's still fairly simple to brick your device.

@eureka: I had been thinking about a less "brutal" way for semi-translated resources. Forcing 100% completion is technically ok, but it's not ideal, because people will probably tend to copy over the english strings, instead of leaving them untranslated. This makes it harder to find strings which actually are, in fact, still untranslated inside a "100% translated" resource.

For gettext, this is not a big issue, as there are only few strings. For Java, the problem is that untranslated keys get assigned an empty string as the value. However, since I'm parsing the resources anyway, I could simply throw away such key/value pairs. This would have the advantage that one can also test-drive semi-translated languages. Should I implement this?

[miguelos]

what's the difference between pl and pl_PL
which team should I create ?

I'm in pl_PL team but can't translate above entries (you put them as an example for thai - I just changed last letters of link from th to pl or pl_PL - doesn't work)

I got this message :
The following resources don't have any translations in Polish (Poland) and also don't have a team associated with this language: