Internationalization: Difference between revisions

Latest revision as of 20:31, 27 July 2021

This article deals with the localisation/translation of both TDM itself, as well as Fan Missions.

The I18N support in TDM will only work with v1.07 or newer.

I18N, L10N?

I18N stands for internationalisation (the 18 means there are 18 letters missing between I and N), and L10N stands for localisation.

In TDM this means that both the menu, the HUD and all the readable content (shown in in-game GUIs) in a mission can be shown in the language the user selects.

Note: There is not yet support for subtitles. This will come with the next version.

For Users

You can switch the language of TDM in the Settings menu under Video, General. The language is stored in the CVAR sys_lang (Note: in v1.07, it was stored in tdm_lang).

Switching the language does not require a restart, except for switching between Russian and Western languages. There the game will inform you that you need to restart it manually, or the display will not look correct.

Changing the language will change the language of the HUD, as well as the inventory immidiately. It can also be done during a mission.

Note: Not all FMs are yet multi-lingual, and for these certain inventory items, the objectives, or readables might appear in English. See the list of already internationalized FMs and their status.

For Mappers

As a mapper, you want to reach an audience as wide as possible with your FM. One way is to provide alternative language versions of your map. There are two ways that can be achieved:

You can ignore the issue, and hope that somebody else modifies your FM and provides a translation. The past shows that the changes that someone does this are slim, tho. Only a handful of FMs were every translated, most of them only into one other language. That is because modifying and translating an FM are quite a lot of work, and translators would rather just translate things than to muck around with your PK4 file.
You can (from v1.07 on) build in support for translation from the beginning, and make the work for translators easier.

The second point can achieved in two ways:

You build the FM with hard-coded strings like 'Silver Key', and then use I18N.pl to transform the FM into a package that can have external dictionaries. That might be easier to work with in DR (as DR itself has no support for dictionaries yet), but the script might not be 100% correct and you may need to manually help it out. However, this is a step that needs to be done only once.
You build your FM with string templates like "#str_23456" (use the range 20000 … 89999) instead of hard-coded names, and supply an external dictionary with these strings.

In both cases, translating your mission becomes super easy, there is only one file with all the strings in place, and this can be easily edited.

FM specific dictionaries

Please see the main article about about translating FMs for details.

Choosing Names

The TDM core dictionary contains a lot of strings, both for inventory items like keys, for locations, monster names, jobs, titles and descriptions. If possible, use one of these names. That has the advantage that the translation is already provided in many languages. For instance, instead of "a silvery key", use if possible "Silver Key" (if you plan on converting your mission with I18N.pl) or "#str_10000" (if you are doing it manually).

Browse through strings/all.lang to see which strings are available.

Inventory Item Names with multiple lines

If you have an inventory item name which is very long, you can insert \n to split it up into two (but not more!) lines:

"#str_20000" "This is a very very\nlong item name."

You should try in the game to see how it looks, so that both lines are roughly the same length.

Support in DR

Darkradiant currently does not support the dictionaries, so strings will look like "#str_07118" instead of the "OK". This is tracked here: https://bugs.thedarkmod.com/view.php?id=2845

For Developers

CVAR tdm_lang

tdm_lang was only used before TDM 2.0 and is obsolete.

CVAR sys_lang

The CVAR sys_lang contains the current language as English lower-case string, e.g. "english", "german", "portuguese" etc. This variable is controlled by the GUI via a call back into the SDK code, which switches the language underneath, and also reloads the GUI.

If you change the CVAR manually (via the console f.i.), you need to restart TDM so it works!

Code

The main translation object is of the class CI18N (which is short for "Class Internationalization") and lives in framework/I18N.cpp and framework/I18N.h (in v1.07, it was in Darkmod/I18N.cpp and Darkmod/I18N.h).

Charset

The TDM (old D3) code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, an special charset (based on ISO 8859-1 with modifications) is used. The details can be found on this page.

New Strings

If you need to add a new string, follow these steps:

search for a fitting, free ID in strings/all.lang.
Add your string there. If possible, add any translation you can, or ask the translators to translate it for you.
re-generate the language files with perl devel/gen_lang.pl

Note that if the translations for the new strings (or any other string) are missing, the script will substitute the English variant - so you should always have at least the English variant.

The other translations are optional, but if they are added later, you need to re-run perl devel/gen_lang.pl! See also the next point:

Always edit all.lang with a UTF-8 editor and keep it in UTF-8.

Never edit the other language files manually, they will be regenerated and your edits will be lost.

Make sure you use the latest versions of all.lang and gen_lang.pl. If you don't have SVN access, work with someone who does and can provide you with the latest versions. When your changes are ready to be checked in, make sure that not only the new *.lang files get checked in, but also the new gen_lang.pl

Adding a new i18n language

The TDM i18n system works by having the translated strings given individual numbers. These numbers are included from separate language files at compile time, and then replaced with the language strings at runtime.

Relevant files

all.lang - all the languages
devel/gen_lang.pl - perl script for breaking up the all.lang master file
(language).lang - the individual language files, eg english.lang or german.lang
(language).map - a character mapping file, eg german.map

Very basic overview of the process

Edit the file "all.lang". Just copy the english language section to the end of the file and rename the new section. Make sure you use a Unicode editor and all.lang stays in UTF-8!
Add the correct entry for the language name in strings/all.lang after the names in the range of 2460 .. 2500 (unless it already exists). The language name is to be expected in the local name, e.g. it is "Deutsch" in the english translation, not "German". If the language name contains characters that cannot be represented in the current charset of the language, please use either the special characters from ISO-8859-1, or an translitertion like "Bulgarian"
if possible, add the same entries in languages that do not use ISO-8859-1
Run the perl script devel/gen_lang.pl To do this, you need to have commit access to the developer SVN. This will "break up" the all.lang file into separate files for each translated language.
If nec, create a new ".map" file by copying one of the existing .map files in "strings/".
Add the new ".lang" file (and possible ".map" file) to SVN.

The new strings should now be accessible at the next start of TDM.

Missing Translations

The following will generate files called "missing_language.txt" in the darkmod directory:

perl devel/gen_lang.pl --missing

Note that the script can no differentiate between a string that was not translated, and a string that is the same even after translation. An example is "Normal" in English, and German. This will be listed as missing string in missing_german.txt.

To avoid that, add the following comment:

"#str_03005"    "Normal"                        // stays the same

to each language variant where the string is unchanged after translation.

Since it can be a bother to ask translators for a new string translation every other day, the current solution is to gather strings for a while, then regenerate the missing_xyz.txt files, and then ask the translators for a translation.

Relocating String IDs

Note: Relocation of string IDs can only be done either before release, or when it is clear that this string is not used by any FM, otherwise you break FMs!

perl devel/rename_string.pl FIRSTOLDID FRISTNEWID COUNT

Example:

perl devel/rename_string.pl 2236 2298 2

GUI

In v1.07 there was a bug in choiceDef inside the GUI, which did not allow other characters than a-z. To work around this choiceDef bug, two new main menu commands were added, called "initChoice" and "stepChoice". However, for v1.08, the bug is fixed in the main code (due to the source code now being available), so the work-arounds were removed.

Known bugs

At the moment only a few things are still not working:

Not all fonts contain accented characters yet.
DarkRadiant does not support the #str_01234 strings yet.
Doom3 does not render/support all characters (f.i. 0xFF in Russian, 0xF7 .. 0x9F in Western). To work around this, the characters are remapped (moved) during load. See I18N - Charset for details on this.

@@ Line 2: / Line 2: @@
 {{infobox|{{red|The I18N support in TDM will only work with v1.07 or newer.}}}}
+== I18N, L10N? ==
+I18N stands for '''internationalisation''' (the 18 means there are 18 letters missing between I and N), and L10N stands for '''localisation'''.
+In TDM this means that both the menu, the HUD and all the readable content (shown in in-game GUIs) in a mission can be shown in the language the user selects.
+{{infobox|Note: There is not yet support for subtitles. This will come with the next version.}}
 == For Users ==
-You can switch the language of TDM in the '''Settings''' menu under '''Video''', '''General'''. The language is stored in the CVAR '''tdm_lang'''. Ignore the CVAR '''sys_lang'''.
+You can switch the language of TDM in the '''Settings''' menu under '''Video''', '''General'''. The language is stored in the CVAR '''sys_lang''' (Note: in v1.07, it was stored in tdm_lang).
-Switching the language does not require a newstart, except for some languages like Russian. There the game will inform you that you need to restart it manually, or the display will not look correct.
+Switching the language does not require a restart, except for switching between Russian and Western languages. There the game will inform you that you need to restart it manually, or the display will not look correct.
 Changing the language will change the language of the HUD, as well as the inventory immidiately. It can also be done during a mission.
-{{infobox|'''Note:''' Not all FMs are yet multi-lingual, and for these certain inventory items, the objectives, or readables might appear in English.}}
+{{infobox|'''Note:''' Not all FMs are yet multi-lingual, and for these certain inventory items, the objectives, or readables might appear in English. See the '''[[I18N Status|list of already internationalized FMs]]''' and their status.}}
 == For Mappers ==
@@ Line 17: / Line 27: @@
 As a mapper, you want to reach an audience as wide as possible with your FM. One way is to provide alternative language versions of your map. There are two ways that can be achieved:
-# You can ignore the issue, and hope that somebody else modifies your FM and provides a translation. The past shows that the changes that someone does this are slim, tho. Only a handful of FMs were every translated, most of them only into one other languages. That is because modifying and translating an FM are quite a lot of work, and translators would rather just translate things than to muck around with your PK4 file.
+# You can ignore the issue, and hope that somebody else modifies your FM and provides a translation. The past shows that the changes that someone does this are slim, tho. Only a handful of FMs were every translated, most of them only into one other language. That is because modifying and translating an FM are quite a lot of work, and translators would rather just translate things than to muck around with your PK4 file.
 # You can (from v1.07 on) build in support for translation from the beginning, and make the work for translators easier.
 The second point can achieved in two ways:
-# You build the FM with hard-coded strings like 'Silver Key', and then use [[I18N.pl]] to transform the FM into a package that can have external dictionaries. That might be easier to work with in DR (as DR itself has no support for dictionaries yet), but the script might not be 100% correct and you need to manually help it out.
+# You build the FM with hard-coded strings like 'Silver Key', and then use [[I18N.pl]] to transform the FM into a package that can have external dictionaries. That might be easier to work with in DR (as DR itself has no support for dictionaries yet), but the script might not be 100% correct and you may need to manually help it out. However, this is a step that needs to be done only once.
-# You build your FM with string templates like '''"#str_12345"''' instead of hard-coded names, and supply an external dictionary with these strings.
+# You build your FM with string templates like '''"#str_23456"''' (use the range 20000 … 89999) instead of hard-coded names, and supply an external dictionary with these strings.
+In both cases, translating your mission becomes super easy, there is only one file with all the strings in place, and this can be easily edited.
 === FM specific dictionaries ===
-These dictionaries live in the directory '''strings/fm/''' and must follow a specific format, explained below. The name for such a dictionary is always '''LANGUAGE.lang''', f.i. "english.lang".
+Please see the main article about [[I18N - Translating Fan Missions|about translating FMs]] for details.
-The string IDs for FMs are 20000 .. 89999, using these ensures that they do not conflict with strings from the TDM core. Note that if you have a string that matches one of the other core strings (like "Silver Key", "#str_10000"), then you should use just the core string ID. This saves the translators to translate the string again.
+=== Choosing Names ===
-You '''MUST''' also separate them from the rest of your FM into their own package called "YOURFM_i18n.pk4" where YOURFM is the name of your original PK4 file. This ensures that TDM can install/update the I18N package properly.
+The TDM core dictionary contains a lot of strings, both for inventory items like keys, for locations, monster names, jobs, titles and descriptions. If possible, use one of these names. That has the advantage that the translation is already provided in many languages. For instance, instead of "a silvery key", use if possible "Silver Key" (if you plan on converting your mission with [[I18N.pl]]) or "#str_10000" (if you are doing it manually).
-Let's see how this looks for the FM named ''Outpost'', which provides two translations (English and French):
+Browse through ''strings/all.lang'' to see which strings are available.
-<pre>
+=== Inventory Item Names with multiple lines ===
- outpost.pk4:
-  xdata/outpost.xd
+If you have an inventory item name which is very long, you can insert '''\n''' to split it up into two (but not more!) lines:
-  maps/outpost.map
-  maps/outpost.cm
-  maps/outpost.proc
-  maps/outpost.script
-  maps/outpost.aas32
-  outpost_i18n.pk4:
+  "#str_20000" "This is a very very\nlong item name."
-  strings/fm/english.lang
+You should try in the game to see how it looks, so that both lines are roughly the same length.
-  strings/fm/french.lang
-</pre>
-Note that all files names should be in lowercase, otherwise there can be problems under Linux.
+=== Support in DR ===
-Example dictionary, note the entire content is wrapped in "{ }"
+Darkradiant currently does not support the dictionaries, so strings will look like "#str_07118" instead of the "OK". This is tracked here: https://bugs.thedarkmod.com/view.php?id=2845
-<pre>
+== For Developers ==
-// String table english (iso-8859-1) for: The Outpost
-// This file was generated automatically on 2011-08-20 13:56 UTC by i18n.pl v0.02
+=== CVAR tdm_lang ===
-{
+'''tdm_lang''' was only used before TDM 2.0 and is obsolete.
-        "#str_20000"    "The Outpost"
-        "#str_20001"    "Break into a builder outpost to retrieve a gold ingot."
-        "#str_20002"    "Gold Ingot"
-        "#str_20003"    "Leave the outpost after your main objectives are done and get back to the front gates."
-        "#str_20004"    "You won't find bags full of money in a builder outpost, but take away any valuables you can get hold of. Find valuables worth at least 600."
-        "#str_20005"    "You won't find bags full of money in a builder outpost, but take away any valuables you can get hold of. Find valuables worth at least 700."
-        "#str_20006"    "No knock outs too."
-        "#str_20007"    "Killing is not your style. No corpses tonight."
-        "#str_20008"    "You won't find bags full of money in a builder outpost, but take away any valuables you can get hold of. Find valuables worth at least 800."
-        "#str_20009"    "The massive gold ingot should be worth a fair amount of money. Find out where the ingot is kept and change its ownership."
-}
-</pre>
-This setup makes it easier to translate the FM, because an translator needs only to copy ''english.lang'' to ''german.lang'', change the strings, repackage the ''outpost_i18n.pk4'' file, and it works!
+=== CVAR sys_lang ===
+The [[CVAR]] '''sys_lang''' contains the current language as English lower-case string, e.g. "english", "german", "portuguese" etc. This variable is controlled by the GUI via a call back into the SDK code, which switches the language underneath, and also reloads the GUI.
-=== Inventory Item Names with multiple lines ===
+{{infobox|If you change the CVAR manually (via the console f.i.), you need to restart TDM so it works!}}
+=== Code ===
+The main translation object is of the class CI18N (which is short for "Class Internationalization") and lives in ''framework/I18N.cpp'' and ''framework/I18N.h'' (in v1.07, it was in ''Darkmod/I18N.cpp'' and ''Darkmod/I18N.h'').
+=== Charset ===
+The TDM (old D3) code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, an special charset (based on ISO 8859-1 with modifications) is used. The details can be found [[I18N - Charset|on this page]].
+=== New Strings ===
+If you need to add a new string, follow these steps:
+* search for a fitting, free ID in '''strings/all.lang'''.
+* Add your string there. If possible, add any translation you can, or ask the translators to translate it for you.
+* re-generate the language files with '''perl devel/gen_lang.pl'''
+Note that if the translations for the new strings (or any other string) are missing, the script will substitute the English variant - so you should always have at least the English variant.
+The other translations are optional, but if they are added later, you need to re-run '''perl devel/gen_lang.pl'''! See also the next point:
+{{infobox|{{red|Always edit all.lang with a UTF-8 editor and keep it in UTF-8.}}}}
+{{infobox|{{red|Never edit the other language files manually, they will be regenerated and your edits will be lost.}}}}
+{{infobox|{{red|Make sure you use the latest versions of all.lang and gen_lang.pl. If you don't have SVN access, work with someone who does and can provide you with the latest versions. When your changes are ready to be checked in, make sure that not only the new *.lang files get checked in, but also the new gen_lang.pl}}}}
+=== Adding a new i18n language ===
+The TDM i18n system works by having the translated strings given individual numbers. These numbers are included from separate language files at compile time, and then replaced with the language strings at runtime.
+==== Relevant files ====
+* all.lang - all the languages
+* devel/gen_lang.pl - perl script for breaking up the all.lang master file
+* (language).lang - the individual language files, eg english.lang or german.lang
+* (language).map - a [[character mapping file]], eg german.map
+==== Very basic overview of the process ====
+* Edit the file "all.lang". Just copy the english language section to the end of the file and rename the new section. '''Make sure you use a Unicode editor and all.lang stays in UTF-8!'''
+* Add the correct entry for the language name in '''strings/all.lang''' after the names in the range of 2460 .. 2500 (unless it already exists). The language name is to be expected in the local name, e.g. it is "Deutsch" in the english translation, not "German". If the language name contains characters that cannot be represented in the current charset of the language, please use either the special characters from ISO-8859-1, or an translitertion like "Bulgarian"
+* if possible, add the same entries in languages that do not use ISO-8859-1
+* Run the perl script '''devel/gen_lang.pl''' To do this, you need to have commit access to the developer SVN. This will "break up" the all.lang file into separate files for each translated language.
+* If nec, create a new ".map" file by copying one of the existing .map files in "strings/".
+* Add the new ".lang" file (and possible ".map" file) to SVN.
+The new strings should now be accessible at the next start of TDM.
+=== Missing Translations ===
+The following will generate files called "missing_language.txt" in the darkmod directory:
+ perl devel/gen_lang.pl --missing
+Note that the script can no differentiate between a string that was not translated, and a string that is the same even after translation. An example is "Normal" in English, and German. This will be listed as missing string in '''missing_german.txt'''.
+To avoid that, add the following comment:
+ "#str_03005"    "Normal"                        // stays the same
-If you have an inventor item name which is very long, you can insert '''\n''' to split it up into two (but not more!) lines:
+to each language variant where the string is unchanged after translation.
- "#str_20000" "This is a very very\nlong item name."
+Since it can be a bother to ask translators for a new string translation every other day, the current solution is to gather strings for a while, then regenerate the missing_xyz.txt files, and then ask the translators for a translation.
-You should try in game how it looks, so that both lines are roughly the same length.
+See also the [[I18N - List of TDM translators|List of TDM translators]].
-== For Developers ==
+=== Relocating String IDs ===
-=== CVAR tdm_lang ===
+{{infobox|Note: Relocation of string IDs can only be done either before release, or when it is clear that this string is not used by any FM, otherwise you break FMs!}}
-The D3 CVAR '''sys_lang''' is restricted to a few hard-coded values, and is thus not usable. We have added a new CVAR '''tdm_lang''', which contains the current language as English lower-case string, e.g. "english", "german", "portuguese" etc. This variable is controlled by the GUI via a call back into the SDK code, which switches the language underneath, and also reloads the GUI.
+ perl devel/rename_string.pl FIRSTOLDID FRISTNEWID COUNT
-{{infobox|If you change the CVAR manually (via the console f.i.), you need to restart TDM so it works!}}
+Example:
-=== Code ===
+ perl devel/rename_string.pl 2236 2298 2
-The main translation object is of the class CI18N (which is short for "Class Internationalization") and lives in ''Darkmod/I18N.cpp'' and ''Darkmod/I18N.h''.
+=== GUI ===
-== Known bugs ==
+In v1.07 there was a bug in '''choiceDef''' inside the GUI, which did not allow other characters than a-z. To work around this '''choiceDef''' bug, two new main menu commands were added, called "initChoice" and "stepChoice". However, for v1.08, the bug is fixed in the main code (due to the source code now being available), so the work-arounds were removed.
-Sadly, there are quite a lot of limitations in the D3 SDK that prevent a few things from fully working. Once D3 goes open source, these things will be fixed, in the meantime we hope that you can live with the quirks.
-{{red|TODO}}
+== Known bugs ==
-== See also ==
+At the moment only a few things are still not working:
-* [http://modetwo.net/darkmod/index.php?/topic/12863-translating-the-tdm-gui/ Forum Thread]
+* Not all fonts contain accented characters yet.
-* [http://bugs.angua.at/view.php?id=2779 Bug Tracker entry #2779]
+* [[DarkRadiant]] does not support the '''#str_01234''' strings yet.
-* [[I18N.pl]] - a script to transform a FM into a mission and I18N data
+* Doom3 does not render/support all characters (f.i. 0xFF in Russian, 0xF7 .. 0x9F in Western). To work around this, the characters are remapped (moved) during load. See [[I18N - Charset]] for details on this.
-* [[I18N Status]] - Which FMs are translated into which language
-{{editing}} {{sdk}}
+{{i18n}} {{editing}} {{sdk}}