Multilanguage Display - Revision history

Geep: /* TDM & ISO Char Sets - Details, Potential Tricks, Limitations */

2026-06-19T17:44:18Z

TDM & ISO Char Sets - Details, Potential Tricks, Limitations

← Older revision		Revision as of 17:44, 19 June 2026
Line 493:		Line 493:
	\|-		\|-

	\| U+011E \|\| Ğ \|\| 0x88 \|\| \|\| \|\| style="background:#ebbb54;" \| 0xD0 \|\| \|\| \|\| latin capital letter g with breve \|\| As of TDM 2.13		\| U+011E \|\| Ğ \|\| 0x88 \|\| \|\| \|\| style="background:#ebbb54;" \| 0xD0 \|\| \|\| \|\| latin capital letter g with breve \|\| As of TDM 2.13 (TDM codemap), 2.15 (turkish.map)
	\|-		\|-

	\| U+011F \|\| ğ \|\| 0x98 \|\| \|\| \|\| style="background:#ebbb54;" \| 0xF0 \|\| \|\| \|\| latin small letter g with breve \|\| As of TDM 2.13		\| U+011F \|\| ğ \|\| 0x98 \|\| \|\| \|\| style="background:#ebbb54;" \| 0xF0 \|\| \|\| \|\| latin small letter g with breve \|\| As of TDM 2.13 (TDM codemap), 2.15 (turkish.map)
	\|- style="color:#8c8c8c;"		\|- style="color:#8c8c8c;"

Geep: /* Background */

2026-06-19T17:35:13Z

Background

← Older revision		Revision as of 17:35, 19 June 2026
Line 4:		Line 4:
	In TDM's main menu, the Settings/Video/Language page attempts to list all the supported languages in their native form with respect to character set, that is, untranslated. Showing multiple language strings together is difficult, but some tricks are available. The result has been a reasonable near-term workaround for this particular page, applicable to comparable cases. (Comprehensive support for multilanguage display likely involves a major restructuring, for instance, moving to a native Unicode architecture with combined Latin and Cyrillic bitmaps.)		In TDM's main menu, the Settings/Video/Language page attempts to list all the supported languages in their native form with respect to character set, that is, untranslated. Showing multiple language strings together is difficult, but some tricks are available. The result has been a reasonable near-term workaround for this particular page, applicable to comparable cases. (Comprehensive support for multilanguage display likely involves a major restructuring, for instance, moving to a native Unicode architecture with combined Latin and Cyrillic bitmaps.)

	Recall that strings for the main menu are found in the utf8 file tdm_base01.pk4/strings/all.lang, where there is are sections for [English], [German], etc. A particular section is used to generate for distribution the corresponding file, e.g., french.lang, german.lang, etc., in a language-specific 8-bit encoding. When you play TDM, and select a given language, the corresponding .lang file (if ~~needed~~) is read.		Recall that strings for the main menu are found in the utf8 file tdm_base01.pk4/strings/all.lang, where there are sections for [English], [German], etc. A particular section is used to generate for distribution the corresponding file, e.g., french.lang, german.lang, etc., in a language-specific 8-bit encoding. When you play TDM, and select a given language, the corresponding .lang file (if provided) is read.

	Except for Cyrillic Russian, these files use encodings within the ISO-8859 family (as detailed in [[I18N_-_Charset]]). Any specific encoding is a bottleneck... by design, not all characters in any one member of the ISO-8859 family will be accessible by some other members.		Except for Cyrillic Russian, these files use encodings within the ISO-8859 family (as detailed in [[I18N_-_Charset]]). Any specific encoding is a bottleneck... by design, not all characters in any one member of the ISO-8859 family will be accessible by some other members.

Geep: /* For More */ adds section

2026-06-19T17:30:54Z

For More: adds section

← Older revision		Revision as of 17:30, 19 June 2026
Line 675:		Line 675:
	\| U+20AC \|\| € \|\| \|\| \|\| \|\| \|\| 0xA4 \|\| 0xA4 \|\| euro sign \|\|		\| U+20AC \|\| € \|\| \|\| \|\| \|\| \|\| 0xA4 \|\| 0xA4 \|\| euro sign \|\|
	\|}		\|}

			== For More ==
			* [[I18N - Character mapping]] sketches the format and location of <language>.map files.
			* [[I18N - Charset]] is the main article about TDM language use and various encodings.

Geep: finalize for now

2026-06-19T15:50:36Z

finalize for now

← Older revision		Revision as of 15:50, 19 June 2026
Line 1:		Line 1:
	~~[PAGE IN PROGRESS]~~

	== Background ==		== Background ==
	''This describes a particular challenging use case of a TDM methodology that has been around for some time. The motivation to detail this was a 2026 update by Geep to the language map files, following a revision of Carleton 24pt font used throughout the Main Menu.''		''This describes a particular challenging use case of a TDM methodology that has been around for some time. The motivation to detail this was a 2026 update by Geep to the language map files, following a revision of Carleton 24pt font used throughout the Main Menu.''
Line 104:		Line 102:
	*for iso 2: Put ´ (acute accent 0xBD)		*for iso 2: Put ´ (acute accent 0xBD)

	== ~~Details of~~ TDM ~~and~~ ISO Char Sets ~~and~~ Potential Tricks ~~and~~ Limitations ==		== TDM & ISO Char Sets - Details, Potential Tricks, Limitations ==
	~~[WORK IN PROGRESS][COLOR CODING NOT RIGHT YET]~~

	The main table below (after the Key) was drafted by Google AI, from a prompt that defined the header and an example row, and specified:		The main table below (after the Key) was drafted by Google AI, from a prompt that defined the header and an example row, and specified:

	"Rows are a ~~combine~~ of all the printable 8-bit codepoints in the range 0x80-0xff defined in ISO-8859-1, -2, -9, -15, and -16. The rows are ordered by Unicode number. If a particular ISO standard does not include the character, leave that cell blank."		"Rows are a combination of all the printable 8-bit codepoints in the range 0x80-0xff defined in ISO-8859-1, -2, -9, -15, and -16. The rows are ordered by Unicode number. If a particular ISO standard does not include the character, leave that cell blank."

			Subsequently, after Excel import, the TDM and Comments columns were filled in by Geep, and color coding added. Language mappings were independently color coded, then verified against existing language maps. Finally converted to wikitable format.

	~~Subsequently, the TDM and Comments columns were filled in by Geep, and color coding added.~~
	~~Language mappings were independently color coded, then verified against existing language maps.~~
	This table skips codepoints in the range 0x00-0x7f, because they are the same for all ISO-8859 standards. No mapping required.		This table skips codepoints in the range 0x00-0x7f, because they are the same for all ISO-8859 standards. No mapping required.

Geep: /* Details of TDM and ISO Char Sets and Potential Tricks and Limitations */ fixup colors

2026-06-19T15:18:23Z

Details of TDM and ISO Char Sets and Potential Tricks and Limitations: fixup colors

Show changes

Geep: /* Background */ link I18n - Charset page

2026-06-19T13:32:31Z

Background: link I18n - Charset page

← Older revision		Revision as of 13:32, 19 June 2026
Line 2:		Line 2:

	== Background ==		== Background ==
	''This describes a particular challenging use case of a TDM methodology that has been around for some time. The motivation was a 2026 update by Geep to the language map files, following a revision of Carleton 24pt font used throughout the Main Menu.''		''This describes a particular challenging use case of a TDM methodology that has been around for some time. The motivation to detail this was a 2026 update by Geep to the language map files, following a revision of Carleton 24pt font used throughout the Main Menu.''

	In TDM main menu, the Settings/Video/Language page attempts to list all the supported languages in their native form with respect to character set, that is, untranslated. Showing multiple language strings together is difficult, but some tricks are available. The result has been a reasonable near-term workaround for this particular page, applicable to comparable cases. (Comprehensive support for multilanguage display likely involves a major restructuring, for instance, moving to a native Unicode architecture with combined Latin and Cyrillic bitmaps.)		In TDM's main menu, the Settings/Video/Language page attempts to list all the supported languages in their native form with respect to character set, that is, untranslated. Showing multiple language strings together is difficult, but some tricks are available. The result has been a reasonable near-term workaround for this particular page, applicable to comparable cases. (Comprehensive support for multilanguage display likely involves a major restructuring, for instance, moving to a native Unicode architecture with combined Latin and Cyrillic bitmaps.)

	Recall that strings for the main menu are found in the utf8 file tdm_base01.pk4/strings/all.lang, where there is are sections for [English], [German], etc. A particular section is used to generate for distribution the corresponding ~~language-specific~~ file, e.g., french.lang, german.lang, etc. When you play TDM, and select a given language, the corresponding .lang file (if needed) is read.		Recall that strings for the main menu are found in the utf8 file tdm_base01.pk4/strings/all.lang, where there is are sections for [English], [German], etc. A particular section is used to generate for distribution the corresponding file, e.g., french.lang, german.lang, etc., in a language-specific 8-bit encoding. When you play TDM, and select a given language, the corresponding .lang file (if needed) is read.

	Except for Russian, these files use ~~a specific 8-bit encoding~~ within the ISO-8859 family. (~~The i18n – Char page has specifics here~~.~~) This~~ specific encoding is a bottleneck... by design, not all characters in any one member of the ISO-8859 family will be accessible by some other members.		Except for Cyrillic Russian, these files use encodings within the ISO-8859 family (as detailed in [[I18N_-_Charset]]). Any specific encoding is a bottleneck... by design, not all characters in any one member of the ISO-8859 family will be accessible by some other members.

	== Language Names in Idealized UTF8 Form ==		== Language Names in Idealized UTF8 Form ==

Geep: Correct colors in table key

2026-06-19T13:17:56Z

Correct colors in table key

← Older revision		Revision as of 13:17, 19 June 2026
Line 126:		Line 126:
	\|-		\|-

	\| style="background:#~~ffe4b5~~;" \| 0xNN, amber hilite \|\| Type desired utf8 char in all.lang. Generation + mapping (in <language>.map file) goes to TDM codepoint via codepoint shown.		\| style="background:#ebbb54;" \| 0xNN, amber hilite \|\| Type desired utf8 char in all.lang. Generation + mapping (in <language>.map file) goes to TDM codepoint via codepoint shown.
	\|-		\|-

	\| style="background:#~~ffe4b5~~;color:#606060;" \| 0xNN in gray \|\| Type desired utf8 char in all.lang. Generation + mapping (in <language>.map file) goes to TDM codepoint OF SUBSTITUTE CHARACTER via codepoint shown.		\| style="background:#ebbb54;color:#606060;" \| 0xNN in gray \|\| Type desired utf8 char in all.lang. Generation + mapping (in <language>.map file) goes to TDM codepoint OF SUBSTITUTE CHARACTER via codepoint shown.
	\|-		\|-

	\| style="background:#~~ffe4b5~~;color:#ff0000;" \| 0xNN in red \|\| Character not part of this ISO. "0xnn" shown is not ISO, but for trick mapping (in <language.map), to support main menu's multilingual "Languages" page.		\| style="background:#ebbb54;color:#ff0000;" \| 0xNN in red \|\| Character not part of this ISO. "0xnn" shown is not ISO, but for trick mapping (in <language.map), to support main menu's multilingual "Languages" page.
	\|-		\|-

	\| style="background:#~~ffe4b5~~;" \| Substitute Char \|\| Character not part of this ISO. But as trick, you can stuff in this substitute char at the needed TDM codepoint.		\| style="background:#9ab4e3;" \| Substitute Char \|\| Character not part of this ISO. But as trick, you can stuff in this substitute char at the needed TDM codepoint.
	\|-		\|-

	\| style="background:#~~ffe4b5~~;" \| n/a \|\| Not available to use substitute character, due to trick mapping (in <language>.map) elsewhere.		\| style="background:#43a8cc;" \| n/a \|\| Not available to use substitute character, due to trick mapping (in <language>.map) elsewhere.
	\|-		\|-

Line 144:		Line 144:
	\|-		\|-

	\| style="background:#~~ffe4b5~~;" \| Tan Row \|\| Orphaned character in TDM set, not part of 5 ISO-8859 standards for current TDM-supported languages.		\| style="background:#dbab8a;" \| Tan Row \|\| Orphaned character in TDM set, not part of 5 ISO-8859 standards for current TDM-supported languages.
	\|-		\|-
	\|}		\|}

Geep: /* Details of TDM and ISO Char Sets and Potential Tricks and Limitations */ start to fix colors

2026-06-18T20:23:22Z

Details of TDM and ISO Char Sets and Potential Tricks and Limitations: start to fix colors

@@ Line 106: / Line 106: @@
 == Details of TDM and ISO Char Sets and Potential Tricks and Limitations ==
 [WORK IN PROGRESS][COLOR CODING NOT RIGHT YET]
 {| class="wikitable" style="font-size: 95%; line-height: 1.3em;"

Geep: Add spreadsheet, fix columns

2026-06-18T19:38:55Z

Add spreadsheet, fix columns

Show changes

Geep: create article

2026-06-18T19:18:13Z

create article

New page

Title: Multilanguage Display
[PAGE IN PROGRESS]

== Background ==
''This describes a particular challenging use case of a TDM methodology that has been around for some time. The motivation was a 2026 update by Geep to the language map files, following a revision of Carleton 24pt font used throughout the Main Menu.''

In TDM main menu, the Settings/Video/Language page attempts to list all the supported languages in their native form with respect to character set, that is, untranslated. Showing multiple language strings together is difficult, but some tricks are available. The result has been a reasonable near-term workaround for this particular page, applicable to comparable cases. (Comprehensive support for multilanguage display likely involves a major restructuring, for instance, moving to a native Unicode architecture with combined Latin and Cyrillic bitmaps.)

Recall that strings for the main menu are found in the utf8 file tdm_base01.pk4/strings/all.lang, where there is are sections for [English], [German], etc. A particular section is used to generate for distribution the corresponding language-specific file, e.g., french.lang, german.lang, etc. When you play TDM, and select a given language, the corresponding .lang file (if needed) is read.

Except for Russian, these files use a specific 8-bit encoding within the ISO-8859 family. (The i18n – Char page has specifics here.) This specific encoding is a bottleneck... by design, not all characters in any one member of the ISO-8859 family will be accessible by some other members.

== Language Names in Idealized UTF8 Form ==
Only the TDM-exposed languages are shown here. The all.lang file has additional strings for other potential Latin and Cyrillic languages. For reference, the generated encoding for the <language>.lang file is shown in the comment.

"#str_02460" "English" // English [ISO-8859-1]
"#str_02461" "Deutsch" // German [ISO-8859-1]
"#str_02462" "Español" // Spanish [ISO-8859-1]
"#str_02463" "Français" // French [ISO-8859-15]
"#str_02464" "Português" // Portuguese [ISO-8859-1]
"#str_02465" "Polski" // Polish [ISO-8859-2]
"#str_02466" "Italiano" // Italian [ISO-8859-1]
"#str_02467" "Česky" // Czech [ISO-8859-2]
"#str_02468" "Русский" // Russian [WIN-1251]
"#str_02469" "Català" // Catalan [ISO-8859-1]
"#str_02470" "Dansk" // Danish [ISO-8859-1]
"#str_02472" "Nederlands" // Dutch [ISO-8859-1]
"#str_02474" "Magyar" // Hungarian [ISO-8859-2]
"#str_02476" "Svenska" // Swedish [ISO-8859-1]
"#str_02477" "Türkçe" // Turkish [ISO-8859-9]
"#str_02479" "Română" // Romanian [ISO-8859-16]
"#str_02480" "Slovenčina" // Slovak [ISO-8859-2]

== First Compromise – Cyrillic vs. Latin Font ==
While there is some overlap in characters between Cyrillic and Latin encodings, it is insufficient for our purposes. So, for [English] and other European sections, a Latin transliteration from Cyrillic is used:

"#str_02468" "Russkiy" // Russian

Going the other way, for [Russian], the current treatment is not to do a Cyrillic transliteration from Latin, but instead to drop accents, rendering just in ASCII (except of course “Russian”), e.g.:

"#str_02460" "English" // English
"#str_02461" "Deutsch" // German
"#str_02462" "Espanol" // Spanish
"#str_02463" "Francais" // French
"#str_02464" "Portugues" // Portuguese
"#str_02465" "Polski" // Polish
"#str_02466" "Italiano" // Italian
"#str_02467" "Cesky" // Czech
"#str_02468" "Русский" // Russian
"#str_02469" "Catala" // Catalan
"#str_02470" "Dansk" // Danish
"#str_02472" "Nederlands" // Dutch
"#str_02474" "Magyar" // Hungarian
"#str_02476" "Svenska" // Swedish
"#str_02477" "Turkce" // Turkish
"#str_02479" "Romana" // Romanian
"#str_02480" "Slovencina" // Slovak

== Moving between ISO-8859 Members ==
In the discussion here, let “L1” be the current language (from the player’s perspective) or the [language] section of interest in all.lang (from the translator’s perspective). Let “L2” be the language of the foreign word, i.e., L1 and L2 differ.

=== Type and Go ===
In several cases, the translator merely puts the ideal character in the all.lang strings. After <language>.lang generation, everything works. Cases are:

# L1 and L2 are both use the same ISO-8859 encoding. (Specifically, they are either both ISO-8859-1, which doesn’t need <language>.map files, or both ISO-8859-2, and have <language>.map files with, by convention, identical contents.) NOTE: For TDM’s purposes, ISO-8859-15 (French) can be considered part of the ISO-8859-1 family.
# L2 character is ASCII (i.e., in ISO range 0x00-0x7f). This is always the case if L2 = English. Characters in this range have identical codepoints in all five TDM-supported ISO-8859 and in the TDM target encoding.
# L2 character is in the ISO range 0xA0-0xFF, with additional constraints.

For the last case, the L2 character must be either:
* present at the same codepoint in all five TDM-supported ISO-8859 encodings, and in the TDM target encoding. For our Latin language names, these accented characters are like that:
ç in "Français" and "Türkçe" at 0xE7
à in "Català" at 0xEA
â in "Română" at 0xE2
ü in "Türkçe" at 0xFC
* present at the same codepoint in ISO-8859 encodings for L1 and L2, and in the TDM target encoding.
* present in different codepoints in ISO-8859 encodings for L1 and L2, but with each being either the same as the TDM target encoding, or mapped to the TDM target encoding by the <language>.map file.

=== Tricks ===
If the L2 character is not represented in the ISO encoding for L1, then at <L1>.lang generation time, it will be replaced by a “?”. To work around this, the translator has two methods:

====Trick Method 1 – Direct Stuffing====
When L1 encoding is ISO-8859-1 (or -9 or -15, which all have identical encodings for the 0xA0-BF ranges) and L2 character is in TDM’s 0xA0-0xBF range (or D7, F7), then you can lookup the ISO character associated with that codepoint (shown in parentheses in cells of the i18n- charmap table), and type it. Examples:

[English] // and other ISO-8859-1, -9, or -15
"#str_02467" "¬esky" // our default mapping is ISO 8859-1, so ¬ is shown as Č (¬)
"#str_02480" "Sloven®ina" // Slovak (® in ISO-8859-1 is č in our font)

Here’s another example of that last stuffing from a TDM-unexposed language, showing also a diaresis (0xA8):

"#str_02481" "Sloven¨®ina" // Slovenian (southern slovenia) (¨ => š)

In this case, ISO-8859-1 would need to use the diaresis, but ISO-8859-15 (French) would not, i.e., could type and go with š.
The spreadsheet shows all available substitutions in light blue.

====Trick Method 2 – Special Mapping with Repurposed Character====
If all else fails, a little-used character in L1 can be redirected to the L2 codepoint TDM wants, by an extra mapping command in the <L1>.map file. Current such tricks are...

To get ñ for "Español":

* for iso 2, put ¨ (diaeresis 0xA8)
* for iso 16, put ¶ (pilcrow 0xB6)

To get ê for "Português":

*for iso 2: Put ´ (acute accent 0xBD)

== Details of TDM and ISO Char Sets and Potential Tricks and Limitations ==
[TO DO – Insert big table]