I18N - Charset - Revision history

Geep: /* Priority Early-On: the "Top 50" */ Add Show/Hide to table, hide by default

2026-06-28T20:00:06Z

Priority Early-On: the "Top 50": Add Show/Hide to table, hide by default

← Older revision		Revision as of 20:00, 28 June 2026
Line 412:		Line 412:

	== European Character Implementation ==		== European Character Implementation ==
	=== Priority Early-On - the "Top 50" ===		=== Priority Early-On: the "Top 50" ===

	Some of the special characters are used more often than others. Here is a statistic over the entire string set of the TDM core, from TDM v1.08, showing the top 50 most-used characters (~~excluding~~ a-z, 0-9 and ~~russian~~ characters):		Some of the special characters are used more often than others. Here is a statistic over the entire string set of the TDM core, from TDM v1.08, showing the top 50 most-used characters, exclusive of ASCII (e.g., a-z, 0-9) and Russian characters:

	{\|class="wikitable" border=1 style="border-collapse: collapse; font-size: 85%" cellspacing=0 cellpadding=2		{\|class="wikitable mw-collapsible mw-collapsed" border=1 style="border-collapse: collapse; font-size: 85%" cellspacing=0 cellpadding=2 data-expandtext="Show" data-collapsetext="Hide"

	\|-		\|-
Line 426:		Line 426:
	\|Occurances		\|Occurances
	\|Letter		\|Letter
	\|Remarks		\|Remarks .

	\|-		\|-

Geep: /* All other language files */ note variation in depth of support, with link

2026-06-28T19:34:55Z

All other language files: note variation in depth of support, with link

← Older revision		Revision as of 19:34, 28 June 2026
Line 27:		Line 27:

	{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}		{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}
			Be aware that among these languages, there is considerable variation in the actual depth of support. See [[I18N Status]] for more about that.

	== Character Remapping ==		== Character Remapping ==

Geep: /* Future Extensions to Unicode and/or Asian Languages (Korean, Chinese, Japanese)? */ Restructure, list other languages

2026-06-28T19:24:32Z

Future Extensions to Unicode and/or Asian Languages (Korean, Chinese, Japanese)?: Restructure, list other languages

← Older revision		Revision as of 19:24, 28 June 2026
Line 710:		Line 710:
	In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints, and included in TDM 2.14.		In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints, and included in TDM 2.14.

	== Future Extensions ~~to Unicode~~ and~~/or Asian~~ Languages ~~(Korean, Chinese, Japanese)~~? ==		== Possible Future Extensions ==
			=== To Additional Latin and Cyrillic Languages? ===
			TDM's core all.lang file has #str_ for the names of a few additional languages, not yet supported or exposed in the game interface, that could be built out:

	~~It is possible to visualize a transition to a DAT format~~ (~~and engine code~~) ~~that supported UCS (16-bit) Unicode, allowing the existing fonts to then be expanded to cover more characters. Then perhaps all.lang would be the only .lang needed.~~		==== Latin Alphabet ====
			* Croatian / Hrvatski
			* Norse / Norsk
			* Finnish / Suomi
			* Slovenian (differs from Slovak; for southern Slovenia) / Slovenšcina

	As for Asian languages, the original D3 had support for them, so it might be possible to add them to TDM, too, but with a heavy burden on font development and translation. Also, writing from right-to-left (~~Hebrew~~) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.		==== Cyrillic Alphabet ====
			* Bulgarian / Bugarian / Български
			* Serbian / Srpski / Српски
			* Ukrainian (regional) / Українська

			There are no doubt other languages that could be supported by TDM's existing suite of 8-bit encodings.
			=== To Unicode? ===
			It is possible to visualize a transition to a DAT format (and engine code) that supported UCS (16-bit) Unicode, allowing the existing fonts to then be expanded to cover more characters. Then perhaps all.lang would be the only .lang needed.
			=== To Asian Languages (Korean, Chinese, Japanese)? ===
			The original D3 had support for Asian languages, so it might be possible to add them to TDM, too, but with a heavy burden on font development and translation. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

	[[Category:Fonts]]		[[Category:Fonts]]

	{{i18n}}		{{i18n}}

Geep: /* Access to TDM Characters from Various Languages */ add section

2026-06-19T16:07:45Z

Access to TDM Characters from Various Languages: add section

← Older revision		Revision as of 16:07, 19 June 2026
Line 397:		Line 397:
	* ISO 8859-x sourcing of each character.		* ISO 8859-x sourcing of each character.
	* alternative representations of some European and control characters.		* alternative representations of some European and control characters.

			==== Access to TDM Characters from Various Languages ====
			See [[Multilanguage Display]], which includes an itemized table of characters across TDM/ISO codemaps. This discusses which characters (assuming the font implements them) the translator can...
			* enter directly
			* enter by tricks
			* not currently access.

	==== Fonts with Special Characters ====		==== Fonts with Special Characters ====

Geep: /* Font Encoding for European Languages */ Note at top about 0x80-0x9f range

2026-06-11T19:59:49Z

Font Encoding for European Languages: Note at top about 0x80-0x9f range

← Older revision		Revision as of 19:59, 11 June 2026
Line 40:		Line 40:
	Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!		Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!

	In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.		In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character. In addition, printable characters in the range 0x80-0x9f, shown in light or dark green, are also TDM characters, replacing ISO-8859 control characters (not shown).

	'''Color code:'''		'''Color code:'''

Geep: /* Character Remapping */ mention ISO-8859-2

2026-05-31T21:35:16Z

Character Remapping: mention ISO-8859-2

← Older revision		Revision as of 21:35, 31 May 2026
Line 30:		Line 30:
	== Character Remapping ==		== Character Remapping ==

	The characters are remapped upon loading the dictionary/readable, from their native encoding to the two special ones (respectively for Latin and Cyrillic) that TDM uses and that are described next. Responsible for the remapping are [[I18N - Character mapping\|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found in "tdm_base01.pk4/strings" (and there is no "default.map" there, which is the case these days), then no remapping takes place. Generally, ISO-8859-1 languages do not require remapping.		The characters are remapped upon loading the dictionary/readable, from their native encoding to the two special ones (respectively for Latin and Cyrillic) that TDM uses and that are described next. Responsible for the remapping are [[I18N - Character mapping\|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found in "tdm_base01.pk4/strings" (and there is no "default.map" there, which is the case these days), then no remapping takes place. Generally, ISO-8859-1 languages do not require remapping. The map files for the four ISO-8859-2 languages have identical content.

	See '''[[I18N - Character mapping\|Character mapping]]''' for more information.		See '''[[I18N - Character mapping\|Character mapping]]''' for more information.

Geep: Rearrange sections

2026-05-31T20:54:10Z

Rearrange sections

← Older revision		Revision as of 20:54, 31 May 2026
Line 3:		Line 3:
	The D3 code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, special charsets are used. The fonts (Carleton for the menu f.i.) are build/patched so that the right characters appear in the right place.		The D3 code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, special charsets are used. The fonts (Carleton for the menu f.i.) are build/patched so that the right characters appear in the right place.

	== Encodings ==		== LANG File Encodings ==

	=== all.lang ===		=== all.lang ===
Line 34:		Line 34:
	See '''[[I18N - Character mapping\|Character mapping]]''' for more information.		See '''[[I18N - Character mapping\|Character mapping]]''' for more information.

	== Encoding for European Languages ==		== Font Encoding for European Languages ==

	This TDM-specific 8-bit encoding applies to all TDM fonts for English/European languages, that is, everything except the Cyrillic-based Russian.		This TDM-specific 8-bit encoding applies to all TDM fonts for English/European languages, that is, everything except the Cyrillic-based Russian. This encoding controls the ordering within a font's binary DAT file.

	Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!		Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!
Line 403:		Line 403:
	* Carleton font (regular, bold, glow) uses codepoint 0x23, ordinarily "#", to represent "superscript #", which is used in the main menu system to indicate FMs for which a translation pack is available.		* Carleton font (regular, bold, glow) uses codepoint 0x23, ordinarily "#", to represent "superscript #", which is used in the main menu system to indicate FMs for which a translation pack is available.
	* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".		* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".

	~~== Encoding for Russian Cyrillic ==~~

	Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).

	~~Within any given font, character coverage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.~~

	~~== Future Extensions to Unicode and/or Asian Languages (Korean, Chinese, Japanese)? ==~~

	It is possible to visualize a transition to a DAT format (and engine code) that supported UCS (16-bit) Unicode, allowing the existing fonts to then be expanded to cover more characters. Then perhaps all.lang would be the only .lang needed.

	As for Asian languages, the original D3 had support for them, so it might be possible to add them to TDM, too, but with a heavy burden on font development and translation. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

	== European Character Implementation ==		== European Character Implementation ==
Line 697:		Line 685:

	In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14.		In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14.

	~~In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints, and included in TDM 2.14.~~

	In 2026, English/European Carleton 24pt, widely used in the main menu and also some readables, was improved to:		In 2026, English/European Carleton 24pt, widely used in the main menu and also some readables, was improved to:
Line 708:		Line 694:

	Stone 48pt improvements are under consideration for TDM 2.15.		Stone 48pt improvements are under consideration for TDM 2.15.

			== Font Encoding for Russian Cyrillic ==

			Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).

			=== Cyrillic Character Implementation ===
			Within any given font, character coverage may be incomplete.

			In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints, and included in TDM 2.14.

			== Future Extensions to Unicode and/or Asian Languages (Korean, Chinese, Japanese)? ==

			It is possible to visualize a transition to a DAT format (and engine code) that supported UCS (16-bit) Unicode, allowing the existing fonts to then be expanded to cover more characters. Then perhaps all.lang would be the only .lang needed.

			As for Asian languages, the original D3 had support for them, so it might be possible to add them to TDM, too, but with a heavy burden on font development and translation. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.


	[[Category:Fonts]]		[[Category:Fonts]]

	{{i18n}}		{{i18n}}

Geep: /* Character remapping */ Rewrite intro & future extensions, change heading nesting

2026-05-31T20:29:59Z

Character remapping: Rewrite intro & future extensions, change heading nesting

← Older revision		Revision as of 20:29, 31 May 2026
Line 28:		Line 28:
	{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}		{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}

	== Character ~~remapping~~ ==		== Character Remapping ==

	The characters are remapped upon loading the dictionary/readable, from their native encoding to the special ~~one~~ that TDM uses and that is described ~~here~~. Responsible for the remapping are [[I18N - Character mapping\|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found, "strings/default.map" ~~is used instead~~, ~~if this~~ is ~~not found~~, no remapping takes place.		The characters are remapped upon loading the dictionary/readable, from their native encoding to the two special ones (respectively for Latin and Cyrillic) that TDM uses and that are described next. Responsible for the remapping are [[I18N - Character mapping\|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found in "tdm_base01.pk4/strings" (and there is no "default.map" there, which is the case these days), then no remapping takes place. Generally, ISO-8859-1 languages do not require remapping.

	See '''[[I18N - Character mapping\|Character mapping]]''' for more information.		See '''[[I18N - Character mapping\|Character mapping]]''' for more information.

	=== European Languages ===		== Encoding for European Languages ==

	This ~~mapping is used~~ for European languages, ~~f.i. '''Czech''', '''French''', '''German''', '''Spanish''', '''Portuguese'''~~, ~~'''Polish'''~~. Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!		This TDM-specific 8-bit encoding applies to all TDM fonts for English/European languages, that is, everything except the Cyrillic-based Russian.

			Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!

	In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.		In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.
Line 402:		Line 404:
	* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".		* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".

	=== Russian ===		== Encoding for Russian Cyrillic ==

	Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).		Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).

	Within any given font, character ~~overage~~ may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.		Within any given font, character coverage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.

			== Future Extensions to Unicode and/or Asian Languages (Korean, Chinese, Japanese)? ==

	~~=== Asian Languages~~ (~~Korean~~, ~~Chinese, Japanese) ===~~		It is possible to visualize a transition to a DAT format (and engine code) that supported UCS (16-bit) Unicode, allowing the existing fonts to then be expanded to cover more characters. Then perhaps all.lang would be the only .lang needed.

	~~The~~ original D3 had support for ~~these languages~~, so it might be possible to add them to TDM, too~~. At the moment, however~~, ~~we lack the fonts~~ and ~~translators~~. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.		As for Asian languages, the original D3 had support for them, so it might be possible to add them to TDM, too, but with a heavy burden on font development and translation. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

	== European Character Implementation ==		== European Character Implementation ==

Geep: /* All other language files */ Update/correct language list

2026-05-31T19:42:25Z

All other language files: Update/correct language list

← Older revision		Revision as of 19:42, 31 May 2026
Line 17:		Line 17:
	=== All other language files ===		=== All other language files ===

	~~Note that the language~~ files (f.i. '''~~strings/~~german.lang''') ~~as well as~~ the ~~readables and the FM dictionariaries~~ are expected to be in the following encodings:		Language-specific files (f.i. '''german.lang''') provide string dictionaries. A system-wide set is provided in tdm_base01.pk4/strings/. A deployed FM may offer similar files in its /strings/ folder... at the very least '''english.lang'''. All such files are expected to be in the following 8-bit encodings:

	* '''Czech~~'''~~, ~~'''~~Hungarian~~'''~~, ~~'''~~Slovak~~'''~~, ~~'''~~Polish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-2 ISO-8859-2] ('''not WIN-1250!)		* '''Czech, Hungarian, Slovak, Polish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-2 ISO-8859-2] ('''not WIN-1250!)
	* '''Russian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/Win-1251 WIN-1251]		* '''Russian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/Win-1251 WIN-1251]
	* '''French:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-15 ISO-8859-15]		* '''French:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-15 ISO-8859-15]
	* '''Romanian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-16 ISO-8859-16]		* '''Romanian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-16 ISO-8859-16]
	* '''All other languages:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-1 ISO-8859-1] (German, ~~Dutch~~, ~~Danish~~, Swedish, ~~Portuguese~~, ~~etc~~.)		* '''Turkish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-9 ISO-8859-9]
			* '''All other languages:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-1 ISO-8859-1]. This covers TDM-supported languages English, German, Italian, Spanish, Portuguese, Swedish, Danish, Dutch, and Catalan.

	{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}		{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}

Geep: /* Russian */ explain remapping, link to bug report

2026-05-31T16:42:46Z

Russian: explain remapping, link to bug report

← Older revision		Revision as of 16:42, 31 May 2026
Line 404:		Line 404:
	=== Russian ===		=== Russian ===

	Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6.		Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).

	Within any given font, character overage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.		Within any given font, character overage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.