I18N - Charset - Revision history

Geep: /* Font Encoding for European Languages */ Note at top about 0x80-0x9f range

2026-06-11T19:59:49Z

Font Encoding for European Languages: Note at top about 0x80-0x9f range

← Older revision		Revision as of 19:59, 11 June 2026
Line 40:		Line 40:
	Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!		Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!

	In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.		In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character. In addition, printable characters in the range 0x80-0x9f, shown in light or dark green, are also TDM characters, replacing ISO-8859 control characters (not shown).

	'''Color code:'''		'''Color code:'''

Geep: /* Character Remapping */ mention ISO-8859-2

2026-05-31T21:35:16Z

Character Remapping: mention ISO-8859-2

← Older revision		Revision as of 21:35, 31 May 2026
Line 30:		Line 30:
	== Character Remapping ==		== Character Remapping ==

	The characters are remapped upon loading the dictionary/readable, from their native encoding to the two special ones (respectively for Latin and Cyrillic) that TDM uses and that are described next. Responsible for the remapping are [[I18N - Character mapping\|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found in "tdm_base01.pk4/strings" (and there is no "default.map" there, which is the case these days), then no remapping takes place. Generally, ISO-8859-1 languages do not require remapping.		The characters are remapped upon loading the dictionary/readable, from their native encoding to the two special ones (respectively for Latin and Cyrillic) that TDM uses and that are described next. Responsible for the remapping are [[I18N - Character mapping\|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found in "tdm_base01.pk4/strings" (and there is no "default.map" there, which is the case these days), then no remapping takes place. Generally, ISO-8859-1 languages do not require remapping. The map files for the four ISO-8859-2 languages have identical content.

	See '''[[I18N - Character mapping\|Character mapping]]''' for more information.		See '''[[I18N - Character mapping\|Character mapping]]''' for more information.

Geep: Rearrange sections

2026-05-31T20:54:10Z

Rearrange sections

← Older revision		Revision as of 20:54, 31 May 2026
Line 3:		Line 3:
	The D3 code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, special charsets are used. The fonts (Carleton for the menu f.i.) are build/patched so that the right characters appear in the right place.		The D3 code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, special charsets are used. The fonts (Carleton for the menu f.i.) are build/patched so that the right characters appear in the right place.

	== Encodings ==		== LANG File Encodings ==

	=== all.lang ===		=== all.lang ===
Line 34:		Line 34:
	See '''[[I18N - Character mapping\|Character mapping]]''' for more information.		See '''[[I18N - Character mapping\|Character mapping]]''' for more information.

	== Encoding for European Languages ==		== Font Encoding for European Languages ==

	This TDM-specific 8-bit encoding applies to all TDM fonts for English/European languages, that is, everything except the Cyrillic-based Russian.		This TDM-specific 8-bit encoding applies to all TDM fonts for English/European languages, that is, everything except the Cyrillic-based Russian. This encoding controls the ordering within a font's binary DAT file.

	Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!		Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!
Line 403:		Line 403:
	* Carleton font (regular, bold, glow) uses codepoint 0x23, ordinarily "#", to represent "superscript #", which is used in the main menu system to indicate FMs for which a translation pack is available.		* Carleton font (regular, bold, glow) uses codepoint 0x23, ordinarily "#", to represent "superscript #", which is used in the main menu system to indicate FMs for which a translation pack is available.
	* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".		* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".

	~~== Encoding for Russian Cyrillic ==~~

	Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).

	~~Within any given font, character coverage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.~~

	~~== Future Extensions to Unicode and/or Asian Languages (Korean, Chinese, Japanese)? ==~~

	It is possible to visualize a transition to a DAT format (and engine code) that supported UCS (16-bit) Unicode, allowing the existing fonts to then be expanded to cover more characters. Then perhaps all.lang would be the only .lang needed.

	As for Asian languages, the original D3 had support for them, so it might be possible to add them to TDM, too, but with a heavy burden on font development and translation. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

	== European Character Implementation ==		== European Character Implementation ==
Line 697:		Line 685:

	In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14.		In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14.

	~~In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints, and included in TDM 2.14.~~

	In 2026, English/European Carleton 24pt, widely used in the main menu and also some readables, was improved to:		In 2026, English/European Carleton 24pt, widely used in the main menu and also some readables, was improved to:
Line 708:		Line 694:

	Stone 48pt improvements are under consideration for TDM 2.15.		Stone 48pt improvements are under consideration for TDM 2.15.

			== Font Encoding for Russian Cyrillic ==

			Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).

			=== Cyrillic Character Implementation ===
			Within any given font, character coverage may be incomplete.

			In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints, and included in TDM 2.14.

			== Future Extensions to Unicode and/or Asian Languages (Korean, Chinese, Japanese)? ==

			It is possible to visualize a transition to a DAT format (and engine code) that supported UCS (16-bit) Unicode, allowing the existing fonts to then be expanded to cover more characters. Then perhaps all.lang would be the only .lang needed.

			As for Asian languages, the original D3 had support for them, so it might be possible to add them to TDM, too, but with a heavy burden on font development and translation. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.


	[[Category:Fonts]]		[[Category:Fonts]]

	{{i18n}}		{{i18n}}

Geep: /* Character remapping */ Rewrite intro & future extensions, change heading nesting

2026-05-31T20:29:59Z

Character remapping: Rewrite intro & future extensions, change heading nesting

← Older revision		Revision as of 20:29, 31 May 2026
Line 28:		Line 28:
	{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}		{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}

	== Character ~~remapping~~ ==		== Character Remapping ==

	The characters are remapped upon loading the dictionary/readable, from their native encoding to the special ~~one~~ that TDM uses and that is described ~~here~~. Responsible for the remapping are [[I18N - Character mapping\|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found, "strings/default.map" ~~is used instead~~, ~~if this~~ is ~~not found~~, no remapping takes place.		The characters are remapped upon loading the dictionary/readable, from their native encoding to the two special ones (respectively for Latin and Cyrillic) that TDM uses and that are described next. Responsible for the remapping are [[I18N - Character mapping\|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found in "tdm_base01.pk4/strings" (and there is no "default.map" there, which is the case these days), then no remapping takes place. Generally, ISO-8859-1 languages do not require remapping.

	See '''[[I18N - Character mapping\|Character mapping]]''' for more information.		See '''[[I18N - Character mapping\|Character mapping]]''' for more information.

	=== European Languages ===		== Encoding for European Languages ==

	This ~~mapping is used~~ for European languages, ~~f.i. '''Czech''', '''French''', '''German''', '''Spanish''', '''Portuguese'''~~, ~~'''Polish'''~~. Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!		This TDM-specific 8-bit encoding applies to all TDM fonts for English/European languages, that is, everything except the Cyrillic-based Russian.

			Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!

	In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.		In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.
Line 402:		Line 404:
	* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".		* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".

	=== Russian ===		== Encoding for Russian Cyrillic ==

	Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).		Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).

	Within any given font, character ~~overage~~ may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.		Within any given font, character coverage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.

			== Future Extensions to Unicode and/or Asian Languages (Korean, Chinese, Japanese)? ==

	~~=== Asian Languages~~ (~~Korean~~, ~~Chinese, Japanese) ===~~		It is possible to visualize a transition to a DAT format (and engine code) that supported UCS (16-bit) Unicode, allowing the existing fonts to then be expanded to cover more characters. Then perhaps all.lang would be the only .lang needed.

	~~The~~ original D3 had support for ~~these languages~~, so it might be possible to add them to TDM, too~~. At the moment, however~~, ~~we lack the fonts~~ and ~~translators~~. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.		As for Asian languages, the original D3 had support for them, so it might be possible to add them to TDM, too, but with a heavy burden on font development and translation. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

	== European Character Implementation ==		== European Character Implementation ==

Geep: /* All other language files */ Update/correct language list

2026-05-31T19:42:25Z

All other language files: Update/correct language list

← Older revision		Revision as of 19:42, 31 May 2026
Line 17:		Line 17:
	=== All other language files ===		=== All other language files ===

	~~Note that the language~~ files (f.i. '''~~strings/~~german.lang''') ~~as well as~~ the ~~readables and the FM dictionariaries~~ are expected to be in the following encodings:		Language-specific files (f.i. '''german.lang''') provide string dictionaries. A system-wide set is provided in tdm_base01.pk4/strings/. A deployed FM may offer similar files in its /strings/ folder... at the very least '''english.lang'''. All such files are expected to be in the following 8-bit encodings:

	* '''Czech~~'''~~, ~~'''~~Hungarian~~'''~~, ~~'''~~Slovak~~'''~~, ~~'''~~Polish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-2 ISO-8859-2] ('''not WIN-1250!)		* '''Czech, Hungarian, Slovak, Polish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-2 ISO-8859-2] ('''not WIN-1250!)
	* '''Russian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/Win-1251 WIN-1251]		* '''Russian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/Win-1251 WIN-1251]
	* '''French:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-15 ISO-8859-15]		* '''French:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-15 ISO-8859-15]
	* '''Romanian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-16 ISO-8859-16]		* '''Romanian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-16 ISO-8859-16]
	* '''All other languages:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-1 ISO-8859-1] (German, ~~Dutch~~, ~~Danish~~, Swedish, ~~Portuguese~~, ~~etc~~.)		* '''Turkish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-9 ISO-8859-9]
			* '''All other languages:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-1 ISO-8859-1]. This covers TDM-supported languages English, German, Italian, Spanish, Portuguese, Swedish, Danish, Dutch, and Catalan.

	{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}		{{infobox\|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}

Geep: /* Russian */ explain remapping, link to bug report

2026-05-31T16:42:46Z

Russian: explain remapping, link to bug report

← Older revision		Revision as of 16:42, 31 May 2026
Line 404:		Line 404:
	=== Russian ===		=== Russian ===

	Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6.		Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6. (This was to overcome a historical Doom3 bug that is now fixed. But to date "the remapping for Russian is still in place, tho, to avoid having to patch all the russian fonts", says bug report [https://bugs.thedarkmod.com/view.php?id=2812 "0002812: Character 0xFF does not work in fonts"]).

	Within any given font, character overage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.		Within any given font, character overage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.

Geep: /* Coverage Expansion & Remaining Limitations */

2026-05-11T19:14:12Z

Coverage Expansion & Remaining Limitations

← Older revision		Revision as of 19:14, 11 May 2026
Line 694:		Line 694:
	In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14.		In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14.

	In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints.		In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints, and included in TDM 2.14.

	In 2026, English/European Carleton 24pt, widely used in the main menu and also some readables, was improved to:		In 2026, English/European Carleton 24pt, widely used in the main menu and also some readables, was improved to:

Geep: /* all.lang */ Include gen_lang_plus

2026-05-11T19:08:28Z

all.lang: Include gen_lang_plus

← Older revision		Revision as of 19:08, 11 May 2026
Line 7:		Line 7:
	=== all.lang ===		=== all.lang ===

	This file is in '''UTF-8''', and converted with the help of the script '''devel/gen_lang.pl''':		This file is in '''UTF-8''', and converted with the help of either the script '''devel/gen_lang.pl''':

	perl devel/gen_lang.pl		perl devel/gen_lang.pl

	This ensures that the generated language files are in their proper encodings (see below).		or, more recently, the Windows C++ utility program '''gen_lang_plus'''. [[Gen_Lang_Programs]] has details of these.

			This process ensures that the generated language files are in their proper encodings (see below).

	=== All other language files ===		=== All other language files ===

Geep: /* Coverage Expansion & Remaining Limitations */ Update Mason 48pt, Carleton 24pt

2026-05-11T17:48:09Z

Coverage Expansion & Remaining Limitations: Update Mason 48pt, Carleton 24pt

← Older revision		Revision as of 17:48, 11 May 2026
Line 690:		Line 690:
	This was confirmed more specifically in 2024, as part of an [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ Analysis of 2.12 Fonts]. This analysis indicated that, unlike the system fonts, the FM fonts generally did not provide specific glyphs beyond ASCII.		This was confirmed more specifically in 2024, as part of an [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ Analysis of 2.12 Fonts]. This analysis indicated that, unlike the system fonts, the FM fonts generally did not provide specific glyphs beyond ASCII.

	In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14. ~~Similar Stone~~ 48pt and Carleton 24pt improvements are ~~planned~~ for TDM 2.15.		In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14.

			In 2025, the Russian Mason 48pt font was improved and extended to cover all TDM Russian codepoints.

			In 2026, English/European Carleton 24pt, widely used in the main menu and also some readables, was improved to:
			* extend coverage to all codepoints
			* replace sloppy hand-drawn Latin-1 glyphs with fresh glyphs, most of them generated from TTF then edge-darkened
			* for simplicity, re-implement the red "glow" effect as if a drop-shadow.

			Testing of this is planned with TDM 2.15 betas.

			Stone 48pt improvements are under consideration for TDM 2.15.

	[[Category:Fonts]]		[[Category:Fonts]]

	{{i18n}}		{{i18n}}

Geep: create "Fonts with Special Characters" section

2026-05-02T16:19:45Z

create "Fonts with Special Characters" section

← Older revision		Revision as of 16:19, 2 May 2026
Line 393:		Line 393:
	* ISO 8859-x sourcing of each character.		* ISO 8859-x sourcing of each character.
	* alternative representations of some European and control characters.		* alternative representations of some European and control characters.

			==== Fonts with Special Characters ====
			Certain fonts have a few codepoints that purposefully diverge from the table entry.

			* Carleton font (regular, bold, glow) uses codepoint 0x23, ordinarily "#", to represent "superscript #", which is used in the main menu system to indicate FMs for which a translation pack is available.
			* Treasure Map font has characters for "pirate skull and crossbones" and "bottle of booze (or poison)".

	=== Russian ===		=== Russian ===