The DarkMod Wiki - User contributions [en]

ExportUnicodeFontToDoom3

2026-04-01T00:42:21Z

Geep: /* Relationship to Other Parameters */

''By Geep, 2025''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints]. As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

More broadly, this program supersedes ExportFontToDoom3 and ExportFontToDoom3All256.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

IMPORTANT: If you edit lines, the remaining active glyph lines MUST be in ascending order by the 2-hex-digit codepoint value that starts each line. Otherwise, results will go astray.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

Note: The spacing value is relative to 256x256 bitmaps. If you specify "hirez", then for proportionality, your specified value is doubled internally when laying out the 512x512 bitmaps.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== hirez or hires ===
''Added with v1.1, March 2026''

Instead of 256x256 bitmaps (and associated DAT), this exports double-sized 512x512 bitmaps with larger, thus potentially-crisper glyphs. (TDM has had a few such 512x512 bitmaps for redrawn fonts since 2014.) Specifically, it:
* outputs requested 512x512 TGAs and their DAT(s).
* internally, doubles the pointsize requested of the TTF, resulting in 2x character glyph sizes. Because both the container bitmap and each glyph is (nominally) doubled in size, the [0...1] s, s2, t, t2 values stored in the DAT would be very similar to those for a 256x256 bitmap.
* Other DAT layout metrics remain as if scaled to a 256x256 layout:
** imageHeight
** imageWidth
** skip
** top (with bottom ignored as usual)
** pitch

For example, if you request –hirez with a carleton font in just 24pt type, it will create 512x512 files with usual names like carleton_0_24.tga, etc., and fontimage_24.dat, with the console showing “Exporting carleton 24pt”.
But internally, 48pt characters are drawn on the bitmaps. The imageHeight and imageWidth metrics will be as if the requested 24pt size was generated, so in effect causing the game engine to scale down by a factor of 2 from the 48pt glyph.

''TIPS on Using a HiRez DAT file''

If subsequently using Refont to make a REF file from this DAT, add the “scaling_ok” option, to suppress otherwise voluminous console warnings about per-character scaling.

If you further plan to generate bounds visualizations with datBounds:
* For 512x512 bitmaps, you must generate from the REF file, not directly from DAT.
* Be sure to first edit the REF to add headers of the form:

# bitmap_<n>_size=512

, where <n> is 0 to 9, and refers to shaders present in the REF file with names like, e.g., carleton_3_48.tga; n would be 3 for this.

* datBounds also has a worthwhile “scaling_ok” option.
* Uncommonly, the raw_metrics option may be of interest.

=== bolding <n> ===
''Added with v1.1, March 2026''

Sometimes, a TTF font does not offer a “bold” variant, or, more generally, a font’s overall stroke thickness is not ideal. It is possible to alter character outlines, then render those as bitmap glyphs.
The result is usually better than post-bitmap thickening or thinning. The FreeType library offers an embolding function for outline alteration, used here.

The integer parameter <n> is in units of 1/64th pixel. A positive value adjusts the outline to embolden the character. Negative thins. The change affects both the overall height and width of the outline.
You may think of the left and bottom character borders as unchanged. The allowed range of <n> is from -128 (i.e., -2 pixels) to 256 (4 pixels). The -2 limit is arbitrary, but for sanity. The 4 limit is the maximum FreeType permits.

Caution: The adjustment algorithm moves the outline controls points, but doesn't change the number of such points;
this means that certain situations like acute angles or intersections are sometimes handled incorrectly.

=== inPlace <filename of input layout file> ===
''This experimental feature, new in v 1.2, reads a DAT file for input layout. It may be extended to REF and FTN filetypes in the future. Specifications are subject to change.''

Rather than having an algorithm decide where each glyph should go within a set of bitmaps, this option lets a pre-existing layout file determine it. This is designed to be used with only one point size at a time, so deploy it in conjunction with the “-size” parameter.
This is helpful in building up aligned layers in a GIMP file, particularly as a step in making a “hybrid” or composite TDM font from multiple sources.
This forces a particular layout, great as a “first draft”, but improvable with further bitmap and DAT editing. You can expect some glyph collisions will occur; perhaps incrementally manage this by suppressing some glyphs with a custom unicodeMap?

====File Generation & Naming====
The number of TGA files created, and their names, and their references in the output DAT, will exactly match that specified by the input layout file.
To avoid confusion or name conflict between input and output layout files, either rename the input file, e.g., from “fontimage_24.dat” to “fontimage_24_in.dat”, or place the input layout file in a separate child directory, e.g., “placeAt/fontimage_24.dat”

====Glyph Alignment & Placement====
During placement, for each glyph, only 4 input DAT values are used:
* s
* t
* top
* shaderName

The rest are derived from the TTF file, as influenced by other command-line parameters. Specifically, an output glyph is aligned with the baseline and left-boundary of the corresponding input glyph. That is, the upper-left origin of the output glyph is given by (s, t+top_input-top_TTF), placed into bitmap “shaderName”. (Exception: Negative origin coordinates are replaced by zero.) Aligning vertically by baseline in this way seemed more helpful than aligning by t value alone.

If a codepoint requested by the input DAT has no glyph in the TTF, this codepoint will be skipped during output, i.e., no empty box is generated in its place... because the empty box just gets in the way during subsequent GIMP work. Exception: Codepoint 0x0000 does generate an empty box.

====Relationship to Other Parameters====
These are required:
* The input TTF filename (in addition to inPlace's input layout file; as noted, the contents of the latter file will determine what TGA files are generated.)
* The size <fontsize> option. As usual, this will determine the font size requested from the TTF file, as well as the name of the output fontImage_<filesize>.dat file. However, unlike the usual case, not the naming of the output TGA files.

These options are optional:
* unicodeMap <filename>
* hirez [with DAT file as input, to generate 512x512 output files; situation may change with future REF file.]
* bolding <value>
* name <alternative font name>. As usual, this will name the (generated if necessary) folder into which output files are placed. However, unlike the usual case, the naming of the output TGA files will be as given in the input DAT file.

These options are ignored, by design:
* compact [whether present or absent; no algorithimic layout applied.]
* awayFromEdge <value>

This option is probably problematic and should be avoided for now:
* startFileNumber

== Changed Behavior for Font Title from ExportFontToDoom3All256 ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using or revising the -name argument.

== Downloads ==
Release v1.2 of 2025-03-31 ''(which adds inPlace option; makes clearer a command-line message about xOffset and pitch).''
* [https://drive.google.com/file/d/1CG_BXsTTYi7E3nQPKtBWi-94Cj3R5eOY/view?usp=sharing ExportUnicodeFontToDoom3_release_1_2_exe.zip] . Windows binary and required archived versions of FreeType and DevIL DLLs.
* [https://drive.google.com/file/d/1Nw4RdKKx2YHoPR6D-YQzdnEFx-vyYUhk/view?usp=sharing ExportUnicodeFontToDoom3_release_1_2_source_project.zip] . The C++ project file with source code, including source & build project of archived library.

Release v1.1 of 2025-03-03:
* [https://drive.google.com/file/d/1f3Jj53qtJGIp0_mz7Riing9Fi8ESFDr5/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_exe.zip]
* [https://drive.google.com/file/d/1nutMBfcIWw63wNTUdnlnzP397tGVpnpd/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_source_project.zip] .

Release 1 of 2024-09-10:
* [https://drive.google.com/file/d/1MqoOfhc5-ovoxoHfRquYAcIEdHEOJAQV/view?usp=sharing ExportUnicodeFontToDoom3_release_1_exe.zip]
* [https://drive.google.com/file/d/1Frk_byfdtCcMqdr27ulw1XyuFj8vYp9x/view?usp=sharing ExportUnicodeFontToDoom3_release_1_source_project.zip]

== See Also ==
The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

ExportUnicodeFontToDoom3

2026-04-01T00:41:57Z

Geep: /* Relationship to Other Parameters */

''By Geep, 2025''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints]. As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

More broadly, this program supersedes ExportFontToDoom3 and ExportFontToDoom3All256.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

IMPORTANT: If you edit lines, the remaining active glyph lines MUST be in ascending order by the 2-hex-digit codepoint value that starts each line. Otherwise, results will go astray.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

Note: The spacing value is relative to 256x256 bitmaps. If you specify "hirez", then for proportionality, your specified value is doubled internally when laying out the 512x512 bitmaps.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== hirez or hires ===
''Added with v1.1, March 2026''

Instead of 256x256 bitmaps (and associated DAT), this exports double-sized 512x512 bitmaps with larger, thus potentially-crisper glyphs. (TDM has had a few such 512x512 bitmaps for redrawn fonts since 2014.) Specifically, it:
* outputs requested 512x512 TGAs and their DAT(s).
* internally, doubles the pointsize requested of the TTF, resulting in 2x character glyph sizes. Because both the container bitmap and each glyph is (nominally) doubled in size, the [0...1] s, s2, t, t2 values stored in the DAT would be very similar to those for a 256x256 bitmap.
* Other DAT layout metrics remain as if scaled to a 256x256 layout:
** imageHeight
** imageWidth
** skip
** top (with bottom ignored as usual)
** pitch

For example, if you request –hirez with a carleton font in just 24pt type, it will create 512x512 files with usual names like carleton_0_24.tga, etc., and fontimage_24.dat, with the console showing “Exporting carleton 24pt”.
But internally, 48pt characters are drawn on the bitmaps. The imageHeight and imageWidth metrics will be as if the requested 24pt size was generated, so in effect causing the game engine to scale down by a factor of 2 from the 48pt glyph.

''TIPS on Using a HiRez DAT file''

If subsequently using Refont to make a REF file from this DAT, add the “scaling_ok” option, to suppress otherwise voluminous console warnings about per-character scaling.

If you further plan to generate bounds visualizations with datBounds:
* For 512x512 bitmaps, you must generate from the REF file, not directly from DAT.
* Be sure to first edit the REF to add headers of the form:

# bitmap_<n>_size=512

, where <n> is 0 to 9, and refers to shaders present in the REF file with names like, e.g., carleton_3_48.tga; n would be 3 for this.

* datBounds also has a worthwhile “scaling_ok” option.
* Uncommonly, the raw_metrics option may be of interest.

=== bolding <n> ===
''Added with v1.1, March 2026''

Sometimes, a TTF font does not offer a “bold” variant, or, more generally, a font’s overall stroke thickness is not ideal. It is possible to alter character outlines, then render those as bitmap glyphs.
The result is usually better than post-bitmap thickening or thinning. The FreeType library offers an embolding function for outline alteration, used here.

The integer parameter <n> is in units of 1/64th pixel. A positive value adjusts the outline to embolden the character. Negative thins. The change affects both the overall height and width of the outline.
You may think of the left and bottom character borders as unchanged. The allowed range of <n> is from -128 (i.e., -2 pixels) to 256 (4 pixels). The -2 limit is arbitrary, but for sanity. The 4 limit is the maximum FreeType permits.

Caution: The adjustment algorithm moves the outline controls points, but doesn't change the number of such points;
this means that certain situations like acute angles or intersections are sometimes handled incorrectly.

=== inPlace <filename of input layout file> ===
''This experimental feature, new in v 1.2, reads a DAT file for input layout. It may be extended to REF and FTN filetypes in the future. Specifications are subject to change.''

Rather than having an algorithm decide where each glyph should go within a set of bitmaps, this option lets a pre-existing layout file determine it. This is designed to be used with only one point size at a time, so deploy it in conjunction with the “-size” parameter.
This is helpful in building up aligned layers in a GIMP file, particularly as a step in making a “hybrid” or composite TDM font from multiple sources.
This forces a particular layout, great as a “first draft”, but improvable with further bitmap and DAT editing. You can expect some glyph collisions will occur; perhaps incrementally manage this by suppressing some glyphs with a custom unicodeMap?

====File Generation & Naming====
The number of TGA files created, and their names, and their references in the output DAT, will exactly match that specified by the input layout file.
To avoid confusion or name conflict between input and output layout files, either rename the input file, e.g., from “fontimage_24.dat” to “fontimage_24_in.dat”, or place the input layout file in a separate child directory, e.g., “placeAt/fontimage_24.dat”

====Glyph Alignment & Placement====
During placement, for each glyph, only 4 input DAT values are used:
* s
* t
* top
* shaderName

The rest are derived from the TTF file, as influenced by other command-line parameters. Specifically, an output glyph is aligned with the baseline and left-boundary of the corresponding input glyph. That is, the upper-left origin of the output glyph is given by (s, t+top_input-top_TTF), placed into bitmap “shaderName”. (Exception: Negative origin coordinates are replaced by zero.) Aligning vertically by baseline in this way seemed more helpful than aligning by t value alone.

If a codepoint requested by the input DAT has no glyph in the TTF, this codepoint will be skipped during output, i.e., no empty box is generated in its place... because the empty box just gets in the way during subsequent GIMP work. Exception: Codepoint 0x0000 does generate an empty box.

====Relationship to Other Parameters====
These are required:
* The input TTF filename (in addition to inPlace's input layout file; as noted, the contents of the latter file will determine what TGA files are generated.)
* The size <fontsize> option. As usual, this will determine the font size requested from the TTF file, as well as the name of the output fontImage_<filesize>.dat file. However, unlike the usual case, not the naming of the output TGA files.

These options are optional:
* unicodeMap <filename>
* hirez [with DAT file as input, to generate 512x512 output files; situation may change with future REF file.]
* bolding <value>
* name <alternative font name>. As usual, this will name the (generated if necessary) folder into which output files are placed. However, unlike the usual case, the naming of the output TGA files will be as given in the input DAT file.

These options are ignored, by design:
* compact [whether present or absent; no algorithimic layout applied.]
* awayFromEdge <value>

Thes option is probably problematic and should be avoided for now:
* startFileNumber

== Changed Behavior for Font Title from ExportFontToDoom3All256 ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using or revising the -name argument.

== Downloads ==
Release v1.2 of 2025-03-31 ''(which adds inPlace option; makes clearer a command-line message about xOffset and pitch).''
* [https://drive.google.com/file/d/1CG_BXsTTYi7E3nQPKtBWi-94Cj3R5eOY/view?usp=sharing ExportUnicodeFontToDoom3_release_1_2_exe.zip] . Windows binary and required archived versions of FreeType and DevIL DLLs.
* [https://drive.google.com/file/d/1Nw4RdKKx2YHoPR6D-YQzdnEFx-vyYUhk/view?usp=sharing ExportUnicodeFontToDoom3_release_1_2_source_project.zip] . The C++ project file with source code, including source & build project of archived library.

Release v1.1 of 2025-03-03:
* [https://drive.google.com/file/d/1f3Jj53qtJGIp0_mz7Riing9Fi8ESFDr5/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_exe.zip]
* [https://drive.google.com/file/d/1nutMBfcIWw63wNTUdnlnzP397tGVpnpd/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_source_project.zip] .

Release 1 of 2024-09-10:
* [https://drive.google.com/file/d/1MqoOfhc5-ovoxoHfRquYAcIEdHEOJAQV/view?usp=sharing ExportUnicodeFontToDoom3_release_1_exe.zip]
* [https://drive.google.com/file/d/1Frk_byfdtCcMqdr27ulw1XyuFj8vYp9x/view?usp=sharing ExportUnicodeFontToDoom3_release_1_source_project.zip]

== See Also ==
The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

ExportUnicodeFontToDoom3

2026-04-01T00:40:14Z

Geep: /* Downloads */ updates to release 1.2

''By Geep, 2025''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints]. As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

More broadly, this program supersedes ExportFontToDoom3 and ExportFontToDoom3All256.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

IMPORTANT: If you edit lines, the remaining active glyph lines MUST be in ascending order by the 2-hex-digit codepoint value that starts each line. Otherwise, results will go astray.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

Note: The spacing value is relative to 256x256 bitmaps. If you specify "hirez", then for proportionality, your specified value is doubled internally when laying out the 512x512 bitmaps.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== hirez or hires ===
''Added with v1.1, March 2026''

Instead of 256x256 bitmaps (and associated DAT), this exports double-sized 512x512 bitmaps with larger, thus potentially-crisper glyphs. (TDM has had a few such 512x512 bitmaps for redrawn fonts since 2014.) Specifically, it:
* outputs requested 512x512 TGAs and their DAT(s).
* internally, doubles the pointsize requested of the TTF, resulting in 2x character glyph sizes. Because both the container bitmap and each glyph is (nominally) doubled in size, the [0...1] s, s2, t, t2 values stored in the DAT would be very similar to those for a 256x256 bitmap.
* Other DAT layout metrics remain as if scaled to a 256x256 layout:
** imageHeight
** imageWidth
** skip
** top (with bottom ignored as usual)
** pitch

For example, if you request –hirez with a carleton font in just 24pt type, it will create 512x512 files with usual names like carleton_0_24.tga, etc., and fontimage_24.dat, with the console showing “Exporting carleton 24pt”.
But internally, 48pt characters are drawn on the bitmaps. The imageHeight and imageWidth metrics will be as if the requested 24pt size was generated, so in effect causing the game engine to scale down by a factor of 2 from the 48pt glyph.

''TIPS on Using a HiRez DAT file''

If subsequently using Refont to make a REF file from this DAT, add the “scaling_ok” option, to suppress otherwise voluminous console warnings about per-character scaling.

If you further plan to generate bounds visualizations with datBounds:
* For 512x512 bitmaps, you must generate from the REF file, not directly from DAT.
* Be sure to first edit the REF to add headers of the form:

# bitmap_<n>_size=512

, where <n> is 0 to 9, and refers to shaders present in the REF file with names like, e.g., carleton_3_48.tga; n would be 3 for this.

* datBounds also has a worthwhile “scaling_ok” option.
* Uncommonly, the raw_metrics option may be of interest.

=== bolding <n> ===
''Added with v1.1, March 2026''

Sometimes, a TTF font does not offer a “bold” variant, or, more generally, a font’s overall stroke thickness is not ideal. It is possible to alter character outlines, then render those as bitmap glyphs.
The result is usually better than post-bitmap thickening or thinning. The FreeType library offers an embolding function for outline alteration, used here.

The integer parameter <n> is in units of 1/64th pixel. A positive value adjusts the outline to embolden the character. Negative thins. The change affects both the overall height and width of the outline.
You may think of the left and bottom character borders as unchanged. The allowed range of <n> is from -128 (i.e., -2 pixels) to 256 (4 pixels). The -2 limit is arbitrary, but for sanity. The 4 limit is the maximum FreeType permits.

Caution: The adjustment algorithm moves the outline controls points, but doesn't change the number of such points;
this means that certain situations like acute angles or intersections are sometimes handled incorrectly.

=== inPlace <filename of input layout file> ===
''This experimental feature, new in v 1.2, reads a DAT file for input layout. It may be extended to REF and FTN filetypes in the future. Specifications are subject to change.''

Rather than having an algorithm decide where each glyph should go within a set of bitmaps, this option lets a pre-existing layout file determine it. This is designed to be used with only one point size at a time, so deploy it in conjunction with the “-size” parameter.
This is helpful in building up aligned layers in a GIMP file, particularly as a step in making a “hybrid” or composite TDM font from multiple sources.
This forces a particular layout, great as a “first draft”, but improvable with further bitmap and DAT editing. You can expect some glyph collisions will occur; perhaps incrementally manage this by suppressing some glyphs with a custom unicodeMap?

====File Generation & Naming====
The number of TGA files created, and their names, and their references in the output DAT, will exactly match that specified by the input layout file.
To avoid confusion or name conflict between input and output layout files, either rename the input file, e.g., from “fontimage_24.dat” to “fontimage_24_in.dat”, or place the input layout file in a separate child directory, e.g., “placeAt/fontimage_24.dat”

====Glyph Alignment & Placement====
During placement, for each glyph, only 4 input DAT values are used:
* s
* t
* top
* shaderName

The rest are derived from the TTF file, as influenced by other command-line parameters. Specifically, an output glyph is aligned with the baseline and left-boundary of the corresponding input glyph. That is, the upper-left origin of the output glyph is given by (s, t+top_input-top_TTF), placed into bitmap “shaderName”. (Exception: Negative origin coordinates are replaced by zero.) Aligning vertically by baseline in this way seemed more helpful than aligning by t value alone.

If a codepoint requested by the input DAT has no glyph in the TTF, this codepoint will be skipped during output, i.e., no empty box is generated in its place... because the empty box just gets in the way during subsequent GIMP work. Exception: Codepoint 0x0000 does generate an empty box.

====Relationship to Other Parameters====
These are required:
* The input TTF filename (in addition to inPlace's input layout file; as noted, the contents of the latter file will determine what TGA files are generated.)
* The size <fontsize> option. As usual, this will determine the font size requested from the TTF file, as well as the name of the output fontImage_<filesize>.dat file. However, unlike the usual case, not the naming of the output TGA files.

These options are optional:
* unicodeMap <filename>
* hirez [with DAT file as input, to generate 512x512 output files; situation may change with future REF file.]
* bolding <value>
* name <alternative font name>. As usual, this will name the (generated if necessary) folder into which output files are placed. However, unlike the usual case, the naming of the output TGA files will be as given in the input DAT file.

These options are ignored, by design:
* compact [whether present or absent; no algorithimic layout applied.]
* awayFromEdge <value>

Theis option is probably problematic and should be avoided for now:
* startFileNumber

== Changed Behavior for Font Title from ExportFontToDoom3All256 ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using or revising the -name argument.

== Downloads ==
Release v1.2 of 2025-03-31 ''(which adds inPlace option; makes clearer a command-line message about xOffset and pitch).''
* [https://drive.google.com/file/d/1CG_BXsTTYi7E3nQPKtBWi-94Cj3R5eOY/view?usp=sharing ExportUnicodeFontToDoom3_release_1_2_exe.zip] . Windows binary and required archived versions of FreeType and DevIL DLLs.
* [https://drive.google.com/file/d/1Nw4RdKKx2YHoPR6D-YQzdnEFx-vyYUhk/view?usp=sharing ExportUnicodeFontToDoom3_release_1_2_source_project.zip] . The C++ project file with source code, including source & build project of archived library.

Release v1.1 of 2025-03-03:
* [https://drive.google.com/file/d/1f3Jj53qtJGIp0_mz7Riing9Fi8ESFDr5/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_exe.zip]
* [https://drive.google.com/file/d/1nutMBfcIWw63wNTUdnlnzP397tGVpnpd/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_source_project.zip] .

Release 1 of 2024-09-10:
* [https://drive.google.com/file/d/1MqoOfhc5-ovoxoHfRquYAcIEdHEOJAQV/view?usp=sharing ExportUnicodeFontToDoom3_release_1_exe.zip]
* [https://drive.google.com/file/d/1Frk_byfdtCcMqdr27ulw1XyuFj8vYp9x/view?usp=sharing ExportUnicodeFontToDoom3_release_1_source_project.zip]

== See Also ==
The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

ExportUnicodeFontToDoom3

2026-03-31T21:24:13Z

Geep: Add inPlace option

''By Geep, 2025''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints]. As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

More broadly, this program supersedes ExportFontToDoom3 and ExportFontToDoom3All256.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

IMPORTANT: If you edit lines, the remaining active glyph lines MUST be in ascending order by the 2-hex-digit codepoint value that starts each line. Otherwise, results will go astray.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

Note: The spacing value is relative to 256x256 bitmaps. If you specify "hirez", then for proportionality, your specified value is doubled internally when laying out the 512x512 bitmaps.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== hirez or hires ===
''Added with v1.1, March 2026''

Instead of 256x256 bitmaps (and associated DAT), this exports double-sized 512x512 bitmaps with larger, thus potentially-crisper glyphs. (TDM has had a few such 512x512 bitmaps for redrawn fonts since 2014.) Specifically, it:
* outputs requested 512x512 TGAs and their DAT(s).
* internally, doubles the pointsize requested of the TTF, resulting in 2x character glyph sizes. Because both the container bitmap and each glyph is (nominally) doubled in size, the [0...1] s, s2, t, t2 values stored in the DAT would be very similar to those for a 256x256 bitmap.
* Other DAT layout metrics remain as if scaled to a 256x256 layout:
** imageHeight
** imageWidth
** skip
** top (with bottom ignored as usual)
** pitch

For example, if you request –hirez with a carleton font in just 24pt type, it will create 512x512 files with usual names like carleton_0_24.tga, etc., and fontimage_24.dat, with the console showing “Exporting carleton 24pt”.
But internally, 48pt characters are drawn on the bitmaps. The imageHeight and imageWidth metrics will be as if the requested 24pt size was generated, so in effect causing the game engine to scale down by a factor of 2 from the 48pt glyph.

''TIPS on Using a HiRez DAT file''

If subsequently using Refont to make a REF file from this DAT, add the “scaling_ok” option, to suppress otherwise voluminous console warnings about per-character scaling.

If you further plan to generate bounds visualizations with datBounds:
* For 512x512 bitmaps, you must generate from the REF file, not directly from DAT.
* Be sure to first edit the REF to add headers of the form:

# bitmap_<n>_size=512

, where <n> is 0 to 9, and refers to shaders present in the REF file with names like, e.g., carleton_3_48.tga; n would be 3 for this.

* datBounds also has a worthwhile “scaling_ok” option.
* Uncommonly, the raw_metrics option may be of interest.

=== bolding <n> ===
''Added with v1.1, March 2026''

Sometimes, a TTF font does not offer a “bold” variant, or, more generally, a font’s overall stroke thickness is not ideal. It is possible to alter character outlines, then render those as bitmap glyphs.
The result is usually better than post-bitmap thickening or thinning. The FreeType library offers an embolding function for outline alteration, used here.

The integer parameter <n> is in units of 1/64th pixel. A positive value adjusts the outline to embolden the character. Negative thins. The change affects both the overall height and width of the outline.
You may think of the left and bottom character borders as unchanged. The allowed range of <n> is from -128 (i.e., -2 pixels) to 256 (4 pixels). The -2 limit is arbitrary, but for sanity. The 4 limit is the maximum FreeType permits.

Caution: The adjustment algorithm moves the outline controls points, but doesn't change the number of such points;
this means that certain situations like acute angles or intersections are sometimes handled incorrectly.

=== inPlace <filename of input layout file> ===
''This experimental feature, new in v 1.2, reads a DAT file for input layout. It may be extended to REF and FTN filetypes in the future. Specifications are subject to change.''

Rather than having an algorithm decide where each glyph should go within a set of bitmaps, this option lets a pre-existing layout file determine it. This is designed to be used with only one point size at a time, so deploy it in conjunction with the “-size” parameter.
This is helpful in building up aligned layers in a GIMP file, particularly as a step in making a “hybrid” or composite TDM font from multiple sources.
This forces a particular layout, great as a “first draft”, but improvable with further bitmap and DAT editing. You can expect some glyph collisions will occur; perhaps incrementally manage this by suppressing some glyphs with a custom unicodeMap?

====File Generation & Naming====
The number of TGA files created, and their names, and their references in the output DAT, will exactly match that specified by the input layout file.
To avoid confusion or name conflict between input and output layout files, either rename the input file, e.g., from “fontimage_24.dat” to “fontimage_24_in.dat”, or place the input layout file in a separate child directory, e.g., “placeAt/fontimage_24.dat”

====Glyph Alignment & Placement====
During placement, for each glyph, only 4 input DAT values are used:
* s
* t
* top
* shaderName

The rest are derived from the TTF file, as influenced by other command-line parameters. Specifically, an output glyph is aligned with the baseline and left-boundary of the corresponding input glyph. That is, the upper-left origin of the output glyph is given by (s, t+top_input-top_TTF), placed into bitmap “shaderName”. (Exception: Negative origin coordinates are replaced by zero.) Aligning vertically by baseline in this way seemed more helpful than aligning by t value alone.

If a codepoint requested by the input DAT has no glyph in the TTF, this codepoint will be skipped during output, i.e., no empty box is generated in its place... because the empty box just gets in the way during subsequent GIMP work. Exception: Codepoint 0x0000 does generate an empty box.

====Relationship to Other Parameters====
These are required:
* The input TTF filename (in addition to inPlace's input layout file; as noted, the contents of the latter file will determine what TGA files are generated.)
* The size <fontsize> option. As usual, this will determine the font size requested from the TTF file, as well as the name of the output fontImage_<filesize>.dat file. However, unlike the usual case, not the naming of the output TGA files.

These options are optional:
* unicodeMap <filename>
* hirez [with DAT file as input, to generate 512x512 output files; situation may change with future REF file.]
* bolding <value>
* name <alternative font name>. As usual, this will name the (generated if necessary) folder into which output files are placed. However, unlike the usual case, the naming of the output TGA files will be as given in the input DAT file.

These options are ignored, by design:
* compact [whether present or absent; no algorithimic layout applied.]
* awayFromEdge <value>

Theis option is probably problematic and should be avoided for now:
* startFileNumber

== Changed Behavior for Font Title from ExportFontToDoom3All256 ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using or revising the -name argument.

== Downloads ==
Release v1.2 of 2025-03-31
* [REAL SOON]

Release v1.1 of 2025-03-03:
* [https://drive.google.com/file/d/1f3Jj53qtJGIp0_mz7Riing9Fi8ESFDr5/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_exe.zip] . Windows binary and required archived versions of FreeType and DevIL DLLs.
* [https://drive.google.com/file/d/1nutMBfcIWw63wNTUdnlnzP397tGVpnpd/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_source_project.zip] . The C++ project file with source code, including source & build project of archived library.

Release 1 of 2024-09-10:
* [https://drive.google.com/file/d/1MqoOfhc5-ovoxoHfRquYAcIEdHEOJAQV/view?usp=sharing ExportUnicodeFontToDoom3_release_1_exe.zip]
* [https://drive.google.com/file/d/1Frk_byfdtCcMqdr27ulw1XyuFj8vYp9x/view?usp=sharing ExportUnicodeFontToDoom3_release_1_source_project.zip]

== See Also ==
The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

Refont

2026-03-21T22:16:13Z

Geep: /* Downloads */ Add comments about 2.7, 2.8 improvements

''By Geep, 2024, and similar to the article about Q3Font.''

Introduced in 2024, the "refont" command-line utility allows inspection and alteration of [[Font Metrics & DAT File Format | font metrics]].
It is designed to be an easier-to-use version of the inspection/alteration functionalities of [[Q3Font]].
It also provides a font analysis feature.

== Comparison with Font Patcher ==

Refont can be seen as complementary to [[Font Patcher]]. The latter has additional features that may be better if planning a wholesale rearrangement or expansion of characters within a bitmap, particularly for a new font immediately after conversion from a TTF font. But refont is easier to use in later years, and Font Patcher requires installation of Perl, while refont.exe is more drop and go.

== Comparison with Q3Font ==

For metric alteration, q3font has a native "*.fnt" human-readable format, and supports DAT <==> FNT conversions. For compatibility, so does refont.

=== Advantages of Q3Font ===

* This has been an important traditional tool for Doom3/TDM font manipulation.
* In addition to production/consumption of FNT, it -
** Allows creation of DAT & TGA files from a TrueType font (although for this purpose, [[ExportFontToDoom3]] has been preferred in practice).
** Can write (but not read) an alternative "compact report".

=== Advantages of Refont ===

* It's open-source, so further C++ development is possible.
* It has its own native human-readable format, "*.ref", similar to .fnt but better.
* The REF format can be read by [[datBounds]] to generate bitmap boundary & metrics visualizations.

The better features of REF are:

* Codepoints are enumerated in not just decimal, but also hex.
* Coordinates of the overall image box within its bitmap are expressed in pixel units, rather than [0...1] fractional floating point. So the user is spared having to manually do this tedious and error-prone calculation.
* This overcomes a drawback of FNT, that expresses the coordinates with only 6 digits after the decimal point, not quite enough to avoid minor mis-precisions.
* if optionally provided, a separate, read-only annotation file will automatically decorate the .ref file with helpful information as the file is generated. This is explained below.

Because the code is open-source, there's better clarity about -
* how parsing and calculations are done.
* what warnings are emitted.

Also, using "-stats", refont can write (but not read) an analysis of a DAT file, particularly looking for problematic/unimplemented Western-language characters.

== For Inspection & Correction of DAT files ==

''To Get the Data in Readable, Editable Form''

For viewing and altering in a text editor, two formats are provided. Both list every codepoint, in order from 0 to 255, as a block of data. Within each block, every item gets its own line.

=== As a FNT File ===

To create a q3font-compatable "<dat-name>.fnt" file:

refont -decompile <path to given .dat file> -fnt

where the path can be a full path, a relative path (e.g., starting with "./" or "../"), or just the file name in the current working directory. The resulting .fnt file will appear in the same directory. It have the same name as the .dat file, except for the extension. As a convenience to keep track of versions, it is not required that the file name be in the standard fontImage_nn.dat form.

Example code block for the letter A:
...
char 65
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
s 0.769531
t 0.250000
s2 0.847656
t2 0.320313
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Note that Q3Font FNT expresses the s, t, s2, and t2 coordinates with only 6 digits after the decimal point, causing minor low-precision rounding during DAT --> FNT --> DAT roundtrips. For compatibility, Refont FNT does likewise.

=== As a REF File ===
''Starting with Refont v2.5, a mixture of 256x256 and 512x512 bitmap shaders, encountered with Carleton and Mason family fonts, can be handled. See further below.''

To create a refont-specific "<dat-name>.ref" file:

refont -decompile <path to given .dat file>

The same path and naming conventions as for .fnt generation discussed above apply here.

Example code block for the letter A when no supplemental annotation file is present (nor any per-line comments, e.g., with warnings):

...
char 65 (0x41)
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Items bolded differ from FNT format. The starting line expresses the value in hex. And rather than s, t, s2, t2 in fractional [0...1] floating point, the "coord_..." values are expressed (and editable) as integer coordinates of the bitmap, with a range reflecting bitmap size. The default TDM bitmap size is 256x256, but a few fonts use 512x512 and require special treatment discussed further below.

== When Editing ==
Refont has a minimal parser. To keep it happy, preserve the file's line structure. In particular:
* every REF and FNT file has the same number of lines. ''(Exception: REF may have additional starting lines to flag 512x512 bitmaps, discussed below. Generally, you won't edit these.)''
* all keywords within a block must be present, and in a particular order.
* each block must have exactly the same number of lines (except for the special one at the end).

The meaning of the font metrics are explained in [[Font_Metrics_%26_DAT_File_Format]]. The [[Q3Font]] article has an example of simple metric editing that is also pertinent to refont.

== To Move Changed FNT or REF Data Back to a DAT ==

refont -compile <path to given .fnt or .ref file>

This reverses the -decompile process, to create a .dat file whose name is derived from the .fnt or .ref file. Importantly, this decompile/compile cycle leaves .tga/.dds files untouched.

== The Annotation File ==
When told to generate a REF file (but not FNT), refont.exe also looks for a file called "refont_char_annotations.txt", in the same folder as refont.exe. That file should have exactly 256 data lines in it, one for each of the codepoints, in order. As the REF is generated, the line that starts a codepoint's data block gets the corresponding annotation line appended to it. Having the annotations come from an external file, instead of hard-coded into refont, allows required flexibility as to what information is presented. It even potentially allows refont to be used with other idTech3/4 games that don't necessarily used the TDM-specific codepoints. Also, the annotation files contain helpful specific comments (as lines that start with ";").

Each annotation:
* indicates what character symbol is ''expected''; it is oblivious as to what character if any is actually within the bounding box described by the DAT data.
* lives only in a REF file, and is not retained when converted to a DAT file. This also true for any manual supplementary annotations you might make.

This is clearer with examples. In 'Downloads' below are 4 different versions of "refont_char_annotations[...].txt". Three of these, specific to TDM's custom 'english' (i.e., composite European) codepage, are discussed here. To deploy, edit the filename of one of them to remove the square bracket text.

=== English with Symbol Only ===

This simple format is best for most purposes. The file entry for letter 'A' is just:

A

Then the block in a generated REF file would have that string appended, but with " // " inserted.
...
char 65 (0x41) // A
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

=== English with Unicode-16 Codes and Names ===

This supplies the Unicode Consortium's official 16-bit codepoints (where U+NNNN = 0xNNNN) and their names, in the formal all-caps convention.

The file entry for letter 'A' is:

U+0041 LATIN CAPITAL LETTER A

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // U+0041 LATIN CAPITAL LETTER A
...
Here's another example from that REF:
...
char 129 (0x81) // U+015A LATIN CAPITAL LETTER S WITH ACUTE
...
For control or TDM undefined characters, this annotation file assumes a hollow box glyph is the appropriate mapping:
...
char 0 (0x00) // U+24A1 WHITE SQUARE
...
But there are other choices, so edit the annotation file as you see fit.
=== English with 8859-x Source ===

This verbose format may be useful during font analysis. For letters that are in their expected locations as defined by ASCII or ISO 8859-1 (Latin-1) or Windows-1252 ("Western Europe"), the annotation is fairly minimal. Thus, letter 'A' has this:

0x41 is A

(TIP: If authoring your own annotation file, consider including a codepoint number on each line, like "0x41" here, to make the process easier.)

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // 0x41 is A
...
Here are a few more interesting examples from that REF, with more expansive annotations:
...
char 5 (0x05) // 0x05 is a control character, unusable in TDM
...
char 166 (0xa6) // 0xA6 is Š (not ISO 8859-1 ¦) (from A9 in ISO 8859-2) (TDM only)
...

== Revising or Making Your Own Annotations File ==
If creating your own refont_char_annotations.txt file, note that encoding upper-range characters in UTF-8 will be more universal than using Windows region-specific codepages. To enter a UTF-8 character using its four digit hex Unicode (say, in a range of interest 0x0080-0x00FF):
* Under Windows, type the four hex digits followed by Alt-X.
* Under Linux, hold down Ctrl+Shift, type U followed by the 4 digits; release Ctrl+Shift.

When refont reads refont_char_annotations.txt, any line that starts with ";" will be ignored. The latter is so that you can put comments into the file, that will not propagate to the generated REF file. TIP: If such commenting is unwanted, start the line with <space> before ";".

Other than that, be sure to have exactly 256 lines of annotations (though refont will check and issue a warning if not). A line can be empty, i.e., have only the linebreak.

When generating a REF file, refont will automatically add " // " before each non-empty annotation, so don't include that.

== Adding Your Own Annotations to a REF File ==
Beyond the refont_char_annotation.txt content, you can add certain commentary to the REF file without it affecting its ability to convert to a DAT file. Here are the rules:
* Don't add or subtract lines, or comment out non-blank lines
* Feel free to append a comment to the end of a line, by beginning the comment with " //".
* Likewise, on a line starting with "char" that had an annotation automatically added, you can revise the annotation as you see fit.

Example with revision in bold:
...
char 65 (0x41) // '''Capital A was slightly clipped. Decremented s, s2 by 1. Looks OK now.'''
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 196 '''// WAS: 197'''
coord_t 64
coord_s2 216 '''// WAS: 217'''
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

In this way, you can markup a REF file as a record of work to do, work in progress, or work completed.

==Errors, Warnings, and Auto-Corrections==
''This covers the major overhaul of Refont v 2.1, and its added options -no_warn_comments and -scaling_ok.''

During the compile or decompile, errors and warnings may be generated in the console. Fatal errors are largely due to:
* file I/O problems;
* deviations from the strict DAT, REF, and FNT formats.

Warnings, which are not fatal, are largely due to anomolous numeric values. The TDM engine can tolerant some of these, so they don't necessarily need fixing.

===Fatal Errors when Parsing REF or FNT as Input===
A fatal error, due to divergence from the expected format while parsing, stops the program. While potentially due to many causes (e.g., adding or deleting lines), errors will be reported - with line number - as failures of:

''Variable Name Checks.'' The parser expects every variable name in a particular order. A mismatch happened.

''Integer Checks.'' Integer value was unparsable, or out of range for integer type. Checked for per-character variables height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and glyph. Plus REF’s coord_s, coord_t, coord_s2, and coord_ t2.

''Float Checks.'' Floating point value was unparsable, or out of range for float type. Checked for glyphScale, and FNT’s per-character variables s, t, s2, t2.

===Post-Read MAJOR Warnings about REF, FNT, and DAT Input Values===
After parsing, the input process concludes with a separate loop of non-fatal warning analysis. Warnings during input, broadly categorized as major or minor, report to the console, but don’t do any value corrections (which is instead left to the subsequent file-write stage).

''For FNT’s and DAT’s s, t, s2, t2''
* Inexact Integer. The given bounding box value, expected in the range 0-1, when multiplied by 256.0 should be exactly an integer. If not, the divergence is likely due to miscalculation. (Actually, a divergence from an integer of under +/- 0.001 is tolerated by Refont and considered a minor warning, discussed below.)
* Float Out of Range 0.0 - 1.0. Or equivalently, post-multiplication, 0.0 - 256.0. (See the discussion of the edge cases under minor warnings below.)

''For Other Metrics of DAT, FNT, and REF''
* Integer Out of Range 0-256. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and REF’s coord_s, coord_t, coord_s2, and coord_t2.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).
* Inconsistent Metrics. For a given character, this compares two metrics and looks for inconsistencies. We’ll state this here in REF terms, e.g., using “coord_s” rather than “ROUND_TO_INT(s * 256.0f)”:
** imageHeight == height?
** imageWidth == coord_s2 - coord_s?
** imageHeight == coord_t2 - coord_t?

However, warnings about those last two "Inconsistent Metrics" criteria are suppressed if:
* you use refont's "-scaling_ok" option. This is helpful when there is extensive per-character scaling. (Only TDM’s mason and mason-glow use a lot of per-character scaling, specifically for all upper-case characters.)
* you have a 512x512 bitmap, where throughout, the imageWidth and imageHeight (and some other metrics) are expected to be scaled to "half" of the corresponding coord difference. As of refont 2.8, 1 pixel of wiggle-room is allowed for integer rounding. Only warnings will be issued that fail these checks:
** abs(imageWidth * 2 - (coord_s2 - coord_s)) > 1
** abs(imageHeight * 2 - (coord_t2 - coord_t)) > 1

===Post-Read MINOR Warnings for FNT’s and DAT’s s, t, s2, t2===
''Low Precision.'' This value, when multiplied by 256.0, should be exactly an integer. However, some generation or revision procedure (e.g., Q3Font’s DAT  FNT  DAT) may have not used sufficient precision to ensure this. An offset from an integer value of less than +/- 0.001 is considered a minor problem; otherwise, major as above.

This problem, when it occurs with a file, typically happens a lot. Consequently, it is not reported in an itemized way, but instead, the console will show total counts of affected characters and values.

Edge cases: a value of {s, s2, t, or t2} * 256.0 that is negative but closer to zero than -0.001000 is considered this type of minor problem (and is not considered out of range, which would be major). Likewise, if that value is greater than 256.000000, but less than 256.001000
===Automatic Corrections during Output to a REF File===
Certain problems, first detected during DAT input, are further addressed at write time. Because these problems were already reported to the console during the DAT read, nothing additional is written to console. Instead, there may be reporting within the REF file itself.

''Automatic Corrections.'' When writing a REF file, the integer values of coord_s, coord_t, coord_s2, and coord_t2 are calculated from DAT floats s, t, s2, and t2 respectively. A calculated coord_... value, if not already resulting in an integer, is automatically forced to the nearest integer. This is so whether it was earlier reported to the console as a Major or Minor Warning. (Before refont v 2.1, the REF might assign an unrounded decimal value; no longer true.)

''Warnings Not Automatically Corrected.'' Automatic fixing of other warnings is unwise and not offered. Why? Because the fix is not obvious or complex, and sometimes, the corresponding character glyph in the bitmap needs adjustment. This pertains to Major Warnings about:
* Integer values outside their expected range (e.g., 0-256), including post-rounding coord_... values.
* Inconsistent Metrics
===Generated Within-REF Comments during Output===
For most causes of a Major Warning during DAT read, during subsequent writing of the REF file, a similar comment is also generated within the REF, on the same line as the parameter assignment. For example:
char 95 (0x5f) // _
{
...
top -6 // WARNING: Not in range 0-256. Does bitmap or other metrics need adjusting?
...
}
...
char 140 (0x8c) // Ń
{
...
coord_t -10 // WARNING: Underlying DAT value -0.03906250, * 256.0 gave -10.000000000, not in range 0-256 once rounded. Does bitmap or other metrics need adjusting?
...
}
''Suppressing Comment Generation.'' Use the refont option “-no_warn_comments”. ''New to v 2.1.''

TIP: When editing a REF file with comments, you can keep them, change them, or delete them as you prefer. They have no impact on subsequent compiles to DAT.

''Coverage of Generated Within-REF Comments.'' A calculated value of coord_s, coord_t, coord_s2, or coord_t2 would have been flagged as a “Major Warning” during read if:
* the discrepancy from being an integer was non-trivial (+/- 0.001 or more), or
* the calculated and rounded integer is outside the range given by bitmap size, e.g., 0-256.

For these, a comment is generated. Other causes are:
* Integer Out of Range 0-256 ''(or 0-512 as applicable)''. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).

''Major Warnings that Don’t Cause Within-REF Comments.'' Currently, this is limited to “Inconsistent Metrics”, which would probably require tagging on multiple lines.

== Special Handling of Fonts with 512x512 Bitmaps ==
Some DAT files reference bitmap shaders where some or all are of size 512x512, instead of the default 256x256. Refont v2.5+ can support this (replacing an insufficient first attempt in v2.4).

Font developers went to double-sized bitmaps to:
* in the case of Carleton, allow freehand drawing of missing European characters, which could be drawn more easily if twice standard size.
* in the case of Mason, have larger ASCII alphanumeric glyphs that would render with crisper edges.

In either case, the font characters when rendered in, say, the game main menu will be the same size as if from 256x256. This is because the imageWidth, imageHeight, height, and top metrics are not doubled.
In effect, a scaling by half (approximately, as constrained by integer values) is requested for all characters in the DAT file and applied by the engine.

=== When generating a REF file from a DAT file ===

refont -decompile <path to given .dat file> -512_<n> ''where <n> is a single digit, e.g. '1' in referenced shader name "font/Carleton_1_24.tga" ''

You should use multiple instances of the -512_<n> option, to cover all the 512x512 bitmaps found in the DAT file.

If ''all'' of a given DATs shaders are 512x512, you can use this shorter form:

refont -decompile <path to given .dat file> -512

Exercising -512... options will mean that in the generated REF file:

* There will be a one or more starting lines like:
# bitmap_<n>_size=512 ''(Refont will look for this if doing a subsequent compile to DAT)''
* If you specified the short form, it will expand to just the right number of starting lines.
* Values for coord_s, coord_s2, coord_t, and coord_t2 will have a normal range of 0-512 when appropriate. [COMING SOON: Those values may then be read properly by datBounds.].
* Any warnings expressed (within the file or within the command-line window) will be in terms of the correct range of values.
* Warning checks will expect a scaling by half for all characters. That is, while 512... requires scaling down by half, that scaling will not generate a warning (unless it's not EXACTLY by half, or within 1 pixel of that), so the -scaling_ok option is not needed just for the exactly-by-half scaling.

=== When generating a DAT file from a REF file ===

During refont -compile, no special command line options are needed. Instead, REF's starting lines of form "# bitmap_<n>_size=512" are inspected and coord ranges will be interpreted as 0-512 whenever appropriate.

=== Special Handling Recommendations for Carleton and Mason ===
''For fonts shipped with TDM 2.12 in tdm_fonts01.pk4. Illustrative command paths are shown; adjust to your working copy.''
==== Mason Family ====
The [[Mason Font]] uses within-DAT per-character scaling, as well as some but not all 512x512 bitmaps. Only 48pt Mason is shipped.

Because, under /dds/fonts/english/mason/, bitmaps masonalternate_0_48.dds to _2_ are 512x512, but _3_ to _5_ are not, use:

-decompile /fonts/english/mason/fontimage_48.dat -512_0 -512_1 -512_2 -scaling_ok

Because, under /dds/fonts/english/mason_glow/, bitmaps masonalternate_1_48.dds and _2_ are 512x512, but _0_ and _3_ to _5_ are not, use:

-decompile /fonts/english/mason_glow/fontimage_48.dat -512_1 -512_2 -scaling_ok

Russian versions use the default 256x256. [These possibly benefit from -scaling_ok; didn't investigate.]
==== Carleton Family ====
The 24pt size of carleton has two bitmaps under /dds/fonts/english/carleton/, namely carleton_0_24.dds and _1_, both of size 512x512. So use:

-decompile /fonts/english/carleton/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton/fontimage_24.dat -512

The 24pt size of carleton_bold has bitmaps with identical content to carleton's, so use the same options:

-decompile /fonts/english/carleton_bold/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton_bold/fontimage_24.dat -512

Other members of this family use the default 256x256 bitmaps:
* Sizes 12pt & 48pt
* carleton_condensed and carleton_glow. (Note: while helpful, it is not required that the base & glow have identical bitmap sizes.)
* Russian versions

==== These Fonts with Stats ====
With -stats discussed next, if reading a DAT file, you may use the -512... and scaling_ok options to be most correct, although neglecting to specify them will likely have no significant impact on the analytical results. If reading a REF file, the included "# bitmap_..." lines will take care of that for you.

== For Statistics on Unimplemented or Problematic Font Characters ==
DAT or (as of v2.3) REF file analysis here is primarily designed to benefit english/european fonts. It will use the optional annotation file if provided. Syntax:

refont -stats <path to given .dat or .ref file>

Use "-stats" by itself, not combined with "-compile" or "-decompile".

The resulting analysis report will have the same name and location as the input file, but with a ".txt" suffix.
The report provides categorized totals and (for problematic characters) itemizations across all 256 character codepoints.
The counts of problems should be considered minimums, since no inspection of bitmap files (TGA or DDS) is done.
But a perhaps surprising amount of insight can be gleaned just from DAT/REF file metrics.
Itemizations include annotations from the 'ref_char_annotation' file, if provided.

In the hunt for missing characters, 3 signatures are important:

* Presumed 'Hollow Box' = glyph's 'shadername' is <fontname>_0_<size>.dds only, and s & t are zero, but not s2 & t2
* '<Space>' = same test as 'Hollow Box', but pointed to by char 32 (0x20)
* 'Zero Box' = No glyph box (s, t, s2, t2 all zero)

A given analysis will assume either 'Hollow Box' or '<Space>' but not both.

The analysis provides totals and itemizations - grouped into lower (0-127) and upper ranges (128-255) - in several passes:

* Pass 1 - Handling of unprintable/unsupported/missing codepoints, indicated by Hollow Box, Zero Box, or <Space>. Some of these are not a problem, but some are tagged as "Undesirable".
* Pass 2 - Bad glyph box (negative s, t, s2, or t2; or s2 <= s, t2 <= t) or good glyph box with dubious metrics (imageHeight <= 0, imageWidth <=0, imageHeight != height). Excludes those already counted as "Undesirable" in Pass 1.
* Pass 3 - Detection of duplicate glyph boxes (other than Hollow Box, Zero Box, or <Space>). Detected by: glyph's values for shadername, s, t, s2, & t2 exactly match those of another codepoint. These are grouped into "Dup Sets".

Then, across passes 1-3, a count if given of the minimum "Total Glyphs Needing Work".

Refont version 2.3 adds additional passes:

* Pass 4 - Reports "Overlap Sets" rather than just "Dup Sets". A set here may include an unaccented "base character" that has been serving as a substitute work-around target for one or more missing accented characters. (What constitutes an "overlap" is idiosyncratic; see the source code for details.)
* Pass 5 - Same information as Pass 4, but sorted first by shader name (i.e., bitmap). Helpful for organizing bitmap-editing work.
* Pass 6 - An analysis focusing just on the planned migration of duplicate O/o-circumflex codepoints to G/g-cup glyphs.

== Downloads ==
Release 2.8 of March 19, 2026:
* Supports partial coverage regeneration (i.e., unicodeMap with not all 256 represented) by ExportUnicodeFontToDoom3.
* With 512x512 bitmaps, suppresses scaling warnings if scaled by "half", now +/- 1 pixel.
* [https://drive.google.com/file/d/1BHGkCOhgXP7bFp1mXyfgoRBUtqkbCVRb/view?usp=sharing refont.exe]
* [https://drive.google.com/file/d/1Cca_Xo3W5jYB_40RVo4aIPHAGIufA9V0/view?usp=sharing refont_source_v2.8.zip, with refont.cpp, refont.sln, open-source license]

Version 2.7: Fixes poor detection of _<n>_ pattern in shadername, and allow <n> to be 1 or 2 digits, not just 1.

Release 2.6 of July 28, 2025:
* [https://drive.google.com/file/d/1ABf4771Jr1C1hKbLx4xWq8lx8Kn7HdKP/view?usp=sharing refont.exe]
* [https://drive.google.com/file/d/1-k68eo0CHKZ-sM7JGP05axm49cc5De0T/view?usp=sharing source code: refont.cpp]

===Annotations===
''After download, to use any one of these, edit filename to remove "[...]" substring.''

Update of June 20, 2024. This reflects a change to the TDM charmap for 2.13, to replace duplicate O/o circumflex characters with G/g breve:

* refont_char_annotations[english with symbol only. With G-breve update].txt [https://drive.google.com/file/d/1rrt-O9DgD7Zn2Hxt7nBtViIlO4kv_gQB/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes & names. With G-breve update].txt [https://drive.google.com/file/d/1AMxDKQc_DE33gsExiEkGp3x67eb4aGFh/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source. With G-breve update].txt [https://drive.google.com/file/d/1xoeZ1poTbmkls-gE5PnbbMxdoz5KnMD7/view?usp=sharing].
* For Russian, see the previous update next.

Update of April 13, 2024:

* refont_char_annotations[english with symbol only].txt [https://drive.google.com/file/d/1ItM-aXps0xQE655HdaJM7Aje5lHewUfC/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes and names].txt [https://drive.google.com/file/d/1gW7oO_wLqCEPSdm6b05OaTNYHnyuUSA-/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source].txt [https://drive.google.com/file/d/1ZTcgA4QJYwXzNdqbolYkn0CCEDILD1bX/view?usp=sharing here]. Corrects and supersedes original english file.
* refont_char_annotations[russian].txt [https://drive.google.com/file/d/10txNXsSCtaTiT2iGT9o3iDxBU7HVYNxi/view?usp=sharing here]. Recommended for 'russian' fonts.

== For More ==
* See the [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ summary analysis of 2.12 TDM fonts], based on applying 'refont -stats ...' to all 'english' DAT files.

{{tutorial}} [[Category:Fonts]]

Refont

2026-03-19T17:46:28Z

Geep: /* Downloads */ v 2.8 released

''By Geep, 2024, and similar to the article about Q3Font.''

Introduced in 2024, the "refont" command-line utility allows inspection and alteration of [[Font Metrics & DAT File Format | font metrics]].
It is designed to be an easier-to-use version of the inspection/alteration functionalities of [[Q3Font]].
It also provides a font analysis feature.

== Comparison with Font Patcher ==

Refont can be seen as complementary to [[Font Patcher]]. The latter has additional features that may be better if planning a wholesale rearrangement or expansion of characters within a bitmap, particularly for a new font immediately after conversion from a TTF font. But refont is easier to use in later years, and Font Patcher requires installation of Perl, while refont.exe is more drop and go.

== Comparison with Q3Font ==

For metric alteration, q3font has a native "*.fnt" human-readable format, and supports DAT <==> FNT conversions. For compatibility, so does refont.

=== Advantages of Q3Font ===

* This has been an important traditional tool for Doom3/TDM font manipulation.
* In addition to production/consumption of FNT, it -
** Allows creation of DAT & TGA files from a TrueType font (although for this purpose, [[ExportFontToDoom3]] has been preferred in practice).
** Can write (but not read) an alternative "compact report".

=== Advantages of Refont ===

* It's open-source, so further C++ development is possible.
* It has its own native human-readable format, "*.ref", similar to .fnt but better.
* The REF format can be read by [[datBounds]] to generate bitmap boundary & metrics visualizations.

The better features of REF are:

* Codepoints are enumerated in not just decimal, but also hex.
* Coordinates of the overall image box within its bitmap are expressed in pixel units, rather than [0...1] fractional floating point. So the user is spared having to manually do this tedious and error-prone calculation.
* This overcomes a drawback of FNT, that expresses the coordinates with only 6 digits after the decimal point, not quite enough to avoid minor mis-precisions.
* if optionally provided, a separate, read-only annotation file will automatically decorate the .ref file with helpful information as the file is generated. This is explained below.

Because the code is open-source, there's better clarity about -
* how parsing and calculations are done.
* what warnings are emitted.

Also, using "-stats", refont can write (but not read) an analysis of a DAT file, particularly looking for problematic/unimplemented Western-language characters.

== For Inspection & Correction of DAT files ==

''To Get the Data in Readable, Editable Form''

For viewing and altering in a text editor, two formats are provided. Both list every codepoint, in order from 0 to 255, as a block of data. Within each block, every item gets its own line.

=== As a FNT File ===

To create a q3font-compatable "<dat-name>.fnt" file:

refont -decompile <path to given .dat file> -fnt

where the path can be a full path, a relative path (e.g., starting with "./" or "../"), or just the file name in the current working directory. The resulting .fnt file will appear in the same directory. It have the same name as the .dat file, except for the extension. As a convenience to keep track of versions, it is not required that the file name be in the standard fontImage_nn.dat form.

Example code block for the letter A:
...
char 65
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
s 0.769531
t 0.250000
s2 0.847656
t2 0.320313
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Note that Q3Font FNT expresses the s, t, s2, and t2 coordinates with only 6 digits after the decimal point, causing minor low-precision rounding during DAT --> FNT --> DAT roundtrips. For compatibility, Refont FNT does likewise.

=== As a REF File ===
''Starting with Refont v2.5, a mixture of 256x256 and 512x512 bitmap shaders, encountered with Carleton and Mason family fonts, can be handled. See further below.''

To create a refont-specific "<dat-name>.ref" file:

refont -decompile <path to given .dat file>

The same path and naming conventions as for .fnt generation discussed above apply here.

Example code block for the letter A when no supplemental annotation file is present (nor any per-line comments, e.g., with warnings):

...
char 65 (0x41)
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Items bolded differ from FNT format. The starting line expresses the value in hex. And rather than s, t, s2, t2 in fractional [0...1] floating point, the "coord_..." values are expressed (and editable) as integer coordinates of the bitmap, with a range reflecting bitmap size. The default TDM bitmap size is 256x256, but a few fonts use 512x512 and require special treatment discussed further below.

== When Editing ==
Refont has a minimal parser. To keep it happy, preserve the file's line structure. In particular:
* every REF and FNT file has the same number of lines. ''(Exception: REF may have additional starting lines to flag 512x512 bitmaps, discussed below. Generally, you won't edit these.)''
* all keywords within a block must be present, and in a particular order.
* each block must have exactly the same number of lines (except for the special one at the end).

The meaning of the font metrics are explained in [[Font_Metrics_%26_DAT_File_Format]]. The [[Q3Font]] article has an example of simple metric editing that is also pertinent to refont.

== To Move Changed FNT or REF Data Back to a DAT ==

refont -compile <path to given .fnt or .ref file>

This reverses the -decompile process, to create a .dat file whose name is derived from the .fnt or .ref file. Importantly, this decompile/compile cycle leaves .tga/.dds files untouched.

== The Annotation File ==
When told to generate a REF file (but not FNT), refont.exe also looks for a file called "refont_char_annotations.txt", in the same folder as refont.exe. That file should have exactly 256 data lines in it, one for each of the codepoints, in order. As the REF is generated, the line that starts a codepoint's data block gets the corresponding annotation line appended to it. Having the annotations come from an external file, instead of hard-coded into refont, allows required flexibility as to what information is presented. It even potentially allows refont to be used with other idTech3/4 games that don't necessarily used the TDM-specific codepoints. Also, the annotation files contain helpful specific comments (as lines that start with ";").

Each annotation:
* indicates what character symbol is ''expected''; it is oblivious as to what character if any is actually within the bounding box described by the DAT data.
* lives only in a REF file, and is not retained when converted to a DAT file. This also true for any manual supplementary annotations you might make.

This is clearer with examples. In 'Downloads' below are 4 different versions of "refont_char_annotations[...].txt". Three of these, specific to TDM's custom 'english' (i.e., composite European) codepage, are discussed here. To deploy, edit the filename of one of them to remove the square bracket text.

=== English with Symbol Only ===

This simple format is best for most purposes. The file entry for letter 'A' is just:

A

Then the block in a generated REF file would have that string appended, but with " // " inserted.
...
char 65 (0x41) // A
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

=== English with Unicode-16 Codes and Names ===

This supplies the Unicode Consortium's official 16-bit codepoints (where U+NNNN = 0xNNNN) and their names, in the formal all-caps convention.

The file entry for letter 'A' is:

U+0041 LATIN CAPITAL LETTER A

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // U+0041 LATIN CAPITAL LETTER A
...
Here's another example from that REF:
...
char 129 (0x81) // U+015A LATIN CAPITAL LETTER S WITH ACUTE
...
For control or TDM undefined characters, this annotation file assumes a hollow box glyph is the appropriate mapping:
...
char 0 (0x00) // U+24A1 WHITE SQUARE
...
But there are other choices, so edit the annotation file as you see fit.
=== English with 8859-x Source ===

This verbose format may be useful during font analysis. For letters that are in their expected locations as defined by ASCII or ISO 8859-1 (Latin-1) or Windows-1252 ("Western Europe"), the annotation is fairly minimal. Thus, letter 'A' has this:

0x41 is A

(TIP: If authoring your own annotation file, consider including a codepoint number on each line, like "0x41" here, to make the process easier.)

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // 0x41 is A
...
Here are a few more interesting examples from that REF, with more expansive annotations:
...
char 5 (0x05) // 0x05 is a control character, unusable in TDM
...
char 166 (0xa6) // 0xA6 is Š (not ISO 8859-1 ¦) (from A9 in ISO 8859-2) (TDM only)
...

== Revising or Making Your Own Annotations File ==
If creating your own refont_char_annotations.txt file, note that encoding upper-range characters in UTF-8 will be more universal than using Windows region-specific codepages. To enter a UTF-8 character using its four digit hex Unicode (say, in a range of interest 0x0080-0x00FF):
* Under Windows, type the four hex digits followed by Alt-X.
* Under Linux, hold down Ctrl+Shift, type U followed by the 4 digits; release Ctrl+Shift.

When refont reads refont_char_annotations.txt, any line that starts with ";" will be ignored. The latter is so that you can put comments into the file, that will not propagate to the generated REF file. TIP: If such commenting is unwanted, start the line with <space> before ";".

Other than that, be sure to have exactly 256 lines of annotations (though refont will check and issue a warning if not). A line can be empty, i.e., have only the linebreak.

When generating a REF file, refont will automatically add " // " before each non-empty annotation, so don't include that.

== Adding Your Own Annotations to a REF File ==
Beyond the refont_char_annotation.txt content, you can add certain commentary to the REF file without it affecting its ability to convert to a DAT file. Here are the rules:
* Don't add or subtract lines, or comment out non-blank lines
* Feel free to append a comment to the end of a line, by beginning the comment with " //".
* Likewise, on a line starting with "char" that had an annotation automatically added, you can revise the annotation as you see fit.

Example with revision in bold:
...
char 65 (0x41) // '''Capital A was slightly clipped. Decremented s, s2 by 1. Looks OK now.'''
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 196 '''// WAS: 197'''
coord_t 64
coord_s2 216 '''// WAS: 217'''
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

In this way, you can markup a REF file as a record of work to do, work in progress, or work completed.

==Errors, Warnings, and Auto-Corrections==
''This covers the major overhaul of Refont v 2.1, and its added options -no_warn_comments and -scaling_ok.''

During the compile or decompile, errors and warnings may be generated in the console. Fatal errors are largely due to:
* file I/O problems;
* deviations from the strict DAT, REF, and FNT formats.

Warnings, which are not fatal, are largely due to anomolous numeric values. The TDM engine can tolerant some of these, so they don't necessarily need fixing.

===Fatal Errors when Parsing REF or FNT as Input===
A fatal error, due to divergence from the expected format while parsing, stops the program. While potentially due to many causes (e.g., adding or deleting lines), errors will be reported - with line number - as failures of:

''Variable Name Checks.'' The parser expects every variable name in a particular order. A mismatch happened.

''Integer Checks.'' Integer value was unparsable, or out of range for integer type. Checked for per-character variables height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and glyph. Plus REF’s coord_s, coord_t, coord_s2, and coord_ t2.

''Float Checks.'' Floating point value was unparsable, or out of range for float type. Checked for glyphScale, and FNT’s per-character variables s, t, s2, t2.

===Post-Read MAJOR Warnings about REF, FNT, and DAT Input Values===
After parsing, the input process concludes with a separate loop of non-fatal warning analysis. Warnings during input, broadly categorized as major or minor, report to the console, but don’t do any value corrections (which is instead left to the subsequent file-write stage).

''For FNT’s and DAT’s s, t, s2, t2''
* Inexact Integer. The given bounding box value, expected in the range 0-1, when multiplied by 256.0 should be exactly an integer. If not, the divergence is likely due to miscalculation. (Actually, a divergence from an integer of under +/- 0.001 is tolerated by Refont and considered a minor warning, discussed below.)
* Float Out of Range 0.0 - 1.0. Or equivalently, post-multiplication, 0.0 - 256.0. (See the discussion of the edge cases under minor warnings below.)

''For Other Metrics of DAT, FNT, and REF''
* Integer Out of Range 0-256. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and REF’s coord_s, coord_t, coord_s2, and coord_t2.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).
* Inconsistent Metrics. For a given character, this compares two metrics and looks for inconsistencies. We’ll state this here in REF terms, e.g., using “coord_s” rather than “ROUND_TO_INT(s * 256.0f)”:
** imageHeight == height?
** imageWidth == coord_s2 - coord_s?
** imageHeight == coord_t2 - coord_t?

However, warnings about those last two "Inconsistent Metrics" criteria are suppressed if:
* you use refont's "-scaling_ok" option. This is helpful when there is extensive per-character scaling. (Only TDM’s mason and mason-glow use a lot of per-character scaling, specifically for all upper-case characters.)
* you have a 512x512 bitmap, where throughout, the imageWidth and imageHeight (and some other metrics) are expected to be scaled to "half" of the corresponding coord difference. As of refont 2.8, 1 pixel of wiggle-room is allowed for integer rounding. Only warnings will be issued that fail these checks:
** abs(imageWidth * 2 - (coord_s2 - coord_s)) > 1
** abs(imageHeight * 2 - (coord_t2 - coord_t)) > 1

===Post-Read MINOR Warnings for FNT’s and DAT’s s, t, s2, t2===
''Low Precision.'' This value, when multiplied by 256.0, should be exactly an integer. However, some generation or revision procedure (e.g., Q3Font’s DAT  FNT  DAT) may have not used sufficient precision to ensure this. An offset from an integer value of less than +/- 0.001 is considered a minor problem; otherwise, major as above.

This problem, when it occurs with a file, typically happens a lot. Consequently, it is not reported in an itemized way, but instead, the console will show total counts of affected characters and values.

Edge cases: a value of {s, s2, t, or t2} * 256.0 that is negative but closer to zero than -0.001000 is considered this type of minor problem (and is not considered out of range, which would be major). Likewise, if that value is greater than 256.000000, but less than 256.001000
===Automatic Corrections during Output to a REF File===
Certain problems, first detected during DAT input, are further addressed at write time. Because these problems were already reported to the console during the DAT read, nothing additional is written to console. Instead, there may be reporting within the REF file itself.

''Automatic Corrections.'' When writing a REF file, the integer values of coord_s, coord_t, coord_s2, and coord_t2 are calculated from DAT floats s, t, s2, and t2 respectively. A calculated coord_... value, if not already resulting in an integer, is automatically forced to the nearest integer. This is so whether it was earlier reported to the console as a Major or Minor Warning. (Before refont v 2.1, the REF might assign an unrounded decimal value; no longer true.)

''Warnings Not Automatically Corrected.'' Automatic fixing of other warnings is unwise and not offered. Why? Because the fix is not obvious or complex, and sometimes, the corresponding character glyph in the bitmap needs adjustment. This pertains to Major Warnings about:
* Integer values outside their expected range (e.g., 0-256), including post-rounding coord_... values.
* Inconsistent Metrics
===Generated Within-REF Comments during Output===
For most causes of a Major Warning during DAT read, during subsequent writing of the REF file, a similar comment is also generated within the REF, on the same line as the parameter assignment. For example:
char 95 (0x5f) // _
{
...
top -6 // WARNING: Not in range 0-256. Does bitmap or other metrics need adjusting?
...
}
...
char 140 (0x8c) // Ń
{
...
coord_t -10 // WARNING: Underlying DAT value -0.03906250, * 256.0 gave -10.000000000, not in range 0-256 once rounded. Does bitmap or other metrics need adjusting?
...
}
''Suppressing Comment Generation.'' Use the refont option “-no_warn_comments”. ''New to v 2.1.''

TIP: When editing a REF file with comments, you can keep them, change them, or delete them as you prefer. They have no impact on subsequent compiles to DAT.

''Coverage of Generated Within-REF Comments.'' A calculated value of coord_s, coord_t, coord_s2, or coord_t2 would have been flagged as a “Major Warning” during read if:
* the discrepancy from being an integer was non-trivial (+/- 0.001 or more), or
* the calculated and rounded integer is outside the range given by bitmap size, e.g., 0-256.

For these, a comment is generated. Other causes are:
* Integer Out of Range 0-256 ''(or 0-512 as applicable)''. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).

''Major Warnings that Don’t Cause Within-REF Comments.'' Currently, this is limited to “Inconsistent Metrics”, which would probably require tagging on multiple lines.

== Special Handling of Fonts with 512x512 Bitmaps ==
Some DAT files reference bitmap shaders where some or all are of size 512x512, instead of the default 256x256. Refont v2.5+ can support this (replacing an insufficient first attempt in v2.4).

Font developers went to double-sized bitmaps to:
* in the case of Carleton, allow freehand drawing of missing European characters, which could be drawn more easily if twice standard size.
* in the case of Mason, have larger ASCII alphanumeric glyphs that would render with crisper edges.

In either case, the font characters when rendered in, say, the game main menu will be the same size as if from 256x256. This is because the imageWidth, imageHeight, height, and top metrics are not doubled.
In effect, a scaling by half (approximately, as constrained by integer values) is requested for all characters in the DAT file and applied by the engine.

=== When generating a REF file from a DAT file ===

refont -decompile <path to given .dat file> -512_<n> ''where <n> is a single digit, e.g. '1' in referenced shader name "font/Carleton_1_24.tga" ''

You should use multiple instances of the -512_<n> option, to cover all the 512x512 bitmaps found in the DAT file.

If ''all'' of a given DATs shaders are 512x512, you can use this shorter form:

refont -decompile <path to given .dat file> -512

Exercising -512... options will mean that in the generated REF file:

* There will be a one or more starting lines like:
# bitmap_<n>_size=512 ''(Refont will look for this if doing a subsequent compile to DAT)''
* If you specified the short form, it will expand to just the right number of starting lines.
* Values for coord_s, coord_s2, coord_t, and coord_t2 will have a normal range of 0-512 when appropriate. [COMING SOON: Those values may then be read properly by datBounds.].
* Any warnings expressed (within the file or within the command-line window) will be in terms of the correct range of values.
* Warning checks will expect a scaling by half for all characters. That is, while 512... requires scaling down by half, that scaling will not generate a warning (unless it's not EXACTLY by half, or within 1 pixel of that), so the -scaling_ok option is not needed just for the exactly-by-half scaling.

=== When generating a DAT file from a REF file ===

During refont -compile, no special command line options are needed. Instead, REF's starting lines of form "# bitmap_<n>_size=512" are inspected and coord ranges will be interpreted as 0-512 whenever appropriate.

=== Special Handling Recommendations for Carleton and Mason ===
''For fonts shipped with TDM 2.12 in tdm_fonts01.pk4. Illustrative command paths are shown; adjust to your working copy.''
==== Mason Family ====
The [[Mason Font]] uses within-DAT per-character scaling, as well as some but not all 512x512 bitmaps. Only 48pt Mason is shipped.

Because, under /dds/fonts/english/mason/, bitmaps masonalternate_0_48.dds to _2_ are 512x512, but _3_ to _5_ are not, use:

-decompile /fonts/english/mason/fontimage_48.dat -512_0 -512_1 -512_2 -scaling_ok

Because, under /dds/fonts/english/mason_glow/, bitmaps masonalternate_1_48.dds and _2_ are 512x512, but _0_ and _3_ to _5_ are not, use:

-decompile /fonts/english/mason_glow/fontimage_48.dat -512_1 -512_2 -scaling_ok

Russian versions use the default 256x256. [These possibly benefit from -scaling_ok; didn't investigate.]
==== Carleton Family ====
The 24pt size of carleton has two bitmaps under /dds/fonts/english/carleton/, namely carleton_0_24.dds and _1_, both of size 512x512. So use:

-decompile /fonts/english/carleton/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton/fontimage_24.dat -512

The 24pt size of carleton_bold has bitmaps with identical content to carleton's, so use the same options:

-decompile /fonts/english/carleton_bold/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton_bold/fontimage_24.dat -512

Other members of this family use the default 256x256 bitmaps:
* Sizes 12pt & 48pt
* carleton_condensed and carleton_glow. (Note: while helpful, it is not required that the base & glow have identical bitmap sizes.)
* Russian versions

==== These Fonts with Stats ====
With -stats discussed next, if reading a DAT file, you may use the -512... and scaling_ok options to be most correct, although neglecting to specify them will likely have no significant impact on the analytical results. If reading a REF file, the included "# bitmap_..." lines will take care of that for you.

== For Statistics on Unimplemented or Problematic Font Characters ==
DAT or (as of v2.3) REF file analysis here is primarily designed to benefit english/european fonts. It will use the optional annotation file if provided. Syntax:

refont -stats <path to given .dat or .ref file>

Use "-stats" by itself, not combined with "-compile" or "-decompile".

The resulting analysis report will have the same name and location as the input file, but with a ".txt" suffix.
The report provides categorized totals and (for problematic characters) itemizations across all 256 character codepoints.
The counts of problems should be considered minimums, since no inspection of bitmap files (TGA or DDS) is done.
But a perhaps surprising amount of insight can be gleaned just from DAT/REF file metrics.
Itemizations include annotations from the 'ref_char_annotation' file, if provided.

In the hunt for missing characters, 3 signatures are important:

* Presumed 'Hollow Box' = glyph's 'shadername' is <fontname>_0_<size>.dds only, and s & t are zero, but not s2 & t2
* '<Space>' = same test as 'Hollow Box', but pointed to by char 32 (0x20)
* 'Zero Box' = No glyph box (s, t, s2, t2 all zero)

A given analysis will assume either 'Hollow Box' or '<Space>' but not both.

The analysis provides totals and itemizations - grouped into lower (0-127) and upper ranges (128-255) - in several passes:

* Pass 1 - Handling of unprintable/unsupported/missing codepoints, indicated by Hollow Box, Zero Box, or <Space>. Some of these are not a problem, but some are tagged as "Undesirable".
* Pass 2 - Bad glyph box (negative s, t, s2, or t2; or s2 <= s, t2 <= t) or good glyph box with dubious metrics (imageHeight <= 0, imageWidth <=0, imageHeight != height). Excludes those already counted as "Undesirable" in Pass 1.
* Pass 3 - Detection of duplicate glyph boxes (other than Hollow Box, Zero Box, or <Space>). Detected by: glyph's values for shadername, s, t, s2, & t2 exactly match those of another codepoint. These are grouped into "Dup Sets".

Then, across passes 1-3, a count if given of the minimum "Total Glyphs Needing Work".

Refont version 2.3 adds additional passes:

* Pass 4 - Reports "Overlap Sets" rather than just "Dup Sets". A set here may include an unaccented "base character" that has been serving as a substitute work-around target for one or more missing accented characters. (What constitutes an "overlap" is idiosyncratic; see the source code for details.)
* Pass 5 - Same information as Pass 4, but sorted first by shader name (i.e., bitmap). Helpful for organizing bitmap-editing work.
* Pass 6 - An analysis focusing just on the planned migration of duplicate O/o-circumflex codepoints to G/g-cup glyphs.

== Downloads ==
Release 2.8 of March 19, 2026:
* [https://drive.google.com/file/d/1BHGkCOhgXP7bFp1mXyfgoRBUtqkbCVRb/view?usp=sharing refont.exe]
* [https://drive.google.com/file/d/1Cca_Xo3W5jYB_40RVo4aIPHAGIufA9V0/view?usp=sharing refont_source_v2.8.zip, with refont.cpp, refont.sln, open-source license]

Release 2.6 of July 28, 2025:
* [https://drive.google.com/file/d/1ABf4771Jr1C1hKbLx4xWq8lx8Kn7HdKP/view?usp=sharing refont.exe]
* [https://drive.google.com/file/d/1-k68eo0CHKZ-sM7JGP05axm49cc5De0T/view?usp=sharing source code: refont.cpp]

===Annotations===
''After download, to use any one of these, edit filename to remove "[...]" substring.''

Update of June 20, 2024. This reflects a change to the TDM charmap for 2.13, to replace duplicate O/o circumflex characters with G/g breve:

* refont_char_annotations[english with symbol only. With G-breve update].txt [https://drive.google.com/file/d/1rrt-O9DgD7Zn2Hxt7nBtViIlO4kv_gQB/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes & names. With G-breve update].txt [https://drive.google.com/file/d/1AMxDKQc_DE33gsExiEkGp3x67eb4aGFh/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source. With G-breve update].txt [https://drive.google.com/file/d/1xoeZ1poTbmkls-gE5PnbbMxdoz5KnMD7/view?usp=sharing].
* For Russian, see the previous update next.

Update of April 13, 2024:

* refont_char_annotations[english with symbol only].txt [https://drive.google.com/file/d/1ItM-aXps0xQE655HdaJM7Aje5lHewUfC/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes and names].txt [https://drive.google.com/file/d/1gW7oO_wLqCEPSdm6b05OaTNYHnyuUSA-/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source].txt [https://drive.google.com/file/d/1ZTcgA4QJYwXzNdqbolYkn0CCEDILD1bX/view?usp=sharing here]. Corrects and supersedes original english file.
* refont_char_annotations[russian].txt [https://drive.google.com/file/d/10txNXsSCtaTiT2iGT9o3iDxBU7HVYNxi/view?usp=sharing here]. Recommended for 'russian' fonts.

== For More ==
* See the [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ summary analysis of 2.12 TDM fonts], based on applying 'refont -stats ...' to all 'english' DAT files.

{{tutorial}} [[Category:Fonts]]

Refont

2026-03-18T17:25:03Z

Geep: /* Special Handling of Fonts with 512x512 Bitmaps */ v2.8 adds 1-pixel slack, suppress warnings

''By Geep, 2024, and similar to the article about Q3Font.''

Introduced in 2024, the "refont" command-line utility allows inspection and alteration of [[Font Metrics & DAT File Format | font metrics]].
It is designed to be an easier-to-use version of the inspection/alteration functionalities of [[Q3Font]].
It also provides a font analysis feature.

== Comparison with Font Patcher ==

Refont can be seen as complementary to [[Font Patcher]]. The latter has additional features that may be better if planning a wholesale rearrangement or expansion of characters within a bitmap, particularly for a new font immediately after conversion from a TTF font. But refont is easier to use in later years, and Font Patcher requires installation of Perl, while refont.exe is more drop and go.

== Comparison with Q3Font ==

For metric alteration, q3font has a native "*.fnt" human-readable format, and supports DAT <==> FNT conversions. For compatibility, so does refont.

=== Advantages of Q3Font ===

* This has been an important traditional tool for Doom3/TDM font manipulation.
* In addition to production/consumption of FNT, it -
** Allows creation of DAT & TGA files from a TrueType font (although for this purpose, [[ExportFontToDoom3]] has been preferred in practice).
** Can write (but not read) an alternative "compact report".

=== Advantages of Refont ===

* It's open-source, so further C++ development is possible.
* It has its own native human-readable format, "*.ref", similar to .fnt but better.
* The REF format can be read by [[datBounds]] to generate bitmap boundary & metrics visualizations.

The better features of REF are:

* Codepoints are enumerated in not just decimal, but also hex.
* Coordinates of the overall image box within its bitmap are expressed in pixel units, rather than [0...1] fractional floating point. So the user is spared having to manually do this tedious and error-prone calculation.
* This overcomes a drawback of FNT, that expresses the coordinates with only 6 digits after the decimal point, not quite enough to avoid minor mis-precisions.
* if optionally provided, a separate, read-only annotation file will automatically decorate the .ref file with helpful information as the file is generated. This is explained below.

Because the code is open-source, there's better clarity about -
* how parsing and calculations are done.
* what warnings are emitted.

Also, using "-stats", refont can write (but not read) an analysis of a DAT file, particularly looking for problematic/unimplemented Western-language characters.

== For Inspection & Correction of DAT files ==

''To Get the Data in Readable, Editable Form''

For viewing and altering in a text editor, two formats are provided. Both list every codepoint, in order from 0 to 255, as a block of data. Within each block, every item gets its own line.

=== As a FNT File ===

To create a q3font-compatable "<dat-name>.fnt" file:

refont -decompile <path to given .dat file> -fnt

where the path can be a full path, a relative path (e.g., starting with "./" or "../"), or just the file name in the current working directory. The resulting .fnt file will appear in the same directory. It have the same name as the .dat file, except for the extension. As a convenience to keep track of versions, it is not required that the file name be in the standard fontImage_nn.dat form.

Example code block for the letter A:
...
char 65
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
s 0.769531
t 0.250000
s2 0.847656
t2 0.320313
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Note that Q3Font FNT expresses the s, t, s2, and t2 coordinates with only 6 digits after the decimal point, causing minor low-precision rounding during DAT --> FNT --> DAT roundtrips. For compatibility, Refont FNT does likewise.

=== As a REF File ===
''Starting with Refont v2.5, a mixture of 256x256 and 512x512 bitmap shaders, encountered with Carleton and Mason family fonts, can be handled. See further below.''

To create a refont-specific "<dat-name>.ref" file:

refont -decompile <path to given .dat file>

The same path and naming conventions as for .fnt generation discussed above apply here.

Example code block for the letter A when no supplemental annotation file is present (nor any per-line comments, e.g., with warnings):

...
char 65 (0x41)
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Items bolded differ from FNT format. The starting line expresses the value in hex. And rather than s, t, s2, t2 in fractional [0...1] floating point, the "coord_..." values are expressed (and editable) as integer coordinates of the bitmap, with a range reflecting bitmap size. The default TDM bitmap size is 256x256, but a few fonts use 512x512 and require special treatment discussed further below.

== When Editing ==
Refont has a minimal parser. To keep it happy, preserve the file's line structure. In particular:
* every REF and FNT file has the same number of lines. ''(Exception: REF may have additional starting lines to flag 512x512 bitmaps, discussed below. Generally, you won't edit these.)''
* all keywords within a block must be present, and in a particular order.
* each block must have exactly the same number of lines (except for the special one at the end).

The meaning of the font metrics are explained in [[Font_Metrics_%26_DAT_File_Format]]. The [[Q3Font]] article has an example of simple metric editing that is also pertinent to refont.

== To Move Changed FNT or REF Data Back to a DAT ==

refont -compile <path to given .fnt or .ref file>

This reverses the -decompile process, to create a .dat file whose name is derived from the .fnt or .ref file. Importantly, this decompile/compile cycle leaves .tga/.dds files untouched.

== The Annotation File ==
When told to generate a REF file (but not FNT), refont.exe also looks for a file called "refont_char_annotations.txt", in the same folder as refont.exe. That file should have exactly 256 data lines in it, one for each of the codepoints, in order. As the REF is generated, the line that starts a codepoint's data block gets the corresponding annotation line appended to it. Having the annotations come from an external file, instead of hard-coded into refont, allows required flexibility as to what information is presented. It even potentially allows refont to be used with other idTech3/4 games that don't necessarily used the TDM-specific codepoints. Also, the annotation files contain helpful specific comments (as lines that start with ";").

Each annotation:
* indicates what character symbol is ''expected''; it is oblivious as to what character if any is actually within the bounding box described by the DAT data.
* lives only in a REF file, and is not retained when converted to a DAT file. This also true for any manual supplementary annotations you might make.

This is clearer with examples. In 'Downloads' below are 4 different versions of "refont_char_annotations[...].txt". Three of these, specific to TDM's custom 'english' (i.e., composite European) codepage, are discussed here. To deploy, edit the filename of one of them to remove the square bracket text.

=== English with Symbol Only ===

This simple format is best for most purposes. The file entry for letter 'A' is just:

A

Then the block in a generated REF file would have that string appended, but with " // " inserted.
...
char 65 (0x41) // A
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

=== English with Unicode-16 Codes and Names ===

This supplies the Unicode Consortium's official 16-bit codepoints (where U+NNNN = 0xNNNN) and their names, in the formal all-caps convention.

The file entry for letter 'A' is:

U+0041 LATIN CAPITAL LETTER A

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // U+0041 LATIN CAPITAL LETTER A
...
Here's another example from that REF:
...
char 129 (0x81) // U+015A LATIN CAPITAL LETTER S WITH ACUTE
...
For control or TDM undefined characters, this annotation file assumes a hollow box glyph is the appropriate mapping:
...
char 0 (0x00) // U+24A1 WHITE SQUARE
...
But there are other choices, so edit the annotation file as you see fit.
=== English with 8859-x Source ===

This verbose format may be useful during font analysis. For letters that are in their expected locations as defined by ASCII or ISO 8859-1 (Latin-1) or Windows-1252 ("Western Europe"), the annotation is fairly minimal. Thus, letter 'A' has this:

0x41 is A

(TIP: If authoring your own annotation file, consider including a codepoint number on each line, like "0x41" here, to make the process easier.)

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // 0x41 is A
...
Here are a few more interesting examples from that REF, with more expansive annotations:
...
char 5 (0x05) // 0x05 is a control character, unusable in TDM
...
char 166 (0xa6) // 0xA6 is Š (not ISO 8859-1 ¦) (from A9 in ISO 8859-2) (TDM only)
...

== Revising or Making Your Own Annotations File ==
If creating your own refont_char_annotations.txt file, note that encoding upper-range characters in UTF-8 will be more universal than using Windows region-specific codepages. To enter a UTF-8 character using its four digit hex Unicode (say, in a range of interest 0x0080-0x00FF):
* Under Windows, type the four hex digits followed by Alt-X.
* Under Linux, hold down Ctrl+Shift, type U followed by the 4 digits; release Ctrl+Shift.

When refont reads refont_char_annotations.txt, any line that starts with ";" will be ignored. The latter is so that you can put comments into the file, that will not propagate to the generated REF file. TIP: If such commenting is unwanted, start the line with <space> before ";".

Other than that, be sure to have exactly 256 lines of annotations (though refont will check and issue a warning if not). A line can be empty, i.e., have only the linebreak.

When generating a REF file, refont will automatically add " // " before each non-empty annotation, so don't include that.

== Adding Your Own Annotations to a REF File ==
Beyond the refont_char_annotation.txt content, you can add certain commentary to the REF file without it affecting its ability to convert to a DAT file. Here are the rules:
* Don't add or subtract lines, or comment out non-blank lines
* Feel free to append a comment to the end of a line, by beginning the comment with " //".
* Likewise, on a line starting with "char" that had an annotation automatically added, you can revise the annotation as you see fit.

Example with revision in bold:
...
char 65 (0x41) // '''Capital A was slightly clipped. Decremented s, s2 by 1. Looks OK now.'''
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 196 '''// WAS: 197'''
coord_t 64
coord_s2 216 '''// WAS: 217'''
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

In this way, you can markup a REF file as a record of work to do, work in progress, or work completed.

==Errors, Warnings, and Auto-Corrections==
''This covers the major overhaul of Refont v 2.1, and its added options -no_warn_comments and -scaling_ok.''

During the compile or decompile, errors and warnings may be generated in the console. Fatal errors are largely due to:
* file I/O problems;
* deviations from the strict DAT, REF, and FNT formats.

Warnings, which are not fatal, are largely due to anomolous numeric values. The TDM engine can tolerant some of these, so they don't necessarily need fixing.

===Fatal Errors when Parsing REF or FNT as Input===
A fatal error, due to divergence from the expected format while parsing, stops the program. While potentially due to many causes (e.g., adding or deleting lines), errors will be reported - with line number - as failures of:

''Variable Name Checks.'' The parser expects every variable name in a particular order. A mismatch happened.

''Integer Checks.'' Integer value was unparsable, or out of range for integer type. Checked for per-character variables height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and glyph. Plus REF’s coord_s, coord_t, coord_s2, and coord_ t2.

''Float Checks.'' Floating point value was unparsable, or out of range for float type. Checked for glyphScale, and FNT’s per-character variables s, t, s2, t2.

===Post-Read MAJOR Warnings about REF, FNT, and DAT Input Values===
After parsing, the input process concludes with a separate loop of non-fatal warning analysis. Warnings during input, broadly categorized as major or minor, report to the console, but don’t do any value corrections (which is instead left to the subsequent file-write stage).

''For FNT’s and DAT’s s, t, s2, t2''
* Inexact Integer. The given bounding box value, expected in the range 0-1, when multiplied by 256.0 should be exactly an integer. If not, the divergence is likely due to miscalculation. (Actually, a divergence from an integer of under +/- 0.001 is tolerated by Refont and considered a minor warning, discussed below.)
* Float Out of Range 0.0 - 1.0. Or equivalently, post-multiplication, 0.0 - 256.0. (See the discussion of the edge cases under minor warnings below.)

''For Other Metrics of DAT, FNT, and REF''
* Integer Out of Range 0-256. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and REF’s coord_s, coord_t, coord_s2, and coord_t2.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).
* Inconsistent Metrics. For a given character, this compares two metrics and looks for inconsistencies. We’ll state this here in REF terms, e.g., using “coord_s” rather than “ROUND_TO_INT(s * 256.0f)”:
** imageHeight == height?
** imageWidth == coord_s2 - coord_s?
** imageHeight == coord_t2 - coord_t?

However, warnings about those last two "Inconsistent Metrics" criteria are suppressed if:
* you use refont's "-scaling_ok" option. This is helpful when there is extensive per-character scaling. (Only TDM’s mason and mason-glow use a lot of per-character scaling, specifically for all upper-case characters.)
* you have a 512x512 bitmap, where throughout, the imageWidth and imageHeight (and some other metrics) are expected to be scaled to "half" of the corresponding coord difference. As of refont 2.8, 1 pixel of wiggle-room is allowed for integer rounding. Only warnings will be issued that fail these checks:
** abs(imageWidth * 2 - (coord_s2 - coord_s)) > 1
** abs(imageHeight * 2 - (coord_t2 - coord_t)) > 1

===Post-Read MINOR Warnings for FNT’s and DAT’s s, t, s2, t2===
''Low Precision.'' This value, when multiplied by 256.0, should be exactly an integer. However, some generation or revision procedure (e.g., Q3Font’s DAT  FNT  DAT) may have not used sufficient precision to ensure this. An offset from an integer value of less than +/- 0.001 is considered a minor problem; otherwise, major as above.

This problem, when it occurs with a file, typically happens a lot. Consequently, it is not reported in an itemized way, but instead, the console will show total counts of affected characters and values.

Edge cases: a value of {s, s2, t, or t2} * 256.0 that is negative but closer to zero than -0.001000 is considered this type of minor problem (and is not considered out of range, which would be major). Likewise, if that value is greater than 256.000000, but less than 256.001000
===Automatic Corrections during Output to a REF File===
Certain problems, first detected during DAT input, are further addressed at write time. Because these problems were already reported to the console during the DAT read, nothing additional is written to console. Instead, there may be reporting within the REF file itself.

''Automatic Corrections.'' When writing a REF file, the integer values of coord_s, coord_t, coord_s2, and coord_t2 are calculated from DAT floats s, t, s2, and t2 respectively. A calculated coord_... value, if not already resulting in an integer, is automatically forced to the nearest integer. This is so whether it was earlier reported to the console as a Major or Minor Warning. (Before refont v 2.1, the REF might assign an unrounded decimal value; no longer true.)

''Warnings Not Automatically Corrected.'' Automatic fixing of other warnings is unwise and not offered. Why? Because the fix is not obvious or complex, and sometimes, the corresponding character glyph in the bitmap needs adjustment. This pertains to Major Warnings about:
* Integer values outside their expected range (e.g., 0-256), including post-rounding coord_... values.
* Inconsistent Metrics
===Generated Within-REF Comments during Output===
For most causes of a Major Warning during DAT read, during subsequent writing of the REF file, a similar comment is also generated within the REF, on the same line as the parameter assignment. For example:
char 95 (0x5f) // _
{
...
top -6 // WARNING: Not in range 0-256. Does bitmap or other metrics need adjusting?
...
}
...
char 140 (0x8c) // Ń
{
...
coord_t -10 // WARNING: Underlying DAT value -0.03906250, * 256.0 gave -10.000000000, not in range 0-256 once rounded. Does bitmap or other metrics need adjusting?
...
}
''Suppressing Comment Generation.'' Use the refont option “-no_warn_comments”. ''New to v 2.1.''

TIP: When editing a REF file with comments, you can keep them, change them, or delete them as you prefer. They have no impact on subsequent compiles to DAT.

''Coverage of Generated Within-REF Comments.'' A calculated value of coord_s, coord_t, coord_s2, or coord_t2 would have been flagged as a “Major Warning” during read if:
* the discrepancy from being an integer was non-trivial (+/- 0.001 or more), or
* the calculated and rounded integer is outside the range given by bitmap size, e.g., 0-256.

For these, a comment is generated. Other causes are:
* Integer Out of Range 0-256 ''(or 0-512 as applicable)''. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).

''Major Warnings that Don’t Cause Within-REF Comments.'' Currently, this is limited to “Inconsistent Metrics”, which would probably require tagging on multiple lines.

== Special Handling of Fonts with 512x512 Bitmaps ==
Some DAT files reference bitmap shaders where some or all are of size 512x512, instead of the default 256x256. Refont v2.5+ can support this (replacing an insufficient first attempt in v2.4).

Font developers went to double-sized bitmaps to:
* in the case of Carleton, allow freehand drawing of missing European characters, which could be drawn more easily if twice standard size.
* in the case of Mason, have larger ASCII alphanumeric glyphs that would render with crisper edges.

In either case, the font characters when rendered in, say, the game main menu will be the same size as if from 256x256. This is because the imageWidth, imageHeight, height, and top metrics are not doubled.
In effect, a scaling by half (approximately, as constrained by integer values) is requested for all characters in the DAT file and applied by the engine.

=== When generating a REF file from a DAT file ===

refont -decompile <path to given .dat file> -512_<n> ''where <n> is a single digit, e.g. '1' in referenced shader name "font/Carleton_1_24.tga" ''

You should use multiple instances of the -512_<n> option, to cover all the 512x512 bitmaps found in the DAT file.

If ''all'' of a given DATs shaders are 512x512, you can use this shorter form:

refont -decompile <path to given .dat file> -512

Exercising -512... options will mean that in the generated REF file:

* There will be a one or more starting lines like:
# bitmap_<n>_size=512 ''(Refont will look for this if doing a subsequent compile to DAT)''
* If you specified the short form, it will expand to just the right number of starting lines.
* Values for coord_s, coord_s2, coord_t, and coord_t2 will have a normal range of 0-512 when appropriate. [COMING SOON: Those values may then be read properly by datBounds.].
* Any warnings expressed (within the file or within the command-line window) will be in terms of the correct range of values.
* Warning checks will expect a scaling by half for all characters. That is, while 512... requires scaling down by half, that scaling will not generate a warning (unless it's not EXACTLY by half, or within 1 pixel of that), so the -scaling_ok option is not needed just for the exactly-by-half scaling.

=== When generating a DAT file from a REF file ===

During refont -compile, no special command line options are needed. Instead, REF's starting lines of form "# bitmap_<n>_size=512" are inspected and coord ranges will be interpreted as 0-512 whenever appropriate.

=== Special Handling Recommendations for Carleton and Mason ===
''For fonts shipped with TDM 2.12 in tdm_fonts01.pk4. Illustrative command paths are shown; adjust to your working copy.''
==== Mason Family ====
The [[Mason Font]] uses within-DAT per-character scaling, as well as some but not all 512x512 bitmaps. Only 48pt Mason is shipped.

Because, under /dds/fonts/english/mason/, bitmaps masonalternate_0_48.dds to _2_ are 512x512, but _3_ to _5_ are not, use:

-decompile /fonts/english/mason/fontimage_48.dat -512_0 -512_1 -512_2 -scaling_ok

Because, under /dds/fonts/english/mason_glow/, bitmaps masonalternate_1_48.dds and _2_ are 512x512, but _0_ and _3_ to _5_ are not, use:

-decompile /fonts/english/mason_glow/fontimage_48.dat -512_1 -512_2 -scaling_ok

Russian versions use the default 256x256. [These possibly benefit from -scaling_ok; didn't investigate.]
==== Carleton Family ====
The 24pt size of carleton has two bitmaps under /dds/fonts/english/carleton/, namely carleton_0_24.dds and _1_, both of size 512x512. So use:

-decompile /fonts/english/carleton/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton/fontimage_24.dat -512

The 24pt size of carleton_bold has bitmaps with identical content to carleton's, so use the same options:

-decompile /fonts/english/carleton_bold/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton_bold/fontimage_24.dat -512

Other members of this family use the default 256x256 bitmaps:
* Sizes 12pt & 48pt
* carleton_condensed and carleton_glow. (Note: while helpful, it is not required that the base & glow have identical bitmap sizes.)
* Russian versions

==== These Fonts with Stats ====
With -stats discussed next, if reading a DAT file, you may use the -512... and scaling_ok options to be most correct, although neglecting to specify them will likely have no significant impact on the analytical results. If reading a REF file, the included "# bitmap_..." lines will take care of that for you.

== For Statistics on Unimplemented or Problematic Font Characters ==
DAT or (as of v2.3) REF file analysis here is primarily designed to benefit english/european fonts. It will use the optional annotation file if provided. Syntax:

refont -stats <path to given .dat or .ref file>

Use "-stats" by itself, not combined with "-compile" or "-decompile".

The resulting analysis report will have the same name and location as the input file, but with a ".txt" suffix.
The report provides categorized totals and (for problematic characters) itemizations across all 256 character codepoints.
The counts of problems should be considered minimums, since no inspection of bitmap files (TGA or DDS) is done.
But a perhaps surprising amount of insight can be gleaned just from DAT/REF file metrics.
Itemizations include annotations from the 'ref_char_annotation' file, if provided.

In the hunt for missing characters, 3 signatures are important:

* Presumed 'Hollow Box' = glyph's 'shadername' is <fontname>_0_<size>.dds only, and s & t are zero, but not s2 & t2
* '<Space>' = same test as 'Hollow Box', but pointed to by char 32 (0x20)
* 'Zero Box' = No glyph box (s, t, s2, t2 all zero)

A given analysis will assume either 'Hollow Box' or '<Space>' but not both.

The analysis provides totals and itemizations - grouped into lower (0-127) and upper ranges (128-255) - in several passes:

* Pass 1 - Handling of unprintable/unsupported/missing codepoints, indicated by Hollow Box, Zero Box, or <Space>. Some of these are not a problem, but some are tagged as "Undesirable".
* Pass 2 - Bad glyph box (negative s, t, s2, or t2; or s2 <= s, t2 <= t) or good glyph box with dubious metrics (imageHeight <= 0, imageWidth <=0, imageHeight != height). Excludes those already counted as "Undesirable" in Pass 1.
* Pass 3 - Detection of duplicate glyph boxes (other than Hollow Box, Zero Box, or <Space>). Detected by: glyph's values for shadername, s, t, s2, & t2 exactly match those of another codepoint. These are grouped into "Dup Sets".

Then, across passes 1-3, a count if given of the minimum "Total Glyphs Needing Work".

Refont version 2.3 adds additional passes:

* Pass 4 - Reports "Overlap Sets" rather than just "Dup Sets". A set here may include an unaccented "base character" that has been serving as a substitute work-around target for one or more missing accented characters. (What constitutes an "overlap" is idiosyncratic; see the source code for details.)
* Pass 5 - Same information as Pass 4, but sorted first by shader name (i.e., bitmap). Helpful for organizing bitmap-editing work.
* Pass 6 - An analysis focusing just on the planned migration of duplicate O/o-circumflex codepoints to G/g-cup glyphs.

== Downloads ==
Release 2.6 of July 28, 2025:
* [https://drive.google.com/file/d/1ABf4771Jr1C1hKbLx4xWq8lx8Kn7HdKP/view?usp=sharing refont.exe]
* [https://drive.google.com/file/d/1-k68eo0CHKZ-sM7JGP05axm49cc5De0T/view?usp=sharing source code: refont.cpp]

===Annotations===
''After download, to use any one of these, edit filename to remove "[...]" substring.''

Update of June 20, 2024. This reflects a change to the TDM charmap for 2.13, to replace duplicate O/o circumflex characters with G/g breve:

* refont_char_annotations[english with symbol only. With G-breve update].txt [https://drive.google.com/file/d/1rrt-O9DgD7Zn2Hxt7nBtViIlO4kv_gQB/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes & names. With G-breve update].txt [https://drive.google.com/file/d/1AMxDKQc_DE33gsExiEkGp3x67eb4aGFh/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source. With G-breve update].txt [https://drive.google.com/file/d/1xoeZ1poTbmkls-gE5PnbbMxdoz5KnMD7/view?usp=sharing].
* For Russian, see the previous update next.

Update of April 13, 2024:

* refont_char_annotations[english with symbol only].txt [https://drive.google.com/file/d/1ItM-aXps0xQE655HdaJM7Aje5lHewUfC/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes and names].txt [https://drive.google.com/file/d/1gW7oO_wLqCEPSdm6b05OaTNYHnyuUSA-/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source].txt [https://drive.google.com/file/d/1ZTcgA4QJYwXzNdqbolYkn0CCEDILD1bX/view?usp=sharing here]. Corrects and supersedes original english file.
* refont_char_annotations[russian].txt [https://drive.google.com/file/d/10txNXsSCtaTiT2iGT9o3iDxBU7HVYNxi/view?usp=sharing here]. Recommended for 'russian' fonts.

== For More ==
* See the [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ summary analysis of 2.12 TDM fonts], based on applying 'refont -stats ...' to all 'english' DAT files.

{{tutorial}} [[Category:Fonts]]

Refont

2026-03-18T17:18:00Z

Geep: /* Post-Read MAJOR Warnings about REF, FNT, and DAT Input Values */

''By Geep, 2024, and similar to the article about Q3Font.''

Introduced in 2024, the "refont" command-line utility allows inspection and alteration of [[Font Metrics & DAT File Format | font metrics]].
It is designed to be an easier-to-use version of the inspection/alteration functionalities of [[Q3Font]].
It also provides a font analysis feature.

== Comparison with Font Patcher ==

Refont can be seen as complementary to [[Font Patcher]]. The latter has additional features that may be better if planning a wholesale rearrangement or expansion of characters within a bitmap, particularly for a new font immediately after conversion from a TTF font. But refont is easier to use in later years, and Font Patcher requires installation of Perl, while refont.exe is more drop and go.

== Comparison with Q3Font ==

For metric alteration, q3font has a native "*.fnt" human-readable format, and supports DAT <==> FNT conversions. For compatibility, so does refont.

=== Advantages of Q3Font ===

* This has been an important traditional tool for Doom3/TDM font manipulation.
* In addition to production/consumption of FNT, it -
** Allows creation of DAT & TGA files from a TrueType font (although for this purpose, [[ExportFontToDoom3]] has been preferred in practice).
** Can write (but not read) an alternative "compact report".

=== Advantages of Refont ===

* It's open-source, so further C++ development is possible.
* It has its own native human-readable format, "*.ref", similar to .fnt but better.
* The REF format can be read by [[datBounds]] to generate bitmap boundary & metrics visualizations.

The better features of REF are:

* Codepoints are enumerated in not just decimal, but also hex.
* Coordinates of the overall image box within its bitmap are expressed in pixel units, rather than [0...1] fractional floating point. So the user is spared having to manually do this tedious and error-prone calculation.
* This overcomes a drawback of FNT, that expresses the coordinates with only 6 digits after the decimal point, not quite enough to avoid minor mis-precisions.
* if optionally provided, a separate, read-only annotation file will automatically decorate the .ref file with helpful information as the file is generated. This is explained below.

Because the code is open-source, there's better clarity about -
* how parsing and calculations are done.
* what warnings are emitted.

Also, using "-stats", refont can write (but not read) an analysis of a DAT file, particularly looking for problematic/unimplemented Western-language characters.

== For Inspection & Correction of DAT files ==

''To Get the Data in Readable, Editable Form''

For viewing and altering in a text editor, two formats are provided. Both list every codepoint, in order from 0 to 255, as a block of data. Within each block, every item gets its own line.

=== As a FNT File ===

To create a q3font-compatable "<dat-name>.fnt" file:

refont -decompile <path to given .dat file> -fnt

where the path can be a full path, a relative path (e.g., starting with "./" or "../"), or just the file name in the current working directory. The resulting .fnt file will appear in the same directory. It have the same name as the .dat file, except for the extension. As a convenience to keep track of versions, it is not required that the file name be in the standard fontImage_nn.dat form.

Example code block for the letter A:
...
char 65
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
s 0.769531
t 0.250000
s2 0.847656
t2 0.320313
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Note that Q3Font FNT expresses the s, t, s2, and t2 coordinates with only 6 digits after the decimal point, causing minor low-precision rounding during DAT --> FNT --> DAT roundtrips. For compatibility, Refont FNT does likewise.

=== As a REF File ===
''Starting with Refont v2.5, a mixture of 256x256 and 512x512 bitmap shaders, encountered with Carleton and Mason family fonts, can be handled. See further below.''

To create a refont-specific "<dat-name>.ref" file:

refont -decompile <path to given .dat file>

The same path and naming conventions as for .fnt generation discussed above apply here.

Example code block for the letter A when no supplemental annotation file is present (nor any per-line comments, e.g., with warnings):

...
char 65 (0x41)
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Items bolded differ from FNT format. The starting line expresses the value in hex. And rather than s, t, s2, t2 in fractional [0...1] floating point, the "coord_..." values are expressed (and editable) as integer coordinates of the bitmap, with a range reflecting bitmap size. The default TDM bitmap size is 256x256, but a few fonts use 512x512 and require special treatment discussed further below.

== When Editing ==
Refont has a minimal parser. To keep it happy, preserve the file's line structure. In particular:
* every REF and FNT file has the same number of lines. ''(Exception: REF may have additional starting lines to flag 512x512 bitmaps, discussed below. Generally, you won't edit these.)''
* all keywords within a block must be present, and in a particular order.
* each block must have exactly the same number of lines (except for the special one at the end).

The meaning of the font metrics are explained in [[Font_Metrics_%26_DAT_File_Format]]. The [[Q3Font]] article has an example of simple metric editing that is also pertinent to refont.

== To Move Changed FNT or REF Data Back to a DAT ==

refont -compile <path to given .fnt or .ref file>

This reverses the -decompile process, to create a .dat file whose name is derived from the .fnt or .ref file. Importantly, this decompile/compile cycle leaves .tga/.dds files untouched.

== The Annotation File ==
When told to generate a REF file (but not FNT), refont.exe also looks for a file called "refont_char_annotations.txt", in the same folder as refont.exe. That file should have exactly 256 data lines in it, one for each of the codepoints, in order. As the REF is generated, the line that starts a codepoint's data block gets the corresponding annotation line appended to it. Having the annotations come from an external file, instead of hard-coded into refont, allows required flexibility as to what information is presented. It even potentially allows refont to be used with other idTech3/4 games that don't necessarily used the TDM-specific codepoints. Also, the annotation files contain helpful specific comments (as lines that start with ";").

Each annotation:
* indicates what character symbol is ''expected''; it is oblivious as to what character if any is actually within the bounding box described by the DAT data.
* lives only in a REF file, and is not retained when converted to a DAT file. This also true for any manual supplementary annotations you might make.

This is clearer with examples. In 'Downloads' below are 4 different versions of "refont_char_annotations[...].txt". Three of these, specific to TDM's custom 'english' (i.e., composite European) codepage, are discussed here. To deploy, edit the filename of one of them to remove the square bracket text.

=== English with Symbol Only ===

This simple format is best for most purposes. The file entry for letter 'A' is just:

A

Then the block in a generated REF file would have that string appended, but with " // " inserted.
...
char 65 (0x41) // A
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

=== English with Unicode-16 Codes and Names ===

This supplies the Unicode Consortium's official 16-bit codepoints (where U+NNNN = 0xNNNN) and their names, in the formal all-caps convention.

The file entry for letter 'A' is:

U+0041 LATIN CAPITAL LETTER A

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // U+0041 LATIN CAPITAL LETTER A
...
Here's another example from that REF:
...
char 129 (0x81) // U+015A LATIN CAPITAL LETTER S WITH ACUTE
...
For control or TDM undefined characters, this annotation file assumes a hollow box glyph is the appropriate mapping:
...
char 0 (0x00) // U+24A1 WHITE SQUARE
...
But there are other choices, so edit the annotation file as you see fit.
=== English with 8859-x Source ===

This verbose format may be useful during font analysis. For letters that are in their expected locations as defined by ASCII or ISO 8859-1 (Latin-1) or Windows-1252 ("Western Europe"), the annotation is fairly minimal. Thus, letter 'A' has this:

0x41 is A

(TIP: If authoring your own annotation file, consider including a codepoint number on each line, like "0x41" here, to make the process easier.)

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // 0x41 is A
...
Here are a few more interesting examples from that REF, with more expansive annotations:
...
char 5 (0x05) // 0x05 is a control character, unusable in TDM
...
char 166 (0xa6) // 0xA6 is Š (not ISO 8859-1 ¦) (from A9 in ISO 8859-2) (TDM only)
...

== Revising or Making Your Own Annotations File ==
If creating your own refont_char_annotations.txt file, note that encoding upper-range characters in UTF-8 will be more universal than using Windows region-specific codepages. To enter a UTF-8 character using its four digit hex Unicode (say, in a range of interest 0x0080-0x00FF):
* Under Windows, type the four hex digits followed by Alt-X.
* Under Linux, hold down Ctrl+Shift, type U followed by the 4 digits; release Ctrl+Shift.

When refont reads refont_char_annotations.txt, any line that starts with ";" will be ignored. The latter is so that you can put comments into the file, that will not propagate to the generated REF file. TIP: If such commenting is unwanted, start the line with <space> before ";".

Other than that, be sure to have exactly 256 lines of annotations (though refont will check and issue a warning if not). A line can be empty, i.e., have only the linebreak.

When generating a REF file, refont will automatically add " // " before each non-empty annotation, so don't include that.

== Adding Your Own Annotations to a REF File ==
Beyond the refont_char_annotation.txt content, you can add certain commentary to the REF file without it affecting its ability to convert to a DAT file. Here are the rules:
* Don't add or subtract lines, or comment out non-blank lines
* Feel free to append a comment to the end of a line, by beginning the comment with " //".
* Likewise, on a line starting with "char" that had an annotation automatically added, you can revise the annotation as you see fit.

Example with revision in bold:
...
char 65 (0x41) // '''Capital A was slightly clipped. Decremented s, s2 by 1. Looks OK now.'''
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 196 '''// WAS: 197'''
coord_t 64
coord_s2 216 '''// WAS: 217'''
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

In this way, you can markup a REF file as a record of work to do, work in progress, or work completed.

==Errors, Warnings, and Auto-Corrections==
''This covers the major overhaul of Refont v 2.1, and its added options -no_warn_comments and -scaling_ok.''

During the compile or decompile, errors and warnings may be generated in the console. Fatal errors are largely due to:
* file I/O problems;
* deviations from the strict DAT, REF, and FNT formats.

Warnings, which are not fatal, are largely due to anomolous numeric values. The TDM engine can tolerant some of these, so they don't necessarily need fixing.

===Fatal Errors when Parsing REF or FNT as Input===
A fatal error, due to divergence from the expected format while parsing, stops the program. While potentially due to many causes (e.g., adding or deleting lines), errors will be reported - with line number - as failures of:

''Variable Name Checks.'' The parser expects every variable name in a particular order. A mismatch happened.

''Integer Checks.'' Integer value was unparsable, or out of range for integer type. Checked for per-character variables height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and glyph. Plus REF’s coord_s, coord_t, coord_s2, and coord_ t2.

''Float Checks.'' Floating point value was unparsable, or out of range for float type. Checked for glyphScale, and FNT’s per-character variables s, t, s2, t2.

===Post-Read MAJOR Warnings about REF, FNT, and DAT Input Values===
After parsing, the input process concludes with a separate loop of non-fatal warning analysis. Warnings during input, broadly categorized as major or minor, report to the console, but don’t do any value corrections (which is instead left to the subsequent file-write stage).

''For FNT’s and DAT’s s, t, s2, t2''
* Inexact Integer. The given bounding box value, expected in the range 0-1, when multiplied by 256.0 should be exactly an integer. If not, the divergence is likely due to miscalculation. (Actually, a divergence from an integer of under +/- 0.001 is tolerated by Refont and considered a minor warning, discussed below.)
* Float Out of Range 0.0 - 1.0. Or equivalently, post-multiplication, 0.0 - 256.0. (See the discussion of the edge cases under minor warnings below.)

''For Other Metrics of DAT, FNT, and REF''
* Integer Out of Range 0-256. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and REF’s coord_s, coord_t, coord_s2, and coord_t2.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).
* Inconsistent Metrics. For a given character, this compares two metrics and looks for inconsistencies. We’ll state this here in REF terms, e.g., using “coord_s” rather than “ROUND_TO_INT(s * 256.0f)”:
** imageHeight == height?
** imageWidth == coord_s2 - coord_s?
** imageHeight == coord_t2 - coord_t?

However, warnings about those last two "Inconsistent Metrics" criteria are suppressed if:
* you use refont's "-scaling_ok" option. This is helpful when there is extensive per-character scaling. (Only TDM’s mason and mason-glow use a lot of per-character scaling, specifically for all upper-case characters.)
* you have a 512x512 bitmap, where throughout, the imageWidth and imageHeight (and some other metrics) are expected to be scaled to "half" of the corresponding coord difference. As of refont 2.8, 1 pixel of wiggle-room is allowed for integer rounding. Only warnings will be issued that fail these checks:
** abs(imageWidth * 2 - (coord_s2 - coord_s)) > 1
** abs(imageHeight * 2 - (coord_t2 - coord_t)) > 1

===Post-Read MINOR Warnings for FNT’s and DAT’s s, t, s2, t2===
''Low Precision.'' This value, when multiplied by 256.0, should be exactly an integer. However, some generation or revision procedure (e.g., Q3Font’s DAT  FNT  DAT) may have not used sufficient precision to ensure this. An offset from an integer value of less than +/- 0.001 is considered a minor problem; otherwise, major as above.

This problem, when it occurs with a file, typically happens a lot. Consequently, it is not reported in an itemized way, but instead, the console will show total counts of affected characters and values.

Edge cases: a value of {s, s2, t, or t2} * 256.0 that is negative but closer to zero than -0.001000 is considered this type of minor problem (and is not considered out of range, which would be major). Likewise, if that value is greater than 256.000000, but less than 256.001000
===Automatic Corrections during Output to a REF File===
Certain problems, first detected during DAT input, are further addressed at write time. Because these problems were already reported to the console during the DAT read, nothing additional is written to console. Instead, there may be reporting within the REF file itself.

''Automatic Corrections.'' When writing a REF file, the integer values of coord_s, coord_t, coord_s2, and coord_t2 are calculated from DAT floats s, t, s2, and t2 respectively. A calculated coord_... value, if not already resulting in an integer, is automatically forced to the nearest integer. This is so whether it was earlier reported to the console as a Major or Minor Warning. (Before refont v 2.1, the REF might assign an unrounded decimal value; no longer true.)

''Warnings Not Automatically Corrected.'' Automatic fixing of other warnings is unwise and not offered. Why? Because the fix is not obvious or complex, and sometimes, the corresponding character glyph in the bitmap needs adjustment. This pertains to Major Warnings about:
* Integer values outside their expected range (e.g., 0-256), including post-rounding coord_... values.
* Inconsistent Metrics
===Generated Within-REF Comments during Output===
For most causes of a Major Warning during DAT read, during subsequent writing of the REF file, a similar comment is also generated within the REF, on the same line as the parameter assignment. For example:
char 95 (0x5f) // _
{
...
top -6 // WARNING: Not in range 0-256. Does bitmap or other metrics need adjusting?
...
}
...
char 140 (0x8c) // Ń
{
...
coord_t -10 // WARNING: Underlying DAT value -0.03906250, * 256.0 gave -10.000000000, not in range 0-256 once rounded. Does bitmap or other metrics need adjusting?
...
}
''Suppressing Comment Generation.'' Use the refont option “-no_warn_comments”. ''New to v 2.1.''

TIP: When editing a REF file with comments, you can keep them, change them, or delete them as you prefer. They have no impact on subsequent compiles to DAT.

''Coverage of Generated Within-REF Comments.'' A calculated value of coord_s, coord_t, coord_s2, or coord_t2 would have been flagged as a “Major Warning” during read if:
* the discrepancy from being an integer was non-trivial (+/- 0.001 or more), or
* the calculated and rounded integer is outside the range given by bitmap size, e.g., 0-256.

For these, a comment is generated. Other causes are:
* Integer Out of Range 0-256 ''(or 0-512 as applicable)''. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).

''Major Warnings that Don’t Cause Within-REF Comments.'' Currently, this is limited to “Inconsistent Metrics”, which would probably require tagging on multiple lines.

== Special Handling of Fonts with 512x512 Bitmaps ==
Some DAT files reference bitmap shaders where some or all are of size 512x512, instead of the default 256x256. Refont v2.5+ can support this (replacing an insufficient first attempt in v2.4).

Font developers went to double-sized bitmaps to:
* in the case of Carleton, allow freehand drawing of missing European characters, which could be drawn more easily if twice standard size.
* in the case of Mason, have larger ASCII alphanumeric glyphs that would render with crisper edges.

In either case, the font characters when rendered in, say, the game main menu will be the same size as if from 256x256. This is because the imageWidth and imageHeight metrics are not doubled.
In effect, a scaling by half is requested for all characters in the DAT file and applied by the engine.

=== When generating a REF file from a DAT file ===

refont -decompile <path to given .dat file> -512_<n> ''where <n> is a single digit, e.g. '1' in referenced shader name "font/Carleton_1_24.tga" ''

You should use multiple instances of the -512_<n> option, to cover all the 512x512 bitmaps found in the DAT file.

If ''all'' of a given DATs shaders are 512x512, you can use this shorter form:

refont -decompile <path to given .dat file> -512

Exercising -512... options will mean that in the generated REF file:

* There will be a one or more starting lines like:
# bitmap_<n>_size=512 ''(Refont will look for this if doing a subsequent compile to DAT)''
* If you specified the short form, it will expand to just the right number of starting lines.
* Values for coord_s, coord_s2, coord_t, and coord_t2 will have a normal range of 0-512 when appropriate. [COMING SOON: Those values may then be read properly by datBounds.].
* Any warnings expressed (within the file or within the command-line window) will be in terms of the correct range of values.
* Warning checks will expect a scaling by half for all characters. That is, while 512... requires scaling down by half, that scaling will not generate a warning (unless it's not EXACTLY by half), so the -scaling_ok option is not needed just for the exactly-by-half scaling.

=== When generating a DAT file from a REF file ===

During refont -compile, no special command line options are needed. Instead, REF's starting lines of form "# bitmap_<n>_size=512" are inspected and coord ranges will be interpreted as 0-512 whenever appropriate.

=== Special Handling Recommendations for Carleton and Mason ===
''For fonts shipped with TDM 2.12 in tdm_fonts01.pk4. Illustrative command paths are shown; adjust to your working copy.''
==== Mason Family ====
The [[Mason Font]] uses within-DAT per-character scaling, as well as some but not all 512x512 bitmaps. Only 48pt Mason is shipped.

Because, under /dds/fonts/english/mason/, bitmaps masonalternate_0_48.dds to _2_ are 512x512, but _3_ to _5_ are not, use:

-decompile /fonts/english/mason/fontimage_48.dat -512_0 -512_1 -512_2 -scaling_ok

Because, under /dds/fonts/english/mason_glow/, bitmaps masonalternate_1_48.dds and _2_ are 512x512, but _0_ and _3_ to _5_ are not, use:

-decompile /fonts/english/mason_glow/fontimage_48.dat -512_1 -512_2 -scaling_ok

Russian versions use the default 256x256. [These possibly benefit from -scaling_ok; didn't investigate.]
==== Carleton Family ====
The 24pt size of carleton has two bitmaps under /dds/fonts/english/carleton/, namely carleton_0_24.dds and _1_, both of size 512x512. So use:

-decompile /fonts/english/carleton/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton/fontimage_24.dat -512

The 24pt size of carleton_bold has bitmaps with identical content to carleton's, so use the same options:

-decompile /fonts/english/carleton_bold/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton_bold/fontimage_24.dat -512

Other members of this family use the default 256x256 bitmaps:
* Sizes 12pt & 48pt
* carleton_condensed and carleton_glow. (Note: while helpful, it is not required that the base & glow have identical bitmap sizes.)
* Russian versions

==== These Fonts with Stats ====
With -stats discussed next, if reading a DAT file, you may use the -512... and scaling_ok options to be most correct, although neglecting to specify them will likely have no significant impact on the analytical results. If reading a REF file, the included "# bitmap_..." lines will take care of that for you.

== For Statistics on Unimplemented or Problematic Font Characters ==
DAT or (as of v2.3) REF file analysis here is primarily designed to benefit english/european fonts. It will use the optional annotation file if provided. Syntax:

refont -stats <path to given .dat or .ref file>

Use "-stats" by itself, not combined with "-compile" or "-decompile".

The resulting analysis report will have the same name and location as the input file, but with a ".txt" suffix.
The report provides categorized totals and (for problematic characters) itemizations across all 256 character codepoints.
The counts of problems should be considered minimums, since no inspection of bitmap files (TGA or DDS) is done.
But a perhaps surprising amount of insight can be gleaned just from DAT/REF file metrics.
Itemizations include annotations from the 'ref_char_annotation' file, if provided.

In the hunt for missing characters, 3 signatures are important:

* Presumed 'Hollow Box' = glyph's 'shadername' is <fontname>_0_<size>.dds only, and s & t are zero, but not s2 & t2
* '<Space>' = same test as 'Hollow Box', but pointed to by char 32 (0x20)
* 'Zero Box' = No glyph box (s, t, s2, t2 all zero)

A given analysis will assume either 'Hollow Box' or '<Space>' but not both.

The analysis provides totals and itemizations - grouped into lower (0-127) and upper ranges (128-255) - in several passes:

* Pass 1 - Handling of unprintable/unsupported/missing codepoints, indicated by Hollow Box, Zero Box, or <Space>. Some of these are not a problem, but some are tagged as "Undesirable".
* Pass 2 - Bad glyph box (negative s, t, s2, or t2; or s2 <= s, t2 <= t) or good glyph box with dubious metrics (imageHeight <= 0, imageWidth <=0, imageHeight != height). Excludes those already counted as "Undesirable" in Pass 1.
* Pass 3 - Detection of duplicate glyph boxes (other than Hollow Box, Zero Box, or <Space>). Detected by: glyph's values for shadername, s, t, s2, & t2 exactly match those of another codepoint. These are grouped into "Dup Sets".

Then, across passes 1-3, a count if given of the minimum "Total Glyphs Needing Work".

Refont version 2.3 adds additional passes:

* Pass 4 - Reports "Overlap Sets" rather than just "Dup Sets". A set here may include an unaccented "base character" that has been serving as a substitute work-around target for one or more missing accented characters. (What constitutes an "overlap" is idiosyncratic; see the source code for details.)
* Pass 5 - Same information as Pass 4, but sorted first by shader name (i.e., bitmap). Helpful for organizing bitmap-editing work.
* Pass 6 - An analysis focusing just on the planned migration of duplicate O/o-circumflex codepoints to G/g-cup glyphs.

== Downloads ==
Release 2.6 of July 28, 2025:
* [https://drive.google.com/file/d/1ABf4771Jr1C1hKbLx4xWq8lx8Kn7HdKP/view?usp=sharing refont.exe]
* [https://drive.google.com/file/d/1-k68eo0CHKZ-sM7JGP05axm49cc5De0T/view?usp=sharing source code: refont.cpp]

===Annotations===
''After download, to use any one of these, edit filename to remove "[...]" substring.''

Update of June 20, 2024. This reflects a change to the TDM charmap for 2.13, to replace duplicate O/o circumflex characters with G/g breve:

* refont_char_annotations[english with symbol only. With G-breve update].txt [https://drive.google.com/file/d/1rrt-O9DgD7Zn2Hxt7nBtViIlO4kv_gQB/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes & names. With G-breve update].txt [https://drive.google.com/file/d/1AMxDKQc_DE33gsExiEkGp3x67eb4aGFh/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source. With G-breve update].txt [https://drive.google.com/file/d/1xoeZ1poTbmkls-gE5PnbbMxdoz5KnMD7/view?usp=sharing].
* For Russian, see the previous update next.

Update of April 13, 2024:

* refont_char_annotations[english with symbol only].txt [https://drive.google.com/file/d/1ItM-aXps0xQE655HdaJM7Aje5lHewUfC/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes and names].txt [https://drive.google.com/file/d/1gW7oO_wLqCEPSdm6b05OaTNYHnyuUSA-/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source].txt [https://drive.google.com/file/d/1ZTcgA4QJYwXzNdqbolYkn0CCEDILD1bX/view?usp=sharing here]. Corrects and supersedes original english file.
* refont_char_annotations[russian].txt [https://drive.google.com/file/d/10txNXsSCtaTiT2iGT9o3iDxBU7HVYNxi/view?usp=sharing here]. Recommended for 'russian' fonts.

== For More ==
* See the [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ summary analysis of 2.12 TDM fonts], based on applying 'refont -stats ...' to all 'english' DAT files.

{{tutorial}} [[Category:Fonts]]

ExportUnicodeFontToDoom3

2026-03-16T20:40:25Z

Geep: /* unicodeMap */ Must keep file ordered

''By Geep, 2025''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints]. As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

More broadly, this program supersedes ExportFontToDoom3 and ExportFontToDoom3All256.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

IMPORTANT: If you edit lines, the remaining active glyph lines MUST be in ascending order by the 2-hex-digit codepoint value that starts each line. Otherwise, results will go astray.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

Note: The spacing value is relative to 256x256 bitmaps. If you specify "hirez", then for proportionality, your specified value is doubled internally when laying out the 512x512 bitmaps.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== hirez or hires ===
''Added with v1.1, March 2026''

Instead of 256x256 bitmaps (and associated DAT), this exports double-sized 512x512 bitmaps with larger, thus potentially-crisper glyphs. (TDM has had a few such 512x512 bitmaps for redrawn fonts since 2014.) Specifically, it:
* outputs requested 512x512 TGAs and their DAT(s).
* internally, doubles the pointsize requested of the TTF, resulting in 2x character glyph sizes. Because both the container bitmap and each glyph is (nominally) doubled in size, the [0...1] s, s2, t, t2 values stored in the DAT would be very similar to those for a 256x256 bitmap.
* Other DAT layout metrics remain as if scaled to a 256x256 layout:
** imageHeight
** imageWidth
** skip
** top (with bottom ignored as usual)
** pitch

For example, if you request –hirez with a carleton font in just 24pt type, it will create 512x512 files with usual names like carleton_0_24.tga, etc., and fontimage_24.dat, with the console showing “Exporting carleton 24pt”.
But internally, 48pt characters are drawn on the bitmaps. The imageHeight and imageWidth metrics will be as if the requested 24pt size was generated, so in effect causing the game engine to scale down by a factor of 2 from the 48pt glyph.

''TIPS on Using a HiRez DAT file''

If subsequently using Refont to make a REF file from this DAT, add the “scaling_ok” option, to suppress otherwise voluminous console warnings about per-character scaling.

If you further plan to generate bounds visualizations with datBounds:
* For 512x512 bitmaps, you must generate from the REF file, not directly from DAT.
* Be sure to first edit the REF to add headers of the form:

# bitmap_<n>_size=512

, where <n> is 0 to 9, and refers to shaders present in the REF file with names like, e.g., carleton_3_48.tga; n would be 3 for this.

* datBounds also has a worthwhile “scaling_ok” option.
* Uncommonly, the raw_metrics option may be of interest.

=== bolding <n> ===
''Added with v1.1, March 2026''

Sometimes, a TTF font does not offer a “bold” variant, or, more generally, a font’s overall stroke thickness is not ideal. It is possible to alter character outlines, then render those as bitmap glyphs.
The result is usually better than post-bitmap thickening or thinning. The FreeType library offers an embolding function for outline alteration, used here.

The integer parameter <n> is in units of 1/64th pixel. A positive value adjusts the outline to embolden the character. Negative thins. The change affects both the overall height and width of the outline.
You may think of the left and bottom character borders as unchanged. The allowed range of <n> is from -128 (i.e., -2 pixels) to 256 (4 pixels). The -2 limit is arbitrary, but for sanity. The 4 limit is the maximum FreeType permits.

Caution: The adjustment algorithm moves the outline controls points, but doesn't change the number of such points;
this means that certain situations like acute angles or intersections are sometimes handled incorrectly.

== Changed Behavior for Font Title from ExportFontToDoom3All256 ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using or revising the -name argument.

== Downloads ==
Release v1.1 of 2025-03-03:
* [https://drive.google.com/file/d/1f3Jj53qtJGIp0_mz7Riing9Fi8ESFDr5/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_exe.zip] . Windows binary and required archived versions of FreeType and DevIL DLLs.
* [https://drive.google.com/file/d/1nutMBfcIWw63wNTUdnlnzP397tGVpnpd/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_source_project.zip] . The C++ project file with source code, including source & build project of archived library.

Release 1 of 2024-09-10:
* [https://drive.google.com/file/d/1MqoOfhc5-ovoxoHfRquYAcIEdHEOJAQV/view?usp=sharing ExportUnicodeFontToDoom3_release_1_exe.zip]
* [https://drive.google.com/file/d/1Frk_byfdtCcMqdr27ulw1XyuFj8vYp9x/view?usp=sharing ExportUnicodeFontToDoom3_release_1_source_project.zip]

== See Also ==
The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

I18N - Charset

2026-03-15T20:59:00Z

Geep: /* Notes */

== Introduction ==

The D3 code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, special charsets are used. The fonts (Carleton for the menu f.i.) are build/patched so that the right characters appear in the right place.

== Encodings ==

=== all.lang ===

This file is in '''UTF-8''', and converted with the help of the script '''devel/gen_lang.pl''':

perl devel/gen_lang.pl

This ensures that the generated language files are in their proper encodings (see below).

=== All other language files ===

Note that the language files (f.i. '''strings/german.lang''') as well as the readables and the FM dictionariaries are expected to be in the following encodings:

* '''Czech''', '''Hungarian''', '''Slovak''', '''Polish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-2 ISO-8859-2] ('''not WIN-1250!)
* '''Russian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/Win-1251 WIN-1251]
* '''French:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-15 ISO-8859-15]
* '''Romanian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-16 ISO-8859-16]
* '''All other languages:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-1 ISO-8859-1] (German, Dutch, Danish, Swedish, Portuguese, etc.)

{{infobox|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}

== Character remapping ==

The characters are remapped upon loading the dictionary/readable, from their native encoding to the special one that TDM uses and that is described here. Responsible for the remapping are [[I18N - Character mapping|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found, "strings/default.map" is used instead, if this is not found, no remapping takes place.

See '''[[I18N - Character mapping|Character mapping]]''' for more information.

=== European Languages ===

This mapping is used for European languages, f.i. '''Czech''', '''French''', '''German''', '''Spanish''', '''Portuguese''', '''Polish'''. Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!

In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.

'''Color code:'''

{{box|#f0d0d0|Character not usable by TDM|Unusable}}{{box|#d0e0d0|Character not yet used in TDM|Unused}}{{box|#c0ffc0|Character displayed in v1.08 or newer|Usable in v1.08}}{{box|#80f080|Character displayed in v2.03 or newer|Usable in v2.03}}{{box|#d0d0f0|Changed from the ISO-8859-1 default, usable by TDM 1.0 or newer|Changed from ISO 8859-1}}

{|class="wikitable" border=1 style="border-collapse: collapse; font-size: 95%" cellspacing=0 cellpadding=2 width=100%

|-
!
!…0
!…1
!…2
!…3
!…4
!…5
!…6
!…7
!…8
!…9
!…A
!…B
!…C
!…D
!…E
!…F

|-
!0…
|align='center' style='background: #f0d0d0'|00 '''–'''
|align='center' style='background: #f0d0d0'|01 '''–'''
|align='center' style='background: #f0d0d0'|02 '''–'''
|align='center' style='background: #f0d0d0'|03 '''–'''
|align='center' style='background: #f0d0d0'|04 '''–'''
|align='center' style='background: #f0d0d0'|05 '''–'''
|align='center' style='background: #f0d0d0'|06 '''–'''
|align='center' style='background: #f0d0d0'|07 '''–'''
|align='center' style='background: #f0d0d0'|08 '''–'''
|align='center' style='background: #f0d0d0'|09 '''–'''
|align='center' style='background: #f0d0d0'|0A '''–'''
|align='center' style='background: #f0d0d0'|0B '''–'''
|align='center' style='background: #f0d0d0'|0C '''–'''
|align='center' style='background: #f0d0d0'|0D '''–'''
|align='center' style='background: #f0d0d0'|0E '''–'''
|align='center' style='background: #f0d0d0'|0F '''–'''

|-
!1…
|align='center' style='background: #f0d0d0'|10 '''–'''
|align='center' style='background: #f0d0d0'|11 '''–'''
|align='center' style='background: #f0d0d0'|12 '''–'''
|align='center' style='background: #f0d0d0'|13 '''–'''
|align='center' style='background: #f0d0d0'|14 '''–'''
|align='center' style='background: #f0d0d0'|15 '''–'''
|align='center' style='background: #f0d0d0'|16 '''–'''
|align='center' style='background: #f0d0d0'|17 '''–'''
|align='center' style='background: #f0d0d0'|18 '''–'''
|align='center' style='background: #f0d0d0'|19 '''–'''
|align='center' style='background: #f0d0d0'|1A '''–'''
|align='center' style='background: #f0d0d0'|1B '''–'''
|align='center' style='background: #f0d0d0'|1C '''–'''
|align='center' style='background: #f0d0d0'|1D '''–'''
|align='center' style='background: #f0d0d0'|1E '''–'''
|align='center' style='background: #f0d0d0'|1F '''–'''

|-
!2…
|align='center'|20 ''' '''
|align='center'|21 '''!'''
|align='center'|22 '''"'''
|align='center'|23 '''#'''
|align='center'|24 '''$'''
|align='center'|25 '''%'''
|align='center'|26 '''&'''
|align='center'|27 '''''''
|align='center'|28 '''('''
|align='center'|29 ''')'''
|align='center'|2A '''*'''
|align='center'|2B '''+'''
|align='center'|2C ''','''
|align='center'|2D '''-'''
|align='center'|2E '''.'''
|align='center'|2F '''/'''

|-
!3…
|align='center'|30 '''0'''
|align='center'|31 '''1'''
|align='center'|32 '''2'''
|align='center'|33 '''3'''
|align='center'|34 '''4'''
|align='center'|35 '''5'''
|align='center'|36 '''6'''
|align='center'|37 '''7'''
|align='center'|38 '''8'''
|align='center'|39 '''9'''
|align='center'|3A ''':'''
|align='center'|3B ''';'''
|align='center'|3C '''<'''
|align='center'|3D '''='''
|align='center'|3E '''>'''
|align='center'|3F '''?'''

|-
!4…
|align='center'|40 '''@'''
|align='center'|41 '''A'''
|align='center'|42 '''B'''
|align='center'|43 '''C'''
|align='center'|44 '''D'''
|align='center'|45 '''E'''
|align='center'|46 '''F'''
|align='center'|47 '''G'''
|align='center'|48 '''H'''
|align='center'|49 '''I'''
|align='center'|4A '''J'''
|align='center'|4B '''K'''
|align='center'|4C '''L'''
|align='center'|4D '''M'''
|align='center'|4E '''N'''
|align='center'|4F '''O'''

|-
!5…
|align='center'|50 '''P'''
|align='center'|51 '''Q'''
|align='center'|52 '''R'''
|align='center'|53 '''S'''
|align='center'|54 '''T'''
|align='center'|55 '''U'''
|align='center'|56 '''V'''
|align='center'|57 '''W'''
|align='center'|58 '''X'''
|align='center'|59 '''Y'''
|align='center'|5A '''Z'''
|align='center'|5B '''['''
|align='center'|5C '''\'''
|align='center'|5D ''']'''
|align='center'|5E '''^'''
|align='center'|5F '''_'''

|-
!6…
|align='center'|60 '''`'''
|align='center'|61 '''a'''
|align='center'|62 '''b'''
|align='center'|63 '''c'''
|align='center'|64 '''d'''
|align='center'|65 '''e'''
|align='center'|66 '''f'''
|align='center'|67 '''g'''
|align='center'|68 '''h'''
|align='center'|69 '''i'''
|align='center'|6A '''j'''
|align='center'|6B '''k'''
|align='center'|6C '''l'''
|align='center'|6D '''m'''
|align='center'|6E '''n'''
|align='center'|6F '''o'''

|-
!7…
|align='center'|70 '''p'''
|align='center'|71 '''q'''
|align='center'|72 '''r'''
|align='center'|73 '''s'''
|align='center'|74 '''t'''
|align='center'|75 '''u'''
|align='center'|76 '''v'''
|align='center'|77 '''w'''
|align='center'|78 '''x'''
|align='center'|79 '''y'''
|align='center'|7A '''z'''
|align='center'|7B '''{'''
|align='center'|7C '''|'''
|align='center'|7D '''}'''
|align='center'|7E '''~'''
|align='center' style='background: #d0e0d0'|7F '''�'''

|-
!8…
|align='center' style='background: #c0ffc0'|80 '''Ň'''
|align='center' style='background: #c0ffc0'|81 '''Ś'''
|align='center' style='background: #c0ffc0'|82 '''Ć'''
|align='center' style='background: #c0ffc0'|83 '''Ż'''
|align='center' style='background: #c0ffc0'|84 '''Ź'''
|align='center' style='background: #c0ffc0'|85 '''Ŝ'''
|align='center' style='background: #c0ffc0'|86 '''Ĉ'''
|align='center' style='background: #c0ffc0'|87 '''Ẑ'''
|align='center' style='background: #c0ffc0'|88 '''Ô''' [1]
|align='center' style='background: #c0ffc0'|89 '''Ŕ'''
|align='center' style='background: #c0ffc0'|8A '''Ǔ'''
|align='center' style='background: #c0ffc0'|8B '''Ă'''
|align='center' style='background: #c0ffc0'|8C '''Ń'''
|align='center' style='background: #80f080'|8D '''Ș'''
|align='center' style='background: #80f080'|8E '''Ț'''
|align='center' style='background: #d0e0d0'|8F '''�'''

|-
!9…
|align='center' style='background: #80f080'|90 '''đ'''
|align='center' style='background: #c0ffc0'|91 '''ś'''
|align='center' style='background: #c0ffc0'|92 '''ć'''
|align='center' style='background: #c0ffc0'|93 '''ż'''
|align='center' style='background: #c0ffc0'|94 '''ź'''
|align='center' style='background: #c0ffc0'|95 '''ŝ'''
|align='center' style='background: #c0ffc0'|96 '''ĉ'''
|align='center' style='background: #c0ffc0'|97 '''ẑ'''
|align='center' style='background: #c0ffc0'|98 '''ô''' [1]
|align='center' style='background: #c0ffc0'|99 '''ŕ'''
|align='center' style='background: #c0ffc0'|9A '''ǔ'''
|align='center' style='background: #c0ffc0'|9B '''ă'''
|align='center' style='background: #c0ffc0'|9C '''ń'''
|align='center' style='background: #80f080'|9D '''ș'''
|align='center' style='background: #80f080'|9E '''ț'''
|align='center' style='background: #d0e0d0'|9F '''�'''

|-
!A…
|align='center'|A0 '''[https://secure.wikimedia.org/wikipedia/en/wiki/Non-breaking_space NBSP]''' [2]
|align='center' style='background: #d0d0f0'|A1 '''ň''' (¡)'''
|align='center' style='background: #d0d0f0'|A2 '''Ű''' (¢)'''
|align='center' style='background: #d0d0f0'|A3 '''ě''' (£)'''
|align='center' style='background: #d0d0f0'|A4 '''ű''' (¤)'''
|align='center' style='background: #d0d0f0'|A5 '''Ě''' (¥)'''
|align='center' style='background: #d0d0f0'|A6 '''Š''' (¦)'''
|align='center'|A7 '''§'''
|align='center' style='background: #d0d0f0'|A8 '''š''' (¨)'''
|align='center' style='background: #d0d0f0'|A9 '''Ů''' (©)'''
|align='center' style='background: #d0d0f0'|AA '''Ą''' (ª)'''
|align='center' style='background: #d0d0f0'|AB '''Ę''' («)'''
|align='center' style='background: #d0d0f0'|AC '''Č''' (¬)'''
|align='center'|AD '''[https://secure.wikimedia.org/wikipedia/en/wiki/Soft_hyphen SHY]''' [2]
|align='center' style='background: #d0d0f0'|AE '''č''' (®)'''
|align='center' style='background: #d0d0f0'|AF '''ů''' (¯)'''

|-
!B…
|align='center' style='background: #d0d0f0'|B0 '''Ő''' (°)'''
|align='center' style='background: #d0d0f0'|B1 '''Ł''' (±)'''
|align='center' style='background: #d0d0f0'|B2 '''Ť''' (²)'''
|align='center' style='background: #d0d0f0'|B3 '''Ď''' (³)'''
|align='center' style='background: #d0d0f0'|B4 '''Ž''' (´)'''
|align='center' style='background: #d0d0f0'|B5 '''ł''' (µ)'''
|align='center' style='background: #d0d0f0'|B6 '''ť''' (¶)'''
|align='center' style='background: #d0d0f0'|B7 '''ď''' (·)'''
|align='center' style='background: #d0d0f0'|B8 '''ž''' (¸)'''
|align='center' style='background: #d0d0f0'|B9 '''ő''' (¹)'''
|align='center' style='background: #d0d0f0'|BA '''ą''' (º)'''
|align='center' style='background: #d0d0f0'|BB '''ę''' (»)'''
|align='center' style='background: #d0d0f0'|BC '''Œ''' (¼)'''
|align='center' style='background: #d0d0f0'|BD '''œ''' (½)'''
|align='center' style='background: #d0d0f0'|BE '''Ÿ''' (¾)'''
|align='center'|BF '''¿'''

|-
!C…
|align='center'|C0 '''À'''
|align='center'|C1 '''Á'''
|align='center'|C2 '''Â'''
|align='center'|C3 '''Ã'''
|align='center'|C4 '''Ä'''
|align='center'|C5 '''Å'''
|align='center'|C6 '''Æ'''
|align='center'|C7 '''Ç'''
|align='center'|C8 '''È'''
|align='center'|C9 '''É'''
|align='center'|CA '''Ê'''
|align='center'|CB '''Ë'''
|align='center'|CC '''Ì'''
|align='center'|CD '''Í'''
|align='center'|CE '''Î'''
|align='center'|CF '''Ï'''

|-
!D…
|align='center'|D0 '''Ð'''
|align='center'|D1 '''Ñ'''
|align='center'|D2 '''Ò'''
|align='center'|D3 '''Ó'''
|align='center'|D4 '''Ô'''
|align='center'|D5 '''Õ'''
|align='center'|D6 '''Ö'''
|align='center' style='background: #d0d0f0'|D7 '''Ř''' (×)'''
|align='center'|D8 '''Ø'''
|align='center'|D9 '''Ù'''
|align='center'|DA '''Ú'''
|align='center'|DB '''Û'''
|align='center'|DC '''Ü'''
|align='center'|DD '''Ý'''
|align='center'|DE '''Þ'''
|align='center'|DF '''ß'''

|-
!E…
|align='center'|E0 '''à'''
|align='center'|E1 '''á'''
|align='center'|E2 '''â'''
|align='center'|E3 '''ã'''
|align='center'|E4 '''ä'''
|align='center'|E5 '''å'''
|align='center'|E6 '''æ'''
|align='center'|E7 '''ç'''
|align='center'|E8 '''è'''
|align='center'|E9 '''é'''
|align='center'|EA '''ê'''
|align='center'|EB '''ë'''
|align='center'|EC '''ì'''
|align='center'|ED '''í'''
|align='center'|EE '''î'''
|align='center'|EF '''ï'''

|-
!F…
|align='center'|F0 '''ð'''
|align='center'|F1 '''ñ'''
|align='center'|F2 '''ò'''
|align='center'|F3 '''ó'''
|align='center'|F4 '''ô'''
|align='center'|F5 '''õ'''
|align='center'|F6 '''ö'''
|align='center' style='background: #d0d0f0'|F7 '''ř''' (÷)'''
|align='center'|F8 '''ø'''
|align='center'|F9 '''ù'''
|align='center'|FA '''ú'''
|align='center'|FB '''û'''
|align='center'|FC '''ü'''
|align='center'|FD '''ý'''
|align='center'|FE '''þ'''
|align='center'|FF '''ÿ'''

|}

==== Table Notes ====

'''[1]''' As discussed [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/&do=findComment&comment=494855 here], the TDM char set has a redundant treatment of 2 characters:

* Ô appears at 0x88 and 0xD4
* ô appears at 0x98 and 0xF4

Starting with TDM 2.13, the redundancy can be removed and these new characters (from ISO-8859-3) introduced:

* Ğ appears at 0x88
* ğ appears at 0x98

'''[2]''' Avoid using the non-breaking space (NBSP, 0xA0) and the soft hyphen (SHY, 0xAD) in your strings. The TDM engine has no code to respect these during word wrap. Font maintainers: probably map NBSP to the <space> glyph, SHY to undefined/hollow box or zero-sized box.

==== Corresponding Unicode ====

For a mapping of these 256 codepoints to Unicode U+NNNN values and formal names, download 'TDM 8859-Style Font Map to Unicode-16.txt':

* [https://drive.google.com/file/d/1wLEtuFvnrZ-WHK8ign8i3diIXZdriZ4F/view?usp=sharing for TDM 2.13]
* [https://drive.google.com/file/d/1UAz9jSZpT_j33STP3So_Re8JWa8QuAmz/view?usp=sharing for TDM 2.12 and earlier]

Each of these files (by Geep, 2024) is in a standardized format so that it can also be imported into font design programs like FontForge as a custom 256-position map. In the comments, there is additional information about:
* ISO 8859-x sourcing of each character.
* alternative representations of some European and control characters.

=== Russian ===

Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6.

Within any given font, character overage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.

=== Asian Languages (Korean, Chinese, Japanese) ===

The original D3 had support for these languages, so it might be possible to add them to TDM, too. At the moment, however, we lack the fonts and translators. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

== European Character Implementation ==
=== Priority Early-On - the "Top 50" ===

Some of the special characters are used more often than others. Here is a statistic over the entire string set of the TDM core, from TDM v1.08, showing the top 50 most-used characters (excluding a-z, 0-9 and russian characters):

{|class="wikitable" border=1 style="border-collapse: collapse; font-size: 85%" cellspacing=0 cellpadding=2

|-
|Rank
|Occurances
|Letter
|Remarks
|Rank
|Occurances
|Letter
|Remarks

|-
|1
|í
|715
|
|25
|ć
|67
|

|-
|2
|é
|674
|
|26
|è
|65
|

|-
|3
|á
|524
|
|27
|ú
|56
|

|-
|4
|ø
|303
|Danish
|28
|ê
|52
|

|-
|5
|č
|288
|
|29
|ö
|48
|German

|-
|6
|ó
|283
|
|30
|É
|46
|

|-
|7
|ü
|270
|German
|31
|ñ
|37
|

|-
|8
|ł
|203
|Polish
|32
|õ
|32
|

|-
|9
|æ
|200
|Danish
|33
|ń
|26
|

|-
|10
|ě
|182
|
|34
|Ł
|24
|

|-
|11
|ř
|175
|Czech
|35
|Š
|21
|

|-
|12
|ã
|168
|
|36
|â
|21
|

|-
|13
|ž
|148
|Czech
|37
|ź
|20
|

|-
|14
|ý
|142
|
|38
|ß
|18
|German

|-
|15
|ę
|141
|
|39
|Ó
|18
|

|-
|16
|ą
|140
|
|40
|ň
|15
|

|-
|17
|ż
|119
|
|41
|Ú
|15
|

|-
|18
|å
|109
|Danish
|42
|Á
|13
|

|-
|19
|š
|99
|
|43
|î
|12
|

|-
|20
|ś
|97
|
|44
|ť
|11
|

|-
|21
|ç
|91
|
|45
|ô
|9
|

|-
|22
|ä
|86
|German
|46
|Ž
|8
|

|-
|23
|à
|83
|
|47
|Ż
|7
|

|-
|24
|ů
|77
|
|48
|Č
|7
|

|-
|25
|ć
|67
|
|49
|ù
|6
|

|}

Although ö, ä and ü do not appear that often, with only these and Ü, Ö, Ä and ß, the entire German language works. So adding these letters to the fonts is quite important.

Preferably, all foreign letters would be added to the fonts (see [[Font Patcher]] or [[Refont]]). However, if time permits only adding a few, '''í''' would be more important than, say, '''ô'''.

It is commonplace for missing accented letters to be redirected in the .dat file to the corresponding unaccented base letter.

=== Coverage Expansion & Remaining Limitations ===

By 2014, all the system (e.g., main menu) fonts (Carleton, Carleton_condensed, Stone in sizes 24pt and 48pt; Mason and Mason_glow in 48pt) had good coverage of the "Top 50" and beyond, although not all 256 codepoints.
This was confirmed more specifically in 2024, as part of an [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ Analysis of 2.12 Fonts]. This analysis indicated that, unlike the system fonts, the FM fonts generally did not provide specific glyphs beyond ASCII.

In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14. Similar Stone 48pt and Carleton 24pt improvements are planned for TDM 2.15.

[[Category:Fonts]]

{{i18n}}

I18N - Charset

2026-03-15T20:58:07Z

Geep: Added note 2 about avoiding non-breaking space & soft hyphen

== Introduction ==

The D3 code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, special charsets are used. The fonts (Carleton for the menu f.i.) are build/patched so that the right characters appear in the right place.

== Encodings ==

=== all.lang ===

This file is in '''UTF-8''', and converted with the help of the script '''devel/gen_lang.pl''':

perl devel/gen_lang.pl

This ensures that the generated language files are in their proper encodings (see below).

=== All other language files ===

Note that the language files (f.i. '''strings/german.lang''') as well as the readables and the FM dictionariaries are expected to be in the following encodings:

* '''Czech''', '''Hungarian''', '''Slovak''', '''Polish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-2 ISO-8859-2] ('''not WIN-1250!)
* '''Russian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/Win-1251 WIN-1251]
* '''French:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-15 ISO-8859-15]
* '''Romanian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-16 ISO-8859-16]
* '''All other languages:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-1 ISO-8859-1] (German, Dutch, Danish, Swedish, Portuguese, etc.)

{{infobox|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}

== Character remapping ==

The characters are remapped upon loading the dictionary/readable, from their native encoding to the special one that TDM uses and that is described here. Responsible for the remapping are [[I18N - Character mapping|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found, "strings/default.map" is used instead, if this is not found, no remapping takes place.

See '''[[I18N - Character mapping|Character mapping]]''' for more information.

=== European Languages ===

This mapping is used for European languages, f.i. '''Czech''', '''French''', '''German''', '''Spanish''', '''Portuguese''', '''Polish'''. Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!

In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.

'''Color code:'''

{{box|#f0d0d0|Character not usable by TDM|Unusable}}{{box|#d0e0d0|Character not yet used in TDM|Unused}}{{box|#c0ffc0|Character displayed in v1.08 or newer|Usable in v1.08}}{{box|#80f080|Character displayed in v2.03 or newer|Usable in v2.03}}{{box|#d0d0f0|Changed from the ISO-8859-1 default, usable by TDM 1.0 or newer|Changed from ISO 8859-1}}

{|class="wikitable" border=1 style="border-collapse: collapse; font-size: 95%" cellspacing=0 cellpadding=2 width=100%

|-
!
!…0
!…1
!…2
!…3
!…4
!…5
!…6
!…7
!…8
!…9
!…A
!…B
!…C
!…D
!…E
!…F

|-
!0…
|align='center' style='background: #f0d0d0'|00 '''–'''
|align='center' style='background: #f0d0d0'|01 '''–'''
|align='center' style='background: #f0d0d0'|02 '''–'''
|align='center' style='background: #f0d0d0'|03 '''–'''
|align='center' style='background: #f0d0d0'|04 '''–'''
|align='center' style='background: #f0d0d0'|05 '''–'''
|align='center' style='background: #f0d0d0'|06 '''–'''
|align='center' style='background: #f0d0d0'|07 '''–'''
|align='center' style='background: #f0d0d0'|08 '''–'''
|align='center' style='background: #f0d0d0'|09 '''–'''
|align='center' style='background: #f0d0d0'|0A '''–'''
|align='center' style='background: #f0d0d0'|0B '''–'''
|align='center' style='background: #f0d0d0'|0C '''–'''
|align='center' style='background: #f0d0d0'|0D '''–'''
|align='center' style='background: #f0d0d0'|0E '''–'''
|align='center' style='background: #f0d0d0'|0F '''–'''

|-
!1…
|align='center' style='background: #f0d0d0'|10 '''–'''
|align='center' style='background: #f0d0d0'|11 '''–'''
|align='center' style='background: #f0d0d0'|12 '''–'''
|align='center' style='background: #f0d0d0'|13 '''–'''
|align='center' style='background: #f0d0d0'|14 '''–'''
|align='center' style='background: #f0d0d0'|15 '''–'''
|align='center' style='background: #f0d0d0'|16 '''–'''
|align='center' style='background: #f0d0d0'|17 '''–'''
|align='center' style='background: #f0d0d0'|18 '''–'''
|align='center' style='background: #f0d0d0'|19 '''–'''
|align='center' style='background: #f0d0d0'|1A '''–'''
|align='center' style='background: #f0d0d0'|1B '''–'''
|align='center' style='background: #f0d0d0'|1C '''–'''
|align='center' style='background: #f0d0d0'|1D '''–'''
|align='center' style='background: #f0d0d0'|1E '''–'''
|align='center' style='background: #f0d0d0'|1F '''–'''

|-
!2…
|align='center'|20 ''' '''
|align='center'|21 '''!'''
|align='center'|22 '''"'''
|align='center'|23 '''#'''
|align='center'|24 '''$'''
|align='center'|25 '''%'''
|align='center'|26 '''&'''
|align='center'|27 '''''''
|align='center'|28 '''('''
|align='center'|29 ''')'''
|align='center'|2A '''*'''
|align='center'|2B '''+'''
|align='center'|2C ''','''
|align='center'|2D '''-'''
|align='center'|2E '''.'''
|align='center'|2F '''/'''

|-
!3…
|align='center'|30 '''0'''
|align='center'|31 '''1'''
|align='center'|32 '''2'''
|align='center'|33 '''3'''
|align='center'|34 '''4'''
|align='center'|35 '''5'''
|align='center'|36 '''6'''
|align='center'|37 '''7'''
|align='center'|38 '''8'''
|align='center'|39 '''9'''
|align='center'|3A ''':'''
|align='center'|3B ''';'''
|align='center'|3C '''<'''
|align='center'|3D '''='''
|align='center'|3E '''>'''
|align='center'|3F '''?'''

|-
!4…
|align='center'|40 '''@'''
|align='center'|41 '''A'''
|align='center'|42 '''B'''
|align='center'|43 '''C'''
|align='center'|44 '''D'''
|align='center'|45 '''E'''
|align='center'|46 '''F'''
|align='center'|47 '''G'''
|align='center'|48 '''H'''
|align='center'|49 '''I'''
|align='center'|4A '''J'''
|align='center'|4B '''K'''
|align='center'|4C '''L'''
|align='center'|4D '''M'''
|align='center'|4E '''N'''
|align='center'|4F '''O'''

|-
!5…
|align='center'|50 '''P'''
|align='center'|51 '''Q'''
|align='center'|52 '''R'''
|align='center'|53 '''S'''
|align='center'|54 '''T'''
|align='center'|55 '''U'''
|align='center'|56 '''V'''
|align='center'|57 '''W'''
|align='center'|58 '''X'''
|align='center'|59 '''Y'''
|align='center'|5A '''Z'''
|align='center'|5B '''['''
|align='center'|5C '''\'''
|align='center'|5D ''']'''
|align='center'|5E '''^'''
|align='center'|5F '''_'''

|-
!6…
|align='center'|60 '''`'''
|align='center'|61 '''a'''
|align='center'|62 '''b'''
|align='center'|63 '''c'''
|align='center'|64 '''d'''
|align='center'|65 '''e'''
|align='center'|66 '''f'''
|align='center'|67 '''g'''
|align='center'|68 '''h'''
|align='center'|69 '''i'''
|align='center'|6A '''j'''
|align='center'|6B '''k'''
|align='center'|6C '''l'''
|align='center'|6D '''m'''
|align='center'|6E '''n'''
|align='center'|6F '''o'''

|-
!7…
|align='center'|70 '''p'''
|align='center'|71 '''q'''
|align='center'|72 '''r'''
|align='center'|73 '''s'''
|align='center'|74 '''t'''
|align='center'|75 '''u'''
|align='center'|76 '''v'''
|align='center'|77 '''w'''
|align='center'|78 '''x'''
|align='center'|79 '''y'''
|align='center'|7A '''z'''
|align='center'|7B '''{'''
|align='center'|7C '''|'''
|align='center'|7D '''}'''
|align='center'|7E '''~'''
|align='center' style='background: #d0e0d0'|7F '''�'''

|-
!8…
|align='center' style='background: #c0ffc0'|80 '''Ň'''
|align='center' style='background: #c0ffc0'|81 '''Ś'''
|align='center' style='background: #c0ffc0'|82 '''Ć'''
|align='center' style='background: #c0ffc0'|83 '''Ż'''
|align='center' style='background: #c0ffc0'|84 '''Ź'''
|align='center' style='background: #c0ffc0'|85 '''Ŝ'''
|align='center' style='background: #c0ffc0'|86 '''Ĉ'''
|align='center' style='background: #c0ffc0'|87 '''Ẑ'''
|align='center' style='background: #c0ffc0'|88 '''Ô''' [1]
|align='center' style='background: #c0ffc0'|89 '''Ŕ'''
|align='center' style='background: #c0ffc0'|8A '''Ǔ'''
|align='center' style='background: #c0ffc0'|8B '''Ă'''
|align='center' style='background: #c0ffc0'|8C '''Ń'''
|align='center' style='background: #80f080'|8D '''Ș'''
|align='center' style='background: #80f080'|8E '''Ț'''
|align='center' style='background: #d0e0d0'|8F '''�'''

|-
!9…
|align='center' style='background: #80f080'|90 '''đ'''
|align='center' style='background: #c0ffc0'|91 '''ś'''
|align='center' style='background: #c0ffc0'|92 '''ć'''
|align='center' style='background: #c0ffc0'|93 '''ż'''
|align='center' style='background: #c0ffc0'|94 '''ź'''
|align='center' style='background: #c0ffc0'|95 '''ŝ'''
|align='center' style='background: #c0ffc0'|96 '''ĉ'''
|align='center' style='background: #c0ffc0'|97 '''ẑ'''
|align='center' style='background: #c0ffc0'|98 '''ô''' [1]
|align='center' style='background: #c0ffc0'|99 '''ŕ'''
|align='center' style='background: #c0ffc0'|9A '''ǔ'''
|align='center' style='background: #c0ffc0'|9B '''ă'''
|align='center' style='background: #c0ffc0'|9C '''ń'''
|align='center' style='background: #80f080'|9D '''ș'''
|align='center' style='background: #80f080'|9E '''ț'''
|align='center' style='background: #d0e0d0'|9F '''�'''

|-
!A…
|align='center'|A0 '''[https://secure.wikimedia.org/wikipedia/en/wiki/Non-breaking_space NBSP]''' [2]
|align='center' style='background: #d0d0f0'|A1 '''ň''' (¡)'''
|align='center' style='background: #d0d0f0'|A2 '''Ű''' (¢)'''
|align='center' style='background: #d0d0f0'|A3 '''ě''' (£)'''
|align='center' style='background: #d0d0f0'|A4 '''ű''' (¤)'''
|align='center' style='background: #d0d0f0'|A5 '''Ě''' (¥)'''
|align='center' style='background: #d0d0f0'|A6 '''Š''' (¦)'''
|align='center'|A7 '''§'''
|align='center' style='background: #d0d0f0'|A8 '''š''' (¨)'''
|align='center' style='background: #d0d0f0'|A9 '''Ů''' (©)'''
|align='center' style='background: #d0d0f0'|AA '''Ą''' (ª)'''
|align='center' style='background: #d0d0f0'|AB '''Ę''' («)'''
|align='center' style='background: #d0d0f0'|AC '''Č''' (¬)'''
|align='center'|AD '''[https://secure.wikimedia.org/wikipedia/en/wiki/Soft_hyphen SHY]''' [2]
|align='center' style='background: #d0d0f0'|AE '''č''' (®)'''
|align='center' style='background: #d0d0f0'|AF '''ů''' (¯)'''

|-
!B…
|align='center' style='background: #d0d0f0'|B0 '''Ő''' (°)'''
|align='center' style='background: #d0d0f0'|B1 '''Ł''' (±)'''
|align='center' style='background: #d0d0f0'|B2 '''Ť''' (²)'''
|align='center' style='background: #d0d0f0'|B3 '''Ď''' (³)'''
|align='center' style='background: #d0d0f0'|B4 '''Ž''' (´)'''
|align='center' style='background: #d0d0f0'|B5 '''ł''' (µ)'''
|align='center' style='background: #d0d0f0'|B6 '''ť''' (¶)'''
|align='center' style='background: #d0d0f0'|B7 '''ď''' (·)'''
|align='center' style='background: #d0d0f0'|B8 '''ž''' (¸)'''
|align='center' style='background: #d0d0f0'|B9 '''ő''' (¹)'''
|align='center' style='background: #d0d0f0'|BA '''ą''' (º)'''
|align='center' style='background: #d0d0f0'|BB '''ę''' (»)'''
|align='center' style='background: #d0d0f0'|BC '''Œ''' (¼)'''
|align='center' style='background: #d0d0f0'|BD '''œ''' (½)'''
|align='center' style='background: #d0d0f0'|BE '''Ÿ''' (¾)'''
|align='center'|BF '''¿'''

|-
!C…
|align='center'|C0 '''À'''
|align='center'|C1 '''Á'''
|align='center'|C2 '''Â'''
|align='center'|C3 '''Ã'''
|align='center'|C4 '''Ä'''
|align='center'|C5 '''Å'''
|align='center'|C6 '''Æ'''
|align='center'|C7 '''Ç'''
|align='center'|C8 '''È'''
|align='center'|C9 '''É'''
|align='center'|CA '''Ê'''
|align='center'|CB '''Ë'''
|align='center'|CC '''Ì'''
|align='center'|CD '''Í'''
|align='center'|CE '''Î'''
|align='center'|CF '''Ï'''

|-
!D…
|align='center'|D0 '''Ð'''
|align='center'|D1 '''Ñ'''
|align='center'|D2 '''Ò'''
|align='center'|D3 '''Ó'''
|align='center'|D4 '''Ô'''
|align='center'|D5 '''Õ'''
|align='center'|D6 '''Ö'''
|align='center' style='background: #d0d0f0'|D7 '''Ř''' (×)'''
|align='center'|D8 '''Ø'''
|align='center'|D9 '''Ù'''
|align='center'|DA '''Ú'''
|align='center'|DB '''Û'''
|align='center'|DC '''Ü'''
|align='center'|DD '''Ý'''
|align='center'|DE '''Þ'''
|align='center'|DF '''ß'''

|-
!E…
|align='center'|E0 '''à'''
|align='center'|E1 '''á'''
|align='center'|E2 '''â'''
|align='center'|E3 '''ã'''
|align='center'|E4 '''ä'''
|align='center'|E5 '''å'''
|align='center'|E6 '''æ'''
|align='center'|E7 '''ç'''
|align='center'|E8 '''è'''
|align='center'|E9 '''é'''
|align='center'|EA '''ê'''
|align='center'|EB '''ë'''
|align='center'|EC '''ì'''
|align='center'|ED '''í'''
|align='center'|EE '''î'''
|align='center'|EF '''ï'''

|-
!F…
|align='center'|F0 '''ð'''
|align='center'|F1 '''ñ'''
|align='center'|F2 '''ò'''
|align='center'|F3 '''ó'''
|align='center'|F4 '''ô'''
|align='center'|F5 '''õ'''
|align='center'|F6 '''ö'''
|align='center' style='background: #d0d0f0'|F7 '''ř''' (÷)'''
|align='center'|F8 '''ø'''
|align='center'|F9 '''ù'''
|align='center'|FA '''ú'''
|align='center'|FB '''û'''
|align='center'|FC '''ü'''
|align='center'|FD '''ý'''
|align='center'|FE '''þ'''
|align='center'|FF '''ÿ'''

|}

==== Notes ====

'''[1]''' As discussed [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/&do=findComment&comment=494855 here], the TDM char set has a redundant treatment of 2 characters:

* Ô appears at 0x88 and 0xD4
* ô appears at 0x98 and 0xF4

Starting with TDM 2.13, the redundancy can be removed and these new characters (from ISO-8859-3) introduced:

* Ğ appears at 0x88
* ğ appears at 0x98

'''[2]''' Avoid using the non-breaking space (NBSP, 0xA0) and the soft hyphen (SHY, 0xAD) in your strings. The TDM engine has no code to respect these during word wrap. Font maintainers: probably map NBSP to the <space> glyph, SHY to undefined/hollow box or zero-sized box.

==== Corresponding Unicode ====

For a mapping of these 256 codepoints to Unicode U+NNNN values and formal names, download 'TDM 8859-Style Font Map to Unicode-16.txt':

* [https://drive.google.com/file/d/1wLEtuFvnrZ-WHK8ign8i3diIXZdriZ4F/view?usp=sharing for TDM 2.13]
* [https://drive.google.com/file/d/1UAz9jSZpT_j33STP3So_Re8JWa8QuAmz/view?usp=sharing for TDM 2.12 and earlier]

Each of these files (by Geep, 2024) is in a standardized format so that it can also be imported into font design programs like FontForge as a custom 256-position map. In the comments, there is additional information about:
* ISO 8859-x sourcing of each character.
* alternative representations of some European and control characters.

=== Russian ===

Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6.

Within any given font, character overage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.

=== Asian Languages (Korean, Chinese, Japanese) ===

The original D3 had support for these languages, so it might be possible to add them to TDM, too. At the moment, however, we lack the fonts and translators. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

== European Character Implementation ==
=== Priority Early-On - the "Top 50" ===

Some of the special characters are used more often than others. Here is a statistic over the entire string set of the TDM core, from TDM v1.08, showing the top 50 most-used characters (excluding a-z, 0-9 and russian characters):

{|class="wikitable" border=1 style="border-collapse: collapse; font-size: 85%" cellspacing=0 cellpadding=2

|-
|Rank
|Occurances
|Letter
|Remarks
|Rank
|Occurances
|Letter
|Remarks

|-
|1
|í
|715
|
|25
|ć
|67
|

|-
|2
|é
|674
|
|26
|è
|65
|

|-
|3
|á
|524
|
|27
|ú
|56
|

|-
|4
|ø
|303
|Danish
|28
|ê
|52
|

|-
|5
|č
|288
|
|29
|ö
|48
|German

|-
|6
|ó
|283
|
|30
|É
|46
|

|-
|7
|ü
|270
|German
|31
|ñ
|37
|

|-
|8
|ł
|203
|Polish
|32
|õ
|32
|

|-
|9
|æ
|200
|Danish
|33
|ń
|26
|

|-
|10
|ě
|182
|
|34
|Ł
|24
|

|-
|11
|ř
|175
|Czech
|35
|Š
|21
|

|-
|12
|ã
|168
|
|36
|â
|21
|

|-
|13
|ž
|148
|Czech
|37
|ź
|20
|

|-
|14
|ý
|142
|
|38
|ß
|18
|German

|-
|15
|ę
|141
|
|39
|Ó
|18
|

|-
|16
|ą
|140
|
|40
|ň
|15
|

|-
|17
|ż
|119
|
|41
|Ú
|15
|

|-
|18
|å
|109
|Danish
|42
|Á
|13
|

|-
|19
|š
|99
|
|43
|î
|12
|

|-
|20
|ś
|97
|
|44
|ť
|11
|

|-
|21
|ç
|91
|
|45
|ô
|9
|

|-
|22
|ä
|86
|German
|46
|Ž
|8
|

|-
|23
|à
|83
|
|47
|Ż
|7
|

|-
|24
|ů
|77
|
|48
|Č
|7
|

|-
|25
|ć
|67
|
|49
|ù
|6
|

|}

Although ö, ä and ü do not appear that often, with only these and Ü, Ö, Ä and ß, the entire German language works. So adding these letters to the fonts is quite important.

Preferably, all foreign letters would be added to the fonts (see [[Font Patcher]] or [[Refont]]). However, if time permits only adding a few, '''í''' would be more important than, say, '''ô'''.

It is commonplace for missing accented letters to be redirected in the .dat file to the corresponding unaccented base letter.

=== Coverage Expansion & Remaining Limitations ===

By 2014, all the system (e.g., main menu) fonts (Carleton, Carleton_condensed, Stone in sizes 24pt and 48pt; Mason and Mason_glow in 48pt) had good coverage of the "Top 50" and beyond, although not all 256 codepoints.
This was confirmed more specifically in 2024, as part of an [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ Analysis of 2.12 Fonts]. This analysis indicated that, unlike the system fonts, the FM fonts generally did not provide specific glyphs beyond ASCII.

In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14. Similar Stone 48pt and Carleton 24pt improvements are planned for TDM 2.15.

[[Category:Fonts]]

{{i18n}}

ExportUnicodeFontToDoom3

2026-03-03T21:57:54Z

Geep: /* Downloads */ Add 1.1 links. Also fix 1.0 source project link

''By Geep, 2025''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints]. As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

More broadly, this program supersedes ExportFontToDoom3 and ExportFontToDoom3All256.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

Note: The spacing value is relative to 256x256 bitmaps. If you specify "hirez", then for proportionality, your specified value is doubled internally when laying out the 512x512 bitmaps.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== hirez or hires ===
''Added with v1.1, March 2026''

Instead of 256x256 bitmaps (and associated DAT), this exports double-sized 512x512 bitmaps with larger, thus potentially-crisper glyphs. (TDM has had a few such 512x512 bitmaps for redrawn fonts since 2014.) Specifically, it:
* outputs requested 512x512 TGAs and their DAT(s).
* internally, doubles the pointsize requested of the TTF, resulting in 2x character glyph sizes. Because both the container bitmap and each glyph is (nominally) doubled in size, the [0...1] s, s2, t, t2 values stored in the DAT would be very similar to those for a 256x256 bitmap.
* Other DAT layout metrics remain as if scaled to a 256x256 layout:
** imageHeight
** imageWidth
** skip
** top (with bottom ignored as usual)
** pitch

For example, if you request –hirez with a carleton font in just 24pt type, it will create 512x512 files with usual names like carleton_0_24.tga, etc., and fontimage_24.dat, with the console showing “Exporting carleton 24pt”.
But internally, 48pt characters are drawn on the bitmaps. The imageHeight and imageWidth metrics will be as if the requested 24pt size was generated, so in effect causing the game engine to scale down by a factor of 2 from the 48pt glyph.

''TIPS on Using a HiRez DAT file''

If subsequently using Refont to make a REF file from this DAT, add the “scaling_ok” option, to suppress otherwise voluminous console warnings about per-character scaling.

If you further plan to generate bounds visualizations with datBounds:
* For 512x512 bitmaps, you must generate from the REF file, not directly from DAT.
* Be sure to first edit the REF to add headers of the form:

# bitmap_<n>_size=512

, where <n> is 0 to 9, and refers to shaders present in the REF file with names like, e.g., carleton_3_48.tga; n would be 3 for this.

* datBounds also has a worthwhile “scaling_ok” option.
* Uncommonly, the raw_metrics option may be of interest.

=== bolding <n> ===
''Added with v1.1, March 2026''

Sometimes, a TTF font does not offer a “bold” variant, or, more generally, a font’s overall stroke thickness is not ideal. It is possible to alter character outlines, then render those as bitmap glyphs.
The result is usually better than post-bitmap thickening or thinning. The FreeType library offers an embolding function for outline alteration, used here.

The integer parameter <n> is in units of 1/64th pixel. A positive value adjusts the outline to embolden the character. Negative thins. The change affects both the overall height and width of the outline.
You may think of the left and bottom character borders as unchanged. The allowed range of <n> is from -128 (i.e., -2 pixels) to 256 (4 pixels). The -2 limit is arbitrary, but for sanity. The 4 limit is the maximum FreeType permits.

Caution: The adjustment algorithm moves the outline controls points, but doesn't change the number of such points;
this means that certain situations like acute angles or intersections are sometimes handled incorrectly.

== Changed Behavior for Font Title from ExportFontToDoom3All256 ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using or revising the -name argument.

== Downloads ==
Release v1.1 of 2025-03-03:
* [https://drive.google.com/file/d/1f3Jj53qtJGIp0_mz7Riing9Fi8ESFDr5/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_exe.zip] . Windows binary and required archived versions of FreeType and DevIL DLLs.
* [https://drive.google.com/file/d/1nutMBfcIWw63wNTUdnlnzP397tGVpnpd/view?usp=sharing ExportUnicodeFontToDoom3_release_1_1_source_project.zip] . The C++ project file with source code, including source & build project of archived library.

Release 1 of 2024-09-10:
* [https://drive.google.com/file/d/1MqoOfhc5-ovoxoHfRquYAcIEdHEOJAQV/view?usp=sharing ExportUnicodeFontToDoom3_release_1_exe.zip]
* [https://drive.google.com/file/d/1Frk_byfdtCcMqdr27ulw1XyuFj8vYp9x/view?usp=sharing ExportUnicodeFontToDoom3_release_1_source_project.zip]

== See Also ==
The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

ExportUnicodeFontToDoom3

2026-03-02T21:01:07Z

Geep: Added 512x512 for awayFromEdge, hirez, bolding

''By Geep, 2025''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints]. As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

More broadly, this program supersedes ExportFontToDoom3 and ExportFontToDoom3All256.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

Note: The spacing value is relative to 256x256 bitmaps. If you specify "hirez", then for proportionality, your specified value is doubled internally when laying out the 512x512 bitmaps.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== hirez or hires ===
''Added with v1.1, March 2026''

Instead of 256x256 bitmaps (and associated DAT), this exports double-sized 512x512 bitmaps with larger, thus potentially-crisper glyphs. (TDM has had a few such 512x512 bitmaps for redrawn fonts since 2014.) Specifically, it:
* outputs requested 512x512 TGAs and their DAT(s).
* internally, doubles the pointsize requested of the TTF, resulting in 2x character glyph sizes. Because both the container bitmap and each glyph is (nominally) doubled in size, the [0...1] s, s2, t, t2 values stored in the DAT would be very similar to those for a 256x256 bitmap.
* Other DAT layout metrics remain as if scaled to a 256x256 layout:
** imageHeight
** imageWidth
** skip
** top (with bottom ignored as usual)
** pitch

For example, if you request –hirez with a carleton font in just 24pt type, it will create 512x512 files with usual names like carleton_0_24.tga, etc., and fontimage_24.dat, with the console showing “Exporting carleton 24pt”.
But internally, 48pt characters are drawn on the bitmaps. The imageHeight and imageWidth metrics will be as if the requested 24pt size was generated, so in effect causing the game engine to scale down by a factor of 2 from the 48pt glyph.

''TIPS on Using a HiRez DAT file''

If subsequently using Refont to make a REF file from this DAT, add the “scaling_ok” option, to suppress otherwise voluminous console warnings about per-character scaling.

If you further plan to generate bounds visualizations with datBounds:
* For 512x512 bitmaps, you must generate from the REF file, not directly from DAT.
* Be sure to first edit the REF to add headers of the form:

# bitmap_<n>_size=512

, where <n> is 0 to 9, and refers to shaders present in the REF file with names like, e.g., carleton_3_48.tga; n would be 3 for this.

* datBounds also has a worthwhile “scaling_ok” option.
* Uncommonly, the raw_metrics option may be of interest.

=== bolding <n> ===
''Added with v1.1, March 2026''

Sometimes, a TTF font does not offer a “bold” variant, or, more generally, a font’s overall stroke thickness is not ideal. It is possible to alter character outlines, then render those as bitmap glyphs.
The result is usually better than post-bitmap thickening or thinning. The FreeType library offers an embolding function for outline alteration, used here.

The integer parameter <n> is in units of 1/64th pixel. A positive value adjusts the outline to embolden the character. Negative thins. The change affects both the overall height and width of the outline.
You may think of the left and bottom character borders as unchanged. The allowed range of <n> is from -128 (i.e., -2 pixels) to 256 (4 pixels). The -2 limit is arbitrary, but for sanity. The 4 limit is the maximum FreeType permits.

Caution: The adjustment algorithm moves the outline controls points, but doesn't change the number of such points;
this means that certain situations like acute angles or intersections are sometimes handled incorrectly.

== Changed Behavior for Font Title from ExportFontToDoom3All256 ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using or revising the -name argument.

== Downloads ==
[COMING SOON - v1.1]

Release 1 of 2024-09-10:
* [https://drive.google.com/file/d/1MqoOfhc5-ovoxoHfRquYAcIEdHEOJAQV/view?usp=sharing ExportUnicodeFontToDoom3_release_1_exe.zip] . Windows binary and required archived versions of FreeType and DevIL DLLs.
* [https://drive.google.com/file/d/1Frk_byfdtCcMqdr27ulw1XyuFj8vYp9x/view?usp=sharing ExportUnicodeFontToDoom3_release_1_source_project.zip] . The C++ project file with source code, including source & build project of archived library.

The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

Rebuild all lang Program

2026-02-14T16:15:29Z

Geep: /* Downloads */ Update contents of rebuilt_all_lang.zip to include LICENSE.txt

''by Geep, 2026''

==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of most helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use '''rebuild_all_lang'''. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
* deciding in advance which *.lang files are the current directory
* if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As examples, the program distribution includes 2 typical stock preambles.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are ''always'' processed in the order shown. If a particular *.lang is missing, that is merely mentioned to the console, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

Within each of the source *.lang files, comments take only two forms:
* as a preamble; as discussed above, the first line is harvested, the rest ignored.
* as trailing "// ..." at the ends of lines. These will be propagated to the corresponding lines of the rebuilt all.lang. (These will be treated as in the same 8-bit language encoding as the rest of the text in the block. Because all supported encodings include ASCII, English comments will be preserved.)

==== Downloads ====
v 1.1 [Recommended]. This upgrades to a non-deprecated interface to Windows 10/11's built-in encoding support, and adds custom code to fix 1.0's problem with [Romanian]. It was roundtrip tested against AirPocket's new all.lang. See the release notes in the .cpp file for details.
* [https://drive.google.com/file/d/1j7941Sjy969XZvKEkHuHxs2Ikz5DVp46/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/178UsLBFpPPce6N-LPwSrG84VRGeLO35i/view?usp=sharing rebuilt_all_lang_1_1.zip - source code: a .cpp and .sln file]

v 1.0 Source Code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code], consisting of a .cpp, .sln, and open-source license file.

Example preamble files (rename to preamble_utf8.txt to use):
* [https://drive.google.com/file/d/1bOWXv8Ju1DktaNYe2M8Bo-V7zAg2bNJ9/view?usp=sharing preamble_utf8[traditional<nowiki>]</nowiki>.txt]
* [https://drive.google.com/file/d/1QKJvG6FoAzuI7njO5fnOQCgwbXQdWqQe/view?usp=sharing preamble_utf8[modern for FM<nowiki>]</nowiki>.txt]

==== See Also ====
* [[Gen_Lang_Programs]]

{{i18n}} {{editing}}

Rebuild all lang Program

2026-02-01T20:42:37Z

Geep: /* Downloads */ links to 1.1 distribution

''by Geep, 2026''

==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of most helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use '''rebuild_all_lang'''. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
* deciding in advance which *.lang files are the current directory
* if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As examples, the program distribution includes 2 typical stock preambles.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are ''always'' processed in the order shown. If a particular *.lang is missing, that is merely mentioned to the console, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

Within each of the source *.lang files, comments take only two forms:
* as a preamble; as discussed above, the first line is harvested, the rest ignored.
* as trailing "// ..." at the ends of lines. These will be propagated to the corresponding lines of the rebuilt all.lang. (These will be treated as in the same 8-bit language encoding as the rest of the text in the block. Because all supported encodings include ASCII, English comments will be preserved.)

==== Downloads ====
v 1.1 [Recommended]. This upgrades to a non-deprecated interface to Windows 10/11's built-in encoding support, and adds custom code to fix 1.0's problem with [Romanian]. It was roundtrip tested against AirPocket's new all.lang. See the release notes in the .cpp file for details.
* [https://drive.google.com/file/d/1j7941Sjy969XZvKEkHuHxs2Ikz5DVp46/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/178UsLBFpPPce6N-LPwSrG84VRGeLO35i/view?usp=sharing rebuilt_all_lang_1_1.zip - source code: a .cpp and .sln file]

v 1.0 Source Code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code: a .cpp and .sln file]

Example preamble files (rename to preamble_utf8.txt to use):
* [https://drive.google.com/file/d/1bOWXv8Ju1DktaNYe2M8Bo-V7zAg2bNJ9/view?usp=sharing preamble_utf8[traditional<nowiki>]</nowiki>.txt]
* [https://drive.google.com/file/d/1QKJvG6FoAzuI7njO5fnOQCgwbXQdWqQe/view?usp=sharing preamble_utf8[modern for FM<nowiki>]</nowiki>.txt]

==== See Also ====
* [[Gen_Lang_Programs]]

{{i18n}} {{editing}}

Rebuild all lang Program

2026-02-01T20:38:54Z

Geep: Update for v 1.1

''by Geep, 2026''

==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of most helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use '''rebuild_all_lang'''. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
* deciding in advance which *.lang files are the current directory
* if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As examples, the program distribution includes 2 typical stock preambles.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are ''always'' processed in the order shown. If a particular *.lang is missing, that is merely mentioned to the console, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

Within each of the source *.lang files, comments take only two forms:
* as a preamble; as discussed above, the first line is harvested, the rest ignored.
* as trailing "// ..." at the ends of lines. These will be propagated to the corresponding lines of the rebuilt all.lang. (These will be treated as in the same 8-bit language encoding as the rest of the text in the block. Because all supported encodings include ASCII, English comments will be preserved.)

==== Downloads ====
v 1.1 [Recommended]. This upgrades to a non-deprecated interface to Windows 10/11's built-in encoding support, and adds custom code to fix 1.0's problem with [Romanian]. It was roundtrip tested against AirPocket's new all.lang. See the release notes in the .cpp file for details.
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code: a .cpp and .sln file]

v 1.0 Source Code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code: a .cpp and .sln file]

Example preamble files (rename to preamble_utf8.txt to use):
* [https://drive.google.com/file/d/1bOWXv8Ju1DktaNYe2M8Bo-V7zAg2bNJ9/view?usp=sharing preamble_utf8[traditional<nowiki>]</nowiki>.txt]
* [https://drive.google.com/file/d/1QKJvG6FoAzuI7njO5fnOQCgwbXQdWqQe/view?usp=sharing preamble_utf8[modern for FM<nowiki>]</nowiki>.txt]
==== See Also ====
* [[Gen_Lang_Programs]]

{{i18n}} {{editing}}

CBinaryFrobMover

2026-01-27T16:08:20Z

Geep: /* Spawnargs */ lock_picktype can't use *

== CBinaryFrobMover ==
The corresponding entityDef is '''atdm:mover_binarymover_base'''

This is the base class for all kind of TDM movers like doors, levers and buttons. The class is designed for movers with two final states, namely open and closed. "Open" and "closed" are abstract definitions and just refer to the "one" and the "other" state of the mover. All movers are considered to spawn "closed", until specifically specified otherwise.

The class provides a set of virtual C++ methods which can be specialised by their subclasses to implement the correct behaviour. It also supports a number if spawnargs which allow for easy customisation right in the editor.

=== Spawnargs ===
* '''rotate''': (angles) Describes the angles (pitch, yaw, roll) the mover will rotate from its closed to its open position (default is: "90 0 0").
* '''translate''': (vector) Describes the relative movement of the origin from the closed position to the open position (default is "0 0 0" = no movement).
* '''translate_speed''': (float) Is the linear velocity in units per second (defaults to "0", which means that the idMover's "move_time" value is in effect).

* '''open''': (1/0) Specifies whether this frobmover spawns at the open position. Can be used to let doors be open right at map start (default is "0").
* '''locked''': (1/0) Specifies whether this frobmover is locked (default is "0").

* '''pickable''' (1/0) If set to 1, is pickable by the player's lockpicks.
* '''lock_picktype''' (string) This is a sequence of characters defining the lockpick types which need to be used on this door. For instance, 'ts' means that the player must use the "triangle" lockpick ('t') and the "snake" lockpick ('s') - in this order - to pick this lock. [At one time, it was said that it's also possible to define a star "*", which matches all lockpick types. Doesn't work now.]
* '''lock_pins''' (string) Defines the number of samples per pin to be created for this lock, example: "56" will create two patterns, one with 5 samples, the other with 6 samples. The minimum is 5.

* '''open_on_unlock''': (1/0) When set to 1, the mover will automatically open when unlocked.
* '''interruptable''': (1/0) When set to 1, the mover can be stopped by frobbing it while moving. Doors are interruptable by default, whereas levers are not.
* '''stop_when_blocked''': (1/0) Set stop_when_blocked to 0 to let this entity keep moving after the blocking entity is out of the way. Default is "1".
* '''push_player''': (1/0) Set this to 1 to let this entity push the player (default is 0). Note: belongs to idMover class.

* '''start_rotate''': (Angles) Defines the angles (pitch, yaw, roll) the mover should have at spawn time (to let the mover start out half-open, for instance). Defaults to "0 0 0".
* "start_position": (Vector) Defines the position the mover should have at spawn time (to let the mover start out open, for instance). Defaults to "0 0 0".
* '''first_frob_open''': (1/0) This helps to disambiguate the behaviour when a mover starts at a half-open position. When this is set to "1" the first frob will make the mover open. Defaults to "0", which means that half-open doors will close on first frob by default.

* '''snd_open''' (Soundshader) The sound to be played when the door starts to open after being fully closed.
* '''snd_close''' (Soundshader) The sound to be played when the door reaches its fully closed position.
* '''snd_locked''' (Soundshader) The sound to be played when the mover becomes locked.
* '''snd_unlock''' (Soundshader) The sound to be played when the mover unlocks.

* '''snd_lockpick_pin_NN''' (soundshader) Defines the sample sound for the pin with NN being in the range 00..14. The number must have a leading zero in it (e.g. '02').
* '''snd_lockpick_pin_sweetspot''' (soundshader) Defines the sweet spot sound to be inserted into a pattern.
* '''snd_lockpick_lock_picked''' (soundshader) Is played when the entire lock is successfully picked.
* '''snd_lockpick_pin_success''' (soundshader) Is played when a single pin is successfully picked.
* '''snd_lockpick_pin_fail''' (soundshader) Is played when the player fails to pick a pin.
* '''snd_lockpick_pick_wrong''' (soundshader) Is played when the user either attempts to use the wrong lockpick type or the door is not locked or already picked.

* '''trigger_on_open''': (1/0) If set to "1", this binary mover will trigger its targets when it starts out completely closed and is opened. Default is "0".
* '''trigger_on_close''': (1/0) If set to 1, this binary mover will trigger its targets when it completely closes after being open. Default is "0".
* '''trigger_when_opened''': (1/0) If set to 1, this binary mover will trigger its targets when it is completely opened. Default is "0".

* '''auto_close_time''': (float) Set this to something >= 0 to let the mover automatically close (again) after this time when being fully opened (measured in seconds). Defaults to -1, which means 'do not autoclose'. The event is also activated when the mover starts at the open position at map start.
* '''auto_open_time''': (float) Set this to something >= 0 to let the mover automatically open (again) after this time when being fully closed (measured in seconds). Defaults to -1, which means 'do not autoopen'. The event is also activated when the mover starts at the closed position at map start.

* '''impulse_thresh_open''': (float) Defines the minimum impulse which needs to be applied to this door before it starts to open. If 0, no open impulse is defined. Default is "0".
* '''impulse_thresh_close''': (float) Defines the minimum impulse which needs to be applied to this door before it starts to close. If 0, no close impulse is defined. Default is "0".
* '''impulse_dir_open''': (vector) Defines the direction the open impulse needs to be applied in. Default is '0 0 0'.
* '''impulse_dir_close''': (vector) Defines the direction the close impulse needs to be applied in. Default is '0 0 0'.

* '''state_change_callback''': (script function) Defines the (global) script function which gets called when the mover changes its state. Function signature is: '''void statChangeCallback(entity self, bool open, bool locked, bool interrupted);'''

=== Script Events ===

;scriptEvent void Open();
: Opens the frobmover, regardless of its previous state. The mover will not move when it's locked.

;scriptEvent void Close();
: Closes the frobmover, regardless of its previous state.

;scriptEvent void ToggleOpen();
: Toggles the mover state. Closes when fully open, opens when fully closed. If the mover is "interrupted" (e.g. when the player frobbed the mover in between), the move direction depends on the state of the internal "intent_open" flag.

; scriptEvent float IsOpen();
: Returns true (nonzero) if the mover is open, which is basically the same as "not closed". A mover is considered closed when it is at its close position.

;scriptEvent void Lock();
: Locks the mover. Calls to Open() will not succeed after this call.
;scriptEvent void Unlock();
: Unlocks the mover. Calls to Open() will succeed after this call. Depending on the value of the spawnarg "open_on_unlock" the mover might automatically open after this call.
; scriptEvent void ToggleLock();
: Toggles the lock state. Unlocked movers will be locked and vice versa. The notes above concerning Unlock() still apply if this call unlocks the mover.
; scriptEvent float IsLocked();
: Returns true (nonzero) if the mover is currently locked.
; scriptEvent float IsPickable();
: Returns true (nonzero) if this mover is pickable.

=== C++ Methods and Events ===
The class declaration can be found in ''DarkMod/BinaryFrobMover.h''. Apart from the mandatory Spawn/Save/Restore triple, a set of virtual functions are defined, which can be overridden by the subclasses to specialise the mover.

Note: This is just an abstract. A detailed description of these events can be found in the BinaryFrobMover.h header itself.

As many spawnargs refer to other entities, most of the setup work can be performed after all entities have been spawned. This is why the FrobMover base class provides a virtual PostSpawn() method which can be overridden by the subclasses to do work after all map entities have been spawned. Note: don't forget to call the base class!
// Gets called 16 ms after all entities have been spawned
virtual void PostSpawn();

The class provides a few virtual "interceptor" methods, which can be used to prevent the mover from changing its state. These are:
virtual bool PreOpen();
virtual bool PreClose();
virtual bool PreInterrupt();
virtual bool PreLock(bool bMaster);
virtual bool PreUnlock(bool bMaster);
These functions are called when the mover is about to perform the respective action. If these are returning TRUE, the mover has green light and it will start to open/close/lock/unlock/get interrupted. Returning FALSE will prevent the calling method from continuing. An example of such a use case is the CBinaryFrobMover::PreOpen() routine, which returns FALSE when the mover is locked. It also playes the "I'm locked" sound before returning.

There are a couple of "state change events", which get invoked when the mover has reached a certain state:
// Gets called when the mover actually starts to move, regardless what the state was beforehand.
// The boolean tells which state the mover is heading towards.
virtual void OnMoveStart(bool open);

// Gets called when the mover is about to move towards its "open" and "closed" position, resp..
virtual void OnStartOpen(bool wasClosed, bool bMaster);
virtual void OnStartClose(bool wasOpen, bool bMaster);

// Gets called when the mover has just reached its fully open/closed position. Does not get called when the mover didn't move.
virtual void OnOpenPositionReached();
virtual void OnClosedPositionReached();

// Gets called when the mover has just been interrupted.
virtual void OnInterrupt();

// Is called when the mover has just been locked/unlocked.
virtual void OnLock(bool bMaster);
virtual void OnUnlock(bool bMaster);

General note on overriding these events: Althought most default implementations are empty, it's still advisable to call the base class from the overriding function. Default behaviour might be changed in the future, so always call the base class, unless you really need to prevent the default action from being executed.

{{coding}} {{editing}}

Rebuild all lang Program

2026-01-23T18:15:07Z

Geep: /* Usage and Example */

''by Geep, 2026''

'''HOLD OFF ON USING THIS! Further round-trip testing indicates a problem needing more work.'''

==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use '''rebuild_all_lang'''. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
* deciding in advance which *.lang files are the current directory
* if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As an example, the program distribution includes a typical stock preamble.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are always processed in the order shown. If a particular *.lang is missing, that is handled quietly, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

==== Downloads ====
v 1.0 Source Code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code: a .cpp and .sln file]

Example preamble files (rename to preamble_utf8.txt to use):
* [https://drive.google.com/file/d/1bOWXv8Ju1DktaNYe2M8Bo-V7zAg2bNJ9/view?usp=sharing preamble_utf8[traditional<nowiki>]</nowiki>.txt]
* [https://drive.google.com/file/d/1QKJvG6FoAzuI7njO5fnOQCgwbXQdWqQe/view?usp=sharing preamble_utf8[modern for FM<nowiki>]</nowiki>.txt]
==== See Also ====
* [[Gen_Lang_Programs]]

{{i18n}} {{editing}}

Rebuild all lang Program

2026-01-23T18:07:16Z

Geep: HOLD OFF ON USING THIS warning added

''by Geep, 2026''

'''HOLD OFF ON USING THIS! Further round-trip testing indicates a problem needing more work.'''

==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use '''rebuild_all_lang'''. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
- deciding in advance which *.lang files are the current directory
- if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As an example, the program distribution includes a typical stock preamble.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are always processed in the order shown. If a particular *.lang is missing, that is handled quietly, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

==== Downloads ====
v 1.0 Source Code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code: a .cpp and .sln file]

Example preamble files (rename to preamble_utf8.txt to use):
* [https://drive.google.com/file/d/1bOWXv8Ju1DktaNYe2M8Bo-V7zAg2bNJ9/view?usp=sharing preamble_utf8[traditional<nowiki>]</nowiki>.txt]
* [https://drive.google.com/file/d/1QKJvG6FoAzuI7njO5fnOQCgwbXQdWqQe/view?usp=sharing preamble_utf8[modern for FM<nowiki>]</nowiki>.txt]
==== See Also ====
* [[Gen_Lang_Programs]]

{{i18n}} {{editing}}

I18N - Charset

2026-01-21T20:26:23Z

Geep: /* Russian */

== Introduction ==

The D3 code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, special charsets are used. The fonts (Carleton for the menu f.i.) are build/patched so that the right characters appear in the right place.

== Encodings ==

=== all.lang ===

This file is in '''UTF-8''', and converted with the help of the script '''devel/gen_lang.pl''':

perl devel/gen_lang.pl

This ensures that the generated language files are in their proper encodings (see below).

=== All other language files ===

Note that the language files (f.i. '''strings/german.lang''') as well as the readables and the FM dictionariaries are expected to be in the following encodings:

* '''Czech''', '''Hungarian''', '''Slovak''', '''Polish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-2 ISO-8859-2] ('''not WIN-1250!)
* '''Russian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/Win-1251 WIN-1251]
* '''French:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-15 ISO-8859-15]
* '''Romanian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-16 ISO-8859-16]
* '''All other languages:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-1 ISO-8859-1] (German, Dutch, Danish, Swedish, Portuguese, etc.)

{{infobox|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}

== Character remapping ==

The characters are remapped upon loading the dictionary/readable, from their native encoding to the special one that TDM uses and that is described here. Responsible for the remapping are [[I18N - Character mapping|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found, "strings/default.map" is used instead, if this is not found, no remapping takes place.

See '''[[I18N - Character mapping|Character mapping]]''' for more information.

=== European Languages ===

This mapping is used for European languages, f.i. '''Czech''', '''French''', '''German''', '''Spanish''', '''Portuguese''', '''Polish'''. Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!

In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.

'''Color code:'''

{{box|#f0d0d0|Character not usable by TDM|Unusable}}{{box|#d0e0d0|Character not yet used in TDM|Unused}}{{box|#c0ffc0|Character displayed in v1.08 or newer|Usable in v1.08}}{{box|#80f080|Character displayed in v2.03 or newer|Usable in v2.03}}{{box|#d0d0f0|Changed from the ISO-8859-1 default, usable by TDM 1.0 or newer|Changed from ISO 8859-1}}

{|class="wikitable" border=1 style="border-collapse: collapse; font-size: 95%" cellspacing=0 cellpadding=2 width=100%

|-
!
!…0
!…1
!…2
!…3
!…4
!…5
!…6
!…7
!…8
!…9
!…A
!…B
!…C
!…D
!…E
!…F

|-
!0…
|align='center' style='background: #f0d0d0'|00 '''–'''
|align='center' style='background: #f0d0d0'|01 '''–'''
|align='center' style='background: #f0d0d0'|02 '''–'''
|align='center' style='background: #f0d0d0'|03 '''–'''
|align='center' style='background: #f0d0d0'|04 '''–'''
|align='center' style='background: #f0d0d0'|05 '''–'''
|align='center' style='background: #f0d0d0'|06 '''–'''
|align='center' style='background: #f0d0d0'|07 '''–'''
|align='center' style='background: #f0d0d0'|08 '''–'''
|align='center' style='background: #f0d0d0'|09 '''–'''
|align='center' style='background: #f0d0d0'|0A '''–'''
|align='center' style='background: #f0d0d0'|0B '''–'''
|align='center' style='background: #f0d0d0'|0C '''–'''
|align='center' style='background: #f0d0d0'|0D '''–'''
|align='center' style='background: #f0d0d0'|0E '''–'''
|align='center' style='background: #f0d0d0'|0F '''–'''

|-
!1…
|align='center' style='background: #f0d0d0'|10 '''–'''
|align='center' style='background: #f0d0d0'|11 '''–'''
|align='center' style='background: #f0d0d0'|12 '''–'''
|align='center' style='background: #f0d0d0'|13 '''–'''
|align='center' style='background: #f0d0d0'|14 '''–'''
|align='center' style='background: #f0d0d0'|15 '''–'''
|align='center' style='background: #f0d0d0'|16 '''–'''
|align='center' style='background: #f0d0d0'|17 '''–'''
|align='center' style='background: #f0d0d0'|18 '''–'''
|align='center' style='background: #f0d0d0'|19 '''–'''
|align='center' style='background: #f0d0d0'|1A '''–'''
|align='center' style='background: #f0d0d0'|1B '''–'''
|align='center' style='background: #f0d0d0'|1C '''–'''
|align='center' style='background: #f0d0d0'|1D '''–'''
|align='center' style='background: #f0d0d0'|1E '''–'''
|align='center' style='background: #f0d0d0'|1F '''–'''

|-
!2…
|align='center'|20 ''' '''
|align='center'|21 '''!'''
|align='center'|22 '''"'''
|align='center'|23 '''#'''
|align='center'|24 '''$'''
|align='center'|25 '''%'''
|align='center'|26 '''&'''
|align='center'|27 '''''''
|align='center'|28 '''('''
|align='center'|29 ''')'''
|align='center'|2A '''*'''
|align='center'|2B '''+'''
|align='center'|2C ''','''
|align='center'|2D '''-'''
|align='center'|2E '''.'''
|align='center'|2F '''/'''

|-
!3…
|align='center'|30 '''0'''
|align='center'|31 '''1'''
|align='center'|32 '''2'''
|align='center'|33 '''3'''
|align='center'|34 '''4'''
|align='center'|35 '''5'''
|align='center'|36 '''6'''
|align='center'|37 '''7'''
|align='center'|38 '''8'''
|align='center'|39 '''9'''
|align='center'|3A ''':'''
|align='center'|3B ''';'''
|align='center'|3C '''<'''
|align='center'|3D '''='''
|align='center'|3E '''>'''
|align='center'|3F '''?'''

|-
!4…
|align='center'|40 '''@'''
|align='center'|41 '''A'''
|align='center'|42 '''B'''
|align='center'|43 '''C'''
|align='center'|44 '''D'''
|align='center'|45 '''E'''
|align='center'|46 '''F'''
|align='center'|47 '''G'''
|align='center'|48 '''H'''
|align='center'|49 '''I'''
|align='center'|4A '''J'''
|align='center'|4B '''K'''
|align='center'|4C '''L'''
|align='center'|4D '''M'''
|align='center'|4E '''N'''
|align='center'|4F '''O'''

|-
!5…
|align='center'|50 '''P'''
|align='center'|51 '''Q'''
|align='center'|52 '''R'''
|align='center'|53 '''S'''
|align='center'|54 '''T'''
|align='center'|55 '''U'''
|align='center'|56 '''V'''
|align='center'|57 '''W'''
|align='center'|58 '''X'''
|align='center'|59 '''Y'''
|align='center'|5A '''Z'''
|align='center'|5B '''['''
|align='center'|5C '''\'''
|align='center'|5D ''']'''
|align='center'|5E '''^'''
|align='center'|5F '''_'''

|-
!6…
|align='center'|60 '''`'''
|align='center'|61 '''a'''
|align='center'|62 '''b'''
|align='center'|63 '''c'''
|align='center'|64 '''d'''
|align='center'|65 '''e'''
|align='center'|66 '''f'''
|align='center'|67 '''g'''
|align='center'|68 '''h'''
|align='center'|69 '''i'''
|align='center'|6A '''j'''
|align='center'|6B '''k'''
|align='center'|6C '''l'''
|align='center'|6D '''m'''
|align='center'|6E '''n'''
|align='center'|6F '''o'''

|-
!7…
|align='center'|70 '''p'''
|align='center'|71 '''q'''
|align='center'|72 '''r'''
|align='center'|73 '''s'''
|align='center'|74 '''t'''
|align='center'|75 '''u'''
|align='center'|76 '''v'''
|align='center'|77 '''w'''
|align='center'|78 '''x'''
|align='center'|79 '''y'''
|align='center'|7A '''z'''
|align='center'|7B '''{'''
|align='center'|7C '''|'''
|align='center'|7D '''}'''
|align='center'|7E '''~'''
|align='center' style='background: #d0e0d0'|7F '''�'''

|-
!8…
|align='center' style='background: #c0ffc0'|80 '''Ň'''
|align='center' style='background: #c0ffc0'|81 '''Ś'''
|align='center' style='background: #c0ffc0'|82 '''Ć'''
|align='center' style='background: #c0ffc0'|83 '''Ż'''
|align='center' style='background: #c0ffc0'|84 '''Ź'''
|align='center' style='background: #c0ffc0'|85 '''Ŝ'''
|align='center' style='background: #c0ffc0'|86 '''Ĉ'''
|align='center' style='background: #c0ffc0'|87 '''Ẑ'''
|align='center' style='background: #c0ffc0'|88 '''Ô''' [note]
|align='center' style='background: #c0ffc0'|89 '''Ŕ'''
|align='center' style='background: #c0ffc0'|8A '''Ǔ'''
|align='center' style='background: #c0ffc0'|8B '''Ă'''
|align='center' style='background: #c0ffc0'|8C '''Ń'''
|align='center' style='background: #80f080'|8D '''Ș'''
|align='center' style='background: #80f080'|8E '''Ț'''
|align='center' style='background: #d0e0d0'|8F '''�'''

|-
!9…
|align='center' style='background: #80f080'|90 '''đ'''
|align='center' style='background: #c0ffc0'|91 '''ś'''
|align='center' style='background: #c0ffc0'|92 '''ć'''
|align='center' style='background: #c0ffc0'|93 '''ż'''
|align='center' style='background: #c0ffc0'|94 '''ź'''
|align='center' style='background: #c0ffc0'|95 '''ŝ'''
|align='center' style='background: #c0ffc0'|96 '''ĉ'''
|align='center' style='background: #c0ffc0'|97 '''ẑ'''
|align='center' style='background: #c0ffc0'|98 '''ô''' [note]
|align='center' style='background: #c0ffc0'|99 '''ŕ'''
|align='center' style='background: #c0ffc0'|9A '''ǔ'''
|align='center' style='background: #c0ffc0'|9B '''ă'''
|align='center' style='background: #c0ffc0'|9C '''ń'''
|align='center' style='background: #80f080'|9D '''ș'''
|align='center' style='background: #80f080'|9E '''ț'''
|align='center' style='background: #d0e0d0'|9F '''�'''

|-
!A…
|align='center'|A0 '''[https://secure.wikimedia.org/wikipedia/en/wiki/Non-breaking_space NBSP]'''
|align='center' style='background: #d0d0f0'|A1 '''ň''' (¡)'''
|align='center' style='background: #d0d0f0'|A2 '''Ű''' (¢)'''
|align='center' style='background: #d0d0f0'|A3 '''ě''' (£)'''
|align='center' style='background: #d0d0f0'|A4 '''ű''' (¤)'''
|align='center' style='background: #d0d0f0'|A5 '''Ě''' (¥)'''
|align='center' style='background: #d0d0f0'|A6 '''Š''' (¦)'''
|align='center'|A7 '''§'''
|align='center' style='background: #d0d0f0'|A8 '''š''' (¨)'''
|align='center' style='background: #d0d0f0'|A9 '''Ů''' (©)'''
|align='center' style='background: #d0d0f0'|AA '''Ą''' (ª)'''
|align='center' style='background: #d0d0f0'|AB '''Ę''' («)'''
|align='center' style='background: #d0d0f0'|AC '''Č''' (¬)'''
|align='center'|AD '''[https://secure.wikimedia.org/wikipedia/en/wiki/Soft_hyphen SHY]'''
|align='center' style='background: #d0d0f0'|AE '''č''' (®)'''
|align='center' style='background: #d0d0f0'|AF '''ů''' (¯)'''

|-
!B…
|align='center' style='background: #d0d0f0'|B0 '''Ő''' (°)'''
|align='center' style='background: #d0d0f0'|B1 '''Ł''' (±)'''
|align='center' style='background: #d0d0f0'|B2 '''Ť''' (²)'''
|align='center' style='background: #d0d0f0'|B3 '''Ď''' (³)'''
|align='center' style='background: #d0d0f0'|B4 '''Ž''' (´)'''
|align='center' style='background: #d0d0f0'|B5 '''ł''' (µ)'''
|align='center' style='background: #d0d0f0'|B6 '''ť''' (¶)'''
|align='center' style='background: #d0d0f0'|B7 '''ď''' (·)'''
|align='center' style='background: #d0d0f0'|B8 '''ž''' (¸)'''
|align='center' style='background: #d0d0f0'|B9 '''ő''' (¹)'''
|align='center' style='background: #d0d0f0'|BA '''ą''' (º)'''
|align='center' style='background: #d0d0f0'|BB '''ę''' (»)'''
|align='center' style='background: #d0d0f0'|BC '''Œ''' (¼)'''
|align='center' style='background: #d0d0f0'|BD '''œ''' (½)'''
|align='center' style='background: #d0d0f0'|BE '''Ÿ''' (¾)'''
|align='center'|BF '''¿'''

|-
!C…
|align='center'|C0 '''À'''
|align='center'|C1 '''Á'''
|align='center'|C2 '''Â'''
|align='center'|C3 '''Ã'''
|align='center'|C4 '''Ä'''
|align='center'|C5 '''Å'''
|align='center'|C6 '''Æ'''
|align='center'|C7 '''Ç'''
|align='center'|C8 '''È'''
|align='center'|C9 '''É'''
|align='center'|CA '''Ê'''
|align='center'|CB '''Ë'''
|align='center'|CC '''Ì'''
|align='center'|CD '''Í'''
|align='center'|CE '''Î'''
|align='center'|CF '''Ï'''

|-
!D…
|align='center'|D0 '''Ð'''
|align='center'|D1 '''Ñ'''
|align='center'|D2 '''Ò'''
|align='center'|D3 '''Ó'''
|align='center'|D4 '''Ô'''
|align='center'|D5 '''Õ'''
|align='center'|D6 '''Ö'''
|align='center' style='background: #d0d0f0'|D7 '''Ř''' (×)'''
|align='center'|D8 '''Ø'''
|align='center'|D9 '''Ù'''
|align='center'|DA '''Ú'''
|align='center'|DB '''Û'''
|align='center'|DC '''Ü'''
|align='center'|DD '''Ý'''
|align='center'|DE '''Þ'''
|align='center'|DF '''ß'''

|-
!E…
|align='center'|E0 '''à'''
|align='center'|E1 '''á'''
|align='center'|E2 '''â'''
|align='center'|E3 '''ã'''
|align='center'|E4 '''ä'''
|align='center'|E5 '''å'''
|align='center'|E6 '''æ'''
|align='center'|E7 '''ç'''
|align='center'|E8 '''è'''
|align='center'|E9 '''é'''
|align='center'|EA '''ê'''
|align='center'|EB '''ë'''
|align='center'|EC '''ì'''
|align='center'|ED '''í'''
|align='center'|EE '''î'''
|align='center'|EF '''ï'''

|-
!F…
|align='center'|F0 '''ð'''
|align='center'|F1 '''ñ'''
|align='center'|F2 '''ò'''
|align='center'|F3 '''ó'''
|align='center'|F4 '''ô'''
|align='center'|F5 '''õ'''
|align='center'|F6 '''ö'''
|align='center' style='background: #d0d0f0'|F7 '''ř''' (÷)'''
|align='center'|F8 '''ø'''
|align='center'|F9 '''ù'''
|align='center'|FA '''ú'''
|align='center'|FB '''û'''
|align='center'|FC '''ü'''
|align='center'|FD '''ý'''
|align='center'|FE '''þ'''
|align='center'|FF '''ÿ'''

|}

[note] As discussed [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/&do=findComment&comment=494855 here], the TDM char set has a redundant treatment of 2 characters:

* Ô appears at 0x88 and 0xD4
* ô appears at 0x98 and 0xF4

Planned for TDM 2.13, the redundancy can be removed and these new characters (from ISO-8859-3) introduced:

* Ğ appears at 0x88
* ğ appears at 0x98

For a mapping of these 256 codepoints to Unicode U+NNNN values and formal names, download 'TDM 8859-Style Font Map to Unicode-16.txt':

* [https://drive.google.com/file/d/1wLEtuFvnrZ-WHK8ign8i3diIXZdriZ4F/view?usp=sharing for TDM 2.13]
* [https://drive.google.com/file/d/1UAz9jSZpT_j33STP3So_Re8JWa8QuAmz/view?usp=sharing for TDM 2.12 and earlier]

Each of these files (by Geep, 2024) is in a standardized format so that it can also be imported into font design programs like FontForge as a custom 256-position map. In the comments, there is additional information about:
* ISO 8859-x sourcing of each character.
* alternative representations of some European and control characters.

=== Russian ===

Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6.

Within any given font, character overage may be incomplete. A 2025 effort extended the Mason 48pt font to all 256 codepoints.

=== Asian Languages (Korean, Chinese, Japanese) ===

The original D3 had support for these languages, so it might be possible to add them to TDM, too. At the moment, however, we lack the fonts and translators. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

== European Character Implementation ==
=== Priority Early-On - the "Top 50" ===

Some of the special characters are used more often than others. Here is a statistic over the entire string set of the TDM core, from TDM v1.08, showing the top 50 most-used characters (excluding a-z, 0-9 and russian characters):

{|class="wikitable" border=1 style="border-collapse: collapse; font-size: 85%" cellspacing=0 cellpadding=2

|-
|Rank
|Occurances
|Letter
|Remarks
|Rank
|Occurances
|Letter
|Remarks

|-
|1
|í
|715
|
|25
|ć
|67
|

|-
|2
|é
|674
|
|26
|è
|65
|

|-
|3
|á
|524
|
|27
|ú
|56
|

|-
|4
|ø
|303
|Danish
|28
|ê
|52
|

|-
|5
|č
|288
|
|29
|ö
|48
|German

|-
|6
|ó
|283
|
|30
|É
|46
|

|-
|7
|ü
|270
|German
|31
|ñ
|37
|

|-
|8
|ł
|203
|Polish
|32
|õ
|32
|

|-
|9
|æ
|200
|Danish
|33
|ń
|26
|

|-
|10
|ě
|182
|
|34
|Ł
|24
|

|-
|11
|ř
|175
|Czech
|35
|Š
|21
|

|-
|12
|ã
|168
|
|36
|â
|21
|

|-
|13
|ž
|148
|Czech
|37
|ź
|20
|

|-
|14
|ý
|142
|
|38
|ß
|18
|German

|-
|15
|ę
|141
|
|39
|Ó
|18
|

|-
|16
|ą
|140
|
|40
|ň
|15
|

|-
|17
|ż
|119
|
|41
|Ú
|15
|

|-
|18
|å
|109
|Danish
|42
|Á
|13
|

|-
|19
|š
|99
|
|43
|î
|12
|

|-
|20
|ś
|97
|
|44
|ť
|11
|

|-
|21
|ç
|91
|
|45
|ô
|9
|

|-
|22
|ä
|86
|German
|46
|Ž
|8
|

|-
|23
|à
|83
|
|47
|Ż
|7
|

|-
|24
|ů
|77
|
|48
|Č
|7
|

|-
|25
|ć
|67
|
|49
|ù
|6
|

|}

Although ö, ä and ü do not appear that often, with only these and Ü, Ö, Ä and ß, the entire German language works. So adding these letters to the fonts is quite important.

Preferably, all foreign letters would be added to the fonts (see [[Font Patcher]] or [[Refont]]). However, if time permits only adding a few, '''í''' would be more important than, say, '''ô'''.

It is commonplace for missing accented letters to be redirected in the .dat file to the corresponding unaccented base letter.

=== Coverage Expansion & Remaining Limitations ===

By 2014, all the system (e.g., main menu) fonts (Carleton, Carleton_condensed, Stone in sizes 24pt and 48pt; Mason and Mason_glow in 48pt) had good coverage of the "Top 50" and beyond, although not all 256 codepoints.
This was confirmed more specifically in 2024, as part of an [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ Analysis of 2.12 Fonts]. This analysis indicated that, unlike the system fonts, the FM fonts generally did not provide specific glyphs beyond ASCII.

In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14. Similar Stone 48pt and Carleton 24pt improvements are planned for TDM 2.15.

[[Category:Fonts]]

{{i18n}}

I18N - Charset

2026-01-21T20:09:12Z

Geep: /* European Character Implementation */ subsection headers, minor updates

== Introduction ==

The D3 code that handles the GUI bitmap font can only load a specific range of bytes as characters. To get the most out of the available entries, special charsets are used. The fonts (Carleton for the menu f.i.) are build/patched so that the right characters appear in the right place.

== Encodings ==

=== all.lang ===

This file is in '''UTF-8''', and converted with the help of the script '''devel/gen_lang.pl''':

perl devel/gen_lang.pl

This ensures that the generated language files are in their proper encodings (see below).

=== All other language files ===

Note that the language files (f.i. '''strings/german.lang''') as well as the readables and the FM dictionariaries are expected to be in the following encodings:

* '''Czech''', '''Hungarian''', '''Slovak''', '''Polish:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-2 ISO-8859-2] ('''not WIN-1250!)
* '''Russian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/Win-1251 WIN-1251]
* '''French:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-15 ISO-8859-15]
* '''Romanian:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-16 ISO-8859-16]
* '''All other languages:''' [https://secure.wikimedia.org/wikipedia/en/wiki/ISO/IEC_8859-1 ISO-8859-1] (German, Dutch, Danish, Swedish, Portuguese, etc.)

{{infobox|The core dictionaries are automatically generated in the right encoding, but make sure that you use the right encoding for the FM dictionary, too!}}

== Character remapping ==

The characters are remapped upon loading the dictionary/readable, from their native encoding to the special one that TDM uses and that is described here. Responsible for the remapping are [[I18N - Character mapping|mapping files]], f.i. "strings/czech.map". If a map file for a specific language is not found, "strings/default.map" is used instead, if this is not found, no remapping takes place.

See '''[[I18N - Character mapping|Character mapping]]''' for more information.

=== European Languages ===

This mapping is used for European languages, f.i. '''Czech''', '''French''', '''German''', '''Spanish''', '''Portuguese''', '''Polish'''. Note that the double accented characters in Hungarian '''Ő, ő, Ű and ű''' look a bit different from '''Ö, ö, Ü and ü'''!

In the table below, the original ISO 8859-1 characters are given in ''()'' below the TDM character.

'''Color code:'''

{{box|#f0d0d0|Character not usable by TDM|Unusable}}{{box|#d0e0d0|Character not yet used in TDM|Unused}}{{box|#c0ffc0|Character displayed in v1.08 or newer|Usable in v1.08}}{{box|#80f080|Character displayed in v2.03 or newer|Usable in v2.03}}{{box|#d0d0f0|Changed from the ISO-8859-1 default, usable by TDM 1.0 or newer|Changed from ISO 8859-1}}

{|class="wikitable" border=1 style="border-collapse: collapse; font-size: 95%" cellspacing=0 cellpadding=2 width=100%

|-
!
!…0
!…1
!…2
!…3
!…4
!…5
!…6
!…7
!…8
!…9
!…A
!…B
!…C
!…D
!…E
!…F

|-
!0…
|align='center' style='background: #f0d0d0'|00 '''–'''
|align='center' style='background: #f0d0d0'|01 '''–'''
|align='center' style='background: #f0d0d0'|02 '''–'''
|align='center' style='background: #f0d0d0'|03 '''–'''
|align='center' style='background: #f0d0d0'|04 '''–'''
|align='center' style='background: #f0d0d0'|05 '''–'''
|align='center' style='background: #f0d0d0'|06 '''–'''
|align='center' style='background: #f0d0d0'|07 '''–'''
|align='center' style='background: #f0d0d0'|08 '''–'''
|align='center' style='background: #f0d0d0'|09 '''–'''
|align='center' style='background: #f0d0d0'|0A '''–'''
|align='center' style='background: #f0d0d0'|0B '''–'''
|align='center' style='background: #f0d0d0'|0C '''–'''
|align='center' style='background: #f0d0d0'|0D '''–'''
|align='center' style='background: #f0d0d0'|0E '''–'''
|align='center' style='background: #f0d0d0'|0F '''–'''

|-
!1…
|align='center' style='background: #f0d0d0'|10 '''–'''
|align='center' style='background: #f0d0d0'|11 '''–'''
|align='center' style='background: #f0d0d0'|12 '''–'''
|align='center' style='background: #f0d0d0'|13 '''–'''
|align='center' style='background: #f0d0d0'|14 '''–'''
|align='center' style='background: #f0d0d0'|15 '''–'''
|align='center' style='background: #f0d0d0'|16 '''–'''
|align='center' style='background: #f0d0d0'|17 '''–'''
|align='center' style='background: #f0d0d0'|18 '''–'''
|align='center' style='background: #f0d0d0'|19 '''–'''
|align='center' style='background: #f0d0d0'|1A '''–'''
|align='center' style='background: #f0d0d0'|1B '''–'''
|align='center' style='background: #f0d0d0'|1C '''–'''
|align='center' style='background: #f0d0d0'|1D '''–'''
|align='center' style='background: #f0d0d0'|1E '''–'''
|align='center' style='background: #f0d0d0'|1F '''–'''

|-
!2…
|align='center'|20 ''' '''
|align='center'|21 '''!'''
|align='center'|22 '''"'''
|align='center'|23 '''#'''
|align='center'|24 '''$'''
|align='center'|25 '''%'''
|align='center'|26 '''&'''
|align='center'|27 '''''''
|align='center'|28 '''('''
|align='center'|29 ''')'''
|align='center'|2A '''*'''
|align='center'|2B '''+'''
|align='center'|2C ''','''
|align='center'|2D '''-'''
|align='center'|2E '''.'''
|align='center'|2F '''/'''

|-
!3…
|align='center'|30 '''0'''
|align='center'|31 '''1'''
|align='center'|32 '''2'''
|align='center'|33 '''3'''
|align='center'|34 '''4'''
|align='center'|35 '''5'''
|align='center'|36 '''6'''
|align='center'|37 '''7'''
|align='center'|38 '''8'''
|align='center'|39 '''9'''
|align='center'|3A ''':'''
|align='center'|3B ''';'''
|align='center'|3C '''<'''
|align='center'|3D '''='''
|align='center'|3E '''>'''
|align='center'|3F '''?'''

|-
!4…
|align='center'|40 '''@'''
|align='center'|41 '''A'''
|align='center'|42 '''B'''
|align='center'|43 '''C'''
|align='center'|44 '''D'''
|align='center'|45 '''E'''
|align='center'|46 '''F'''
|align='center'|47 '''G'''
|align='center'|48 '''H'''
|align='center'|49 '''I'''
|align='center'|4A '''J'''
|align='center'|4B '''K'''
|align='center'|4C '''L'''
|align='center'|4D '''M'''
|align='center'|4E '''N'''
|align='center'|4F '''O'''

|-
!5…
|align='center'|50 '''P'''
|align='center'|51 '''Q'''
|align='center'|52 '''R'''
|align='center'|53 '''S'''
|align='center'|54 '''T'''
|align='center'|55 '''U'''
|align='center'|56 '''V'''
|align='center'|57 '''W'''
|align='center'|58 '''X'''
|align='center'|59 '''Y'''
|align='center'|5A '''Z'''
|align='center'|5B '''['''
|align='center'|5C '''\'''
|align='center'|5D ''']'''
|align='center'|5E '''^'''
|align='center'|5F '''_'''

|-
!6…
|align='center'|60 '''`'''
|align='center'|61 '''a'''
|align='center'|62 '''b'''
|align='center'|63 '''c'''
|align='center'|64 '''d'''
|align='center'|65 '''e'''
|align='center'|66 '''f'''
|align='center'|67 '''g'''
|align='center'|68 '''h'''
|align='center'|69 '''i'''
|align='center'|6A '''j'''
|align='center'|6B '''k'''
|align='center'|6C '''l'''
|align='center'|6D '''m'''
|align='center'|6E '''n'''
|align='center'|6F '''o'''

|-
!7…
|align='center'|70 '''p'''
|align='center'|71 '''q'''
|align='center'|72 '''r'''
|align='center'|73 '''s'''
|align='center'|74 '''t'''
|align='center'|75 '''u'''
|align='center'|76 '''v'''
|align='center'|77 '''w'''
|align='center'|78 '''x'''
|align='center'|79 '''y'''
|align='center'|7A '''z'''
|align='center'|7B '''{'''
|align='center'|7C '''|'''
|align='center'|7D '''}'''
|align='center'|7E '''~'''
|align='center' style='background: #d0e0d0'|7F '''�'''

|-
!8…
|align='center' style='background: #c0ffc0'|80 '''Ň'''
|align='center' style='background: #c0ffc0'|81 '''Ś'''
|align='center' style='background: #c0ffc0'|82 '''Ć'''
|align='center' style='background: #c0ffc0'|83 '''Ż'''
|align='center' style='background: #c0ffc0'|84 '''Ź'''
|align='center' style='background: #c0ffc0'|85 '''Ŝ'''
|align='center' style='background: #c0ffc0'|86 '''Ĉ'''
|align='center' style='background: #c0ffc0'|87 '''Ẑ'''
|align='center' style='background: #c0ffc0'|88 '''Ô''' [note]
|align='center' style='background: #c0ffc0'|89 '''Ŕ'''
|align='center' style='background: #c0ffc0'|8A '''Ǔ'''
|align='center' style='background: #c0ffc0'|8B '''Ă'''
|align='center' style='background: #c0ffc0'|8C '''Ń'''
|align='center' style='background: #80f080'|8D '''Ș'''
|align='center' style='background: #80f080'|8E '''Ț'''
|align='center' style='background: #d0e0d0'|8F '''�'''

|-
!9…
|align='center' style='background: #80f080'|90 '''đ'''
|align='center' style='background: #c0ffc0'|91 '''ś'''
|align='center' style='background: #c0ffc0'|92 '''ć'''
|align='center' style='background: #c0ffc0'|93 '''ż'''
|align='center' style='background: #c0ffc0'|94 '''ź'''
|align='center' style='background: #c0ffc0'|95 '''ŝ'''
|align='center' style='background: #c0ffc0'|96 '''ĉ'''
|align='center' style='background: #c0ffc0'|97 '''ẑ'''
|align='center' style='background: #c0ffc0'|98 '''ô''' [note]
|align='center' style='background: #c0ffc0'|99 '''ŕ'''
|align='center' style='background: #c0ffc0'|9A '''ǔ'''
|align='center' style='background: #c0ffc0'|9B '''ă'''
|align='center' style='background: #c0ffc0'|9C '''ń'''
|align='center' style='background: #80f080'|9D '''ș'''
|align='center' style='background: #80f080'|9E '''ț'''
|align='center' style='background: #d0e0d0'|9F '''�'''

|-
!A…
|align='center'|A0 '''[https://secure.wikimedia.org/wikipedia/en/wiki/Non-breaking_space NBSP]'''
|align='center' style='background: #d0d0f0'|A1 '''ň''' (¡)'''
|align='center' style='background: #d0d0f0'|A2 '''Ű''' (¢)'''
|align='center' style='background: #d0d0f0'|A3 '''ě''' (£)'''
|align='center' style='background: #d0d0f0'|A4 '''ű''' (¤)'''
|align='center' style='background: #d0d0f0'|A5 '''Ě''' (¥)'''
|align='center' style='background: #d0d0f0'|A6 '''Š''' (¦)'''
|align='center'|A7 '''§'''
|align='center' style='background: #d0d0f0'|A8 '''š''' (¨)'''
|align='center' style='background: #d0d0f0'|A9 '''Ů''' (©)'''
|align='center' style='background: #d0d0f0'|AA '''Ą''' (ª)'''
|align='center' style='background: #d0d0f0'|AB '''Ę''' («)'''
|align='center' style='background: #d0d0f0'|AC '''Č''' (¬)'''
|align='center'|AD '''[https://secure.wikimedia.org/wikipedia/en/wiki/Soft_hyphen SHY]'''
|align='center' style='background: #d0d0f0'|AE '''č''' (®)'''
|align='center' style='background: #d0d0f0'|AF '''ů''' (¯)'''

|-
!B…
|align='center' style='background: #d0d0f0'|B0 '''Ő''' (°)'''
|align='center' style='background: #d0d0f0'|B1 '''Ł''' (±)'''
|align='center' style='background: #d0d0f0'|B2 '''Ť''' (²)'''
|align='center' style='background: #d0d0f0'|B3 '''Ď''' (³)'''
|align='center' style='background: #d0d0f0'|B4 '''Ž''' (´)'''
|align='center' style='background: #d0d0f0'|B5 '''ł''' (µ)'''
|align='center' style='background: #d0d0f0'|B6 '''ť''' (¶)'''
|align='center' style='background: #d0d0f0'|B7 '''ď''' (·)'''
|align='center' style='background: #d0d0f0'|B8 '''ž''' (¸)'''
|align='center' style='background: #d0d0f0'|B9 '''ő''' (¹)'''
|align='center' style='background: #d0d0f0'|BA '''ą''' (º)'''
|align='center' style='background: #d0d0f0'|BB '''ę''' (»)'''
|align='center' style='background: #d0d0f0'|BC '''Œ''' (¼)'''
|align='center' style='background: #d0d0f0'|BD '''œ''' (½)'''
|align='center' style='background: #d0d0f0'|BE '''Ÿ''' (¾)'''
|align='center'|BF '''¿'''

|-
!C…
|align='center'|C0 '''À'''
|align='center'|C1 '''Á'''
|align='center'|C2 '''Â'''
|align='center'|C3 '''Ã'''
|align='center'|C4 '''Ä'''
|align='center'|C5 '''Å'''
|align='center'|C6 '''Æ'''
|align='center'|C7 '''Ç'''
|align='center'|C8 '''È'''
|align='center'|C9 '''É'''
|align='center'|CA '''Ê'''
|align='center'|CB '''Ë'''
|align='center'|CC '''Ì'''
|align='center'|CD '''Í'''
|align='center'|CE '''Î'''
|align='center'|CF '''Ï'''

|-
!D…
|align='center'|D0 '''Ð'''
|align='center'|D1 '''Ñ'''
|align='center'|D2 '''Ò'''
|align='center'|D3 '''Ó'''
|align='center'|D4 '''Ô'''
|align='center'|D5 '''Õ'''
|align='center'|D6 '''Ö'''
|align='center' style='background: #d0d0f0'|D7 '''Ř''' (×)'''
|align='center'|D8 '''Ø'''
|align='center'|D9 '''Ù'''
|align='center'|DA '''Ú'''
|align='center'|DB '''Û'''
|align='center'|DC '''Ü'''
|align='center'|DD '''Ý'''
|align='center'|DE '''Þ'''
|align='center'|DF '''ß'''

|-
!E…
|align='center'|E0 '''à'''
|align='center'|E1 '''á'''
|align='center'|E2 '''â'''
|align='center'|E3 '''ã'''
|align='center'|E4 '''ä'''
|align='center'|E5 '''å'''
|align='center'|E6 '''æ'''
|align='center'|E7 '''ç'''
|align='center'|E8 '''è'''
|align='center'|E9 '''é'''
|align='center'|EA '''ê'''
|align='center'|EB '''ë'''
|align='center'|EC '''ì'''
|align='center'|ED '''í'''
|align='center'|EE '''î'''
|align='center'|EF '''ï'''

|-
!F…
|align='center'|F0 '''ð'''
|align='center'|F1 '''ñ'''
|align='center'|F2 '''ò'''
|align='center'|F3 '''ó'''
|align='center'|F4 '''ô'''
|align='center'|F5 '''õ'''
|align='center'|F6 '''ö'''
|align='center' style='background: #d0d0f0'|F7 '''ř''' (÷)'''
|align='center'|F8 '''ø'''
|align='center'|F9 '''ù'''
|align='center'|FA '''ú'''
|align='center'|FB '''û'''
|align='center'|FC '''ü'''
|align='center'|FD '''ý'''
|align='center'|FE '''þ'''
|align='center'|FF '''ÿ'''

|}

[note] As discussed [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/&do=findComment&comment=494855 here], the TDM char set has a redundant treatment of 2 characters:

* Ô appears at 0x88 and 0xD4
* ô appears at 0x98 and 0xF4

Planned for TDM 2.13, the redundancy can be removed and these new characters (from ISO-8859-3) introduced:

* Ğ appears at 0x88
* ğ appears at 0x98

For a mapping of these 256 codepoints to Unicode U+NNNN values and formal names, download 'TDM 8859-Style Font Map to Unicode-16.txt':

* [https://drive.google.com/file/d/1wLEtuFvnrZ-WHK8ign8i3diIXZdriZ4F/view?usp=sharing for TDM 2.13]
* [https://drive.google.com/file/d/1UAz9jSZpT_j33STP3So_Re8JWa8QuAmz/view?usp=sharing for TDM 2.12 and earlier]

Each of these files (by Geep, 2024) is in a standardized format so that it can also be imported into font design programs like FontForge as a custom 256-position map. In the comments, there is additional information about:
* ISO 8859-x sourcing of each character.
* alternative representations of some European and control characters.

=== Russian ===

Characters conform to the [https://en.wikipedia.org/wiki/Win-1251 WIN-1251 native encoding, shown in the Wikipedia article]. Exception: the character '''0xFF''' (я) is mapped to '''0xB6''' upon loading. Therefore any Russian font must contain я at the place 0xB6.

=== Asian Languages (Korean, Chinese, Japanese) ===

The original D3 had support for these languages, so it might be possible to add them to TDM, too. At the moment, however, we lack the fonts and translators. Also, writing from right-to-left (Hebrew) or top-down (Japanese) might be tricky or outright impossible in our GUI without more work in the C++ code. Plus, these languages use more than 256 different characters, and an 8 bit table will not hold these.

== European Character Implementation ==
=== Priority Early-On - the "Top 50" ===

Some of the special characters are used more often than others. Here is a statistic over the entire string set of the TDM core, from TDM v1.08, showing the top 50 most-used characters (excluding a-z, 0-9 and russian characters):

{|class="wikitable" border=1 style="border-collapse: collapse; font-size: 85%" cellspacing=0 cellpadding=2

|-
|Rank
|Occurances
|Letter
|Remarks
|Rank
|Occurances
|Letter
|Remarks

|-
|1
|í
|715
|
|25
|ć
|67
|

|-
|2
|é
|674
|
|26
|è
|65
|

|-
|3
|á
|524
|
|27
|ú
|56
|

|-
|4
|ø
|303
|Danish
|28
|ê
|52
|

|-
|5
|č
|288
|
|29
|ö
|48
|German

|-
|6
|ó
|283
|
|30
|É
|46
|

|-
|7
|ü
|270
|German
|31
|ñ
|37
|

|-
|8
|ł
|203
|Polish
|32
|õ
|32
|

|-
|9
|æ
|200
|Danish
|33
|ń
|26
|

|-
|10
|ě
|182
|
|34
|Ł
|24
|

|-
|11
|ř
|175
|Czech
|35
|Š
|21
|

|-
|12
|ã
|168
|
|36
|â
|21
|

|-
|13
|ž
|148
|Czech
|37
|ź
|20
|

|-
|14
|ý
|142
|
|38
|ß
|18
|German

|-
|15
|ę
|141
|
|39
|Ó
|18
|

|-
|16
|ą
|140
|
|40
|ň
|15
|

|-
|17
|ż
|119
|
|41
|Ú
|15
|

|-
|18
|å
|109
|Danish
|42
|Á
|13
|

|-
|19
|š
|99
|
|43
|î
|12
|

|-
|20
|ś
|97
|
|44
|ť
|11
|

|-
|21
|ç
|91
|
|45
|ô
|9
|

|-
|22
|ä
|86
|German
|46
|Ž
|8
|

|-
|23
|à
|83
|
|47
|Ż
|7
|

|-
|24
|ů
|77
|
|48
|Č
|7
|

|-
|25
|ć
|67
|
|49
|ù
|6
|

|}

Although ö, ä and ü do not appear that often, with only these and Ü, Ö, Ä and ß, the entire German language works. So adding these letters to the fonts is quite important.

Preferably, all foreign letters would be added to the fonts (see [[Font Patcher]] or [[Refont]]). However, if time permits only adding a few, '''í''' would be more important than, say, '''ô'''.

It is commonplace for missing accented letters to be redirected in the .dat file to the corresponding unaccented base letter.

=== Coverage Expansion & Remaining Limitations ===

By 2014, all the system (e.g., main menu) fonts (Carleton, Carleton_condensed, Stone in sizes 24pt and 48pt; Mason and Mason_glow in 48pt) had good coverage of the "Top 50" and beyond, although not all 256 codepoints.
This was confirmed more specifically in 2024, as part of an [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ Analysis of 2.12 Fonts]. This analysis indicated that, unlike the system fonts, the FM fonts generally did not provide specific glyphs beyond ASCII.

In 2024, Stone 24pt, important for subtitles and a number of readables, was further extended to cover all 256 codepoints. This was tested with TDM 2.13[betas] and released with 2.14. Similar Stone 48pt and Carleton 24pt improvements are planned for TDM 2.15.

[[Category:Fonts]]

{{i18n}}

Rebuild all lang Program

2026-01-21T19:29:52Z

Geep: Add ending tags

''by Geep, 2026''
==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use '''rebuild_all_lang'''. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
- deciding in advance which *.lang files are the current directory
- if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As an example, the program distribution includes a typical stock preamble.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are always processed in the order shown. If a particular *.lang is missing, that is handled quietly, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

==== Downloads ====
v 1.0 Source Code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code: a .cpp and .sln file]

Example preamble files (rename to preamble_utf8.txt to use):
* [https://drive.google.com/file/d/1bOWXv8Ju1DktaNYe2M8Bo-V7zAg2bNJ9/view?usp=sharing preamble_utf8[traditional<nowiki>]</nowiki>.txt]
* [https://drive.google.com/file/d/1QKJvG6FoAzuI7njO5fnOQCgwbXQdWqQe/view?usp=sharing preamble_utf8[modern for FM<nowiki>]</nowiki>.txt]
==== See Also ====
* [[Gen_Lang_Programs]]

{{i18n}} {{editing}}

Rebuild all lang Program

2026-01-21T19:27:08Z

Geep: Tweak "Downloads" & "See Also"

''by Geep, 2026''
==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use '''rebuild_all_lang'''. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
- deciding in advance which *.lang files are the current directory
- if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As an example, the program distribution includes a typical stock preamble.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are always processed in the order shown. If a particular *.lang is missing, that is handled quietly, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

==== Downloads ====
v 1.0 Source Code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code: a .cpp and .sln file]

Example preamble files (rename to preamble_utf8.txt to use):
* [https://drive.google.com/file/d/1bOWXv8Ju1DktaNYe2M8Bo-V7zAg2bNJ9/view?usp=sharing preamble_utf8[traditional<nowiki>]</nowiki>.txt]
* [https://drive.google.com/file/d/1QKJvG6FoAzuI7njO5fnOQCgwbXQdWqQe/view?usp=sharing preamble_utf8[modern for FM<nowiki>]</nowiki>.txt]
==== See Also ====
* [[Gen_Lang_Programs]]

Rebuild all lang Program

2026-01-21T19:19:33Z

Geep: /* Downloads */ added links

''by Geep, 2026''
==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use '''rebuild_all_lang'''. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
- deciding in advance which *.lang files are the current directory
- if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As an example, the program distribution includes a typical stock preamble.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are always processed in the order shown. If a particular *.lang is missing, that is handled quietly, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

==== Downloads ====
Source code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
* [https://drive.google.com/file/d/1kKunDZfVcolZgU4xMsYIpWc5VQgjd3-_/view?usp=sharing Windows executable rebuild_all_lang.exe]
* [https://drive.google.com/file/d/1sNbWZRznhmgUeGJR8GXK6uiMGAstTusY/view?usp=sharing rebuilt_all_lang.zip - source code: a .cpp and .sln file]

Example preamble files (rename to preamble_utf8.txt to use):
* [https://drive.google.com/file/d/1bOWXv8Ju1DktaNYe2M8Bo-V7zAg2bNJ9/view?usp=sharing preamble_utf8[traditional<nowiki>]</nowiki>.txt]
* [https://drive.google.com/file/d/1QKJvG6FoAzuI7njO5fnOQCgwbXQdWqQe/view?usp=sharing preamble_utf8[modern for FM<nowiki>]</nowiki>.txt]
==== See Also ====
[COMING SOON]

Rebuild all lang Program

2026-01-20T18:55:36Z

Geep: create article

''by Geep, 2026''
==== Introduction ====
Ordinarily, an FM that has been internationalized has a UTF8 "all.lang" text file, from which "*.lang" files in various 8-bit ISO-8859-x (or Windows) encodings were generated and distributed within the "strings" folder.

Sometimes, an attempt is made to improve translations, or extend them to additional languages. The ideal way is to alter content in all.lang, then regenerate the *.lang files (e.g., with gen_lang_plus). This regeneration could be for every language section specified in all.lang, or more selectively, just the ones for languages newly added or with changed content.

Suppose "all.lang" was not distributed with the FM. The original FM author may be able to provide it, or you can ask about it being in some TDM SVN repo.

If those routes fail, you can fabricate a replacement all.lang from the *.lang files. Note that the latter are stripped of helpful comments, including grouping header lines, often found in the original all.lang. So this approach is driven by necessity.

One way that fabrication can be done is first make a new partial all.lang from the existing english.lang file (which being in ASCII, can be viewed as also UTF8), Then add new language sections in UTF8 form. Regenerate just those *.lang files, while retaining older unchanged *.lang files. Each new UTF8 language section can be -
* typed from scratch, while consulting the ISO-8859-x version.
* possibly generated using some third-party ISO to UTF8 translator tool.

Or, as a convenience, use rebuild_all_lang. When run from within a give folder (e.g., "strings"), it will take every *.lang it finds (starting with english.lang), convert it from that language's ISO to UTF8, and basically glue them together into the required form.

==== Usage and Example ====
The Windows command-line program rebuild_all_lang is a simple C++ utility that takes no parameters, and if successful generates "all.lang" in the current directory. Control it by -
- deciding in advance which *.lang files are the current directory
- if you want the generated file to embed a preamble, you can provide it as a file named "preamble_utf8.txt" in the same directory. This is optional. As an example, the program distribution includes a typical stock preamble.

A typical output to console is:

Running rebuild_all_lang, v. 1.0
Note: Did not find an optional preamble file preamble_utf8.txt
Processing...
English
German
French
Italian
Spanish
Polish
Romanian
Russian
Portuguese
Czech
Hungarian
Slovak
Swedish
Danish
Dutch
Turkish
Catalan
Contents of *.lang files were converted successfully to UTF-8 and merged into new 'all.lang'.

Languages are always processed in the order shown. If a particular *.lang is missing, that is handled quietly, and that section is not added to all.lang.

At the start of all.lang, typical text is...

// Rebuilt all.lang file - created by rebuild_all_lang v. 1.0
// Strings in each section in same order as in *.lang, but converted from various iso-8859s encodings to UTF8.

// Note - did not find an optional file preamble_utf8.txt with preamble comments to place here.

{
[English]
// Sourced from english.lang of 2025-07-16T16:08:26+00:00, whose first line was...
// String table - english (iso-8859-1) - created on 2025-07-16T16:08:26Z with gen_lang_plus v. 1.1
"#str_fm_airpocket_xd_sheet_appointment_to_service_pg1__title" "Appointment to Service\n"
....

Within each language section - like [English] here - some variant of the first two commented lines are always generated. As you see, the "// Sourced from..." line is followed by the first line copied from the *.lang file. In this example, the comments show the external file modification datetime, and then the datetime written at generation time. The latter will not always be present, depending upon creation method.

==== Details ====
As required, the generated all.lang text file has Linux linebreaks (e.g., LF instead of CRFL), independent of linebreak form in input files *.lang or preamble_UTF8.txt

If all.lang already exists, you will see...

Running rebuild_all_lang, v. 1.0
File 'all.lang' already exists.
Continue and overwrite it? [y/n]

If you indicate "n" for "no", the program quits with message "Bye".

This program has some minimal file I/O and conversion error checking, with reporting to console. It is not thought necessary to do character-level checking of the UTF8 results against characters supported by TDM's character map... such checking is better done elsewhere. This is a simple program. (One could imagine a more elaborate version that would, say, allow english.lang to be annotated with comments (e.g., to group lines into subsections). And then reorder lines within the non-English language sections to match, embedding those same annotations.)

==== Download ====
[COMING SOON]
The distribution includes an example stock "preamble_utf8.txt" file.
Source code: This C++ v20 program was developed under Windows 11 with VS 2022 Community Edition (updated through 2025).
==== See Also ====
[COMING SOON]

Subtitles

2026-01-17T18:20:01Z

Geep: /* internationalization */ section added

Support for subtitles (or closed captions) was added in TDM 2.10.

=== What the player sees ===
In 2.10+, the player can control subtitle visualization under Settings/Audio/Subtitles, which takes one of these values:
* Story - display only story-related subtitles
* On - display subtitles for all speech
* Off - disable subtitles
"Story" is the default. If "On" is chosen, story and non-story speech subtitles (like for AI "barks") appear.

Each subtitle has an implicit or explicit start and end time relative to its audio clip. To handle multiple overlapping sounds, the text is not interwoven; instead, there are 3 separate, non-overlapping fields, stacked in the lower part of the screen. When a subtitle is shown, it is as white text on a translucent field-rectangle background. A text line that is too long is wrapped to two lines.

One immediate purpose for TDM's subtitles is to provide English captions for English audio. This will assist understanding of accented dialog for those who are hearing-impaired or whose English is less fluent. (In theory, subtitles could be relied upon to play an FM silently, but the lack of audio positional and distance cues, e.g., for footsteps, would be detrimental.)

As for translation, i.e., subtitles shown in other TDM-supported languages, the current system provides an important and usable building block. An FM author could employ it to provide "story" subtitles in a particular language other than English. Furthermore, a non-TDM site for a specific non-English audience - one that hosts full ports of TDM and its FMs - could create its own subtitle files in the target language. Other ideas for future capabilities, such as easy switching among multiple subtitle languages, are [https://forums.thedarkmod.com/index.php?/topic/21741-subtitles-possibilities-beyond-211/ under discussion].

=== Rollout so far ===
''As of 2.11''

The "Subtitles" setting affects the CVar "cv_tdm_subtitles", as follows:
* Story --> 1 --- show only story-relevant
* On --> 2 --- show all speech
* Off --> 0 --- hide all

There is an additional cv_tdm_subtitles value, 3. It means "show everything", including additional subtitles for sound effects. No core sound effect subtitles are provided yet.

Subtitles are not generated on the fly. They are pre-encoded into game files (as detailed below), and this will take human effort over time. "Story" subtitles are chiefly associated with custom sound files and typically the responsibility of the FM author. Conversely, subtitles for AI standard speech phrases (as well as standard sound effects) are seen as mainly core resources.

For the FMs included in the standard distribution, "Tears of Saint Lucia" (in 2.10) and "A New Job" (planned in 2.11) have "Story" speech English subtitles.

Non-story speech (as of 2.10) is limited to a dozen phrases of the "Cynic" voice, used by some attacking guards. Since this is part of the core distribution, these English subtitles become available to existing games automatically. [https://forums.thedarkmod.com/index.php?/topic/21740-english-subtitles-for-ai-barks/ A project to expand this coverage for 2.12] has been started.

See also the section "displaying text with extended duration (new for 2.12)" below.

=== How it works ===

Subtitles work on a very low level: where the engine manages sound samples and sound channels.
This approach allows to reliably add subtitles for sounds of any kind, but has some downsides too: the system does '''not''' know anything at all about entities, spawnargs, scripts, and other gameplay stuff.
The system allows to assign chunks of text directly to sound samples (which are usually .ogg and .wav files), so that when a sound sample is played, the assigned text is shown on the screen.

=== Subtitles decls ===

The assignment of text to sound samples is specified in a new type of decl files. These files must:
# be in '''subtitles''' directory,
# have '''.subs''' extension,
# contain declarations of type '''subtitles'''.
For comparison, materials are another type of decls, which must be in <tt>materials</tt> directory, in files with <tt>.mtr</tt> extension, and be of type <tt>material</tt>.

To avoid any conflicts between core game and missions, FM authors should follow the conventions:
# Names of subtitles decl files must start with '''fm_''', e.g.: <tt>fm_conversations.subs</tt>
# Names of declarations must start with '''fm_''' too, e.g.: <tt>fm_paul_intro_convo</tt>

Here is an example of <tt>fm_conversations.subs</tt> contents:

//note: comments work as they normally do

subtitles fm_intro_convo {
verbosity story
inline "sound/voices/paul/sound1.ogg" "My name is Paul, and I'm a city guard."
inline "sound/voices/bjorn/sound2.ogg" "Welcome to Anonymous City Guards! Call me Bjorn."
inline "sound/voices/paul/sound3.ogg" "I started cutting thieves' throats when I was 17..."
inline "sound/voices/paul/sound4.ogg" "... and now I cannot stop doing that!"

verbosity effect
inline "sound/voices/paul/sound5.ogg" "(starts weeping)"

verbosity story
inline "sound/voices/bjorn/sound6.ogg" "Calm down, Paul! We all were in your shoes!"
inline "sound/voices/bjorn/sound7.ogg" "Let us first watch an educational video about the harm from cutting throats."

srt "sound/voices/bjorn/sound8_long.ogg" "sound/voices/bjorn/sound8_long.srt"
}

subtitles fm_briefing {
verbosity story
srt "fromVideo video/sts_briefing_intro" "video/briefing/briefing.srt"
}

==== displayed text ====

There are two ways to specify subtitle text: inline and srt-based.

'''inline''' command is followed by a name of sound sample and the text to be displayed while it is being played.
In this case, the text is written straight in the decl file, and it is displayed for the whole duration of the sound sample.
This way is convenient for short sounds, which are the majority of sound samples, including phrases of a conversation. A "\n" can be embedded in the text to force a line break.

'''srt''' command is followed by paths to a sound sample and its .srt file, typically with matching filenames. This method, while more complicated than inline, has the flexibility to support a long sound file that needs multiple sequential subtitles, such as the sound sample of a briefing video. An .srt file, in "SubRip" format, is usually placed either with its sound file or in a "subtitles" folder. It contains a sequence of text messages to show during the sound sample, each with start and end HH:MM:SS,sss timestamps within the sample's timeline. For example:
1
00:00:03,612 --> 00:00:10,376
Bugger me!
Something's wrong with this crystal ball.

2
00:00:25,336 --> 00:00:28,167
Ah! Here we go.
Creating .srt files is best done with third-party AV software, not writing them manually. For TDM, files must be in engine-native encoding (internationalization is not supported yet anyway) and have no BOM mark. While implementations of the [https://en.wikipedia.org/wiki/Subrip#SubRip_file_format "SubRip" file format] often include HTML-style markups for font attributes like bold or italics, these are not possible for TDM. What TDM ''does'' have is limited primary-color font markup, applicable to inline and srt subtitles; see [[Text_Decals_for_Signs_etc.#Signs_with_Illuminated_Colored_Letters | Signs_with_Illuminated_Colored_Letters]] for details.

==== displaying text with extended duration (new for 2.12) ====

A TDM inline subtitle appears on screen at the start of its audio clip. By convention, this is true for srt as well; the first message starts at time 00:00:00,000.

Under 2.10/2.11, a subtitle goes away when its audio clip ends. For srt, this is true even if a timestamp specifies a value beyond the audio clip duration.

Starting with 2.12, both automatic and manual extensions are provided.

'''For inline subtitles.''' Automatically, durations are extended by 0.2 seconds past the clip end. This matches the 0.2 second pause automatically put in between each voice clip in the Conversation object, so as to show such subtitles without a visual gap. This degree of extension is considered imperceptable. In addition, 1.0 second is now the minimum duration of an inline subtitle, as widely recommended for captioning. (These extensions apply to all subtitles, not just Conversation.)

For any particular "inline" command, the 0.2 second value can be overridden (e.g., with a larger value) by an additional trailing parameter, with syntax of either:

-durationExtend ''<positive value>''
-dx ''<positive value>''

Example:
inline "sound/voices/bjorn/sound7.ogg" "Let us first watch an educational video about the harm from cutting throats." -dx 0.45
Manual extension may be needed with particularly fast talking (i.e., high presentation rate, as expressed in characters-per-second or words-per-minute). Conversely, to avoid irritating the reader with noticeably lagging subtitles, use manual extension sparingly. In particular, don't be keen to go longer than 1/2 second.

There is a sanity upper limit of 5.0 seconds on the extension. So overall, the calculated duration is:
duration = max(c + 0.2, 1.0) ''if x not defined''
duration = min(max(c + x, 1.0), c + 5.0) ''if x defined''
where c is the clip length, and x is the -dx value

'''For srt subtitles.''' The automatic extensions provided for inline do not apply to srt. Instead, the srt timestamp values can now extend beyond the clip end.

It is the srt author's responsibility to apply best-practices such as:
* Start the first message at time 0
* Avoid messages less than 1 second duration
* Try to cut messages between sentences, or at other vocal pauses
* Don't overlap messages in time
* Usually, messages are abutted; sometimes, significant time gaps are appropriate
* Avoid ending the last message much beyond the clip length (analogous to -dx considerations)
* While such a duration extension directly benefits the final message, it is sometimes possible to "distribute the benefit" to an earlier message by small shifts in message breaks.

==== internationalization ====

As mentioned above, a mechanism for support of translations, e.g., use of #str IDs, has not been developed. As of TDM 2.14 [alpha], one pre-requisite for European languages exists: build-out of the font used for subtitles, "english" Stone 24pt, to include complete coverage of all 256 characters in the TDM character map.

==== sound sample ====

In order to specify sound sample, write path to the file the same way you do e.g. in sound shaders.

One complicated case is when you have an [[Cutscene_video_with_FFmpeg#Play_video_with_sound|FFmpeg-based video with embedded sound]].
In this case write material name, prepended with '''fromVideo''' keyword and space, just like you do in the sound shader.
See <tt>fm_briefing</tt> in the example above.

==== verbosity ====

Currently there are three verbosity levels for subtitles:

* '''story''': This level should be applied to all story-related subtitles, everything that player should absolutely notice and understand.
* '''speech''': This is for subtitles of speech which is usually of little interest to the player, or is repeated too often. For instance, regular AI barks (e.g. "I'll get ya!") belong here.
* '''effect''': This level is reserved for non-speech subtitles which have little interest to the player, like "(bang)" or "(ding)", etc.

It is not yet clear how exactly these levels will be used in the long term. For now:

# By default, players only see <tt>story</tt>-level subtitles, although they can enable the other two categories in settings, or disable subtitles entirely.
# It is expected that almost all FM-specific subtitles get into <tt>story</tt> category. The other two exist mostly for core TDM sounds, which include a lot of AI barks (<tt>speech</tt> level) and trivial sound effects like arrow hit (<tt>effect</tt> level).

Every subtitles declaration must start with '''verbosity''' command.
This command applies to all the subsequent commands, until the other <tt>verbosity</tt> is met or declaration ends (whichever happens first).
For instance, all subtitles have <tt>story</tt> verbosity level in the example above, except for the "(starts weeping)" text, which has <tt>effect</tt> verbosity.

==== include ====

While the engine parses all declarations that are present in the files, the subtitles system only uses one subtitles decl named <tt>tdm_root</tt>. ''Include'' commands are used in order to put other subtitle decls into it. (The <tt>tdm_root</tt> decl is in file <tt>tdm_root.subs</tt>. As of TDM 2.11, this file, and all core non-game-specific tdm_*.subs, can be found in tdm_sound_vocals_decls01.pk4/subtitles/. An ''fm_root.subs'' file exists there, whose stub fm_root decl is over-ridden by your FM's fm_root as described next.)

'''include''' command has one argument: the name of the subtitles decl to be included. Note that it is the name of the decl, not the name of the file which contains it! Included decl can in turn include other decls.

FM-specific subtitles are always included from the decl named '''fm_root''', which must be in the '''fm_root.subs''' file. For instance, the contents of the <tt>fm_conversations.subs</tt> file from the example above will have no effect until we add the file <tt>fm_root.subs</tt> with contents:

/**
* This file should be overridden in order to provide mission-specific subtitles.
* When doing so, please follow the conventions:
* 1) The root decl is called "fm_root" and is located in file "fm_root.subs".
* 2) All other subtitle decls start with "fm_" prefix and are located in files starting with "fm_".
*/

subtitles fm_root {
include "fm_intro_convo"
include "fm_briefing"
include "fm_much_later" // points to the decl below
}

// Decls and their "inline"/"srt" commands can be in the fm_root.subs file, not just in other .subs files:
subtitles fm_much_later {
verbosity story
inline "sound/voices/paul/sound14.ogg" "Where did I put that knife sharpener?"
}

Notice that one decl can be used to do any number of subtitle assignments, so it is not necessary to have many decls and decl files. The easiest way to do FM-specific subtitles is to have one <tt>fm_root.subs</tt> file with one <tt>fm_root</tt> decl, and specify all your subtitles right there. Splitting subtitles across decls and files is entirely optional and can be used for grouping subtitles. It was added mainly for core TDM sounds: there are several thousands of them, so keeping all subtitles in a single file can become a headache quickly.

=== Output ===

The subtitles which should be displayed right now are provided to GUI code as '''gui::subtitleN''' variables, where N ranges from 0 to 3 (although only 0 to 2 will be made visible by the standard GUI scripting described next).
GUI scripts display these variables as non-overlapping text messages. Given a logical screen height of 480, each of the 3 fields is 45 units high. N = 0 is the lowest, at Y-origin 400; 1 at 350; 2 at 300.
The responsible code is located in <tt>tdm_subtitles_common.gui</tt> file, and is already used in several places:
* in the following states of the main menu: briefing, briefing video, and debriefing video
* in the always-on GUI overlay during gameplay

For example, here is how it is used in <tt>mainmenu_briefing.gui</tt>:

//note: each include of tdm_subtitles_common must have different name prefix!
#define SUBTITLES_NAMEPREFIX Briefing
#include "guis/tdm_subtitles_common.gui"

Usually, you don't need to do anything about it. However, this information can be relevant if you write some custom GUI and want to see subtitles there.

Note: it is not clear yet which kind of customization is necessary for the subtitles GUI.
If you have any ideas, please share them on [https://forums.thedarkmod.com/index.php?/topic/21741-subtitles-possibilities-beyond-211/ forums].

{{GUI}}

Mason Font

2025-10-21T22:05:06Z

Geep: Add new section: 2025 Improvements to Russian [Proposed for TDM 2.14]

''By Geep, 2024''

== Introduction ==
Mason is a highly stylized font found in the large headlines at the top of the main menu pages. In English these read as:
* The Dark Mod ''(on main page)''
* Missions List
* Online Mission Archive ''(with Online rendered smaller)''
* Download Status
* Mission Details ''(and subheadings Description and Screenshots)''
* Settings
* Load + Save

Headline characters are rendered in black, surrounded by a static golden-white "glow" (or a different effect if Russian). The styling features top-alignment of characters, achieved by giving capital letters a higher-than-normal baseline. As a TDM font, provided in English (including Latin/European) and Russian, only the 48pt size is supplied, not 24pt or 12pt. That's because this font is not positioned for general use in FMs.

The information here will be of most interest to anyone seeking to extend or manipulate this font. Historical context hopefully sheds light on current complexities.

== Origins ==
Back when the founders created TDM, the main menu headlines (including glow) were burnt into the parchment backgrounds of main pages. This was no doubt done with a font (likely Adobe's "Mason Alternate Serif") available within a graphics editor. And presumably with different page images for English and Russian, the two initial languages. [Please expand/refine this section if you were around then.]

== 2010-14 Era - A New Font to Support Translations ==
''Drawing from Springheel's [https://forums.thedarkmod.com/index.php?/topic/13825-menu-concerns-for-108/ "Menu concerns for 1.08"] and Tels' "Translating the TDM GUI" thread, particularly [https://forums.thedarkmod.com/index.php?/topic/12863-translating-the-tdm-gui/page/14/#findComment-274467 here].''

When it came time to expand main menu headlines of that time to European languages (an effort Springheel and Tels led in 2010-14), the original method was replaced by a more flexible one: blank parchment backgrounds overlaid by a text font, plus the glow.

=== TTF Manipulation and Conversion ===
The starting point for that font [one assumes] was some Mason-like TTF font. It was likely Jonathan Barnbrook's 1992 "Mason Alternate", which has no separately styled lower case characters; lower-case is just a smaller version of upper-case.

Doom-era tools existed to programmatically convert a TTF file into a starting version of a TDM font, namely, a DAT file and a set of TGA (and ultimately DDS) images. (12pt and 24pt sizes were also generated, but only 48pt was pursued.) Furthermore, Tels reported working with graphics apps GIMP and Paintshop, that work directly with TTF fonts as well as TGA image files.

=== Bitmap Tweaking and Character Set Extension ===
The 48pt font was then further adapted to "TDM style" (internally, "MasonAlternate"), with some characters morphed to look more like their burned-in TDM predecessors. Substantial additional bitmap editing added Cyrillic and the full panoply of accented Latin characters. The result was a duplicated set of (initially) seven 256x256 bitmaps that would cover English, European, and Russian languages. While the bitmaps were "shared" (by duplication - unlike the case with other TDM fonts), the metadata DAT files for English+European and for Russian had different content (i.e., diverging in their included glyphs and codepoint mapping), and were generated by separate [[Font Patcher]] scripts. (Subsequently, the sharing became partial; see "Further Changes..." below.)

=== Enlarging Upper-Case Character of the New Font ===
Font Patcher scripting, as it generates a new, modified DAT file from an old one, can be asked to apply a scaling factor to specific characters. This does not affect the bitmap glyphs in any way, but instructs the TDM engine to scale at render-time. (See elsewhere for more about scaling within DAT files.)

Recall the starting Mason TTF font had lower case-letters that were the same as upper-case, except for size difference. But that difference seemed insufficient. After experimentation, it was decided to make the upper-case letter nominally 20% larger, by applying a scaling factor of 1.20 to them. (Actual scaling ratios recalculated from resulting DAT values will vary due to rounding-to-integer quantification.)
This was applied to A-Z, as well as accented uppercase characters at codepoints 0x80-84, a2, a5, a6, a9-ac, b0, b2-b4, bc, be, and c0-dd.

=== Finalizing Russian ===
At this point, the /russian/ versions of the seven bitmap shaders were frozen, and remain unchanged to the current day (unless DarkFate independently has made further alterations; beyond our scope here). [There was a 2021 touch of all font file dates... was anything changed then?]

While a Russian "mason_glow" font was created and shipped, it is merely a placeholder: an exact duplicate of "mason", in regard to the content of their DAT files and their sets of DDS files. Instead, the Russian deployment provides a substitute visual effect by rendering each character twice.

=== Further Changes to English+European Mason During 2010-14 ===
==== Bitmap Consolation ====
While /russian/ kept 7 bitmaps, as discussed, the /english/ set was compacted to 6, by combining characters from Masonalternate_5_48.dds and Masonalternate_6_48.dds into the former alone. The DAT file was appropriated adjusted.

Be aware that the compacted "english" set and the uncompacted "russian" set still encompass the same character glyphs; as before, their DAT files differ in what they expose.

==== Introducing Glow ====
TDM's Mason (as well as Carleton) supports a "glow" around each character. As deployed with the /english/ version of Mason, the glow is a static yellowish-white halo around each black character; it does not vary with mouse-over. Indeed, TDM's Mason headlines (as opposed to Carleton) are not mouse-selectable. The glow is not done with special engine coding, but rather by using the GUI system and separate bitmap sets for the base characters ("mason") and their glows ("mason_glow"). Each pair of base and glow characters will be aligned at render time, so must have compatible DAT metrics. (More about this below in a modern context.)

To accommodate the extra spacing needed for a glow, some glyphs were moved further away from a shader edge, where they had been placed by the programmatic conversion. Significant adjustments for glow spacing:
* Masonalternate_0_48.dds. No change.
* Masonalternate_1_48.dds. Top row english moved down.
* Masonalternate_2_48.dds. Top row english moved down.
* Masonalternate_3_48.dds. No change.
* Masonalternate_4_48.dds. Top row accented moved down. Left column accented moved right.
* Masonalternate_5_48.dds. (combines russian 5 & 6) Top row accented moved down.

The general GIMP method of creating a glow font bitmap from a copy of a base font bitmap is described
[https://forums.thedarkmod.com/index.php?/topic/12863-translating-the-tdm-gui/page/5/#findComment-262661 here].

This results in paired base and glow bitmaps having the same character positioning, which while not required, is best practice and simplifies understanding when viewing in GIMP.

Be aware that though paired DDS files in /english/mason/ and /english/mason_glow/ have identical filenames, the content is different. For example, mason/masonalternative_0_48.dds and mason_glow/masonalternative_0_48.dds have respectively base content and glow content.

==== Resulting Bitmaps within GIMP, as Source ====
This discusses masonalternate_48.xcf, available...
* with filedate of May 26, 2012, unversioned in TDM SVN
* with filedate of March 28, 2014, within Tels' [http://bloodgate.com/mirrors/tdm/pub/tdm_i18n.zip bloodgate archive of translated FMs]. Font Patcher scripts (as .txt files) are found here too.

All 12 /english/ bitmaps are found as separate layers within a GIMP .xcf (project) file. The base character layers are called simply "0" to "5". The glow characters are called "New Layer", "New Layer #2", and so on. (Unfortunately, in this .xcf, they are unhelpfully misnumbered. Just know that a "New Layer" pairs with the base numbered layer immediately below it.) In addition, the early base seventh bitmap is represented here by the extra "Pasted Layer", whose characters (for /english/ not /russian/) were subsequently copied into layer 5.

So, as layers are individually exported as DDS files, they must be manually distributed appropriately:
* "New Layer" above "5" ==> mason_glow/fontimage_5_48.dds
* "5" ==> mason/fontimage_5_48.dds
* "New Layer" above "4" ==> mason_glow/fontimage_4_48.dds
* "4" ==> mason/fontimage_4_48.dds
* "New Layer" above "3" ==> mason_glow/fontimage_3_48.dds
* "3" ==> mason/fontimage_3_48.dds
* "New Layer" above "2" ==> mason_glow/fontimage_2_48.dds
* "2" ==> mason/fontimage_2_48.dds
* "New Layer" above "1" ==> mason_glow/fontimage_1_48.dds
* "1" ==> mason/fontimage_1_48.dds
* "New Layer" above "0" ==> mason_glow/fontimage_0_48.dds
* "0" ==> mason/fontimage_0_48.dds

==== Deployment and European String Support ====
Regarding Mason, it was the plan to provide translated text and GUI methodology to all the main menu headlings, e.g., "Settings", "Load/Save", and "Mission Archive". [There are indications this was only realized for some headlines, and finally completed during the 2017-18 update.]
== 2017-18 Era - Improving English Appearance ==
''Adapted from Springheel's thread [https://forums.thedarkmod.com/index.php?/topic/19129-menu-update/#comment-412921 "Menu Update"].''

As time went on, it was observed that the 48pt-sized characters used in Mason, while of adequate crispness for the screen resolutions of Doom and early TDM, were showing their age. In 2017-8, Springheel undertook a revision to fix this. This effort focused on just the English alphabetic characters In the /english/ set. The method involved:
* Doubling the size of selected bitmap shaders from 256 x 256 to 512 x 512. This involved enlarging the bitmap image, so doubling the size of every glyph contained within (since the goal here was not to pack in more glyphs per bitmap.) Fortunately, the size the characters appearance in-game is unchanged, if the DAT file metrics are unaltered.
* Within the doubled shaders, selecting and modifying individual glyphs to make them render more crisply.

=== Selective Doubling of "Mason" ===
An analysis showed that only Mason's first three shader bitmaps contained the characters of interest.
So expeditiously, only those were doubled to 512x512. This doubled the size of glyphs of all characters within, including those not of improvement interest. (By not doubling all six bitmaps, the resulting mixed-bitmap dimensions in the set has drawbacks for GIMP layering and tool usage, discussed later.)

In spite of this doubling, the characters must not change in size when rendered. This is done by keeping the DAT's metrics – particularly imageHeight and imageWidth - unchanged, as well as floating-point boundaries s, t, s2, and t2 expressed as 0..1, but now within the doubled 512 x 512 size.

Expressed in REF terms, the values of imageHeight and imageWidth are unchanged from their 256 x 256 values, but coord_s, coord_t, coord_s2, and coord_t2 integers are doubled. So, as integer coordinates, this may be viewed as applying a 0.50 scaling factor – or for upper-case ASCII characters, composing 0.50 and 1.20 scaling factors.

=== Selective Doubling of "Mason_glow" ===
Surprisingly, of the corresponding 3 glow bitmaps, only 2 of them were doubled. There are pluses and minuses to doubling glow bitmaps. The main argument "for" is that it permits easy visualization as layers in apps like GIMP. The main argument "against" is that, since the glows are supposed to be fuzzy, they don't really need to be sharpened.

In any event, the base and glow don't require the same size shader for proper alignment. The specific requirement is that the following must match for each base and glow character in their respective DAT files:
* horizontal border dimension s2-s, expressed as a floating point value 0...1
* vertical border dimension t2-t, expressed as a floating point value 0...1
* All other numeric metadata.

If border dimensions are expressed as pixel integer in REF terms, this is more complicated, as each border must reflect an appropriate range of 0...256 or 0...512, depending on its bitmap shader.

=== Black Borders Appear ===
In general, TDM fonts represent every glyph as pure white (RGB 1,1,1), delineated only by alpha channel edges. This was true for mason within the previous-era's /english/ and /russian/ DDS files.

But in the 3 /english/ base mason dds files that were doubled, every character somehow got a ragged black border (when viewed in GIMP). Specifically, these "black border" glyphs are gray-scale (or maybe black/white dithered) with smudgy white interiors. Perhaps that doesn't matter so much when game-rendered in black, but still, unhelpful.

=== Selective Sharpening/Redrawing ===
Finally, the bitmap glyphs were editing to make crisper edges. This was limited to "regular keyboard characters", not "alternative language characters" (e.g., accented or Cyrillic). For these characters, the "black borders" problems was overwritten: the black borders were minimized (though not eliminated), and the interiors were cleaned to be pure white. [Done with flood fill?]. But the problem remains for other characters.

=== Coverage ===
Coverage was completed so that all headlines (e.g., Settings, Load/Save, Mission Archive) were rendered translated into European languages. But only the unaccented English characters were uniformly improved.

=== Resulting Bitmaps as Files ===
To summarize the /english/ base results...

Now of size 512x512 (342 KB), and containing (though not exclusively) redrawn English characters:
* Masonalternate_0_48.dds
* Masonalternate_1_48.dds
* Masonalternate_2_48.dds

Of size 256 x 256 (86 KB), and not containing English characters, so no redrawing:
* Masonalternate_3_48.dds
* Masonalternate_4_48.dds
* Masonalternate_5_48.dds

== 2025 Improvements to Russian [Proposed for TDM 2.14] ==
''For more about this project by kalinovka and Geep, including motivations, approaches, and resulting screenshots, test FMs and details of individual characters,, see the "Font localization" thread, particularly this [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization/#comment-501602 this post with links and screenshots].''

=== Coverage, Appearance, and Bitmaps ===

Mason /russian/ was missing a number of alphabetic glyphs for TDM-defined Cyrillic characters. This upgrade corrected that, so that Mason can be used not just for the main menu system, but in-game (albeit with still only 48pt distributed). When easy to do, some characters also got crisper glyph rendering or improved spacing. (Some characters will remain soft-edged or not-ideally spaced.)

The main approach was to first replace the /russian/ Mason DDS set with the TDM 2.13 /english/ Mason set (with corresponding DAT file changes), then - as fill-in - add newly-generated DDS from a "Lora Regular" font. While less stylish, Lora (in adjusted point sizes) is adequate to "pass" as Mason 48pt. The original "Mason Alternate" TTF source did not include Cyrillic characters.

Specifically, the DDS set, distributed under "tdm_fonts01/dds/fonts/russian/mason", became as follows.
Copied from the TDM 2.13 /english/ set (replacing the 2.13 /russian/ set), with Jan. 17, 2021 file dates:

* masonalternate_0_48.dds
* masonalternate_1_48.dds
* masonalternate_2_48.dds
* masonalternate_3_48.dds
* masonalternate_4_48.dds
* masonalternate_5_48.dds

Newly generated lower-case and symbols for /russian/, with 2025 file dates:

* lora45_as_mason_6_48.dds
* lora45_as_mason_7_48.dds

Newly generated upper-case for /russian/, with 2025 file dates:

* lora54_as_mason_8_48.dds
* lora54_as_mason_9_48.dds
* lora54_as_mason_10_48.dds

=== Revised DAT File ===
The contents of several DAT files were hand-merged, to reach the final DAT that mixes shaders for "Mason Alternate" & "Lora Regular" fonts.

At codepoints 0x00-7f, the TDM 2.13 Mason ''/english/'' ASCII characters appear. These render crisply, due to previous bitmap improvements: doubling of bitmap size, followed by glyph redrawing to limit anti-aliasing.

At codepoints 0x80-ff, for Cyrillic, there's a mix of:

* Cyrillic characters already existing in the Mason /english/ set (some the same glyphs as ASCII, a few the same as European, some previously hand-drawn)
* New non-capitals generated from Lora 45pt (matching the height of Mason 48pt).
* New capitals generated from Lora 54pt. Because these were generated at a larger point size, they didn’t require additional 120% DAT scaling (with its added edge-softening) to reach the target size. But they did need DAT adjustments to be top-aligned in the TDM-specific style for this font.

A few characters got additional special treatments.

=== Mason_glow ===
Finally, the /russian/ mason_glow font was also updated with a clone of these updated /russian/ Mason resources. (This differs from how /english/ glows are implemented.)

== Working with Mason Today ==
=== Improved Tools to Generate Bitmaps from TTF and to View Font Coverage ===
The 2025 Improvements to Russian included:

* Creating the [[ExportUnicodeFontToDoom3]] utility, used to generate the new Lora DDS and DAT files.
* Deploying several test FMs. The final one, testMasonLora3Way.pk4, displays the entirety of the proposed 2.14 /russian/mason font. (This is available via the "Font localization" link above. It is trivial to clone and adapt this FM to show comparable coverage for other /russian/ or /english/ fonts.)

=== Drawbacks of Mixed-Size Bitmaps ===
==== With GIMP ====
While you can view multiple images in GIMP, if they are of different sizes, a view of them in aligned layers becomes impractical. This weighs against the use of a single .xcf file to serve as overall source for the set. Instead, individual TGA images make more sense. These can still be edited one at a time in GIMP.

==== With [[Refont]] and [[datBounds]] ====

Refont v2.5 and datBounds v1.6 now supports mixed-sized bitmaps. See in particular [[Refont#Mason Family]].

=== How Mason's "Glow" is Implemented ===
A typical usage is the mainmenu_download.gui, where two representations of the current string are overlaid in an aligned manner. One is the text, rendered as opaque black (using RGBA 0,0,0,1). The other is the glow, rendered in pale yellow with some transparency (RGBA 1,1,0.80,0.8). The glow source already has some transparency, so this is even more. For a given character, the glow bitmap has wider and more diffuse strokes than the base font. A link about glow bitmap creation is above.

Here's one representative gui code snippet, showing placement of the words "Mission Archive" (or it's non-English equivalent), on a parchment background:

...
// Mission Archive Headline Translation
windowDef MissionArchiveHeadlineTextGlow
{
rect 10, 15 + "gui::headline_offset", 300, 100
text "#str_menu_mission_archive" // Mission Archive
font "fonts/mason_glow"
forecolor 1,1,0.80,0.8
textscale 0.65
textalign 1
textaligny -1
visible 1
}
windowDef MissionArchiveHeadlineText
{
rect 10, 15 + "gui::headline_offset", 300, 100
text "#str_menu_mission_archive" // Mission Archive
font "fonts/mason"
forecolor 0,0,0,1.0
textscale 0.65
textalign 1
visible 1
}
...

Observe that the similarity of rect, textscale, and textalign values keeps the two overlays exactly aligned, except for a 1 pixel vertical offset given by textaligny -1.

(Additional GUI event handling, not shown, allows the paired textscale values to be reduced to accommodate a more-verbose European language.)

{{tutorial}} [[Category:Fonts]]

ExportUnicodeFontToDoom3

2025-09-11T21:35:55Z

Geep: Add release 1 links

ExportUnicodeFontToDoom3

2025-09-10T20:06:33Z

Geep: /* Changed Behavior */

''By Geep, 2025 [WORK IN PROGRESS]''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints].
As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== (hirez - another possible argument) ===
''Geep: This is a future idea, but no promises on this.''

Change code to allow export of 512x512 bitmaps with larger, thus crisper glyphs. Specifically, add a new command-line option, -hirez, that would
* output 512x512 TGAs and their DAT.
* internally, double the character glyph size; so if your requested 48pt, it would make 96pt glyphs. (This is more or less how /english/ 512x512 treats its hand-made glyphs.)
* make whatever other layout & metric adjustments needed. The imageHeight and imageWidth metrics will be as if the requested 48pt size was generated, so in effect scaling down by a factor of 2 from the 96pt glyph.

== Changed Behavior ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using or revising the -name argument.

== Downloads ==
[links to do]

Windows binary and required archived versions of FreeType and DevIL DLLs:

ExportUnicodeFontToDoom3 Release 1 of 09-10-2025.zip

The C++ project file with source code:

ExportUnicodeFontToDoom3_source_project.zip Release 1 of 09-10-2025. This zip is ... MB, includes source & build project of archived library.

The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

ExportUnicodeFontToDoom3

2025-09-10T20:05:07Z

Geep: /* Downloads */

''By Geep, 2025 [WORK IN PROGRESS]''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints].
As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== (hirez - another possible argument) ===
''Geep: This is a future idea, but no promises on this.''

Change code to allow export of 512x512 bitmaps with larger, thus crisper glyphs. Specifically, add a new command-line option, -hirez, that would
* output 512x512 TGAs and their DAT.
* internally, double the character glyph size; so if your requested 48pt, it would make 96pt glyphs. (This is more or less how /english/ 512x512 treats its hand-made glyphs.)
* make whatever other layout & metric adjustments needed. The imageHeight and imageWidth metrics will be as if the requested 48pt size was generated, so in effect scaling down by a factor of 2 from the 96pt glyph.

== Changed Behavior ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using the -name argument.

== Downloads ==
[links to do]

Windows binary and required archived versions of FreeType and DevIL DLLs:

ExportUnicodeFontToDoom3 Release 1 of 09-10-2025.zip

The C++ project file with source code:

ExportUnicodeFontToDoom3_source_project.zip Release 1 of 09-10-2025. This zip is ... MB, includes source & build project of archived library.

The Cyrillic coverage of Lora Regular TTF font, projected to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

ExportUnicodeFontToDoom3

2025-09-10T20:04:47Z

Geep: /* Downloads */

''By Geep, 2025 [WORK IN PROGRESS]''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints].
As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== (hirez - another possible argument) ===
''Geep: This is a future idea, but no promises on this.''

Change code to allow export of 512x512 bitmaps with larger, thus crisper glyphs. Specifically, add a new command-line option, -hirez, that would
* output 512x512 TGAs and their DAT.
* internally, double the character glyph size; so if your requested 48pt, it would make 96pt glyphs. (This is more or less how /english/ 512x512 treats its hand-made glyphs.)
* make whatever other layout & metric adjustments needed. The imageHeight and imageWidth metrics will be as if the requested 48pt size was generated, so in effect scaling down by a factor of 2 from the 96pt glyph.

== Changed Behavior ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using the -name argument.

== Downloads ==
[links to do]

Windows binary and required archived versions of FreeType and DevIL DLLs:

ExportUnicodeFontToDoom3 Release 1 of 09-10-2025.zip

The C++ project file with source code:

ExportUnicodeFontToDoom3_source_project.zip Release 1 of 09-10-2025. This zip is ... MB, includes source & build project of archived library.

The Cyrillic coverage of Lora Regular TTF font, project to used as Russian Mason supplement in TDM 2.14, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

ExportUnicodeFontToDoom3

2025-09-10T20:02:44Z

Geep: Add Changed Behavior, Downloads [draft] sections

''By Geep, 2025 [WORK IN PROGRESS]''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints].
As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is good because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

=== (hirez - another possible argument) ===
''Geep: This is a future idea, but no promises on this.''

Change code to allow export of 512x512 bitmaps with larger, thus crisper glyphs. Specifically, add a new command-line option, -hirez, that would
* output 512x512 TGAs and their DAT.
* internally, double the character glyph size; so if your requested 48pt, it would make 96pt glyphs. (This is more or less how /english/ 512x512 treats its hand-made glyphs.)
* make whatever other layout & metric adjustments needed. The imageHeight and imageWidth metrics will be as if the requested 48pt size was generated, so in effect scaling down by a factor of 2 from the 96pt glyph.

== Changed Behavior ==
In ExportFontToDoom3All256, if the font title was more that 20 characters long, it would truncate it to fit in the DAT file as part of the shader names (i.e., paths to TGA files). But this threshold was wrong. Now, truncation and a warning happens if the font title is more that 17 characters long (or 16 if generated TGAs include any with a 2-digit count instead of 1-digit). If truncation occurs, the user is asked to consider using the -name argument.

== Downloads ==
[links to do]

Windows binary and required archived versions of FreeType and DevIL DLLs:

ExportUnicodeFontToDoom3 Release 1 of 09-10-2025.zip

The C++ project file with source code:

ExportUnicodeFontToDoom3_source_project.zip Release 1 of 09-10-2025. This zip is ... MB, includes source & build project of archived library.

The Cyrillic coverage of Lora Regular TTF font, used as Russian Mason supplement, may be viewed and font downloaded [https://www.fontsquirrel.com/fonts/lora?filter%5Bclassifications%5D%5B0%5D=serif&filter%5Blanguages%5D%5B0%5D=cyrillic#eula here].

ExportFontToDoom3

2025-09-10T19:48:47Z

Geep: Add See Also: ExportUnicodeFontToDoom3

== Intro to ExportFontToDoom3 & ExportFontToDoom3All256 ==
The Windows command-line utility "ExportFontToDoom3" was created by Australian modder Grant Davies in 2005 to get TrueType Fonts, standard on Windows, into the metadata+bitmaps form needed by the Doom 3 engine. His final version, 1.02, was released as open-source and substantially used for TDM. It relies on the open-source DevIL and Freetype2 libraries.

'''ExportFontToDoom3''' exports only the bitmap glyphs assigned to codepoints in the range 0-127, routinely the ASCII characters. The output actually generates metadata for 256 characters, but, due to a bug, upper-half characters are just mapped to the "undefined character" glyph.

'''ExportFontToDoom3All256''' fixes that bug; it exports the glyphs assigned to codepoints in the range 0-255, routinely ANSI characters. It is also easier to run on modern Windows PCs, so recommended. It takes the same command-line arguments as ExportFontToDoom3. Credits to Crispy (for 2009 fix) and (in 2024 form) Geep.

'''See Also: successor project [[ExportUnicodeFontToDoom3]]''', that can export Cyrillic and other Unicode characters, and adds further useful options.

== Using the Tool ==
''The information here is adapted from the description doc included with the ExportFontToDoom3 download.''
=== Basic Command ===
To export with the common settings of 12, 24, and 48 point fonts, simply type:
ExportFontToDoom3All256 ''font_filename''
For example (and assuming here a font "badtimes" with no legal issues):
ExportFontToDoom3All256 badtimes.ttf
Most standard font formats (from the 2009 timeframe) can be used for export, but TrueType is most common.
=== Size Option ===
To export the font in a certain size, use the option:
-size ''font_pointsize''
For Doom3/TDM, only the sizes 12, 24, and 48 directly produce valid end-products. More than one "-size" arguments may be specified, for example:
ExportFontToDoom3All256 badtimes.ttf -size 12 -size 24
As indicated above, if no "-size" argument is specified, 12, 24, and 48 point fonts are exported.

The set of sizes in a particular generation run will jointly determine the "glyphScale" value specified at the end of bottom of a DAT/REF file, as follows:

glyphScale[for size n] = largest-size-in-set / size-n

So, depending on how you generate, you may have to hand-adjust the glyphScale later.

It is possible to specify one or more non-standard sizes for special processing, e.g., to do TTF-to-DAT overall scaling. To use in TDM, this will need subsequent file renaming to one of the 3 valid sizes, and glyphScale correction.

=== Name Option ===
To (re)name the font to be exported, use the option:
-name ''fontname''
By default, the fontname is instead extracted from the font file. In either case, the fontname string will be truncated if too long. It then begins each of the TGA filename(s), which are referenced in turn in the per-character "shaderName" fields of the DAT file. The shaderName field in a DAT file can have at most 32 characters; when viewed as a REF or FNT file, it has this template:
shaderName fontname_<1- or 2-digit count>_<2-digit fontsize>.tga

''TIP: If the default fontname is more than 16 characters, it is recommended you use the -name option, to prevent ugly truncation or (corrected in ExportUnicodeFontToDoom3) aborts.''

=== Other Options ===
In the description doc, there is a great deal of technical discussion about these next options, but of little likely relevance to TDM nowadays. Just ignore them in favor of their default values.
-textureFormat ''format''
where the default is .tga, which is what you want. Use another tool to convert the output from .tga to .dds, since ExportFontToDoom3 won't do it correctly.
-xOffsetFix ''type''
where the default is "none", which causes the font's X offset value to be exported to the "pitch" member of TDM .dat file. There is also the related:
-noXOffsetWarnings

=== Output ===
A new folder will be created and all output files will be stored there. The folder will be named according to the font being exported.
The output will consist of Targa image (.tga) files and .dat files.

Afterwards, from each TGA, you should create a corresponding DDS file.

== Deployment & Further Adaptions ==
After export, assuming you have an FM set up to test them, rename the files into standard form and move or copy them to the expected locations...
* For DAT files: fonts/english/<your font name>/
* For DDS files: dds/english/<your font name>/
You'll undoubtedly have to make further iterative adjustments to both the DAT files (using other methods listed in [[Font Conversion & Repair]]) and bitmap files.
For instance, the "glyphScale" value will likely have to be changed.

== Tutorials that Include ExportFontToDoom3 ==
Doom3-era tutorials, once posted on www.doom3world, can still be found with the "wayback machine" links given here:
* [https://web.archive.org/web/20081028224423/http://www.doom3world.org/phpbb2/viewtopic.php?t=11923 Tutorial 10 - Using custom fonts]
* [https://web.archive.org/web/20100522173219/http://www.doom3world.org/phpbb2/viewtopic.php?t=11924 Tutorial 11 - Editing fonts]

== Downloads of ExportFontToDoom3All256 ==
Windows binary and required archived versions of FreeType and DevIL DLLs:
* [https://drive.google.com/file/d/1HzyyuZv0AVEhw4czJEwQ4nDMuqCogRyi/view?usp=sharing 'ExportFontToDoom3All256 Release 1 of 04-26-2024.zip']

This version, having been rebuilt, does not require legacy msvcp71.dll & msvcr71.dll files needed to run ExportFontToDoom3 v 1.02.

The C++ project file with source code:
* [https://drive.google.com/file/d/1nkXJpBPa8qDqQxVQd9c1HlsSMKgohIw-/view?usp=sharing ExportFontToDoom3All256_source_project.zip] Release 1 of 04-26-2024. This zip is 177 MB; includes source & build project of archived library. Also sample TTF font (distributed with TDM) for debug testing.

For more about the modification and rebuild process, see "ReadMe_about_All256.txt", included in either of these two .zip downloads.

== Downloads of ExportFontToDoom3 v 1.02 ==
Windows binary:
* [https://grantheant.com/products/exportfonttodoom3 Grant Davie's project page]

Required DLLs:
To use ExportFontToDoom3 v 1.02 binary, you must have msvcp71.dll & msvcr71.dll installed in Windows or (more practical and safe now) in the same folder as the .exe. Once common, these files are no longer available from Microsoft (and purged from GitHub projects too). If you have an old machine with pre-Win10 OS, you can probably copy them from there. Otherwise, TDM's ATI Compressonator zip has them, at [[DDS Creation with ATI Compressonator]]; or various sketchy dll sites.

The C++ project file with source code:
* [https://grantheant.com/products/exportfonttodoom3 Grant Davie's project page]
* [https://www.moddb.com/downloads/export-font-to-doom-3-v102 Moddb downloads page]
* also from wayback machine.
The project makes use of the circa-2005 FreeType font library (www.freetype.org) and DevIL image library.

== See Also: [[ExportUnicodeFontToDoom3]] ==
This project, the successor to ExportFontToDoom3All256, can export Cyrillic and other Unicode characters, and adds further useful options.

{{tutorial}} [[Category:Fonts]]

ExportFontToDoom3

2025-09-10T19:42:27Z

Geep: /* Intro to ExportFontToDoom3 & ExportFontToDoom3All256 */ Add link to ExportUnicdoeFontToDoom3

ExportFontToDoom3

2025-09-10T19:34:41Z

Geep: /* Name Option */ Mention truncation, show template, add tip

== Intro to ExportFontToDoom3 & ExportFontToDoom3All256 ==
The Windows command-line utility "ExportFontToDoom3" was created by Australian modder Grant Davies in 2005 to get TrueType Fonts, standard on Windows, into the metadata+bitmaps form needed by the Doom 3 engine. His final version, 1.02, was released as open-source and substantially used for TDM. It relies on the open-source DevIL and Freetype2 libraries.

'''ExportFontToDoom3''' exports only the bitmap glyphs assigned to codepoints in the range 0-127, routinely the ASCII characters. The output actually generates metadata for 256 characters, but, due to a bug, upper-half characters are just mapped to the "undefined character" glyph.

'''ExportFontToDoom3All256''' fixes that bug; it exports the glyphs assigned to codepoints in the range 0-255, routinely ANSI characters. It is also easier to run on modern Windows PCs, so recommended. It takes the same command-line arguments as ExportFontToDoom3. Credits to Crispy (for 2009 fix) and (in 2024 form) Geep.

== Using the Tool ==
''The information here is adapted from the description doc included with the ExportFontToDoom3 download.''
=== Basic Command ===
To export with the common settings of 12, 24, and 48 point fonts, simply type:
ExportFontToDoom3All256 ''font_filename''
For example (and assuming here a font "badtimes" with no legal issues):
ExportFontToDoom3All256 badtimes.ttf
Most standard font formats (from the 2009 timeframe) can be used for export, but TrueType is most common.
=== Size Option ===
To export the font in a certain size, use the option:
-size ''font_pointsize''
For Doom3/TDM, only the sizes 12, 24, and 48 directly produce valid end-products. More than one "-size" arguments may be specified, for example:
ExportFontToDoom3All256 badtimes.ttf -size 12 -size 24
As indicated above, if no "-size" argument is specified, 12, 24, and 48 point fonts are exported.

The set of sizes in a particular generation run will jointly determine the "glyphScale" value specified at the end of bottom of a DAT/REF file, as follows:

glyphScale[for size n] = largest-size-in-set / size-n

So, depending on how you generate, you may have to hand-adjust the glyphScale later.

It is possible to specify one or more non-standard sizes for special processing, e.g., to do TTF-to-DAT overall scaling. To use in TDM, this will need subsequent file renaming to one of the 3 valid sizes, and glyphScale correction.

=== Name Option ===
To (re)name the font to be exported, use the option:
-name ''fontname''
By default, the fontname is instead extracted from the font file. In either case, the fontname string will be truncated if too long. It then begins each of the TGA filename(s), which are referenced in turn in the per-character "shaderName" fields of the DAT file. The shaderName field in a DAT file can have at most 32 characters; when viewed as a REF or FNT file, it has this template:
shaderName fontname_<1- or 2-digit count>_<2-digit fontsize>.tga

''TIP: If the default fontname is more than 16 characters, it is recommended you use the -name option, to prevent ugly truncation or (corrected in ExportUnicodeFontToDoom3) aborts.''

=== Other Options ===
In the description doc, there is a great deal of technical discussion about these next options, but of little likely relevance to TDM nowadays. Just ignore them in favor of their default values.
-textureFormat ''format''
where the default is .tga, which is what you want. Use another tool to convert the output from .tga to .dds, since ExportFontToDoom3 won't do it correctly.
-xOffsetFix ''type''
where the default is "none", which causes the font's X offset value to be exported to the "pitch" member of TDM .dat file. There is also the related:
-noXOffsetWarnings

=== Output ===
A new folder will be created and all output files will be stored there. The folder will be named according to the font being exported.
The output will consist of Targa image (.tga) files and .dat files.

Afterwards, from each TGA, you should create a corresponding DDS file.

== Deployment & Further Adaptions ==
After export, assuming you have an FM set up to test them, rename the files into standard form and move or copy them to the expected locations...
* For DAT files: fonts/english/<your font name>/
* For DDS files: dds/english/<your font name>/
You'll undoubtedly have to make further iterative adjustments to both the DAT files (using other methods listed in [[Font Conversion & Repair]]) and bitmap files.
For instance, the "glyphScale" value will likely have to be changed.

== Tutorials that Include ExportFontToDoom3 ==
Doom3-era tutorials, once posted on www.doom3world, can still be found with the "wayback machine" links given here:
* [https://web.archive.org/web/20081028224423/http://www.doom3world.org/phpbb2/viewtopic.php?t=11923 Tutorial 10 - Using custom fonts]
* [https://web.archive.org/web/20100522173219/http://www.doom3world.org/phpbb2/viewtopic.php?t=11924 Tutorial 11 - Editing fonts]

== Downloads of ExportFontToDoom3All256 ==
Windows binary and required archived versions of FreeType and DevIL DLLs:
* [https://drive.google.com/file/d/1HzyyuZv0AVEhw4czJEwQ4nDMuqCogRyi/view?usp=sharing 'ExportFontToDoom3All256 Release 1 of 04-26-2024.zip']

This version, having been rebuilt, does not require legacy msvcp71.dll & msvcr71.dll files needed to run ExportFontToDoom3 v 1.02.

The C++ project file with source code:
* [https://drive.google.com/file/d/1nkXJpBPa8qDqQxVQd9c1HlsSMKgohIw-/view?usp=sharing ExportFontToDoom3All256_source_project.zip] Release 1 of 04-26-2024. This zip is 177 MB; includes source & build project of archived library. Also sample TTF font (distributed with TDM) for debug testing.

For more about the modification and rebuild process, see "ReadMe_about_All256.txt", included in either of these two .zip downloads.

== Downloads of ExportFontToDoom3 v 1.02 ==
Windows binary:
* [https://grantheant.com/products/exportfonttodoom3 Grant Davie's project page]

Required DLLs:
To use ExportFontToDoom3 v 1.02 binary, you must have msvcp71.dll & msvcr71.dll installed in Windows or (more practical and safe now) in the same folder as the .exe. Once common, these files are no longer available from Microsoft (and purged from GitHub projects too). If you have an old machine with pre-Win10 OS, you can probably copy them from there. Otherwise, TDM's ATI Compressonator zip has them, at [[DDS Creation with ATI Compressonator]]; or various sketchy dll sites.

The C++ project file with source code:
* [https://grantheant.com/products/exportfonttodoom3 Grant Davie's project page]
* [https://www.moddb.com/downloads/export-font-to-doom-3-v102 Moddb downloads page]
* also from wayback machine.
The project makes use of the circa-2005 FreeType font library (www.freetype.org) and DevIL image library.

{{tutorial}} [[Category:Fonts]]

ExportFontToDoom3

2025-08-27T14:59:28Z

Geep: /* Size Option */ Mention glyphScale, non-standard sizes

== Intro to ExportFontToDoom3 & ExportFontToDoom3All256 ==
The Windows command-line utility "ExportFontToDoom3" was created by Australian modder Grant Davies in 2005 to get TrueType Fonts, standard on Windows, into the metadata+bitmaps form needed by the Doom 3 engine. His final version, 1.02, was released as open-source and substantially used for TDM. It relies on the open-source DevIL and Freetype2 libraries.

'''ExportFontToDoom3''' exports only the bitmap glyphs assigned to codepoints in the range 0-127, routinely the ASCII characters. The output actually generates metadata for 256 characters, but, due to a bug, upper-half characters are just mapped to the "undefined character" glyph.

'''ExportFontToDoom3All256''' fixes that bug; it exports the glyphs assigned to codepoints in the range 0-255, routinely ANSI characters. It is also easier to run on modern Windows PCs, so recommended. It takes the same command-line arguments as ExportFontToDoom3. Credits to Crispy (for 2009 fix) and (in 2024 form) Geep.

== Using the Tool ==
''The information here is adapted from the description doc included with the ExportFontToDoom3 download.''
=== Basic Command ===
To export with the common settings of 12, 24, and 48 point fonts, simply type:
ExportFontToDoom3All256 ''font_filename''
For example (and assuming here a font "badtimes" with no legal issues):
ExportFontToDoom3All256 badtimes.ttf
Most standard font formats (from the 2009 timeframe) can be used for export, but TrueType is most common.
=== Size Option ===
To export the font in a certain size, use the option:
-size ''font_pointsize''
For Doom3/TDM, only the sizes 12, 24, and 48 directly produce valid end-products. More than one "-size" arguments may be specified, for example:
ExportFontToDoom3All256 badtimes.ttf -size 12 -size 24
As indicated above, if no "-size" argument is specified, 12, 24, and 48 point fonts are exported.

The set of sizes in a particular generation run will jointly determine the "glyphScale" value specified at the end of bottom of a DAT/REF file, as follows:

glyphScale[for size n] = largest-size-in-set / size-n

So, depending on how you generate, you may have to hand-adjust the glyphScale later.

It is possible to specify one or more non-standard sizes for special processing, e.g., to do TTF-to-DAT overall scaling. To use in TDM, this will need subsequent file renaming to one of the 3 valid sizes, and glyphScale correction.

=== Name Option ===
To name the font to be exported, use the option:
-name ''fontname''
By default, the fontname is instead extracted from the font file. In either case, this fontname string will begin each of the TGA filename(s), which are referenced in turn in the per-character "shaderName" fields of the DAT file.
=== Other Options ===
In the description doc, there is a great deal of technical discussion about these next options, but of little likely relevance to TDM nowadays. Just ignore them in favor of their default values.
-textureFormat ''format''
where the default is .tga, which is what you want. Use another tool to convert the output from .tga to .dds, since ExportFontToDoom3 won't do it correctly.
-xOffsetFix ''type''
where the default is "none", which causes the font's X offset value to be exported to the "pitch" member of TDM .dat file. There is also the related:
-noXOffsetWarnings

=== Output ===
A new folder will be created and all output files will be stored there. The folder will be named according to the font being exported.
The output will consist of Targa image (.tga) files and .dat files.

Afterwards, from each TGA, you should create a corresponding DDS file.

== Deployment & Further Adaptions ==
After export, assuming you have an FM set up to test them, rename the files into standard form and move or copy them to the expected locations...
* For DAT files: fonts/english/<your font name>/
* For DDS files: dds/english/<your font name>/
You'll undoubtedly have to make further iterative adjustments to both the DAT files (using other methods listed in [[Font Conversion & Repair]]) and bitmap files.
For instance, the "glyphScale" value will likely have to be changed.

== Tutorials that Include ExportFontToDoom3 ==
Doom3-era tutorials, once posted on www.doom3world, can still be found with the "wayback machine" links given here:
* [https://web.archive.org/web/20081028224423/http://www.doom3world.org/phpbb2/viewtopic.php?t=11923 Tutorial 10 - Using custom fonts]
* [https://web.archive.org/web/20100522173219/http://www.doom3world.org/phpbb2/viewtopic.php?t=11924 Tutorial 11 - Editing fonts]

== Downloads of ExportFontToDoom3All256 ==
Windows binary and required archived versions of FreeType and DevIL DLLs:
* [https://drive.google.com/file/d/1HzyyuZv0AVEhw4czJEwQ4nDMuqCogRyi/view?usp=sharing 'ExportFontToDoom3All256 Release 1 of 04-26-2024.zip']

This version, having been rebuilt, does not require legacy msvcp71.dll & msvcr71.dll files needed to run ExportFontToDoom3 v 1.02.

The C++ project file with source code:
* [https://drive.google.com/file/d/1nkXJpBPa8qDqQxVQd9c1HlsSMKgohIw-/view?usp=sharing ExportFontToDoom3All256_source_project.zip] Release 1 of 04-26-2024. This zip is 177 MB; includes source & build project of archived library. Also sample TTF font (distributed with TDM) for debug testing.

For more about the modification and rebuild process, see "ReadMe_about_All256.txt", included in either of these two .zip downloads.

== Downloads of ExportFontToDoom3 v 1.02 ==
Windows binary:
* [https://grantheant.com/products/exportfonttodoom3 Grant Davie's project page]

Required DLLs:
To use ExportFontToDoom3 v 1.02 binary, you must have msvcp71.dll & msvcr71.dll installed in Windows or (more practical and safe now) in the same folder as the .exe. Once common, these files are no longer available from Microsoft (and purged from GitHub projects too). If you have an old machine with pre-Win10 OS, you can probably copy them from there. Otherwise, TDM's ATI Compressonator zip has them, at [[DDS Creation with ATI Compressonator]]; or various sketchy dll sites.

The C++ project file with source code:
* [https://grantheant.com/products/exportfonttodoom3 Grant Davie's project page]
* [https://www.moddb.com/downloads/export-font-to-doom-3-v102 Moddb downloads page]
* also from wayback machine.
The project makes use of the circa-2005 FreeType font library (www.freetype.org) and DevIL image library.

{{tutorial}} [[Category:Fonts]]

DatBounds

2025-08-25T15:35:17Z

Geep: /* Downloads */ released version 1.7

''By Geep, 2024''

== Introduction ==

DatBounds is a Windows command-line utility to help visualize TDM/Doom3 bitmap font bounding boxes and related metrics. It generates a special set of .tga image files, which are then brought into a tool like GIMP to view. But only view; any needed metric changes must be applied separately (e.g., by editing a .ref file). Nevertheless, this viewing helps (when used in conjunction with other tools like refont's "stats") to understand, improve, and extend complex and often overlapping metadata relationships.

The command-line format is:

datBounds -in <path to input metrics file> [options]

with options:

-tag = each drawn bounds box has a 2-hex-digit codepoint label.
-hug = show, in tan, horizontal baseline & pitch (as vertical line) that 'hug' each glyph.
-xskip = show xSkip as red dot on rightside of baseline.
-long_baseline_left = draw horizontal baseline from left boundary, instead of from pitch line to right. ''[new v1.3]''
-long_baseline_right = draw horizontal baseline to xSkip red dot, instead of to right boundary. ''[new v1.3]''
-long_pitchline_down = extend vertical pitchline downward to baseline (e.g., below lower boundary). ''[new 1.3]''
-boundless = suppress multicolor outer bounds; e.g., to help make tag-only .tga" ''[new v1.3]''
-from <codepoint num> (default 0)
-to <codepoint num> (default 255)
shows codepoints between 'from' and 'to', inclusive,
where <codepoint num> is 0xHH or decimal from 0 to 255.
-alpha = render background transparent instead of white.
-scaling_ok = suppress console warnings if per-character scaling.
-raw_metrics = show bounds box as imageHeight/Width size, & matching unscaled baseline, pitchline, xSkip.

Running "datbounds" with no arguments, or with "-h" or "?", will show this summary.

Otherwise, the program will generate one special .tga file for every regular glyph-containing .tga/.dds file. To keep datbounds relatively simple:
* the .tga format is used for output, instead of, say, .jpg or .png, that need more code or libraries;
* the names of the output files are not specified, but just derived from shader names contained in the -in file. For example, "stone_0_24" --> "stone_0_24_bounds.tga";
* These special files do not contain images of the font glyphs themselves.

So either view each one side by side with the corresponding glyph-containing file [Figure 1 below], or import copies of those paired files into GIMP as separate layers for combined viewing, as discussed further below.

== File I/O ==

The input file, with metadata for a particular font and fontsize, should be one of these formats:
* TDM/Doom3's '.dat'
* refont's '.ref'
* q3font's '.fnt'

Note that DatBounds does not read bitmap files, nor does it make use of refont’s annotation file.

DatBounds outputs multiple .tga files, into the same directory as the input file. The generated filenames are that of the set of shaderNames given in the input file, but with suffix "_bounds.tga". Within each will be a 256 x 256 image.

Except a few fonts (i.e., in the Mason and Carleton families) incorporate 512 x 512 bitmaps. Currently, for these you must use the REF format as input. A REF file (since refont v2.5) can indicate that any particular shaderName contained within is 512 x 512 by a header line:
# bitmap_<n>_size=512
where <n> is 0 to 9, and refers to shaders present in the REF file with names like, e.g., carleton_3_48.tga; n would be 3 for this.

== Example Output ==

[[File:Screenshot GIMP stone 0 24 bounds (ASCII).jpg|frameless|512px]]
[[File:Screenshot GIMP stone 0 24.jpg|frameless|512px]]

'''Figure 1.''' [Click either to enlarge further.] These show information about the english/stone font of 24 pt size, as distributed with TDM 2.12 in Feb, 2024. That size requires two bitmaps; we see just one of them here. Both images are screenshots of GIMP. The left image shows the datBounds output, framed in the gray scales of the GIMP view/edit panel. The right image shows the corresponding glyph bitmap, with the usual checkerboard indicating a transparent background.

This datBounds Version 1.0 command was used to generate the left image, which has the default opaque white background:

datBounds -in fontImage_24.ref -tag -hug -xskip -to 127

With Version 1.3, this would get the same results:

datBounds -in fontImage_24.ref -tag -hug -xskip -long_baseline_left -to 127

For simplicity, this figure has only bounds for the ASCII codepoints (because of "-to 127"). For each of those, it shows:
* the overall bounds as a green/blue box
* the pitch (aka xOffset) as a vertical tan line, and the baseline as a horizontal tan line (because of "-hug")
* the 2-digit-hex codepoint label, red on black (because of "-tag")
* the horizontal xSkip distance from the pitch line, shown as a red dot near the right end of the baseline (due to "-xskip")

Regarding the baselines in Figures 1 & 3: These were generated with datBounds Version 1.0, and are slightly shorter in subsequent versions. Version 1.3 introduces the long_baseline_left option, to once-again generate the Version 1.0 form. "See "Downloads" at end for more about this.

== Generated Visualizations and Options ==
Within each ...bounds.tga will be an image. (The discussion here is in terms of a 256 x 256 image; you can generalize to 512 x 512 cases.) The image contains, for each codepoint's glyph outer bounds, a two-color box. The walls of the box are defined by integer values in the range 0-255, namely:
* s_coord, t_coord, (s2_coord - 1), (t2_coord - 1) ''in .ref terms''
* round(s*256), round(t*256), round((s2-1)*256), round ((t2-1)*256) ''in .dat or .fnt terms''

Since s2 & t2 are considered to just beyond the glyph’s actual pixels, 1 is subtracted here. So the glyph pixels (if they were seen) would partially lap the box edges (except for the left edge due to pitch offset.) A box is rendered in two colors to help disambiguate abutting glyphs. Green for top & left edges (sharing s & t), blue for right & bottom (s2 & t2).

===Visualization Order===
The visualization algorithm uses a loop in codepoint order. Consequently, visual elements of higher codepoint numbers may obscure or hide those of lower numbers. This often occurs with shared or overlapping bounding boxes. While it can’t be avoided, if you wish you can manage it with the "from" and "to" options.

===Tag Option===
This shows a "tag", i.e., a label, having the codepoint value as a 2-digit hexadecimal number. It appears in the upper-left corner of the bounding box, with red digits on a black background. The tiny "dot matrix" font here is generated within the program. It is legible with practice when zoomed in within GIMP. (Tags will be generated the same size whether the bitmap is 256x256 or 512x512, so in the latter case, will appear relatively smaller compared to glyphs.)

[[File:DatBoundsDigits.jpg|frameless]]

'''Figure 2.''' Closeup of the digits of datBounds.

===Hug Option===
This shows two lines in a tan color, that informally can be said to "hug" the glyph:
* The "baseline", shown as a horizontal line, offset downward from the bounding box top edge by the "top" metric value. For a glyph without descender, this is often just below the bounding box bottom edge (because of latter’s t2-1 subtraction); the glyph is visualized as resting on the baseline.
* The "pitchline", shown as a vertical line, offset rightward from the bounding box left edge by the "pitch" metric value. Most glyphs will not extend leftward beyond the pitch line.

===XSkip Option===
Invoked with either "-xskip" or "-xSkip", this shows a red dot on the rightside of the baseline, corresponding to a horizontal location of:
s_coord + pitch + xSkip

===Long_baseline_left Option===
Ordinarily, drawing of the horizontal baseline starts at the vertical pitch line. The "-long_baseline_left" option moves the start leftward, to begin at the left boundary given by coord_s. (Actually, from the leftmost of the pitchline and the left boundary.)

===Long_baseline_right Option===
Ordinarily, the horizontal baseline ends at the right boundary given by coord_s2 - 1. The "-long_baseline_right" option extends it to abut the xSkip red dot instead. (Actually, to the rightmost of the right boundary and the red dot.)

These "long baseline" options are most useful with odd fonts that use unusual pitch values, e.g., Medusa, that has the pitch line to the right of the character glyph instead of the left.

===Long_pitchline_down Option===
Ordinarily, the vertical pitch line travels downward to the lower boundary given by coord_t2 - 1. The "-long_pitchline_down" option extends it downward to the baseline given by coord_t + top. (Actually, to the lower of the baseline and lower boundary.) This can be helpful particularly for certain punctuation characters, that with many fonts have tight bounding boxes with baselines far below.

===Boundless Option===
The "-boundless" option suppresses drawing of the multicolor outer bounds. Its main use is paired with the "-tag" option, to generate tag-only .tga files for GIMP layers.

===From & To: Codepoint Range Options===
These accept decimal or two-digit hex numbers (the latter preceded by "0x"). If both "-from" and "-to" are given, the value of "-to" should be greater than or equal to "-from". These options can be helpful to reduce the complexity of visualization with densely overlapping bounding boxes.

===Alpha Option – Make Background Transparent.===
By default, the background is opaque white. This provides good contrast if, in GIMP, you prefer to turn on grid lines, which are black. However, a transparent background option is provided in case you want to also see the glyphs and metrics superimposed, as discussed next.

===Scaling_ok Option===
This option (also found in refont) suppresses console warnings when per-character scaling is present, e.g., for the Mason font.

===Raw_metrics Option===
This option might be of occasional diagnostic interest for fonts with scaling, including large bitmaps (e.g., Carleton and Mason).

Ordinarily, a bounding box is drawn at {coord_s, coord_t, coord_s2, coord_t2}. Conversely, '''not''' drawn is a box representing the render-target dimensions, given by metrics imageWidth and imageHeight. Most often, the sizes of these two boxes are the same. If not, horizontal and/or vertical scaling factors (i.e., other than 1.0) can be derived. In that case, the other drawn items (pitchline, baseline [from "top"], xSkip) are routinely placed using such scaling.

If you specify this option, the bounding box is instead shown with imageWidth and imageHeight dimensions, arbitrarily with upper-left origin at {coord_s, coord_t). As before, the other drawn items (pitchline, baseline, and xSkip) are placed relative to the origin, but now at unscaled distances given directly by REF/DAT/FTN numeric values.

== A Combined View of Both Metrics and Glyphs in GIMP ==
For some fonts and font sizes, there is an existing TDM-created GIMP .xcf project file, that contains each bitmap of a glyph set as a separate layer, and that serves as the source file for TDM’s version of the font. Make a working copy of that for what is described here (as shown in Figure 3).

Or you can create your own version of this, e.g., importing both the _bounds file(s) and the corresponding glyph-containing bitmap file(s) into GIMP, as separate layers.

In either case, GIMP’s "Open as Layers..." is what you use. Be sure the _bounds files were created with the –alpha option. Once files are imported, hide all the layers except those of a particular pair (i.e., matching bounds & glyphs) of interest. Which one you place in front of the other depends on whether it’s more important to see the glyphs unclipped or the tags (if present) un-obscured.

More elaborately, you can import and stack multiple versions of a _bounds file, e.g., with different ranges, to toggle for differences.

TIP: Viewing glyphs against the checkerboard "transparency" pattern can be tedious. Consider adding an all-black background layer. To do so in GIMP, set the foreground color to black, then, when creating a new layer, you can ask it to fill with the foreground color.

You are creating a "snapshot in time", great for planning and quality control. But of course any subsequent edits to the DAT metrics renders the _bounds file(s) within the .xcf somewhat obsolete.

== Example Combined View ==
[[File:Screenshot GIMP layers stone 24.jpg|frameless|1000px]]

'''Figure 3.''' In GIMP, this shows the native project file (.xcf) that serves as master source for english stone 24pt bitmaps. This size has two bitmaps (_0_ and _1_). Only layers associated with _0_ are momentarily visible in the left panel. They are the glyph bitmap, and two datBounds bitmaps (lower 128 and upper 128 characters). While these are all transparent, the lowest "black background" layer renders the stack as shown.

This snapshot shows a few early changes towards font completion for TDM 2.13. So, for instance, the N in row 4 now has a tilde above it. Note how N and Ñ have overlapping bounds, allowing them to share the base letter but not the accent. This is common with hand-done font bitmaps to conserve bitmap real estate. (Subsequent to this image, the two bitmaps, further revised, were renamed to be more apt, as "_0_ Mostly ASCII" and "_1_ Mostly ANSI".)

== Downloads ==
''Version 1.7 of 2025, August 18''. Fixes fatal bug introduced in v1.6 when reading DAT format file.
* [https://drive.google.com/file/d/1SYaKyKXXCFHAYE5LH4e9XkSuJCGLvFOl/view?usp=sharing executable: datBounds.exe]
* [https://drive.google.com/file/d/10lHc_LXNublaMdgyZdXpomjmQYbwXXLt/view?usp=sharing Source code datBounds.cpp]

''Earlier Versions''

''Version 1.6 of 2025, January 22.'' Fixes shaderName and other issues preventing .tga generation. Shows filenames during write. Supports REF files with refont v2.5 headers for 512x512 bitmaps, plus options scaling_ok and raw_metrics.
* [https://drive.google.com/file/d/1NiFcAC0cA5-t8_nWH3pUzT5viNihA-u7/view?usp=sharing executable: datBounds.exe]
* [https://drive.google.com/file/d/1Tit5WwHSaVV7kXsw7xvsKfbC6fhUJiMw/view?usp=sharing Source code: datBounds.cpp]

''Version 1.3 of 2024, August 16.'' Prevents a crash if drawn lines are out of bounds. Fixes a bug where xSkip dot was always showing, even if -xskip not specified. Adds "0x" before hex form output of warnings and errors. Adds command-line options -boundless, -long_baseline_left, -long_baseline_right, and -long_pitchline_down.
* [https://drive.google.com/file/d/16tVHiXvCuG8pi6WLSOUFrNF0R4-_fWZA/view?usp=sharing executable: datBounds.exe]
* [https://drive.google.com/file/d/1_qfoH9FTmSBIqys2d1o-ZwD_bYJDzO3t/view?usp=sharing Source code: datBounds.cpp]

''Version 1.2 of 2024, June 9.'' Fixes a bug with codepoint 255 being skipped. Adds metadata consistency checking.
* [https://drive.google.com/file/d/1mMYc7pAjVDSiWF21a4vdtE3ROBSDzmUb/view?usp=sharing executable: datBounds.exe]
* [https://drive.google.com/file/d/1zcmBML14eessYvjlK4UjKMt13FOQ5N4u/view?usp=sharing Source code: datBounds.cpp]

''Version 1.1 of 2024, May 20.'' This shortens baselines. The left end of a baseline now begins at the pitch vertical. So that the red xSkip dot will be less likely to be obscured by the baseline of an adjoining character.
* [https://drive.google.com/file/d/1I5j8Az6bDZ4461_02sbh0RiVfm_1d6fD/view?usp=sharing Windows executable: datBounds.exe]
* [https://drive.google.com/file/d/1jN-Uvr9QYg1YOs8no3fNvuvnjOGPf48p/view?usp=sharing Source code: datBounds.cpp]

''Version 1.0 of 2024, May 10.'' Original.
* [https://drive.google.com/file/d/1nL7gTbPAaossPrL_0b39b9P_r7HiB6zq/view?usp=sharing Windows executable: datBounds.exe]
* [https://drive.google.com/file/d/1208Ce1bvbwXR9R8mY6cHP3fPBPKwtWpc/view?usp=sharing Source code: datBounds.cpp]

== See Also ==
* [[Font Bitmaps in DDS Files]] provides a broader context to the examples here.
* [[Font Files]] describes DAT and DDS file naming, directory location, scaling, and usage by TDM.
* [[Font Metrics & DAT File Format]] provides an Overview and helpful details.
* [[Font Conversion & Repair]] touches on tools to generate, visualize, and adjust DAT and DSS files, including utilities [[ExportFontToDoom3]], [[Q3Font]], [[Refont]], and [[Font Patcher]].

{{tutorial}} [[Category:Fonts]]

Text Decals for Signs etc.

2025-08-11T15:51:23Z

Geep: Move #str_ infobox, mention \n in *.lang

From [https://forums.thedarkmod.com/index.php?/topic/11159-add-text-to-your-signs-signposts-walls-etc/&tab=comments#comment-218526 Fidcal's 2010 Posting]. Updated by Geep, 2020.
==Introduction==

This article describes how to add non-graphic simple text to signs, signposts, walls, etc. using gui decals. In this image you can see some examples. This method only provides the application of the text itself (ie, does not include the signs, signposts, and so on). For the latter, one starting point is the model collection under darkmod/decorative/signs (avoiding any sign that has an embedded GUI).

<center>[[Image:text_decals.jpg]]</center> 

Dark Mod (from 1.03 onward) provides convenient prefab assets; or alternatively, see [[#Customising and Making your Own]] below.

For other types of in-game text see [[Text]]

==Ready-made Prefabs==

This is by far the easiest method. You simply insert a prefab and change the default text and size to what you want:

* This is described for the Dark Radiant editor.
* For general use all you need do is to find or make your blank wall, sign, etc.
* RMB in grid view, select "insert prefab", and navigate to readables/sign_text_decals/.
* Select a prefab with a desired font (eg, sign_text_carleton.pfb) and one of two sizes (with or without "_small" suffix).
* Of the Dark Mod fonts shown in [[Fonts Screenshots]], two-thirds are available as prefabs; for others, see [[#Customising and Making your Own]] below.
* You now have a working text decal, seen in DR as a blue rectangular "Entity GUI".
* To set your text, change the spawnarg value gui_parm1 in Entity Inspector to whatever you want.
* The defaults are all black text with 66% transparency so they blend well onto most surfaces but see [[#Customising and Making your Own]] below for how to change this.

==Text Size, Text Composition, and Multi-line Text==

by Geep, 2020

To change the size of the text, just resize the patch. To do this, first press the TAB key. Remember to press TAB again to get back.

Since you can't see the rendered text content results directly in DR, you will have to iterate to get what you want. The gui_parm1 value takes a single line of text (with no enveloping quotes). Within the spawnarg, there is no direct way to indicate line breaks; "\n" will not do it.

{{infobox|Note: Instead of using a hard-coded English text, consider using "#str_...", such as "#str_myFM_brewery_sign", and create a custom FM dictionary english.lang (and other translations). Strings in *.lang can effectively contain "\n". See [[I18N|here]] for details.}}

The examples shown here rely entirely upon word-wrapping for multi-line text. In practice, that means inserting spaces to force wordwrap (or see Tip below).

Often, you need to resize the Entity GUI patch to cover a larger area than the visible sign; the patch size and aspect ratio will affect font size and aspect ratio, but not wordwrap and number of visible lines. The decal patch is non-solid and invisible - except for the text itself - so can overlap any visible surface with no problem.

You can offset the Entity GUI with respect to the underlying sign to adjust left and top margins, including to fake centering.

===An Example===

To demonstrate some of this (except use of extra spaces in the text), here's a series of signs on a wall, all of which have:
* the same empty wall plaque
* an Entity GUI with 1:2 ratio (so the aspect ratio of the font characters is kept constant for demonstration purposes)
* use of a common font family, Carleton

The photos shows the series in DR (top) and in-game (bottom). Each row of signs has the same Entity GUI size and placement. Each column of signs has the same text string and choice of font size. The top left sign is an plaque-centered placement of a 16x32 "Carleton" Entity GUI. This typically results in poor placement, so all the other signs have the upper-left corner of the Entity GUI down and slightly to the right. Going down the first column, as the Entity GUI becomes larger (20x40 in the 3rd row, 28x56 in the 4th row), the font enlarges proportionately to the Entity GUI size, so wordwrap is the same in all cases. In the second column, the gui_parm1 text counts up to ten, but some of it is not rendered due to the Entity GUI boundary. Again, this line clipping is the same in all cases. The third and fourth columns repeat all that, but with the "Carleton Small" font. This allows both more words per line, and more lines. The specifics of those will vary with font family chosen.

<center>[[File:Example sign Entity GUIs in DR.jpg]]</center> 

<center>[[File:Example_sign_Entity_GUIs_in-game.jpg| 1100px]]</center>

{{infobox|Tip 1: Rather than relying on wordwrap for multi-lining, or using #str_ with "\n", you can create multiple overlapping, offset Entity GUIs, one per desired line, and adjust them independently. This technique also allows different lines to have different fonts and/or scalings.}}
{{infobox|Tip 2: Want to have text horizontally centered rather than left-justified? You'll have to do this manually. For a single line of text, either slide the Entity GUI sideways or add leading spaces or both. But for multi-lines, word wrap will ignore excess spaces between words, so spacing can't be used for centering after line 1. Instead, use the Tip 1 method.}}

== Signs with Illuminated Colored Letters ==
by Geep 2021

Within the GUI text, you can add Caret Control Codes (see [[Xdata file format]]) around particular words or phrases to give you the look of ''backlit'' letters in bright primary colors, perhaps to indicate wizardry afoot, evoke carnival, or provide eye-catching text for a custom in-game control.

^0 default
^1 bright red
^2 bright green
^3 bright yellow
^4 bright blue
^5 bright cyan
^6 bright magenta (pink)
^7 bright white
^8 light grey
^9 black

Example: "This is default black. ^1This is bright red."

The scope of a code terminates at the end of the line as formatted in the game, at which point it defaults back to the gui default color. So get your word wrapping in shape first, then apply codes. Or bracket around each word if you want it to work no matter the word wrap, e.g.:

"This is default black. ^1This^0 ^1is^0 ^1bright^0 ^1red.^0 Back to black."

==Customising and Making your Own==

Here are the steps to make your own text decals or customise the ones provided (say, to use a different font size by changing the gui's "textscale" value).

===The Decal===

* Create a patch and give it the texture: textures/darkmod/decals/signs/decal_gui . This is only available from Dark Mod update 1.03 onward but if you want it earlier this is what to do:
** Use textures/common/entityGui instead if it doesn't matter if the decal is solid.
** Create the custom texture listed below at [[#The Texture]]
* In Surface Inspector, use these in order: natural, fit, flip horizontal, rotate left 180 degrees.
* You should now see a single occurrence of the words Entity GUI on the patch and the words should be upright and not mirrored. If not, adjust, rotate, until they do.
* Note that the text size will be affected by the size of the patch
* Convert the patch to a func_static entity
* Give it the spawnargs:
** '''gui''' with the path and name of your gui (see [[#The GUI]] below) as its value.
** '''gui_parm1''' with your own text as its value.

===The GUI===

This is the gui file that defines the font. If you have Dark Mod update 1.03 you can modify an existing one. Alternatively, just copy and paste this into a text file, and modify:
windowDef Desktop
{
rect 0, 0, 640, 480
backcolor 0, 0, 0, 0

windowDef SignText
{
rect 0, 0, 640,480
backcolor 0, 0, 0, 0
text "gui::gui_parm1"
font "fonts/stone"
textscale 2
forecolor 0, 0, 0, 0.8
visible 1
}
}

Save it with your FM in a "guis" (sub)folder. For instance, as guis/readables/sign_text_decals/my_stone_text.gui.
You will need to add that path and name to the gui spawnarg of the decal entity above, eg,
gui guis/readables/sign_text_decals/my_stone_text.gui

In the above GUI-script definition, these are some of the things you can change:

* Set your font in the font line as shown, it must be in quotation marks as above eg "fonts/stone". And to change the font just replace the word stone with the new font name - font choices are shown in [[Fonts Screenshots]]. Eg in my case Carleton - "fonts/carleton"
* Change the font size in textscale. Fractions can be used, eg, 1.5 but remember, resizing the patch or texture scale in Dark Radiant also changes the font size.
* Change the colour and transparency of the text in the forecolor line. Because of the way Doom 3 blends imagery, for all practical purposes this is limited to very dark colours because they do not react to local game lighting. If you choose lighter colours then they will glow in the dark. (it might be possible to match a colour to a static light situation.) The forecolor values are:
** red, green, blue, transparency; each in the range 0 to 1.
** For colours, 0 to 1 is zero to full intensity; for transparency it is invisible (0) to fully opaque (1)
** For example:
*** 0.9, 0.9, 0, 1 would be high red, high green, no blue, and fully opaque. Red and green light make yellow so this would give glowing yellow text that would not blend well with its background.
*** 0.1, 0, 0, 0.66 This is more practical and would give a dark red that blends well onto its background with 66% opacity.
* A background colour can be set with the backcolor line exactly the same as for forecolor but is probably of limited use for our purposes. 0.9, 0.9, 0, 1 would have the text on a solid bright yellow background that glows in the dark. 0, 0.1,0, 0.33 would give a faint green background of 33% opacity.

===The Texture===

This is the non-solid decal gui texture. Copy and paste it into a text file and save it with your map in a materials folder as eg, mymap.mtr. You will need to reload shaders if you already have Dark Radiant running before you can select it.

textures/mymap/decals/signs/decal_gui
{
qer_editorimage textures/editor/entityGui.tga
DECAL_MACRO
nonsolid
noimpact
guiSurf entity
}

where DECAL_MACRO defines these additional keywords for you:

polygonOffset 1
discrete
sort decal
noShadows

== See also ==

* [[I18N|Internationalization]]

{{GUI}}

Text Decals for Signs etc.

2025-08-11T02:55:08Z

Geep: /* The GUI */ provide more typical path example

From [https://forums.thedarkmod.com/index.php?/topic/11159-add-text-to-your-signs-signposts-walls-etc/&tab=comments#comment-218526 Fidcal's 2010 Posting]. Updated by Geep, 2020.
==Introduction==

This article describes how to add non-graphic simple text to signs, signposts, walls, etc. using gui decals. In this image you can see some examples. This method only provides the application of the text itself (ie, does not include the signs, signposts, and so on). For the latter, one starting point is the model collection under darkmod/decorative/signs (avoiding any sign that has an embedded GUI).

<center>[[Image:text_decals.jpg]]</center> 

Dark Mod (from 1.03 onward) provides convenient prefab assets; or alternatively, see [[#Customising and Making your Own]] below.

For other types of in-game text see [[Text]]

==Ready-made Prefabs==

This is by far the easiest method. You simply insert a prefab and change the default text and size to what you want:

* This is described for the Dark Radiant editor.
* For general use all you need do is to find or make your blank wall, sign, etc.
* RMB in grid view, select "insert prefab", and navigate to readables/sign_text_decals/.
* Select a prefab with a desired font (eg, sign_text_carleton.pfb) and one of two sizes (with or without "_small" suffix).
* Of the Dark Mod fonts shown in [[Fonts Screenshots]], two-thirds are available as prefabs; for others, see [[#Customising and Making your Own]] below.
* You now have a working text decal, seen in DR as a blue rectangular "Entity GUI".
* To set your text, change the spawnarg value gui_parm1 in Entity Inspector to whatever you want.
* The defaults are all black text with 66% transparency so they blend well onto most surfaces but see [[#Customising and Making your Own]] below for how to change this.

==Text Size, Text Composition, and Multi-line Text==

by Geep, 2020

To change the size of the text, just resize the patch. To do this, first press the TAB key. Remember to press TAB again to get back.

Since you can't see the rendered text content results directly in DR, you will have to iterate to get what you want. The gui_parm1 value takes a single line of text (with no enveloping quotes). It is without a way to indicate line breaks; "\n" will not do it. The examples shown here rely entirely upon word-wrapping for multi-line text. In practice, that means inserting spaces to force wordwrap (or see Tip below).

Often, you need to resize the Entity GUI patch to cover a larger area than the visible sign; the patch size and aspect ratio will affect font size and aspect ratio, but not wordwrap and number of visible lines. The decal patch is non-solid and invisible - except for the text itself - so can overlap any visible surface with no problem.

You can offset the Entity GUI with respect to the underlying sign to adjust left and top margins, including to fake centering.

===An Example===

To demonstrate some of this (except use of extra spaces in the text), here's a series of signs on a wall, all of which have:
* the same empty wall plaque
* an Entity GUI with 1:2 ratio (so the aspect ratio of the font characters is kept constant for demonstration purposes)
* use of a common font family, Carleton

The photos shows the series in DR (top) and in-game (bottom). Each row of signs has the same Entity GUI size and placement. Each column of signs has the same text string and choice of font size. The top left sign is an plaque-centered placement of a 16x32 "Carleton" Entity GUI. This typically results in poor placement, so all the other signs have the upper-left corner of the Entity GUI down and slightly to the right. Going down the first column, as the Entity GUI becomes larger (20x40 in the 3rd row, 28x56 in the 4th row), the font enlarges proportionately to the Entity GUI size, so wordwrap is the same in all cases. In the second column, the gui_parm1 text counts up to ten, but some of it is not rendered due to the Entity GUI boundary. Again, this line clipping is the same in all cases. The third and fourth columns repeat all that, but with the "Carleton Small" font. This allows both more words per line, and more lines. The specifics of those will vary with font family chosen.

<center>[[File:Example sign Entity GUIs in DR.jpg]]</center> 

<center>[[File:Example_sign_Entity_GUIs_in-game.jpg| 1100px]]</center>

{{infobox|Tip 1: Rather than relying on wordwrap for multi-lining, you can create multiple overlapping, offset Entity GUIs, one per desired line, and adjust them independently. This technique also allows different lines to have different fonts and/or scalings.}}
{{infobox|Tip 2: Want to have text horizontally centered rather than left-justified? You'll have to do this manually. For a single line of text, either slide the Entity GUI sideways or add leading spaces or both. But for multi-lines, word wrap will ignore excess spaces between words, so spacing can't be used for centering after line 1. Instead, use the Tip 1 method.}}

== Signs with Illuminated Colored Letters ==
by Geep 2021

Within the GUI text, you can add Caret Control Codes (see [[Xdata file format]]) around particular words or phrases to give you the look of ''backlit'' letters in bright primary colors, perhaps to indicate wizardry afoot, evoke carnival, or provide eye-catching text for a custom in-game control.

^0 default
^1 bright red
^2 bright green
^3 bright yellow
^4 bright blue
^5 bright cyan
^6 bright magenta (pink)
^7 bright white
^8 light grey
^9 black

Example: "This is default black. ^1This is bright red."

The scope of a code terminates at the end of the line as formatted in the game, at which point it defaults back to the gui default color. So get your word wrapping in shape first, then apply codes. Or bracket around each word if you want it to work no matter the word wrap, e.g.:

"This is default black. ^1This^0 ^1is^0 ^1bright^0 ^1red.^0 Back to black."

==Customising and Making your Own==

Here are the steps to make your own text decals or customise the ones provided (say, to use a different font size by changing the gui's "textscale" value).

===The Decal===

* Create a patch and give it the texture: textures/darkmod/decals/signs/decal_gui . This is only available from Dark Mod update 1.03 onward but if you want it earlier this is what to do:
** Use textures/common/entityGui instead if it doesn't matter if the decal is solid.
** Create the custom texture listed below at [[#The Texture]]
* In Surface Inspector, use these in order: natural, fit, flip horizontal, rotate left 180 degrees.
* You should now see a single occurrence of the words Entity GUI on the patch and the words should be upright and not mirrored. If not, adjust, rotate, until they do.
* Note that the text size will be affected by the size of the patch
* Convert the patch to a func_static entity
* Give it the spawnargs:
** '''gui''' with the path and name of your gui (see [[#The GUI]] below) as its value.
** '''gui_parm1''' with your own text as its value.

{{infobox|Note: Instead of using a hard-coded English text, consider using "#str_20000" and create a custom FM dictionary. See [[I18N|here]] for details.}}

===The GUI===

This is the gui file that defines the font. If you have Dark Mod update 1.03 you can modify an existing one. Alternatively, just copy and paste this into a text file, and modify:
windowDef Desktop
{
rect 0, 0, 640, 480
backcolor 0, 0, 0, 0

windowDef SignText
{
rect 0, 0, 640,480
backcolor 0, 0, 0, 0
text "gui::gui_parm1"
font "fonts/stone"
textscale 2
forecolor 0, 0, 0, 0.8
visible 1
}
}

Save it with your FM in a "guis" (sub)folder. For instance, as guis/readables/sign_text_decals/my_stone_text.gui.
You will need to add that path and name to the gui spawnarg of the decal entity above, eg,
gui guis/readables/sign_text_decals/my_stone_text.gui

In the above GUI-script definition, these are some of the things you can change:

* Set your font in the font line as shown, it must be in quotation marks as above eg "fonts/stone". And to change the font just replace the word stone with the new font name - font choices are shown in [[Fonts Screenshots]]. Eg in my case Carleton - "fonts/carleton"
* Change the font size in textscale. Fractions can be used, eg, 1.5 but remember, resizing the patch or texture scale in Dark Radiant also changes the font size.
* Change the colour and transparency of the text in the forecolor line. Because of the way Doom 3 blends imagery, for all practical purposes this is limited to very dark colours because they do not react to local game lighting. If you choose lighter colours then they will glow in the dark. (it might be possible to match a colour to a static light situation.) The forecolor values are:
** red, green, blue, transparency; each in the range 0 to 1.
** For colours, 0 to 1 is zero to full intensity; for transparency it is invisible (0) to fully opaque (1)
** For example:
*** 0.9, 0.9, 0, 1 would be high red, high green, no blue, and fully opaque. Red and green light make yellow so this would give glowing yellow text that would not blend well with its background.
*** 0.1, 0, 0, 0.66 This is more practical and would give a dark red that blends well onto its background with 66% opacity.
* A background colour can be set with the backcolor line exactly the same as for forecolor but is probably of limited use for our purposes. 0.9, 0.9, 0, 1 would have the text on a solid bright yellow background that glows in the dark. 0, 0.1,0, 0.33 would give a faint green background of 33% opacity.

===The Texture===

This is the non-solid decal gui texture. Copy and paste it into a text file and save it with your map in a materials folder as eg, mymap.mtr. You will need to reload shaders if you already have Dark Radiant running before you can select it.

textures/mymap/decals/signs/decal_gui
{
qer_editorimage textures/editor/entityGui.tga
DECAL_MACRO
nonsolid
noimpact
guiSurf entity
}

where DECAL_MACRO defines these additional keywords for you:

polygonOffset 1
discrete
sort decal
noShadows

== See also ==

* [[I18N|Internationalization]]

{{GUI}}

ExportUnicodeFontToDoom3

2025-08-06T15:43:06Z

Geep: article created

''By Geep, 2025 [WORK IN PROGRESS]''

== Introduction ==

In 2025, kalinovka in [https://forums.thedarkmod.com/index.php?/topic/22736-font-localization Font localization] sought to extend TDM’s Mason 48pt font, used in the main menu, to include missing Cyrillic glyphs. This would allow this font to be used in an FM and still support a Russian translation.

This offshoot of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256] is a generalized response to that. The main idea to select an input TTF font that includes not just ASCII or ANSI, but offers additional Unicode characters, like Cyrillic. The program still limits output to a maximum of 256 characters, consistent with the DAT format.

To achieve this, it reads in an external human-readable "unicodeMap" file, in Unicode.org’s "Format A". This format provides a mapping from the traditional 8-bit encoding that TDM uses to the corresponding UCS (Unicode 16-bit) value. Unicode.org provides stock files for standard ISO and Windows 8-bit encodings. Of specific interest here is [https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT cp1251.txt for Cyrillic]. This requires just trivial editing for [https://wiki.thedarkmod.com/index.php?title=I18N_-_Charset TDM-specific codepoints].
As discussed below, you can further edit a unicodeMap file in advance, if you want to generate just a subset of glyphs.

== New Command Line Arguments ==

These are in-addition to those of [https://wiki.thedarkmod.com/index.php?title=ExportFontToDoom3 ExportFontToDoom3All256].

=== unicodeMap <file> ===
Example:

-unicodeMap "./Test/cp1251.txt"

You can edit the unicodeMap file in advance, to specify which particular glyphs you want to generate by suppressing unwanted glyphs, either by:

* Deleting an unwanted line (or block of lines) entirely, or
* Prepending a "#" to comment-out the line.

Also, this file format allows the Unicode value (e.g., "0x1234") to be replaced by 6 space characters, when the ISO or Windows standard leaves that character undefined.

In all 3 of those cases, every suppressed character in the output DAT file will be represented by the glyph that the font uses for U+0000. A hollow box is common.

=== startFileNumber <n> ===
When a set of TGA/DDS files are generated for a particular font size, by default the first file's name contains _0_, which increments for additional files. You can optionally specify a different 1- or 2-digit start number, e.g., "-startFileNumber 7". Why? Because...

* the file(s) you are generating are supplemental to, not replacements for, existing TGA/DDS files; and
* in this run, you are generating just one of the 3 possible font sizes (assuming each size has a different count of existing TGA/DDS files).

When you generate a supplemental set, the new DAT file has no knowledge of what's in the original set and its DAT file. You will have to interleave-edit the old and new DAT files (e.g., with reFont's REF files) to create a single DAT file spanning old and new glyphs. There may be a redundant NULL glyph in your new TGA set; just ignore it.

=== awayFromEdge <n> ===

Allows specifying in positive integer pixels (e.g., "-awayFromEdge 2") a small extra boundary at the top and left edges of the overall bitmap. This is for fonts like Mason where extra space (typically 2 pixels) is needed around every character to create a "glow variant". Routine glyph layout provides usually enough space for this except at the top and (arguably) left edges. See also the "compact" parameter.

=== compact ===
If specified (i.e, "-compact"), packs the character glyphs more closely together. Particularly recommended for 48pt fonts; it will generate roughly half the files.

Whether "compact" is specified or not, packing is controlled by simply "padding" each glyph’s bounding box. Specifically, this doesn’t change the bounding box, but adds pixel "padding" to just the right and bottom of the bounding box, a simple though asymmetric process. One reason for padding was given ExportFontToDoom3 author Grant Davies in code comments:

"Because of the way Doom 3 Indexes the texture page (using floating point numbers), it is unable to reference exact pixels – there is error. How much error, I don’t know. For this reason, the rectangles needs to be padded out."

Geep suspects that such error with the TDM engine is not really a problem. Likely some error in the past was due to editing with Q3Font and its low floating-point precision. Nevertheless, some padding is gppd because:

* For the Mason font, 2 pixels around each glyph are needed for "glow" effects (either done as a separate smudge font for /english/ or as a bidirection offset effect for /russian/.)
* Keeping glyphs separated improves visualization of boundaries with datBounds, and makes editing tweaks easier.

The "compact" setting restores Grant Davies' first algorithm, which just pads the bounding box by 5 pixels to the right and 5 pixels below.

Without "compact", the default is Grant Davies' second algorithm found in ExportFontToDoom3 v1.02. This likewise first pads each bounding box by 5 pixels to the right and 5 pixels below. It then further pads to make both the width and height a factor of 2; this makes 48pt quite sparse. (It is possible this improved rendering performance in 2005; not important now. Also, in practice for major fonts, much of this "factor of 2" spacing was overridden by subsequent Font Patcher work.)

''[MORE TO COME]''

Refont

2025-07-28T20:03:08Z

Geep: /* Downloads */ release 2.6. Link contents refreshed

''By Geep, 2024, and similar to the article about Q3Font.''

Introduced in 2024, the "refont" command-line utility allows inspection and alteration of [[Font Metrics & DAT File Format | font metrics]].
It is designed to be an easier-to-use version of the inspection/alteration functionalities of [[Q3Font]].
It also provides a font analysis feature.

== Comparison with Font Patcher ==

Refont can be seen as complementary to [[Font Patcher]]. The latter has additional features that may be better if planning a wholesale rearrangement or expansion of characters within a bitmap, particularly for a new font immediately after conversion from a TTF font. But refont is easier to use in later years, and Font Patcher requires installation of Perl, while refont.exe is more drop and go.

== Comparison with Q3Font ==

For metric alteration, q3font has a native "*.fnt" human-readable format, and supports DAT <==> FNT conversions. For compatibility, so does refont.

=== Advantages of Q3Font ===

* This has been an important traditional tool for Doom3/TDM font manipulation.
* In addition to production/consumption of FNT, it -
** Allows creation of DAT & TGA files from a TrueType font (although for this purpose, [[ExportFontToDoom3]] has been preferred in practice).
** Can write (but not read) an alternative "compact report".

=== Advantages of Refont ===

* It's open-source, so further C++ development is possible.
* It has its own native human-readable format, "*.ref", similar to .fnt but better.
* The REF format can be read by [[datBounds]] to generate bitmap boundary & metrics visualizations.

The better features of REF are:

* Codepoints are enumerated in not just decimal, but also hex.
* Coordinates of the overall image box within its bitmap are expressed in pixel units, rather than [0...1] fractional floating point. So the user is spared having to manually do this tedious and error-prone calculation.
* This overcomes a drawback of FNT, that expresses the coordinates with only 6 digits after the decimal point, not quite enough to avoid minor mis-precisions.
* if optionally provided, a separate, read-only annotation file will automatically decorate the .ref file with helpful information as the file is generated. This is explained below.

Because the code is open-source, there's better clarity about -
* how parsing and calculations are done.
* what warnings are emitted.

Also, using "-stats", refont can write (but not read) an analysis of a DAT file, particularly looking for problematic/unimplemented Western-language characters.

== For Inspection & Correction of DAT files ==

''To Get the Data in Readable, Editable Form''

For viewing and altering in a text editor, two formats are provided. Both list every codepoint, in order from 0 to 255, as a block of data. Within each block, every item gets its own line.

=== As a FNT File ===

To create a q3font-compatable "<dat-name>.fnt" file:

refont -decompile <path to given .dat file> -fnt

where the path can be a full path, a relative path (e.g., starting with "./" or "../"), or just the file name in the current working directory. The resulting .fnt file will appear in the same directory. It have the same name as the .dat file, except for the extension. As a convenience to keep track of versions, it is not required that the file name be in the standard fontImage_nn.dat form.

Example code block for the letter A:
...
char 65
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
s 0.769531
t 0.250000
s2 0.847656
t2 0.320313
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Note that Q3Font FNT expresses the s, t, s2, and t2 coordinates with only 6 digits after the decimal point, causing minor low-precision rounding during DAT --> FNT --> DAT roundtrips. For compatibility, Refont FNT does likewise.

=== As a REF File ===
''Starting with Refont v2.5, a mixture of 256x256 and 512x512 bitmap shaders, encountered with Carleton and Mason family fonts, can be handled. See further below.''

To create a refont-specific "<dat-name>.ref" file:

refont -decompile <path to given .dat file>

The same path and naming conventions as for .fnt generation discussed above apply here.

Example code block for the letter A when no supplemental annotation file is present (nor any per-line comments, e.g., with warnings):

...
char 65 (0x41)
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

Items bolded differ from FNT format. The starting line expresses the value in hex. And rather than s, t, s2, t2 in fractional [0...1] floating point, the "coord_..." values are expressed (and editable) as integer coordinates of the bitmap, with a range reflecting bitmap size. The default TDM bitmap size is 256x256, but a few fonts use 512x512 and require special treatment discussed further below.

== When Editing ==
Refont has a minimal parser. To keep it happy, preserve the file's line structure. In particular:
* every REF and FNT file has the same number of lines. ''(Exception: REF may have additional starting lines to flag 512x512 bitmaps, discussed below. Generally, you won't edit these.)''
* all keywords within a block must be present, and in a particular order.
* each block must have exactly the same number of lines (except for the special one at the end).

The meaning of the font metrics are explained in [[Font_Metrics_%26_DAT_File_Format]]. The [[Q3Font]] article has an example of simple metric editing that is also pertinent to refont.

== To Move Changed FNT or REF Data Back to a DAT ==

refont -compile <path to given .fnt or .ref file>

This reverses the -decompile process, to create a .dat file whose name is derived from the .fnt or .ref file. Importantly, this decompile/compile cycle leaves .tga/.dds files untouched.

== The Annotation File ==
When told to generate a REF file (but not FNT), refont.exe also looks for a file called "refont_char_annotations.txt", in the same folder as refont.exe. That file should have exactly 256 data lines in it, one for each of the codepoints, in order. As the REF is generated, the line that starts a codepoint's data block gets the corresponding annotation line appended to it. Having the annotations come from an external file, instead of hard-coded into refont, allows required flexibility as to what information is presented. It even potentially allows refont to be used with other idTech3/4 games that don't necessarily used the TDM-specific codepoints. Also, the annotation files contain helpful specific comments (as lines that start with ";").

Each annotation:
* indicates what character symbol is ''expected''; it is oblivious as to what character if any is actually within the bounding box described by the DAT data.
* lives only in a REF file, and is not retained when converted to a DAT file. This also true for any manual supplementary annotations you might make.

This is clearer with examples. In 'Downloads' below are 4 different versions of "refont_char_annotations[...].txt". Three of these, specific to TDM's custom 'english' (i.e., composite European) codepage, are discussed here. To deploy, edit the filename of one of them to remove the square bracket text.

=== English with Symbol Only ===

This simple format is best for most purposes. The file entry for letter 'A' is just:

A

Then the block in a generated REF file would have that string appended, but with " // " inserted.
...
char 65 (0x41) // A
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 197
coord_t 64
coord_s2 217
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

=== English with Unicode-16 Codes and Names ===

This supplies the Unicode Consortium's official 16-bit codepoints (where U+NNNN = 0xNNNN) and their names, in the formal all-caps convention.

The file entry for letter 'A' is:

U+0041 LATIN CAPITAL LETTER A

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // U+0041 LATIN CAPITAL LETTER A
...
Here's another example from that REF:
...
char 129 (0x81) // U+015A LATIN CAPITAL LETTER S WITH ACUTE
...
For control or TDM undefined characters, this annotation file assumes a hollow box glyph is the appropriate mapping:
...
char 0 (0x00) // U+24A1 WHITE SQUARE
...
But there are other choices, so edit the annotation file as you see fit.
=== English with 8859-x Source ===

This verbose format may be useful during font analysis. For letters that are in their expected locations as defined by ASCII or ISO 8859-1 (Latin-1) or Windows-1252 ("Western Europe"), the annotation is fairly minimal. Thus, letter 'A' has this:

0x41 is A

(TIP: If authoring your own annotation file, consider including a codepoint number on each line, like "0x41" here, to make the process easier.)

Then the first line of the corresponding block in a generated REF file would show:
...
char 65 (0x41) // 0x41 is A
...
Here are a few more interesting examples from that REF, with more expansive annotations:
...
char 5 (0x05) // 0x05 is a control character, unusable in TDM
...
char 166 (0xa6) // 0xA6 is Š (not ISO 8859-1 ¦) (from A9 in ISO 8859-2) (TDM only)
...

== Revising or Making Your Own Annotations File ==
If creating your own refont_char_annotations.txt file, note that encoding upper-range characters in UTF-8 will be more universal than using Windows region-specific codepages. To enter a UTF-8 character using its four digit hex Unicode (say, in a range of interest 0x0080-0x00FF):
* Under Windows, type the four hex digits followed by Alt-X.
* Under Linux, hold down Ctrl+Shift, type U followed by the 4 digits; release Ctrl+Shift.

When refont reads refont_char_annotations.txt, any line that starts with ";" will be ignored. The latter is so that you can put comments into the file, that will not propagate to the generated REF file. TIP: If such commenting is unwanted, start the line with <space> before ";".

Other than that, be sure to have exactly 256 lines of annotations (though refont will check and issue a warning if not). A line can be empty, i.e., have only the linebreak.

When generating a REF file, refont will automatically add " // " before each non-empty annotation, so don't include that.

== Adding Your Own Annotations to a REF File ==
Beyond the refont_char_annotation.txt content, you can add certain commentary to the REF file without it affecting its ability to convert to a DAT file. Here are the rules:
* Don't add or subtract lines, or comment out non-blank lines
* Feel free to append a comment to the end of a line, by beginning the comment with " //".
* Likewise, on a line starting with "char" that had an annotation automatically added, you can revise the annotation as you see fit.

Example with revision in bold:
...
char 65 (0x41) // '''Capital A was slightly clipped. Decremented s, s2 by 1. Looks OK now.'''
{
height 18
top 18
bottom 0
pitch 3
xSkip 18
imageWidth 20
imageHeight 18
coord_s 196 '''// WAS: 197'''
coord_t 64
coord_s2 216 '''// WAS: 217'''
coord_t2 82
glyph 0
shaderName fonts/stone_0_24.tga
}
...

In this way, you can markup a REF file as a record of work to do, work in progress, or work completed.

==Errors, Warnings, and Auto-Corrections==
''This covers the major overhaul of Refont v 2.1, and its added options -no_warn_comments and -scaling_ok.''

During the compile or decompile, errors and warnings may be generated in the console. Fatal errors are largely due to:
* file I/O problems;
* deviations from the strict DAT, REF, and FNT formats.

Warnings, which are not fatal, are largely due to anomolous numeric values. The TDM engine can tolerant some of these, so they don't necessarily need fixing.

===Fatal Errors when Parsing REF or FNT as Input===
A fatal error, due to divergence from the expected format while parsing, stops the program. While potentially due to many causes (e.g., adding or deleting lines), errors will be reported - with line number - as failures of:

''Variable Name Checks.'' The parser expects every variable name in a particular order. A mismatch happened.

''Integer Checks.'' Integer value was unparsable, or out of range for integer type. Checked for per-character variables height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and glyph. Plus REF’s coord_s, coord_t, coord_s2, and coord_ t2.

''Float Checks.'' Floating point value was unparsable, or out of range for float type. Checked for glyphScale, and FNT’s per-character variables s, t, s2, t2.

===Post-Read MAJOR Warnings about REF, FNT, and DAT Input Values===
After parsing, the input process concludes with a separate loop of non-fatal warning analysis. Warnings during input, broadly categorized as major or minor, report to the console, but don’t do any value corrections (which is instead left to the subsequent file-write stage).

''For FNT’s and DAT’s s, t, s2, t2''
* Inexact Integer. The given bounding box value, expected in the range 0-1, when multiplied by 256.0 should be exactly an integer. If not, the divergence is likely due to miscalculation. (Actually, a divergence from an integer of under +/- 0.001 is tolerated by Refont and considered a minor warning, discussed below.)
* Float Out of Range 0.0 - 1.0. Or equivalently, post-multiplication, 0.0 - 256.0. (See the discussion of the edge cases under minor warnings below.)

''For Other Metrics of DAT, FNT, and REF''
* Integer Out of Range 0-256. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and REF’s coord_s, coord_t, coord_s2, and coord_t2.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).
* Inconsistent Metrics. For a given character, this compares two metrics and looks for inconsistencies. We’ll state this here in REF terms, e.g., using “coord_s” rather than “ROUND_TO_INT(s * 256.0f)”:
** imageHeight == height?
** imageWidth == coord_s2 - coord_s?
** imageHeight == coord_t2 - coord_t?

''Suppressing Warnings Due to Scaling.'' The last two "Inconsistent Metrics" criteria will not be true if there is per-character scaling. (Only TDM’s mason and mason-glow use per-character scaling, specifically for all upper-case characters.) In such a case, to avoid generating a lot of useless major warnings, use the refont option "-scaling_ok".
===Post-Read MINOR Warnings for FNT’s and DAT’s s, t, s2, t2===
''Low Precision.'' This value, when multiplied by 256.0, should be exactly an integer. However, some generation or revision procedure (e.g., Q3Font’s DAT  FNT  DAT) may have not used sufficient precision to ensure this. An offset from an integer value of less than +/- 0.001 is considered a minor problem; otherwise, major as above.

This problem, when it occurs with a file, typically happens a lot. Consequently, it is not reported in an itemized way, but instead, the console will show total counts of affected characters and values.

Edge cases: a value of {s, s2, t, or t2} * 256.0 that is negative but closer to zero than -0.001000 is considered this type of minor problem (and is not considered out of range, which would be major). Likewise, if that value is greater than 256.000000, but less than 256.001000
===Automatic Corrections during Output to a REF File===
Certain problems, first detected during DAT input, are further addressed at write time. Because these problems were already reported to the console during the DAT read, nothing additional is written to console. Instead, there may be reporting within the REF file itself.

''Automatic Corrections.'' When writing a REF file, the integer values of coord_s, coord_t, coord_s2, and coord_t2 are calculated from DAT floats s, t, s2, and t2 respectively. A calculated coord_... value, if not already resulting in an integer, is automatically forced to the nearest integer. This is so whether it was earlier reported to the console as a Major or Minor Warning. (Before refont v 2.1, the REF might assign an unrounded decimal value; no longer true.)

''Warnings Not Automatically Corrected.'' Automatic fixing of other warnings is unwise and not offered. Why? Because the fix is not obvious or complex, and sometimes, the corresponding character glyph in the bitmap needs adjustment. This pertains to Major Warnings about:
* Integer values outside their expected range (e.g., 0-256), including post-rounding coord_... values.
* Inconsistent Metrics
===Generated Within-REF Comments during Output===
For most causes of a Major Warning during DAT read, during subsequent writing of the REF file, a similar comment is also generated within the REF, on the same line as the parameter assignment. For example:
char 95 (0x5f) // _
{
...
top -6 // WARNING: Not in range 0-256. Does bitmap or other metrics need adjusting?
...
}
...
char 140 (0x8c) // Ń
{
...
coord_t -10 // WARNING: Underlying DAT value -0.03906250, * 256.0 gave -10.000000000, not in range 0-256 once rounded. Does bitmap or other metrics need adjusting?
...
}
''Suppressing Comment Generation.'' Use the refont option “-no_warn_comments”. ''New to v 2.1.''

TIP: When editing a REF file with comments, you can keep them, change them, or delete them as you prefer. They have no impact on subsequent compiles to DAT.

''Coverage of Generated Within-REF Comments.'' A calculated value of coord_s, coord_t, coord_s2, or coord_t2 would have been flagged as a “Major Warning” during read if:
* the discrepancy from being an integer was non-trivial (+/- 0.001 or more), or
* the calculated and rounded integer is outside the range given by bitmap size, e.g., 0-256.

For these, a comment is generated. Other causes are:
* Integer Out of Range 0-256 ''(or 0-512 as applicable)''. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight.
* Integer Non Zero. For glyph.
* Float Less Than Zero. For glyphScale (which is independent of a particular character).

''Major Warnings that Don’t Cause Within-REF Comments.'' Currently, this is limited to “Inconsistent Metrics”, which would probably require tagging on multiple lines.

== Special Handling of Fonts with 512x512 Bitmaps ==
Some DAT files reference bitmap shaders where some or all are of size 512x512, instead of the default 256x256. Refont v2.5+ can support this (replacing an insufficient first attempt in v2.4).

Font developers went to double-sized bitmaps to:
* in the case of Carleton, allow freehand drawing of missing European characters, which could be drawn more easily if twice standard size.
* in the case of Mason, have larger ASCII alphanumeric glyphs that would render with crisper edges.

In either case, the font characters when rendered in, say, the game main menu will be the same size as if from 256x256. This is because the imageWidth and imageHeight metrics are not doubled.
In effect, a scaling by half is requested for all characters in the DAT file and applied by the engine.

=== When generating a REF file from a DAT file ===

refont -decompile <path to given .dat file> -512_<n> ''where <n> is a single digit, e.g. '1' in referenced shader name "font/Carleton_1_24.tga" ''

You should use multiple instances of the -512_<n> option, to cover all the 512x512 bitmaps found in the DAT file.

If ''all'' of a given DATs shaders are 512x512, you can use this shorter form:

refont -decompile <path to given .dat file> -512

Exercising -512... options will mean that in the generated REF file:

* There will be a one or more starting lines like:
# bitmap_<n>_size=512 ''(Refont will look for this if doing a subsequent compile to DAT)''
* If you specified the short form, it will expand to just the right number of starting lines.
* Values for coord_s, coord_s2, coord_t, and coord_t2 will have a normal range of 0-512 when appropriate. [COMING SOON: Those values may then be read properly by datBounds.].
* Any warnings expressed (within the file or within the command-line window) will be in terms of the correct range of values.
* Warning checks will expect a scaling by half for all characters. That is, while 512... requires scaling down by half, that scaling will not generate a warning (unless it's not EXACTLY by half), so the -scaling_ok option is not needed just for the exactly-by-half scaling.

=== When generating a DAT file from a REF file ===

During refont -compile, no special command line options are needed. Instead, REF's starting lines of form "# bitmap_<n>_size=512" are inspected and coord ranges will be interpreted as 0-512 whenever appropriate.

=== Special Handling Recommendations for Carleton and Mason ===
''For fonts shipped with TDM 2.12 in tdm_fonts01.pk4. Illustrative command paths are shown; adjust to your working copy.''
==== Mason Family ====
The [[Mason Font]] uses within-DAT per-character scaling, as well as some but not all 512x512 bitmaps. Only 48pt Mason is shipped.

Because, under /dds/fonts/english/mason/, bitmaps masonalternate_0_48.dds to _2_ are 512x512, but _3_ to _5_ are not, use:

-decompile /fonts/english/mason/fontimage_48.dat -512_0 -512_1 -512_2 -scaling_ok

Because, under /dds/fonts/english/mason_glow/, bitmaps masonalternate_1_48.dds and _2_ are 512x512, but _0_ and _3_ to _5_ are not, use:

-decompile /fonts/english/mason_glow/fontimage_48.dat -512_1 -512_2 -scaling_ok

Russian versions use the default 256x256. [These possibly benefit from -scaling_ok; didn't investigate.]
==== Carleton Family ====
The 24pt size of carleton has two bitmaps under /dds/fonts/english/carleton/, namely carleton_0_24.dds and _1_, both of size 512x512. So use:

-decompile /fonts/english/carleton/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton/fontimage_24.dat -512

The 24pt size of carleton_bold has bitmaps with identical content to carleton's, so use the same options:

-decompile /fonts/english/carleton_bold/fontimage_24.dat -512_0 -512_1
''or''
-decompile /fonts/english/carleton_bold/fontimage_24.dat -512

Other members of this family use the default 256x256 bitmaps:
* Sizes 12pt & 48pt
* carleton_condensed and carleton_glow. (Note: while helpful, it is not required that the base & glow have identical bitmap sizes.)
* Russian versions

==== These Fonts with Stats ====
With -stats discussed next, if reading a DAT file, you may use the -512... and scaling_ok options to be most correct, although neglecting to specify them will likely have no significant impact on the analytical results. If reading a REF file, the included "# bitmap_..." lines will take care of that for you.

== For Statistics on Unimplemented or Problematic Font Characters ==
DAT or (as of v2.3) REF file analysis here is primarily designed to benefit english/european fonts. It will use the optional annotation file if provided. Syntax:

refont -stats <path to given .dat or .ref file>

Use "-stats" by itself, not combined with "-compile" or "-decompile".

The resulting analysis report will have the same name and location as the input file, but with a ".txt" suffix.
The report provides categorized totals and (for problematic characters) itemizations across all 256 character codepoints.
The counts of problems should be considered minimums, since no inspection of bitmap files (TGA or DDS) is done.
But a perhaps surprising amount of insight can be gleaned just from DAT/REF file metrics.
Itemizations include annotations from the 'ref_char_annotation' file, if provided.

In the hunt for missing characters, 3 signatures are important:

* Presumed 'Hollow Box' = glyph's 'shadername' is <fontname>_0_<size>.dds only, and s & t are zero, but not s2 & t2
* '<Space>' = same test as 'Hollow Box', but pointed to by char 32 (0x20)
* 'Zero Box' = No glyph box (s, t, s2, t2 all zero)

A given analysis will assume either 'Hollow Box' or '<Space>' but not both.

The analysis provides totals and itemizations - grouped into lower (0-127) and upper ranges (128-255) - in several passes:

* Pass 1 - Handling of unprintable/unsupported/missing codepoints, indicated by Hollow Box, Zero Box, or <Space>. Some of these are not a problem, but some are tagged as "Undesirable".
* Pass 2 - Bad glyph box (negative s, t, s2, or t2; or s2 <= s, t2 <= t) or good glyph box with dubious metrics (imageHeight <= 0, imageWidth <=0, imageHeight != height). Excludes those already counted as "Undesirable" in Pass 1.
* Pass 3 - Detection of duplicate glyph boxes (other than Hollow Box, Zero Box, or <Space>). Detected by: glyph's values for shadername, s, t, s2, & t2 exactly match those of another codepoint. These are grouped into "Dup Sets".

Then, across passes 1-3, a count if given of the minimum "Total Glyphs Needing Work".

Refont version 2.3 adds additional passes:

* Pass 4 - Reports "Overlap Sets" rather than just "Dup Sets". A set here may include an unaccented "base character" that has been serving as a substitute work-around target for one or more missing accented characters. (What constitutes an "overlap" is idiosyncratic; see the source code for details.)
* Pass 5 - Same information as Pass 4, but sorted first by shader name (i.e., bitmap). Helpful for organizing bitmap-editing work.
* Pass 6 - An analysis focusing just on the planned migration of duplicate O/o-circumflex codepoints to G/g-cup glyphs.

== Downloads ==
Release 2.6 of July 28, 2025:
* [https://drive.google.com/file/d/1ABf4771Jr1C1hKbLx4xWq8lx8Kn7HdKP/view?usp=sharing refont.exe]
* [https://drive.google.com/file/d/1-k68eo0CHKZ-sM7JGP05axm49cc5De0T/view?usp=sharing source code: refont.cpp]

===Annotations===
''After download, to use any one of these, edit filename to remove "[...]" substring.''

Update of June 20, 2024. This reflects a change to the TDM charmap for 2.13, to replace duplicate O/o circumflex characters with G/g breve:

* refont_char_annotations[english with symbol only. With G-breve update].txt [https://drive.google.com/file/d/1rrt-O9DgD7Zn2Hxt7nBtViIlO4kv_gQB/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes & names. With G-breve update].txt [https://drive.google.com/file/d/1AMxDKQc_DE33gsExiEkGp3x67eb4aGFh/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source. With G-breve update].txt [https://drive.google.com/file/d/1xoeZ1poTbmkls-gE5PnbbMxdoz5KnMD7/view?usp=sharing].
* For Russian, see the previous update next.

Update of April 13, 2024:

* refont_char_annotations[english with symbol only].txt [https://drive.google.com/file/d/1ItM-aXps0xQE655HdaJM7Aje5lHewUfC/view?usp=sharing here]. Recommended for most purposes with 'english' fonts.
* refont_char_annotations[english with Unicode-16 codes and names].txt [https://drive.google.com/file/d/1gW7oO_wLqCEPSdm6b05OaTNYHnyuUSA-/view?usp=sharing here].
* refont_char_annotations[english with 8859-x source].txt [https://drive.google.com/file/d/1ZTcgA4QJYwXzNdqbolYkn0CCEDILD1bX/view?usp=sharing here]. Corrects and supersedes original english file.
* refont_char_annotations[russian].txt [https://drive.google.com/file/d/10txNXsSCtaTiT2iGT9o3iDxBU7HVYNxi/view?usp=sharing here]. Recommended for 'russian' fonts.

== For More ==
* See the [https://forums.thedarkmod.com/index.php?/topic/22427-analysis-of-212-tdm-fonts/ summary analysis of 2.12 TDM fonts], based on applying 'refont -stats ...' to all 'english' DAT files.

{{tutorial}} [[Category:Fonts]]

Gen Lang Programs

2025-07-16T20:21:25Z

Geep: Add [Catalan]

By Geep, from February 2025
== Introduction to Gen_lang.pl ("PL") and Gen_lang_plus ("PLUS") ==
This describes two software implementations of command-line programs that essentially do the same thing;
# The Perl program "gen_lang.pl" (dubbed here PL), created by Tels in the 2010-14 timeframe. This is easily deployed under Linux that has Perl capabilities. Version 17 Is current.
# The C++ program "gen_lang_plus" (dubbed here PLUS), a re-implementation with minor improvements created by Geep in 2025. This is lightly Windows-specific at this time. (Please volunteer if you want to make it run under Linux.) Version 1.0 is current.

The purpose of both is as follows.
* Read and write files in child folder "strings". (Caution: "not strings/fm", even if you specify the "fm" option.)
* Read an "all.lang" file, either that associated with the main-menu-system, or with a specific FM under development. This UTF-8 input file has translations of strings for all TDM-supported languages (currently 17). There are individual sections for each target language. Each string is of form

"#str_<id>" "<string content>" // optional comment.
* Generate a set of individual "<language>.lang" files, with the appropriate translated strings. Each file is generated in a standard 8-bit encoding (e.g., defined by one of the iso-8859 standards) designated for the target language. If a translator did not provide a given string in the target language, the master English string is automatically substituted.
* Finally, as requested by command-line options, various analyses can be performed. For example, a set of "missing_<language>.txt" files can be generated, to suggest where translation efforts are still needed.
This article details both programs, with important differences noted. Running this functionality is intended to be done, infrequently, by the person overseeing translation improvements to all.lang.

For more quick overviews of this process, see
* the wiki article [[Internationalization]]. This also covers the i18n.pl program, which is beyond the scope here.
* the preamble of the all.lang file, found in tdm_base01.pk4's \strings\.

It is helpful to mention what these two programs do '''not''' take into account:
* the <language>.map file that reroutes European codepoints from a standardized encoding to the TDM-specific custom 8-bit encoding discussed in [[I18N_-_Charset]].
* what UTF8 characters are allowed by that TDM-specific coding (as opposed to the ISO-8859 encoding).
* the font and fontsize a specific string will use, and what limitations that entails, and what within-DAT glyph-substitutions may be in effect.

== Runtime Environment for PLUS ==
To function in all aspects, PLUS requires Windows with certain UTF8 capabilities, introduced in later versions of Windows 10. Because the PLUS build is told explicitly to use UTF8, in theory what your "system locale" is set to should not matter. In particular, you are not required you set the system locale to the Windows 11 beta UTF8 codepage. (PLUS was developed on a dev box with the English (US) locale.)

You may run it under the traditional Windows console. However, that will not display UTF8 characters correctly on screen, which is mainly an issue with the -charstats and -charpunts options. Instead, use Windows Terminal (or Powershell with UTF8 encoding). Or pipe the console output to a file (i.e., "> myfile.txt"), that can be viewed as UTF8 with Notepad and other text editors.

== Overview of Command-Line Options ==

===For both PL and PLUS===
--csv Generate CSV file
--missing Generate files with lists of missing strings for each language.
--charstats Generate statistics about foreign characters.
--show-ranges Show occupied string index ranges.
--show-holes Show holes in the string index ranges.

With PLUS, one may use a single "-" before an option, instead of "--", and/or leave off the "show-" prefix. (Regarding "charstats", for consistency, since this option outputs to screen, PL should have used the "show-" prefix, but didn't. PLUS does, optionally.)

===For PLUS Only===
-old-preamble Keep <language>.lang preamble just like that of gen_lang.pl. Don't include file generation date or PLUS version number.
-charpunts ''or'' --show-charpunts Reports lines in which "?" was substituted for any character.
-charpunts-extra ''or'' --show-charpunts-extra Like charpunt, but includes "// comment" characters.
-fm Program used for an fm, not main menu; affects what's considered a valid numeric #str_<id> range.
-cvs-unsorted Like -csv, but row order stays like that of all.lang [English].

== Charpunts and Charpunts-Extra Options [PLUS Only] ==
During generation of a particular <language>.lang file, if a valid data line contains 1 or more characters that can't be rendered in the target encoding, then a 3-line report is issued to console. Specifically:
-charpunts Considers characters within the line's content (excluding comments).
-charpunts-extra Same as charpunts, but also considers characters within the line's "// comment".

When entering these options, you may use the "show-" prefix if you'd like, with 1 or 2 leading dashes.

By "// comment", is meant the optional comment that starts with "//" at the end of a #str_<id> line (but still excluding any other comment forms; these latter won't be present in <language>.lang).

Note – it is assumed that #str_<id> contains only 0-9, A-Z, a-z, or underline... so is not inspected by these options.

An example report for [Hungarian] encoded as iso-8859-2:

Output line 177...
from: "#str_02208" "Küldetési idõ"
to: "#str_02208" "K�ldet�si id?"

Important: in this example, only õ, the last character in the "from:" line, could not be encoded in iso-8859-2 and so is replaced by "?" in the "to:" line. The "to:" line characters indicated by � were successfully encoded, but now are not valid UTF8, so can no longer be shown as such. That is, the "to:" line has the identical string placed into the hungarian.lang file.

Here is an -charpunts-extra example where only the comment has characters that don't encode, in this case into iso-8859-1 for [English]:

Output line 396...
from: "#str_02482" "Srpski"// Serbian (Српски - same exception as Russian)
to: "#str_02482" "Srpski"// Serbian (?????? - same exception as Russian)

TIP: If the results of this option is too voluminous, recall that output from a console app can be directed into a file using "... > filename.txt".

These reports give the output line number within <language>.lang, rather than the input line in all.lang, because of difficulties getting the latter value to be correct in the face of, for instance, multiline comments. (Any line numbers that PL reports are usually off.)

== Old Preamble Option [PLUS Only] ==
Each <language>.lang file gets a standard preamble. For PL, the first line looks like this example:
// String table - english (iso-8859-1)

For PLUS, the first line by default also has UTC time and program version, like this example:
// String table - english (iso-8859-1) - created on 2025-02-15T04:00:21Z with gen_lang_plus v. 1.0

But you can ask PLUS to use PL's format, with the "-old-preamble" option. Why? To simplify comparing a particular <language>.lang file generated with both programs.
Since this is more a debugging option, it's not listed in the help, just here.

== Missing Option ==
This option alerts translators to untranslated strings. For each language (other than English, the master) it generates a file named "missing_<language>.txt, with lists of missing UTF8 strings. Each line has the quoted #str_<id> and quoted content, but not comments.

===What's Considered "Missing"?===
This categorization was established in PL and now mimicked in PLUS.

''If for a given master [English] #str_<id>, there's no matching #str_<id> in the target language.''

It's "missing", unless there's a (hard-coded) exception:

* The master string's content is in the "master_non_translatable" group, namely:
** An empty or whitespaces-only string.
** "TDM" or "The Dark Mod".
** A prefix designating a function key ("F") or port ("AUX" or "JOY") followed by 1 or 2 digits.
** A special numeric format like "320x400;640x480" or "123;345" or "1x;2x".
* The master string's content is a font selector, i.e., beginning with "fonts/".
* The #str_<id> has a numeric <id> within a "master_id_non_translatable" range. Just one range, 02460 – 02490, is so reserved, for language names.

''If the #str_<id> in English and the target language have identical content (not counting comments)''

It's "missing", unless there's an exception:

* The target line has a "//" comment indicating "stays the same"
* The "master_non_translatable", font selector, or "master_id_non_translatable" exclusions apply.

''Otherwise, it's not missing.''

That covers this additional case, where for the matching #str_<id> in English and the target language, there's different content (ignoring comments). Surprisingly, this is considered "not missing" even when the English content exists and the target language content is an empty string.

Rarely, the master content can be empty, but the translated string may or may not be. An example is #str_02313, representing an optional second line needed for some languages.

== CSV Options ==
These options read the contents of all.lang and rearrange them for export as a comma-separated-values "tdm_all_lang.csv" UTF8 file in the "strings" folder. The options have different generated row order:

-csv Rows sorted by #str_<id>; seriously buggy ordering in PL, works in PLUS.
-csv-unsorted PLUS only. Keeps row order of [English] in all.lang

The exported CSV file can then be imported into other tools for translators, e.g., Excel spreadsheets. Be aware as you process this data further: maintaining the UTF8 (or other Unicode) encoding ensures character integrity.

=== Generated CVS File Format ===
The first line of the file has the field (aka column) headers, with languages in alphabetic order:

ID,catalan,czech,danish,dutch,english,french,german,hungarian,italian,polish,portuguese,romanian,russian,slovak,spanish,swedish,turkish,comment
There are 19 comma-separated fields here, so the results is very wide. (Recall in a spreadsheet, you can hide columns not of current interest.)

Broadly, the .csv format comes in different variants. Traditionally, as here, comma is the separator. If a field contains a comma, the field overall is wrapped in double quotes. PL also so wraps if there's a contained double-quote, colon, or semi-colon. PLUS mimics that policy. PL unfortunately only considers the language fields for wrapping. PLUS goes a bit further, to consider the "comment" field.

So, after the header each subsequent line has fields separated by commas, where:

* The first field, "ID", has #str_<id>. Unlike in all.lang, there are no enclosing double quotes.
* Remaining fields have translations. As expected, language fields are in the alphabetic header order shown, not necessarily the [language] section order in all.lang. English is in field 5. A field might be empty, have translation text, or have a copy of the master [English] text. (see "What's in Each Language Field" below). There are enclosing double quotes as needed.
* The last field has any comment from the master {English], beginning with "//". If there was no comment, a trailing comma ends the line.

An example row with some empty fields, for the English word "Error", with no comment:

#str_02000,Error.,,Fejl,,Error,Erreur,Fehler,Hiba,Errore,Błąd,Erro,Eroare,Ошибка,Chyba,,,Hata,
Comments on source lines in sections other than [english] are not included.

=== Sorted and Unsorted Row Orders ===

With the -csv option:

* For PLUS, the row order is sorted by #str_<id>.
* While PL intended the row order to be sorted by #str_<id>, the actual sorting is scattershot. (A cause may be the more-recent presence of numerous non-numeric IDs. The sort uses a custom comparator, which knows only about numeric IDs. Other IDs are "tied" in order, not good for many sort algorithms. The Perl code that writes the <language>.lang files uses a better comparator.) The workaround is to import the data into your spreadsheet (keeping UTF8 encoding) and do row sorting there.

With the -csv-unsorted option [PLUS only]:

* Sometimes, it is more helpful to have rows in the same order as in the master [english] section of all.lang, e.g., not necessarily sorted; so perhaps meaningfully grouped. (Caution: in all.lang, there is sometimes a preceding full-line comment that describes a given grouping. That will not be present in the .csv file.)

=== What's in Each Language Field ===
As discussed above, a field will have enclosing double quotes as needed.

The master "english" field, will simply have the [English] content (which could in rare cases be empty).

The situation for other languages is more complex. PL appears, in its separate csv implementation, to be following a middle course, between:
* the "always use translation if available; otherwise English" policy for <language>.lang generation.
* the categorization of the Missing Option.

To make clear the difference, we repeat the Missing Options description, but with '''lines crossed out''' below. (PLUS follows what PL does here, but FYI, there is commented-out code that could implement the crossed-out conditions.)

A field considered "missing" will be empty (e.g., between 2 commas).

''If for a given master [English] #str_<id>, there's no matching #str_<id> in the target language.''

It's "missing", unless there's a (hard-coded) exception:
* The master string's content is in the "master_non_translatable" group, namely:
** An empty or whitespaces-only string.
** "TDM" or "The Dark Mod".
** A prefix designating a function key ("F") or port ("AUX" or "JOY") followed by 1 or 2 digits.
** A special numeric format like "320x400;640x480" or "123;345" or "1x;2x".
* <s>The master string's content is a font selector, i.e., beginning with "fonts/".</s>
* <s>The #str_<id> has a numeric <id> within a "master_id_non_translatable" range. Just one range, 2460 – 2490, is so reserved, for language names.</s>

In this case, if it's not missing, the '''master string's content''' will be used.

''If the #str_<id> in English and the target language have identical content (not counting comments)''

It's "missing", unless there's an exception:
* <s>The target line has a "// comment" indicating "stays the same".</s>
* The "master_non_translatable", <s>font selector, or "master_id_non_translatable"</s> exclusions apply.

In this case, if it's not missing, the '''translated string content (same as master string's content)''' will be used. As indicated, any "stays the same" phrase in the line's comment within the [language] section is ignored.

''Otherwise, it's not missing.''

The '''translated string's content''' will be used. This is the normal "translation provided" case, where for the matching #str_<id> in English and the target language, there's different content (ignoring comments). Note that if English content exists but the target language content is an empty string, while this categorization would call it "not missing", it still has the same result: nothing between the commmas.

=== Differences in Comments between PL and PLUS ===
As noted above , PLUS will enclose the comment field in double quotes if needed. Also, if the comment field contains any unescaped double-quotes, they are backslash-escaped.
PL will do none of that. This is a bug. When the .csv file is imported into a spreadsheet, extra spurious columns may result, requiring manual fix-up.
Also, there is a difference in handling of how characters appear in comments. PLUS will preserve UTF8 characters. PL will replace characters not in ISO-8859-1 with "?" (resulting in the same appearance as in english.lang).

== Charstats Option ==
Creating bitmap font glyphs can be a time-consuming process. This option provides information to suggest, among European (i.e., non-ASCII, non-Cyrillic) characters, which ones should take priority in that process, based on what translators have provided to all.lang.

The results go to the console. PL and PLUS formats differ. PL groups compactly by "Top 5", "Top 10", and so on, and is not particularly Unicode-aware.
PLUS gives each character its own line, with rank and additional data. (And the output here is in utf8 form, better viewed in Windows Terminal than the traditional console. Or pipe to a text file that, e.g., Notepad can treat as UTF8.) Example output:

Non-ASCII UTF8 characters in 'all.lang' across all languages except [Russian],
by descending frequency. (Excludes characters in comments.)

Rank, unicode codepoint, utf8 char, frequency (i.e., count out of 13563 total listed)

1 U+00e1 á 1504
2 U+00e9 é 1420
3 U+00ed í 1099
4 U+0131 ı 1075
5 U+00fc ü 755
6 U+010d č 644
7 U+00f3 ó 498
8 U+00f6 ö 418
9 U+00fa ú 407
....
93 U+0152 Œ 1
94 U+0158 Ř 1
95 U+0179 Ź 1

== Ranges Option and Holes Option ==
Traditionally, #str_<id>s had only 5-digit numeric values. When a new string needed to be added (i.e., to the [English] master), it was helpful to know what numeric ranges were already fully in use, and conversely, what ranges are unoccupied, dubbed "holes". With these options, that info gets reported to the console. Note that for PL, "holes" reports only if "ranges" is not given as an option. PLUS removes that restriction.
PL processes and reports <id>s in [English] listing order; if that is non-monotonic, this can cause anomalies for the "holes" output. To fix that, PLUS does a pre-sort into monotonic order.
PL and PLUS formats differ slightly. An example PLUS output for "-ranges":

There are 48 occupied numeric #str_ ranges in [English]:

1000-1019 (with 20 entries)
1050-1062 (with 13 entries)
1500-1503 (with 4 entries)
2000-2016 (with 17 entries)
...
8980-8988 (with 9 entries)
10000-10162 (with 163 entries)
10180-10185 (with 6 entries)

Also, there are 54 non-numeric #str_ entries.

In the last line, you see that, unlike PL, PLUS is aware of non-numeric <id>s, but just reports them as a total.
An example PLUS output for "-holes":

There are 44 unoccupied numeric #str_ ranges (aka holes) in [English]:

0-999 (with 1000 openings)
1020-1049 (with 30 openings)
1063-1499 (with 437 openings)
1504-1999 (with 496 openings)
....
8989-9999 (with 1011 openings)
10163-10179 (with 17 openings)
10185-19999 (with 9814 openings)

The range from 20000-99999 is reserved for FMs. Using #str_<id> in that range in all.lang will trigger a fatal error (with the –range or –hole option). (See also the FM Option.)

== FM Option [PLUS only] ==
By default, when either the -ranges or -holes option is in effect, the valid range for #str_<id> numeric values is enforced by PLUS as 0-19999. If you are using this for FM strings, select this option to make the valid range 20000-99999.

''These days, many but not all main menu #str_<id>s contain a substring "_menu_". There could be future enforcement of alphanumeric substrings.''
== Details about Main File Formats and Processing ==

=== All.lang File Input Format ===
The file begins with a preamble (in C-style comment form) of notes for translators, some of which we restate here. Then an opening "{" by itself, followed by a series of language sections. Each starts with a line with one of these headers (shown in the order currently seen in the main menu all.lang file):

[English]
[German]
[French]
[Italian]
[Spanish
[Polish]
[Romanian]
[Russian]
[Portuguese}
[Czech]
[Hungarian]
[Slovak]
[Swedish]
[Danish]
[Dutch]
[Turkish]
[Catalan]

Case of the language name doesn't matter, nor does order in which sections are listed. But it is a convention to list English first. After the header comes a series of convertible lines and comment lines, as discussed in next sections. The last line of the file has the closing "}".

=== Converting Each Line ===
Within any particular language section of all.lang, a convertible line will have in this order:

# A #str_<id>, in double quotes, where <id> is an unsigned integer, or a string composed of ASCII characters a-z, A-Z, 0-9, or underscore. Preferred for the latter are all lower case with underscores as word separators. Camel-case is tolerated, but not spaces or dashes. Ignoring the "#str_" prefix, the <id> must be no less than 5 and no more than 63 characters long. Leading zeros should be used to pad a numeric <id> to 5 digits. [Note: program i18n.pl possibly deals only with numeric <id>s.]
# The language-specific UTF8 <string content>, in double quotes
# An optional comment beginning with "//". The next section covers overall treatment of comments.

These are led and separated by whitespace. A leading tab is normal, with tab separators.

No particular ordering of #str_<id>s is required, though ordering numeric <id>s (at least within sub-groups) is desirable.
==== Duplicates Across Lines Within a Section ====
If a particular <id> value appears more than once in a section, the last value is the effective one. A console warning will generally occur during parsing, in the case of PLUS like:
Warning - #str_02518 redefined within [english]...
from: malformed
to: 100

In the [English] section, there might be separate <id>s with the same content. This will generally cause a warning, in the case of PLUS like:
Caution - These strings have the same content ('Master'): #str_08206, #str_03008

To avoid the latter warning, a group of duplicates can be assigned a group number and hard-coded in the "master_double_exceptions" list. Only one group of ten members is so defined currently, those with a required empty string as content. More generally (says a PL comment): "Sometimes it can make sense to have two different strings as when they are translated into different strings in another language. A [theoretical] example would be "Saw" (I saw something, ich hab etwas gesehen) and "Saw" (Saw, Säge)."

==== Substituting English when No Translation ====
Ideally, a #str_<id> in the master [English] section has a match in a target language's section. Then the translated content will always be used for encoding and listing in the <language>.lang file. If no match, the English content is substituted. (Note that only a subset of these substituted strings will categorized as "Missing" if the -missing option is used. Also note that the substitution policy is slightly different with the -csv option.)

==== Conversion Differences between PL and PLUS ====
* To do encoding, PL uses the Perl Encode module, that can autoload language-specific encode/decode tables.
* In the C++ implementation, source file "UTF8_8859_convert.cpp" includes similar (albeit custom) tables for the codepages TDM supports. The overall approach is convert from UTF8 to UTF16, then to the target ISO encoding. That process does not use <locale> facets (which are problematic for our purposes), but standard C++ library "codecvt_utf8_utf16" and "wstring_convert" capabilities.

=== The <language>.Lang File Output Format ===
* A preamble states the language name and specific ISO encoding.
* This is followed by the "#str_<id> ..." lines; sorted in alphabetic order; <id> in numeric form will be listed before those in alphabetic form. Among the latter, an upper case letter will sort before all lower case letters.
* Trailing "//" comments on "#str_<id>" lines from all.lang are preserved.
* Comments of the form /*...*/ in all.lang are skipped, whether spanning multiple lines or part of a single line.
* Blank lines, or lines with only whitespace or a "//" comment, are likewise skipped. The latter comments typically contain headers to organize groups of items in all.lang, but that organization is not retained, replaced by alphanumeric ordering as mentioned.

==== Output File Differences between PL and PLUS ====
These differences are not thought to have functional implications:

* The first line of the preamble will differ unless PLUS's -old_preamble option is used.
* Between a string content and any comment, PLUS preserves tabs and spaces found in all.lang, while PL substitutes two tabs. This is means that simple file comparison programs (like Windows "fc", even with /T or /W options) will find lots of trivial differences. It is suggested you use a more sophisticated comparison tool, for instance, Notepad++ with the ComparePlus plugin and "Ignore changed spaces" selected.

== Notes on PLUS Internals ==
A hand-crafted conversion from a static analysis of PL, PLUS preserves many variable names, some reimagined as functions. Befitting a straightforward conversion, it offers a procedural, C-like style, without much in the way of a class hierarchy, overarching object-oriented design pattern, or advanced C++ features. (This may be of benefit if part or all of its functionality is ever integrated into the TDM engine and/or DR.)
Not much effort was spent on performance optimization. However, as a compiled language, it will of course run much faster than PL. Also, the C++ implementation avoids use of regular expressions, which are heavily featured in PL and are notoriously slow and also hard for most coders to read.

Platform-independent C++ standard-library structures like vectors, (ordered) sets, maps, and multimaps are employed. However, to do the UTF8 to ISO conversions, a Windows-specific approach is taken, to convert via unencoding from utf8 to wide characters and then re-encoding to target ISOs. (While it is possible to do such conversions entirely within 8-bit characters, the resulting code is a puzzle box, best left to third-party libraries.) Instead of using a third-party library, the encode/decode implementation here includes use of the "deprecated" C++ standard "codecvt" feature; while deprecated, the C++ standard committee has not (as of 2025) come up with a viable replacement.

Like PL, PLUS has some DEBUG statements defined (requiring recompile for C++). These reflect specific concerns during development and testing and are of limited general interest.

== Downloads ==
For PLUS v1.1 of 2025, July 16 (mainly adds Catalan as a supported language)

* [https://drive.google.com/file/d/1R_qajkfq5_drIGEMYQ8ahpJwfN1LdkiT/view?usp=sharing gen_lang_plus_source_v1.1.zip]. This is a Visual Studio project. Includes open-source license.
* [https://drive.google.com/file/d/1kj_zE9VN7l1DMztnOHx-UaNbIAECIE6M/view?usp=sharing gen_lang_plus.exe]

For PLUS v1.0 of 2025, Feb 28:

* [https://drive.google.com/file/d/1V_FSMEBYsinXPMKlH3_OmzcS8f1zjVeL/view?usp=sharing gen_lang_plus_source_v1.0.zip]. This is a Visual Studio project.
* [https://drive.google.com/file/d/1xsas9kZ-6tRdNdFhKHS736xYK5-aYGMg/view?usp=sharing gen_lang_plus.exe]

== See Also ==
The Perl program "i18n.pl" contains similar functionality for FM-specific strings, but also parses and edits the FM .map file, so is considerably more complicated.

Gen Lang Programs

2025-07-16T18:24:19Z

Geep: /* Downloads */ Add links for v1.1

By Geep, from February 2025
== Introduction to Gen_lang.pl ("PL") and Gen_lang_plus ("PLUS") ==
This describes two software implementations of command-line programs that essentially do the same thing;
# The Perl program "gen_lang.pl" (dubbed here PL), created by Tels in the 2010-14 timeframe. This is easily deployed under Linux that has Perl capabilities. Version 17 Is current.
# The C++ program "gen_lang_plus" (dubbed here PLUS), a re-implementation with minor improvements created by Geep in 2025. This is lightly Windows-specific at this time. (Please volunteer if you want to make it run under Linux.) Version 1.0 is current.

The purpose of both is as follows.
* Read and write files in child folder "strings". (Caution: "not strings/fm", even if you specify the "fm" option.)
* Read an "all.lang" file, either that associated with the main-menu-system, or with a specific FM under development. This UTF-8 input file has translations of strings for all TDM-supported languages (currently 17). There are individual sections for each target language. Each string is of form

"#str_<id>" "<string content>" // optional comment.
* Generate a set of individual "<language>.lang" files, with the appropriate translated strings. Each file is generated in a standard 8-bit encoding (e.g., defined by one of the iso-8859 standards) designated for the target language. If a translator did not provide a given string in the target language, the master English string is automatically substituted.
* Finally, as requested by command-line options, various analyses can be performed. For example, a set of "missing_<language>.txt" files can be generated, to suggest where translation efforts are still needed.
This article details both programs, with important differences noted. Running this functionality is intended to be done, infrequently, by the person overseeing translation improvements to all.lang.

For more quick overviews of this process, see
* the wiki article [[Internationalization]]. This also covers the i18n.pl program, which is beyond the scope here.
* the preamble of the all.lang file, found in tdm_base01.pk4's \strings\.

It is helpful to mention what these two programs do '''not''' take into account:
* the <language>.map file that reroutes European codepoints from a standardized encoding to the TDM-specific custom 8-bit encoding discussed in [[I18N_-_Charset]].
* what UTF8 characters are allowed by that TDM-specific coding (as opposed to the ISO-8859 encoding).
* the font and fontsize a specific string will use, and what limitations that entails, and what within-DAT glyph-substitutions may be in effect.

== Runtime Environment for PLUS ==
To function in all aspects, PLUS requires Windows with certain UTF8 capabilities, introduced in later versions of Windows 10. Because the PLUS build is told explicitly to use UTF8, in theory what your "system locale" is set to should not matter. In particular, you are not required you set the system locale to the Windows 11 beta UTF8 codepage. (PLUS was developed on a dev box with the English (US) locale.)

You may run it under the traditional Windows console. However, that will not display UTF8 characters correctly on screen, which is mainly an issue with the -charstats and -charpunts options. Instead, use Windows Terminal (or Powershell with UTF8 encoding). Or pipe the console output to a file (i.e., "> myfile.txt"), that can be viewed as UTF8 with Notepad and other text editors.

== Overview of Command-Line Options ==

===For both PL and PLUS===
--csv Generate CSV file
--missing Generate files with lists of missing strings for each language.
--charstats Generate statistics about foreign characters.
--show-ranges Show occupied string index ranges.
--show-holes Show holes in the string index ranges.

With PLUS, one may use a single "-" before an option, instead of "--", and/or leave off the "show-" prefix. (Regarding "charstats", for consistency, since this option outputs to screen, PL should have used the "show-" prefix, but didn't. PLUS does, optionally.)

===For PLUS Only===
-old-preamble Keep <language>.lang preamble just like that of gen_lang.pl. Don't include file generation date or PLUS version number.
-charpunts ''or'' --show-charpunts Reports lines in which "?" was substituted for any character.
-charpunts-extra ''or'' --show-charpunts-extra Like charpunt, but includes "// comment" characters.
-fm Program used for an fm, not main menu; affects what's considered a valid numeric #str_<id> range.
-cvs-unsorted Like -csv, but row order stays like that of all.lang [English].

== Charpunts and Charpunts-Extra Options [PLUS Only] ==
During generation of a particular <language>.lang file, if a valid data line contains 1 or more characters that can't be rendered in the target encoding, then a 3-line report is issued to console. Specifically:
-charpunts Considers characters within the line's content (excluding comments).
-charpunts-extra Same as charpunts, but also considers characters within the line's "// comment".

When entering these options, you may use the "show-" prefix if you'd like, with 1 or 2 leading dashes.

By "// comment", is meant the optional comment that starts with "//" at the end of a #str_<id> line (but still excluding any other comment forms; these latter won't be present in <language>.lang).

Note – it is assumed that #str_<id> contains only 0-9, A-Z, a-z, or underline... so is not inspected by these options.

An example report for [Hungarian] encoded as iso-8859-2:

Output line 177...
from: "#str_02208" "Küldetési idõ"
to: "#str_02208" "K�ldet�si id?"

Important: in this example, only õ, the last character in the "from:" line, could not be encoded in iso-8859-2 and so is replaced by "?" in the "to:" line. The "to:" line characters indicated by � were successfully encoded, but now are not valid UTF8, so can no longer be shown as such. That is, the "to:" line has the identical string placed into the hungarian.lang file.

Here is an -charpunts-extra example where only the comment has characters that don't encode, in this case into iso-8859-1 for [English]:

Output line 396...
from: "#str_02482" "Srpski"// Serbian (Српски - same exception as Russian)
to: "#str_02482" "Srpski"// Serbian (?????? - same exception as Russian)

TIP: If the results of this option is too voluminous, recall that output from a console app can be directed into a file using "... > filename.txt".

These reports give the output line number within <language>.lang, rather than the input line in all.lang, because of difficulties getting the latter value to be correct in the face of, for instance, multiline comments. (Any line numbers that PL reports are usually off.)

== Old Preamble Option [PLUS Only] ==
Each <language>.lang file gets a standard preamble. For PL, the first line looks like this example:
// String table - english (iso-8859-1)

For PLUS, the first line by default also has UTC time and program version, like this example:
// String table - english (iso-8859-1) - created on 2025-02-15T04:00:21Z with gen_lang_plus v. 1.0

But you can ask PLUS to use PL's format, with the "-old-preamble" option. Why? To simplify comparing a particular <language>.lang file generated with both programs.
Since this is more a debugging option, it's not listed in the help, just here.

== Missing Option ==
This option alerts translators to untranslated strings. For each language (other than English, the master) it generates a file named "missing_<language>.txt, with lists of missing UTF8 strings. Each line has the quoted #str_<id> and quoted content, but not comments.

===What's Considered "Missing"?===
This categorization was established in PL and now mimicked in PLUS.

''If for a given master [English] #str_<id>, there's no matching #str_<id> in the target language.''

It's "missing", unless there's a (hard-coded) exception:

* The master string's content is in the "master_non_translatable" group, namely:
** An empty or whitespaces-only string.
** "TDM" or "The Dark Mod".
** A prefix designating a function key ("F") or port ("AUX" or "JOY") followed by 1 or 2 digits.
** A special numeric format like "320x400;640x480" or "123;345" or "1x;2x".
* The master string's content is a font selector, i.e., beginning with "fonts/".
* The #str_<id> has a numeric <id> within a "master_id_non_translatable" range. Just one range, 02460 – 02490, is so reserved, for language names.

''If the #str_<id> in English and the target language have identical content (not counting comments)''

It's "missing", unless there's an exception:

* The target line has a "//" comment indicating "stays the same"
* The "master_non_translatable", font selector, or "master_id_non_translatable" exclusions apply.

''Otherwise, it's not missing.''

That covers this additional case, where for the matching #str_<id> in English and the target language, there's different content (ignoring comments). Surprisingly, this is considered "not missing" even when the English content exists and the target language content is an empty string.

Rarely, the master content can be empty, but the translated string may or may not be. An example is #str_02313, representing an optional second line needed for some languages.

== CSV Options ==
These options read the contents of all.lang and rearrange them for export as a comma-separated-values "tdm_all_lang.csv" UTF8 file in the "strings" folder. The options have different generated row order:

-csv Rows sorted by #str_<id>; seriously buggy ordering in PL, works in PLUS.
-csv-unsorted PLUS only. Keeps row order of [English] in all.lang

The exported CSV file can then be imported into other tools for translators, e.g., Excel spreadsheets. Be aware as you process this data further: maintaining the UTF8 (or other Unicode) encoding ensures character integrity.

=== Generated CVS File Format ===
The first line of the file has the field (aka column) headers, with languages in alphabetic order:

ID,catalan,czech,danish,dutch,english,french,german,hungarian,italian,polish,portuguese,romanian,russian,slovak,spanish,swedish,turkish,comment
There are 19 comma-separated fields here, so the results is very wide. (Recall in a spreadsheet, you can hide columns not of current interest.)

Broadly, the .csv format comes in different variants. Traditionally, as here, comma is the separator. If a field contains a comma, the field overall is wrapped in double quotes. PL also so wraps if there's a contained double-quote, colon, or semi-colon. PLUS mimics that policy. PL unfortunately only considers the language fields for wrapping. PLUS goes a bit further, to consider the "comment" field.

So, after the header each subsequent line has fields separated by commas, where:

* The first field, "ID", has #str_<id>. Unlike in all.lang, there are no enclosing double quotes.
* Remaining fields have translations. As expected, language fields are in the alphabetic header order shown, not necessarily the [language] section order in all.lang. English is in field 5. A field might be empty, have translation text, or have a copy of the master [English] text. (see "What's in Each Language Field" below). There are enclosing double quotes as needed.
* The last field has any comment from the master {English], beginning with "//". If there was no comment, a trailing comma ends the line.

An example row with some empty fields, for the English word "Error", with no comment:

#str_02000,Error.,,Fejl,,Error,Erreur,Fehler,Hiba,Errore,Błąd,Erro,Eroare,Ошибка,Chyba,,,Hata,
Comments on source lines in sections other than [english] are not included.

=== Sorted and Unsorted Row Orders ===

With the -csv option:

* For PLUS, the row order is sorted by #str_<id>.
* While PL intended the row order to be sorted by #str_<id>, the actual sorting is scattershot. (A cause may be the more-recent presence of numerous non-numeric IDs. The sort uses a custom comparator, which knows only about numeric IDs. Other IDs are "tied" in order, not good for many sort algorithms. The Perl code that writes the <language>.lang files uses a better comparator.) The workaround is to import the data into your spreadsheet (keeping UTF8 encoding) and do row sorting there.

With the -csv-unsorted option [PLUS only]:

* Sometimes, it is more helpful to have rows in the same order as in the master [english] section of all.lang, e.g., not necessarily sorted; so perhaps meaningfully grouped. (Caution: in all.lang, there is sometimes a preceding full-line comment that describes a given grouping. That will not be present in the .csv file.)

=== What's in Each Language Field ===
As discussed above, a field will have enclosing double quotes as needed.

The master "english" field, will simply have the [English] content (which could in rare cases be empty).

The situation for other languages is more complex. PL appears, in its separate csv implementation, to be following a middle course, between:
* the "always use translation if available; otherwise English" policy for <language>.lang generation.
* the categorization of the Missing Option.

To make clear the difference, we repeat the Missing Options description, but with '''lines crossed out''' below. (PLUS follows what PL does here, but FYI, there is commented-out code that could implement the crossed-out conditions.)

A field considered "missing" will be empty (e.g., between 2 commas).

''If for a given master [English] #str_<id>, there's no matching #str_<id> in the target language.''

It's "missing", unless there's a (hard-coded) exception:
* The master string's content is in the "master_non_translatable" group, namely:
** An empty or whitespaces-only string.
** "TDM" or "The Dark Mod".
** A prefix designating a function key ("F") or port ("AUX" or "JOY") followed by 1 or 2 digits.
** A special numeric format like "320x400;640x480" or "123;345" or "1x;2x".
* <s>The master string's content is a font selector, i.e., beginning with "fonts/".</s>
* <s>The #str_<id> has a numeric <id> within a "master_id_non_translatable" range. Just one range, 2460 – 2490, is so reserved, for language names.</s>

In this case, if it's not missing, the '''master string's content''' will be used.

''If the #str_<id> in English and the target language have identical content (not counting comments)''

It's "missing", unless there's an exception:
* <s>The target line has a "// comment" indicating "stays the same".</s>
* The "master_non_translatable", <s>font selector, or "master_id_non_translatable"</s> exclusions apply.

In this case, if it's not missing, the '''translated string content (same as master string's content)''' will be used. As indicated, any "stays the same" phrase in the line's comment within the [language] section is ignored.

''Otherwise, it's not missing.''

The '''translated string's content''' will be used. This is the normal "translation provided" case, where for the matching #str_<id> in English and the target language, there's different content (ignoring comments). Note that if English content exists but the target language content is an empty string, while this categorization would call it "not missing", it still has the same result: nothing between the commmas.

=== Differences in Comments between PL and PLUS ===
As noted above , PLUS will enclose the comment field in double quotes if needed. Also, if the comment field contains any unescaped double-quotes, they are backslash-escaped.
PL will do none of that. This is a bug. When the .csv file is imported into a spreadsheet, extra spurious columns may result, requiring manual fix-up.
Also, there is a difference in handling of how characters appear in comments. PLUS will preserve UTF8 characters. PL will replace characters not in ISO-8859-1 with "?" (resulting in the same appearance as in english.lang).

== Charstats Option ==
Creating bitmap font glyphs can be a time-consuming process. This option provides information to suggest, among European (i.e., non-ASCII, non-Cyrillic) characters, which ones should take priority in that process, based on what translators have provided to all.lang.

The results go to the console. PL and PLUS formats differ. PL groups compactly by "Top 5", "Top 10", and so on, and is not particularly Unicode-aware.
PLUS gives each character its own line, with rank and additional data. (And the output here is in utf8 form, better viewed in Windows Terminal than the traditional console. Or pipe to a text file that, e.g., Notepad can treat as UTF8.) Example output:

Non-ASCII UTF8 characters in 'all.lang' across all languages except [Russian],
by descending frequency. (Excludes characters in comments.)

Rank, unicode codepoint, utf8 char, frequency (i.e., count out of 13563 total listed)

1 U+00e1 á 1504
2 U+00e9 é 1420
3 U+00ed í 1099
4 U+0131 ı 1075
5 U+00fc ü 755
6 U+010d č 644
7 U+00f3 ó 498
8 U+00f6 ö 418
9 U+00fa ú 407
....
93 U+0152 Œ 1
94 U+0158 Ř 1
95 U+0179 Ź 1

== Ranges Option and Holes Option ==
Traditionally, #str_<id>s had only 5-digit numeric values. When a new string needed to be added (i.e., to the [English] master), it was helpful to know what numeric ranges were already fully in use, and conversely, what ranges are unoccupied, dubbed "holes". With these options, that info gets reported to the console. Note that for PL, "holes" reports only if "ranges" is not given as an option. PLUS removes that restriction.
PL processes and reports <id>s in [English] listing order; if that is non-monotonic, this can cause anomalies for the "holes" output. To fix that, PLUS does a pre-sort into monotonic order.
PL and PLUS formats differ slightly. An example PLUS output for "-ranges":

There are 48 occupied numeric #str_ ranges in [English]:

1000-1019 (with 20 entries)
1050-1062 (with 13 entries)
1500-1503 (with 4 entries)
2000-2016 (with 17 entries)
...
8980-8988 (with 9 entries)
10000-10162 (with 163 entries)
10180-10185 (with 6 entries)

Also, there are 54 non-numeric #str_ entries.

In the last line, you see that, unlike PL, PLUS is aware of non-numeric <id>s, but just reports them as a total.
An example PLUS output for "-holes":

There are 44 unoccupied numeric #str_ ranges (aka holes) in [English]:

0-999 (with 1000 openings)
1020-1049 (with 30 openings)
1063-1499 (with 437 openings)
1504-1999 (with 496 openings)
....
8989-9999 (with 1011 openings)
10163-10179 (with 17 openings)
10185-19999 (with 9814 openings)

The range from 20000-99999 is reserved for FMs. Using #str_<id> in that range in all.lang will trigger a fatal error (with the –range or –hole option). (See also the FM Option.)

== FM Option [PLUS only] ==
By default, when either the -ranges or -holes option is in effect, the valid range for #str_<id> numeric values is enforced by PLUS as 0-19999. If you are using this for FM strings, select this option to make the valid range 20000-99999.

''These days, many but not all main menu #str_<id>s contain a substring "_menu_". There could be future enforcement of alphanumeric substrings.''
== Details about Main File Formats and Processing ==

=== All.lang File Input Format ===
The file begins with a preamble (in C-style comment form) of notes for translators, some of which we restate here. Then an opening "{" by itself, followed by a series of language sections. Each starts with a line with one of these headers (shown in the order currently seen in the main menu all.lang file):

[English]
[German]
[French]
[Italian]
[Spanish
[Polish]
[Romanian]
[Russian]
[Portuguese}
[Czech]
[Hungarian]
[Slovak]
[Swedish]
[Danish]
[Dutch]
[Turkish]

Case of the language name doesn't matter, nor does order in which sections are listed. But it is a convention to list English first. After the header comes a series of convertible lines and comment lines, as discussed in next sections. The last line of the file has the closing "}".

=== Converting Each Line ===
Within any particular language section of all.lang, a convertible line will have in this order:

# A #str_<id>, in double quotes, where <id> is an unsigned integer, or a string composed of ASCII characters a-z, A-Z, 0-9, or underscore. Preferred for the latter are all lower case with underscores as word separators. Camel-case is tolerated, but not spaces or dashes. Ignoring the "#str_" prefix, the <id> must be no less than 5 and no more than 63 characters long. Leading zeros should be used to pad a numeric <id> to 5 digits. [Note: program i18n.pl possibly deals only with numeric <id>s.]
# The language-specific UTF8 <string content>, in double quotes
# An optional comment beginning with "//". The next section covers overall treatment of comments.

These are led and separated by whitespace. A leading tab is normal, with tab separators.

No particular ordering of #str_<id>s is required, though ordering numeric <id>s (at least within sub-groups) is desirable.
==== Duplicates Across Lines Within a Section ====
If a particular <id> value appears more than once in a section, the last value is the effective one. A console warning will generally occur during parsing, in the case of PLUS like:
Warning - #str_02518 redefined within [english]...
from: malformed
to: 100

In the [English] section, there might be separate <id>s with the same content. This will generally cause a warning, in the case of PLUS like:
Caution - These strings have the same content ('Master'): #str_08206, #str_03008

To avoid the latter warning, a group of duplicates can be assigned a group number and hard-coded in the "master_double_exceptions" list. Only one group of ten members is so defined currently, those with a required empty string as content. More generally (says a PL comment): "Sometimes it can make sense to have two different strings as when they are translated into different strings in another language. A [theoretical] example would be "Saw" (I saw something, ich hab etwas gesehen) and "Saw" (Saw, Säge)."

==== Substituting English when No Translation ====
Ideally, a #str_<id> in the master [English] section has a match in a target language's section. Then the translated content will always be used for encoding and listing in the <language>.lang file. If no match, the English content is substituted. (Note that only a subset of these substituted strings will categorized as "Missing" if the -missing option is used. Also note that the substitution policy is slightly different with the -csv option.)

==== Conversion Differences between PL and PLUS ====
* To do encoding, PL uses the Perl Encode module, that can autoload language-specific encode/decode tables.
* In the C++ implementation, source file "UTF8_8859_convert.cpp" includes similar (albeit custom) tables for the codepages TDM supports. The overall approach is convert from UTF8 to UTF16, then to the target ISO encoding. That process does not use <locale> facets (which are problematic for our purposes), but standard C++ library "codecvt_utf8_utf16" and "wstring_convert" capabilities.

=== The <language>.Lang File Output Format ===
* A preamble states the language name and specific ISO encoding.
* This is followed by the "#str_<id> ..." lines; sorted in alphabetic order; <id> in numeric form will be listed before those in alphabetic form. Among the latter, an upper case letter will sort before all lower case letters.
* Trailing "//" comments on "#str_<id>" lines from all.lang are preserved.
* Comments of the form /*...*/ in all.lang are skipped, whether spanning multiple lines or part of a single line.
* Blank lines, or lines with only whitespace or a "//" comment, are likewise skipped. The latter comments typically contain headers to organize groups of items in all.lang, but that organization is not retained, replaced by alphanumeric ordering as mentioned.

==== Output File Differences between PL and PLUS ====
These differences are not thought to have functional implications:

* The first line of the preamble will differ unless PLUS's -old_preamble option is used.
* Between a string content and any comment, PLUS preserves tabs and spaces found in all.lang, while PL substitutes two tabs. This is means that simple file comparison programs (like Windows "fc", even with /T or /W options) will find lots of trivial differences. It is suggested you use a more sophisticated comparison tool, for instance, Notepad++ with the ComparePlus plugin and "Ignore changed spaces" selected.

== Notes on PLUS Internals ==
A hand-crafted conversion from a static analysis of PL, PLUS preserves many variable names, some reimagined as functions. Befitting a straightforward conversion, it offers a procedural, C-like style, without much in the way of a class hierarchy, overarching object-oriented design pattern, or advanced C++ features. (This may be of benefit if part or all of its functionality is ever integrated into the TDM engine and/or DR.)
Not much effort was spent on performance optimization. However, as a compiled language, it will of course run much faster than PL. Also, the C++ implementation avoids use of regular expressions, which are heavily featured in PL and are notoriously slow and also hard for most coders to read.

Platform-independent C++ standard-library structures like vectors, (ordered) sets, maps, and multimaps are employed. However, to do the UTF8 to ISO conversions, a Windows-specific approach is taken, to convert via unencoding from utf8 to wide characters and then re-encoding to target ISOs. (While it is possible to do such conversions entirely within 8-bit characters, the resulting code is a puzzle box, best left to third-party libraries.) Instead of using a third-party library, the encode/decode implementation here includes use of the "deprecated" C++ standard "codecvt" feature; while deprecated, the C++ standard committee has not (as of 2025) come up with a viable replacement.

Like PL, PLUS has some DEBUG statements defined (requiring recompile for C++). These reflect specific concerns during development and testing and are of limited general interest.

== Downloads ==
For PLUS v1.1 of 2025, July 16 (mainly adds Catalan as a supported language)

* [https://drive.google.com/file/d/1R_qajkfq5_drIGEMYQ8ahpJwfN1LdkiT/view?usp=sharing gen_lang_plus_source_v1.1.zip]. This is a Visual Studio project. Includes open-source license.
* [https://drive.google.com/file/d/1kj_zE9VN7l1DMztnOHx-UaNbIAECIE6M/view?usp=sharing gen_lang_plus.exe]

For PLUS v1.0 of 2025, Feb 28:

* [https://drive.google.com/file/d/1V_FSMEBYsinXPMKlH3_OmzcS8f1zjVeL/view?usp=sharing gen_lang_plus_source_v1.0.zip]. This is a Visual Studio project.
* [https://drive.google.com/file/d/1xsas9kZ-6tRdNdFhKHS736xYK5-aYGMg/view?usp=sharing gen_lang_plus.exe]

== See Also ==
The Perl program "i18n.pl" contains similar functionality for FM-specific strings, but also parses and edits the FM .map file, so is considerably more complicated.

Gen Lang Programs

2025-07-16T16:48:56Z

Geep: /* Charpunts and Charpunts-Extra Options [PLUS Only] */

By Geep, from February 2025
== Introduction to Gen_lang.pl ("PL") and Gen_lang_plus ("PLUS") ==
This describes two software implementations of command-line programs that essentially do the same thing;
# The Perl program "gen_lang.pl" (dubbed here PL), created by Tels in the 2010-14 timeframe. This is easily deployed under Linux that has Perl capabilities. Version 17 Is current.
# The C++ program "gen_lang_plus" (dubbed here PLUS), a re-implementation with minor improvements created by Geep in 2025. This is lightly Windows-specific at this time. (Please volunteer if you want to make it run under Linux.) Version 1.0 is current.

The purpose of both is as follows.
* Read and write files in child folder "strings". (Caution: "not strings/fm", even if you specify the "fm" option.)
* Read an "all.lang" file, either that associated with the main-menu-system, or with a specific FM under development. This UTF-8 input file has translations of strings for all TDM-supported languages (currently 17). There are individual sections for each target language. Each string is of form

"#str_<id>" "<string content>" // optional comment.
* Generate a set of individual "<language>.lang" files, with the appropriate translated strings. Each file is generated in a standard 8-bit encoding (e.g., defined by one of the iso-8859 standards) designated for the target language. If a translator did not provide a given string in the target language, the master English string is automatically substituted.
* Finally, as requested by command-line options, various analyses can be performed. For example, a set of "missing_<language>.txt" files can be generated, to suggest where translation efforts are still needed.
This article details both programs, with important differences noted. Running this functionality is intended to be done, infrequently, by the person overseeing translation improvements to all.lang.

For more quick overviews of this process, see
* the wiki article [[Internationalization]]. This also covers the i18n.pl program, which is beyond the scope here.
* the preamble of the all.lang file, found in tdm_base01.pk4's \strings\.

It is helpful to mention what these two programs do '''not''' take into account:
* the <language>.map file that reroutes European codepoints from a standardized encoding to the TDM-specific custom 8-bit encoding discussed in [[I18N_-_Charset]].
* what UTF8 characters are allowed by that TDM-specific coding (as opposed to the ISO-8859 encoding).
* the font and fontsize a specific string will use, and what limitations that entails, and what within-DAT glyph-substitutions may be in effect.

== Runtime Environment for PLUS ==
To function in all aspects, PLUS requires Windows with certain UTF8 capabilities, introduced in later versions of Windows 10. Because the PLUS build is told explicitly to use UTF8, in theory what your "system locale" is set to should not matter. In particular, you are not required you set the system locale to the Windows 11 beta UTF8 codepage. (PLUS was developed on a dev box with the English (US) locale.)

You may run it under the traditional Windows console. However, that will not display UTF8 characters correctly on screen, which is mainly an issue with the -charstats and -charpunts options. Instead, use Windows Terminal (or Powershell with UTF8 encoding). Or pipe the console output to a file (i.e., "> myfile.txt"), that can be viewed as UTF8 with Notepad and other text editors.

== Overview of Command-Line Options ==

===For both PL and PLUS===
--csv Generate CSV file
--missing Generate files with lists of missing strings for each language.
--charstats Generate statistics about foreign characters.
--show-ranges Show occupied string index ranges.
--show-holes Show holes in the string index ranges.

With PLUS, one may use a single "-" before an option, instead of "--", and/or leave off the "show-" prefix. (Regarding "charstats", for consistency, since this option outputs to screen, PL should have used the "show-" prefix, but didn't. PLUS does, optionally.)

===For PLUS Only===
-old-preamble Keep <language>.lang preamble just like that of gen_lang.pl. Don't include file generation date or PLUS version number.
-charpunts ''or'' --show-charpunts Reports lines in which "?" was substituted for any character.
-charpunts-extra ''or'' --show-charpunts-extra Like charpunt, but includes "// comment" characters.
-fm Program used for an fm, not main menu; affects what's considered a valid numeric #str_<id> range.
-cvs-unsorted Like -csv, but row order stays like that of all.lang [English].

== Charpunts and Charpunts-Extra Options [PLUS Only] ==
During generation of a particular <language>.lang file, if a valid data line contains 1 or more characters that can't be rendered in the target encoding, then a 3-line report is issued to console. Specifically:
-charpunts Considers characters within the line's content (excluding comments).
-charpunts-extra Same as charpunts, but also considers characters within the line's "// comment".

When entering these options, you may use the "show-" prefix if you'd like, with 1 or 2 leading dashes.

By "// comment", is meant the optional comment that starts with "//" at the end of a #str_<id> line (but still excluding any other comment forms; these latter won't be present in <language>.lang).

Note – it is assumed that #str_<id> contains only 0-9, A-Z, a-z, or underline... so is not inspected by these options.

An example report for [Hungarian] encoded as iso-8859-2:

Output line 177...
from: "#str_02208" "Küldetési idõ"
to: "#str_02208" "K�ldet�si id?"

Important: in this example, only õ, the last character in the "from:" line, could not be encoded in iso-8859-2 and so is replaced by "?" in the "to:" line. The "to:" line characters indicated by � were successfully encoded, but now are not valid UTF8, so can no longer be shown as such. That is, the "to:" line has the identical string placed into the hungarian.lang file.

Here is an -charpunts-extra example where only the comment has characters that don't encode, in this case into iso-8859-1 for [English]:

Output line 396...
from: "#str_02482" "Srpski"// Serbian (Српски - same exception as Russian)
to: "#str_02482" "Srpski"// Serbian (?????? - same exception as Russian)

TIP: If the results of this option is too voluminous, recall that output from a console app can be directed into a file using "... > filename.txt".

These reports give the output line number within <language>.lang, rather than the input line in all.lang, because of difficulties getting the latter value to be correct in the face of, for instance, multiline comments. (Any line numbers that PL reports are usually off.)

== Old Preamble Option [PLUS Only] ==
Each <language>.lang file gets a standard preamble. For PL, the first line looks like this example:
// String table - english (iso-8859-1)

For PLUS, the first line by default also has UTC time and program version, like this example:
// String table - english (iso-8859-1) - created on 2025-02-15T04:00:21Z with gen_lang_plus v. 1.0

But you can ask PLUS to use PL's format, with the "-old-preamble" option. Why? To simplify comparing a particular <language>.lang file generated with both programs.
Since this is more a debugging option, it's not listed in the help, just here.

== Missing Option ==
This option alerts translators to untranslated strings. For each language (other than English, the master) it generates a file named "missing_<language>.txt, with lists of missing UTF8 strings. Each line has the quoted #str_<id> and quoted content, but not comments.

===What's Considered "Missing"?===
This categorization was established in PL and now mimicked in PLUS.

''If for a given master [English] #str_<id>, there's no matching #str_<id> in the target language.''

It's "missing", unless there's a (hard-coded) exception:

* The master string's content is in the "master_non_translatable" group, namely:
** An empty or whitespaces-only string.
** "TDM" or "The Dark Mod".
** A prefix designating a function key ("F") or port ("AUX" or "JOY") followed by 1 or 2 digits.
** A special numeric format like "320x400;640x480" or "123;345" or "1x;2x".
* The master string's content is a font selector, i.e., beginning with "fonts/".
* The #str_<id> has a numeric <id> within a "master_id_non_translatable" range. Just one range, 02460 – 02490, is so reserved, for language names.

''If the #str_<id> in English and the target language have identical content (not counting comments)''

It's "missing", unless there's an exception:

* The target line has a "//" comment indicating "stays the same"
* The "master_non_translatable", font selector, or "master_id_non_translatable" exclusions apply.

''Otherwise, it's not missing.''

That covers this additional case, where for the matching #str_<id> in English and the target language, there's different content (ignoring comments). Surprisingly, this is considered "not missing" even when the English content exists and the target language content is an empty string.

Rarely, the master content can be empty, but the translated string may or may not be. An example is #str_02313, representing an optional second line needed for some languages.

== CSV Options ==
These options read the contents of all.lang and rearrange them for export as a comma-separated-values "tdm_all_lang.csv" UTF8 file in the "strings" folder. The options have different generated row order:

-csv Rows sorted by #str_<id>; seriously buggy ordering in PL, works in PLUS.
-csv-unsorted PLUS only. Keeps row order of [English] in all.lang

The exported CSV file can then be imported into other tools for translators, e.g., Excel spreadsheets. Be aware as you process this data further: maintaining the UTF8 (or other Unicode) encoding ensures character integrity.

=== Generated CVS File Format ===
The first line of the file has the field (aka column) headers, with languages in alphabetic order:

ID,catalan,czech,danish,dutch,english,french,german,hungarian,italian,polish,portuguese,romanian,russian,slovak,spanish,swedish,turkish,comment
There are 19 comma-separated fields here, so the results is very wide. (Recall in a spreadsheet, you can hide columns not of current interest.)

Broadly, the .csv format comes in different variants. Traditionally, as here, comma is the separator. If a field contains a comma, the field overall is wrapped in double quotes. PL also so wraps if there's a contained double-quote, colon, or semi-colon. PLUS mimics that policy. PL unfortunately only considers the language fields for wrapping. PLUS goes a bit further, to consider the "comment" field.

So, after the header each subsequent line has fields separated by commas, where:

* The first field, "ID", has #str_<id>. Unlike in all.lang, there are no enclosing double quotes.
* Remaining fields have translations. As expected, language fields are in the alphabetic header order shown, not necessarily the [language] section order in all.lang. English is in field 5. A field might be empty, have translation text, or have a copy of the master [English] text. (see "What's in Each Language Field" below). There are enclosing double quotes as needed.
* The last field has any comment from the master {English], beginning with "//". If there was no comment, a trailing comma ends the line.

An example row with some empty fields, for the English word "Error", with no comment:

#str_02000,Error.,,Fejl,,Error,Erreur,Fehler,Hiba,Errore,Błąd,Erro,Eroare,Ошибка,Chyba,,,Hata,
Comments on source lines in sections other than [english] are not included.

=== Sorted and Unsorted Row Orders ===

With the -csv option:

* For PLUS, the row order is sorted by #str_<id>.
* While PL intended the row order to be sorted by #str_<id>, the actual sorting is scattershot. (A cause may be the more-recent presence of numerous non-numeric IDs. The sort uses a custom comparator, which knows only about numeric IDs. Other IDs are "tied" in order, not good for many sort algorithms. The Perl code that writes the <language>.lang files uses a better comparator.) The workaround is to import the data into your spreadsheet (keeping UTF8 encoding) and do row sorting there.

With the -csv-unsorted option [PLUS only]:

* Sometimes, it is more helpful to have rows in the same order as in the master [english] section of all.lang, e.g., not necessarily sorted; so perhaps meaningfully grouped. (Caution: in all.lang, there is sometimes a preceding full-line comment that describes a given grouping. That will not be present in the .csv file.)

=== What's in Each Language Field ===
As discussed above, a field will have enclosing double quotes as needed.

The master "english" field, will simply have the [English] content (which could in rare cases be empty).

The situation for other languages is more complex. PL appears, in its separate csv implementation, to be following a middle course, between:
* the "always use translation if available; otherwise English" policy for <language>.lang generation.
* the categorization of the Missing Option.

To make clear the difference, we repeat the Missing Options description, but with '''lines crossed out''' below. (PLUS follows what PL does here, but FYI, there is commented-out code that could implement the crossed-out conditions.)

A field considered "missing" will be empty (e.g., between 2 commas).

''If for a given master [English] #str_<id>, there's no matching #str_<id> in the target language.''

It's "missing", unless there's a (hard-coded) exception:
* The master string's content is in the "master_non_translatable" group, namely:
** An empty or whitespaces-only string.
** "TDM" or "The Dark Mod".
** A prefix designating a function key ("F") or port ("AUX" or "JOY") followed by 1 or 2 digits.
** A special numeric format like "320x400;640x480" or "123;345" or "1x;2x".
* <s>The master string's content is a font selector, i.e., beginning with "fonts/".</s>
* <s>The #str_<id> has a numeric <id> within a "master_id_non_translatable" range. Just one range, 2460 – 2490, is so reserved, for language names.</s>

In this case, if it's not missing, the '''master string's content''' will be used.

''If the #str_<id> in English and the target language have identical content (not counting comments)''

It's "missing", unless there's an exception:
* <s>The target line has a "// comment" indicating "stays the same".</s>
* The "master_non_translatable", <s>font selector, or "master_id_non_translatable"</s> exclusions apply.

In this case, if it's not missing, the '''translated string content (same as master string's content)''' will be used. As indicated, any "stays the same" phrase in the line's comment within the [language] section is ignored.

''Otherwise, it's not missing.''

The '''translated string's content''' will be used. This is the normal "translation provided" case, where for the matching #str_<id> in English and the target language, there's different content (ignoring comments). Note that if English content exists but the target language content is an empty string, while this categorization would call it "not missing", it still has the same result: nothing between the commmas.

=== Differences in Comments between PL and PLUS ===
As noted above , PLUS will enclose the comment field in double quotes if needed. Also, if the comment field contains any unescaped double-quotes, they are backslash-escaped.
PL will do none of that. This is a bug. When the .csv file is imported into a spreadsheet, extra spurious columns may result, requiring manual fix-up.
Also, there is a difference in handling of how characters appear in comments. PLUS will preserve UTF8 characters. PL will replace characters not in ISO-8859-1 with "?" (resulting in the same appearance as in english.lang).

== Charstats Option ==
Creating bitmap font glyphs can be a time-consuming process. This option provides information to suggest, among European (i.e., non-ASCII, non-Cyrillic) characters, which ones should take priority in that process, based on what translators have provided to all.lang.

The results go to the console. PL and PLUS formats differ. PL groups compactly by "Top 5", "Top 10", and so on, and is not particularly Unicode-aware.
PLUS gives each character its own line, with rank and additional data. (And the output here is in utf8 form, better viewed in Windows Terminal than the traditional console. Or pipe to a text file that, e.g., Notepad can treat as UTF8.) Example output:

Non-ASCII UTF8 characters in 'all.lang' across all languages except [Russian],
by descending frequency. (Excludes characters in comments.)

Rank, unicode codepoint, utf8 char, frequency (i.e., count out of 13563 total listed)

1 U+00e1 á 1504
2 U+00e9 é 1420
3 U+00ed í 1099
4 U+0131 ı 1075
5 U+00fc ü 755
6 U+010d č 644
7 U+00f3 ó 498
8 U+00f6 ö 418
9 U+00fa ú 407
....
93 U+0152 Œ 1
94 U+0158 Ř 1
95 U+0179 Ź 1

== Ranges Option and Holes Option ==
Traditionally, #str_<id>s had only 5-digit numeric values. When a new string needed to be added (i.e., to the [English] master), it was helpful to know what numeric ranges were already fully in use, and conversely, what ranges are unoccupied, dubbed "holes". With these options, that info gets reported to the console. Note that for PL, "holes" reports only if "ranges" is not given as an option. PLUS removes that restriction.
PL processes and reports <id>s in [English] listing order; if that is non-monotonic, this can cause anomalies for the "holes" output. To fix that, PLUS does a pre-sort into monotonic order.
PL and PLUS formats differ slightly. An example PLUS output for "-ranges":

There are 48 occupied numeric #str_ ranges in [English]:

1000-1019 (with 20 entries)
1050-1062 (with 13 entries)
1500-1503 (with 4 entries)
2000-2016 (with 17 entries)
...
8980-8988 (with 9 entries)
10000-10162 (with 163 entries)
10180-10185 (with 6 entries)

Also, there are 54 non-numeric #str_ entries.

In the last line, you see that, unlike PL, PLUS is aware of non-numeric <id>s, but just reports them as a total.
An example PLUS output for "-holes":

There are 44 unoccupied numeric #str_ ranges (aka holes) in [English]:

0-999 (with 1000 openings)
1020-1049 (with 30 openings)
1063-1499 (with 437 openings)
1504-1999 (with 496 openings)
....
8989-9999 (with 1011 openings)
10163-10179 (with 17 openings)
10185-19999 (with 9814 openings)

The range from 20000-99999 is reserved for FMs. Using #str_<id> in that range in all.lang will trigger a fatal error (with the –range or –hole option). (See also the FM Option.)

== FM Option [PLUS only] ==
By default, when either the -ranges or -holes option is in effect, the valid range for #str_<id> numeric values is enforced by PLUS as 0-19999. If you are using this for FM strings, select this option to make the valid range 20000-99999.

''These days, many but not all main menu #str_<id>s contain a substring "_menu_". There could be future enforcement of alphanumeric substrings.''
== Details about Main File Formats and Processing ==

=== All.lang File Input Format ===
The file begins with a preamble (in C-style comment form) of notes for translators, some of which we restate here. Then an opening "{" by itself, followed by a series of language sections. Each starts with a line with one of these headers (shown in the order currently seen in the main menu all.lang file):

[English]
[German]
[French]
[Italian]
[Spanish
[Polish]
[Romanian]
[Russian]
[Portuguese}
[Czech]
[Hungarian]
[Slovak]
[Swedish]
[Danish]
[Dutch]
[Turkish]

Case of the language name doesn't matter, nor does order in which sections are listed. But it is a convention to list English first. After the header comes a series of convertible lines and comment lines, as discussed in next sections. The last line of the file has the closing "}".

=== Converting Each Line ===
Within any particular language section of all.lang, a convertible line will have in this order:

# A #str_<id>, in double quotes, where <id> is an unsigned integer, or a string composed of ASCII characters a-z, A-Z, 0-9, or underscore. Preferred for the latter are all lower case with underscores as word separators. Camel-case is tolerated, but not spaces or dashes. Ignoring the "#str_" prefix, the <id> must be no less than 5 and no more than 63 characters long. Leading zeros should be used to pad a numeric <id> to 5 digits. [Note: program i18n.pl possibly deals only with numeric <id>s.]
# The language-specific UTF8 <string content>, in double quotes
# An optional comment beginning with "//". The next section covers overall treatment of comments.

These are led and separated by whitespace. A leading tab is normal, with tab separators.

No particular ordering of #str_<id>s is required, though ordering numeric <id>s (at least within sub-groups) is desirable.
==== Duplicates Across Lines Within a Section ====
If a particular <id> value appears more than once in a section, the last value is the effective one. A console warning will generally occur during parsing, in the case of PLUS like:
Warning - #str_02518 redefined within [english]...
from: malformed
to: 100

In the [English] section, there might be separate <id>s with the same content. This will generally cause a warning, in the case of PLUS like:
Caution - These strings have the same content ('Master'): #str_08206, #str_03008

To avoid the latter warning, a group of duplicates can be assigned a group number and hard-coded in the "master_double_exceptions" list. Only one group of ten members is so defined currently, those with a required empty string as content. More generally (says a PL comment): "Sometimes it can make sense to have two different strings as when they are translated into different strings in another language. A [theoretical] example would be "Saw" (I saw something, ich hab etwas gesehen) and "Saw" (Saw, Säge)."

==== Substituting English when No Translation ====
Ideally, a #str_<id> in the master [English] section has a match in a target language's section. Then the translated content will always be used for encoding and listing in the <language>.lang file. If no match, the English content is substituted. (Note that only a subset of these substituted strings will categorized as "Missing" if the -missing option is used. Also note that the substitution policy is slightly different with the -csv option.)

==== Conversion Differences between PL and PLUS ====
* To do encoding, PL uses the Perl Encode module, that can autoload language-specific encode/decode tables.
* In the C++ implementation, source file "UTF8_8859_convert.cpp" includes similar (albeit custom) tables for the codepages TDM supports. The overall approach is convert from UTF8 to UTF16, then to the target ISO encoding. That process does not use <locale> facets (which are problematic for our purposes), but standard C++ library "codecvt_utf8_utf16" and "wstring_convert" capabilities.

=== The <language>.Lang File Output Format ===
* A preamble states the language name and specific ISO encoding.
* This is followed by the "#str_<id> ..." lines; sorted in alphabetic order; <id> in numeric form will be listed before those in alphabetic form. Among the latter, an upper case letter will sort before all lower case letters.
* Trailing "//" comments on "#str_<id>" lines from all.lang are preserved.
* Comments of the form /*...*/ in all.lang are skipped, whether spanning multiple lines or part of a single line.
* Blank lines, or lines with only whitespace or a "//" comment, are likewise skipped. The latter comments typically contain headers to organize groups of items in all.lang, but that organization is not retained, replaced by alphanumeric ordering as mentioned.

==== Output File Differences between PL and PLUS ====
These differences are not thought to have functional implications:

* The first line of the preamble will differ unless PLUS's -old_preamble option is used.
* Between a string content and any comment, PLUS preserves tabs and spaces found in all.lang, while PL substitutes two tabs. This is means that simple file comparison programs (like Windows "fc", even with /T or /W options) will find lots of trivial differences. It is suggested you use a more sophisticated comparison tool, for instance, Notepad++ with the ComparePlus plugin and "Ignore changed spaces" selected.

== Notes on PLUS Internals ==
A hand-crafted conversion from a static analysis of PL, PLUS preserves many variable names, some reimagined as functions. Befitting a straightforward conversion, it offers a procedural, C-like style, without much in the way of a class hierarchy, overarching object-oriented design pattern, or advanced C++ features. (This may be of benefit if part or all of its functionality is ever integrated into the TDM engine and/or DR.)
Not much effort was spent on performance optimization. However, as a compiled language, it will of course run much faster than PL. Also, the C++ implementation avoids use of regular expressions, which are heavily featured in PL and are notoriously slow and also hard for most coders to read.

Platform-independent C++ standard-library structures like vectors, (ordered) sets, maps, and multimaps are employed. However, to do the UTF8 to ISO conversions, a Windows-specific approach is taken, to convert via unencoding from utf8 to wide characters and then re-encoding to target ISOs. (While it is possible to do such conversions entirely within 8-bit characters, the resulting code is a puzzle box, best left to third-party libraries.) Instead of using a third-party library, the encode/decode implementation here includes use of the "deprecated" C++ standard "codecvt" feature; while deprecated, the C++ standard committee has not (as of 2025) come up with a viable replacement.

Like PL, PLUS has some DEBUG statements defined (requiring recompile for C++). These reflect specific concerns during development and testing and are of limited general interest.

== Downloads ==
For PLUS v1.1 of 2025, July 16 (mainly adds Catalan as a supported language)

[soon]

For PLUS v1.0 of 2025, Feb 28:

* [https://drive.google.com/file/d/1V_FSMEBYsinXPMKlH3_OmzcS8f1zjVeL/view?usp=sharing gen_lang_plus_source_v1.0.zip]. This is a Visual Studio project.
* [https://drive.google.com/file/d/1xsas9kZ-6tRdNdFhKHS736xYK5-aYGMg/view?usp=sharing gen_lang_plus.exe]

== See Also ==
The Perl program "i18n.pl" contains similar functionality for FM-specific strings, but also parses and edits the FM .map file, so is considerably more complicated.

Gen Lang Programs

2025-07-16T16:43:51Z

Geep: Clarify use of strings folder; Get ready for v1.1 release.with Catalan

By Geep, from February 2025
== Introduction to Gen_lang.pl ("PL") and Gen_lang_plus ("PLUS") ==
This describes two software implementations of command-line programs that essentially do the same thing;
# The Perl program "gen_lang.pl" (dubbed here PL), created by Tels in the 2010-14 timeframe. This is easily deployed under Linux that has Perl capabilities. Version 17 Is current.
# The C++ program "gen_lang_plus" (dubbed here PLUS), a re-implementation with minor improvements created by Geep in 2025. This is lightly Windows-specific at this time. (Please volunteer if you want to make it run under Linux.) Version 1.0 is current.

The purpose of both is as follows.
* Read and write files in child folder "strings". (Caution: "not strings/fm", even if you specify the "fm" option.)
* Read an "all.lang" file, either that associated with the main-menu-system, or with a specific FM under development. This UTF-8 input file has translations of strings for all TDM-supported languages (currently 17). There are individual sections for each target language. Each string is of form

"#str_<id>" "<string content>" // optional comment.
* Generate a set of individual "<language>.lang" files, with the appropriate translated strings. Each file is generated in a standard 8-bit encoding (e.g., defined by one of the iso-8859 standards) designated for the target language. If a translator did not provide a given string in the target language, the master English string is automatically substituted.
* Finally, as requested by command-line options, various analyses can be performed. For example, a set of "missing_<language>.txt" files can be generated, to suggest where translation efforts are still needed.
This article details both programs, with important differences noted. Running this functionality is intended to be done, infrequently, by the person overseeing translation improvements to all.lang.

For more quick overviews of this process, see
* the wiki article [[Internationalization]]. This also covers the i18n.pl program, which is beyond the scope here.
* the preamble of the all.lang file, found in tdm_base01.pk4's \strings\.

It is helpful to mention what these two programs do '''not''' take into account:
* the <language>.map file that reroutes European codepoints from a standardized encoding to the TDM-specific custom 8-bit encoding discussed in [[I18N_-_Charset]].
* what UTF8 characters are allowed by that TDM-specific coding (as opposed to the ISO-8859 encoding).
* the font and fontsize a specific string will use, and what limitations that entails, and what within-DAT glyph-substitutions may be in effect.

== Runtime Environment for PLUS ==
To function in all aspects, PLUS requires Windows with certain UTF8 capabilities, introduced in later versions of Windows 10. Because the PLUS build is told explicitly to use UTF8, in theory what your "system locale" is set to should not matter. In particular, you are not required you set the system locale to the Windows 11 beta UTF8 codepage. (PLUS was developed on a dev box with the English (US) locale.)

You may run it under the traditional Windows console. However, that will not display UTF8 characters correctly on screen, which is mainly an issue with the -charstats and -charpunts options. Instead, use Windows Terminal (or Powershell with UTF8 encoding). Or pipe the console output to a file (i.e., "> myfile.txt"), that can be viewed as UTF8 with Notepad and other text editors.

== Overview of Command-Line Options ==

===For both PL and PLUS===
--csv Generate CSV file
--missing Generate files with lists of missing strings for each language.
--charstats Generate statistics about foreign characters.
--show-ranges Show occupied string index ranges.
--show-holes Show holes in the string index ranges.

With PLUS, one may use a single "-" before an option, instead of "--", and/or leave off the "show-" prefix. (Regarding "charstats", for consistency, since this option outputs to screen, PL should have used the "show-" prefix, but didn't. PLUS does, optionally.)

===For PLUS Only===
-old-preamble Keep <language>.lang preamble just like that of gen_lang.pl. Don't include file generation date or PLUS version number.
-charpunts ''or'' --show-charpunts Reports lines in which "?" was substituted for any character.
-charpunts-extra ''or'' --show-charpunts-extra Like charpunt, but includes "// comment" characters.
-fm Program used for an fm, not main menu; affects what's considered a valid numeric #str_<id> range.
-cvs-unsorted Like -csv, but row order stays like that of all.lang [English].

== Charpunts and Charpunts-Extra Options [PLUS Only] ==
During generation of a particular <language>.lang file, if a valid data line contains 1 or more characters that can't be rendered in the target encoding, then a 3-line report is issued to console. Specifically:
-charpunts Considers characters within the line's content (excluding comments).
-charpunts-extra Same as charpunts, but also considers characters within the line's "// comment".

When entering these options, you may use the "show-" prefix if you'd like, with 1 or 2 leading dashes.

By "// comment", is meant the optional comment that starts with "//" at the end of a #str_<id> line (but still excluding any other comment forms; these latter won't be present in <language>.lang).

Note – it is assumed that #str_<id> contains only 0-9, A-Z, a-z, or underline... so is not inspected by these options.

An example report for [Hungarian] encoded as iso-8859-2:

Output line 177...
from: "#str_02208" "Küldetési idõ"
to: "#str_02208" "K�ldet�si id?"

Important: in this example, only õ, the last character in the "from:" line, could not be encoded in iso-8859-2 and so is replaced by "?" in the "to:" line. The "to:" line characters indicated by � were successfully encoded, but now are not valid UTF8, so can no longer be shown as such. That is, the "to:" line has the identical string placed into the hungarian.lang file.

Here is an -charpunts-extra example where only the comment has characters that don't encode, in this case into iso-8859-1 for [English]:

Output line 396...
from: "#str_02482" "Srpski"// Serbian (Српски - same exception as Russian)
to: "#str_02482" "Srpski"// Serbian (?????? - same exception as Russian)

TIP: If the results of this option is too voluminous, recall that output from a console app can be directed into a file using "... > filename.txt".

These reports give the output line number within <language>.lang, rather than the input line in all.lang, because of difficulties getting the latter value right in the face of, for instance, multiline comments. (Any line numbers that PL reports are usually off.)

== Old Preamble Option [PLUS Only] ==
Each <language>.lang file gets a standard preamble. For PL, the first line looks like this example:
// String table - english (iso-8859-1)

For PLUS, the first line by default also has UTC time and program version, like this example:
// String table - english (iso-8859-1) - created on 2025-02-15T04:00:21Z with gen_lang_plus v. 1.0

But you can ask PLUS to use PL's format, with the "-old-preamble" option. Why? To simplify comparing a particular <language>.lang file generated with both programs.
Since this is more a debugging option, it's not listed in the help, just here.

== Missing Option ==
This option alerts translators to untranslated strings. For each language (other than English, the master) it generates a file named "missing_<language>.txt, with lists of missing UTF8 strings. Each line has the quoted #str_<id> and quoted content, but not comments.

===What's Considered "Missing"?===
This categorization was established in PL and now mimicked in PLUS.

''If for a given master [English] #str_<id>, there's no matching #str_<id> in the target language.''

It's "missing", unless there's a (hard-coded) exception:

* The master string's content is in the "master_non_translatable" group, namely:
** An empty or whitespaces-only string.
** "TDM" or "The Dark Mod".
** A prefix designating a function key ("F") or port ("AUX" or "JOY") followed by 1 or 2 digits.
** A special numeric format like "320x400;640x480" or "123;345" or "1x;2x".
* The master string's content is a font selector, i.e., beginning with "fonts/".
* The #str_<id> has a numeric <id> within a "master_id_non_translatable" range. Just one range, 02460 – 02490, is so reserved, for language names.

''If the #str_<id> in English and the target language have identical content (not counting comments)''

It's "missing", unless there's an exception:

* The target line has a "//" comment indicating "stays the same"
* The "master_non_translatable", font selector, or "master_id_non_translatable" exclusions apply.

''Otherwise, it's not missing.''

That covers this additional case, where for the matching #str_<id> in English and the target language, there's different content (ignoring comments). Surprisingly, this is considered "not missing" even when the English content exists and the target language content is an empty string.

Rarely, the master content can be empty, but the translated string may or may not be. An example is #str_02313, representing an optional second line needed for some languages.

== CSV Options ==
These options read the contents of all.lang and rearrange them for export as a comma-separated-values "tdm_all_lang.csv" UTF8 file in the "strings" folder. The options have different generated row order:

-csv Rows sorted by #str_<id>; seriously buggy ordering in PL, works in PLUS.
-csv-unsorted PLUS only. Keeps row order of [English] in all.lang

The exported CSV file can then be imported into other tools for translators, e.g., Excel spreadsheets. Be aware as you process this data further: maintaining the UTF8 (or other Unicode) encoding ensures character integrity.

=== Generated CVS File Format ===
The first line of the file has the field (aka column) headers, with languages in alphabetic order:

ID,catalan,czech,danish,dutch,english,french,german,hungarian,italian,polish,portuguese,romanian,russian,slovak,spanish,swedish,turkish,comment
There are 19 comma-separated fields here, so the results is very wide. (Recall in a spreadsheet, you can hide columns not of current interest.)

Broadly, the .csv format comes in different variants. Traditionally, as here, comma is the separator. If a field contains a comma, the field overall is wrapped in double quotes. PL also so wraps if there's a contained double-quote, colon, or semi-colon. PLUS mimics that policy. PL unfortunately only considers the language fields for wrapping. PLUS goes a bit further, to consider the "comment" field.

So, after the header each subsequent line has fields separated by commas, where:

* The first field, "ID", has #str_<id>. Unlike in all.lang, there are no enclosing double quotes.
* Remaining fields have translations. As expected, language fields are in the alphabetic header order shown, not necessarily the [language] section order in all.lang. English is in field 5. A field might be empty, have translation text, or have a copy of the master [English] text. (see "What's in Each Language Field" below). There are enclosing double quotes as needed.
* The last field has any comment from the master {English], beginning with "//". If there was no comment, a trailing comma ends the line.

An example row with some empty fields, for the English word "Error", with no comment:

#str_02000,Error.,,Fejl,,Error,Erreur,Fehler,Hiba,Errore,Błąd,Erro,Eroare,Ошибка,Chyba,,,Hata,
Comments on source lines in sections other than [english] are not included.

=== Sorted and Unsorted Row Orders ===

With the -csv option:

* For PLUS, the row order is sorted by #str_<id>.
* While PL intended the row order to be sorted by #str_<id>, the actual sorting is scattershot. (A cause may be the more-recent presence of numerous non-numeric IDs. The sort uses a custom comparator, which knows only about numeric IDs. Other IDs are "tied" in order, not good for many sort algorithms. The Perl code that writes the <language>.lang files uses a better comparator.) The workaround is to import the data into your spreadsheet (keeping UTF8 encoding) and do row sorting there.

With the -csv-unsorted option [PLUS only]:

* Sometimes, it is more helpful to have rows in the same order as in the master [english] section of all.lang, e.g., not necessarily sorted; so perhaps meaningfully grouped. (Caution: in all.lang, there is sometimes a preceding full-line comment that describes a given grouping. That will not be present in the .csv file.)

=== What's in Each Language Field ===
As discussed above, a field will have enclosing double quotes as needed.

The master "english" field, will simply have the [English] content (which could in rare cases be empty).

The situation for other languages is more complex. PL appears, in its separate csv implementation, to be following a middle course, between:
* the "always use translation if available; otherwise English" policy for <language>.lang generation.
* the categorization of the Missing Option.

To make clear the difference, we repeat the Missing Options description, but with '''lines crossed out''' below. (PLUS follows what PL does here, but FYI, there is commented-out code that could implement the crossed-out conditions.)

A field considered "missing" will be empty (e.g., between 2 commas).

''If for a given master [English] #str_<id>, there's no matching #str_<id> in the target language.''

It's "missing", unless there's a (hard-coded) exception:
* The master string's content is in the "master_non_translatable" group, namely:
** An empty or whitespaces-only string.
** "TDM" or "The Dark Mod".
** A prefix designating a function key ("F") or port ("AUX" or "JOY") followed by 1 or 2 digits.
** A special numeric format like "320x400;640x480" or "123;345" or "1x;2x".
* <s>The master string's content is a font selector, i.e., beginning with "fonts/".</s>
* <s>The #str_<id> has a numeric <id> within a "master_id_non_translatable" range. Just one range, 2460 – 2490, is so reserved, for language names.</s>

In this case, if it's not missing, the '''master string's content''' will be used.

''If the #str_<id> in English and the target language have identical content (not counting comments)''

It's "missing", unless there's an exception:
* <s>The target line has a "// comment" indicating "stays the same".</s>
* The "master_non_translatable", <s>font selector, or "master_id_non_translatable"</s> exclusions apply.

In this case, if it's not missing, the '''translated string content (same as master string's content)''' will be used. As indicated, any "stays the same" phrase in the line's comment within the [language] section is ignored.

''Otherwise, it's not missing.''

The '''translated string's content''' will be used. This is the normal "translation provided" case, where for the matching #str_<id> in English and the target language, there's different content (ignoring comments). Note that if English content exists but the target language content is an empty string, while this categorization would call it "not missing", it still has the same result: nothing between the commmas.

=== Differences in Comments between PL and PLUS ===
As noted above , PLUS will enclose the comment field in double quotes if needed. Also, if the comment field contains any unescaped double-quotes, they are backslash-escaped.
PL will do none of that. This is a bug. When the .csv file is imported into a spreadsheet, extra spurious columns may result, requiring manual fix-up.
Also, there is a difference in handling of how characters appear in comments. PLUS will preserve UTF8 characters. PL will replace characters not in ISO-8859-1 with "?" (resulting in the same appearance as in english.lang).

== Charstats Option ==
Creating bitmap font glyphs can be a time-consuming process. This option provides information to suggest, among European (i.e., non-ASCII, non-Cyrillic) characters, which ones should take priority in that process, based on what translators have provided to all.lang.

The results go to the console. PL and PLUS formats differ. PL groups compactly by "Top 5", "Top 10", and so on, and is not particularly Unicode-aware.
PLUS gives each character its own line, with rank and additional data. (And the output here is in utf8 form, better viewed in Windows Terminal than the traditional console. Or pipe to a text file that, e.g., Notepad can treat as UTF8.) Example output:

Non-ASCII UTF8 characters in 'all.lang' across all languages except [Russian],
by descending frequency. (Excludes characters in comments.)

Rank, unicode codepoint, utf8 char, frequency (i.e., count out of 13563 total listed)

1 U+00e1 á 1504
2 U+00e9 é 1420
3 U+00ed í 1099
4 U+0131 ı 1075
5 U+00fc ü 755
6 U+010d č 644
7 U+00f3 ó 498
8 U+00f6 ö 418
9 U+00fa ú 407
....
93 U+0152 Œ 1
94 U+0158 Ř 1
95 U+0179 Ź 1

== Ranges Option and Holes Option ==
Traditionally, #str_<id>s had only 5-digit numeric values. When a new string needed to be added (i.e., to the [English] master), it was helpful to know what numeric ranges were already fully in use, and conversely, what ranges are unoccupied, dubbed "holes". With these options, that info gets reported to the console. Note that for PL, "holes" reports only if "ranges" is not given as an option. PLUS removes that restriction.
PL processes and reports <id>s in [English] listing order; if that is non-monotonic, this can cause anomalies for the "holes" output. To fix that, PLUS does a pre-sort into monotonic order.
PL and PLUS formats differ slightly. An example PLUS output for "-ranges":

There are 48 occupied numeric #str_ ranges in [English]:

1000-1019 (with 20 entries)
1050-1062 (with 13 entries)
1500-1503 (with 4 entries)
2000-2016 (with 17 entries)
...
8980-8988 (with 9 entries)
10000-10162 (with 163 entries)
10180-10185 (with 6 entries)

Also, there are 54 non-numeric #str_ entries.

In the last line, you see that, unlike PL, PLUS is aware of non-numeric <id>s, but just reports them as a total.
An example PLUS output for "-holes":

There are 44 unoccupied numeric #str_ ranges (aka holes) in [English]:

0-999 (with 1000 openings)
1020-1049 (with 30 openings)
1063-1499 (with 437 openings)
1504-1999 (with 496 openings)
....
8989-9999 (with 1011 openings)
10163-10179 (with 17 openings)
10185-19999 (with 9814 openings)

The range from 20000-99999 is reserved for FMs. Using #str_<id> in that range in all.lang will trigger a fatal error (with the –range or –hole option). (See also the FM Option.)

== FM Option [PLUS only] ==
By default, when either the -ranges or -holes option is in effect, the valid range for #str_<id> numeric values is enforced by PLUS as 0-19999. If you are using this for FM strings, select this option to make the valid range 20000-99999.

''These days, many but not all main menu #str_<id>s contain a substring "_menu_". There could be future enforcement of alphanumeric substrings.''
== Details about Main File Formats and Processing ==

=== All.lang File Input Format ===
The file begins with a preamble (in C-style comment form) of notes for translators, some of which we restate here. Then an opening "{" by itself, followed by a series of language sections. Each starts with a line with one of these headers (shown in the order currently seen in the main menu all.lang file):

[English]
[German]
[French]
[Italian]
[Spanish
[Polish]
[Romanian]
[Russian]
[Portuguese}
[Czech]
[Hungarian]
[Slovak]
[Swedish]
[Danish]
[Dutch]
[Turkish]

Case of the language name doesn't matter, nor does order in which sections are listed. But it is a convention to list English first. After the header comes a series of convertible lines and comment lines, as discussed in next sections. The last line of the file has the closing "}".

=== Converting Each Line ===
Within any particular language section of all.lang, a convertible line will have in this order:

# A #str_<id>, in double quotes, where <id> is an unsigned integer, or a string composed of ASCII characters a-z, A-Z, 0-9, or underscore. Preferred for the latter are all lower case with underscores as word separators. Camel-case is tolerated, but not spaces or dashes. Ignoring the "#str_" prefix, the <id> must be no less than 5 and no more than 63 characters long. Leading zeros should be used to pad a numeric <id> to 5 digits. [Note: program i18n.pl possibly deals only with numeric <id>s.]
# The language-specific UTF8 <string content>, in double quotes
# An optional comment beginning with "//". The next section covers overall treatment of comments.

These are led and separated by whitespace. A leading tab is normal, with tab separators.

No particular ordering of #str_<id>s is required, though ordering numeric <id>s (at least within sub-groups) is desirable.
==== Duplicates Across Lines Within a Section ====
If a particular <id> value appears more than once in a section, the last value is the effective one. A console warning will generally occur during parsing, in the case of PLUS like:
Warning - #str_02518 redefined within [english]...
from: malformed
to: 100

In the [English] section, there might be separate <id>s with the same content. This will generally cause a warning, in the case of PLUS like:
Caution - These strings have the same content ('Master'): #str_08206, #str_03008

To avoid the latter warning, a group of duplicates can be assigned a group number and hard-coded in the "master_double_exceptions" list. Only one group of ten members is so defined currently, those with a required empty string as content. More generally (says a PL comment): "Sometimes it can make sense to have two different strings as when they are translated into different strings in another language. A [theoretical] example would be "Saw" (I saw something, ich hab etwas gesehen) and "Saw" (Saw, Säge)."

==== Substituting English when No Translation ====
Ideally, a #str_<id> in the master [English] section has a match in a target language's section. Then the translated content will always be used for encoding and listing in the <language>.lang file. If no match, the English content is substituted. (Note that only a subset of these substituted strings will categorized as "Missing" if the -missing option is used. Also note that the substitution policy is slightly different with the -csv option.)

==== Conversion Differences between PL and PLUS ====
* To do encoding, PL uses the Perl Encode module, that can autoload language-specific encode/decode tables.
* In the C++ implementation, source file "UTF8_8859_convert.cpp" includes similar (albeit custom) tables for the codepages TDM supports. The overall approach is convert from UTF8 to UTF16, then to the target ISO encoding. That process does not use <locale> facets (which are problematic for our purposes), but standard C++ library "codecvt_utf8_utf16" and "wstring_convert" capabilities.

=== The <language>.Lang File Output Format ===
* A preamble states the language name and specific ISO encoding.
* This is followed by the "#str_<id> ..." lines; sorted in alphabetic order; <id> in numeric form will be listed before those in alphabetic form. Among the latter, an upper case letter will sort before all lower case letters.
* Trailing "//" comments on "#str_<id>" lines from all.lang are preserved.
* Comments of the form /*...*/ in all.lang are skipped, whether spanning multiple lines or part of a single line.
* Blank lines, or lines with only whitespace or a "//" comment, are likewise skipped. The latter comments typically contain headers to organize groups of items in all.lang, but that organization is not retained, replaced by alphanumeric ordering as mentioned.

==== Output File Differences between PL and PLUS ====
These differences are not thought to have functional implications:

* The first line of the preamble will differ unless PLUS's -old_preamble option is used.
* Between a string content and any comment, PLUS preserves tabs and spaces found in all.lang, while PL substitutes two tabs. This is means that simple file comparison programs (like Windows "fc", even with /T or /W options) will find lots of trivial differences. It is suggested you use a more sophisticated comparison tool, for instance, Notepad++ with the ComparePlus plugin and "Ignore changed spaces" selected.

== Notes on PLUS Internals ==
A hand-crafted conversion from a static analysis of PL, PLUS preserves many variable names, some reimagined as functions. Befitting a straightforward conversion, it offers a procedural, C-like style, without much in the way of a class hierarchy, overarching object-oriented design pattern, or advanced C++ features. (This may be of benefit if part or all of its functionality is ever integrated into the TDM engine and/or DR.)
Not much effort was spent on performance optimization. However, as a compiled language, it will of course run much faster than PL. Also, the C++ implementation avoids use of regular expressions, which are heavily featured in PL and are notoriously slow and also hard for most coders to read.

Platform-independent C++ standard-library structures like vectors, (ordered) sets, maps, and multimaps are employed. However, to do the UTF8 to ISO conversions, a Windows-specific approach is taken, to convert via unencoding from utf8 to wide characters and then re-encoding to target ISOs. (While it is possible to do such conversions entirely within 8-bit characters, the resulting code is a puzzle box, best left to third-party libraries.) Instead of using a third-party library, the encode/decode implementation here includes use of the "deprecated" C++ standard "codecvt" feature; while deprecated, the C++ standard committee has not (as of 2025) come up with a viable replacement.

Like PL, PLUS has some DEBUG statements defined (requiring recompile for C++). These reflect specific concerns during development and testing and are of limited general interest.

== Downloads ==
For PLUS v1.1 of 2025, July 16 (mainly adds Catalan as a supported language)

[soon]

For PLUS v1.0 of 2025, Feb 28:

* [https://drive.google.com/file/d/1V_FSMEBYsinXPMKlH3_OmzcS8f1zjVeL/view?usp=sharing gen_lang_plus_source_v1.0.zip]. This is a Visual Studio project.
* [https://drive.google.com/file/d/1xsas9kZ-6tRdNdFhKHS736xYK5-aYGMg/view?usp=sharing gen_lang_plus.exe]

== See Also ==
The Perl program "i18n.pl" contains similar functionality for FM-specific strings, but also parses and edits the FM .map file, so is considerably more complicated.

Installing and Running Fan Missions

2025-07-01T19:53:54Z

Geep: /* By installing with the in-game downloader */

= Installing a New Mission =
<big>There are two ways to install a new mission:
# By downloading a mission pk4 file and placing it in the fms folder
# By using the in-game mission downloader
</big>

== By downloading the mission PK4 ==
# Go to [https://www.thedarkmod.com/missions/ thedarkmod.com/missions] (say, by selecting "Missions" from the site top menu strip) to see a list of available FMs.
# For an FM of interest, click on "Download Details".
# Each FM is contained in a PK4 file, which contains all the files necessary to run the mission. Choose one of the equivalent links to download, and drop the PK4 file into the fms/ folder, e.g. <tt>C:\Games\darkmod\fms\</tt>.
# Launch The Dark Mod game to '''automatically install the mission'''. That is, upon launch, the game will detect the new mission package in the fms/ folder and will add it to the list of available missions. (Note: the PK4 file will be automatically moved into a subfolder. This is normal operation).

== By installing with the in-game downloader ==
[[Image:Download-mission210.jpg|right|thumb]]
# Launch The Dark Mod.
# From the main menu, click ''Mission List'' (''formerly ''New Mission'', as shown in figures here).
# Click ''Download Missions'' in the lower border of the Mission List.

In this screen on the left is the ''Online Mission Archive'', with a list of FMs that:
* you don't yet have, but are available to download and install; and
* you '''do''' have already installed, but that have an update or new (or at least not-yet-installed) translation available.
If you click on a listed mission, you see information under it. You can click on ''More...'' to get a more detailed screen with a description and screenshots from the mission.

Last steps:
# With a row selected, click on ''Select for download'' to move a listing to the right-hand ''Download Status/Selected For DownLoad'' pane. Repeat to build up a list to download.
# Or ''Select all'' to download all missions, but '''not recommended''' because it will stress the download mirrors unnecessary.
# Then click "Start Download". FMs will be queued to download and automatically install.

= Selecting a New or Different Mission to Play =
[[Image:new-mission210.jpg|right|thumb]]
# From the main menu, click ''Mission List''
# On the right of the screen, you'll see a list of available missions that you've installed.
# Click on a mission within the list. Information about it will pop up in the left pane (as shown in figure), and ''Notes'' often has more.
# When you've made your choice, click ''Select Mission'' at the lower right corner.
# The game will restart itself automatically. 
== Starting the mission ==
After the restart, click ''Start this Mission''. You'll be taken to the mission's briefing. 

== Resuming a mission ==
Once installed, a Fan Mission will stay installed until you uninstall it (click ''deselect mission'') or install a different FM. As each Fan Mission is "run" in a separate folder, your savegames '''will be preserved''' when switching between them. (However, a savefile will generally not be functional after an update to its FM or core TDM.)

To resume a mission from a savegame, make sure the FM is installed, then just go to the Load/Save game menu and select one from the list.

== Updating a mission ==
Whenever the mission author releases a new version of his FM, you need to re-install the mission. Missions do not auto-update. See above for instructions.

= Does a Mission Have Translations? =
Some FMs support additional languages besides English, contained in a "Localization Pack". This is a separate PK4 file, routinely named <myFM>_l10n_<checksum>.pk4, where "l10n" abbreviates "localization".

== Overviews across FMs ==
For an overview of which FMs have Localization Packs, see [[Fan Missions for The Dark Mod]], and look at the table's 'Translation Pack' column.

Unfortunately, at this time it is difficult to know which FMs have a Localization Pack that includes ''your'' particular language of interest. The [[I18N Status]] page (where "I18N" stands for Internationalization) gives information about some older FMs, but has not been updated for quite awhile.

== For a Given FM ==
It's easy enough to verify if a Localization Pack is available; the installation process described below will let you know if not.

As for support for a particular language, after you install the Localization Pack:
* look within the .pk4 (temporarily renamed to .zip if necessary) and its /strings/ folder. The presence of a *.lang (e.g., french.lang) is required for support.
* or just play the game to find out. Be aware that there are degrees of translation support.

= Installing a Mission's Translations =
The basic install above provides English. '''Do that first'''. There are then two official ways to get and install a Localization Pack.

== By downloading the localization pack ==
Starting from the site's ''Mission'' page and then to the ''Download Details'' page for given FM, the Localization Packs section will tell you if a pack of translations is available, and if so provide a choice of equivalent links. [I guess download into the /fms folder and get it auto-installed on game restart? Or download it into the particular /fm folder? TBD]

== By installing with the in-game downloader ==
For this process to work, you must have already installed the FM. Then, revisit the ''Online Mission Archive'' page. Is there an "#" annotation (for "# Translation") next to your FM name? Then '''download again''' to get the Pack and have it auto-installed. Note - Only in the 'Online Mission Archive' page will there be "#" and "*" (for "* Update") annotations; not in the Mission List page.

== See also ==
* [[The Dark Mod Gameplay]]
* [[FAQ]] (for troubleshooting etc.)
[[Category:Gameplay]]
[[Category:Fan Missions]]
[[Category:Gameplay concepts and settings]]

I18N.pl

2025-06-19T02:02:47Z

Geep: In Intro, correct "fm_translate.pl" to "I18N.pl". Add missing step to make strings folder & copy english.lang

== Introduction ==

The script I18N.pl separates the main FM data from all the texts and other assets that need to be changed in order for translation. It can either transform an English FM, or match an English FM to other languages.

== Requirements ==

* Perl. For Windows, use [http://strawberryperl.com/ Strawberry Perl].
* The script from [http://bloodgate.com/mirrors/tdm/pub/scripts/ Bloodgate].
* The TDM-distribution language files under '''<tt>strings/*.lang</tt>'''

== Usage ==

In the following examples we will use ''myfm'' in place of your FM name:

=== Windows ===

* Install Perl from [http://strawberryperl.com/ Strawberry Perl]. You may need to reboot
* Download '''I18N_script.zip''' from [http://bloodgate.com/mirrors/tdm/pub/scripts/ here].
* Extract the contents of this file to somewhere, f.i. <tt>'''C:\'''</tt>, so that you now have there an I18N directory with a directory ''I18N.pl'' in it. The text below assumes you have choosen that path. If not, replace it everywhere it occurs.
* Open this directory in Explorer, and copy your FM's PK4 file to it (for example "myfm.pk4")
* If you have other versions of your FM in a different language, copy them there, too. For instance if you have a file named "myfm_italian.pk4" which is '''the same version of your FM, but in Italian''', then copy it there, too.
* Open a console aka a "DOS Window" (see [http://www.computerhope.com/issues/chdos.htm here] if you do not know how to do this)
* Navigate to the directory you have uncompressed the script to, e.g. by typing

chdir c:\I18N\I18N.pl

* Make a "\strings\" subdirectory, and copy TDM's "english.lang" there (and any other .lang files for which you have myfm_<lang>.pk4 versions).
* Run the script (we assume here you have only the english version:)

perl I18N.pl --english myfm.pk4

The output packages will be created under output, as:

<pre>
C:\I18N\I18N.pl\output\myfm.pk4
C:\I18N\I18N.pl\output\myfm_l10n.pk4
</pre>

=== Linux ===

* Install Perl (it is probably already installed)
* Download '''I18N_script.zip''' from [http://bloodgate.com/mirrors/tdm/pub/scripts/ here].
* Extract the contents of this file to somewhere, f.i. <tt>'''/home/yourusernamehere/'''</tt>, so that you now have there an I18N directory with a file ''I18N.pl'' in it. The text below assumes you have choosen that path. If not, replace it everywhere it occurs.
* Open this directory in your favourite file manager, and copy your FM's PK4 file to it (for example '''<tt>myfm.pk4</tt>''')
* If you have other versions of your FM in a different language, copy them there, too. For instance if you have a file named <tt>'''myfm_italian.pk4'''</tt> which is '''the same version of your FM, but in Italian''', then copy it there, too.
* Open a console
* Navigate to the directory you have uncompressed the script to, e.g. by typing

cd ~/I18N/I18N.pl

* Make a "/strings/" subdirectory, and copy TDM's "english.lang" there (and any other .lang files for which you have myfm_<lang>.pk4 versions).
* Run the script (we assume here you have only the english version:)

perl I18N.pl --english myfm.pk4

The output packages will be created under output, as:

<pre>
./output/myfm.pk4
./output/myfm_l10n.pk4
</pre>

== The FM dictionaries ==

Once the script is finished, it will replace as much hard-coded strings like "Silver Key" with string templates like "#str_20000" and place these in a FM-specific dictionary. These dictionaries are put into a sep. PK4 file (named ''myfm_l10n.pk4''), they can be found inside this file under the directory '''strings/fm/'''.

See '''[[I18N - Translating Fan Missions|here]]''' for examples and how to translate these files.

== Bugs ==

=== GUI files ===

Currently GUI files are not parsed, and thus their strings not entered into the FM dictionary. Mainly this means you need to manually edit '''guis/maps/FMNAME.gui''' and replace things like:

text "Mission Loading"

with

text "#str_02236" // Mission Loading

If there are other texts in it, like "Loading finished", and these cannot be found in strings/all.lang in the TDM core, then you need to use numbers above 20000 (best use 20100 and above) and then also add the strings into the '''strings/fm/english.lang''' etc. file.

{{infobox|Also note that the main PK4 file should '''always''' contain a copy of the strings/fm/english.lang file, so that people who do not have the _l10n.pk4 file can still play the FM.}}

=== README.txt and darkmod.txt ===

Due to [http://bugs.thedarkmod.com/view.php?id=3012 bug #3012] README.txt and darkmod.txt cannot yet be translated.

== FAQ ==

=== Do I need different languages of my FM? ===

No, the script can also just use the English version. But the English version is mandatory, you cannot just use a French FM!

=== How must the PK4 files be named? ===

The names can be anything, they are not important.

=== Do I need to increase the version of my FM? ===

The FM database on thedarkmod.com uses an arbitrary integer to denote the current version, everytime this is incremented, users which have the same FM with a lesser number get asked to upgrade it.

However, it is a wise idea to increase the version in your README, too, because that is the one that users will refer to when talking about your FM.

=== My mission has image files with text on it. How do I translate them? ===

See this article [[I18N - Translating Fan Missions|Translating Fan Missions]].

=== Who can do the translation? ===

Anybody who wants to. There is a [[I18N_-_List_of_TDM_translators|list of translators]], these people might be able to help you.

== See also ==

* [[Internationalization]] - Main article
* [[I18N - Translating Fan Missions|Translating Fan Missions]]
* [http://strawberryperl.com/ Strawberry Perl].
* [http://bloodgate.com/mirrors/tdm/pub/scripts/I18N_script.zip The script itself]

{{editing}}

Installing and Running Fan Missions

2025-06-09T21:54:14Z

Geep: Add translation info

= Installing a New Mission =
<big>There are two ways to install a new mission:
# By downloading a mission pk4 file and placing it in the fms folder
# By using the in-game mission downloader
</big>

== By downloading the mission PK4 ==
# Go to [https://www.thedarkmod.com/missions/ thedarkmod.com/missions] (say, by selecting "Missions" from the site top menu strip) to see a list of available FMs.
# For an FM of interest, click on "Download Details".
# Each FM is contained in a PK4 file, which contains all the files necessary to run the mission. Choose one of the equivalent links to download, and drop the PK4 file into the fms/ folder, e.g. <tt>C:\Games\darkmod\fms\</tt>.
# Launch The Dark Mod game to '''automatically install the mission'''. That is, upon launch, the game will detect the new mission package in the fms/ folder and will add it to the list of available missions. (Note: the PK4 file will be automatically moved into a subfolder. This is normal operation).

== By installing with the in-game downloader ==
[[Image:Download-mission210.jpg|right|thumb]]
# Launch The Dark Mod.
# From the main menu, click ''Mission List'' (''formerly ''New Mission'', as shown in figures here).
# Click ''Download Missions'' in the lower border of the Mission List.

In this screen on the left is the ''Online Mission Archive'', with a list of FMs that:
* you don't yet have, but are available to download and install; and
* you '''do''' have already installed, but that have an update or new (or at least not-yet-installed) translation available.
If you click on a listed mission, you see information under it. You can click on ''More...'' to get a more detailed screen with a description and screenshots from the mission.

Last steps:
# With a row selected, click on ''Select for download'' to move a listing to the right-hand ''Download Status/Selected For DownLoad'' pane. Repeat to build up a list to download.
# Or ''Select all'' to download all missions, but '''not recommended''' because it will stress the download mirrors unnecessary.
# Then click "Start Download". FMs will be queued to download and automatically install.

= Selecting a New or Different Mission to Play =
[[Image:new-mission210.jpg|right|thumb]]
# From the main menu, click ''Mission List''
# On the right of the screen, you'll see a list of available missions that you've installed.
# Click on a mission within the list. Information about it will pop up in the left pane (as shown in figure), and ''Notes'' often has more.
# When you've made your choice, click ''Select Mission'' at the lower right corner.
# The game will restart itself automatically. 
== Starting the mission ==
After the restart, click ''Start this Mission''. You'll be taken to the mission's briefing. 

== Resuming a mission ==
Once installed, a Fan Mission will stay installed until you uninstall it (click ''deselect mission'') or install a different FM. As each Fan Mission is "run" in a separate folder, your savegames '''will be preserved''' when switching between them. (However, a savefile will generally not be functional after an update to its FM or core TDM.)

To resume a mission from a savegame, make sure the FM is installed, then just go to the Load/Save game menu and select one from the list.

== Updating a mission ==
Whenever the mission author releases a new version of his FM, you need to re-install the mission. Missions do not auto-update. See above for instructions.

= Does a Mission Have Translations? =
Some FMs support additional languages besides English, contained in a "Localization Pack". This is a separate PK4 file, routinely named <myFM>_l10n_<checksum>.pk4, where "l10n" abbreviates "localization".

== Overviews across FMs ==
For an overview of which FMs have Localization Packs, see [[Fan Missions for The Dark Mod]], and look at the table's 'Translation Pack' column.

Unfortunately, at this time it is difficult to know which FMs have a Localization Pack that includes ''your'' particular language of interest. The [[I18N Status]] page (where "I18N" stands for Internationalization) gives information about some older FMs, but has not been updated for quite awhile.

== For a Given FM ==
It's easy enough to verify if a Localization Pack is available; the installation process described below will let you know if not.

As for support for a particular language, after you install the Localization Pack:
* look within the .pk4 (temporarily renamed to .zip if necessary) and its /strings/ folder. The presence of a *.lang (e.g., french.lang) is required for support.
* or just play the game to find out. Be aware that there are degrees of translation support.

= Installing a Mission's Translations =
The basic install above provides English. '''Do that first'''. There are then two official ways to get and install a Localization Pack.

== By downloading the localization pack ==
Starting from the site's ''Mission'' page and then to the ''Download Details'' page for given FM, the Localization Packs section will tell you if a pack of translations is available, and if so provide a choice of equivalent links. [I guess download into the /fms folder and get it auto-installed on game restart? Or download it into the particular /fm folder? TBD]

== By installing with the in-game downloader ==
For this process to work, you must have already installed the FM. Then, revisit the ''Online Mission Archive'' page. Is there an "#" annotation (for "# Translation") next to your FM name? Then '''download again''' to get the Pack and have it auto-installed. Note - Only in the 'Online Mission Archive' page will there be "#" and "*" (for "* Update" annotations; not in the Mission List page.

== See also ==
* [[The Dark Mod Gameplay]]
* [[FAQ]] (for troubleshooting etc.)
[[Category:Gameplay]]
[[Category:Fan Missions]]
[[Category:Gameplay concepts and settings]]

Installing and Running Fan Missions

2025-06-09T20:19:37Z

Geep: Revise headings, freshen text

Skybox Tutorial

2025-05-10T19:31:46Z

Geep: /* See Also */

= Skybox Tutorial =
This is the tutorial for skybox creation by Sotha.
== Motivation ==
Why bother creating a custom skybox? The standard starry prefab is just fine!

It indeed is, but everyone has seen it so many times. Creating a specific sky just for your mission gives your mission personality. With the sky, you can relate atmosphere and information on the location where the mission takes place. In the plains just outside the City? In the top of a high mountain, in the eternal snow?
With a some hours invested in sky creation you can make your mission seem real as it seems to be located somewhere. It does not exist alone in the world, you can see the world around it. You can see the tiny hamlets and the fields surrounding the wealthy merchants manor.

== Introduction ==
In this tutorial you will learn
*how to create a sky texture using Terragen.
*how to build a skybox using the landscape texture you created.
*how to add details (buildings) to the skybox.

== Preparing ==
*Get aquainted with the basic idea what we are going to do. https://modwiki.dhewm3.org/Cube_maps
*Download and install Terragen. ''[Geep comments in 2025: The original (16-bit?) version of free Terragen for which this tutorial was written is no longer available from vendor planetside.co.uk. Instead now offered is a [https://planetside.co.uk/free-downloads/terragen-4-free-download/ free version of Terragen 4.] Video tutorials about Terragen 4 are available. In particular, within the 12-part series by Vladimir Chopine [GeekatPlay], [https://www.youtube.com/watch?v=_A4mrkVLs88 part 3 discusses skyboxes.] If anyone tries Terragen 4 for TDM, please update our tutorial here as needed.]''

== Basic Scenery Building ==
The program window is shown in the picture 1.
[[Image:Skybox_tutorial_terra1.jpg|200px|thumb|right|Picture 1]]

First create your first terrain. Click the Generate Terrain -button and click again the Generate Terrain button in the appearing window. You'll see a grey topology map appear. Close the Terrain Genesis window.

Since we are doing a skybox, we need the camera to be located near the middle of the map, so we are surrounded by the scene. Click left mouse button on the topology map on the Rendering Control -window to place camera in the middle of the scene. You can change the direction with the right mouse button. The grayscale topography image may not give much information on the resulting scene, so you can use the 3D preview button on the left. You can also use Rendering Preview and the actual Render Image -buttons to check what the camera sees.

Now that you know how to see what you've created, you need to try to select the parameters so that you'll get close the kind of scenery you want (plains with distant mountains or barren mountain range). Hold your mouse above the settings (Realism, Smoothing, etc) to see what each parameters does. Experiment until you get something close what you want. The closer you get with the parameters, the easier the sculpting phase is.
== Sculpting the Scenery ==
When you're close with the scenery you want, save it so you can start from where you were is something goes wrong. Landscape Window -> Terrain -> Save. Now we will sculpt the map. Click the topology map on the Landscape window. Now you can sculpt the scene with the Basic Sculpting Tool. Raise a mountain, remove a mountain, smoothen the plains and add a pit with smooth edges for a lake, or make a river.
== Water, Surface and Skies ==
Next we set the water level. Click the water-button on the left. Set the water level and press update maps. You'll see light blue colors in areas with water. Set the levels so that you've realistic looking sea. Sculpt shores to be gently sloping.

Then we set the surface. In the landscape window there is Surface Map -section. Select open and try to pick up the kind of terrain textures you need. For plains "grass and sand" is just fine. For others you probably need to create your own surface map. I cannot help with this at this time.

You can edit the sky's clouds by selecting the Clouds button on the left and set the ambience haze from the atmosphere button on the left.
== Buildings? ==
''Geep 2025 comments:''

Terragen 4 and its predecessors have no native way to create distant buildings. One approach, before capturing the skybox cubemap:
* Acquire such open-source assets in one of the Terragen object import formats (.obj, .lwo, .tgo). To import, see TerraTuts' "Terragen 4 Basics" series, particularly [https://www.youtube.com/watch?v=Fj0VdxEL7bQ "Importing a Single 3D Object"] and [https://www.youtube.com/watch?v=kc1tyruEwKM "Positioning a Single 3D Object".]
* Or use a tool like Blender to build your own model of building(s).
An advantage is that the building and landscape will share lighting, so match easily.

Alternatively, after cubemap texture capture, create miniatures in DR to add near the walls of your skybox, discussed under "Creating the Skybox" below.
== Lighting ==
Finally we set the lighting conditions with the button on the left. Since TDM is a sneaking game you probably want to set the sun below the horizon. Maybe the mission occurs at midnight? Or just after sunset? Set the sun appropriately. Check the rendering preview. It probably looks too bright with the sun gone, so decrease Shadow Lightness (located in Lighting Conditions -> Background Light). If you end up in pitch black scenery, increase the Exposure time in Rendering Control -> Camera Settings.
== Rendering the Scenery ==
When you have everything the way you want we can proceed to the image capture phase. DOUBLE CHECK your work with the Render Image -button in every direction with low detail settings so everything looks good. Once the images have been captured you need to capture everything again if you notice something unpleasing in the resulting work.

Set all detail knobs you can find to maximum (Rendering Control -> Render Settings, Rendering Control -> Detail). Set image size to 960x960. This is the maximum resolution you'll get from the free version. Double check your camera settings. Is the camera in the correct position? Does it capture the scenery details you want? Is the camera height from the surface correct? I used 6m for the plains skybox in picture 3. '''Check the camera ZOOM in Camera Settings: if this is any other than 1.0 your skybox will not align correctly (it defaults to 1.4 so change it!)'''

''Terragen 2+ evidently would use "FOV 90" instead of Zoom, [https://forums.thedarkmod.com/index.php?/topic/9082-newbie-darkradiant-questions/page/130/#findComment-309011 says post by someTaff.]''

Then capture the images.
*Set Camera orientation: head 0, pitch 0, bank 0
*Render image and save it as yoursky_'''forward'''.bmp
*Set Camera orientation: head 90, pitch 0, bank 0
*Render image and save it as yoursky_'''right'''.bmp
*Set Camera orientation: head 180, pitch 0, bank 0
*Render image and save it as yoursky_'''back'''.bmp
*Set Camera orientation: head 270, pitch 0, bank 0
*Render image and save it as yoursky_'''left'''.bmp
*Set Camera orientation: head 0, pitch 90, bank 0
*Render image and save it as yoursky_'''up'''.bmp
*Set Camera orientation: head 0, pitch -90, bank 0
*Render image and save it as yoursky_'''down'''.bmp

Open each of your image files with GIMP or whatever and save them as .tga in to folder darkmod\env\custom

== Create Material ==
Create a material file for your sky in folder darkmod\materials. Create a text file called Customskies.mtr. Fill in the following
textures/skies/custom/yoursky
{
qer_editorimage textures/custom/skies/yoursky.tga
noFragment
noshadows
nooverlays
forceOpaque // so transparent windows can draw on top of it
{
blend add // so transparent windows can draw on top of it
cameraCubeMap env/custom/yoursky
forceHighQuality
texgen skybox
texgen wobblesky .0 .0 .0
}
}

== Creating the Skybox ==
Fire up DR. You could just apply the sky texture you created (it should be in DR textures/skies/custom) on the ceilings on your map, but for distant buildings it is better to put smf/portal_sky where your sky should be.

Create somewhere away from your main map a box shaped room with the texture you created. Place an entity info_portalsky exactly in the middle of the room. Now all the surfaces with the portal_sky textures show the contents of the info_portalsky room. Now you add details to your skybox by adding buildings in the small info_portalsky -room you created. You will notice that you have to make small toy-houses and place them just slightly below the info_portalsky and next to the wall in the skybox -room. See picture 3 to see a suggested setup. Notice the position and orientation of the moon slab, the size scale of the buildings and the position of the info_portalsky.
[[Image:Skybox_tutorial_skybox.jpg|200px|thumb|right|Picture 2]]
Add light source to make the buildings bathe in moonlight. Create the moon itself: add a simple square brush with nodraw on all sides. Rotate the brush so that it's wide face is pointed straight towards the info_portalsky. To this face add the moon texture and FIT it there with surface inspector. Add more buildings and details. You'll get a decent illusion of distant structures (see picture 3). You can also add particle effects which show in the sky portals as well. Important note: if you have a sunset in your skybox, go ingame and check the compass if the sunset is in an incorrect direction. You can remedy this by a) turn the sky texture or b) rotate the info_portalsky entity. Also check that your moon's light side is pointed towards the sunset.
[[Image:Skybox_tutorial_result.png|200px|thumb|right|Picture 3. Zoom in to see night-time details.]]

I've tried scaling existing models down with the rotation parameter, but they do not light up correctly by light sources. Therefore you have to create all small detail objects from normal brushes as the ones seen in picture 2.

''Geep comments, 2020: This finding predated DR's model rescaling tool, which now offers another way to downsize existing models... more compatible with light sources?''

== Rotating the Skybox ==
If your compass shows that north is not in the correct direction you can rotate the info_portalsky. This is not a good solution, since this changes the north of the map. Objects in the skybox north won't be in game world north. It might get really confusing to make changes in the skybox later if you take this path. To maintain consistency, you should rotate the skybox instead. It is done as follows:
*Check the cubemap picture here for reference: [https://modwiki.dhewm3.org/Cube_maps]. Basically we are going nudge each direction file one slot leftwise.
*Cut your skybox files to a temporary safe folder. Copy your _up.tga and _down.tga files back to the env/custom -folder. I'll talk only in shorthand form. "_up.tga" means "yoursky_up.tga"
*Copy _forward.tga back to your env/custom/ folder. Rename it to _left.
*Copy _right -> _forward
*Copy _back -> _right
*Copy _left -> _back
*Go ingame and check direction. If its still wrong, repeat these steps until you get it right.
*When you got the surrounding correctly, go ingame and check how badly the sky texture and ground texture are unaligned with the rest of the scenery. Pay attention how many 90 degree turns to what direction is needed to make it match with the surroundings.
*Open them with you favourite image manipulation software and rotate the files accordingly.
''Geep comments, 2020: You might also need to do this procedure if, rather than creating your own skybox texture, you are using one acquired from a third-party sharing repository. This is because the skybox format is not universal across game platforms. Expect to both re-name and rotate some of the images individually.''

This concludes the tutorial and you should now be able to create your very own scenery! Go forth and create!
== See Also ==
* [[Skybox Basic Details]] has more about using the standard prefab tdm_sky_starry1.pfb. But of pertinence here, it touches on inside-skybox lighting considerations. There is also a quick backgrounder on using a skybox best in TDM.
* Besides the projected terrain of the sky box, there is world-scale terrain within TDM itself. Generally, this is not created with Terragen-like tools, but by hand-building DR patches. Some terrain may be inaccessible, outside city walls; this can present visportalling and performance challenges. Accessible terrain will need additional work for path finding. For more: [[Systematic Method for Adding Pathfinding to Uneven Terrain]]; [[Pathfinding]]; [[Roads]].
{{tutorial-editing}}
[[Category:Skybox]]