Refont: Difference between revisions

Latest revision as of 17:36, 14 July 2024

By Geep, 2024, and similar to the article about Q3Font.

As of 2024, the new "refont" command-line utility allows inspection and alteration of font metrics. It is designed to be an easier-to-use version of the inspection/alteration parts of Q3Font. It also provides a font analysis feature.

Comparison with Font Patcher

Refont can be seen as complementary to Font Patcher. The latter has additional features that may be better if planning a wholesale rearrangement or expansion of characters within a bitmap. But Font Patcher requires installation of Perl, while refont.exe is more drop and go.

Comparison with Q3Font

For metric alteration, q3font has a native "*.fnt" human-readable format, and supports DAT <==> FNT conversions. For compatability, so does refont.

Advantages of Q3Font

This has been an important traditional tool for Doom3/TDM font manipulation.
In addition to production/consumption of FNT, it -
- Allows creation of DAT & TGA files from a TrueType font (although for this purpose, ExportFontToDoom3 has been preferred in practice).
- Can write (but not read) an alternative "compact report".

Advantages of Refont

It's open-source, so further C++ development is possible.
It has its own native human-readable format, "*.ref", similar to .fnt but better.

The better features of REF are:

Codepoints are enumerated in not just decimal, but also hex.
Coordinates of the overall image box within its bitmap are expressed in pixel units, rather than [0...1] fractional floating point. So the user is spared having to manually do this tedious and error-prone calculation.
This overcomes a drawback of FNT, that expresses the coordinates with only 6 digits after the decimal point, not quite enough to avoid minor mis-precisions.
if optionally provided, a separate, read-only annotation file will automatically decorate the .ref file with helpful information as the file is generated. This is explained below.

Because the code is open-source, there's better clarity about -

how parsing and calculations are done.
what warnings are emitted.

Also, using "-stats", refont can write (but not read) an analysis of a DAT file, particularly looking for problematic/unimplemented Western-language characters.

For Inspection & Correction of DAT files

To Get the Data in Readable, Editable Form

For viewing and altering in a text editor, two formats are provided. Both list every codepoint, in order from 0 to 255, as a block of data. Within each block, every item gets its own line.

As a FNT File

To create a q3font-compatable "<dat-name>.fnt" file:

refont -decompile <path to given .dat file> -fnt

where the path can be a full path, a relative path (e.g., starting with "./" or "../"), or just the file name in the current working directory. The resulting .fnt file will appear in the same directory. It have the same name as the .dat file, except for the extension. As a convenience to keep track of versions, it is not required that the file name be in the standard fontImage_nn.dat form.

Example code block for the letter A:

	...
	char 65
	{
		height 18
		top 18
		bottom 0
		pitch 3
		xSkip 18
		imageWidth 20
		imageHeight 18
		s 0.769531
		t 0.250000
		s2 0.847656
		t2 0.320313
		glyph 0
		shaderName fonts/stone_0_24.tga
	}
 	...

Note that Q3Font FNT expresses the s, t, s2, and t2 coordinates with only 6 digits after the decimal point, causing minor low-precision rounding during DAT --> FNT --> DAT roundtrips. For compatibility, Refont FNT does likewise.

As a REF File

To create a refont-specific "<dat-name>.ref" file:

refont -decompile <path to given .dat file>

The same path and naming conventions as for .fnt generation discussed above apply here.

Example code block for the letter A when no supplemental annotation file is present (nor any per-line comments, e.g., with warnings):

	...
	char 65 (0x41)
	{
		height 18
		top 18
		bottom 0
		pitch 3
		xSkip 18
		imageWidth 20
		imageHeight 18
		coord_s 197
		coord_t 64
		coord_s2 217
		coord_t2 82
		glyph 0
		shaderName fonts/stone_0_24.tga
	}
	...

Items bolded differ from FNT format. The starting line expresses the value in hex. And rather than s, t, s2, t2 in fractional [0...1] floating point, the "coord_..." values are expressed (and editable) in pixel units.

When Editing

Refont has a minimal parser. To keep it happy, preserve the file's line structure. In particular:

every REF and FNT file has the same number of lines.
all keywords within a block must be present, and in a particular order.
each block must have exactly the same number of lines (except for the special one at the end).

The meaning of the font metrics are explained in Font_Metrics_&_DAT_File_Format. The Q3Font article has an example of simple metric editing that is also pertinent to refont.

To Move Changed FNT or REF Data Back to a DAT

q3font -compile <path to given .fnt or .ref file>

This reverses the -decompile process, to create a .dat file whose name is derived from the .fnt or .ref file. Importantly, this decompile/compile cycle leaves .tga/.dds files untouched.

The Annotation File

When told to generate a REF file (but not FNT), refont.exe also looks for a file called "refont_char_annotations.txt", in the same folder as refont.exe. That file should have exactly 256 data lines in it, one for each of the codepoints, in order. As the REF is generated, the line that starts a codepoint's data block gets the corresponding annotation line appended to it. Having the annotations come from an external file, instead of hard-coded into refont, allows required flexibility as to what information is presented. It even potentially allows refont to be used with other idTech3/4 games that don't necessarily used the TDM-specific codepoints. Also, the annotation files contain helpful specific comments (as lines that start with ";").

Each annotation:

indicates what character symbol is expected; it is oblivious as to what character if any is actually within the bounding box described by the DAT data.
lives only in a REF file, and is not retained when converted to a DAT file. This also true for any manual supplementary annotations you might make.

This is clearer with examples. In 'Downloads' below are 4 different versions of "refont_char_annotations[...].txt". Three of these, specific to TDM's custom 'english' (i.e., composite European) codepage, are discussed here. To deploy, edit the filename of one of them to remove the square bracket text.

English with Symbol Only

This simple format is best for most purposes. The file entry for letter 'A' is just:

Then the block in a generated REF file would have that string appended, but with " // " inserted.

	...
	char 65 (0x41) // A
	{
		height 18
		top 18
		bottom 0
		pitch 3
		xSkip 18
		imageWidth 20
		imageHeight 18
		coord_s 197
		coord_t 64
		coord_s2 217
		coord_t2 82
		glyph 0
		shaderName fonts/stone_0_24.tga
	}
	...

English with Unicode-16 Codes and Names

This supplies the Unicode Consortium's official 16-bit codepoints (where U+NNNN = 0xNNNN) and their names, in the formal all-caps convention.

The file entry for letter 'A' is:

U+0041 LATIN CAPITAL LETTER A

Then the first line of the corresponding block in a generated REF file would show:

	...
	char 65 (0x41) // U+0041 LATIN CAPITAL LETTER A
	...

Here's another example from that REF:

	...
	char 129 (0x81) // U+015A LATIN CAPITAL LETTER S WITH ACUTE
	...

For control or TDM undefined characters, this annotation file assumes a hollow box glyph is the appropriate mapping:

	...
	char 0 (0x00) // U+24A1 WHITE SQUARE
	...

But there are other choices, so edit the annotation file as you see fit.

English with 8859-x Source

This verbose format may be useful during font analysis. For letters that are in their expected locations as defined by ASCII or ISO 8859-1 (Latin-1) or Windows-1252 ("Western Europe"), the annotation is fairly minimal. Thus, letter 'A' has this:

0x41 is A

(TIP: If authoring your own annotation file, consider including a codepoint number on each line, like "0x41" here, to make the process easier.)

Then the first line of the corresponding block in a generated REF file would show:

	...
	char 65 (0x41) // 0x41 is A
	...

Here are a few more interesting examples from that REF, with more expansive annotations:

	...
	char 5 (0x05) // 0x05 is a control character, unusable in TDM
	...
	char 166 (0xa6) // 0xA6 is Š (not ISO 8859-1 ¦) (from A9 in ISO 8859-2) (TDM only)
	...

Revising or Making Your Own Annotations File

If creating your own refont_char_annotations.txt file, note that encoding upper-range characters in UTF-8 will be more universal than using Windows region-specific codepages. To enter a UTF-8 character using its four digit hex Unicode (say, in a range of interest 0x0080-0x00FF):

Under Windows, type the four hex digits followed by Alt-X.
Under Linux, hold down Ctrl+Shift, type U followed by the 4 digits; release Ctrl+Shift.

When refont reads refont_char_annotations.txt, any line that starts with ";" will be ignored. The latter is so that you can put comments into the file, that will not propagate to the generated REF file. TIP: If such commenting is unwanted, start the line with <space> before ";".

Other than that, be sure to have exactly 256 lines of annotations (though refont will check and issue a warning if not). A line can be empty, i.e., have only the linebreak.

When generating a REF file, refont will automatically add " // " before each non-empty annotation, so don't include that.

Adding Your Own Annotations to a REF File

Beyond the refont_char_annotation.txt content, you can add certain commentary to the REF file without it affecting its ability to convert to a DAT file. Here are the rules:

Don't add or subtract lines, or comment out non-blank lines
Feel free to append a comment to the end of a line, by beginning the comment with " //".
Likewise, on a line starting with "char" that had an annotation automatically added, you can revise the annotation as you see fit.

Example with revision in bold:

	...
	char 65 (0x41) // Capital A was slightly clipped. Decremented s, s2 by 1. Looks OK now.
	{
		height 18
		top 18
		bottom 0
		pitch 3
		xSkip 18
		imageWidth 20
		imageHeight 18
		coord_s 196 // WAS: 197
		coord_t 64
		coord_s2 216 // WAS: 217
		coord_t2 82
		glyph 0
		shaderName fonts/stone_0_24.tga
 	}
	...

In this way, you can markup a REF file as a record of work to do, work in progress, or work completed.

Errors, Warnings, and Auto-Corrections

This covers the major overhaul of Refont v 2.1, and its added options -no_warn_comments and -scaling_ok.

During the compile or decompile, errors and warnings may be generated in the console. Fatal errors are largely due to:

file I/O problems;
deviations from the strict DAT, REF, and FNT formats.

Warnings, which are not fatal, are largely due to anomolous numeric values. The TDM engine can tolerant some of these, so they don't necessarily need fixing.

Fatal Errors when Parsing REF or FNT as Input

A fatal error, due to divergence from the expected format while parsing, stops the program. While potentially due to many causes (e.g., adding or deleting lines), errors will be reported - with line number - as failures of:

Variable Name Checks. The parser expects every variable name in a particular order. A mismatch happened.

Integer Checks. Integer value was unparsable, or out of range for integer type. Checked for per-character variables height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and glyph. Plus REF’s coord_s, coord_t, coord_s2, and coord_ t2.

Float Checks. Floating point value was unparsable, or out of range for float type. Checked for glyphScale, and FNT’s per-character variables s, t, s2, t2.

Post-Read MAJOR Warnings about REF, FNT, and DAT Input Values

After parsing, the input process concludes with a separate loop of non-fatal warning analysis. Warnings during input, broadly categorized as major or minor, report to the console, but don’t do any value corrections (which is instead left to the subsequent file-write stage).

For FNT’s and DAT’s s, t, s2, t2

Inexact Integer. The given bounding box value, expected in the range 0-1, when multiplied by 256.0 should be exactly an integer. If not, the divergence is likely due to miscalculation. (Actually, a divergence from an integer of under +/- 0.001 is tolerated by Refont and considered a minor warning, discussed below.)
Float Out of Range 0.0 - 1.0. Or equivalently, post-multiplication, 0.0 - 256.0. (See the discussion of the edge cases under minor warnings below.)

For Other Metrics of DAT, FNT, and REF

Integer Out of Range 0-256. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight, and REF’s coord_s, coord_t, coord_s2, and coord_t2.
Integer Non Zero. For glyph.
Float Less Than Zero. For glyphScale (which is independent of a particular character).
Inconsistent Metrics. For a given character, this compares two metrics and looks for inconsistencies. We’ll state this here in REF terms, e.g., using “coord_s” rather than “ROUND_TO_INT(s * 256.0f)”:
- imageHeight == height?
- imageWidth == coord_s2 - coord_s?
- imageHeight == coord_t2 - coord_t?

Suppressing Warnings Due to Scaling. The last two "Inconsistent Metrics" criteria will not be true if there is per-character scaling. (Only TDM’s mason and mason-glow use per-character scaling, specifically for all upper-case characters.) In such a case, to avoid generating a lot of useless major warnings, use the refont option "-scaling_ok".

Post-Read MINOR Warnings for FNT’s and DAT’s s, t, s2, t2

Low Precision. This value, when multiplied by 256.0, should be exactly an integer. However, some generation or revision procedure (e.g., Q3Font’s DAT  FNT  DAT) may have not used sufficient precision to ensure this. An offset from an integer value of less than +/- 0.001 is considered a minor problem; otherwise, major as above.

This problem, when it occurs with a file, typically happens a lot. Consequently, it is not reported in an itemized way, but instead, the console will show total counts of affected characters and values.

Edge cases: a value of {s, s2, t, or t2} * 256.0 that is negative but closer to zero than -0.001000 is considered this type of minor problem (and is not considered out of range, which would be major). Likewise, if that value is greater than 256.000000, but less than 256.001000

Automatic Corrections during Output to a REF File

Certain problems, first detected during DAT input, are further addressed at write time. Because these problems were already reported to the console during the DAT read, nothing additional is written to console. Instead, there may be reporting within the REF file itself.

Automatic Corrections. When writing a REF file, the integer values of coord_s, coord_t, coord_s2, and coord_t2 are calculated from DAT floats s, t, s2, and t2 respectively. A calculated coord_... value, if not already resulting in an integer, is automatically forced to the nearest integer. This is so whether it was earlier reported to the console as a Major or Minor Warning. (Before refont v 2.1, the REF might assign an unrounded decimal value; no longer true.)

Warnings Not Automatically Corrected. Automatic fixing of other warnings is unwise and not offered. Why? Because the fix is not obvious or complex, and sometimes, the corresponding character glyph in the bitmap needs adjustment. This pertains to Major Warnings about:

Integer values outside their expected range (e.g., 0-256), including post-rounding coord_... values.
Inconsistent Metrics

Generated Within-REF Comments during Output

For most causes of a Major Warning during DAT read, during subsequent writing of the REF file, a similar comment is also generated within the REF, on the same line as the parameter assignment. For example:

char 95 (0x5f) // _
{
   ...
   top -6 // WARNING: Not in range 0-256. Does bitmap or other metrics need adjusting?
   ...
}
...
char 140 (0x8c) // Ń
{
   ...
   coord_t -10 // WARNING: Underlying DAT value -0.03906250, * 256.0 gave -10.000000000, not in range 0-256 once rounded. Does bitmap or other metrics need adjusting?
   ...
}

Suppressing Comment Generation. Use the refont option “-no_warn_comments”. New to v 2.1.

TIP: When editing a REF file with comments, you can keep them, change them, or delete them as you prefer. They have no impact on subsequent compiles to DAT.

Coverage of Generated Within-REF Comments. A calculated value of coord_s, coord_t, coord_s2, or coord_t2 would have been flagged as a “Major Warning” during read if:

the discrepancy from being an integer was non-trivial (+/- 0.001 or more), or
the calculated and rounded integer is outside the range 0-256.

For these, a comment is generated. Other causes are:

Integer Out of Range 0-256. For height, top, bottom, pitch, xSkip, imageWidth, imageHeight.
Integer Non Zero. For glyph.
Float Less Than Zero. For glyphScale (which is independent of a particular character).

Major Warnings that Don’t Cause Within-REF Comments. Currently, this is limited to “Inconsistent Metrics”, which would probably require tagging on multiple lines.

For Statistics on Unimplemented or Problematic Font Characters

DAT file analysis here is primarily designed to benefit english/european fonts. It will use the optional annotation file if provided. Syntax:

    refont -stats <path to given .dat file>

Use "-stats" by itself, not combined with "-decompile".

The resulting analysis report will have the same name and location as the DAT file, but with a ".txt" suffix. The report provides categorized totals and (for problematic characters) itemizations across all 256 character codepoints. The counts of problems should be considered minimums, since no inspection of bitmap files (TGA or DSS) is done. But a perhaps surprising amount of insight can be gleaned just from DAT file metrics. Itemizations include annotations from the 'ref_char_annotation' file, if provided.

In the hunt for missing characters, 3 signatures are important:

Presumed 'Hollow Box' = glyph's 'shadername' is <fontname>_0_<size>.dds only, and s & t are zero, but not s2 & t2
'<Space>' = same test as 'Hollow Box', but pointed to by char 32 (0x20)
'Zero Box' = No glyph box (s, t, s2, t2 all zero)

A given analysis will assume either 'Hollow Box' or '<Space>' but not both.

The analysis provides totals and itemizations - grouped into lower (0-127) and upper ranges (128-255) - in 3 passes:

Pass 1 - Handling of unprintable/unsupported/missing codepoints, indicated by Hollow Box, Zero Box, or <Space>. Some of these are not a problem, but some are tagged as "Undesirable".
Pass 2 - Bad glyph box (negative s, t, s2, or t2; or s2 <= s, t2 <= t) or good glyph box with dubious metrics (imageHeight <= 0, imageWidth <=0, imageHeight != height). Excludes those already counted as "Undesirable" in Pass 1.
Pass 3 - Detection of duplicate glyph boxes (other than Hollow Box, Zero Box, or <Space>). Detected by: glyph's values for shadername, s, t, s2, & t2 exactly match those of another codepoint. These are grouped into "Dup Sets".

Finally, across all passes, a count if given of the minimum "Total Glyphs Needing Work".

Downloads

Release 2.1 of July 14, 2024:

Annotations

After download, to use any one of these, edit filename to remove "[...]" substring.

Update of June 20, 2024. This reflects a change to the TDM charmap for 2.13, to replace duplicate O/o circumflex characters with G/g breve:

refont_char_annotations[english with symbol only. With G-breve update].txt here. Recommended for most purposes with 'english' fonts.
refont_char_annotations[english with Unicode-16 codes & names. With G-breve update].txt here.
refont_char_annotations[english with 8859-x source. With G-breve update].txt [1].
For Russian, see the previous update next.

Update of April 13, 2024:

refont_char_annotations[english with symbol only].txt here. Recommended for most purposes with 'english' fonts.
refont_char_annotations[english with Unicode-16 codes and names].txt here.
refont_char_annotations[english with 8859-x source].txt here. Corrects and supersedes original english file.
refont_char_annotations[russian].txt here. Recommended for 'russian' fonts.

For More

See the summary analysis of 2.12 TDM fonts, based on applying 'refont -stats ...' to all 'english' DAT files.