Converting Translations with all lang to po

Geep — Sat, 24 Oct 2026 19:39:45 GMT

Overview

Converting Translations with all lang to po

Geep — Fri, 23 Oct 2026 20:01:05 GMT

create article

New page

=== Moving from LANG to POT and PO Files===
''By Geep, 2026''
==Overview==
Traditionally, TDM's i18n approach has used a UTF8 "all.lang" file, which has an "[English]" section as master, followed by individual sections for each TDM-supported language. Across sections, phrases have been tied together by #str_ids. Then, from all.lang, language files (e.g., english.lang, german.lang, etc) in language-specific 8-bit encodings are generated and distributed.

Consider specifically tdm_base01.pk4/strings/all.lang, that provides TDM's main menu and other core phrases.

For [upcoming] TDM 2.15, the [English] portion of this was substantially reworked, to provide greater and more accurate context for human and AI translators, and so assist remaining translation updates and fill-ins.

Building off that, Geep and stgatilov are undertaking a move towards the popular GNU "gettext" translation architecture, with:
* an English template file (english.pot)
* various individual language files (*.po). The target encoding for these remains UTF8, unlike the *.lang files. Instead, the TDM engine internally converts to 8-bit form (at least for now).

All these can be edited by third-party apps like "poedit".

It is desirable to retain the advantages of the reworked all.lang. So, rather than the stock workflow of using "xgettext" to generate a .pot file from marked-up *.cpp, *.hpp, and *.gui source, a further-marked-up version of all.lang [English] is used instead, as in effect the "source" material.

In turn, this used by all_lang_to_po to generate:
* "english.pot", as structured from all.lang [English].
* Various non-english "*.po" files. These also reflect the all.lang [English] structure, because the creation of each begins with copying the english.pot template, then adding all existing translations from the appropriate language section of all.lang .
* Optionally, an "english.po" file, where each English string appears twice. This is not required, since TDM will consume "english.pot" for the English phrases.

[LATER: The exact workflow for further revisions after this transition is still being determined. Likewise applicability to FM translations.]
==Treatment of Unique IDs==
As background, some gettext-based translation projects do not use symbolic ids (like #str_), but just use unique English strings as keys. This would be represented in the english.pot file as:

msgid "Mine"
msgstr ""

Then, in each corresponding language-specific .po, msgstr would be filled in with the translation.

If a particular English key phrase appears multiple times with different meanings, then an additional "context" field can distinguish them:

msgctxt "weapon"
msgid "Mine"
msgstr ""

msgctxt "location"
msgid "Mine"
msgstr ""

In this case, pot/po considers msgctxt+msgid to be the unique key (while msgctxt by itself is not unique.)

For projects like main-menu TDM with symbolic keys, an obvious configuration of the format would be:

msgid "#str_12345"
msgstr ""

But this has a significant drawback. Because the .pot file doesn't contain any English strings, features of poedit that use AI (or community resources) to do draft string translations can't be used. There are ways to overcome this; a reportedly popular one, adopted here, is to move the symbolic link into msgctxt:

msgctxt "#str_12345"
msgid "Mine"
msgstr ""

If further context is needed, it could be included as a comment, discussed further below.

==Editing & Markup in all.lang [English]==
===Programmatic Considerations===
On each line within the [English] section, leading tabs and spaces are ignored.
====Po-Visible Lines====
We define here a "po-visible" line as one that appears as a visible, selectable row in poedit. There are two types, which we name here:
* "Translatable" lines
* "Structure" lines

Translatable lines are indicated by retaining the traditional TDM all.lang layout:
"#str_12345" "Some text here"
Structure lines (which will not be expected to be translated) serve as headlines to group translatable lines together within poedit. Arbitrarily, they are flagged with "//|", e.g.
//| SETTINGS VIDEO SUBMENU
====Comments====
Comments must be associated with, and anchored by, a particular po-visible line. In all.lang, they are arbitrarily flagged as follows:
* Lines that start with "//" are BEFORE comments, associated with a po-visible line that immediately follows the group of them. (Excluded: lines starting with "//|" or "///".)
* Lines that start with "///" are AFTER comments, associated with a po-visible line that immediately precedes the group of them.
When processed by all_lang_to_po, both types will be emitted together (as a group of consecutive individual comment lines, all preceding the po-visible line). Each comment line be flagged by starting with "#." and omit the "//" or "///". In poedit, these will appear in the "Note to Translators" pane, that is shown when the po-visible line is selected.

It was found necessary to have both types to maximize readability of both all.lang and .pot, and provide flexibility for anchoring.

Special treatment: In all.lang, it is common with a group of before-comments for the last line to end in ":", referring to the next line with its po-visible anchoring item. However, once converted into a "Note to Translators", the colon is either useless or erroneously seems to address the first after-comment. So the conversion process automatically removes it. (If the all.lang editor needs to retain it, add a space after it.)

Note: a third possibility, a "//" comment at the trailing end of a translatable line within the [English] section, is deprecated. If present, all_lang_to_po will issue a console warning, and turn it into a "before" comment line. The reworked all.lang [English] avoids this form entirely.
===Style Considerations===
====Headlines====
Currently, the "structure" lines are used for primarily for 3 levels of headlines/outlines:
1) ALL CAPS WITH UNDERLINE
=========================
2) ALL CAPS
3) Camel Case
In a few spots, a fourth style, just a sentence, is used for a group comment that needs to be visible.

Note that the first style uses two po-visible lines, and is used sparingly to denote major chapters.

Translators should be encouraged to click on structure lines, because often times, there are "Notes to Translators" associated with them.
====Comments about Source Code====
When there is editor-provided information about what source code file (e.g., cpp, gui) a particular #str_id appears in, this usually becomes the first line of a Notes to Translators. (This is in the spirit of, but different from, the specific-line annotation that xgettext provides. The latter, tagged with "#:", has a very strict syntax and is only reasonable if programmatically generated, i.e., by xgettext.)
====Comments about #str_ID References====
Within a comment, all.lang uses a leading "&" followed by the seeming text of a translatable line, as a "reference" to that #str_id, e.g.:
//& "#str_12345" "Some text here"
or
///& "#str_12345" "Some text here"
The meaning: that #str_id was already uniquely defined elsewhere in all.lang [English], and is being listed here solely to mark for the reader where it is being used again within, say, a menu's layout. (Without requiring any special processing, in the .pot/.po file, the "&" is retained within a comment line, and when viewed in poedit, becomes the first letter on the corresponding line within "Notes to Translators".)
==Generated Preamble Header==
===In the POT File===
Below is a 2026 example of a gettext-style header in the english.pot file. By gettext convention, the fields starting with "X" are customized additions, e.g., for TDM.
# TDM does not support po plural forms at this time, so this field
# (with language variants) is excluded (which may cause warnings):
# "Plural-Forms: nplurals=2; plural=(n != 1);\n"
msgid ""
msgstr ""
"Project-Id-Version: TDM Main Menu, 2.15\n"
"POT-Creation-Date: 2026-10-22T20:59:54Z\n"
"Language: en\n"
"Content-Type: text/plain; charset=UTF-8\n"
"MIME-Version: 1.0\n"
"Content-Transfer-Encoding: 8bit\n"
"Language-Team: none\n"
"X-TDM-All.Lang-Format-Input-File-Name: all.lang\n"
"X-TDM-All.Lang-Format-Input-File-Date: 2026-10-08 16:05:35\n"
"X-TDM-POT-Output-File-Name: english.pot\n"
"X-TDM-Generator: Program all_lang_to_po v. 0.13\n"
===In the PO Files===
The *.po files have the same preamble, except the Language field has the appropriate code.
==Data Conversion from all.lang [English] to english.pot==
[TO DO - THIS INFO GOES INTO TABLE CELLS]
From all.lang [English] To .pot [Note 1] Notes
"#str_whatever" "text" msgctxt "#str_whatever"
msgid "text"
msgstr ""
Same, but "text" has printf specifiers Adds line: "#, c-format" [Note 2]
Same, but "if line > maxlength chars..." msgid ""
"if line > maxlength chars, ... then "
"chunked, i.e., word wrapped" [Note 3]
//| Structure text msgctxt "##"
msgid "Structure text"
msgstr "" [Note 3][Note 4]
// Before-comment #. Before-comment [Note 5]
/// After-comment #. After-comment [Note 5]

===Notes===
[1] Following a poedit convention, each block of data associated with a given phrase is separated by a blank line in the file (but not visible in the poedit UI). Also, in the optional english.po generated by all_lang_to_po, the "msgstr" is not an empty string, but instead the same string as "msgid".

[2] Only specifier substrings actually used in TDM are searched for: %d, %i, %s, %0 .

[3] Chunking applies to msgctxt, msgid, and msgstr, and for both translatable and structure text (though the latter is not likely to need it). You can suppress chunking with an all_lang_to_po "-long" option. And change poedit to allow long lines. (Note however: poedit appears to have its own treatment of word wrap that seems idiosyncratic.) [maxLength is in theory 79 printable chars; my code is a little more conservative at moment.]

[4] Note that msgctxt is "##" (rather than all.lang's "//|" or other considered alternatives). This style choice, eventually arrived at, reflecting that:
* comments in po generally start with "#"
* in poedit, for a po-visible line, the msgctxt content is visually separated from the msgid content by a "|", so already has that hint.
* Keeping this as short as possible was visually helpful in poedit.

Also, the "Structure Text" phrase must be unique to the file (since the msgctxt "##" is not unique).

[5] Before and after cases differ in only where lines are defined in all.lang; both are emitted into .pot before the corresponding po-visible anchor line. Also, any too-line will be broken up into multiple comment lines – similar to chunking in spirit if not details.
==Data Conversion from all.lang [<language>] to <language>.po==
The process begins by copying the english.pot file, then filing in the "msgstr" fields with all the existing translations. Within the all.lang <language> section, full-line comments are ignored. The only comments retained are those that trail phrase lines. In the .po file, such a comment becomes a separate line beginning with "#<space>" instead of "//". In poedit this appears in the translator's "Comment" pane. This treatment covers the traditional translator notation of "stays the same" specified by TDM.

If the content of msgctxt, msgid, or msgstr is too long, and the -long option is not specified, "chunking" (word-wrap) will occur.
==Release Notes & Possible Further Work==
See the start of the all_lang_to_po.cpp file.
==Downloadable Files==
[This section coming soon]

Source Code

Executable

==See More==
For more about the headers, see
* https://po-file.com/blog/po-file-header-explained
* https://www.gnu.org/software/gettext/manual/html_node/Header-Entry.html

The DarkMod Wiki - Recent changes [en]

Converting Translations with all lang to po

Converting Translations with all lang to po