Converting Translations with all lang to po
Moving from LANG to POT and PO Files
By Geep, 2026. Overall project discussion with stgatilov is here.
Overview
Traditionally, TDM's i18n approach has used a UTF8 "all.lang" file, which has an "[English]" section as master, followed by individual sections for each TDM-supported language. Across sections, phrases have been tied together by #str_ids. Then, from all.lang, language files (e.g., english.lang, german.lang, etc) in language-specific 8-bit encodings are generated and distributed.
Consider specifically tdm_base01.pk4/strings/all.lang, that provides TDM's main menu and other core phrases.
For [upcoming] TDM 2.15, the [English] portion of this was substantially reworked, to provide greater and more accurate context for human and AI translators, and so assist remaining translation updates and fill-ins.
Building off that, Geep and stgatilov are undertaking a move towards the popular GNU "gettext" translation architecture, with:
- an English template file (english.pot)
- various individual language files (*.po). The target encoding for these remains UTF8, unlike the *.lang files. Instead, the TDM engine internally converts to 8-bit form (at least for now).
All these can be edited by third-party apps like "poedit".
It is desirable to retain the advantages of the reworked all.lang. So, rather than the stock workflow of using "xgettext" to generate a .pot file from marked-up *.cpp, *.hpp, and *.gui source, a further-marked-up version of all.lang [English] is used instead, as in effect the "source" material.
In turn, this is read by all_lang_to_po to generate:
- "english.pot", as structured from all.lang [English].
- Various non-english "*.po" files. These also reflect the all.lang [English] structure, because the creation of each begins with copying the english.pot template, then adding all existing translations from the appropriate language section of all.lang .
- Optionally, an "english.po" file, where each English string appears twice. This is not required, since TDM will consume "english.pot" for the English phrases.
[LATER: The exact workflow for further revisions after this transition is still being determined. Likewise applicability to FM translations.]
Running the Program
Setup
This is a Windows program. If you run it with the "-h" or "?" option, help says:
Usage: all_lang_to_po [OPTIONS], with all.lang in 'strings' child folder. Output: english.pot (and .po files unless skipped) into same folder. Options: -long Allow long lines in .pot and .po files, instead of gettext-compatable chunking (i.e. wordwrap). -pot_only Skip output of .po files. -en_po When outputting .po files, include english.po.
So, set up your 'strings' folder relative to the program first, with marked-up all.lang in it.
Important Limitations
At this time, you cannot enter paths or filenames. Also, output files overwrite existing files WITHOUT WARNING.
Typical Output
TDM 'all_lang_to_po' v 1.00, build of Oct 24 2026 12:21:26.
By Geep for TDM; adapted from pivot_all_lang & gen_lang_plus.
Command line request: all_lang_to_po -long
Parsing language sections...
0) [english]
Caution - These strings have the same content ('Broadhead Arrow'): #str_02430, #str_02448
Caution - These strings have the same content ('Objectives'): #str_02105, #str_07312
[etc.]
1) [german]
2) [french]
[etc.]
15) [turkish]
16) [catalan]
Generating english.pot and *.po, each with #str in all.lang [English] order...
Wrote english.pot
Wrote catalan.po
Wrote czech.po
[etc.]
Wrote swedish.po
Wrote turkish.po
Also, if chunking is active and an encountered comment is too long for a single line, a "split" warning will be seen. You might edit all.lang to shorten the comment.
Treatment of Unique IDs
As background, some gettext-based translation projects do not use symbolic ids (like #str_), but just use unique English strings as keys. This would be represented in the english.pot file as:
msgid "Mine" msgstr ""
Then, in each corresponding language-specific .po, msgstr would be filled in with the translation.
If a particular English key phrase appears multiple times with different meanings, then an additional "context" field can distinguish them:
msgctxt "weapon" msgid "Mine" msgstr "" msgctxt "location" msgid "Mine" msgstr ""
In this case, pot/po considers msgctxt+msgid to be the unique key (while msgctxt by itself is not unique.)
For projects like main-menu TDM with symbolic keys, an obvious configuration of the format would be:
msgid "#str_12345" msgstr ""
But this has a significant drawback. Because the .pot file doesn't contain any English strings, features of poedit that use AI (or community resources) to do draft string translations can't be used. There are ways to overcome this; a reportedly popular one, adopted here, is to move the symbolic link into msgctxt:
msgctxt "#str_12345" msgid "Mine" msgstr ""
If further context is needed, it could be included as a comment, discussed further below.
Editing & Markup in all.lang [English]
Programmatic Considerations
On each line within the [English] section, leading tabs and spaces are ignored.
Po-Visible Lines
We define here a "po-visible" line as one that appears as a visible, selectable row in poedit. There are two types, which we name here:
- "Translatable" lines
- "Structure" lines
Translatable lines are indicated by retaining the traditional TDM all.lang layout:
"#str_12345" "Some text here"
Structure lines (which will not be expected to be translated) serve as headlines to group translatable lines together within poedit. Arbitrarily, they are flagged with "//|", e.g.
//| SETTINGS VIDEO SUBMENU
Comments
Comments must be associated with, and anchored by, a particular po-visible line. In all.lang, they are arbitrarily flagged as follows:
- Lines that start with "//" are BEFORE comments, associated with a po-visible line that immediately follows the group of them. (Excluded: lines starting with "//|" or "///".)
- Lines that start with "///" are AFTER comments, associated with a po-visible line that immediately precedes the group of them.
When processed by all_lang_to_po, both types will be emitted together (as a group of consecutive individual comment lines, all preceding the po-visible line). Each comment line be flagged by starting with "#." and omit the "//" or "///". In poedit, these will appear in the "Note to Translators" pane, that is shown when the po-visible line is selected.
It was found necessary to have both types to maximize readability of both all.lang and .pot, and provide flexibility for anchoring.
Special treatment: In all.lang, it is common with a group of before-comments for the last line to end in ":", referring to the next line with its po-visible anchoring item. However, once converted into a "Note to Translators", the colon is either useless or erroneously seems to address the first after-comment. So the conversion process automatically removes it. (If the all.lang editor needs to retain it, add a space after it.)
Note: a third possibility, a "//" comment at the trailing end of a translatable line within the [English] section, is deprecated. If present, all_lang_to_po will issue a console warning, and turn it into a "before" comment line. The reworked all.lang [English] avoids this form entirely.
Style Considerations
Headlines
Currently, the "structure" lines are used for primarily for 3 levels of headlines/outlines:
1) ALL CAPS WITH UNDERLINE ========================= 2) ALL CAPS 3) Camel Case
In a few spots, a fourth style, just a sentence, is used for a group comment that needs to be visible.
Note that the first style uses two po-visible lines, and is used sparingly to denote major chapters.
Translators should be encouraged to click on structure lines, because often times, there are "Notes to Translators" associated with them.
Comments about Source Code
When there is editor-provided information about what source code file (e.g., cpp, gui) a particular #str_id appears in, this usually becomes the first line of a Notes to Translators. (This is in the spirit of, but different from, the specific-line annotation that xgettext provides. The latter, tagged with "#:", has a very strict syntax and is only reasonable if programmatically generated, i.e., by xgettext.)
Comments about #str_ID References
Within a comment, all.lang uses a leading "&" followed by the seeming text of a translatable line, as a "reference" to that #str_id, e.g.:
//& "#str_12345" "Some text here"
or
///& "#str_12345" "Some text here"
The meaning: that #str_id was already uniquely defined elsewhere in all.lang [English], and is being listed here solely to mark for the reader where it is being used again within, say, a menu's layout. (Without requiring any special processing, in the .pot/.po file, the "&" is retained within a comment line, and when viewed in poedit, becomes the first letter on the corresponding line within "Notes to Translators".)
Generated Preamble Header
While all.lang has a preamble of comments for translators, this is not carried over into the POT and PO files.
In the POT File
Below is a 2026 example of a gettext-style header in the english.pot file. By gettext convention, the fields starting with "X" are customized additions, e.g., for TDM.
# TDM does not support po plural forms at this time, so this field # (with language variants) is excluded (which may cause warnings): # "Plural-Forms: nplurals=2; plural=(n != 1);\n" msgid "" msgstr "" "Project-Id-Version: TDM Main Menu, 2.15\n" "POT-Creation-Date: 2026-10-22T20:59:54Z\n" "Language: en\n" "Content-Type: text/plain; charset=UTF-8\n" "MIME-Version: 1.0\n" "Content-Transfer-Encoding: 8bit\n" "Language-Team: none\n" "X-TDM-All.Lang-Format-Input-File-Name: all.lang\n" "X-TDM-All.Lang-Format-Input-File-Date: 2026-10-08 16:05:35\n" "X-TDM-POT-Output-File-Name: english.pot\n" "X-TDM-Generator: Program all_lang_to_po v. 0.13\n"
In the PO Files
The *.po files have the same preamble, except the Language field has the appropriate code.
Data Conversion from all.lang [English] to english.pot
Basic Conversion to Blocks
"#str_whatever" "text"
becomes, as noted above,
msgctxt "#str_whatever" msgid "text" msgstr ""
Following a poedit convention, each block of data associated with a given phrase is separated by a blank line in the file (but not visible in the poedit UI).
Text with Specifiers
If the "text" contains printf specifiers, then a preceding line is added:
#, c-format
Only printf specifiers actually used in TDM are searched for by all_lang_to_po: %d, %i, %s, %0 .
Long Text and Chunking
If "text...is...very...very long", then by default it is chunked, i.e., word-wrapped:
msgid "" "text...is...very" "very long"
By gettext convention:
- the first line is always ""
- word-wrap occurs if a line would have more than 79 characters [if not otherwise specified; & all_lang_to_po is a little more conservative than that].
- an embedded "\n" in the string will force a wrap.
Chunking applies to msgctxt, msgid, and msgstr, and for both translatable and structure text (though the latter is not likely to need it).
You can suppress chunking with the all_lang_to_po "-long" option. And change poedit to allow long lines. (Note however: poedit appears to have its own treatment of word wrap that seems idiosyncratic.)
Group Structuring Headlines
//| Structure text
converts to
msgctxt "##" msgid "Structure text" msgstr ""
Note that msgctxt is "##" (rather than all.lang's "//|" or other considered alternatives). This style choice, eventually arrived at, reflects that:
- comments in po generally start with "#"
- in poedit, for a po-visible line, the msgctxt content is visually separated from the msgid content by a "|", so already has that hint.
- Keeping this as short as possible was visually helpful in poedit.
Also, the "Structure text" phrase must be unique to the file (since the msgctxt "##" is not unique).
Comments
// Before-comment "#str_whatever" "text" /// After-comment
becomes
#. Before-comment #. After-comment msgctxt "#str_whatever" msgid "text" msgstr ""
Before and after cases differ in only where lines are defined in all.lang; both are emitted into .pot before the corresponding po-visible anchor line. Also, any too-line will be broken up into multiple comment lines [with "split" warning to console] – similar to chunking in spirit if not details.
Data Conversion from all.lang [<language>] to <language>.po
The process begins by copying the english.pot file, then filing in the "msgstr" fields with all the existing translations. Special case: if all_lang_to_po's "-en_po" option is chosen to also generate english.po, each "msgstr" therein simply gets the same string as "msgid".
Within the all.lang <language> section, full-line comments are ignored. The only comments retained are those that trail phrase lines. In the .po file, such a comment becomes a separate line beginning with "#<space>" instead of "//". In poedit this appears in the translator's "Comment" pane. This treatment covers the traditional translator notation of "stays the same" specified by TDM.
If the content of msgctxt, msgid, or msgstr is too long, and the -long option is not specified, "chunking" (word-wrap) will occur.
Release Notes & Possible Further Work
See the start of the all_lang_to_po.cpp file.
Downloadable File
Release V1.00 is available here: all_lang_to_po_1.00.zip
This contains:
- Current all.lang and generated output files, in both chunked & long-line forms.
- Release executable
- C++/Visual Studio 2022 source; open-source license
See Also
For more about the headers, see
