TDM Release Mechanics

This article describes the process of creating a TDM point release. It will be fleshed out over the next few days/weeks.

The team is using a version control system (SVN) for development, and the objective is to create a set of PK4 file from the repository which can be uploaded to the TDM mirrors. Let's go through this step by step:

Development

Just for the sake of completeness, this step includes the regular development between two TDM releases. This day-to-day development is always happening on the trunk, so all commits are ending up there. Stuff gets broken, stuff gets fixed, stuff gets evaluated, some might be ripped out again.

Branching off

At some point the team decides that it might be time for a new release - a so-called branch date (some sort of semi-freeze) is suggested and agreed upon. To this date, team members are supposed to wrap up their pending changes and commit them to the trunk. About two to three weeks are long enough of a time span for people to get everything into SVN.

Once the deadline arrives, the release branch is created:

• Update your darkmod working copy.
• Update the darkmod_src repository too.
• Right-click the darkmod folder to create the branch
  • Select SVN > Branch/Tag...
  • A new dialog appears asking for a URL. Enter a URL like this: https://<darkmod_server_here>/svn/darkmod/branches/release1.xx where xx is replaced by whatever version the next release will be.
  • You'll probably want to switch to the new branch once it is created, so make sure the check button at the bottom of the dialog is active.
  • Hit OK to let the SVN server create the branch (the server will create a copy of the trunk, placing it at the URL you specified. It might be notable that the files are not physically copied, the branch requires a marginal amount of additional server space).
• The same should happen with the darkmod_src repository, the code needs to be frozen as well. Perform the same steps as above, using the URL scheme https://<darkmod_server_here>/svn/darkmod_src/branches/release1.xx

At this point there are two new branches, one for the darkmod repo, one for darkmod_src. Consider them a snapshot of the trunk at the point in time you created the branch.

Important: commits to the trunk won't affect this newly created branch. The team members not involved in the release can just continue working as usual. Also note that if you as branch creator had the check button ("switch working copy") activated, your working copy has been switched to the release branch. Your commits will go to the branch, keep that in mind. To learn how to merge stuff between the trunk and the release branch, see below.

Compile Game Binaries

Before the PK4 files can be assembled into a release package, the code from darkmod_src needs to be re-compiled for the Windows and Linux platforms. This is to make sure that the release is built from the most recent (and hopefully most stable) source. If you're not a coder, you can either ask your fellow coders to switch to the branch and compile the tdm_gameNN.pk4 for you, or you can bite the bullet and follow the TDM Compilation Guide to compile it yourself. As Windows user, I recommend setting up an Ubuntu 10.04 32-Bit instance in VirtualBox. The downside of using virtualisation is that compilation takes 30-40 Minutes depending on your CPU power, but you at least don't need a dual-boot setup, let alone a second PC.

Make sure you compile a release build (debug builds are terribly slow), I also recommend re-compiling everything from scratch everytime you build a binary:

• Windows/VC++: Menu Build > Rebuild Solution
• Linux: scons -c && scons BUILD="release" BUILD_GAMEPAK="1"

Once compiled, make sure that the tdm_game01.pk4 is updated with the newly compiled DLL and the tdm_game02.pk4 is copied over from darkmod_src (where it is created after compilation) to darkmod and commit these two files to the branch.

Generate the Manifest

Once you know that the game binaries (tdm_gameNN.pk4 files) are up to date and checked into the release branch, go ahead and build the manifest. The manifest is a huge text files listing all the files that are nominated to go into the release. Not all files in SVN are supposed to be released (like test files, test maps, other broken or unfinished stuff), so we need a white list.

To build that manifest there exists a Perl script extract-assets.pl in the devel, which traverses your repository and collects the files. To run that script, one is bound to Linux at the moment, as the "file-is-in-SVN" check fails in Windows. I recommend using VirtualBox again, but be warned that building the manifest takes over an hour this way (I've Ubuntu set up on my second laptop, which takes about 10 Minutes to compile the manifest). Run the command by typing this in your darkmod folder:

perl devel/extract-assets.pl darkmod

After this step the file devel/manifests/darkmod.txt has been updated. You might want to review the modifications the script made to that file and commit it to the release branch if you're satisfied.

Note: you need to re-build the manifest everytime a file gets added or removed from the release branch. You don't need to rebuild it if a texture file gots changed, for example.

What is included in the Manifest

The extract-assets Perl script is using the file devel/manifests/darkmod_maps.txt to decide whether to include or exclude stuff. In a first internal step the script will include all files in specific folders:

# Include all these files (but without parsing them like maps), each
# statement will include files (from SVN) in that folder:
INCLUDE def/
INCLUDE dds/
...
INCLUDE video/
INCLUDE xdata/

As visible in the above example, the sharp character # is used to denote comments. Put that at the beginning of a line to disable the statement.

As next step the algorithm will exclude certain files matching the regular expressions in EXCLUDE statements like this:

EXCLUDE dds/darkmod/test

In 95% of the cases it's enough to just specify the path of the files you want to exclude (use forward slashes), but you can do more fancy stuff like this:

EXCLUDE models/md5/chars/undead/revenant/.*.md5anim
EXCLUDE ^(dds/)?models/md5/chars/undead/revenant

The first line will exclude all MD5ANIM files in the revenant folder. The second line will exclude all files in models/md5/chars/undead/revenant and the ones in dds/models/md5/chars/undead/revenant (the dds/ part is marked to be optional).

In summary, all files that are INCLUDE'd as denoted above and afterwards manage to get through the hundreds of lines of EXCLUDE filters will end up in the manifest file.

Build the Package

To build the package, make sure you have enough space on your hard drive. We'll also need a copy of the previous release, so consider the space needed by that one too. The actual packaging is done using a small helper app named tdm_package which can be found in the darkmod repository in devel/tdm_package.exe.

For this tutorial I'll assume that we'll be packaging 1.03 and are using the following paths:

• c:\games\doom3\darkmod as location of the darkmod SVN repository (switched to the release branch).
• d:\temp\package\darkmod1.03 as target path for the TDM 1.03 package.
• d:\temp\package\darkmod1.02 as location of the "old" TDM 1.02 package (needed for comparison).
• d:\temp\package\update as target folder for the differential package.

Copy the previous release

In TDM 1.03+ the updater can deal with differential update packages. To generate one of these you need to have a copy of the previous release available (e.g. if you're about to release TDM 1.03 you'll need a copy of TDM 1.02) for the packager to compare. The previous release package should consist of (no more, no less):

• The PK4 files (tdm_ai_animals01.pk4 through tdm_textures_wood02.pk4)
• The ZIP files
  • tdm_shared_stuff.zip
  • tdm_update_linux.zip
  • tdm_update_windows.zip
• The fms/ folder with the release training map (and possibly other maps that got released with the base TDM package).

Don't copy over the DoomConfig.cfg, config.spec, currentfm.txt or any other files from a working set, we don't need those.

Copy the tdm_version_info.txt file

All releases from TDM 1.03 upwards will contain a tdm_version_info.txt file - download or copy that file and place it into the d:\temp\package\update folder. We'll need this file to store the version info of the next release, plus the info about the differential packages.

Adjust the package scripts

In the folder darkmod/devel there is a small batch file tdm_package1.03.cmd which is performing the packaging sequence. In this file there are several paths which most probably don't match your system, so you'll need to adjust the paths found in the file. Also, if you're releasing 1.04 you should create a copy of that aforementioned file and rename it tdm_package1.04.cmd, logically. Let's have a look at the contents of such a batch file:

@echo --- Cleaning up previously created package ---
d:
cd d:\temp\package
rd darkmod1.03 /S /Q
rd update /S /Q

@echo --- Switching to darkmod SVN folder ---
c:
cd C:\Games\Doom3\darkmod\devel

@echo --- Creating Package and CRC ---
tdm_package --create-package --darkmoddir=c:\games\doom3\darkmod --outputdir=d:\temp\package\darkmod1.03
tdm_package --create-crc-info --basedir=d:\temp\package\darkmod1.03

@echo --- Creating Differential Update ---
md d:\temp\package\update

tdm_package --create-update-package --basedir=d:\temp\package\darkmod1.02 --baseversion=1.02 --headdir=d:\temp\package\darkmod1.03 --headversion=1.03 --outputdir=d:\temp\package\update

@echo --- Updating Version Info ---
tdm_package --update-version-info-file --basedir=d:\temp\package\darkmod1.02 --baseversion=1.02 --outputdir=d:\temp\package\update
tdm_package --update-version-info-file --basedir=d:\temp\package\darkmod1.03 --baseversion=1.03 --outputdir=d:\temp\package\update

tdm_package --register-update-package --package=tdm_update_1.02_to_1.03.zip --outputdir=d:\temp\package\update

Let's go through that:

• In the first few lines, any previous files found in d:\temp\package\darkmod1.03 are deleted.
• The second part is changing folder to the darkmod SVN directory where the tdm_package app is located.
• The third path will create the 1.03 PK4 set, using the SVN folder as specified by --darkmoddir, writing it to the output folder specified by --outputdir
• After this line, the crc_info.txt file will be created from the PK4 set.
• The next line will create a differential package, comparing --basedir to --headdir. This will write the diff package to d:\temp\package\update.
• As final step, the tdm_version_info.txt file will be updated with the information from 1.02, 1.03 and the diff package.

Glossary

• Release Fileset: the PK4 files forming the actual release.
• SVN: an acronym for Subversion, a version control system
• Commit: The process of a team member uploading files to the SVN repository - a commit always creates a new SVN revision.
• Mirrors: The servers where the PK4 files are stored for download by the updater.
• updater: The TDM Updater executable, used for downloading and updating TDM.
• binary: Source code is compiled into so-called binaries. This can be an executable (.exe) file or a module (.dll/.so).
• darkmod_src: The code repository holding all the TDM source files, including the tdm_update application.
• branch: A tree in the SVN repository. Several branches can co-exist side-by-side without interfering.
• trunk: The main branch where all the ongoing development is happening.