TDM Release Mechanics

From The DarkMod Wiki
Jump to navigationJump to search

This article describes the process of creating a TDM point release. It will be fleshed out over the next few days/weeks.

The team is using a version control system (SVN) for development, and the objective is to create a set of PK4 file from the repository which can be uploaded to the TDM mirrors. Let's go through this step by step:

Development

Just for the sake of completeness, this step includes the regular development between two TDM releases. This day-to-day development is always happening on the trunk, so all commits are ending up there. Stuff gets broken, stuff gets fixed, stuff gets evaluated, some might be ripped out again.

Branching off

At some point the team decides that it might be time for a new release - a so-called branch date (some sort of semi-freeze) is suggested and agreed upon. To this date, team members are supposed to wrap up their pending changes and commit them to the trunk. About two to three weeks are long enough of a time span for people to get everything into SVN.

Once the deadline arrives, the release branch is created:

  • Update your darkmod working copy.
  • Update the darkmod_src repository too.
  • Right-click the darkmod folder to create the branch
    • Select SVN > Branch/Tag...
    • A new dialog appears asking for a URL. Enter a URL like this: https://<darkmod_server_here>/svn/darkmod/branches/release1.xx where xx is replaced by whatever version the next release will be.
    • You'll probably want to switch to the new branch once it is created, so make sure the check button at the bottom of the dialog is active.
    • Hit OK to let the SVN server create the branch (the server will create a copy of the trunk, placing it at the URL you specified. It might be notable that the files are not physically copied, the branch requires a marginal amount of additional server space).
  • The same should happen with the darkmod_src repository, the code needs to be frozen as well. Perform the same steps as above, using the URL scheme https://<darkmod_server_here>/svn/darkmod_src/branches/release1.xx

At this point there are two new branches, one for the darkmod repo, one for darkmod_src. Consider them a snapshot of the trunk at the point in time you created the branch.

Important: commits to the trunk won't affect this newly created branch. The team members not involved in the release can just continue working as usual. Also note that if you as branch creator had the check button ("switch working copy") activated, your working copy has been switched to the release branch. Your commits will go to the branch, keep that in mind. To learn how to merge stuff between the trunk and the release branch, see below.

Compile Game Binaries

Before the PK4 files can be assembled into a release package, the code from darkmod_src needs to be re-compiled for the Windows and Linux platforms. This is to make sure that the release is built from the most recent (and hopefully most stable) source. If you're not a coder, you can either ask your fellow coders to switch to the branch and compile the tdm_gameNN.pk4 for you, or you can bite the bullet and follow the TDM Compilation Guide to compile it yourself. As Windows user, I recommend setting up an Ubuntu 10.04 32-Bit instance in VirtualBox. The downside of using virtualisation is that compilation takes 30-40 Minutes depending on your CPU power, but you at least don't need a dual-boot setup, let alone a second PC.

Make sure you compile a release build (debug builds are terribly slow), I also recommend re-compiling everything from scratch everytime you build a binary:

  • Windows/VC++: Menu Build > Rebuild Solution
  • Linux: scons -c && scons BUILD="release" BUILD_GAMEPAK="1"

Once compiled, make sure that the tdm_game01.pk4 is updated with the newly compiled DLL and the tdm_game02.pk4 is copied over from darkmod_src (where it is created after compilation) to darkmod and commit these two files to the branch.

Also commit TheDarkMod.exe and thedarkmod.x86 to the branch.

Compile tdm_update

tdm_update must be built separately for both Windows and Linux.

Make sure you compile a release build, and recompile everything from scratch:

  • Windows/VC++: Menu Build > Rebuild tdm_update_mfc
  • Linux: descend into the tdm_update directory and then do scons -c && scons BUILD="release"

Once compiled, copy the results into the darkmod folder:

  • Windows/VC++: bin/GuiUpdater/tdm_update.exe -> darkmod/tdm_update.exe
  • Linux: tdm_update.linux -> darkmod/tdm_update.linux

Commit these to the branch.

The Manifest

The "manifest" is a huge text files listing all the files that are nominated to go into the release. Not all files in SVN are supposed to be released (like test files, test maps, other broken or unfinished stuff), so we need a white list. The manifest text file is located in the SVN repository: devel/manifests/darkmod.txt

Building the Manifest

Note: This section is more or less for your understanding, you don't need to manually generate the manifest anymore as this step is part of the automated process running on the TDM server itself.

The manifest file is generated by the tdm_package application. (The tdm_package sources are in the darkmod_src repository, in a subfolder called tdm_update. There is a VC++ 2010 solution and a sconscript for Linux and Mac builds available. You don't need to compile it if you're in Windows, there is a pre-compiled version in the darkmod repo as well, in the devel/packager folder.)

After the generation step the file devel/manifests/darkmod.txt will have been updated. You might want to review the modifications the script made to that file and commit it to the release branch if you're satisfied.

What is included in the Manifest

The package application is using the file devel/manifests/darkmod_maps.txt to decide whether to include or exclude stuff. In a first internal step the script will include all files in specific folders: 

# Include all these files (but without parsing them like maps), each
# statement will include files (from SVN) in that folder:
INCLUDE def/
INCLUDE dds/
...
INCLUDE video/
INCLUDE xdata/

As visible in the above example, the sharp character # is used to denote comments. Put that at the beginning of a line to disable the statement.

As next step the algorithm will exclude certain files matching the regular expressions in EXCLUDE statements like this: 

EXCLUDE dds/darkmod/test

In 95% of the cases it's enough to just specify the path of the files you want to exclude (use forward slashes), but you can do more fancy stuff like this: 

EXCLUDE models/md5/chars/undead/revenant/.*.md5anim
EXCLUDE ^(dds/)?models/md5/chars/undead/revenant

The first line will exclude all MD5ANIM files in the revenant folder. The second line will exclude all files in models/md5/chars/undead/revenant and the ones in dds/models/md5/chars/undead/revenant (the dds/ part is marked to be optional).

In summary, all files that are INCLUDE'd as denoted above and afterwards manage to get through the hundreds of lines of EXCLUDE filters will end up in the manifest file.

Distribute files into PK4s

Each of the roughly 15000 files in the manifest needs to be sorted into the correct PK4. This is done by defining a rules in the file devel/manifests/darkmod_pk4s.txt. Each of the lines there defines a PK4 and which files go into it: 

# Miscellaneous stuff (GL Progs, Script, Language Files, Rope Arrow)
tdm_base01.pk4: ^glprogs, ^script, ^strings, sound\.wav, _emptyname\.wav, ^models/md5/environments, ^dds/models/md5/environments, ...

Again, lines starting with the sharp # character denote comments.

Leftmost is the PK4 filename, followed by a colon character. To the right of the colon a list comma-separated patterns is defined, whereas each pattern is interpreted as regular expression. As with the darkmod_maps.txt you don't need to know very much about regular expressions to define those rules, it's usually enough to write the folder names and use wildcards like .*.md5anim

Build the Package

The package is built right on the TDM server - there is a full darkmod repository checked out on the server's filesystem, and it will be processed through some scripts that are listening to your commands.

Once triggered, the process goes roughly like this, I'll comment on these steps below:

  • The working copy is reverted, such that it is clean.
  • The working copy is switched to the release branch of the version you're about to release (it must exist at this point).
  • The working copy is implicitly updated by the switch operation.
  • The manifest is generated.
  • The full set of PK4 files is generated (it takes a while, the server CPU is not very powerful).
  • The differential package is generated.
  • The files are copied to a public folder reachable through HTTP such that it can be tested by the team. The team will test it by storing a manipulated tdm_mirrors.txt file next to their tdm_update application.

Prepare the packaging process

A few steps need to be done before kicking off the process on the server.

  • The release branch must be created, with a specific naming convention: https://darkmod_server_here/svn/darkmod/branches/releaseY.XX where Y is the major version and XX is the two-digit minor version, e.g. "release1.08".
  • The inclusion statements in the darkmod_maps.txt file need to be up to date, make sure everything is committed to the branch.
  • The tdm_version.txt file should be containing the correct version number, it's located in devel/release/tdm_version.txt. Commit that change to the release branch, needless to say.

Kicking off the packaging process

There is a special admin page on the server which can be used to kick off the process. The URL has been posted on the forums before, greebo or grayman know the URL. It's password protected, so you'll be asked to enter credentials. Once there, you can enter the two version numbers, the one you're building, and the previous one (the latest released one). E.g.: 1.08 and 1.07. It's sufficient to enter the currently released one, e.g. 1.07, into the second field and the website will automatically fill in the next version number in the first field - you can override it if really necessary.

Once you start the process you'll be presented a text window with some progress information, and you can watch the messages emitted by the svn client and the tdm_package application running on the server. It takes a while, about half an hour, until the package is built.

Prepare the internal package test

The team needs to have access to the package as created above for internal testing, so tell them to download a specially prepared tdm_mirrors.txt. You can find one attached to the first post here: http://forums.thedarkmod.com/topic/13179-packaging-and-testing-107/

The linked post above already contains a few instructions you can copy from. Basically, instruct the team to use the --keep-mirrors switch when starting the tdm_update application, and to copy the tdm_mirrors.txt file into their test folder before running it. If the --keep-mirrors option is omitted the updater will connect to the regular TDM update mirrors, which is not the point.

The Test & Bugfix cycle

Once the testers post their bug reports, the dev team needs to get moving to fix them. It's recommended that all fixes will be committed to the trunk (both for darkmod and darkmod_src repos), and the release manager should merge the changes into the release branch.

Merge Fixes into the Release Branch

Let's assume there is a fix that got committed to darkmod SVN in order to fix an issue discovered in the test package. Proceed like this to merge it to the release branch.

  • Make sure your local working copy is switched to the release branch.
  • Now right-click the darkmod folder and choose SVN > Merge
  • Select Merge a range of revisions
  • As URL to merge from use the address of the trunk: https://<darkmod_server>/svn/darkmod/trunk
  • In the field revision range to merge you need to specify which exact change you intend to merge. It's easier if you click the "Show Log" button right next to the entry field to view the SVN Log of the trunk. You can either select a single commit, or you can select a range of commits. Perhaps double-check the changes of that checkin.
  • Hit Next, then Merge.
  • Important: the changes will be incorporated into your working copy. If you want these changes to actually be stored in the release branch, you need to perform the commit.

Special note for code changes: once these are committed to darkmod_src trunk, you need to merge them just like described above. Then you're going to recompile the game binaries (I recommend from scratch) and commit the tdm_game01.pk4 / tdm_game02.pk4 files to the darkmod release branch too.

Update the Test Package

Re-package the TDM package, as described above. Just make sure everything is committed to the release branch and kick off another build process.

Test again

Inform your testers about the update and which things got changed since the last test. Instruct them to update their test installation (don't forget to mention --keep-mirrors) and enter the next round of testing.

Release Time: Upload & Cleanup

Once the day has come to release the next version, connect to the TDM server using a secure shell (SSH) client, like PuTTY.

Follow these steps, there are some things that need to be done afterwards, they are necessary for the next release to function properly:

  • Rebuild binaries from scratch, place in darkmod folder, and commit the darkmod folder.
  • 7-zip the Windows gamex86.pdb and TheDarkMod.pdb files and place them in the trunk darkmod/devel/release/debugging/2.0x folder. Add that folder to SVN and commit it.
  • Update the darkmod 2.0x branch tree and merge the just-placed debugging files to that branch. Commit the 2.0x branch.
  • Create the final package remotely as described above (through the website).
  • The above step will generate a new crc_info.txt and a tdm_version_info.txt file on the server.
  • Copy this crc_info.txt and tdm_version_info.txt to the folder packaging/darkmod/devel/release/version_info/2.0x and commit:
    • cd /usr/local/tdm_releases/packaging/darkmod/devel/release/version_info
    • mkdir 2.0x (replace the x)
    • cd 2.0x
    • cp /usr/local/www/update.thedarkmod.com/crc_info.txt .
    • cp /usr/local/www/update.thedarkmod.com/tdm_version_info.txt .
    • cd ..
    • svn add 2.0x
    • svn ci --username=<YOUR_SVN_USER> -m "<YOUR COMMIT MESSAGE>"
  • Check in the newly generated manifest, it should still be in the folder where the packaging script left it:
    • cd /usr/local/tdm_releases/packaging/darkmod/devel/manifests
    • svn ci --username=<YOUR_SVN_USER> -m "<YOUR COMMIT MESSAGE>"
  • Create a tag for the darkmod and darkmod_src repositories (/tags/2.0x) - copy it from the latest revision in the corresponding release branch.
    • Switch your darkmod working copy to the 2.0x release branch
    • Choose SVN > Branch/Tag and fill in the dialog fields with the following:
    • Same goes for the darkmod_src repository, just replace /darkmod/ with /darkmod_src/ above.
  • Copy package from /usr/local/www/update.thedarkmod.com/ to /usr/local/tdm_releases/2.0x/ for archiving. The next package process will need it there.
    • cd /usr/local/tdm_releases
    • mkdir 2.0x
    • cp /usr/local/www/update.thedarkmod.com/* 2.0x/
    • cp -r /usr/local/www/update.thedarkmod.com/fms 2.0x/
  • Copy /usr/local/tdm_releases/1.05/index.html to /usr/local/tdm_releases/2.0x
  • Move all differential packages back to /usr/local/www/update.thedarkmod.com/
    • mv /usr/local/tdm_releases/2.0x/tdm_update_1.0*.zip /usr/local/www/update.thedarkmod.com/
    • mv /usr/local/tdm_releases/2.0x/tdm_update_2.0*.zip /usr/local/www/update.thedarkmod.com/
  • Copy the one differential update package updating to the new 2.0x version back to /usr/local/tdm_releases/2.0x
    • cp /usr/local/www/update.thedarkmod.com/tdm_update_2.0y_to_2.0x.zip to /usr/local/tdm_releases/2.0x (y = x-1)
  • At this point, the /usr/local/tdm_releases/2.0x folder contains a valid TDM 2.0x release, the one that will be pushed to all the mirrors. Examine it for double safety.
  • Examine the upload_to_all_mirrors.sh shell script, to see if it fits your needs.
  • Start uploading to all mirrors.
    • ./upload_to_all_mirrors.sh 2.0x
  • Update tdm_mirrors.txt and tdm_version.xml on thedarkmod.com/update: ask taaaki to do that. The tdm_mirrors.txt file is probably fine, but the tdm_version.xml needs to be updated.

Update the Wiki

The wiki contains one page for each release. You can find them all in the Category:What's New category.

First update the current release page:

  • Change the "roadmap" link on the current release to a "changelog" link by swapping "roadmap_page.php?version_id=XX" with "changelog_page.php?version=XX".
  • Add the {{released|2.01|2014-01-30}} Template on top of the current release page with the correct date.

Then add a new page for the next release. To do this, add a link to it in this template: Template:Whatsnew by editing it. After saving the template, a link to the new release version page appears in the list and in the category. Edit this page and insert some text into it (you can copy it from an older release page).

Also insert a link to the current roadmap on the bugtracker, you can find out the version_id by looking at the roadmap page.

Glossary

  • Release Fileset: the PK4 files forming the actual release.
  • SVN: an acronym for Subversion, a version control system
  • Commit: The process of a team member uploading files to the SVN repository - a commit always creates a new SVN revision.
  • Mirrors: The servers where the PK4 files are stored for download by the updater.
  • updater: The TDM Updater executable, used for downloading and updating TDM.
  • binary: Source code is compiled into so-called binaries. This can be an executable (.exe) file or a module (.dll/.so).
  • darkmod_src: The code repository holding all the TDM source files, including the tdm_update application.
  • branch: A tree in the SVN repository. Several branches can co-exist side-by-side without interfering.
  • trunk: The main branch where all the ongoing development is happening.
Retrieved from "https://wiki.thedarkmod.com/index.php?title=TDM_Release_Mechanics&oldid=18429"