Tdm installer and zipsync

This article describes the new tdm_installer from 2020 and the underlying zipsync framework. It is intended for tech-savvy users, including mainly programmers, and definitely not ordinary players.

The article starts from far away, you can probably skip some text and jump to ready-to-use recipes. However, I usually prefer to know what I'm doing =)

zipsync

zipsync is the library which comprises the core of tdm_installer. It does not know anything about TheDarkMod game, and has no custom code written specifically for TDM. It's main goal is to solve a generic problem of differential update over HTTP.

Basics

Suppose that our installation is merely a set of zip archives: some of them are in the root directory, and some can be in subdirectories. Remote HTTP server provides a "target" installation, and we want to construct exactly the same set of zips locally. However, we don't want to download the whole of it, since we already have some set of zips locally which looks very similar. How can we synchronize our local installation to the remote one by downloading only the data that we don't have yet?

The algorithm relies on the feature of HTTP 1.1 called "byteranges". It allows to download only the specified part (or parts) of the remote file from HTTP server. Given that every file inside zip archive occupies a contiguous segment of bytes, it is possible to download any chosen subset of files from a remote zip archive.

In order to know which files have to be downloaded, zipsync must be able to compare local files against remote files without downloading them first. This is achieved by comparing hash digests of the files: zipsync uses cryptographical hash function BLAKE2s with 32-byte digest. However, in order to make it work, the hashes of all files in remote zips must be precomputed and be available for efficient download. That's why zipsync needs some sort of manifest, which describes all the files contained in the zips.

A manifest consists of a set of files, every file belonging to some zip. There are two types of manifests in zipsync:

• target manifest describes what the files should be, but does not explain how to get/download them.
• provided manifest gives instructions how to obtain/download the files.

So provided manifest is just a bunch of data which everyone can download, but it does not describe any version/installation. A target manifest describes a complete version/installation, but does not describe where/how to get the necessary files.

Zipsync update takes target manifest of the version that must be installed, and provided manifest showing what is available for download. It matches equal files between these manifests by hash, and learns which files to download, and how to repack the zips when everything is ready.

The target manifest always comes from one location, but the provided manifest is usually merged from several sources:

1. local manifest describing the current state of the local installation.
2. remote manifest of some full version, e.g. full release200 version in TDM.
3. remote manifests of differential packages, e.g. release201_from_release200, ..., release208_from_release207 in TDM.

In a typical scenario of differential update, most of the files are taken from local manifest. However, in case of clean install it is empty, so all the files are downloaded from remote manifests. Note that more than one remote provided manifest is necessary, because we want to store a sequence of versions as differential packages on server.

The manifest is usually stored in manifest.iniz file, which also a normal zip archive with an ini file inside. Some confusion may arise from the fact that both target and provided manifests are combined and stored in a single file. That's because provided manifest is always a subset of the target manifest under normal usage, so there is no point in describing same files twice.

Development status

The zipsync code is available as a separate repo on GitHub. This repo is fully embedded into the TDM repo at tdm_installer\zipsync. Nobody except me (stgatilov) has to bother about updating the separate GitHub repo: if you need to change zipsync, just do it in TDM repo and forget. However, please don't put any TDM-specific changes into zipsync repo! The library is intended to be a generic framework, not tied to the details of TDM.

While tdm_installer links the whole zipsync library into itself, the zipsync repo/directory has a few extra projects. The first one is the executable with unit tests --- it covers most of the core algorithm. The second one is the command-line tool called "zipsync". See readme in zipsync directory for explanations how to build them.

Command-line tool

Normally, it should be built from tdm_installer/zipsync directory, but you can also download Windows prebuilt version. Zipsync tool has the following commands:

zipsync analyze: Initializes the set of zips in a directory, preparing it for zipsync to work with. It goes over all the files inside zips and computes their hashes, producing the manifest in the end (all the files are both target and provided in this manifest). The command is usually run with arguments --clean (removes accidental trash starting with double underscore) and --normalize (fix zips if they are not supported by zipsync). See usage examples in Tdm installer and zipsync#Packaging.

zipsync update: This is very low-level command to perform an update. It is much harder to use than tdm_installer, since you have to list all provided manifests explicitly. And it does not do any TDM-specific finalization. For example, here is the way to update to release208:

   set BASE=http://darkmod.taaaki.za.net/zipsync
   zipsync update --clean -p manifest.iniz ^
     -t %BASE%/release/release208_from_release207/manifest.iniz ^
     -p %BASE%/release/release207_from_release206/manifest.iniz ^
     -p %BASE%/release/release206_from_release205/manifest.iniz ^
     -p %BASE%/release/release205_from_release204/manifest.iniz ^
     -p %BASE%/release/release204_from_release203/manifest.iniz ^
     -p %BASE%/release/release203_from_release202/manifest.iniz ^
     -p %BASE%/release/release202_from_release201/manifest.iniz ^
     -p %BASE%/release/release201_from_release200/manifest.iniz ^
     -p %BASE%/release/release200/manifest.iniz

As you see, the manifest from differential package of 2.08 is passed as --target, and all the rest are passed as --provided. Note that local manifest manifest.iniz is also passed as provided for make update differential --- you have to prepare it first.

zipsync diff: Subtracts installation B from installation A, producing differential package. The A-B differential package consists of all the files provided by A which are not described by B. The target manifest of A-B package remains exactly the same as the target manifest of installation A, so it can still be used to update to full A installation, as long as enough provided manifests are given. See usage examples in Tdm installer and zipsync#Differential package.

zipsync patch: A quick and dirty way to create small patches. Given a freshly analyzed patch package P, it extends its target manifest with all files from target manifest of base installation B, except for the files already present. Basically, P consists of a few files which you want to add/substitute in some base installation B, and the command turns P into a working P-B differential package. See usage examples in Tdm installer and zipsync#Patch approach.

zipsync hashzip: Used to package tdm_installer binary into special "checksummed" zip. Special format of such zip allows to quickly check if binary on remote server is different from the local one. Only used if you want to publish a new version of tdm_installer on the central HTTP server.

tdm_installer

This is cross-platform GUI application to install and update TheDarkMod on player side. It establishes a set of additional rules, somewhat restricting the generic system of zipsync packages and updates. Also it adds some nice pre/post processing specific to TDM.

All TDM versions are organized in a rooted tree (or several trees if wanted). Each node of the tree describes a target version which can be installed. The root node release200 stores/provides the full package for the corresponding 2.00 version. Every other node stores/provides a differential package, with parent version subtracted as base. Obviously, when you choose a version to update to, the installer takes its manifest as target, and manifests of all versions on the path to the root as provided.

Config file

The installer needs two files to work: its own executable tdm_installer.???, and the config file tdm_installer.ini. The config file is automatically downloaded from hardcoded URL. It describes all official versions of TDM and mirrors used to download them. It also describes the structure of the tree by specifying parent versions with depends key. One version is marked as default: that's the latest official release, and all players will get it by default unless they choose custom version. For the sake of presentation to end user, versions are organized in GUI tree using folder key, which can be configured arbitrarily with no relation to structure of the physical tree.

The official mirrors are described in a Mirror sections. It is expected that all mirrors have exactly the same data, synchronized e.g. using rsync. It is also allowed to specify plain URL for manifestUrl key in a version. Several manifestUrl keys are also allowed. Note that every official version must be present on the hardcoded central server (that's the one where the installer and its config file are located). The tdm_installer always downloads the target manifest from this central server, which allows adding untrusted servers to the list of mirrors.

Version conventions

Here is the current convention for TDM official versions:

• Official releases are named like release207, organized in a chain of differential packages in the versions tree. This is the main chain in the versions tree, growing with slowest speed, and needing absolute preservation. In GUI, they are located in releases folder. Usually, the latest such version is marked as the default one.

• Hotfix releases are named like release209a, release209b, etc. If any of them exist for an official release, they are organized in a chain starting at the corresponding official release. Hotfix releases substitute the original official release as the default version, although the original release can be still downloaded if wanted.

• Beta releases are produced during beta-testing phase, named like beta208-04. The beta releases are organized in a chain. I'm not sure yet which version should serve as the start point of a beta chain. Beta versions can start from the previous official release, which is good because it makes them independent of development builds. It also makes sense for a beta chain to continue the chain of development builds, i.e. start from the latest dev build. Finally, the beta chain can also be "rebased" to start from the new official release after beta phase is over (in order to save amount of used space). In GUI tree, beta versions are located in subfolders like beta/2.08.

• Development builds are produced regularly from SVN trunk, when previous official release is already out but next beta phase has not started yet. Each dev build is named like dev15976-8815, which has revisions of both SVNs embedded. The assets repo revision actually describes which state it was packaged from, and the code repo revision is necessary so that people know which revision they should checkout from public SVN to build compatible executable. Dev builds are organized in a long chain starting from the previous official release. They are placed in GUI subfolders like dev/2.09, with version of the next release shown.

• Test builds can be made by developers if they want a player to test some version with some hacky changes. The major property of test builds is that they can be added at will and removed when they are not necessary. All test builds are located in _test folder.

The is also some conventions about directory structure of official versions on the official mirrors. If you want to know it, just go to the official mirrors via FTP and see it for yourself.

Unofficial versions

The tdm_installer allows to update TDM to user-made unofficial versions. In order to create such a version, a user needs:

• enough technical and command line skills
• compatible HTTP server to publish files at

Every unofficial package must be based on some official version, and is stored as a differential package against it. For instance, the first experimental VR version was based on version release208 and published at URL: http://darkmod.taaaki.za.net/zipsync/VR/209a1/manifest.iniz When offering unofficial version to end users, you should post both of these things: the base official version and the custom manifest URL. In order to update to it, player checks "Get Custom Version" on first page, then chooses base official version in GUI tree and copy/pastes the URL into "Custom manifest URL" input field.

Note that unless your unofficial version is published on the central TDM server, players will receive a warning when updating to it. Basically, they will have to agree that they trust you and really want to install something provided by you.

Ownership rules

The main benefit of zipsync algorithm is that it does not care what version user currently has: the algorithm will detect every file that can be reused and perform differential update even if user has modified something with his dirty hands. However, there is a downside: the installer knows which version it needs to update to, but does not know which version it is updating from. Hence, if the new version does not contain some file that the old version had, it is hard to decide if it needs to be deleted or not. Note that the TDM directory contains user data, so removing everything is not an option.

To cope with the problem, the installer has some ownership semantics. If it owns a file, then it will delete it if the new version does not have it.

There are some hardcoded owning rules described at the beginning of Actions.cpp. The following zip archives are owned by TDM (described as path/glob relative to root directory):

• tdm_*.pk4: includes all the stock game packages.
• tdm_*.zip: includes tdm_shared_stuff.zip and zips with legacy tdm_update.
• fms/tdm_*/*.pk4: this is for prepackaged FMs. It is intended that their names start with tdm_.

Initially there were some more hardcoded rules, because I was planning to rely on hardcoded rules only. Some generic mechanism was added later, and I disabled them (#if 0 sections in cpp code). The generic mechanism works like this:

1. When user installs some version, the list of files added/updated/touched by it is saved into .zipsync/lastinstall.ini file.
2. When user performs update next time, this list is loaded, and all the listed files are considered owned.

Basically, the installer owns everything it has installed, and can remove it on the next update/install.

tdm_installer also has to deal with unpacked files like TheDarkModx64.exe or ca-bundle.crt. By hardcoded convention, all these files must be packaged into tdm_shared_stuff.zip archive. The installer simply unpacks all the files form this archive at the end of installation. Note that the same ownership rules apply: every file unpacked by installer will be owned by it and can be deleted on the next install.

Lastly, on Linux platform installer marks as executable a hardcoded set of files: thedarkmod.x86 and thedarkmod.x64. If you provide an unofficial version with different executables, please instruct Linux users to chmod them (sorry).

Packaging

The goal of this section is to explain how to package versions for tdm_installer, at least dev builds and unofficial versions. If you seek a way to package a small patch against available version, e.g. substituting the executables, proceed to Tdm installer and zipsync#Patch approach.

Full clean package

The steps to produce an official full package did not change with the introduction of tdm_installer. This approach allows to package the most recent revision of the assets repo in trunk or a branch.

Preliminary steps:

• Make sure you have command line SVN tools installed, i.e. running svn st in SVN repo works as intended.

• Make sure both darkmod_src and darkmod SVN repos (i.e. code and assets) are checked out and clean. For instance, I have them at G:\TheDarkMod\darkmod_src and G:\TheDarkMod\darkmod respectively.

• Go to darkmod_src/tdm_update directory and build tdm_packager executable on your platform.

• Choose an empty directory where the full package will be written to, e.g.: G:/TheDarkMod/tmp_package

Now it's time to update executables.

• Go to darkmod_src repo and build TDM executable for every platform that you want to publish.

• Commit the executables to the assets repo.

If you want to package a custom modification to an old version, then you cannot commit executables. In such case you cannot use this approach: proceed to the next section then.

In order to actually create a package, run:

   tdm_package --create-manifest --darkmoddir=G:\TheDarkMod\darkmod >update_mani.log
   tdm_package --create-package --darkmoddir=G:\TheDarkMod\darkmod --outputdir=G:/TheDarkMod/tmp_package >create_full.log

The first step creates/updates TDM manifest in the assets SVN repo (note: it has nothing to do with zipsync manifests). Please review the log after this step, it should list files which were added/removed compared to the clean manifest stored in SVN. If you have unversioned files, make sure the log contains phrases like "Skipping unversioned item".

Also you might want to review the updated manifest at darkmod/devel/manifests/darkmod.txt. Commit the updated version to the repo. When committing, you can also see the diff, showing which files were added/removed since previous manifest update.

The second step creates the zip archives from SVN-versioned files. When inspecting its log, pay attention to any warnings in "Sorting files into PK4s..." section. Note that it uses command line svn command to see which files are unversioned and ignore them.

For more information on tdm_packager, see sections TDM_Release_Mechanics#The_Manifest and TDM_Release_Mechanics#Distribute_files_into_PK4s.

Now the package is ready: it should be a bunch of pk4 and zip archives. All the other commands of tdm_packager were used for the old tdm_update, they are not necessary for tdm_installer.

Lastly, you need to make the package usable by zipsync. Go to the directory with package and run:

   zipsync analyze -cn *.pk4 *.zip

You might want to add -j0 to accelerate it if the package is located on SSD (which is recommended). After it finishes, you should see manifest.iniz of approximately 2 MB size, meaning that package is ready.

You can upload the whole directory to HTTP server and update to it with tdm_installer, either using custom manifest URL with any base version, or by adding it to the config file with no dependencies. But it is recommended to create and upload differential package instead to save storage (see Tdm installer and zipsync#Differential package).

Full modified package

This approach explains how to produce a full package for a modified version, in case you cannot commit updated executables and manifest to the assets repo. Possible cases include: you don't have access to assets SVN, or you don't want to commit modifications to the repo.

For a start, you need to obtain a full package of some version. There are two ways to get it:

• Create it yourself as the previous section explains, but without committing anything anywhere.

• Install existing version via tdm_installer, with "Bitwise exact zips" checked on.

Obviously, the second approach is much easier. The "Bitwise exact zips" setting is rather important: otherwise the set of zips that you get may differ from the one stored on server in compression. You might also want to delete all files except for .zip and .pk4 after install: only zip archives are ever handled by packaging process.

Now that you have some base version to start with, modify the zips as you see fit. You can add new files to zips, overwrite files inside zips, remove files inside zips, even add and remove new zips. When you are sure you have crafted the version you want, make it usable by zipsync by running:

   zipsync analyze -cn *.pk4 *.zip

After the new manifest.iniz is saved, the full modified package is ready. If can be uploaded to HTTP server and be used for updates, either using custom manifest URL with any base version, or by adding it to the config file with no dependencies. But it is recommended to create and upload differential package instead to save storage (see Tdm installer and zipsync#Differential package).

Differential package

Suppose that you have a full TDM package locally, and want to turn it into differential package.

First of all, you need to choose some base version, which is official and already published. In order to minimize size of differential package, choose the base version which is most similar to the version you have. Luckily, you don't need to have full base version available locally, only its manifest is needed. You can also pass URL of base manifest to zipsync tool, it will download it itself.

To create differential package, run:

   zipsync diff -r my_new_full_package -s http://darkmod.taaaki.za.net/zipsync/release/release208_from_release207/manifest.iniz -o out_diff_package

Here:

• -r my_new_full_package specifies the path to directory where your full package is located.
• -s http://darkmod.taaaki.za.net/zipsync/release/release208_from_release207/manifest.iniz points to the manifest of the base version.
• -o out_diff_package shows a path to an empty directory which would receive the differential package.

After the command completes, the differential package is ready in out_diff_package directory. The full package is no longer needed, although you can save it for some time in case any issues arise. You can upload the whole directory of differential package to HTTP server and make it available for install in tdm_installer in one of the two ways:

• You can add it as new official version to tdm_installer config file, with base version marked as its dependency. Then users will be able to see and choose it straight in the GUI tree.
• You can post the URL of the manifest and the base version somewhere, and users will be able to install it using "Custom manifest URL" feature.

Patch approach

This is the quickest way to publish some minor modification of an existing version. It is recommended for unofficial versions and quick-test fixes.

First of all, choose some base version which you will make a patch to. You don't need to have it available locally, only its manifest is needed. It is recommended to specify URL to it, pointing to the central TDM server. These URLs can be easily deduced from the tdm_installer config file.

Then decide which files must be added or substituted in the base version. Collect all these files in some empty directory, and pack them into zip archives as you see fit. If you want to substitute a file, it must be placed in exactly the same zip, located in the same subdirectory, and with same filename and path insize zip. For instance, updated Windows 64-bit executable must be located inside tdm_shared_stuff.zip named as TheDarkModx64.exe, and updated map-file of NewJob FM must be in maps/prologue9.map file inside fms/newjob/newjob.pk4 archive. If you want to add a file, then name of zip, its subdirectory and path of file inside zip define where this file will be located after install.

Some sad news are:

• You cannot remove files from base version using this approach (but can substitute them).
• You cannot customize which files must be chmod-ed on Linux machine --- that's limitation of tdm_installer.

Hint: you can often avoid substituting files if you know the following. The game engine merges all base pk4 archives into a single virtual filesystem. If some file is present in several pk4 files, then the version in pk4 file with lexicographically greatest name wins. So if you want to override a file in tdm_gui01.pk4, you can achieve it by putting new version into tdm_gui99.pk4. This approach can be useful if you want to share an unofficial assets update without using tdm_installer.

When you have created all the additional/overriding zip archives, make them usable by zipsync the usual way:

   zipsync analyze -cn *.pk4 *.zip

This will produce a pretty small manifest.iniz file, not yet usable in tdm_installer. Now convert it to differential package against base version:

   zipsync patch -b http://darkmod.taaaki.za.net/zipsync/release/release208_from_release207/manifest.iniz

The only -b argument should point to the base manifest, either as local path or as URL. You can deduce URLs of official version manifests by looking at tdm_installer.ini config file.

After the command is over, the manifest should increase in size to the typical level of about 2 MB. Now you can upload the contents of the directory to HTTP server. If can be TDM's official server if you are part of the team, or your own HTTP server. Then publish the information about this version for your users, which includes:

1. URL of your manifest. It must be entered into "Custom manifest URL" in tdm_installer.
2. The name of base version. It must be chosen in GUI tree in tdm_installer.