Mission Database
All released missions are stored in FM database, which defines what is available in mission downloader in-game.
Requirements
In order for a mission to be included in the database, it must satisfy the following conditions:
- It has gone through beta-testing by several forum members who did not take part in its creation.
- As a rare exceptional case, a mission can be rejected from the database if the majority of beta-testers come to conclusion that it is unplayable or its quality is way too low.
- Does not contain any questionable material, potential intellectual property infridgements. Including:
- using names from the original Thief games
- using assets from other games
- having content which is so bad that it is forbidden even on our forums (TODO: link to forum rules?)
A very few missions are installed along with the game, like Saint Lucia or Training Mission. These official missions are considered to be part of the game, and are not included in the database.
SVN
Starting from 2021, the FM database is stored in SVN repository (#5551). Only TDM team members have access to this SVN.
Here is the SVN address:
But please don't rush to checkout the whole repo yet!
Checkout everything
Yes, you can simply copy the link above to SVN Checkout and get single working copy with all the FMs. But keep in mind that you will have to download ~10 GB of data and the working copy will take ~20 GB of space. In most cases you don't need everything in order to work with the database.
The typical reasons to checkout the whole repo are:
- You want to run automated search or tests over all released FMs.
- You are regular committer of the FM database, added a lot of new missions and updates, so the investment is worth it.
Checkout as needed
Instead of checking out the whole repo, you can checkout only the few FMs you are going to modify. This is the recommended approach.
With this approach, you need the Repo-browser feature of TortoiseSVN. It allows to look through all the directories and files on remote SVN without checking it out first.
The directory structure of SVN is show on the picture. Most importantly, all information about FM with internal name "qwerty" is stored in fms/qwerty subdirectory of the repo. You can checkout only this directory in order to work with FM, just put this checkout address:
Or find the FM directory in Repo-browser, right-click and select Checkout.
Detailed instructions in the rest of the article assume this approach.
Add New Mission
Before adding new FM, negotiate internal name with FM author. It must be different from names of all existing FMs, consist of only lowercase letters and digits, be rather short (aim for 10-20 letters). Among many words in the mission title, prefer rare words and proper nouns over common words when composing the internal name.
Open SVN in Repo-browser, find fms directory which contains all the FMs. Right-click on it and choose Add folder in the context menu. Then enter the internal name of FM as the name of the new directory and proceed with commit. Now that the directory has been created, you can checkout it: find the directory in Repo-browser, right-click it and Checkout.
The second step is to upload pk4 files. Copy all the pk4 files of FM into the working copy directory (i.e. checkout directory). Open the directory in Windows Explorer, right-click and select SVN commit. Select all pk4 files with Shift, right-click and select Add. Also set checkboxes for these files. Write commit message in the text area above: it should start with internal name of FM in brackets. When everything is done, hit OK to do the actual commit.
While the pk4 files are already in the repository, they are not yet visible in the database. A mission is only added when its directory contains fminfo.xml file, so now you need to add it. You can take this file from another FM (find it in Repo-browser, right-click, Open with, select text editor), and adjust it for the FM being added. Here is explanation for some fields:
- internalName defines name of the directory and pk4 file.
- title is the name seen by players in-game.
- author is one or several people who made the mission.
- releaseDate shows when the very first version of the mission was added.
- type is multi if mission contains several playable .map files (i.e. is campaign), and single otherwise.
- size is size of the main pk4 file in megabytes displayed to users.
- version is natural number used by in-game downloader to decide whether update is available or not. Starts with 1.
- description contains text displayed in in-game downloader when player inspects mission details.
- mainPack points to the main pk4 file of the mission. Note that the name of file is fully determined by internal name.
- localisationPack points to _l10n.pk4 file if it exists.
Note that XML cannot directly contains some characters, thus they must be escaped:
- Ampersand (&) quotes (" or ') and angle brackets (< or >): reference
- Line break symbol can be inserted as according to reference
When you have created fminfo.xml file, double-check that all properties are correct. Then use SVN to add and commit the file, just like you did with pk4 file.
If you did something wrong (most likely), then you will see an error saying that "Commit blocked by pre-commit hook". A long stacktrace from Python script is included, and the meaningful message should be at the end of it. Typically, it is either XML validation error saying that something in fminfo.xml is wrong, or a message from some custom failed check. You need to fix the errors and try to commit again --- until you manage to commit successfully.
As the last step, create subdirectory named screenshots in the working copy directory. Put screenshots in .jpg or .png format into the directory: they will be displayed in in-game mission downloader. Then add and commit all the screenshot files into the repository, same way as you did for pk4 and xml files.
Update Mission
This section covers the case if mission has already been released, but new version should be uploaded.
First of all, make sure you have up-to-date working copy of the FM directory. If you already have working copy, do right-click and SVN Update in it in Windows Explorer. If you don't have it yet, then open Repo-browser, find the directory named by FM's internal name, right-click and select Checkout. If you don't know the internal name, you can learn it like this: install the FM in the game, then look what is written in the currentfm.txt file. Suppose internal name is "qwerty".
There are two ways to update the FM:
- Minor update changes one or several lines in some file, e.g. in .map file or in .gui file. Such update is usually done when TDM engine is changed, FM gets broken, and merely a small spawnarg change is enough to return proper behavior. Usually such changes are done by TDM developers with approval of FM authors.
- Major update covers everything else. It happens when FM author sends a new version of pk4 file, with arbitrary changes in .map file, and probably some other changes.
When doing minor update, prefer editing existing pk4 archive instead of creating a new one from scratch. For instance, you can open the archive in 7-zip, extract only one file which you want to modify, change it in text editor, then add it back to the archive in 7-zip (overwriting the old version). It is a bad idea to extract everything, do some changes, then create a brand new pk4 archive.
Regardless of how major the update is, keep in mind SVN storage specifics to avoid rapid growth of server storage requirements. Unless full size of FM package is smaller than 100 MB, please run the diff optimization tool. The tool is located in devel\pk4diff\bin in the assets SVN repo. With Python 3 installed, copy its files to the FM directory and run there:
python pk4diff.py --optimize qwerty.pk4
The tool will check how much data is really changed inside the archive. If the changes account for less than 10% of the full package size, then the tool will repack it to ensure that this update takes only a very little additional space on server. See more details in #Storage Concerns section.
Now you need to edit fminfo.xml file:
- Increment the integer in "version" property by one. Otherwise in-game downloader won't know that the mission has been updated.
- Probably update "size" property if it has changed significantly.
After you have dealt both with pk4 storage concerns and with XML file update, it's time to commit the changes. Open working copy directory in Windows Explorer, right-click and select SVN Commit. You should see at least two files marked as modified with checkboxes set: qwerty.pk4 and fminfo.xml. Write commit message which describes the changes. Start commit message with FM internal name in brackets. If the new version came from the author, write something like "New version: sent to me by XyMegaMapper yesterday". If the update did not come from FM author, then all the changes must be described in full detail! In case of minor update, it can be something like "In guis/mainmenu_custom_defs.gui, removed MM_BRIEFING_VIDEO_MATERIAL_K defines for K > 1". Finally, click OK to commit the changes.
Storage Concerns
TODO
