Git and Github for Mappers

Introduction

Overview

This article is about using Git and Github in the context of developing a TDM fan mission. It is important to note that Git and Github are not the same thing, rather they are used together to accomplish something: the versioning and safe storage of your work in a remote location.

There is a perception that Git is a complicated tool and isn't really needed. While it is certainly true that Git can be complicated and isn't a requirement for FM development, I will attempt to show that a basic understanding of the concepts and minimal use can have a large impact on the way you work, especially if collaboration is involved.

Using Github alongside Git goes a long way to making this easier for those new to version control concepts.

What this article isn't:

• a Git tutorial, although it will cover the basics and will be enough to get going
• an evangelism piece about the benefits of using Git over other VCS tools

What is Git and Github?

Git is a version control system (https://en.wikipedia.org/wiki/Version_control)

Github is a cloud-based hosting service originally developed to host remote Git repositories, but has expanded to include many other services related to the software development cycle.

There are other cloud-based hosting services for Git repositories such as Gitlab and Bitbucket which offer similar functionality.

What is a Git repository?

A Git repository is a versioned collection of related files that (usually) make up a software application, or part of an application stack. In the context of this tutorial, our Git repository will only include the TDM fan mission files and nothing else.

When you create a repository on Github however, a whole bunch of other features related to the repository become available to you.

Benefits

Using Github has the following major benefits to the mapper:

• Versioning and history. Easily tell what's changed in your project, and when. Also comes in handy for fixing mistakes.
• Data loss protection. Your FM code stays in the cloud, protected by a resilient hosting platform owned and operated by Microsoft. There is almost zero risk your project can be lost
• Easier collaboration. If you work with others, using Github has many benefits that will be expanded upon in this article (for example, one-click synchronization of FM files for collaborators instead of passing .pk4 files around). See implications for more details.
• Discovery and visibility. Your FM code will be publicly available in the same place for the foreseeable future (if you choose for it to be). This makes it very easy to share, for others to find on their own, or just read later if you want to remember how you did something. Want to show someone how you wrote that script? Just send them a link!. Can't find that .ogg file you used for secrets anywhere? Just grab it!
• It's free!. See list of features available here.

Other less important features that I have found useful:

• A project board where you can create and manage work and track progress (similar to a Kanban board).
• a built-in Wiki for your project. Useful for fleshing out story ideas.
• the ability to create release artifacts. For example, you can package up your FM as a .pk4 and upload it to the project, providing a public place for others to download
• a 3D model viewer
• an image diff viewer

It is important to note that all the above features are available in the context of your FM project. Everything is in one place.

Why use a VCS and not just Google Drive/Dropbox?

Although storing your files on cloud storage protects them against data loss, there are some drawbacks:

Feature	Git/Github	Traditional Cloud Storage
Resilient Cloud Hosting	yes	yes
Ease of file transfer	one or two clicks to synchronize, no manual compression or extracting required	manual zipping/extracting, uploading/downloading
Versioning	built-in	may be able to version files (e.g. Google Drive, but not a VCS and not practical unless packaged in a single file)
File visibility	all files easily browseable	no, especially if packaged (e.g. .pk4)
Discovery	public links, discoverable by anyone	no, private accounts, non-discoverable public links, may or may not be available
Online Collaboration	yes, via pull request	no

Tutorial

The scope of this tutorial will:

• focus on Windows users only (although Linux uses can follow along just fine - Git and VS Code are available for Linux as well)
• include the use of Visual Studio Code. The reason for using this is to demonstrate (using its Git plugin) that most Git interaction can be done without the use of the command line. If you have another preferred editor, check if it has a Git plugin and you can use that instead.

Tool Setup

Obtain a Github account

If you don't already have an account, go here and create one: https://github.com/

Install Git

You have a couple of options here, both will work:

• Git for Windows (recommended). This includes some other utilities like 'Git Bash' and a 'Git GUI' which some might find useful. Note however these are not used in the tutorial.
• Git stand-alone

First-time Git Setup

After installing Git, open a command prompt and type the following. You can put whatever you want for the values (enclose user.name in double quotes if it has a space)

git config --global user.name <your user name here>
git config --global user.email <your email here>

Now do the following to confirm it worked:

git config --global --list

Install Visual Studio Code

VS Code is a very good, lightweight code editor that has become a professional standard, but is just as useful for hobbyists. I prefer it because it because of its lightweight nature and you can find well-supported plugins for pretty much anything (a lot of which are maintained by Microsoft). You can get it here: https://code.visualstudio.com/

Start using Github

Now we are ready to create a Git repository in Github. What we are going to do here is create the FM folder in Github, and then download it to our local computer. This isn't the only way to do this, but it's the easiest.

Create a new repository

NOTE: if you have an existing FM that you want to add to Github now, see Adding an Existing FM to Github.

Log into your Github account and create a new repository. Depending if this is this first time you've logged in or not, you may see different screens. Either way, you will arrive at this screen:

• The 'Repository name' will be the root folder of the FM, and will appear under darkmod/fms on your local filesystem. So call it something relevant to your FM.
• Public or Private. This is up to you. If you choose public, your code will be visible to the internet. If you choose private, only yourself and collaborators will be able to access it. You can change it later if you want.

IMPORTANT: Choose one or both of the following options. If you don't choose one, the repository will be empty when we 'clone' it later, and be more difficult to work with:

• README file. Readmes are markdown files that render in the Github UI automatically when you browse to a repository. Good as a 'landing page' for your project that is immediately viewable to the user.
• .gitignore. You can list file extensions in here of files that you do not want to store in Git (e.g. .proc or .cm files). If you choose this, it forces you to choose a template - just pick any one for now (e.g. VisualStudio)

When ready, click 'Create repository'

Clone the repository to your local filesystem

Now that we've created the remote Git repository for our project, we can download it ('clone' in Git parlance) to our computer. Essentially we will now have a copy (clone) of the Git repository on our local computer.

Browse to the repository and click the 'Code' button. A pop-up will appear with a number of options. Ensure that 'HTTPS' is selected and click the 'copy' button on the right:

Now, open up a command prompt inside your darkmod/fms directory and type the following:

git clone <paste the item from your clipboard>

For example, using the repo in the screenshot we'd have:

git clone https://github.com/tdmuser/tutorialfm.git

Now, the repository should be present as a child of the FMs directory, named 'tutorialfm'

F:\games\darkmod_testing\fms>dir
Volume in drive F is DATA
Volume Serial Number is D43B-9D4F
Directory of F:\games\darkmod_testing\fms
23/05/2021  15:05    <DIR>          .
23/05/2021  15:05    <DIR>          ..
23/05/2021  15:01                45 consolehistory.dat
23/05/2021  15:01    <DIR>          hareinthesnare_v1_0_3_release
23/05/2021  15:01               593 missions.tdminfo
04/04/2021  20:05    <DIR>          newjob
04/04/2021  20:02    <DIR>          saintlucia
04/04/2021  20:05    <DIR>          stlucia
04/04/2021  20:05    <DIR>          training_mission
23/05/2021  15:05    <DIR>          tutorialfm
              2 File(s)            638 bytes
              8 Dir(s)  674,346,885,120 bytes free

Create a new Map in DarkRadiant

Now that we have our FM directory sorted, we can create a map in DR.

1. Create a 'maps' folder in darkmod\fms\tutorialfm folder
2. Open up DR and create a new map file
3. create a simple brush on the map (or else DR won't be able to open it later)
4. Click 'Save' and make sure you save it to the `darkmod\fms\tutorialfm\maps` folder. We'll call ours 'gitmap.map'

Open the FM folder in Visual Studio Code

1. Start VS Code and click File -> Open Folder...
2. Browse to darkmod\fms and click the 'tutorialfm' folder to open it

You will now see something like this:

• The green highlighted files indicate these files have been modified (added or updated, according to Git)
• The two circled items on the left are the different 'views' in VS Code. The top (current) view is the file explorer where you'll spend most of your time. The bottom one is the 'source control' view, which we'll use to commit/push the files to Git/Github.

Commit the files to local Git repository

Even though our map doesn't have much in it, we can still commit it to source control. This is equivalent to the git commit command.

1. Switch to the 'version control' view in VS Code
2. Enter a commit message in the text box
3. Click the checkmark to commit the file locally

You'll notice that the green files have now disappeared from the source control view. This means that your local files are 'up to date' with those in Git repository on your computer.

Push the files to Github (remote repository)

We have committed the files to our local repository where they are now version-controlled with a commit message. But we also want to make sure we 'push' them to Github so we have a copy of our changes safely stored in the cloud. The command-line equivalent is git push.

At the bottom of your screen, you should see a 'loop' icon like so, with some arrows next to it:

The '1' next to the up arrow means there is one commit ready to push. Click the icon to push the files to Github.

NOTE: At this point, you will need to authenticate with Github. Although you may be logged into the Github UI in your browser, you are not logged in on your PC. We were able to clone the repository anonymously because we made it public. Now that we want to write to it, we must authenticate.

You will get a pop up asking for your Github credentials, and then a request to authorize the VS Code Git extension to allow access to your Github account. Just click through until this is all done (NOTE: If using SSH, all of this goes away). When finished, the code will be there in your Github repo.

Now, if you browse to your Github repository you'll see your new files:

From this point onwards, any files you change, add or delete locally can be similarly committed and pushed to Github where they will be safely stored. If anything happens to your local files (deletion or your computer dies), you can re-clone the Github repository and be back to where you left off.

Running the map in TDM

You will have noticed that the FM folder we cloned from Github is now sitting within your darkmod/fms folder. This means TDM will see it as a fan mission. If you have all the required files for a FM (e.g. startingmap.txt, map file, etc) and your FM is playable, all you need to do is start TDM and install the mission in the same manner as you would if using a .pk4 file, run DMAP and off you go.

For a full example of a fully developed and released mission in Github, see The Hare in the Snare, Part 1 in the TDM Github Community Organization.

Collaboration (Advanced)

Introduction

Let's say we have two mappers who want to work on the same FM together. Without using Git, there would need to be some mechanism to share the FM files (probably by zipping and sharing .pk4 files, exporting and sharing prefab files, etc). Using Git, this process becomes much quicker and cleaner. There is one caveat however, in that it's not recommended to have multiple people working on the map file at the same time. This is a limitation of TDM and DarkRadiant, not Git. To be clear, multiple people can work on it and export parts of it (prefabs, etc), but if more than one person pushes changes, merge conflicts will likely occur in the .map file that will be next to impossible to reconcile.

In our scenario, let's assume the following:

• mapper A: mapping and scripting. Also owns the Git repository in Github
• mapper B: models, textures, readables, etc. Basically anything but mapping. They don't own the Github repository, but they have a Github account and can be added as a collaborator.

A word about branches

I haven't mentioned anything about branching in Git until now because I only wanted to introduce concepts as they are needed. However, if you are planning to collaborate you will need to understand how to use branches. The basics are enough, and you can read about it here first.

Essentially, what we are proposing is the following:

• the default (also called 'main' by default) branch is protected, and nobody (not even mapper A, the owner) can push directly to it. This helps prevent screw ups and is highly recommended for collaborative projects.
• both mapper A and mapper B work on their own branches
• when mapper A and mapper B are finished their current task (or just want to get stuff merged), they merge their work into the default branch using something called a pull request. They can do this at any time and independently of one another (except if approvals are required - more on that later).

Project Initialization

Since mapper A will be the owner of the Github project, they create the repository in Github as described above on this page. They get the map into a good enough state for others to start working on it, and they push their changes to Github. Everything is on the 'main' branch and ready to go.

Protect the default branch

NOTE: this feature is only available to free accounts if you are using public repositories!

As mentioned previously, it is highly recommended to protect the default branch (called 'main' by default) to avoid damaging the code (intentionally or unintentionally). You can read more about this here. To protect the default branch, mapper A should follow the instructions here.

The following settings are recommended at minimum:

• for 'branch name pattern' choose 'main', as this will be the name of the default branch we want to protect (unless for some reason mapper A changed the name of the default branch).
• tick 'require pull requests before merging'
• set 'required approving reviews' to at least 1. You can set this to 2, and then both mapper A and mapper B would need to approve any merge to the default branch but his might become a hassle/hindrance. Entirely up to the mappers to decide.
• tick 'Dismiss stale pull request approvals when new commits are pushed'. This will require a new approval on an existing pull request if new commits are pushed.

Add a collaborator to the Github project

Now mapper A wants to invite mapper B to the project. Assuming mapper B also has a Github account, mapper A should then invite mapper B to the repository by following the instructions here.

Note that adding a collaborator entitles them to many permissions on the repository. See here for more information.

Mapper B will now get an invite in their email to join the repository.

Collaborator clones the repository

Mapper B can now clone the repository by following the instructions above

Create 'feature' branches

A 'feature branch' is just Git jargon to mean any branch that is being used to add new features to the codebase, so anything other than the default branch ('main' in our case), or if you are using other branches for testing or release.

Both mapper A and mapper B should now do the following:

To create a new branch using VS Code, do the following:

1. In the bottom left of VS Code, click the branch icon that says 'main' next to it. NOTE: this will always display the name of the current working branch. If you're not sure what branch you are on, just check here.
2. a pop up will appear. Select '+ Create new branch'
3. type in a name for your branch (no spaces), like 'mapper-b-work' or 'add-model' and press enter.

You should now have a new branch in your local Git repository. Both mapper A and mapper B can now carry on and work, committing and pushing their changes to their respective remote branches in Github.

Merging your work: raising a pull request

Say mapper B has now finished his task and wants to merge his work. Since we protected the default branch, he can't just merge it and push to it to Github. Instead, he needs to create a pull request on Github.

A pull request is essentially just a 'thing' you create that says "Hey, I want to merge these files from this branch into that branch. Can someone please review and approve?"

In essence, mapper B should do the following:

1. Make sure his changes are pushed to Github on his branch
2. Raise a pull request as detailed here

NOTE: If you have recently pushed to Github on a branch and are logged into the UI, you will see a message like the following that makes it even easier to create a pull request:

Merging your work: approving a pull request

Now that mapper B has raised his pull request, mapper A can review and approve it. In our scenario, a merge requires at least 1 approval. This needs to be done by mapper A because mapper B cannot approve his own pull requests.

To approve the pull request, mapper A needs to:

1. in the Github UI, navigate to the repository and then 'pull requests'
2. click on the pull request to review, and then click on 'Files changed' to review the changes
3. click on 'Review changes' and then 'approve'. Mapper A could instead choose to only comment, or request changes (meaning the PR can't be merged until mapper A sees and approves the desired changes).

Merging your work: merging a pull request

Now that mapper A has approved mapper B's pull request, mapper B can then merge it. To do this, follow the instructions here.

Merging your work: quick and dirty version

Back when we set up our branch protection rule to protect the default branch ('main'), notice we didn't tick the box that said 'Include administrators':

What this means is that administrators of the repository (i.e. mapper A) can merge this pull request without bothering with approvals. He will get a screen like this when he views the PR:

This is handy for the following scenarios:

• you are working alone, but still want to protect the default branch against accidental pushes
• in a collaboration scenario where mapper A wants to merge his own work without having to bother to get approvals from other collaborators

Getting the latest version of the code

Now that mapper B has his changes merged, mapper A wants to get his hands on it. He has 2 choices here. He can either:

1. merge main into his feature branch (recommended)
2. merge his changes into main via pull request, update his local main branch, and then resume work on a new feature branch

Option 1: merge from main branch

This is the simplest option. This will just merge changes from the main branch into the feature branch. Mapper A should:

1. Commit all changes to his local branch (and optionally push)
2. Switch to the main branch by clicking the branch icon in the bottom left of VS code (the same icon we used to create a new branch - just choose the main branch instead of creating a new one this time)
3. Update the local 'main' branch by clicking the 'loop' icon in the bottom left of VS Code (the same icon we used for push). This is equivalent of performing a git pull on the command-line.
4. switch back to the feature branch (again, the branch icon)
5. From the source control view, click the 3 dots at the top of the view, and select Branch -> Nerge branch... and then select the 'main' branch

Mapper A will now have the latest changes from main, and can continue working.

Option 2: merge, pull and create new branch

There is no real reason to do this except if you are afraid of screwing up the merge. All the merging will be handled by Github. If you want to do this, then:

1. Commit, push and merge your current work as detailed above
2. switch to the main branch, and do a git pull by clicking the 'loop' icon in the bottom left of VS Code. This will pull the changes from mapper B and your changes from your feature branch that you just merged.
3. create a new branch off main and continue working.

Implications

So why do all this? It seems like a hassle right? Well no, because even though it seems like you have to do a lot here, actually performing the steps takes mere seconds. Let's look at what's happening here:

1. Mapper A and B can both work on the same FM
2. Mapper B can apply his changes to the FM and push them with a couple of clicks
3. Mapper A can then retrieve mapper B's changes and see them in his own map with a couple of clicks
4. Mapper A can run DMAP and play the changes from mapper B immediately

In real time, this process takes less than a minute, easily. No zipping files, no uploading/downloading from Dropbox, no extracting/copying files, etc.

Another useful feature of this is mapper A or mapper B can see each others work by simply switching branches. Say mapper B is working on a new section of the map that he is going to export as a prefab later. Mapper A is getting impatient and wants to look at it. All the team needs to do is:

1. ensure both have committed and pushed their latest code to Github (again, a couple of seconds)
2. Mapper A pulls the latest code (loop icon at bottom left in VS Code) and switches to mapper B's branch
3. Mapper A views the work in DR, or runs DMAP and runs the map in TDM to have a look
4. when finished, Mapper A switches back to his own branch and resumes his work

Again, all of this would only take seconds.

Release Management

Features

Use of Github's other features such as Issues, pull requests and releases is very useful for managing the release cycle of your FM. With it, you can:

1. use it as your own personal bug tracker for just this particular FM. If it's public, then anyone can see it (or raise bugs if they have a Github account)
2. use the projects feature to organise your work into columns (e.g. to do, in progress, done, etc)
3. host the .pk4 files for download by your beta testers
4. create automated changelogs (aka Release Notes) for new releases so everyone knows what's been fixed

Setup

• Add labels to your repo that you think you might need. There are a bunch of defaults, but you may want to add more (e.g. 'gameplay', 'improvement', 'readables', etc.): https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels
• Once you have your labels, add the required YAML file to your repo for generating automated release notes: https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes#configuring-automatically-generated-release-notes

Initial Beta Release

You're ready for beta and want to make the .pk4 available for your beta testers. Do the following:

1. Ensure all your branches are cleaned up and you are only left with the main branch
2. Do a clean DMAP of your FM
3. Package up the .pk4. I use a Powershell script, but you can do whatever suits you.
4. Test the .pk4 on a clean install of TDM. I have a separate 'test' install and just delete/reinstall the .pk4 whenever I have a new version.
5. Once you're happy there are no issues, create a release in your Github repo. Use something like 'v1.0-beta', or 'v0.1' for the tag name to indicate it's a pre-release.
6. Attach the .pk4 file to the release by dropping it into the release page. You can edit the release and add it after if you forget.
7. Share the link to the .pk4 with your beta testers.

Bugfix Workflow

This is my own personal workflow. It suits me, and helps keep me organised and makes it easy to share progress with beta testers.

It is strongly recommended to have branch protection turned on for this workflow. This will prevent any accidental changes.

It looks like a lot of steps, but like other parts of this tutorial it becomes second nature and only takes a few seconds to perform.

1. Once a bug or change is identified, create an issue in your repository. For example, let's say we now have issue #8.
2. When you are ready to tackle the bug, make sure your local main branch is synced and create a new branch off of main in your repo just for this issue. Call the branch something like 'fix/8' to associate the branch with the actual issue you are fixing.
3. If you have DarkRadiant open already, reload the map. If you don't do this, DR will notify you with a pop-up message saying the underlying file has changed.
4. Make your changes in DR for the bugfix and test it, etc.
5. When you are ready to commit the fix, commit and push the new branch to Github.
6. Go to your Github repo in the browser and view the 'code' section. Github should notify you that a new branch has been pushed and ask you if you want to raise a pull request. Do so.
7. In the pull request (PR for short), make sure the title is something descriptive as it will show up in the release notes later. For example 'Fix z-fighting on castle wall'.
8. In the description, reference the issue you are fixing. This will associate the PR with the issue. For example, type 'fixes #8'.
9. Add a label to the pull request (e.g. 'bug'). This will categorise the fix in the automatically generated release notes.
10. Click 'create pull request' and then 'merge pull request'.
11. Before you merge, you can add a merge commit comment. Type 'fixes #8' here as well. What this does is automatically close the issue that the PR is associated with.
12. Click 'confirm merge'
13. Delete the branch
14. At this point, if you have TDM open, close it. If you don't, when you change branches in the next step Git will think the local map file has been modified (not sure why this happens).
15. In your local git repo on your PC, change back to the 'main' branch and sync changes (or 'git pull' if using command line).
16. If you didn't close TDM or for some reason you get a merge conflict at this point, just revert any local changes and pull from Github. Your changes have been stored safely in Github and branch protection will prevent anything from getting lost/overwritten.

Subsequent Beta Releases

This is exactly the same as the [beta release], except you will:

• name the release (and tag) differently, say 'v0.2-beta' or 'v0.2'
• When creating the release in Github, click the 'generate release notes' button. This will auto-generate a changelog based on the pull requests that have been merged since the last tagged release. They should be categorised by label if you added the release YAML config mentioned in the setup.

Tips, Tricks and Advice

Git

the .gitignore file

the .gitignore file contains a list of filename and/or patterns to exclude from versioning. For example, you might not want to bother versioning the following:

• save games
• DMAP-generated files (.proc, .aas32, etc)
• DR backup files
• console history file
• pointfiles

An example .gitignore used for a FM can be found here.

Github

Use SSH for Github Authentication

I prefer using SSH for Github instead of HTTPS, just because I don't have to think about credentials. This is a personal preference however and there is much debate over which is 'better' or more secure. If you want to try this, you can start here.

Delete branches after merging!!!

After you merge a pull request, always delete the branch. There is usually no reason to keep it, and you'll find they pile up and pollute your repository.

Basic Collaboration Workflow

A suggested workflow for raising pull requests and merging goes like this:

1. Mapper A raises pull request and notifies Mapper B
2. Mapper B reviews and approves pull request
3. Mapper A merges pull request and deletes branch.

That's it - it doesn't need to be any more complicated than that.

Binary Files

Contrary to what you may read elsewhere, Git and Github handle binary files just fine. However there are a couple of things you need to be aware of, namely:

• if you have files larger than 100 MB, they will be blocked on push. If you are making cutscenes and brief/debrief videos, keep them under 100 MB. See this forum post for more information.
• If you want to use larger files you can use Git LFS, but there are bandwidth and file quotas and limits.

Adding an Existing FM to Github

If you are already working on a FM and want to add it to Github now, you can do that too. Just do the following:

1. Make sure you have done the prerequisites listed here.
2. Create an empty Git repository by following the instructions here, but don't add a readme or .gitignore file.
3. You will see a page that looks similar to this:

Before you start executing the commands, please read the following:

• what this is doing is creating a local Git repository and then pushing it to the remote repository on Github you just created
• the links it displays for the remote repository will be different from the screenshot above (they will be specific to your repository, so use those)
• You need to be on the command-line inside the root of your FM folder (e.g. darkmod/fms/myfm)
• these commands assume you are creating a new repository from scratch and you have no files. You do have files though, so skip the first echo command to create the readme.
• instead of typing git add README.md (since we skipped creating it), add all files in the current directory by typing git add * instead
• you can follow the rest of the commands as is