Git and Github for Mappers

Introduction

Overview

This article is about using Git and Github in the context of developing a TDM fan mission. It is important to note that Git and Github are not the same thing, rather they are used together to accomplish something: the versioning and safe storage of your work in a remote location.

There is a perception that Git is a complicated tool and isn't really needed. While it is certainly true that Git can be complicated and isn't a requirement for FM development, I will attempt to show that a basic understanding of the concepts and minimal use can have a large impact on the way you work, especially if collaboration is involved.

Using Github alongside Git goes a long way to making this easier for those new to version control concepts.

What this article isn't:

• a Git tutorial, although it will cover the basics and will be enough to get going
• an evangelism piece about the benefits of using Git over other VCS tools

What is Git and Github?

Git is a version control system (https://en.wikipedia.org/wiki/Version_control)

Github is a cloud-based hosting service originally developed to host remote Git repositories, but has expanded to include many other services related to the software development cycle.

There are other cloud-based hosting services for Git repositories such as Gitlab and Bitbucket which offer similar functionality.

Benefits

Using Github has the following major benefits to the mapper:

• Versioning and history. Easily tell what's changed in your project, and when. Also comes in handy for fixing mistakes.
• Data loss protection. Your FM code stays in the cloud, protected by a resilient hosting platform owned and operated by Microsoft. There is almost zero risk your project can be lost
• Easier collaboration. If you work with others, using Github has many benefits that will be expanded upon in this article (for example, one-click synchronization of FM files for collaborators instead of passing .pk4 files around)
• Discovery and visibility. Your FM code will be publicly available in the same place for the foreseeable future (if you choose for it to be). This makes it very easy to share, for others to find on their own, or just read later if you want to remember how you did something.

Other less important features that I have found useful:

• A project board where you can create and manage work and track progress (similar to a Kanban board).
• a built-in Wiki for your project. Useful for fleshing out story ideas.
• the ability to create release artifacts. For example, you can package up your FM as a .pk4 and upload it to the project, providing a public place for others to download
• a 3D model viewer
• an image diff viewer

It is important to note that all the above features are available in the context of your FM project. Everything is in one place.

Why use a VCS and not just Google Drive/Dropbox?

Although storing your files on cloud storage protects them against data loss, there are some drawbacks:

• you need to keep zipping and uploading files all the time
• There is no easy way to tell what version you are working on compared to what is stored in your cloud storage, or what's changed
• assuming you have compressed the files for upload, you can't then browse and read the files very easily on your storage location
• for collaboration purposes, other users need to obtain the link, download and extract the files locally
• links to stored files may or may not stay around forever, or aren't discoverable by others

Tutorial

The scope of this tutorial will:

• focus on Windows users only (although Linux uses can follow along just fine - Git and VS Code are available for Linux as well)
• include the use of Visual Studio Code. The reason for using this is to demonstrate (using its Git plugin) that most Git interaction can be done without the use of the command line. If you have another preferred editor, check if it has a Git plugin and you can use that instead.

Tool Setup

Obtain a Github account

If you don't already have an account, go here and create one: https://github.com/

Install Git

You have a couple of options here, both will work:

• Git for Windows (recommended). This includes some other utilities like 'Git Bash' and a 'Git GUI' which some might find useful. Note however these are not used in the tutorial.
• Git stand-alone

First-time Git Setup

After installing Git, open a command prompt and type the following. You can put whatever you want for the values (enclose user.name in double quotes if it has a space)

git config --global user.name <your user name here>
git config --global user.email <your email here>

Now do the following to confirm it worked:

git config --global --list

Install Visual Studio Code

VS Code is a very good, lightweight code editor that has become a professional standard, but is just as useful for hobbyists. I prefer it because it because of its lightweight nature and you can find well-supported plugins for pretty much anything (a lot of which are maintained by Microsoft). You can get it here: https://code.visualstudio.com/

Start using Github

Now we are ready to create a Git repository in Github. What we are going to do here is create the FM folder in Github, and then download it to our local computer. This isn't the only way to do this, but it's the easiest.

Create a new repository

Log into your Github account and create a new repository. Depending if this is this first time you've logged in or not, you may see different screens. Either way, you will arrive at this screen:

• The 'Repository name' will be the root folder of the FM, and will appear under darkmod/fms on your local filesystem. So call it something relevant to your FM.
• Public or Private. This is up to you. If you choose public, your code will be visible to the internet. If you choose private, only yourself and collaborators will be able to access it. You can change it later if you want.

IMPORTANT: Choose one or both of the following options. If you don't choose one, the repository will be empty when we 'clone' it later, and be more difficult to work with:

• README file. Readmes are markdown files that render in the Github UI automatically when you browse to a repository. Good as a 'landing page' for your project that is immediately viewable to the user.
• .gitignore. You can list file extensions in here of files that you do not want to store in Git (e.g. .proc or .cm files). If you choose this, it forces you to choose a template - just pick any one for now (e.g. VisualStudio)

When ready, click 'Create repository'

Clone the repository to your local filesystem

Now that we've created the remote Git repository for our project, we can download it ('clone' in Git parlance) to our computer. Essentially we will now have a copy (clone) of the Git repository on our local computer.

Browse to the repository and click the 'Code' button. A pop-up will appear with a number of options. Ensure that 'HTTPS' is selected and click the 'copy' button on the right:

Now, open up a command prompt inside your darkmod/fms directory and type the following:

git clone <paste the item from your clipboard>

For example, using the repo in the screenshot we'd have:

git clone https://github.com/tdmuser/tutorialfm.git

Now, the repository should be present as a child of the FMs directory, named 'tutorialfm'

F:\games\darkmod_testing\fms>dir
Volume in drive F is DATA
Volume Serial Number is D43B-9D4F
Directory of F:\games\darkmod_testing\fms
23/05/2021  15:05    <DIR>          .
23/05/2021  15:05    <DIR>          ..
23/05/2021  15:01                45 consolehistory.dat
23/05/2021  15:01    <DIR>          hareinthesnare_v1_0_3_release
23/05/2021  15:01               593 missions.tdminfo
04/04/2021  20:05    <DIR>          newjob
04/04/2021  20:02    <DIR>          saintlucia
04/04/2021  20:05    <DIR>          stlucia
04/04/2021  20:05    <DIR>          training_mission
23/05/2021  15:05    <DIR>          tutorialfm
              2 File(s)            638 bytes
              8 Dir(s)  674,346,885,120 bytes free

Create a new Map in DarkRadiant

Now that we have our FM directory sorted, we can create a map in DR.

1. Create a 'maps' folder in darkmod\fms\tutorialfm folder
2. Open up DR and create a new map file
3. create a simple brush on the map (or else DR won't be able to open it later)
4. Click 'Save' and make sure you save it to the `darkmod\fms\tutorialfm\maps` folder. We'll call ours 'gitmap.map'

Open the FM folder in Visual Studio Code

1. Start VS Code and click File -> Open Folder...
2. Browse to darkmod\fms and click the 'tutorialfm' folder to open it

You will now see something like this:

• The green highlighted files indicate these files have been modified (added or updated, according to Git)
• The two circled items on the left are the different 'views' in VS Code. The top (current) view is the file explorer where you'll spend most of your time. The bottom one is the 'source control' view, which we'll use to commit/push the files to Git/Github.

Commit the files to local Git repository

Even though our map is empty, we can still commit it to source control. This is equivalent to the git commit command.

1. Switch to the 'version control' view in VS Code
2. Enter a commit message in the text box
3. Click the checkmark to commit the file locally

You'll notice that the green files have now disappeared from the source control view. This means that your local files are 'up to date' with those in Git repository on your computer.

Push the files to Github (remote repository)

We have committed the files to our local repository where they are now version-controlled with a commit message. But we also want to make sure we 'push' them to Github so we have a copy of our changes safely stored in the cloud. The command-line equivalent is git push.

At the bottom of your screen, you should see a 'loop' icon like so, with some arrows next to it:

The '1' next to the up arrow means there is one commit ready to push. Click the icon to push the files to Github.

NOTE: At this point, you will need to authenticate with Github. Although you may be logged into the Github UI in your browser, you are not logged in on your PC. We were able to clone the repository anonymously because we made it public. Now that we want to write to it, we must authenticate.

You will get a pop up asking for your Github credentials, and then a request to authorize the VS Code Git extension to allow access to your Github account. Just click through until this is all done. When finished, the code will be there in your Github repo.

Now, if you browse to your Github repository you'll see your new files:

From this point onwards, any files you change, add or delete locally can be similarly committed and pushed to Github where they will be safely stored. If anything happens to your local files (deletion or your computer dies), you can re-clone the Github repository and be back to where you left off.

Running the map in TDM

You will have noticed that the FM folder we cloned from Github is now sitting within your darkmod/fms folder. This means TDM will see it as a fan mission. If you have all the required files for a FM (e.g. startingmap.txt, map file, etc) and your FM is playable, all you need to do is start TDM and install the mission in the same manner as you would if using a .pk4 file, run DMAP and off you go.

For a full example of a fully developed and released mission in Github, see The Hare in the Snare, Part 1 in the TDM Github Community Organization.

Collaboration

Introduction

Let's say we have two mappers who want to work on the same FM together. Without using Git, there would need to be some mechanism to share the FM files (probably by zipping and sharing .pk4 files, exporting and sharing prefab files, etc). Using Git, this process becomes much quicker and cleaner. There is one caveat however, in that it's not recommended to have multiple people working on the map file at the same time. This is a limitation of TDM and DarkRadiant, not Git. To be clear, multiple people can work on it and export parts of it (prefabs, etc), but if more than one person pushes changes, merge conflicts will likely occur in the .map file that will be next to impossible to reconcile.

In our scenario, let's assume the following:

• mapper A: mapping and scripting. Also owns the Git repository in Github
• mapper B: models, textures, readables, etc. Basically anything but mapping. They don't own the Github repository, but they have a Github account and can be added as a collaborator.

A word about branches

I haven't mentioned anything about branching in Git until now because I only wanted to introduce concepts as they are needed. However, if you are planning to collaborate you will need to understand how to use branches. The basics are enough, and you can read about it here first.

Essentially, what we are proposing is the following:

• the default (also called 'main' by default) branch is protected, and nobody (not even mapper A, the owner) can push directly to it. This helps prevent screw ups and is highly recommended for collaborative projects.
• both mapper A and mapper B work on their own branches
• when mapper A and mapper B are finished their current task (or just want to get stuff merged), they merge their work into the default branch using something called a pull request. They can do this at any time and independently of one another (except if approvals are required - more on that later).

Project Initialization

Since mapper A will be the owner of the Github project, they create the repository in Github as described above on this page. They get the map into a good enough state for others to start working on it, and they push their changes to Github. Everything is on the 'main' branch and ready to go.

Protect the default branch

As mentioned previously, it is highly recommended to protect the default branch (called 'main' by default) to avoid damaging the code (intentionally or unintentionally). You can read more about this here. To protect the default branch, mapper A should follow the instructions here.

The following settings are recommended at minimum:

• for 'branch name pattern' choose 'main', as this will be the name of the default branch we want to protect (unless for some reason mapper A changed the name of the default branch).
• tick 'require pull requests before merging'
• set 'required approving reviews' to at least 1. You can set this to 2, and then both mapper A and mapper B would need to approve any merge to the default branch but his might become a hassle/hindrance. Entirely up to the mappers to decide.
• tick 'Dismiss stale pull request approvals when new commits are pushed'. This will require a new approval on an existing pull request if new commits are pushed.

Add a collaborator to the Github project

Now mapper A wants to invite mapper B to the project. Assuming mapper B also has a Github account, mapper A should then invite mapper B to the repository by following the instructions here.

Note that adding a collaborator entitles them to many permissions on the repository. See here for more information.

Mapper B will now get an invite in their email to join the repository.

Collaborator clones the repository

Mapper B can now clone the repository by following the instructions above

Create 'feature' branches

A 'feature branch' is just Git jargon to mean any branch that is being used to add new features to the codebase, so anything other than the default branch ('main' in our case), or if you are using other branches for testing or release.

Both mapper A and mapper B should now do the following:

To create a new branch using VS Code, do the following:

1. In the bottom left of VS Code, click the branch icon that says 'main' next to it. NOTE: this will always display the name of the current working branch. If you're not sure what branch you are on, just check here.
2. a pop up will appear. Select '+ Create new branch'
3. type in a name for your branch (no spaces), like 'mapper-b-work' or 'add-model' and press enter.

You should now have a new branch in your local Git repository. Both mapper A and mapper B can now carry on and work, committing and pushing their changes to their respective remote branches in Github.

Merging your work: raising a pull request

Say mapper B has now finished his task and wants to merge his work. Since we protected the default branch, he can't just merge it and push to it to Github. Instead, he needs to create a pull request on Github.

A pull request is essentially just a 'thing' you create that says "Hey, I want to merge these files from this branch into that branch. Can someone please review and approve?"

In essence, mapper B should do the following:

1. Make sure his changes are pushed to Github on his branch
2. Raise a pull request as detailed here

NOTE: If you have recently pushed to Github on a branch and are logged into the UI, you will see a message like the following that makes it even easier to create a pull request:

Merging your work: approving a pull request

Now that mapper B has raised his pull request, mapper A can review and approve it. In our scenario, a merge requires at least 1 approval. This needs to be done by mapper A because mapper B cannot approve his own pull requests.

To approve the pull request, mapper A needs to:

1. in the Github UI, navigate to the repository and then 'pull requests'
2. click on the pull request to review, and then click on 'Files changed' to review the changes
3. click on 'Review changes' and then 'approve'. Mapper A could instead choose to only comment, or request changes (meaning the PR can't be merged until mapper A sees and approves the desired changes).

Merging your work: merging a pull request

Now that mapper A has approved mapper B's pull request, mapper B can then merge it. To do this, follow the instructions here.

Merging your work: quick and dirty version

Back when we set up our branch protection rule to protect the default branch ('main'), notice we didn't tick the box that said 'Include administrators':

What this means is that administrators of the repository (i.e. mapper A) can merge this pull request without bothering with approvals. He will get a screen like this when he views the PR:

This is handy for the following scenarios:

• you are working alone, but still want to protect the default branch against accidental pushes
• in a collaboration scenario where mapper A wants to merge his own work without having to bother to get approvals from other collaborators

Getting the latest version of the code

Now that mapper B has his changes merged, mapper A wants to get his hands on it. He has 2 choices here. He can either:

1. 'rebase' his feature branch off of main (recommended)
2. merge his changes into main via pull request, update his local main branch, and then resume work on a new feature branch

Option 1: rebase off 'main' branch.

Essentially what this does is apply all mapper B's changes to mapper A's working branch. It takes them from main, applies them to his branch, and then applies all his other changes over top. To do this, mapper A must:

1. Commit all changes to his local branch (and optionally push)
2. Switch to the main branch by clicking the branch icon in the bottom left of VS code (the same icon we used to create a new branch - just choose the main branch instead of creating a new one this time)
3. Update the local 'main' branch by clicking the 'loop' icon in the bottom left of VS Code (the same icon we used for push). This is equivalent of performing a git pull on the command-line.
4. switch back to the feature branch (again, the branch icon)
5. open up a command line terminal (in VS Code: right-click, select 'command palette' and search for 'View: toggle terminal') and type the following:

git rebase main

Mapper A will now have the latest changes in his branch from mapper B

TODO: Git rebase without command-line (probably possible, never tried)

Option 2: merge, pull and create new branch

There is no real reason to do this except if you are afraid of screwing up the rebase. If you want to do this, then:

1. Commit, push and merge your current work as detailed above
2. switch to the main branch, and do a git pull by clicking the 'loop' icon in the bottom left of VS Code. This will pull the changes from mapper B and your changes from your feature branch that you just merged.
3. create a new branch off main and continue working.

Implications

So why do all this? It seems like a hassle right? Well no, because even though it seems like you have to do a lot here, actually performing the steps takes mere seconds. Let's look at what's happening here:

1. Mapper A and B can both work on the same FM
2. Mapper B can apply his changes to the FM and push them with a couple of clicks
3. Mapper A can then retrieve mapper B's changes and see them in his own map with a couple of clicks
4. Mapper A can run DMAP and play the changes from mapper B immediately

In real time, this process takes less than a minute, easily. No zipping files, no uploading/downloading from Dropbox, no extracting/copying files, etc.

Another useful feature of this is mapper A or mapper B can see each others work by simply switching branches. Say mapper B is working on a new section of the map that he is going to export as a prefab later. Mapper A is getting impatient and wants to look at it. All the team needs to do is:

1. ensure both have committed and pushed their latest code to Github
2. Mapper A pulls the latest code (loop icon at bottom left in VS Code) and switches to mapper B's branch
3. Mapper A views the work in DR, or runs DMAP and runs the map in TDM to have a look
4. when finished, Mapper A switches back to his own branch and resumes his work

Again, all of this would only take seconds.

Tips, Tricks and Advice

Git

the .gitignore file

the .gitignore file contains a list of filename and/or patterns to exclude from versioning. For example, you might not want to bother versioning the following:

• save games
• DMAP-generated files (.proc, .aas32, etc)
• DR backup files
• console history file
• pointfiles

An example .gitignore used for a FM can be found here.

Github

Use SSH for Github Authentication

I prefer using SSH for Github instead of HTTPS, just because I don't have to think about credentials. This is a personal preference however and there is much debate over which is 'better' or more secure. If you want to try this, you can start here.

Delete branches after merging!!!

After you merge a pull request, always delete the branch. There is usually no reason to keep it, and you'll find they pile up and pollute your repository.

Basic Collaboration Workflow

A suggested workflow for raising pull requests and merging goes like this:

1. Mapper A raises pull request and notifies Mapper B
2. Mapper B reviews and approves pull request
3. Mapper A merges pull request and deletes branch.

That's it - it doesn't need to be any more complicated than that.

Binary Files

Contrary to what you may read elsewhere, Git and Github handle binary files just fine. However there are a couple of things you need to be aware of, namely:

• if you have files larger than 100 MB, they will be blocked on push.
• If you want to use larger files you can use Git LFS, but there are bandwidth and file quotas and limits.