Hosting The Dark Mod: Difference between revisions

From The DarkMod Wiki
Jump to navigationJump to search
← Older edit
Tels (talk | contribs)
2,033 edits
fix spelling, add image
Revamped the whole article for tdm_installer and SVN-based missions downloader
 
Line 3: Line 3:
== How it works ==
== How it works ==


'''TDM''' is distributed via HTTP or HTTPS only.  
The game and missions are distributed via '''HTTP''' or HTTPS only.
Both tdm_installer and the in-game missions downloader do '''not''' support FTP, HTTP redirects, meta-refreshes or other methods.


{{infobox|Both [[tdm_update]] (downloads and updates the mod) and the in-game downloader (downloads and updates [[Fan Missions]]) '''don't''' support FTP, HTTP redirects, meta-refreshes or other methods.}}
When the game itself is installed or updated, first '''tdm_installer.ini''' is downloaded.
This file contains a list of mirror servers.
Each mirror has weight, which defines probability that it will be used for a request.
The weights can be used to distribute traffic across mirrors.
See also: [[Tdm installer and zipsync]]


 
The in-game mission downloader downloads '''available_missions.xml''', which contains list of URLs and their weights for every mission.
When the mod itself is updated, first '''tdm_mirrors.txt''' is downloaded. This file contains a list of mirror servers and a score, which helps balance the traffic. Here is a list of the currently know mirrors:
The weights also define probability of each URL being used in a request.
 
The XML file is generated from various XML files in the missions SVN repository.
<pre>
See also: [[Mission Database]]
# This file lists mirror sites for The Dark Mod, which are used by the
# tdm_update utility to update your TDM installation.
 
[Mirror KeepOfMetalAndGold]
url = http://www.keepofmetalandgold.com/files/tdm/
weight = 10
 
[Mirror ShadowDarkKeep]
url = http://www.shadowdarkkeep.com/files/tdm/
weight = 3
 
[Mirror SouthQuarter]
url = http://www.southquarter.com/tdm/
weight = 6
 
[Mirror roggen.jmnet.us]
url = http://roggen.jmnet.us/darkmod/
weight = 4
 
[Mirror fidcal.com]
# unlimited traffic
url = http://www.fidcal.com/darkuser/
weight = 6
 
[Mirror robertrueger.de]
url = http://thedarkmod.robertrueger.de/
weight = 10
 
[Mirror bloodgate.com]
# 100 Gbyte traffic
url = http://bloodgate.com/mirrors/tdm/pub/pk4/
weight = 1
 
[Mirror swift-mazes.com]
# 1 Tbyte traffic
url = http://swift-mazes.com/mirrors/tdm/release/
weight = 2
 
[Mirror tunnels-of-danger.com]
# 1 Tbyte traffic (shared with swift-mazes.com)
url = http://tunnels-of-danger.com/mirrors/tdm/release/
weight = 2
 
[Mirror molez.net]
# unlimited traffic
url = http://www.molez.net/mirror/tdm/
weight = 5
</pre>


== What you need ==
== What you need ==


* A server with a fast connection, preferable with either unlimited traffic, or at least 1 Terabyte per month.
* A server with fast connection, preferably with either unlimited traffic, or at least '''1000 GB per month'''.
* Space: <abbr title="2.2 Gbyte MOD plus 0.8 Gbyte differential update archives">'''3 GByte'''</abbr> for the mod itself, or <abbr title="At the moment about 1.5 Gbyte, but the FM archive is growing every month">'''2 Gbyte'''</abbr> for the [[Fan Missions|FMs]] (about '''5 GByte for both''')
* Space: '''6 GB''' for the game mirror, '''10 GB''' for the missions mirror (about '''20 GB''' is enough for both).
* A way to give the TDM core team access to your server, preferable via FTP login. If that cannot be arranged, you need to be available when files need to be updated.
* Synchronization with main server is done via '''rsync''' pull at regular intervals (once per day or week).
* The server must respond to individual file requests via HTTP with a "200 OK" and the content. Things like redirects (302, 307 etc.) as well as intermediate HTML pages (meta refresh), or FTP are currently not supported by the updater nor the in-game downloader.
* The server must respond to individual file requests via HTTP with a "200 OK" and the content. Things like redirects (302, 307 etc.) as well as intermediate HTML pages (meta refresh), or FTP are not supported.
* Game mirror must also support [https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests#multipart_ranges multipart byterange requests]. Missions mirror does not need to support byteranges at all.


== Hosting the Mod ==
== Set up hosting ==


The mod itself needs approximately '''3 Gbyte''' space. It is possible to host only parts of it by deleting the biggest files. This however causes the updater to hit files needlessly, so if possible, the entire mod should be hosted.
Here is the generic process of adding a mirror:
# Get in contact with TDM team about mirror hosting.
# Send IP address of your server to TDM server admin, so that he adds it to the whitelist for rsync access.
# Configure your server to perform rsync pull regularly (e.g. every day). Ask admin about rsync address to pull from.
# After all data is rsync-ed successfully, send base URL (where it is served at) to TDM release manager for testing.
# If everything works fine, then TDM release manager adds your mirror to config file, and players start using it.


Here is a list of files and their MD5 sums that you need: [[TDM filelist]]
== Traffic ==


Once you have added all the files to your server, a TDM developer can add an entry to the ''tdm_mirrors.txt'' file. This file contains the base URL for each server, as well as a score. The score helps balance the traffic between the mirrors.
Please pay attention to additional traffic on your mirror:
* for one month after the mirror is enabled;
* for one month after a major release of TDM;
The initial amount of traffic can be pretty high, and there are typically traffic spikes shortly after every major release of the game.
Make sure you won't have to pay extra money and won't get your server down.


If you look for a fast way to mirror the entire mod or the FMs onto your server, you can use this  [http://swift-mazes.com/mirrors/tdm/release/mirror_tdm.pl.txt Perl script]. Adjust the Path and maybe the mirror list (remove your own host!) at the top and run it on your server.
If amount of traffic is too high for you, contact TDM team.
It is possible to reduce weight of your mirror, then players will use it with less probability, and the traffic should decrease proportionally.
Here is how exactly it can be done:
* For game mirror: file <tt>tdm_installer.ini</tt> stores weights for every mirror. It can only be updated via SSH access to the main TDM server.
* For mission mirror: file <tt>mirrors/mirrorinfo.xml</tt> in missions SVN repository supports "weight" attribute near "rootUrl" (default = 1).


=== If you have not enough space ===
Some interesting tendencies about traffic:
* Tdm_installer always updates the game in differential fashion, so updates are cheap: 20 MB between close versions, hundreds of MB between major releases. However, clean install takes about 4 GB of traffic.
* In-game mission downloader provides an easy way to download all FMs, which is very likely to increase traffic on mission mirrors. Aside from that, some recent missions are quite heavy by themselves.


If you have not enough space for the entire mod, then hosting some files will still help. Here are a few files you can safely delete, because these are almost never used (most people either install from scratch or update from the latest version, v1.07 to v1.08):
== Bots ==
 
* tdm_update_1.02_to_1.03.zip '''(215 Mbyte)'''
* tdm_update_1.03_to_1.04.zip '''(263 Mbyte)'''
* tdm_update_1.04_to_1.05.zip '''(11 Mbyte)'''
* tdm_update_1.05_to_1.06.zip '''(50 Mbyte)'''
* tdm_update_1.06_to_1.07.zip '''(185 Mbyte)'''
 
Together these save you about 750 Mbytes.
 
== Hosting FMs ==
 
All FMs todate need about '''2 Gbyte''' space. It is possible to host only certain FMs (the newest, or the biggest). Since every FM is added individualy to the mirror list, this causes no harm.
 
For a list of all available FMs and their current download locations, see [http://thedarkmod.com/missions/ here].
 
Once you have added an FM file to your server, you need to send the URL to a TDM developer (Bikerdue, nbohr1more, Springheel or Tels) and let them add the URL to the mirror list. The URLs are both used by the in-game downloader as well as made public on the webpage. This happens automatically.
 
== Keeping Traffic under Control ==
 
[[Image:Traffic bloodgate release v1.08.png|right]]
 
Normally, mnost updates of TDM are done by downloading a small differential file (usually only a few hundred Mbytes). Shortly after a release, and from time to time, there are people who install the mod from scratch. This can cause quite a traffic spike, but usually subsides quickly.
 
On the right you can see the effect.


What costs traffic, however, are bots, who are either blindly following links, or traversal directories on your server. Here are a few tips to keep them out and your traffic low:
What costs traffic, however, are bots, who are either blindly following links, or traversal directories on your server. Here are a few tips to keep them out and your traffic low:
* disallow directory traversal on your server. If you can open http://yourserver/thedarkmodfiles/ on your server and see a listing, it is still enabled.
* disallow directory traversal on your server. If you can open http://yourserver/thedarkmodfiles/ on your server and see a listing, it is still enabled.
* use a robots.txt to keep Google, Bing, Yahoo, Baidu etc out of this directory (better disallow all robots)
* use a robots.txt to keep Google, Bing, Yahoo, Baidu etc out of this directory (better disallow all robots)
* disallow anything that has a known bot or spider signature (wget, Java, libwww, common bot names, or anything that has a referer of the file itself (PK4 files cannot be refering to themselves).
* disallow anything that has a known bot or spider signature (wget, Java, libwww, common bot names, or anything that has a referrer of the file itself (PK4 files cannot be referring to themselves).
 
If all else fails, block anything except [[tdm_update]]. Here is its signatures:
 
"GET /pk4/tdm_sound_vocals06.pk4 HTTP/1.1" 200 33639507 "-" "The Dark Mod Updater / libtdm_update v0.58/Win32"
 
The OS (Win32, Linux etc.) can vary, as well as the version number.
 
The same goes for FM hosting. The signature of the in-game downloader is:


"GET /fms/madmountain.pk4 HTTP/1.1" 200 2422893 "-" "The Dark Mod Agent/1.08 Linux"
If all else fails, you can block everything except TDM programs by user agent:
* Tdm_installer has user agent: <tt>tdm_installer/1.04</tt>  (version after slash changes over time).
* In-game mission downloader has user agent: <tt>The Dark Mod Agent/2.09 Win32</tt>  (everything after slash may vary).
Note that blocking by user agent is rather inconvenient, since TDM team will not be able to test your mirror by inserting direct URL into browser or curl tool.


== See also ==
== See also ==


* [[Installation]]
* [[Installation]]

Latest revision as of 17:17, 26 May 2021

If you want to help to distribute The Dark Mod, this article offers help on what you need to do.

How it works

The game and missions are distributed via HTTP or HTTPS only. Both tdm_installer and the in-game missions downloader do not support FTP, HTTP redirects, meta-refreshes or other methods.

When the game itself is installed or updated, first tdm_installer.ini is downloaded. This file contains a list of mirror servers. Each mirror has weight, which defines probability that it will be used for a request. The weights can be used to distribute traffic across mirrors. See also: Tdm installer and zipsync

The in-game mission downloader downloads available_missions.xml, which contains list of URLs and their weights for every mission. The weights also define probability of each URL being used in a request. The XML file is generated from various XML files in the missions SVN repository. See also: Mission Database

What you need

  • A server with fast connection, preferably with either unlimited traffic, or at least 1000 GB per month.
  • Space: 6 GB for the game mirror, 10 GB for the missions mirror (about 20 GB is enough for both).
  • Synchronization with main server is done via rsync pull at regular intervals (once per day or week).
  • The server must respond to individual file requests via HTTP with a "200 OK" and the content. Things like redirects (302, 307 etc.) as well as intermediate HTML pages (meta refresh), or FTP are not supported.
  • Game mirror must also support multipart byterange requests. Missions mirror does not need to support byteranges at all.

Set up hosting

Here is the generic process of adding a mirror:

  1. Get in contact with TDM team about mirror hosting.
  2. Send IP address of your server to TDM server admin, so that he adds it to the whitelist for rsync access.
  3. Configure your server to perform rsync pull regularly (e.g. every day). Ask admin about rsync address to pull from.
  4. After all data is rsync-ed successfully, send base URL (where it is served at) to TDM release manager for testing.
  5. If everything works fine, then TDM release manager adds your mirror to config file, and players start using it.

Traffic

Please pay attention to additional traffic on your mirror:

  • for one month after the mirror is enabled;
  • for one month after a major release of TDM;

The initial amount of traffic can be pretty high, and there are typically traffic spikes shortly after every major release of the game. Make sure you won't have to pay extra money and won't get your server down.

If amount of traffic is too high for you, contact TDM team. It is possible to reduce weight of your mirror, then players will use it with less probability, and the traffic should decrease proportionally. Here is how exactly it can be done:

  • For game mirror: file tdm_installer.ini stores weights for every mirror. It can only be updated via SSH access to the main TDM server.
  • For mission mirror: file mirrors/mirrorinfo.xml in missions SVN repository supports "weight" attribute near "rootUrl" (default = 1).

Some interesting tendencies about traffic:

  • Tdm_installer always updates the game in differential fashion, so updates are cheap: 20 MB between close versions, hundreds of MB between major releases. However, clean install takes about 4 GB of traffic.
  • In-game mission downloader provides an easy way to download all FMs, which is very likely to increase traffic on mission mirrors. Aside from that, some recent missions are quite heavy by themselves.

Bots

What costs traffic, however, are bots, who are either blindly following links, or traversal directories on your server. Here are a few tips to keep them out and your traffic low:

  • disallow directory traversal on your server. If you can open http://yourserver/thedarkmodfiles/ on your server and see a listing, it is still enabled.
  • use a robots.txt to keep Google, Bing, Yahoo, Baidu etc out of this directory (better disallow all robots)
  • disallow anything that has a known bot or spider signature (wget, Java, libwww, common bot names, or anything that has a referrer of the file itself (PK4 files cannot be referring to themselves).

If all else fails, you can block everything except TDM programs by user agent:

  • Tdm_installer has user agent: tdm_installer/1.04 (version after slash changes over time).
  • In-game mission downloader has user agent: The Dark Mod Agent/2.09 Win32 (everything after slash may vary).

Note that blocking by user agent is rather inconvenient, since TDM team will not be able to test your mirror by inserting direct URL into browser or curl tool.

See also

Retrieved from "https://wiki.thedarkmod.com/index.php?title=Hosting_The_Dark_Mod&oldid=28539"