Cutscene video with FFmpeg

History: ROQ

Historically, ROQ was the only video format supported by Doom 3 and TDM. All the missions created before TDM 2.06 use ROQ format for their briefings. The ROQ is rather old and weird, and it is not supported by most software. That's why it is considered a "legacy" option now.

New cinematics

Common formats

Since 2.06, the most popular video formats can be used in TDM directly. In order to achieve it, specify the path to your video file in the videoMap keyword of the material definition (just as usual).

While FFmpeg supports vast variety of formats, the set of supported formats in TDM is intentionally restricted to a very small one. It is done to minimize the risk of breaking missions over years.

Recall that for playback purpose, every video file consists of three parts:

1. Container allows to extract video and sound streams from one file.
2. Video codec is responsible for decoding compressed video stream to get individual ready-to-show frames.
3. Audio codec is responsible for decoding compressed audio stream to get ready-to-play sound samples.

Container is usually determined by file extension (e.g. avi or mp4). As for video and sound codecs, you can see them e.g if you open your video in VLC and go to menu "Tools"->"Codec information".

Starting with TDM 2.10, TDM supports only one combination:

• mp4 container
• h264 video codec (also known as "MPEG-4 AVC (part 10)")
• aac audio codec (also known as "MPEG AAC Audio")

As of 2021, it is universally supported by video processing software. ROQ videos are also supported for backward compatibility.

IMPORTANT: The video file must have .mp4 extension. Otherwise video will not play properly (probably generating a warning "Opening video file "%s", which is compressed inside PK4" or simply refusing to open video).

Play video with sound

By default, videoMap only displays the video stream from the specified file on the material texture, and sound streams are ignored. Now it is possible to play sound stream directly from the video file. This feature requires some additional setup, because graphics and sounds live in two completely different worlds in the engine.

First of all, you have to add withAudio keyword to videoMap command. It tells cinematic to read sound stream from the video file and push its data into internal queue. For example:

video/sts_briefing_intro
{
    qer_editorimage textures/editor/video
    {
        //get texture from the given video file, decode audio data too
        videoMap withAudio video/my_cool_briefing_video.mp4
    }
}

Second, you have to create a sound sample linked to the cinematic, and ensure that it would start playing at approximately the same time as the video. This sound sample would pull sound data from the cinematic's queue and play it. In case of briefing cutscene, a sound sample is created automatically by GUI scripts, you only need to properly configure the sound shader it uses. In order to link sound shader to cinematics, use fromVideo keyword with the name of the material (instead of the path to the sound file):

video/sts_briefing_video_sound
{
    //take sound sample data from the video file attached to the prescribed material
    fromVideo video/sts_briefing_intro
}

IMPORTANT: You must either do both modifications and ensure that sound really plays, or do neither of them.

Cinematics code uses swresample library to convert the sound stream to 44.1KHz Stereo (on-the-fly).

Cinematics cannot play sound stream from looped videos, so keywords withAudio and loop are mutually exclusive.

Exactly one sound sample linked to the cinematics must be playing while the video is playing.

Automatic duration

It is no longer necessary to explicitly specify the duration of a video file, as it was for the ROQ files. The stock GUI script for single-mission briefing would detect automatically when the video ends. Make sure to set briefing video length to 0 in mainmenu_custom_defs.gui:

//set video length to zero (otherwise it would be forcefully terminated after prescribed time)
#define MM_BRIEFING_VIDEO_LENGTH_1 0   //note: zero here =)
#define MM_BRIEFING_VIDEO_LENGTH_2 0

Under the hood, the "end of video" detection works via CinematicEnd named event. The stock script mainmenu_briefing_video.gui listens to this event in order to proceed to the objectives screen automatically. In more complex scenarios (campaign / SDK briefings / custom GUI code), you have to listen to this event yourself in your GUI script. Here are the exact lines from the stock mainmenu_briefing_video.gui script, serving as a good example:

// #4535: this event fires up only for non-legacy (i.e. FFmpeg-based) videos
onNamedEvent CinematicEnd
{
    // Hide video
    set "HideBriefingVideo::notime" "0";
    resetTime "HideBriefingVideo" 0;
}

By the way, you no longer have to break your videos into one-minute chunks, with new cinematics videos can have arbitrary length.

Examples

Single mission

1. In mainmenu_custom_defs.gui of your mission:

...
/**
 * Defines the material shader that points to the ROQ video file. Note that the
 * maximum length of an ROQ file is 1 minute, so break your briefing up (just video)
 * into parts of maximum length 1 minute.
 */
#define MM_BRIEFING_VIDEO_MATERIAL_1 "video/sts_briefing_intro" // !! put name of material here !!
#define MM_BRIEFING_VIDEO_MATERIAL_2 ""
...

...
/**
 * This defines the playback time of your video. After this amount of time, the
 * mainmenu will switch to the briefing page.
 *
 * If you have your video in several parts, then specify lengths below. Be sure
 * to specify lengths by adding on the length of the previous section.
 * eg: You have 3 parts, first is 1600 milliseconds, second 1500 and third 500,
 * then LENGTH_1 will be 1600, LENGTH_2 3100, and LENGTH_3 600. Failing to
 * specify length correctly will result in skipping parts of video.
 *
 * Important: This is specified in milliseconds!
 */
#define MM_BRIEFING_VIDEO_LENGTH_1 0        // !! set to zero !!
#define MM_BRIEFING_VIDEO_LENGTH_2 0
...

...
/**
 * Define the soundshader to be played during the briefing video.
 *
 * Important: The syntax is "music YOURSHADER;", the "music" part is important
 * and should not be deleted, neither the trailing semicolon. 
 * This is the actual GUI command issued to the C++ code.
 */
#define MM_BRIEFING_VIDEO_SOUND_CMD "music video/sts_briefing_video_sound;"  // !! put name of sound shader here !!
...

2. In material definition, e.g. in my_video_materials.mtr:

video/sts_briefing_intro
{
    qer_editorimage textures/editor/video
    {
        videoMap withAudio video/my_cool_briefing_video.mp4    // !! specify path to video file, do not forget "withAudio" !!
    }
}

3. In sound shader definition, e.g. in my_video_sounds.sndshd:

video/sts_briefing_video_sound
{
    fromVideo video/sts_briefing_intro                      // !! put fromVideo keyword with the name of material above !!
}

That's all.

You may name sound shader, material and video file as you wish, you may define material and sound shader wherever you wish. Just make sure that all the names/filenames are set appropriately.

Troubleshooting

All related cvars start with the common prefix "r_cinematic_".

The most useful one is r_cinematic_log: it enables generic logging to a dedicated file log_cinematics.txt. If you implement custom GUI scripts, this log can be helpful to see what happens.

For instance, in order to check that cinematic stops properly in your custom GUI, check that:

1. new messages do not appear after cinematic stops
2. the last section of log is "====== Close ======"

Quick test

Sometimes you want to quickly check that your video plays well in TDM engine, but don't want to waste time setting up all the stuff.

Then you can use console command testVideo for a quick test. This command works only in-game: it cannot be used from the main menu. For faster loading, I suggest starting some small test map or Closemouthed Shadows FM.

The command can be used in two ways:

1. "testVideo video/my_cool_briefing_video.mp4": opens specified video file without sound. It also tries to open equally-named .wav file it is exists.
2. "testVideo video/my_cool_briefing_video.mp4 withAudio": opens specified video file with sound (i.e. in withAudio mode).

After executing the command, the video should start playing in a rectangular area in the bottom on the screen. Also some basic information about the video file is printed to console (it is a good idea to check it for errors/warnings). After the video ends, the rectangular region disappears.

References

Issue 4159: Investigate using videos of common formats

Issue 4534: Playing sound stream from video file

Issue 4535: Showing video for its full duration

Issue 4845: Restrict set of features in FFmpeg cinematics

Issue 4847: Support FFmpeg videos in testVideo command