Cutscene video with FFmpeg

History: ROQ

Historically, ROQ was the only video format supported by Doom 3 and TDM. All the missions created before TDM 2.06 use ROQ format for their briefings. The ROQ is rather old and weird, and it is not supported by most software. That's why it is considered a "legacy" option now.

New cinematics

Common formats

Since 2.06, most of the video formats can be used in TDM directly. In order to achieve it, specify the path to your video file in the videoMap keyword of the material definition (just as usual).

You can encode the video with any codec supported by the libavcodec/FFmpeg (see the approximate list). But please be wise in codec choice: better use common and widespread codecs with default options. Codecs XVID and x264 are good examples here.

Although FFmpeg supports many different containers, TDM can only play a limited set of them from inside a pk4 file. If you use wrong type of container, then TDM would generate a warning "Opening video file "%s", which is compressed inside PK4", and the video will not play properly.

IMPORTANT: Use MPEG-4 or AVI container! So the video file must have one of the following extensions: mp4 or m4v or avi

Play video with sound

By default, videoMap only displays the video stream from the specified file on the material texture, and sound streams are ignored. Now it is possible to play sound stream directly from the video file. This feature requires some additional setup, because graphics and sounds live in two completely different worlds in the engine.

First of all, you have to add withAudio keyword to videoMap command. It tells cinematic to read sound stream from the video file and push its data into internal queue. For example:

video/sts_briefing_intro
{
    qer_editorimage textures/editor/video
    {
        //get texture from the given video file, decode audio data too
        videoMap withAudio video/my_cool_briefing_video.mp4
    }
}

Second, you have to create a sound sample linked to the cinematic, and ensure that it would start playing at approximately the same time as the video. This sound sample would pull sound data from the cinematic's queue and play it. In case of briefing cutscene, a sound sample is created automatically by GUI scripts, you only need to properly configure the sound shader it uses. In order to link sound shader to cinematics, use fromVideo keyword with the name of the material (instead of the path to the sound file):

video/sts_briefing_video_sound
{
    //take sound sample data from the video file attached to the prescribed material
    fromVideo video/sts_briefing_intro
}

IMPORTANT: You must either do both modifications and ensure that sound really plays, or do neither of them.

Cinematics code uses swresample library to convert the sound stream to 44.1KHz Stereo (on-the-fly).

Cinematics cannot play sound stream from looped videos, so keywords withAudio and loop are mutually exclusive.

Exactly one sound sample linked to the cinematics must be playing while the video is playing.

Automatic duration

It is no longer necessary to explicitly specify the duration of a video file, as it was for the ROQ files. The standard GUI script for single-mission briefing would detect automatically when the video ends. Make sure to set briefing video length to 0 in mainmenu_custom_defs.gui:

//set video length to zero (otherwise it would be forcefully terminated after prescribed time)
#define MM_BRIEFING_VIDEO_LENGTH_1 0   //note: zero here =)
#define MM_BRIEFING_VIDEO_LENGTH_2 0

In a more complex scenarios (campaign / SDK briefings / custom GUI code), listen to CinematicEnd named event in the GUI script to detect when cinematic ends. For instance, here is this event handler in standard script mainmenu_briefing_video.gui:

// #4535: this event fires up only for non-legacy (i.e. FFmpeg-based) videos
onNamedEvent CinematicEnd
{
    // Hide video
    set "HideBriefingVideo::notime" "0";
    resetTime "HideBriefingVideo" 0;
}

By the way, you no longer have to break your videos into one-minute chunks, with new cinematics videos can have arbitrary length.

Examples

single mission

1. In mainmenu_custom_defs.gui of your mission:

...
/**
 * Defines the material shader that points to the ROQ video file. Note that the
 * maximum length of an ROQ file is 1 minute, so break your briefing up (just video)
 * into parts of maximum length 1 minute.
 */
#define MM_BRIEFING_VIDEO_MATERIAL_1 "video/sts_briefing_intro" // !! put name of material here !!
#define MM_BRIEFING_VIDEO_MATERIAL_2 ""
...

...
/**
 * This defines the playback time of your video. After this amount of time, the
 * mainmenu will switch to the briefing page.
 *
 * If you have your video in several parts, then specify lengths below. Be sure
 * to specify lengths by adding on the length of the previous section.
 * eg: You have 3 parts, first is 1600 milliseconds, second 1500 and third 500,
 * then LENGTH_1 will be 1600, LENGTH_2 3100, and LENGTH_3 600. Failing to
 * specify length correctly will result in skipping parts of video.
 *
 * Important: This is specified in milliseconds!
 */
#define MM_BRIEFING_VIDEO_LENGTH_1 0        // !! set to zero !!
#define MM_BRIEFING_VIDEO_LENGTH_2 0
...

...
/**
 * Define the soundshader to be played during the briefing video.
 *
 * Important: The syntax is "music YOURSHADER;", the "music" part is important
 * and should not be deleted, neither the trailing semicolon. 
 * This is the actual GUI command issued to the C++ code.
 */
#define MM_BRIEFING_VIDEO_SOUND_CMD "music video/sts_briefing_video_sound;"  // !! put name of sound shader here !!
...

2. In material definition, e.g. in my_video_materials.mtr:

video/sts_briefing_intro
{
    qer_editorimage textures/editor/video
    {
        videoMap withAudio video/my_cool_briefing_video.mp4    // !! specify path to video file, do not forget "withAudio" !!
    }
}

3. In sound shader definition, e.g. in my_video_sounds.sndshd:

video/sts_briefing_video_sound
{
    fromVideo video/sts_briefing_intro                      // !! put fromVideo keyword with the name of material above !!
}

That's all.

You may name sound shader, material and video file as you wish, you may define material and sound shader wherever you wish. Just make sure that all the names/filenames are set appropriately.

References

Forum thread "Why ROQ" about FFmpeg cinematics.

Issue 4159: Investigate using videos of common formats

Issue 4534: Playing sound stream from video file

Issue 4535: Showing video for its full duration

All related cvars start with the common prefix "r_cinematic_".