Cutscene video with FFmpeg

History: ROQ

Historically, ROQ was the only video format supported by Doom 3 and TDM. All the missions created before TDM 2.06 use ROQ format for their briefings. The ROQ is rather old and weird, and it is not supported by most software. That's why it is considered a "legacy" option now.

Common formats

Since 2.06, most of the video formats can be used in TDM directly. In order to achieve it, specify the path to your video file in the videoMap keyword of the material definition (just as usual).

You can encode the video with any codec supported by the libavcodec/FFmpeg (see the approximate list). But please be wise in codec choice: better use common and widespread codecs with default options. Codecs XVID and x264 are good examples here.

Although FFmpeg supports many different containers, TDM can only play a limited set of them from inside a pk4 file. If you use wrong type of container, then TDM would generate a warning "Opening video file "%s", which is compressed inside PK4", and the video will not play properly.

IMPORTANT: Use MPEG-4 or AVI container! So the video file must have one of the following extensions: mp4 or m4v or avi

Play video with sound

By default, videoMap only displays the video stream from the specified file on the material texture, and sound streams are ignored. Now it is possible to play sound stream directly from the video file. This feature requires some additional setup, because graphics and sounds live in two completely different worlds in the engine.

First of all, you have to add withAudio keyword to videoMap command. It tells cinematic to read sound stream from the video file and push its data into internal queue. For example:

video/sts_briefing_intro
{
    qer_editorimage textures/editor/video
    {
        //get texture from the given video file, decode audio data too
        videoMap withAudio video/my_cool_briefing_video.mp4
    }
}

Second, you have to create a sound sample linked to the cinematic, and ensure that it would start playing at approximately the same time as the video. This sound sample would pull sound data from the cinematic's queue and play it. In case of briefing cutscene, a sound sample is created automatically by GUI scripts, you only need to properly configure the sound shader it uses. In order to link sound shader to cinematics, use fromVideo keyword with the name of the material (instead of the path to the sound file):

video/sts_briefing_video_sound
{
    //take sound sample data from the video file attached to the prescribed material
    fromVideo video/sts_briefing_intro
}

IMPORTANT: You must either do both modifications and ensure that sound really plays, or do neither of them.

Cinematics code uses swresample library to convert the sound stream to 44.1KHz Stereo (on-the-fly).

Cinematics cannot play sound stream from looped videos, so keywords withAudio and loop are mutually exclusive.

Exactly one sound sample linked to the cinematics must be playing while the video is playing.

Automatic duration

TODO

References

Forum thread "Why ROQ" about FFmpeg cinematics.

Issue 4159: Investigate using videos of common formats

Issue 4534: Playing sound stream from video file

Issue 4535: Showing video for its full duration

All related cvars start with the common prefix "r_cinematic_".