Full-Screen Video Cutscenes

By Geep, 2020-2022

TEMPORARY CAUTION (March, 2021)

If considering creation of a video cutscene with lipsync or other close synchronization, be sure to consult the bugtracker report "0005575: When Uncapped FPS is Off, Video Cutscene Loses Audio Sync", which includes a limited workaround, also discussed below as Technique 3.

Introduction

Suppose you have created a video for a TDM in-game cutscene, using video capture and AV editing software. (Video Cutscene Creation has thoughts about the why and how of this.) Your video is compliant with video codecs supported in TDM 2.06+ and recommended for new FMs (e.g., MP4 or AVI). Likewise, TDM-supported sound formats. You want to play this cutscene back at a particular point in the game. Here are two methods, both of which necessarily deploy somewhat different software architecture (e.g., GUIs) than that of briefing videos.

GUI Message Overlay Method

This method repurposes the standard message entity that overlays the full screen (e.g., as the TDM main menu) to display a video instead. It is the generally-preferred method, because it:

• is the easiest to implement;
• avoids one additional pixel resampling that the other method uses, so is likely to be more performant and the highest possible image quality;
• while handling the ESC key poorly, still does better than the other method;
• scales easily to multiple disjoint cutscenes;
• does not require the construction of a separate room (but there are advantages to doing so anyway, as noted under the next method).

Movie Theatre Method

This requires the construction of a separate, isolated movie-theatre-like room (e.g, in the void). You teleport the player there, play the cutscene with the player's view locked to a camera looking at the screen, then teleport back. The isolation has these advantages:

• It provides audio and lighting that is separate from the rest of the map.
• It avoids any interactions with AI while the player has no sight control.

Other pluses for this method are:

• If you want a black frame around your cutscene (and to not have to burn it in with your video editing software), that's easy;
• It can be extended to hybrid cutscenes with live action in front of a video backdrop.

Information Common to Examples of Both Methods

Video and Sound Files

In our examples here, we assume an MP4 and a separate OGG audio file, specifically:

• an MPEG 4 video, 16:9 aspect ratio, called <fm>/video/cutscene_video.mp4
• a separate OGG audio file, stereo, called <fm>/sound/video/cutscene_video.ogg
• both have a length of 88 seconds.

Can an audio track that is embedded in the video be used directly? Probably not at this time. (Using embedded audio is possible with briefing videos, because special programming was done in TDM 2.06 to allow it.)

Sound Shader

Create a sound shader for the OGG, for instance <fm>/sound/cutscene_video.sndshd, with content:

video/cutscene_video
{
   sound/video/cutscene_video.ogg
}

Video Shader, GUI, and Script Setup

Create the start of a material shader for the videomap, such as <fm>/materials/cutscene_video.mtr:

video/cutscene_video { // more to come }

Similarly, start a custom GUI, here <fm>/guis/cutscene_video.gui:

windowDef Desktop { // more to come }

Also, add a function to <fm>/maps/<fm>.script to trigger the playback, like:

void dovideo() { // more to come }

GUI Message Overlay Method

Somewhere in your map, create an atdm:guis_message entity, with spawnargs values like:

name	atdm_guis_message_1
show	87                          duration of the video in seconds, minus 1
gui	guis/cutscene_video.gui     overrides the standard gui

You can delete any non-default "text" or "lines" spawnargs as irrelevant.

"Show" is the duration of the video in (presumably decimal) seconds, minus 1. This is because the standard code (in scriptobject tdm_gui_message) adds 1 second for a gui-based fade out, but our gui doesn't use a fade out. (And if you set hidden spawnarg "fade_out_time" to 0, it reinterprets it as 1)

Building out the Material Shader for the Video Map

Fill in cutscene_video.mtr:

video/cutscene_video
{
   qer_editorimage textures/editor/video
   {
      videoMap video/cutscene_video.mp4 // with separate sound file
   }
} 

3 Techniques to Build out the GUI and Trigger Video and Audio

Technique 1 is quickest to implement, but can cause loss of lipsync, depending on which value the gamer sets for "Settings/Video/Advanced/Uncap FPS". So can Technique 2, which might be considered a waystation to Technique 3.

Technique 1: GUI with Localsound

Jan 2022 Note: GUI code example updated to correct punctuation errors that quietly caused problems. Thanks, stgatilov

Fill in cutscene_video.gui with:

windowDef Desktop
{
  rect 0 ,0 ,640 ,480 // Standard element to define nominal grid, e.g., for placing and sizing buttons
  backcolor 1,1,1,0
  matcolor 0, 0, 0, 1
  nocursor 1

  background "video/cutscene_video" // mp4 shader defined in materials/cutscene_video.mtr file.

  onTime 0
  {
     resetCinematics; // reset Video to start
     set "Desktop::matcolor" "1,1,1,1"; // show video
     localsound "video/cutscene_video"; // shader as defined in sound/cutscene_video.sndshd
  }
}

The "nocursor 1" line suppresses the appearance - dead in the center of the screen - of the standard large clock-hand cursor seen in the TDM main menu.

Technique 2: Trigger the Audio Separately

Use the same GUI as above, except remove the "localsound" line.

Then, handle the audio. Someplace in your map, following the tdm_voice approach to a narrator's or player's voice, create two objects:

• an atdm:trigger_voice entity (say, atdm_trigger_voice_1)
• an atdm:voice entity (e.g., atdm_voice_1)

Link from the trigger_voice to the voice entity, either by

• adding spawnarg "target0 atdm_voice_1" to atdm_trigger_voice_1; or
• in Dark Radiant, select atdm_trigger_voice_1, select also voice_1, then Control-K.

In atdm_voice_1, leave this spawnarg at its default of:

s_shader   silence

In atdm_trigger_voice_1, add a spawnarg "snd_say", then use the "Choose sound" button to find your sound shader, e.g.:

snd_say   video/cutscene_video

Note also the spawnarg "as_player", which can left at 1 (the default) to represent the sound as the player's voice, or changed to 0 as a narrator. Either work for our purposes. The difference is presumably:

• sound from the player (but not the narrator) could alert any AI; and
• TDM volume controls distinguish the two, that is, in the TDM Audio menu, "Player Voice Volume" slider versus "Narrator Volume" slider. Similarly for cvars tdm_voice_player_volume versus (for narrator) tdm_voice_from_off_volume.

Building Out the Script Function to Trigger the Playback

Fill in your <fm>.script function:

void dovideo(){
   sys.trigger($atdm_gui_message_1);   //start the cutscene. All you need for Technique 1.
   sys.trigger($adtm_trigger_voice_1); //start the audio, for Technique 2.
}

The dovideo function can be called by any triggering device you prefer. The value of using a script function here (as opposed to a triggering relay) becomes more apparent with Technique 3 or with player isolation described below.

Technique 3 - Preserving Lipsync

Here, you first chop of your audio into segments, each ideally under a dozen seconds. Chops are made at dialog pauses, or generally any place where you want to (in-effect) force resync. Record the duration of each segment. Then each segment separately gets the audio treatment of Technique 2. While you will be defining multiple sound shaders, that can all happen within a single .sndshd file. After adding multiple triggers to your map, call them in a script such as:

void dovideo(){
   sys.trigger($atdm_gui_message_1);   //start the cutscene.
   sys.trigger($adtm_trigger_voice_1); //start audio segment 1
   sys.wait(12.3); // duration of segment 1
   sys.trigger($adtm_trigger_voice_2); //start audio segment 2
   sys.wait(8.4); // duration of segment 2
   sys.trigger($adtm_trigger_voice_3); //start audio segment 3
   sys.wait(4.7); // last wait may not be necessary, depending on what else your script does
}

Tip if planning to use this technique: Make sure when recording your choppable audio track that it contains no music or continuous ambient sounds, because that would become ragged by chopping and re-triggering. Instead, provide any such sound separately, through an ambient or regular speaker in the player isolation room, discussed next.

Optional Player Isolation

It may be a good idea to create an isolated room for the player to reside in during the cutscene. The advantages of this are largely the same as for separate room used in the Movie Theatre method:

• It provides audio that is separate from the rest of the map.
• It avoids any interactions with AI while the player's in the box.

While you can use the same room design as Movie Theatre, the dimensions are more flexible here, so perhaps your existing blue room in the void could be outfitted for the purpose.

While seeing only the cutscene, the player can still move around, so maybe cover the floor with moss or a carpet to make that quiet. Or pin the player, e.g., in player clip. Also, place some named object (e.g., a rug "start_cutscene_spot") on the floor, to use as a convenient teleport target.

Then alter the doVideo function to teleport the player there, play the cutscene, then teleport to where the player needs to be afterwards:

void dovideo(){
   $player1.setOrigin($start_cutscene_spot.getOrigin()); // traditional teleport method
   sys.trigger($atdm_gui_message_1);   //start the cutscene
   sys.trigger($adtm_trigger_voice_1); //and audio
   sys.wait(88);                       //length of video... then teleport away
   $player1.setOrigin($after_cutscene_spot.getOrigin());
}

Movie Theatre Method

Room Setup

Begin by creating the void-sealing room as a cube with an internal dimension of 512 in all 3 directions. Paint the interior dark. (For debugging, it's useful to have a perceptable texture like dark marble, plus a dim private ambient light source.)

Inside this box, near and parallel to one wall, place an internal wall (from a brush converted to func_static) to serve as a screen. Name it "video_wall". Given that we have a 16:9 video, make the size of this wall 512 wide x 288 high. Move it up to be centered vertically inside the room. Select the front face as the screen surface, then apply and fit an entityGUI texture to it. Specifically:

• Select one surface as the screen: Control-Shift LMB.
• Using Texture editor/Media, click on the background, search for "entityGUI". When found (under "common/"), select the text, then right-click and "Apply to Selection".
• Using the Surface editor, hit the "Fit" button.

You will be linking the screen to a custom GUI further below.

Next, create a func_cameraview entity (named here "func_cameraview_1"), and move it to be centered on the screen, at a perpendicular distance of about 184 units. 90-degree-rotate the camera so its arrow faces the screen. Give it the spawnargs:

trigger   1
target    video_wall

You may have to rotate the camera further during debugging the setup. And the distance can be adjusted to taste, knowing that you are trading off edge clipping on various monitors versus black bars. A distance of 200 will provide a black frame (your darkened room walls) around the entire window.

Finally, as discussed with the other method, quiet the player's footfalls and place a teleport target (e.g., "start_cutscene_spot") on the floor under camera.

Build out the VideoMap Material Shader

Updated March, 2021

Fill in cutscene_video.mtr:

video/cutscene_video
{
  qer_editorimage textures/editor/video
    noshadows
    guiSurf entity
    discrete
    translucent
    sort "-2"
  {
    videoMap video/cutscene_video.mp4 // Any embedded sound ignored; TDM playback method uses separate sound file.
      remoteRenderMap 512 266 // matches ratio of projection surface
      red       1
      green     1
      blue      1
      scale     1,1
      translate 1,1
  }
} 

For more about possible color channel parameterization of this, and color/resolution effects, see Obsttortes's video in How to get security camera observation screens working.

Build out the GUI

Updated March 2021

Fill in your custom cutscene_video.gui:

windowDef Desktop
{
  rect 0 ,0 ,640 ,480
  background black
  noevents false // required to make event handlers like onTrigger work

  onTrigger
  {
    // commands within event handlers must end with ";", unlike elsewhere
    set "background" "video/cutscene_video"; // "set" required here to change initial value
    resetCinematics; // reset video to start
    localsound "video/cutscene_video"; // This works too: localsound "sound/video/cutscene_video.ogg";
  }
}

The "background black" command here is optional; it generates a console warning but does show the screen as black before the movie plays. Or delete the line to have the pre-movie screen invisible.

Then link your screen entity "video_wall" to this, by adding the spawnarg:

gui    guis/cutscene_video.gui

Trigger the Playback

Fill in you <fm>.script function like:

void dovideo(){
   $player1.setOrigin($start_cutscene_spot.getOrigin()); // traditional teleport method
   entity cam = $func_cameraview_1;
   cam.activate($player1);   //start the camera
   sys.trigger($video_wall); //start the cutscene
   sys.wait(88);             //length of video, seconds
   cam.activate($player1);   //stop the camera
   $player1.setOrigin($after_cutscene_spot.getOrigin());
}

This can be called by any triggering device you prefer.

Discussion

Interrupting or Skipping a CutScene

Support for this is inadequate with the stated methods. That may not be a problem, particularly with short cutscenes. However, it does rule out during-cutscene volume adjustments by the gamer with the TDM Audio sliders. Perhaps someone's future work will find improvements.

With the GUI Message Overlay Method. If you hit ESC, you will return to the main TDM menu. You can subsequently "Resume Mission", and the video will resume where interrupted, but there will be no audio. The GUI shown above does not support skipping with LMB. (Early experiments adding a doAction() function seemed to show no response to LMB, whether or not the TDM clock-hand cursor was visible. Likewise, doEsc() was ineffective. But this was before awareness of "noevent false", so your mileage may vary.)

With the Movie Theatre Method. If you hit ESC once, you will break out of the camera view, so end up seeing the movie theatre, with the sound still playing. Hitting ESC again kills the audio and brings you do the main menu. "Resume" then returns you to the theatre (but NOT the camera view), and with no sound.

Choosing in Advance to Skip a CutScene

An alternative to these shortcomings is to offer some sort of switch in-game, so the player can indicate that they'd rather not see the upcoming cutscene, presumably because they've already seen it. For example, in "Away 0: Stolen Heart", there's a wall panel with glowing text that says "Skip Cutscene? Frob Me". If the player clicks that, the text disappears and a script variable is set. When, shortly later, the player arrives at where the cutscene would be triggered, a replacement slide is shown instead, to maintain story continuity. More about that in A Full-Screen Slide in Mid-Game.

Multiple Non-Sequential Cutscenes in an FM

With the GUI Message Overlay Method. Give each cutscene it's own atdm_gui_message entity and dovideo function. But they should be able to share an isolation room.

With the Movie Theatre Method. Perhaps some way can be found that doesn't require a separate movie theatre for every cutscene.

Alternative Sound Treatments

Above, we have demonstrated two audio techniques:

• localsound within the GUI;
• using a narrator's or player's voice with tdm_voice.

These have the skipping drawback noted, but are otherwise easy to do and provide the expected stereo separation.

Other possible ways to go might be:

• ambient sound through the location system.
• a separate speaker in the isolated room. Generally, this is not a great choice. Besides reducing the stereo to mono, the sound source will seem to drift around if the player moves. And the player can move in both Overlay and Theatre Methods; even if encased in a playerclip cage around $start_cutscene_spot, rotation still seems possible. But see the next section for a situation where it might be useful.

A typical speaker setup (often placed behind the Movie Theatre screen) has spawnargs:

name             speaker_1
s_maxdistance    250   Adjust to cover room, but not impinge on rest of map
s_waitfortrigger 1
s_shader         video/cutscene_video  usually found with the Entity Inspector's "Select Shader" button

Then call it in the dovideo():

  ...
  sys.trigger($video_wall);	//start the cutscene
  sys.trigger($speaker_1);	//& its audio
  ...

Cutscenes using Live Action with a Video Background

Speculatively, the Movie Theatre method should be combinable with traditional TDM cutscene techniques. The hybrid would have AI and a "movie set" in the foreground/middleground, and a video in the background. This approach mimics one widely used in early cinema (pre-green-screen) with a projected backdrop. Examples:

• a balcony overlooking an expansive vista with flocks of birds passing by;
• doors and windows open to a distance storm arising;
• rolling ship's deck in a roaring sea;
• passengers in a stagecoach or steam-train carriage, with the countryside wisking by.

In this hybrid approach, the camera (and player's) view would always remain centered on and perpendicular to the background screen, as per the Movie Theatre approach. This would avoid any skew or keystoning of the video that would reveal its 2D nature. Other considerations:

• Lighting would be needed for the AI and movie set.
• The player character's position is the "ear or microphone" that picks up the AI's talk, as usual with traditional cutscene techniques;
• Audio associated with the video, if used as all, would likely be quieter. This is a case where using a speaker for the video's soundtrack might be helpful. You could move the speaker around in the map to affect the relative location and volume.
• If the scene is not of fixed duration, there are additional complications. The video may have to be long enough to cover the maximum duration, or have to be looped. In the latter case, the video will need some way to mask the loop ends. For instance, moving scenery viewed from a vehicle could start and end with blurs of closeup trees or a tunnel. For potential looping methodologies to explore, see Playing ROQ Video Files on repeating a video using the GUI, or Cutscene video with FFmpeg about "videomap loop...".

Non-Cutscene Live Action with Video Background

During normal gameplay - with a normal player view, not in a cutscene - could a "video background" be used, with in-game foreground set and characters? Clearly, there are parallax issues with a 2D video. For this to be plausible, the video content and/or viewing locations (i.e., player's movements) may need to be constrained. Consider a video of a distance horizon with, say, a flight of crows. If projected on a wall in front of a skybox surface, and viewed through foreground windows, it might well work.

See Also

Video Cutscene Creation gives pluses and minuses of video versus live-action cutscenes, and some useful tips for scene/set preparations and camera control during capture.

A Full-Screen Slide in Mid-Game shows how to fashion a replacement slide for use when the player wants to skip a video cutscene.

Cutscene video with FFmpeg for video codecs supported in TDM 2.06+ and recommended for new FMs, such as MP4 or AVI. Also ideas on video looping.

Sound File Formats supported in TDM.

Video Briefing showcases related issues.

Cutscenes for traditional TDM live-action cutscene techniques.

Security Camera for live-action camera feed to a video surface

Forum discussions:

Dioramas, with scaled-down ghost-like live action effects.

Glass and Magic Mirrors

Magic Screens or Prey-like portals