Full-Screen Video Cutscenes
By Geep, 2020
TEMPORARY CAUTION (March, 2021)
If considering creation of a video cutscene with lipsync or other close synchronization, be sure to consult the bugtracker report "0005575: When Uncapped FPS is Off, Video Cutscene Loses Audio Sync", which includes a limited workaround.
Introduction
Suppose you have created a video for a TDM in-game cutscene, using video capture and AV editing software. (Video Cutscene Creation has thoughts about the why and how of this.) Your video is compliant with video codecs supported in TDM 2.06+ and recommended for new FMs (e.g., MP4 or AVI). Likewise, TDM-supported sound formats. You want to play this cutscene back at a particular point in the game. Here are two methods, both of which necessarily deploy somewhat different software architecture (e.g., GUIs) than that of briefing videos.
GUI Message Overlay Method
This method repurposes the standard message entity that overlays the full screen (e.g., as the TDM main menu) to display a video instead. It is the generally-preferred method, because it:
- is the easiest to implement;
- avoids one additional pixel resampling that the other method uses, so is likely to be more performant and the highest possible image quality;
- while handling the ESC key poorly, still does better than the other method;
- scales easily to multiple disjoint cutscenes;
- does not require the construction of a separate room (but there are advantages to doing so anyway, as noted under the next method).
Movie Theatre Method
This requires the construction of a separate, isolated movie-theatre-like room (e.g, in the void). You teleport the player there, play the cutscene with the player's view locked to a camera looking at the screen, then teleport back. The isolation has these advantages:
- It provides audio and lighting that is separate from the rest of the map.
- It avoids any interactions with AI while the player has no sight control.
Other pluses for this method are:
- It supports "localsound", the quickest-to-implement audio method;
- If you want a black frame around your cutscene (and to not have to burn it in with your video editing software), that's easy;
- It can be extended to hybrid cutscenes with live action in front of a video backdrop.
Information Common to Examples of Both Methods
Video and Sound Files
In our examples here, we assume an MP4 and a separate OGG audio file, specifically:
- an MPEG 4 video, 16:9 aspect ratio, called <fm>/video/cutscene_video.mp4
- a separate OGG audio file, stereo, called <fm>/sound/video/cutscene_video.ogg
- both have a length of 88 seconds.
Can an audio track that is embedded in the video be used directly? Probably not at this time. (Using embedded audio is possible with briefing videos, because special programming was done in TDM 2.06 to allow it.)
Sound Shader
Create a sound shader for the OGG, for instance <fm>/sound/cutscene_video.sndshd, with content:
video/cutscene_video
{
sound/video/cutscene_video.ogg
}
Video Shader, GUI, and Script Setup
Create the start of a material shader for the videomap, such as <fm>/materials/cutscene_video.mtr:
video/cutscene_video { // more to come }
Similarly, start a custom GUI, here <fm>/guis/cutscene_video.gui:
windowDef Desktop { // more to come }
Also, add a function to <fm>/maps/<fm>.script to trigger the playback, like:
void dovideo() { // more to come }
GUI Message Overlay Method
Somewhere in your map, create an atdm:guis_message entity, with spawnargs values like:
name atdm_guis_message_1 show 87 duration of the video in seconds, minus 1 gui guis/cutscene_video.gui overrides the standard gui
You can delete any non-default "text" or "lines" spawnargs as irrelevant.
"Show" is the duration of the video in (presumably decimal) seconds, minus 1. This is because the standard code (in scriptobject tdm_gui_message) adds 1 second for a gui-based fade out, but our gui doesn't use a fade out. (And if you set hidden spawnarg "fade_out_time" to 0, it reinterprets it as 1)
Building out the Material Shader for the Video Map
Fill in cutscene_video.mtr:
video/cutscene_video
{
qer_editorimage textures/editor/video
{
videoMap video/cutscene_video.mp4 // with separate sound file
}
}
Building out the GUI
Fill in cutscene_video.gui with:
windowDef Desktop
{
rect 0 ,0 ,640 ,480 // Standard element to define nominal grid, e.g., for placing and sizing buttons
backcolor 1,1,1,0
matcolor 0, 0, 0, 1
background "video/cutscene_video"; // mp4 shader defined in materials/cutscene_video.mtr file.
onTime 0
{
nocursor 1 // This work ONLY in this location, not in Desktop or after reset/matcolor here
resetCinematics; // reset Video to start
Desktop::matcolor 1,1,1,1; // show video
// Doesn't work: localsound "video/cutscene_video"; // shader as defined in sound/cutscene_video.sndshd
}
}
Be aware that, while this specific code suppresses the appearance of the standard large clock-hand cursor seen in the TDM main menu, small code changes can cause it to re-appear, dead in the center of the screen.
Handling the Audio
Someplace in your map, following the tdm_voice approach to a narrator's or player's voice, create two objects:
- an atdm:trigger_voice entity (say, atdm_trigger_voice_1)
- an atdm:voice entity (e.g., atdm_voice_1)
Link from the trigger_voice to the voice entity, either by
- adding spawnarg "target0 atdm_voice_1" to atdm_trigger_voice_1; or
- in Dark Radiant, select atdm_trigger_voice_1, select also voice_1, then Control-K.
In atdm_voice_1, leave this spawnarg at its default of:
s_shader silence
In atdm_trigger_voice_1, add a spawnarg "snd_say", then use the "Choose sound" button to find your sound shader, e.g.:
snd_say video/cutscene_video
Note also the spawnarg "as_player", which can left at 1 (the default) to represent the sound as the player's voice, or changed to 0 as a narrator. Either work for our purposes. The difference is presumably:
- sound from the player (but not the narrator) could alert any AI; and
- TDM volume controls distinguish the two, that is, in the TDM Audio menu, "Player Voice Volume" slider versus "Narrator Volume" slider. Similarly for cvars tdm_voice_player_volume versus (for narrator) tdm_voice_from_off_volume.
Building Out the Script Function to Trigger the Playback
Fill in your <fm>.script function:
void dovideo(){
sys.trigger($atdm_gui_message_1); //start the cutscene
sys.trigger($adtm_trigger_voice_1); //and audio
}
This function can be called by any triggering device you prefer. The value of a script function (as opposed to a triggering relay) is more apparent in what follows.
Optional Player Isolation
It may be a good idea to create an isolated room for the player to reside in during the cutscene. The advantages of this are largely the same as for separate room used in the Movie Theatre method:
- It provides audio that is separate from the rest of the map.
- It avoids any interactions with AI while the player's in the box.
While you can use the same room design as Movie Theatre, the dimensions are more flexible here, so perhaps your existing blue room in the void could be outfitted for the purpose.
While seeing only the cutscene, the player can still move around, so maybe cover the floor with moss or a carpet to make that quiet. Or pin the player, e.g., in player clip. Also, place some named object (e.g., a rug "start_cutscene_spot") on the floor, to use as a convenient teleport target.
Then alter the doVideo function to teleport the player there, play the cutscene, then teleport to where the player needs to be afterwards:
void dovideo(){
$player1.setOrigin($start_cutscene_spot.getOrigin()); // traditional teleport method
sys.trigger($atdm_gui_message_1); //start the cutscene
sys.trigger($adtm_trigger_voice_1); //and audio
sys.wait(88); //length of video... then teleport away
$player1.setOrigin($after_cutscene_spot.getOrigin());
}
Movie Theatre Method
Room Setup
Begin by creating the void-sealing room as a cube with an internal dimension of 512 in all 3 directions. Paint the interior dark. (For debugging, it's useful to have a perceptable texture like dark marble, plus a dim private ambient light source.)
Inside this box, near and parallel to one wall, place an internal wall (from a brush converted to func_static) to serve as a screen. Name it "video_wall". Given that we have a 16:9 video, make the size of this wall 512 wide x 288 high. Move it up to be centered vertically inside the room. Select the front face as the screen surface, then apply and fit an entityGUI texture to it. Specifically:
- Select one surface as the screen: Control-Shift LMB.
- Using Texture editor/Media, click on the background, search for "entityGUI". When found (under "common/"), select the text, then right-click and "Apply to Selection".
- Using the Surface editor, hit the "Fit" button.
You will be linking the screen to a custom GUI further below.
Next, create a func_cameraview entity (named here "func_cameraview_1"), and move it to be centered on the screen, at a perpendicular distance of about 184 units. 90-degree-rotate the camera so its arrow faces the screen. Give it the spawnargs:
trigger 1 target video_wall
You may have to rotate the camera further during debugging the setup. And the distance can be adjusted to taste, knowing that you are trading off edge clipping on various monitors versus black bars. A distance of 200 will provide a black frame (your darkened room walls) around the entire window.
Finally, as discussed with the other method, quiet the player's footfalls and place a teleport target (e.g., "start_cutscene_spot") on the floor under camera.
Build out the VideoMap Material Shader
Updated March, 2021
Fill in cutscene_video.mtr:
video/cutscene_video
{
qer_editorimage textures/editor/video
noshadows
guiSurf entity
discrete
translucent
sort "-2"
{
videoMap video/cutscene_video.mp4 // Any embedded sound ignored; TDM playback method uses separate sound file.
remoteRenderMap 512 266 // matches ratio of projection surface
red 1
green 1
blue 1
scale 1,1
translate 1,1
}
}
For more about possible color channel parameterization of this, and color/resolution effects, see Obsttortes's video in How to get security camera observation screens working.
Build out the GUI
Updated March 2021
Fill in your custom cutscene_video.gui:
windowDef Desktop
{
rect 0 ,0 ,640 ,480
background black
noevents false // required to make event handlers like onTrigger work
onTrigger
{
// commands within event handlers must end with ";", unlike elsewhere
set background "video/cutscene_video"; // "set" required here to change initial value
resetCinematics; // reset video to start
localsound "video/cutscene_video"; // This works too: localsound "sound/video/cutscene_video.ogg";
}
}
The "background black" command here is optional; it generates a console warning but does show the screen as black before the movie plays. Or delete the line to have the pre-movie screen invisible.
Then link your screen entity "video_wall" to this, by adding the spawnarg:
gui guis/cutscene_video.gui
Trigger the Playback
Fill in you <fm>.script function like:
void dovideo(){
$player1.setOrigin($start_cutscene_spot.getOrigin()); // traditional teleport method
entity cam = $func_cameraview_1;
cam.activate($player1); //start the camera
sys.trigger($video_wall); //start the cutscene
sys.wait(88); //length of video, seconds
cam.activate($player1); //stop the camera
$player1.setOrigin($after_cutscene_spot.getOrigin());
}
This can be called by any triggering device you prefer.
Discussion
Interrupting or Skipping a CutScene
Support for this is inadequate with the stated methods. That may not be a problem, particularly with short cutscenes. However, it does rule out during-cutscene volume adjustments by the gamer with the TDM Audio sliders. Perhaps someone's future work will find improvements.
With the GUI Message Overlay Method. If you hit ESC, you will return to the main TDM menu. You can subsequently "Resume Mission", and the video will resume where interrupted, but there will be no audio. The GUI shown above does not support skipping with LMB. (Early experiments adding a doAction() function seemed to show no response to LMB, whether or not the TDM clock-hand cursor was visible. Likewise, doEsc() was ineffective. But this was before awareness of "noevent false", so your mileage may vary.)
With the Movie Theatre Method. If you hit ESC once, you will break out of the camera view, so end up seeing the movie theatre, with the sound still playing. Hitting ESC again kills the audio and brings you do the main menu. "Resume" then returns you to the theatre (but NOT the camera view), and with no sound.
Choosing in Advance to Skip a CutScene
An alternative to these shortcomings is to offer some sort of switch in-game, so the player can indicate that they'd rather not see the upcoming cutscene, presumably because they've already seen it. For example, in "Away 0: Stolen Heart", there's a wall panel with glowing text that says "Skip Cutscene? Frob Me". If the player clicks that, the text disappears and a script variable is set. When, shortly later, the player arrives at where the cutscene would be triggered, a replacement slide is shown instead, to maintain story continuity. More about that in A Full-Screen Slide in Mid-Game.
Multiple Non-Sequential Cutscenes in an FM
With the GUI Message Overlay Method. Give each cutscene it's own atdm_gui_message entity and dovideo function. But they should be able to share an isolation room.
With the Movie Theatre Method. Perhaps some way can be found that doesn't require a separate movie theatre for every cutscene.
Alternative Sound Treatments
Above, we have demonstrated two audio techniques:
- localsound within the GUI;
- using a narrator's or player's voice with tdm_voice.
These have the skipping drawback noted, but are otherwise easy to do and provide the expected stereo separation.
Other possible ways to go might be:
- ambient sound through the location system.
- a separate speaker in the isolated room. Generally, this is not a great choice. Besides reducing the stereo to mono, the sound source will seem to drift around if the player moves. And the player can move in both Overlay and Theatre Methods; even if encased in a playerclip cage around $start_cutscene_spot, rotation still seems possible. But see the next section for a situation where it might be useful.
A typical speaker setup (often placed behind the Movie Theatre screen) has spawnargs:
name speaker_1 s_maxdistance 250 Adjust to cover room, but not impinge on rest of map s_waitfortrigger 1 s_shader video/cutscene_video usually found with the Entity Inspector's "Select Shader" button
Then call it in the dovideo():
... sys.trigger($video_wall); //start the cutscene sys.trigger($speaker_1); //& its audio ...
Cutscenes using Live Action with a Video Background
Speculatively, the Movie Theatre method should be combinable with traditional TDM cutscene techniques. The hybrid would have AI and a "movie set" in the foreground/middleground, and a video in the background. This approach mimics one widely used in early cinema (pre-green-screen) with a projected backdrop. Examples:
- a balcony overlooking an expansive vista with flocks of birds passing by;
- doors and windows open to a distance storm arising;
- rolling ship's deck in a roaring sea;
- passengers in a stagecoach or steam-train carriage, with the countryside wisking by.
In this hybrid approach, the camera (and player's) view would always remain centered on and perpendicular to the background screen, as per the Movie Theatre approach. This would avoid any skew or keystoning of the video that would reveal its 2D nature. Other considerations:
- Lighting would be needed for the AI and movie set.
- The player character's position is the "ear or microphone" that picks up the AI's talk, as usual with traditional cutscene techniques;
- Audio associated with the video, if used as all, would likely be quieter. This is a case where using a speaker for the video's soundtrack might be helpful. You could move the speaker around in the map to affect the relative location and volume.
- If the scene is not of fixed duration, there are additional complications. The video may have to be long enough to cover the maximum duration, or have to be looped. In the latter case, the video will need some way to mask the loop ends. For instance, moving scenery viewed from a vehicle could start and end with blurs of closeup trees or a tunnel. For potential looping methodologies to explore, see Playing ROQ Video Files on repeating a video using the GUI, or Cutscene video with FFmpeg about "videomap loop...".
Non-Cutscene Live Action with Video Background
During normal gameplay - with a normal player view, not in a cutscene - could a "video background" be used, with in-game foreground set and characters? Clearly, there are parallax issues with a 2D video. For this to be plausible, the video content and/or viewing locations (i.e., player's movements) may need to be constrained. Consider a video of a distance horizon with, say, a flight of crows. If projected on a wall in front of a skybox surface, and viewed through foreground windows, it might well work.
See Also
Video Cutscene Creation gives pluses and minuses of video versus live-action cutscenes, and some useful tips for scene/set preparations and camera control during capture.
A Full-Screen Slide in Mid-Game shows how to fashion a replacement slide for use when the player wants to skip a video cutscene.
Cutscene video with FFmpeg for video codecs supported in TDM 2.06+ and recommended for new FMs, such as MP4 or AVI. Also ideas on video looping.
Sound File Formats supported in TDM.
Video Briefing showcases related issues.
Cutscenes for traditional TDM live-action cutscene techniques.
Security Camera for live-action camera feed to a video surface
Forum discussions:
Dioramas, with scaled-down ghost-like live action effects.
