Full-Screen Video Cutscenes: Difference between revisions

From The DarkMod Wiki
Jump to navigationJump to search
(44 intermediate revisions by the same user not shown)
Line 1: Line 1:
''By Geep, 2020''
''By Geep, 2020-2022''
 
== TEMPORARY CAUTION (March, 2021) ==
 
If considering creation of a video cutscene with lipsync or other close synchronization, be sure to consult the bugtracker report "0005575: When Uncapped FPS is Off, Video Cutscene Loses Audio Sync", which includes a limited workaround, also discussed below as Technique 3.


'''MARCH 2020 - THIS PAGE UNDER CONSTRUCTION. INFORMATION MAY BE INCOMPLETE OR MISLEADING'''
== Introduction ==
== Introduction ==


Suppose you have created a video for a TDM in-game cutscene, using video capture and AV editing software. Your video is compliant with [[Cutscene_video_with_FFmpeg | video codecs supported in TDM 2.06+ and recommended for new FMs]] (e.g., MP4 or AVI). Likewise, [[Sound_File_Formats| TDM-supported sound formats]].
Suppose you have created a video for a TDM in-game cutscene, using video capture and AV editing software. ([[Video Cutscene Creation]] has thoughts about the why and how of this.) Your video is compliant with [[Cutscene_video_with_FFmpeg | video codecs supported in TDM 2.06+ and recommended for new FMs]] (e.g., MP4 or AVI). Likewise, [[Sound_File_Formats| TDM-supported sound formats]]. You want to play this cutscene back at a particular point in the game. Here are two methods, both of which necessarily deploy somewhat different software architecture (e.g., GUIs) than that of [[Video Briefing|briefing videos]].
 
You want to play this cutscene back at a particular point in the game. Here are two methods:


=== GUI Message Overlay Method ===
=== GUI Message Overlay Method ===
Line 12: Line 13:
* is the easiest to implement;
* is the easiest to implement;
* avoids one additional pixel resampling that the other method uses, so is likely to be more performant and the highest possible image quality;
* avoids one additional pixel resampling that the other method uses, so is likely to be more performant and the highest possible image quality;
* scales easily to multiple disjoint cutscenes.
* while handling the ESC key poorly, still does better than the other method;
* does not ''require'' the construction of a separate room (but there are advantages to doing so, as discussed under the next method);
* scales easily to multiple disjoint cutscenes;
* does not ''require'' the construction of a separate room (but there are advantages to doing so anyway, as noted under the next method).


=== Movie Theatre Method ===
=== Movie Theatre Method ===
Line 20: Line 22:
* It avoids any interactions with AI while the player has no sight control.
* It avoids any interactions with AI while the player has no sight control.
Other pluses for this method are:
Other pluses for this method are:
* It supports "localsound", the quickest-to-implement audio method;
* If you want a black frame around your cutscene (and to not have to burn it in with your video editing software), that's easy;
* If you want a black frame around your cutscene (and to not have to burn it in with your video editing software), that's easy.
* It can be extended to hybrid cutscenes with live action in front of a video backdrop.
 
These methods do <i>not</i> rely on the precisely same software architecture (e.g., same GUIs) as [[Video Briefing|briefing videos]].


== Information Common to Examples of Both Methods ==
== Information Common to Examples of Both Methods ==
Line 45: Line 45:
  }
  }


=== GUI and Script Setup ===
=== Video Shader, GUI, and Script Setup ===


Create the start of a custom GUI, here <fm>/guis/cutscene_video.gui:
Create the start of a material shader for the videomap, such as <fm>/materials/cutscene_video.mtr:
 
video/cutscene_video { // more to come }
 
Similarly, start a custom GUI, here <fm>/guis/cutscene_video.gui:


  windowDef Desktop { // more to come }
  windowDef Desktop { // more to come }
Line 57: Line 61:
== GUI Message Overlay Method ==
== GUI Message Overlay Method ==


Somewhere in your map, create an atdm_guis_message entity, with spawnargs values like:
Somewhere in your map, create an atdm:guis_message entity, with spawnargs values like:
  name atdm_guis_message_1
  name atdm_guis_message_1
  show 87                          <i>duration of the video in seconds, minus 1</i>
  show 87                          <i>duration of the video in seconds, minus 1</i>
Line 65: Line 69:
"Show" is the duration of the video in (presumably decimal) seconds, minus 1. This is because the standard code (in scriptobject tdm_gui_message) adds 1 second for a gui-based fade out, but our gui doesn't use a fade out. (And if you set hidden spawnarg "fade_out_time" to 0, it reinterprets it as 1)
"Show" is the duration of the video in (presumably decimal) seconds, minus 1. This is because the standard code (in scriptobject tdm_gui_message) adds 1 second for a gui-based fade out, but our gui doesn't use a fade out. (And if you set hidden spawnarg "fade_out_time" to 0, it reinterprets it as 1)


=== Building out the GUI ===
 
=== Building out the Material Shader for the Video Map ===
 
Fill in cutscene_video.mtr:
 
video/cutscene_video
{
    qer_editorimage textures/editor/video
    {
      videoMap video/cutscene_video.mp4 // with separate sound file
    }
}
 
=== 3 Techniques to Build out the GUI and Trigger Video and Audio ===
 
Technique 1 is quickest to implement, but can cause loss of lipsync, depending on which value the gamer sets for "Settings/Video/Advanced/Uncap FPS". So can Technique 2, which might be considered a waystation to Technique 3.
 
=== Technique 1: GUI with Localsound ===
''Jan 2022 Note: GUI code example updated to correct punctuation errors that quietly caused problems. Thanks, stgatilov''


Fill in cutscene_video.gui with:
Fill in cutscene_video.gui with:
Line 71: Line 93:
  windowDef Desktop
  windowDef Desktop
  {
  {
   rect 0 ,0 ,640 ,480
   rect 0 ,0 ,640 ,480 // Standard element to define nominal grid, e.g., for placing and sizing buttons
   backcolor 1,1,1,0
   backcolor 1,1,1,0
   matcolor 0, 0, 0, 1
   matcolor 0, 0, 0, 1
  nocursor 1
   
   
   background "video/cutscene_video"; // mp4 shader defined in materials/cutscene_video.mtr file.
   background "video/cutscene_video" // mp4 shader defined in materials/cutscene_video.mtr file.
   
   
   onTime 0
   onTime 0
   {
   {
      nocursor 1 // This work ONLY in this location, not in Desktop or after reset/matcolor here
       resetCinematics; // reset Video to start
       resetCinematics; // reset Video to start
       Desktop::matcolor 1,1,1,1; // show video
       set "Desktop::matcolor" "1,1,1,1"; // show video
       // Doesn't work: localsound "video/cutscene_video"; // shader as defined in sound/cutscene_video.sndshd
       localsound "video/cutscene_video"; // shader as defined in sound/cutscene_video.sndshd
   }
   }
  }
  }


Be aware that, while this specific code suppresses the appearance of the standard large clock-hand cursor seen in the TDM main menu, small code changes can cause it to re-appear, dead in the center of the screen.
The "nocursor 1" line suppresses the appearance - dead in the center of the screen - of the standard large clock-hand cursor seen in the TDM main menu.
 
=== Technique 2: Trigger the Audio Separately ===
Use the same GUI as above, except remove the "localsound" line.


=== Handling the Audio ===
Then, handle the audio. Someplace in your map, following the [[Script_objects| tdm_voice]] approach to a narrator's or player's voice, create two objects:
Someplace in your map, following the [[Script_objects| tdm_voice]] approach to a narrator's or player's voice, create two objects:
* an atdm:trigger_voice entity (say, atdm_trigger_voice_1)
* an atdm:trigger_voice entity (say, atdm_trigger_voice_1)
* an atdm:voice entity (e.g., atdm_voice_1)
* an atdm:voice entity (e.g., atdm_voice_1)
Line 97: Line 121:
* in Dark Radiant, select atdm_trigger_voice_1, select also voice_1, then Control-K.
* in Dark Radiant, select atdm_trigger_voice_1, select also voice_1, then Control-K.


In atdm_voice_1, leave this spawnarg at it's default of:
In atdm_voice_1, leave this spawnarg at its default of:
  s_shader  silence
  s_shader  silence


Line 112: Line 136:


  void dovideo(){
  void dovideo(){
     sys.trigger($atdm_gui_message_1); //start the cutscene
     sys.trigger($atdm_gui_message_1);   //start the cutscene. All you need for Technique 1.
     sys.trigger($adtm_trigger_voice_1); //and audio
    sys.trigger($adtm_trigger_voice_1); //start the audio, for Technique 2.
}
 
The dovideo function can be called by any triggering device you prefer. The value of using a script function here (as opposed to a triggering relay) becomes more apparent with Technique 3 or with player isolation described below.
 
=== Technique 3 - Preserving Lipsync ===
Here, you first chop of your audio into segments, each ideally under a dozen seconds. Chops are made at dialog pauses, or generally any place where you want to (in-effect) force resync. Record the duration of each segment. Then each segment separately gets the audio treatment of Technique 2. While you will be defining multiple sound shaders, that can all happen within a single .sndshd file. After adding multiple triggers to your map, call them in a script such as:
 
void dovideo(){
    sys.trigger($atdm_gui_message_1);  //start the cutscene.
     sys.trigger($adtm_trigger_voice_1); //start audio segment 1
    sys.wait(12.3); // duration of segment 1
    sys.trigger($adtm_trigger_voice_2); //start audio segment 2
    sys.wait(8.4); // duration of segment 2
    sys.trigger($adtm_trigger_voice_3); //start audio segment 3
    sys.wait(4.7); // last wait may not be necessary, depending on what else your script does
  }
  }


This function can be called by any triggering device you prefer. The value of a script function (as opposed to a triggering relay) is more apparent in what follows.)
Tip if planning to use this technique: Make sure when recording your choppable audio track that it contains no music or continuous ambient sounds, because that would become ragged by chopping and re-triggering. Instead, provide any such sound separately, through an ambient or regular speaker in the player isolation room, discussed next.


=== Optional Player Isolation ===
=== Optional Player Isolation ===
Line 130: Line 169:


  void dovideo(){
  void dovideo(){
     sys.teleport($start_cutscene_spot);
     $player1.setOrigin($start_cutscene_spot.getOrigin()); // traditional teleport method
     sys.trigger($atdm_gui_message_1); //start the cutscene
     sys.trigger($atdm_gui_message_1);   //start the cutscene
     sys.trigger($adtm_trigger_voice_1); //and audio
     sys.trigger($adtm_trigger_voice_1); //and audio
     sys.wait(88); // length of video... then teleport away
     sys.wait(88);                       //length of video... then teleport away
     sys.teleport($after_cutscene_spot);
     $player1.setOrigin($after_cutscene_spot.getOrigin());
  }
  }


Line 151: Line 190:


Next, create a func_cameraview entity (named here "func_cameraview_1"), and move it to be centered on the screen, at a perpendicular distance of about 184 units. 90-degree-rotate the camera so its arrow faces the screen. Give it the spawnargs:
Next, create a func_cameraview entity (named here "func_cameraview_1"), and move it to be centered on the screen, at a perpendicular distance of about 184 units. 90-degree-rotate the camera so its arrow faces the screen. Give it the spawnargs:
* trigger 1
trigger   1
* target video_wall
target   video_wall


You may have to rotate the camera further during debugging the setup. And the distance can be adjusted to taste, knowing that you are trading off edge clipping on various monitors versus black bars. A distance of 200 will provide a black frame (your darkened room walls) around the entire window.
You may have to rotate the camera further during debugging the setup. And the distance can be adjusted to taste, knowing that you are trading off edge clipping on various monitors versus black bars. A distance of 200 will provide a black frame (your darkened room walls) around the entire window.


Finally, as discussed with the other method, quiet the player's footfalls and place a teleport target (e.g., "start_cutscene_spot") on the floor under camera.
Finally, as discussed with the other method, quiet the player's footfalls and place a teleport target (e.g., "start_cutscene_spot") on the floor under camera.
=== Build out the VideoMap Material Shader ===
''Updated March, 2021''
Fill in cutscene_video.mtr:
video/cutscene_video
{
  qer_editorimage textures/editor/video
    noshadows
    guiSurf entity
    discrete
    translucent
    sort "-2"
  {
    videoMap video/cutscene_video.mp4 // Any embedded sound ignored; TDM playback method uses separate sound file.
      remoteRenderMap 512 266 // matches ratio of projection surface
      red      1
      green    1
      blue      1
      scale    1,1
      translate 1,1
  }
}
For more about possible color channel parameterization of this, and color/resolution effects, see Obsttortes's video in [https://forums.thedarkmod.com/index.php?/topic/14394-apples-and-peaches-obsttortes-mapping-and-scripting-thread/page/11/&tab=comments#comment-349960 How to get security camera observation screens working].


=== Build out the GUI ===
=== Build out the GUI ===
''Updated March 2021''


Fill in your custom cutscene_video.gui:
Fill in your custom cutscene_video.gui:
Line 165: Line 231:
  {
  {
   rect 0 ,0 ,640 ,480
   rect 0 ,0 ,640 ,480
  backcolor 1,1,1,0
   background black
  Desktop::matcolor 0, 0, 0, 1
   noevents false // required to make event handlers like onTrigger work
   background "textures/darkmod/paint_paper/wallpaper_fancy_red"
   // or use "tdm_black" or "black_background"... doesn't much matter because you'll never see it
  background "video/cutscene_video"; // mp4 shader defined in materials/cutscene_video.mtr file.
  onTime 99999
  {
      // Delay start time of vid playback
  }
   
   
   onTrigger
   onTrigger
   {
   {
      resetCinematics; // reset Video to start
    // commands within event handlers must end with ";", unlike elsewhere
      Desktop::matcolor 1,1,1,1; // show video
    set "background" "video/cutscene_video"; // "set" required here to change initial value
      localsound "video/cutscene_video"; // shader as defined in sound/cutscene_video.sndshd
    resetCinematics; // reset video to start
    localsound "video/cutscene_video"; // This works too: localsound "sound/video/cutscene_video.ogg";
   }
   }
  }
  }


Then link your screen entity video_wall to it, by adding the spawnarg "gui guis/cutscene_video.gui".
The "background black" command here is optional; it generates a console warning but does show the screen as black before the movie plays. Or delete the line to have the pre-movie screen invisible.
 
Then link your screen entity "video_wall" to this, by adding the spawnarg:
gui   guis/cutscene_video.gui


=== Trigger the Playback ===
=== Trigger the Playback ===
Line 192: Line 253:


  void dovideo(){
  void dovideo(){
     sys.teleport($start_cutscene_spot);
     $player1.setOrigin($start_cutscene_spot.getOrigin()); // traditional teleport method
     entity cam = $func_cameraview_1;
     entity cam = $func_cameraview_1;
     cam.activate($player1); //start the camera
     cam.activate($player1);   //start the camera
     sys.trigger($video_wall); //start the cutscene
     sys.trigger($video_wall); //start the cutscene
     sys.wait(88); // length of video
     sys.wait(88);             //length of video, seconds
     cam.activate($player1); //stop the camera
     cam.activate($player1);   //stop the camera
     sys.teleport($after_cutscene_spot);
     $player1.setOrigin($after_cutscene_spot.getOrigin());
  }
  }


Line 206: Line 267:


=== Interrupting or Skipping a CutScene ===
=== Interrupting or Skipping a CutScene ===
Support for this is limited. That may not be a problem, particularly with short cutscenes.
Support for this is inadequate with the stated methods. That may not be a problem, particularly with short cutscenes. However, it does rule out during-cutscene volume adjustments by the gamer with the TDM Audio sliders. Perhaps someone's future work will find improvements.


''With the GUI Message Overlay Method.'' If you hit ESC, you will return to the main TDM menu. You can subsequently "Resume Mission", and the video will resume where interrupted, but there will be no audio. The GUI shown above does not support skipping with LMB. (Experiments adding a doAction() function seemed to show no response to LMB, whether or not the TDM clock-hand cursor was visible. Likewise, doEsc() was ineffective. Your mileage may vary.)
''With the GUI Message Overlay Method.'' If you hit ESC, you will return to the main TDM menu. You can subsequently "Resume Mission", and the video will resume where interrupted, but there will be no audio. The GUI shown above does not support skipping with LMB. (Early experiments adding a doAction() function seemed to show no response to LMB, whether or not the TDM clock-hand cursor was visible. Likewise, doEsc() was ineffective. But this was before awareness of "noevent false", so your mileage may vary.)


''With the Movie Theatre Method.'' [TO DO]
''With the Movie Theatre Method.'' If you hit ESC once, you will break out of the camera view, so end up seeing the movie theatre, with the sound still playing. Hitting ESC again kills the audio and brings you do the main menu. "Resume" then returns you to the theatre (but NOT the camera view), and with no sound.
 
=== Choosing in Advance to Skip a CutScene ===
An alternative to these shortcomings is to offer some sort of switch in-game, so the player can indicate that they'd rather not see the upcoming cutscene, presumably because they've already seen it. For example, in "Away 0: Stolen Heart", there's a wall panel with glowing text that says "Skip Cutscene? Frob Me". If the player clicks that, the text disappears and a script variable is set. When, shortly later, the player arrives at where the cutscene would be triggered, a replacement slide is shown instead, to maintain story continuity. More about that in [[A Full-Screen Slide in Mid-Game]].


=== Multiple Non-Sequential Cutscenes in an FM ===
=== Multiple Non-Sequential Cutscenes in an FM ===
Line 225: Line 289:
Other possible ways to go might be:
Other possible ways to go might be:
* ambient sound through the location system.
* ambient sound through the location system.
* a separate speaker in the isolated room. This is not a great choice. Besides reducing the stereo to mono, the sound source will seem to drift around if the player moves. And the player can move in both methods; even if encased in a playerclip cage around $start_cutscene_spot, rotation still seems possible.
* a separate speaker in the isolated room. Generally, this is not a great choice. Besides reducing the stereo to mono, the sound source will seem to drift around if the player moves. And the player can move in both Overlay and Theatre Methods; even if encased in a playerclip cage around $start_cutscene_spot, rotation still seems possible. But see the next section for a situation where it might be useful.
 
A typical speaker setup (often placed behind the Movie Theatre screen) has spawnargs:
name            speaker_1
s_maxdistance    250  ''Adjust to cover room, but not impinge on rest of map''
s_waitfortrigger 1
s_shader        video/cutscene_video  ''usually found with the Entity Inspector's "Select Shader" button''
 
Then call it in the dovideo():
  ...
  sys.trigger($video_wall); //start the cutscene
  sys.trigger($speaker_1); //& its audio
  ...
 
=== Cutscenes using Live Action with a Video Background ===
Speculatively, the Movie Theatre method should be combinable with [[Cutscenes | traditional TDM cutscene techniques]]. The hybrid would have AI and a "movie set" in the foreground/middleground, and a video in the background. This approach mimics one widely used in early cinema (pre-green-screen) with a projected backdrop. Examples:
* a balcony overlooking an expansive vista with flocks of birds passing by;
* doors and windows open to a distance storm arising;
* rolling ship's deck in a roaring sea;
* passengers in a stagecoach or steam-train carriage, with the countryside wisking by.
 
In this hybrid approach, the camera (and player's) view would always remain centered on and perpendicular to the background screen, as per the Movie Theatre approach. This would avoid any skew or keystoning of the video that would reveal its 2D nature. Other considerations:
* Lighting would be needed for the AI and movie set.
* The player character's position is the "ear or microphone" that picks up the AI's talk, as usual with [[Cutscenes | traditional cutscene techniques]];
* Audio associated with the video, if used as all, would likely be quieter. This is a case where using a speaker for the video's soundtrack might be helpful. You could move the speaker around in the map to affect the relative location and volume.
* If the scene is not of fixed duration, there are additional complications. The video may have to be long enough to cover the maximum duration, or have to be looped. In the latter case, the video will need some way to mask the loop ends. For instance, moving scenery viewed from a vehicle could start and end with blurs of closeup trees or a tunnel. For potential looping methodologies to explore, see [[Playing ROQ Video Files]] on repeating a video using the GUI, or [[Cutscene video with FFmpeg]] about "videomap loop...".
 
=== Non-Cutscene Live Action with Video Background ===
During normal gameplay - with a normal player view, not in a cutscene - could a "video background" be used, with in-game foreground set and characters? Clearly, there are parallax issues with a 2D video. For this to be plausible, the video content and/or viewing locations (i.e., player's movements) may need to be constrained. Consider a video of a distance horizon with, say, a flight of crows. If projected on a wall in front of a skybox surface, and viewed through foreground windows, it might well work.
 
== See Also ==
[[Video Cutscene Creation]] gives pluses and minuses of video versus live-action cutscenes, and some useful tips for scene/set preparations and camera control during capture.
 
[[A Full-Screen Slide in Mid-Game]] shows how to fashion a replacement slide for use when the player wants to skip a video cutscene.
 
[[Cutscene video with FFmpeg]] for video codecs supported in TDM 2.06+ and recommended for new FMs, such as MP4 or AVI. Also ideas on video looping.
 
[[Sound File Formats]] supported in TDM.
 
[[Video Briefing]] showcases related issues.
 
[[Cutscenes]] for traditional TDM live-action cutscene techniques.
 
[[Security Camera]] for live-action camera feed to a video surface
 
Forum discussions:
 
[https://forums.thedarkmod.com/index.php?/topic/10003-so-what-are-you-working-on-right-now/page/327/&tab=comments#comment-442392 Dioramas], with scaled-down ghost-like live action effects.
 
[https://forums.thedarkmod.com/index.php?/topic/11884-mirrors-plz/&tab=comments#comment-236393 Glass and Magic Mirrors]
 
[https://forums.thedarkmod.com/index.php?/topic/10003-so-what-are-you-working-on-right-now/page/329/&tab=comments#comment-444897 Magic Screens or Prey-like portals]
{{editing}}
[[Category:Video and cutscenes]]

Revision as of 17:42, 5 January 2022

By Geep, 2020-2022

TEMPORARY CAUTION (March, 2021)

If considering creation of a video cutscene with lipsync or other close synchronization, be sure to consult the bugtracker report "0005575: When Uncapped FPS is Off, Video Cutscene Loses Audio Sync", which includes a limited workaround, also discussed below as Technique 3.

Introduction

Suppose you have created a video for a TDM in-game cutscene, using video capture and AV editing software. (Video Cutscene Creation has thoughts about the why and how of this.) Your video is compliant with video codecs supported in TDM 2.06+ and recommended for new FMs (e.g., MP4 or AVI). Likewise, TDM-supported sound formats. You want to play this cutscene back at a particular point in the game. Here are two methods, both of which necessarily deploy somewhat different software architecture (e.g., GUIs) than that of briefing videos.

GUI Message Overlay Method

This method repurposes the standard message entity that overlays the full screen (e.g., as the TDM main menu) to display a video instead. It is the generally-preferred method, because it:

  • is the easiest to implement;
  • avoids one additional pixel resampling that the other method uses, so is likely to be more performant and the highest possible image quality;
  • while handling the ESC key poorly, still does better than the other method;
  • scales easily to multiple disjoint cutscenes;
  • does not require the construction of a separate room (but there are advantages to doing so anyway, as noted under the next method).

Movie Theatre Method

This requires the construction of a separate, isolated movie-theatre-like room (e.g, in the void). You teleport the player there, play the cutscene with the player's view locked to a camera looking at the screen, then teleport back. The isolation has these advantages:

  • It provides audio and lighting that is separate from the rest of the map.
  • It avoids any interactions with AI while the player has no sight control.

Other pluses for this method are:

  • If you want a black frame around your cutscene (and to not have to burn it in with your video editing software), that's easy;
  • It can be extended to hybrid cutscenes with live action in front of a video backdrop.

Information Common to Examples of Both Methods

Video and Sound Files

In our examples here, we assume an MP4 and a separate OGG audio file, specifically:

  • an MPEG 4 video, 16:9 aspect ratio, called <fm>/video/cutscene_video.mp4
  • a separate OGG audio file, stereo, called <fm>/sound/video/cutscene_video.ogg
  • both have a length of 88 seconds.

Can an audio track that is embedded in the video be used directly? Probably not at this time. (Using embedded audio is possible with briefing videos, because special programming was done in TDM 2.06 to allow it.)

Sound Shader

Create a sound shader for the OGG, for instance <fm>/sound/cutscene_video.sndshd, with content:

video/cutscene_video
{
   sound/video/cutscene_video.ogg
}

Video Shader, GUI, and Script Setup

Create the start of a material shader for the videomap, such as <fm>/materials/cutscene_video.mtr:

video/cutscene_video { // more to come }

Similarly, start a custom GUI, here <fm>/guis/cutscene_video.gui:

windowDef Desktop { // more to come }

Also, add a function to <fm>/maps/<fm>.script to trigger the playback, like:

void dovideo() { // more to come }

GUI Message Overlay Method

Somewhere in your map, create an atdm:guis_message entity, with spawnargs values like:

name	atdm_guis_message_1
show	87                          duration of the video in seconds, minus 1
gui	guis/cutscene_video.gui     overrides the standard gui

You can delete any non-default "text" or "lines" spawnargs as irrelevant.

"Show" is the duration of the video in (presumably decimal) seconds, minus 1. This is because the standard code (in scriptobject tdm_gui_message) adds 1 second for a gui-based fade out, but our gui doesn't use a fade out. (And if you set hidden spawnarg "fade_out_time" to 0, it reinterprets it as 1)


Building out the Material Shader for the Video Map

Fill in cutscene_video.mtr:

video/cutscene_video
{
   qer_editorimage textures/editor/video
   {
      videoMap video/cutscene_video.mp4 // with separate sound file
   }
} 

3 Techniques to Build out the GUI and Trigger Video and Audio

Technique 1 is quickest to implement, but can cause loss of lipsync, depending on which value the gamer sets for "Settings/Video/Advanced/Uncap FPS". So can Technique 2, which might be considered a waystation to Technique 3.

Technique 1: GUI with Localsound

Jan 2022 Note: GUI code example updated to correct punctuation errors that quietly caused problems. Thanks, stgatilov

Fill in cutscene_video.gui with:

windowDef Desktop
{
  rect 0 ,0 ,640 ,480 // Standard element to define nominal grid, e.g., for placing and sizing buttons
  backcolor 1,1,1,0
  matcolor 0, 0, 0, 1
  nocursor 1

  background "video/cutscene_video" // mp4 shader defined in materials/cutscene_video.mtr file.

  onTime 0
  {
     resetCinematics; // reset Video to start
     set "Desktop::matcolor" "1,1,1,1"; // show video
     localsound "video/cutscene_video"; // shader as defined in sound/cutscene_video.sndshd
  }
}

The "nocursor 1" line suppresses the appearance - dead in the center of the screen - of the standard large clock-hand cursor seen in the TDM main menu.

Technique 2: Trigger the Audio Separately

Use the same GUI as above, except remove the "localsound" line.

Then, handle the audio. Someplace in your map, following the tdm_voice approach to a narrator's or player's voice, create two objects:

  • an atdm:trigger_voice entity (say, atdm_trigger_voice_1)
  • an atdm:voice entity (e.g., atdm_voice_1)

Link from the trigger_voice to the voice entity, either by

  • adding spawnarg "target0 atdm_voice_1" to atdm_trigger_voice_1; or
  • in Dark Radiant, select atdm_trigger_voice_1, select also voice_1, then Control-K.

In atdm_voice_1, leave this spawnarg at its default of:

s_shader   silence

In atdm_trigger_voice_1, add a spawnarg "snd_say", then use the "Choose sound" button to find your sound shader, e.g.:

snd_say   video/cutscene_video

Note also the spawnarg "as_player", which can left at 1 (the default) to represent the sound as the player's voice, or changed to 0 as a narrator. Either work for our purposes. The difference is presumably:

  • sound from the player (but not the narrator) could alert any AI; and
  • TDM volume controls distinguish the two, that is, in the TDM Audio menu, "Player Voice Volume" slider versus "Narrator Volume" slider. Similarly for cvars tdm_voice_player_volume versus (for narrator) tdm_voice_from_off_volume.

Building Out the Script Function to Trigger the Playback

Fill in your <fm>.script function:

void dovideo(){
   sys.trigger($atdm_gui_message_1);   //start the cutscene. All you need for Technique 1.
   sys.trigger($adtm_trigger_voice_1); //start the audio, for Technique 2.
}

The dovideo function can be called by any triggering device you prefer. The value of using a script function here (as opposed to a triggering relay) becomes more apparent with Technique 3 or with player isolation described below.

Technique 3 - Preserving Lipsync

Here, you first chop of your audio into segments, each ideally under a dozen seconds. Chops are made at dialog pauses, or generally any place where you want to (in-effect) force resync. Record the duration of each segment. Then each segment separately gets the audio treatment of Technique 2. While you will be defining multiple sound shaders, that can all happen within a single .sndshd file. After adding multiple triggers to your map, call them in a script such as:

void dovideo(){
   sys.trigger($atdm_gui_message_1);   //start the cutscene.
   sys.trigger($adtm_trigger_voice_1); //start audio segment 1
   sys.wait(12.3); // duration of segment 1
   sys.trigger($adtm_trigger_voice_2); //start audio segment 2
   sys.wait(8.4); // duration of segment 2
   sys.trigger($adtm_trigger_voice_3); //start audio segment 3
   sys.wait(4.7); // last wait may not be necessary, depending on what else your script does
}

Tip if planning to use this technique: Make sure when recording your choppable audio track that it contains no music or continuous ambient sounds, because that would become ragged by chopping and re-triggering. Instead, provide any such sound separately, through an ambient or regular speaker in the player isolation room, discussed next.

Optional Player Isolation

It may be a good idea to create an isolated room for the player to reside in during the cutscene. The advantages of this are largely the same as for separate room used in the Movie Theatre method:

  • It provides audio that is separate from the rest of the map.
  • It avoids any interactions with AI while the player's in the box.

While you can use the same room design as Movie Theatre, the dimensions are more flexible here, so perhaps your existing blue room in the void could be outfitted for the purpose.

While seeing only the cutscene, the player can still move around, so maybe cover the floor with moss or a carpet to make that quiet. Or pin the player, e.g., in player clip. Also, place some named object (e.g., a rug "start_cutscene_spot") on the floor, to use as a convenient teleport target.

Then alter the doVideo function to teleport the player there, play the cutscene, then teleport to where the player needs to be afterwards:

void dovideo(){
   $player1.setOrigin($start_cutscene_spot.getOrigin()); // traditional teleport method
   sys.trigger($atdm_gui_message_1);   //start the cutscene
   sys.trigger($adtm_trigger_voice_1); //and audio
   sys.wait(88);                       //length of video... then teleport away
   $player1.setOrigin($after_cutscene_spot.getOrigin());
}

Movie Theatre Method

Room Setup

Begin by creating the void-sealing room as a cube with an internal dimension of 512 in all 3 directions. Paint the interior dark. (For debugging, it's useful to have a perceptable texture like dark marble, plus a dim private ambient light source.)

Inside this box, near and parallel to one wall, place an internal wall (from a brush converted to func_static) to serve as a screen. Name it "video_wall". Given that we have a 16:9 video, make the size of this wall 512 wide x 288 high. Move it up to be centered vertically inside the room. Select the front face as the screen surface, then apply and fit an entityGUI texture to it. Specifically:

  • Select one surface as the screen: Control-Shift LMB.
  • Using Texture editor/Media, click on the background, search for "entityGUI". When found (under "common/"), select the text, then right-click and "Apply to Selection".
  • Using the Surface editor, hit the "Fit" button.

You will be linking the screen to a custom GUI further below.

Next, create a func_cameraview entity (named here "func_cameraview_1"), and move it to be centered on the screen, at a perpendicular distance of about 184 units. 90-degree-rotate the camera so its arrow faces the screen. Give it the spawnargs:

trigger   1
target    video_wall

You may have to rotate the camera further during debugging the setup. And the distance can be adjusted to taste, knowing that you are trading off edge clipping on various monitors versus black bars. A distance of 200 will provide a black frame (your darkened room walls) around the entire window.

Finally, as discussed with the other method, quiet the player's footfalls and place a teleport target (e.g., "start_cutscene_spot") on the floor under camera.

Build out the VideoMap Material Shader

Updated March, 2021

Fill in cutscene_video.mtr:

video/cutscene_video
{
  qer_editorimage textures/editor/video
    noshadows
    guiSurf entity
    discrete
    translucent
    sort "-2"
  {
    videoMap video/cutscene_video.mp4 // Any embedded sound ignored; TDM playback method uses separate sound file.
      remoteRenderMap 512 266 // matches ratio of projection surface
      red       1
      green     1
      blue      1
      scale     1,1
      translate 1,1
  }
} 

For more about possible color channel parameterization of this, and color/resolution effects, see Obsttortes's video in How to get security camera observation screens working.

Build out the GUI

Updated March 2021

Fill in your custom cutscene_video.gui:

windowDef Desktop
{
  rect 0 ,0 ,640 ,480
  background black
  noevents false // required to make event handlers like onTrigger work

  onTrigger
  {
    // commands within event handlers must end with ";", unlike elsewhere
    set "background" "video/cutscene_video"; // "set" required here to change initial value
    resetCinematics; // reset video to start
    localsound "video/cutscene_video"; // This works too: localsound "sound/video/cutscene_video.ogg";
  }
}

The "background black" command here is optional; it generates a console warning but does show the screen as black before the movie plays. Or delete the line to have the pre-movie screen invisible.

Then link your screen entity "video_wall" to this, by adding the spawnarg:

gui    guis/cutscene_video.gui

Trigger the Playback

Fill in you <fm>.script function like:

void dovideo(){
   $player1.setOrigin($start_cutscene_spot.getOrigin()); // traditional teleport method
   entity cam = $func_cameraview_1;
   cam.activate($player1);   //start the camera
   sys.trigger($video_wall); //start the cutscene
   sys.wait(88);             //length of video, seconds
   cam.activate($player1);   //stop the camera
   $player1.setOrigin($after_cutscene_spot.getOrigin());
}

This can be called by any triggering device you prefer.

Discussion

Interrupting or Skipping a CutScene

Support for this is inadequate with the stated methods. That may not be a problem, particularly with short cutscenes. However, it does rule out during-cutscene volume adjustments by the gamer with the TDM Audio sliders. Perhaps someone's future work will find improvements.

With the GUI Message Overlay Method. If you hit ESC, you will return to the main TDM menu. You can subsequently "Resume Mission", and the video will resume where interrupted, but there will be no audio. The GUI shown above does not support skipping with LMB. (Early experiments adding a doAction() function seemed to show no response to LMB, whether or not the TDM clock-hand cursor was visible. Likewise, doEsc() was ineffective. But this was before awareness of "noevent false", so your mileage may vary.)

With the Movie Theatre Method. If you hit ESC once, you will break out of the camera view, so end up seeing the movie theatre, with the sound still playing. Hitting ESC again kills the audio and brings you do the main menu. "Resume" then returns you to the theatre (but NOT the camera view), and with no sound.

Choosing in Advance to Skip a CutScene

An alternative to these shortcomings is to offer some sort of switch in-game, so the player can indicate that they'd rather not see the upcoming cutscene, presumably because they've already seen it. For example, in "Away 0: Stolen Heart", there's a wall panel with glowing text that says "Skip Cutscene? Frob Me". If the player clicks that, the text disappears and a script variable is set. When, shortly later, the player arrives at where the cutscene would be triggered, a replacement slide is shown instead, to maintain story continuity. More about that in A Full-Screen Slide in Mid-Game.

Multiple Non-Sequential Cutscenes in an FM

With the GUI Message Overlay Method. Give each cutscene it's own atdm_gui_message entity and dovideo function. But they should be able to share an isolation room.

With the Movie Theatre Method. Perhaps some way can be found that doesn't require a separate movie theatre for every cutscene.

Alternative Sound Treatments

Above, we have demonstrated two audio techniques:

  • localsound within the GUI;
  • using a narrator's or player's voice with tdm_voice.

These have the skipping drawback noted, but are otherwise easy to do and provide the expected stereo separation.

Other possible ways to go might be:

  • ambient sound through the location system.
  • a separate speaker in the isolated room. Generally, this is not a great choice. Besides reducing the stereo to mono, the sound source will seem to drift around if the player moves. And the player can move in both Overlay and Theatre Methods; even if encased in a playerclip cage around $start_cutscene_spot, rotation still seems possible. But see the next section for a situation where it might be useful.

A typical speaker setup (often placed behind the Movie Theatre screen) has spawnargs:

name             speaker_1
s_maxdistance    250   Adjust to cover room, but not impinge on rest of map
s_waitfortrigger 1
s_shader         video/cutscene_video  usually found with the Entity Inspector's "Select Shader" button

Then call it in the dovideo():

  ...
  sys.trigger($video_wall);	//start the cutscene
  sys.trigger($speaker_1);	//& its audio
  ...

Cutscenes using Live Action with a Video Background

Speculatively, the Movie Theatre method should be combinable with traditional TDM cutscene techniques. The hybrid would have AI and a "movie set" in the foreground/middleground, and a video in the background. This approach mimics one widely used in early cinema (pre-green-screen) with a projected backdrop. Examples:

  • a balcony overlooking an expansive vista with flocks of birds passing by;
  • doors and windows open to a distance storm arising;
  • rolling ship's deck in a roaring sea;
  • passengers in a stagecoach or steam-train carriage, with the countryside wisking by.

In this hybrid approach, the camera (and player's) view would always remain centered on and perpendicular to the background screen, as per the Movie Theatre approach. This would avoid any skew or keystoning of the video that would reveal its 2D nature. Other considerations:

  • Lighting would be needed for the AI and movie set.
  • The player character's position is the "ear or microphone" that picks up the AI's talk, as usual with traditional cutscene techniques;
  • Audio associated with the video, if used as all, would likely be quieter. This is a case where using a speaker for the video's soundtrack might be helpful. You could move the speaker around in the map to affect the relative location and volume.
  • If the scene is not of fixed duration, there are additional complications. The video may have to be long enough to cover the maximum duration, or have to be looped. In the latter case, the video will need some way to mask the loop ends. For instance, moving scenery viewed from a vehicle could start and end with blurs of closeup trees or a tunnel. For potential looping methodologies to explore, see Playing ROQ Video Files on repeating a video using the GUI, or Cutscene video with FFmpeg about "videomap loop...".

Non-Cutscene Live Action with Video Background

During normal gameplay - with a normal player view, not in a cutscene - could a "video background" be used, with in-game foreground set and characters? Clearly, there are parallax issues with a 2D video. For this to be plausible, the video content and/or viewing locations (i.e., player's movements) may need to be constrained. Consider a video of a distance horizon with, say, a flight of crows. If projected on a wall in front of a skybox surface, and viewed through foreground windows, it might well work.

See Also

Video Cutscene Creation gives pluses and minuses of video versus live-action cutscenes, and some useful tips for scene/set preparations and camera control during capture.

A Full-Screen Slide in Mid-Game shows how to fashion a replacement slide for use when the player wants to skip a video cutscene.

Cutscene video with FFmpeg for video codecs supported in TDM 2.06+ and recommended for new FMs, such as MP4 or AVI. Also ideas on video looping.

Sound File Formats supported in TDM.

Video Briefing showcases related issues.

Cutscenes for traditional TDM live-action cutscene techniques.

Security Camera for live-action camera feed to a video surface

Forum discussions:

Dioramas, with scaled-down ghost-like live action effects.

Glass and Magic Mirrors

Magic Screens or Prey-like portals