IdTech4 Open Source: Difference between revisions

From The DarkMod Wiki
Jump to navigationJump to search
No edit summary
 
(48 intermediate revisions by 4 users not shown)
Line 1: Line 1:
id Software has a long history making the code to their engines open source after a while. Even after being sold to Zenimax, this trend continued. The source code can be found [ftp://ftp.idsoftware.com/idstuff/source/ on the officiel FTP server] from id software.
id Software has a long history making the code to their engines open source after a while. Even after being sold to Zenimax, this trend continued. The source code can be found [ftp://ftp.idsoftware.com/idstuff/source/ on the officiel FTP server] from id software.


So there is hope that idTech4 (the engine behind Doom 3 and Quake 4) will also be open sourced. For The Dark Mod this means that we can fix a long list of issues that are currently not possible.
idTech4 (the engine behind Doom 3 and Quake 4) has been open sourced. For The Dark Mod this means that we can fix a long list of issues that are currently not possible.


Here is a small, incomplete list, in no particular order:
Here is a small, incomplete list, in no particular order. Some of these items have already been accomplished.


'''Todo: add more examples'''
See also: '''[[Untethered_TDM]]'''.
 
== ToDo ==


{|class="wikitable sortable" border=1 cellspacing=0 cellpadding=2 width=100%
{|class="wikitable sortable" border=1 cellspacing=0 cellpadding=2 width=100%


|-
|-
!bgcolor=#d0d0e0 width="12%"|Category
!bgcolor=#d0d0e0 width="10%"|Category
!bgcolor=#d0d0e0|Issue
!bgcolor=#d0d0e0|Issue
!bgcolor=#d0d0e0 width="12%"|Possible outcome
!bgcolor=#d0d0e0|Possible outcome


|-
|-
|Sound
|GUI
|EAX does not work under Linux.
|The GUI is 4:3 (640x480), but covers always the entire screen and is thus [https://bugs.thedarkmod.com/view.php?id=5282 distorted on 16:9].
|Works there, too.
|GUI looks better on 16:9 and on multi-monitor setups. Potential fix: https://forums.thedarkmod.com/index.php?/topic/22325-i-made-a-fix-for-ui-display-for-non-169-resolution/


|-
|-
|Graphic
|Graphic
|The lightgem needs to render to a texture and transfer this texture back to the CPU via a complicated in-memory file scheme.
|Local Cubemaps. (See HL2 env_cubemap.) Granting the ability to have reflective materials more accurately reflect their environment. It may be possible to setup scripted material fades (shaderparm?) to implement this with the current SDK framework but it would be yet another CPU dependent operation. The Skin changing in the LOD system could also be used perhaps? This one is definitely questionable about ''requiring'' Open Source, but is may be beneficial to set this up in the renderer.
|Performance increase.
|More accurate reflective materials. Gold loot (etc). Tracker: https://bugs.thedarkmod.com/view.php?id=5239
 
|-
|Animation
|GPU Skinning. Having the GPU stretch and transform the mesh would be tremendous performance enhancement for animations and would allow for extremely detailed facial animations like Team Fortress 2.
|IceColdDuke has offered assistance porting his skinning code https://bugs.thedarkmod.com/view.php?id=3984


|-
|-
Line 28: Line 35:
|No instancing. Multiple instances of the same model are all rendered one by one, instead of uploading it once and then telling the graphic card to render them all in one go.
|No instancing. Multiple instances of the same model are all rendered one by one, instead of uploading it once and then telling the graphic card to render them all in one go.
|Performance increase.
|Performance increase.
|
 
|-
|Graphic
|No access to Vertex positions. For vertex shader or geometry shader (DX10+ hardware) operations like R2VB, Vertex Texturing or Tessellation the correct Vertex locations will be needed. Fur shaders would also be possible...
|Performance increase, Fur shaders, Higher poly-counts. Note: Some of this functionality was probably already exposed but no ARB programmers could use it because it
would have increased the length of the entire interaction shader for anything "lit" though I don't believe anyone's tried calling a VFP from a material that includes the whole light interaction chain (which might've also worked. Nonetheless this remains open for improvement with robust native implementations.
 
|-
|Graphic
|Support for Light-Maps or Deluxe-Maps. While static lighting methods are not really an ideal usage for this engine, these features can have their utility. You could bake ambient bounced lights for areas that are predominantly statically lit. To make these methods more Doom 3 compatible, these could be implemented as an Ambient Light (3D texture support?) as a more robust extension of the Strombine method that is currently available (but very cumbersome). Maps made in other engines (without game specific assets) could be more easily migrated? The unused megatexture code is very similar to lightmap code. We could use the megatexture mod's reverse engineered vfp as a starting point and add PCX support. We'd want to correct the faulty UV space issues since this was built for terrains initially and stretches UV's vertically (etc).
|Performance increase. Graphic enhancement. Migrated content?
 
|-
|Sound
|Determine why "no_dups" doesn't always work and correct it. Remove the SDK workaround for this problem.
|Better sound randomization.
 
|-
|Entities
|Correct the occasional bad "trace" the engine performs.
|Eliminate spatial relationship problems.
 
 
|-
|Graphic
|Add the ability to have multiple dynamic LightFalloffImage stages
|Use ''Texture Sheets'' to render multiple light passes and maximize the work allocated to each light render. (Less waste.) Animated light texture effects can have all 3 axis of movement.
 
|-
|Mapping
|<s>Brushes cannot have Vertex Colors or Vertex Normals. Adding these attributes would give Brushes near feature parity with Models and allow for more seamless mixtures of the two construction methods. Brushes could have Patch style smoothing and curves yet could be properly scissored by a Visportal. Translucency could be applied to Brush-work prefabs.</s>
|<s>Performance. Ease of use for mappers.</s> suspended. changing the map format has been deemed too risky. could possibly be addressed via a Dark Radiant plugin that automates model export when vertex colors are applied. Currently scripts exist to approximate this functionality.
 
|}
 
== Done ==
 
{|class="wikitable sortable" border=1 cellspacing=0 cellpadding=2 width=100%
 
|-
!bgcolor=#d0d0e0 width="10%"|Category
!bgcolor=#d0d0e0|Issue
!bgcolor=#d0d0e0|Explanation
 
|-
|Sound
|Vis-portals do not diminish sound while opened.
|Open vis-portals would also diminish sound.  Sound loss would scale with the size of the portal up to a certain size, where it would then be large enough to have not require sound loss.
 
|-
|GUI
|Language support (via ''''sys_lang'''') is limited to a fixed set of languages. Also the GUI refues to use non-ASCII strings for choices or values.
|Unicode support, better i18n support and UI translateable to more languages.
 
|-
|Graphic
|The lightgem needs to render to a texture and transfer this texture back to the CPU via a complicated in-memory file scheme.
|TDM 2.05 optimized lightgem rendering by culling non-shadowcasting geometry and using a PBO to store light values. 2.06 further improved this by setting the Lightgem with it's own SMP session.


|-
|-
Line 36: Line 100:


|-
|-
|Coding
|Sound
|Single-threaded coding. It may be possible to divide the sub-systems into separate execution threads for processing on separate cores once the source code is available. The game Prey has implemented this type of change to Id Tech 4. On the extreme side of concept, OpenCL could be used to perform AI path-finding, or physics calculations.
|EAX does not work under Linux.
|Performance increase.
|TDM 2.06 has EFX from Dhewm3


|-
|-
|Graphic
|Graphic
|Shadow Volumes are calculated on the CPU. Moving to a GPU calculated Shadow Volume or a Shadow Mapping technique like Parallel Split Variance Shadow Maps would allow more on-screen geometry with active lighting applied. Soft shadows would also be MUCH less expensive than the current Z-Buffer render and Post-Process method.
|LATC2 Normal Map compression support (AKA OpenGL 3Dc) See: http://www.opengl.org/registry/specs/EXT/texture_compression_latc.txt ... Reduced Normal Map memory requirements both on the storage volume and in graphic RAM.
|Performance increase. Higher poly-counts. More active lights. Soft shadows.
|Performance increase. Graphic enhancement. Bigger maps for less RAM. Code is committed, Normal Maps were not converted due to concerns about Intel compatibility. Serpentine was last looking into RGTC as an alternative.


|-
|-
|Graphic
|Graphic
|No access to Vertex positions. For vertex shader or geometry shader (DX10+ hardware) operations like R2VB, Vertex Texturing or Tessellation the correct Vertex locations will be needed.
|Add keywords for Cubemap light method and 3D texture light methods
|Performance increase. Higher poly-counts.
|TDM 2.06 has cubicLight support in both ARB and GLSL
 
|-
|Coding
|Single-threaded coding. It may be possible to divide the sub-systems into separate execution threads for processing on separate cores once the source code is available. The game Prey has implemented this type of change to Id Tech 4. On the extreme side of this concept, OpenCL could be used to perform AI path-finding, or physics calculations.
|TDM 2.06 has moved the game, lightgem, and renderer to different SMP threads. Further improvements are planned with additional code ported from BFG.
 
|-
|Graphic
|Shadow Volumes (silhouettes) are calculated on the CPU. Moving to a GPU calculated Shadow Volume or a Shadow Mapping technique like Parallel Split Variance Shadow Maps would allow more on-screen geometry with active lighting applied. Soft shadows would also be MUCH less expensive than the current Z-Buffer render and Post-Process method.
|TDM 2.06 has buggy initial support for Shadow Maps and 2.07 (SVN) has further improvements.


|-
|-
|Mapping
|Mapping
|Brushes cannot have Vertex Colors or Vertex Normals. Adding these attributes would give Brushes near feature parity with Models and allow for more seamless mixtures of the two construction methods. Brushes could have Patch style smoothing and curves yet could be properly scissored by a Visportal. Translucency could be applied to Brush-work prefabs.
|<s>Eliminate as many hard-coded limits as possible.</s>
|Performance. Ease of use for mappers.
|<s>Reduce mapper frustration.</s> A number of limits have been removed. Some of these changes have unforeseen consequences so removing all limited is probably not feasible
 
|-
|GUI
|Font rendering is buggy (only alpha map,  [https://bugs.thedarkmod.com/view.php?id=2812 misses character 0xFF], no Unicode etc.).
|Better looking GUI and easier translations.


|}
|}
Line 59: Line 138:
== See also ==
== See also ==


* [[Untethered_TDM]]
* [ftp://ftp.idsoftware.com/idstuff/source/ FTP Server from id software]  
* [ftp://ftp.idsoftware.com/idstuff/source/ FTP Server from id software]  
 
* [http://www.thedarkmod.com/downloads.php Download the source code for TDM]
* [http://www.doom3world.org/phpbb2/viewtopic.php?t=23491 Doom3world Wish List for Open Source Doom 3]
* [http://www.doom3world.org/phpbb2/viewtopic.php?t=23491 Doom3world Wish List for Open Source Doom 3]
* [http://www.humus.name/index.php?page=3D&&start=0 Humus Demo's. Many useful graphic techniques that would require access to the renderer]


{{coding}}
{{coding}} {{standalone}}

Latest revision as of 16:24, 30 April 2024

id Software has a long history making the code to their engines open source after a while. Even after being sold to Zenimax, this trend continued. The source code can be found on the officiel FTP server from id software.

idTech4 (the engine behind Doom 3 and Quake 4) has been open sourced. For The Dark Mod this means that we can fix a long list of issues that are currently not possible.

Here is a small, incomplete list, in no particular order. Some of these items have already been accomplished.

See also: Untethered_TDM.

ToDo

Category Issue Possible outcome
GUI The GUI is 4:3 (640x480), but covers always the entire screen and is thus distorted on 16:9. GUI looks better on 16:9 and on multi-monitor setups. Potential fix: https://forums.thedarkmod.com/index.php?/topic/22325-i-made-a-fix-for-ui-display-for-non-169-resolution/
Graphic Local Cubemaps. (See HL2 env_cubemap.) Granting the ability to have reflective materials more accurately reflect their environment. It may be possible to setup scripted material fades (shaderparm?) to implement this with the current SDK framework but it would be yet another CPU dependent operation. The Skin changing in the LOD system could also be used perhaps? This one is definitely questionable about requiring Open Source, but is may be beneficial to set this up in the renderer. More accurate reflective materials. Gold loot (etc). Tracker: https://bugs.thedarkmod.com/view.php?id=5239
Animation GPU Skinning. Having the GPU stretch and transform the mesh would be tremendous performance enhancement for animations and would allow for extremely detailed facial animations like Team Fortress 2. IceColdDuke has offered assistance porting his skinning code https://bugs.thedarkmod.com/view.php?id=3984
Graphic No instancing. Multiple instances of the same model are all rendered one by one, instead of uploading it once and then telling the graphic card to render them all in one go. Performance increase.
Graphic No access to Vertex positions. For vertex shader or geometry shader (DX10+ hardware) operations like R2VB, Vertex Texturing or Tessellation the correct Vertex locations will be needed. Fur shaders would also be possible... Performance increase, Fur shaders, Higher poly-counts. Note: Some of this functionality was probably already exposed but no ARB programmers could use it because it

would have increased the length of the entire interaction shader for anything "lit" though I don't believe anyone's tried calling a VFP from a material that includes the whole light interaction chain (which might've also worked. Nonetheless this remains open for improvement with robust native implementations.

Graphic Support for Light-Maps or Deluxe-Maps. While static lighting methods are not really an ideal usage for this engine, these features can have their utility. You could bake ambient bounced lights for areas that are predominantly statically lit. To make these methods more Doom 3 compatible, these could be implemented as an Ambient Light (3D texture support?) as a more robust extension of the Strombine method that is currently available (but very cumbersome). Maps made in other engines (without game specific assets) could be more easily migrated? The unused megatexture code is very similar to lightmap code. We could use the megatexture mod's reverse engineered vfp as a starting point and add PCX support. We'd want to correct the faulty UV space issues since this was built for terrains initially and stretches UV's vertically (etc). Performance increase. Graphic enhancement. Migrated content?
Sound Determine why "no_dups" doesn't always work and correct it. Remove the SDK workaround for this problem. Better sound randomization.
Entities Correct the occasional bad "trace" the engine performs. Eliminate spatial relationship problems.


Graphic Add the ability to have multiple dynamic LightFalloffImage stages Use Texture Sheets to render multiple light passes and maximize the work allocated to each light render. (Less waste.) Animated light texture effects can have all 3 axis of movement.
Mapping Brushes cannot have Vertex Colors or Vertex Normals. Adding these attributes would give Brushes near feature parity with Models and allow for more seamless mixtures of the two construction methods. Brushes could have Patch style smoothing and curves yet could be properly scissored by a Visportal. Translucency could be applied to Brush-work prefabs. Performance. Ease of use for mappers. suspended. changing the map format has been deemed too risky. could possibly be addressed via a Dark Radiant plugin that automates model export when vertex colors are applied. Currently scripts exist to approximate this functionality.

Done

Category Issue Explanation
Sound Vis-portals do not diminish sound while opened. Open vis-portals would also diminish sound. Sound loss would scale with the size of the portal up to a certain size, where it would then be large enough to have not require sound loss.
GUI Language support (via 'sys_lang') is limited to a fixed set of languages. Also the GUI refues to use non-ASCII strings for choices or values. Unicode support, better i18n support and UI translateable to more languages.
Graphic The lightgem needs to render to a texture and transfer this texture back to the CPU via a complicated in-memory file scheme. TDM 2.05 optimized lightgem rendering by culling non-shadowcasting geometry and using a PBO to store light values. 2.06 further improved this by setting the Lightgem with it's own SMP session.
Graphic No access to the Z-Buffer. Without access to this data, any depth-based fragment shader effects like SSAO or Depth of Field require an extremely expensive z-pass render. Performance increase. More effect options.
Sound EAX does not work under Linux. TDM 2.06 has EFX from Dhewm3
Graphic LATC2 Normal Map compression support (AKA OpenGL 3Dc) See: http://www.opengl.org/registry/specs/EXT/texture_compression_latc.txt ... Reduced Normal Map memory requirements both on the storage volume and in graphic RAM. Performance increase. Graphic enhancement. Bigger maps for less RAM. Code is committed, Normal Maps were not converted due to concerns about Intel compatibility. Serpentine was last looking into RGTC as an alternative.
Graphic Add keywords for Cubemap light method and 3D texture light methods TDM 2.06 has cubicLight support in both ARB and GLSL
Coding Single-threaded coding. It may be possible to divide the sub-systems into separate execution threads for processing on separate cores once the source code is available. The game Prey has implemented this type of change to Id Tech 4. On the extreme side of this concept, OpenCL could be used to perform AI path-finding, or physics calculations. TDM 2.06 has moved the game, lightgem, and renderer to different SMP threads. Further improvements are planned with additional code ported from BFG.
Graphic Shadow Volumes (silhouettes) are calculated on the CPU. Moving to a GPU calculated Shadow Volume or a Shadow Mapping technique like Parallel Split Variance Shadow Maps would allow more on-screen geometry with active lighting applied. Soft shadows would also be MUCH less expensive than the current Z-Buffer render and Post-Process method. TDM 2.06 has buggy initial support for Shadow Maps and 2.07 (SVN) has further improvements.
Mapping Eliminate as many hard-coded limits as possible. Reduce mapper frustration. A number of limits have been removed. Some of these changes have unforeseen consequences so removing all limited is probably not feasible
GUI Font rendering is buggy (only alpha map, misses character 0xFF, no Unicode etc.). Better looking GUI and easier translations.

See also