TypeInfo and Memory debugging configuration
It is very hard to understand the purpose of TypeInfo project in Doom3 / TheDarkMod. Some might even think that it is required for normal execution of the game, which seems to be wrong. For instance, it is completely independent of idTypeInfo class, which really provides RTTI for idClass hierarchy. The truth is: the sole purpose of TypeInfo project is to support memory debugging configuration.
The following build chain summarizes the TypeInfo workflow:
- Build TypeInfo project to get TypeInfo.exe executable.
- Run TypeInfo.exe: it parses single cpp file in game directory (game/gamesys/TypeInfo_GenHelper.cpp) and generates file game/gamesys/GameTypeInfo.h.
- Build Doom 3 (or The Dark Mod) in configuration "Debug with inlines and memory log".
- Start the game.
Let's look closer at each of these steps.
Generate type info
It seems that typeinfo generator uses some systems of the game (mainly fileSystem, but also common, cmdSystem, sys). Currently these systems are built from sources in typeinfo project: the sources are shared between engine and typeinfo projects. This can cause some problems, especially after filesystem changes (e.g. minizip update). Also note all typeinfo sources include lots of stuff via precompiled header.
The executable is expected to be started from the game directory for two reasons. First, it depends on ExtLibs.dll in case of TDM. Second, it expects to find source code root in ../darkmod_src directory (see SOURCE_CODE_BASE_FOLDER define). When started, typeinfo generator chooses a single source file from game directory (in case of TDM, TypeInfo_GenHelper.cpp is taken). Then it parses this source file, going through all its non-system includes and performing all the substitutions of C preprocessor.
Note that typeinfo generator efficiently parses all the Doom3/TDM code with its own homebrew parser (partly the same one which parses game scripts). This is a very scary thing to do given the amount of weirdness in the modern C++, and surely it has some limitations. First of all, it does not support some syntax, for instance:
- #pragma once --- use #ifdef include guards instead
- #if MYX >= 3 for undefined MYX
- deleted default methods like MyClass(const MyClass&) = delete;
- calling methods in definition of global constants
Luckily, preprocessor allows to mark problematic syntax to be ignored by typeinfo parser. The typeinfo parser defines ID_TYPEINFO macro internally, so you can enclose hard stuff into #ifdef sections. However, this does not help with #if and related preprocessor commands: they are parsed and evaluated even if they are already excluded by preprocessor. One last thing to remember: include path needs to be configured in the code, if anything different from game directory is needed. Note that system includes (surrounded by angle brackets <>) are always ignored.
After all the sources are parsed, the information is put into GameTypeInfo.h header in C++-compilable form. The following information is collected and emitted:
- constants: type, name, value (including enum values and static const members)
- enums: list of all enums, list of name/value pairs for all enums
- class members: type, name, offset, size
- classes: full list, name, base class name, pointer to members list
For template classes, information is also collected, but it is commented in the generated header. Multiple inheritance is not supported: only one base class is captured. It is not necessary for a class to have virtual functions (like in C++ RTTI) or to inherit from idClass (like for ID's RTTI implementation). Member offsets and sizes are printed not as numeric values, but as C++ expressions evaluating to correct constant, which probably allows to use the same GameTypeInfo.h even in 64-bit build.
Here is an example of members information for idStr class:
static classVariableInfo_t idStr_typeInfo[] = {
{ "int", "len", (int)(&((idStr *)0)->len), sizeof( ((idStr *)0)->len ) },
{ "char *", "data", (int)(&((idStr *)0)->data), sizeof( ((idStr *)0)->data ) },
{ "int", "alloced", (int)(&((idStr *)0)->alloced), sizeof( ((idStr *)0)->alloced ) },
{ "char[20]", "baseBuffer", (int)(&((idStr *)0)->baseBuffer), sizeof( ((idStr *)0)->baseBuffer ) },
{ NULL, 0 }
};
Here is the full GameTypeInfo.h file generated for TDM.
Build with memory debugging
After GameTypeInfo.h is ready, you can build the game in Debug with inlines and memory log configuration. This configuration defines macro ID_DEBUG_MEMORY, which influences how the code works in several places. Most importantly, it ensures that GameTypeInfo.h header is included into TypeInfo.cpp. (in all the other configurations NoGameTypeInfo.h is included instead, which is a stub file with no information) This allows to use the collected information during runtime (as enriched RTTI or static reflection). In order to compile GameTypeInfo.h, the following conditions must be met:
- All members must be accessible, which is achieved by #define private public dirty hack.
- All classes must be defined, which is achieved in TDM by including TypeInfo_GenHelper.h beforehand.
The first point is quite messy and can cause issues in some compilers (e.g. GCC). Also, you cannot rely on class members being implicitly private, you have to explicitly mark them as private for successful compilation.
Run with memory debugging
Memory debugging provides the following benefits:
- detect memory leaks and wrong frees
- collect memory allocation statistics
- detect and fill with trash uninitialized members (for objects inherited from idClass)
- dump state of all game entities into human-readable file on game save
- compare saved state of all game entities with their state after game restore
In the memory debugging configuration, special implementation of memory heap is used. To achieve it, global operator new and operator delete are overloaded. In order to pass call site information, "new" macro is defined (Heap.h):
#define ID_DEBUG_NEW new( 0, 0, __FILE__, __LINE__ )
#undef new
#define new ID_DEBUG_NEW
This is a known trick which can cause a lot of issues. Most notably, it does not play well with MFC classes, which comprise the in-game editor of Doom 3. Also it fails to compile placement new syntax. Due to all of these reasons, it is masked out in many source files like this:
#ifdef ID_DEBUG_MEMORY
#undef new
#endif
On the implementation side, all the additional information (debugMemory_t) is prepended to the pointer returned. A doubly-linked list of all allocations is maintained. This allows to detect double frees and dump information about currently allocated blocks.
Here are the commands which use debugging memory heap:
- 1. com_showMemoryUsage 1 enables constantly showing global and per-frame memory stats on screen:
total allocated memory: 339346, 486388kB frame alloc: 266, 117 kB frame free: 266, 117kB
- 2. memoryDump dumps all blocks, one block per line (memorydump_full.txt):
size: 48 B: game/ai/AAS_routing.cpp, line: 894 [____________>_________________;] size: 64 B: c:/thedarkmod/tdm/idlib/containers/List.h, line: 377 [________le_atdm:moveable_candle] size: 64 B: c:/thedarkmod/tdm/idlib/containers/List.h, line: 377 [_'___________f_________________] size: 2 KB: sound/snd_world.cpp, line: 200 [__0___C_;__________D_IA__P_B___] size: 64 B: c:/thedarkmod/tdm/idlib/containers/List.h, line: 377 [_'____________=________________]
- 3. memoryDumpCompressed -s dumps per-site sorted statistics (memorydump_compressed_s.txt):
size: 256912 KB, allocs: 354: c:/thedarkmod/tdm/idlib/Allocators.h, line: 613 size: 133217 KB, allocs: 4206: renderer/tr_main.cpp, line: 297 size: 27157 KB, allocs: 34927: c:/thedarkmod/tdm/idlib/containers/List.h, line: 377 size: 10240 KB, allocs: 10: c:/thedarkmod/tdm/idlib/Allocators.h, line: 378 size: 5494 KB, allocs: 11682: framework/DeclManager.cpp, line: 1907
(most of the memory is allocated in idDynamicBlockAlloc)
Lastly, when the engine terminates, it dumps remaining allocations (tdm_main_leak_size.txt):
size: 40 KB, allocs: 3: idlib/containers/HashIndex.cpp, line: 53
size: 24 KB, allocs: 3: idlib/containers/HashIndex.cpp, line: 50
size: 22 KB, allocs: 212: game/Game_local.cpp, line: 7116
size: 18 KB, allocs: 523: idlib/Str.cpp, line: 90
size: 8 KB, allocs: 38: c:/thedarkmod/tdm/idlib/containers/List.h, line: 377
size: 6 KB, allocs: 138: game/DarkModGlobals.cpp, line: 467
size: 3 KB, allocs: 150: , line: 0
size: 3 KB, allocs: 35: renderer/Model_md5.cpp, line: 743
size: 2 KB, allocs: 110: game/darkModLAS.cpp, line: 1282
size: 1 KB, allocs: 8: game/Objectives/ObjectiveLocation.cpp, line: 72
size: 0 KB, allocs: 39: game/SndPropLoader.cpp, line: 723
size: 0 KB, allocs: 39: game/SndProp.cpp, line: 339
size: 0 KB, allocs: 13: game/SEED.cpp, line: 3449
size: 0 KB, allocs: 8: renderer/draw_arb2.cpp, line: 841
size: 0 KB, allocs: 8: c:/thedarkmod/tdm/idlib/math/Polynomial.h, line: 601
size: 0 KB, allocs: 2: framework/I18N.cpp, line: 428
size: 0 KB, allocs: 1: framework/FileSystem.cpp, line: 2756
size: 0 KB, allocs: 3: game/EscapePointManager.cpp, line: 181
1333 total memory blocks allocated
134 KB memory allocated
(note sure if they are real leaks, most likely they are caused by initialization/shutdown order issues)
There is also a very helpful command printMemInfo, which prints detailed stats about memory used by different assets (models, images, sounds, etc). It does not need memory debugging to be enabled, and it is especially helpful in Release build to better understand what eats most of memory.
All these memory debugging capabilities do not use typeinfo generated into GameTypeInfo.h yet. But the remaining features do.
It is known that Doom 3 game defines idClass base class for all game entities. It forces us to use CLASS_PROTOTYPE and CLASS_DECLARATION macros in game class declaration and definition. As a result, some events and RTTI information is generated for these classes (not dependent on memory debugging).
When memory debugging is enabled, all the members of a game object are filled with 0xcdcdcdcd just before its constructor is called. If the code contains uninitialized memory access, this pattern increases the chance of crashing the game at it (which is good). If an object is created via CreateInstance method, then additionally all its data is checked for still being 0xcdcdcdcd immediately after constructor call. If any 32-bit word still has this "cd pattern", then it is considered to be an uninitialized member. Then typeinfo information is used to see which member it is, and a nice message is printed to the game console:
WARNING:type 'idPlayer' has uninitialized variable idEntity::team (offset 488)
Finally, typeinfo allows to dump a human-readable state of the game objects. When you save game, a file like Quicksave_1_save.gameState.txt is also generated. This is a huge file with contents like this (see Quicksave_1_save.gameState.txt):
entity 0 idPlayer {
idEntity::entityNumber = "0"
idEntity::entityDefNumber = "1795"
idEntity::spawnNode = "<unknown type 'idLinkList < idEntity >'>"
idEntity::activeNode = "<unknown type 'idLinkList < idEntity >'>"
idEntity::snapshotNode = "<unknown type 'idLinkList < idEntity >'>"
idEntity::snapshotSequence = "-1"
idEntity::snapshotBits = "0"
idEntity::name = "player1"
idEntity::spawnArgs[0] = "'spawn_entnum' '0'"
idEntity::spawnArgs[1] = "'name' 'player1'"
idEntity::spawnArgs[2] = "'classname' 'atdm:player_thief'"
idEntity::spawnArgs[3] = "'model' 'model_player_thief'"
idEntity::spawnArgs[4] = "'ragdoll' 'guard_base'"
idEntity::spawnArgs[5] = "'mass' '70'"
idEntity::spawnArgs[6] = "'snd_decompress' 'splash_subtle_01'"
idEntity::spawnArgs[7] = "'snd_recompress' 'splash_subtle_01'"
idEntity::spawnArgs[8] = "'snd_airless' 'underwater'"
idEntity::spawnArgs[9] = "'def_head' 'atdm:ai_head_thief_player'"
idEntity::spawnArgs[10] = "'head_joint' 'Head'"
idEntity::spawnArgs[11] = "'offsetHeadModel' '0 0 -7'"
idEntity::spawnArgs[12] = "'team' '0'"
idEntity::spawnArgs[13] = "'spawnclass' 'idPlayer'"
idEntity::spawnArgs[14] = "'scriptobject' 'player'"
idEntity::spawnArgs[15] = "'hud' 'guis/tdm_hud.gui'"
idEntity::spawnArgs[16] = "'mphud' 'guis/mphud.gui'"
idlib containers like idList, idStr and similar are fully supported, e.g. all the elements of idList are printed. STL structures are of course not supported, but it is possible to support them too. To achieve this, a bit of new code must be written in the idTypeInfoTools::WriteVariable_r method.
When you try to load a save in memory debugging configuration, it restores the game from savefile, and then checks the state of all game objects against the ones stored in Quicksave_1_save.gameState.txt. All the differences and uninitialized members are dumped to game console, e.g.:
WARNING:idEntity::m_LODLevel uses uninitialized memory WARNING:idEntity::m_ModelLODCur uses uninitialized memory WARNING:idEntity::m_SkinLODCur uses uninitialized memory WARNING:file C:\TheDarkMod\tdm\..\darkmod\fms\innbiz\savegames\Quicksave_1_save.gameState.txt, line 84469: state diff for idEntity::cameraTarget
This makes it easier to check that save/load functionality works properly.
Also, there is testSaveGame command, which automatically starts a new game of the specified map, saves it and loads it. It may be convenient for automated testing of save/load.
Conclusion
To recap: TypeInfo is a very complicated thing used to print game objects state in human-readable form. Also it is used to detect which member exactly is not initialized in constructor of idClass-inherited class (if it is not).
Maintaining the typeinfo stuff is hard. It took me a full day to completely resurrect typeinfo & memory debugging after nobody-knows-how-many years of never using it. The question is: are the benefits worth the maintainence cost?
Probably we can use text versions of game state to write savegame converters between versions, who knows...
Note: this article was written against revision 7038 of coding SVN and revision 14847 of the main SVN.
