Analyze a Memory Dump

You received a crash dump and don't know how to analyze it? That's not a big deal, really. Don't get scared by numerous references to WinDbg and CDB in relation to memory dumps! You can open dumps directly in Visual Studio debugger.

Before starting, you might want to read Microsoft docs about memory dumps, so that understand what it is and what is included.

Learn version

First of all, you need to know which exact version of program the memory dump was obtained from. You should learn the exact state of source code, and build platform + configuration.

Here are some options:

• TheDarkMod final release: ask the TDM version, like 2.07.
• TheDarkMod beta release: ask when player has updated TDM last time. Or ask him to attach console dump. Or ask him to open game console and look at the revision number there.
• TheDarkMod developer build: ask developer about source code revision, platform and configuration.

Bitness is not important: you will know it immediately when you open the dump.

Get source code

Get the matching version of source code. In the case of TDM game or updater, you most likely need to switch to specific revision on the release branch/tag.

Get debug symbols

This is the hardest part: you need the debug symbols (i.e. the PDB files) for the binaries from which the memory dump was saved.

Here are some options:

• TheDarkMod final release: find PDBs in devel/release/debugging/X.YZ/ in assets SVN, and unpack them.
• TheDarkMod beta release: ask release manager to upload PDBs somewhere (he usually saves them).
• TheDarkMod developer build: ask developer to upload the matching PDB.

The situation is the same for Linux builds as well. All the Linux builds produce debug symbols in a separate file (e.g. thedarkmod.x64.debug), and this file is treated just like PDB.

On Windows, every PDB and DMP file contains checksum of executable, and normally you cannot use PDB for memory dump if the checksums don't match, just like you cannot use a PDB to debug a program if it does not match. Merely rebuilding the project is enough to break PDB compatibility. Not sure if the same problem exists for Linux + GDB.

If you cannot get the original PDB but have the original executable, then you can also try to force-load freshly built PDBs. Here are the steps:

1. Get exactly matching source code, compiler, target, configuration, whatever else.
2. Build new binaries (fresh rebuild advised).
3. Find the created PDB files and use chkmatch -m to substitute checksum inside PDB.

Of course, using mismatched PDB can lead to all sort of issues during debugging.

Action

Open the solution in Visual Studio (instructions written against VS2017).

In menu: "File -> Open -> File...". Choose the memory dump file.

You will see some general info, most of which is useless. Click Debug with Native Only link in the top-right corner.

Most likely you will see this screen about missing PDB. Click on Browse and find xxx.pdb... link. Find the directory where the PDB file is located.

Visual Studio will accept debug symbols immediately when you open the directory containing them. Make sure the PDB files are unpacked.

Now you should see where execution stopped, probably exception code also, You can navigate readable call stack, inspect global and local variables.

Everything you can normally do in Visual Studio when program is paused, you can also do with memory dump.

Happy debugging!

Linux core dumps

In case of Linux core dump, the initial steps are mostly the same:

1. Learn the exact version of TDM.
2. Get the source code of this version.
3. Get debug symbols for this version (named like thedarkmod.x64.debug).
4. Also obtain TDM executable of this version (e.g. thedarkmod.x64).

Put core dump, executable, debug symbols into current directory, then:

1. Run gdb (without any program).
2. Execute (within gdb): file thedarkmod.x64
3. Make sure gdb reads debug symbols successfully.
4. Load core dump like this: core core.27372
5. Now you are ready to see stacktrace: bt

Here is how it looks in action:

You might also need to set source code path if gdb cannot display the source code. I suppose it is done using directory command... (can't test right now).

System libraries issue

In order to display stacktrace, GDB starts at the current stack position and moves towards less nested functions calls. To achieve it, it needs to somehow detect a chunk of stack which corresponds to the current function call.

Under normal circumstances, it is done using the standard structure of "frame pointers". But unfortunately, binaries in modern Linux distros are often built with "omit frame pointers" flag, because this way they run a tiny little bit faster. An alternative way to stack unwinding is to use debug information, which is usually preserved in some capacity even for the system binaries deployed to our machines. In case of TDM engine it resides in a separate file, but system libraries usually contain debug info within itself.

When you analyze a core dump, it might happen that you don't see a complete backtrace, because GDB can't unwind beyond a function from some system library (usually libc or libpthread). In order to unwind through such a library, you need to obtain its binary, exactly as it was on the machine when/where the core dump was recorded. Note that your local binary most likely has different version and does not match! And the core dump does not contain this info either, so you need to ask these binaries from the person who provided the core dump.

Unfortunately, loading these extra libraries into GDB is very painful! Here is a list of GDB commands which I used the only time I had to do it:

# note: all the .so files obtained from user machine must be put into local directory.
#
# most importantly, the following files are necessary:
#   1. libthread_db.so.1 and libpthread.so.0: required for thread debugging.
#   2. other .so files are required if they occur in call stack.
#
# these files must also be renamed exactly as the symlinks
# i.e. libpthread-2.28.so should be renamed to libpthread.so.0

# load executable file
file ./thedarkmod.x64

# force gdb to forget about local system!
# load all .so files using local directory as root
set sysroot .

# drop dump-recorded paths to .so files
# i.e. load ./libpthread.so.0 instead of ./lib/x86_64-linux-gnu/libpthread.so.0
set solib-search-path .
# disable damn security protection
set auto-load safe-path /

# load core dump file
core core.6487

# print stacktrace
bt

You can see more information about that painful experience in the forum thread Mission downloader issue.