Analyze a Memory Dump - Revision history

Stgatilov: Debug symbols exist for Linux builds too.

2025-04-26T21:01:32Z

Debug symbols exist for Linux builds too.

← Older revision		Revision as of 21:01, 26 April 2025
Line 21:		Line 21:
	===== Get debug symbols =====		===== Get debug symbols =====
	This is the hardest part: you need the debug symbols (i.e. the PDB files) for the binaries from which the memory dump was saved.		This is the hardest part: you need the debug symbols (i.e. the PDB files) for the binaries from which the memory dump was saved.

	Every PDB and DMP file contains checksum of executable, and normally you cannot use PDB for memory dump if the checksums don't match, just like you cannot use a PDB to debug a program if it does not match. Merely rebuilding the project is enough to break PDB compatibility.

	Here are some options:		Here are some options:
Line 28:		Line 26:
	* TheDarkMod beta release: ask release manager to upload PDBs somewhere (he usually saves them).		* TheDarkMod beta release: ask release manager to upload PDBs somewhere (he usually saves them).
	* TheDarkMod developer build: ask developer to upload the matching PDB.		* TheDarkMod developer build: ask developer to upload the matching PDB.

			The situation is the same for Linux builds as well.
			All the Linux builds produce debug symbols in a separate file (e.g. <tt>thedarkmod.x64.debug</tt>), and this file is treated just like PDB.

			On Windows, every PDB and DMP file contains checksum of executable, and normally you cannot use PDB for memory dump if the checksums don't match, just like you cannot use a PDB to debug a program if it does not match. Merely rebuilding the project is enough to break PDB compatibility. Not sure if the same problem exists for Linux + GDB.

	If you cannot get the original PDB but have the original executable, then you can also try to force-load freshly built PDBs. Here are the steps:		If you cannot get the original PDB but have the original executable, then you can also try to force-load freshly built PDBs. Here are the steps:
Line 35:		Line 38:

	Of course, using mismatched PDB can lead to all sort of issues during debugging.		Of course, using mismatched PDB can lead to all sort of issues during debugging.


	=== Action ===		=== Action ===

Stgatilov: Added a section about analysing Linux core dumps

2025-04-12T00:44:03Z

Added a section about analysing Linux core dumps

← Older revision		Revision as of 00:44, 12 April 2025
Line 60:		Line 60:

	Happy debugging!		Happy debugging!


			=== Linux core dumps ===

			In case of Linux core dump, the initial steps are mostly the same:
			# Learn the exact version of TDM.
			# Get the source code of this version.
			# Get debug symbols for this version (named like <tt>thedarkmod.x64.debug</tt>).
			# Also obtain TDM executable of this version (e.g. <tt>thedarkmod.x64</tt>).

			Put core dump, executable, debug symbols into current directory, then:
			# Run gdb (without any program).
			# Execute (within gdb): <tt>file thedarkmod.x64</tt>
			# Make sure gdb reads debug symbols successfully.
			# Load core dump like this: <tt>core core.27372</tt>
			# Now you are ready to see stacktrace: <tt>bt</tt>

			Here is how it looks in action:

			[[File:Coredump 3 analyze.png\|border\|1000px]]

			You might also need to set source code path if gdb cannot display the source code.
			I suppose it is done using <tt>directory</tt> command... (can't test right now).

			===== System libraries issue =====

			In order to display stacktrace, GDB starts at the current stack position and moves towards less nested functions calls.
			To achieve it, it needs to somehow detect a chunk of stack which corresponds to the current function call.

			Under normal circumstances, it is done using the standard structure of "frame pointers".
			But unfortunately, binaries in modern Linux distros are often built with "omit frame pointers" flag, because this way they run a tiny little bit faster.
			An alternative way to stack unwinding is to use debug information, which is usually preserved in some capacity even for the system binaries deployed to our machines.
			In case of TDM engine it resides in a separate file, but system libraries usually contain debug info within itself.

			When you analyze a core dump, it might happen that you don't see a complete backtrace, because GDB can't unwind beyond a function from some system library (usually libc or libpthread).
			In order to unwind through such a library, you need to obtain its binary, exactly as it was on the machine when/where the core dump was recorded.
			Note that your local binary most likely has different version and does not match!
			And the core dump does not contain this info either, so you need to ask these binaries from the person who provided the core dump.

			Unfortunately, loading these extra libraries into GDB is very painful!
			Here is a list of GDB commands which I used the only time I had to do it:

			# note: all the .so files obtained from user machine must be put into local directory.
			#
			# most importantly, the following files are necessary:
			# 1. libthread_db.so.1 and libpthread.so.0: required for thread debugging.
			# 2. other .so files are required if they occur in call stack.
			#
			# these files must also be renamed exactly as the symlinks
			# i.e. libpthread-2.28.so should be renamed to libpthread.so.0

			# load executable file
			file ./thedarkmod.x64

			# force gdb to forget about local system!
			# load all .so files using local directory as root
			set sysroot .

			# drop dump-recorded paths to .so files
			# i.e. load ./libpthread.so.0 instead of ./lib/x86_64-linux-gnu/libpthread.so.0
			set solib-search-path .
			# disable damn security protection
			set auto-load safe-path /

			# load core dump file
			core core.6487

			# print stacktrace
			bt

			You can see more information about that painful experience in the [https://forums.thedarkmod.com/index.php?/topic/21060-mission-downloader-issue/page/2/#findComment-465439 forum thread Mission downloader issue].

	[[Category:Coding]]		[[Category:Coding]]

Greebo: Created page with "You received a crash dump and don't know how to analyze it? That's not a big deal, really. Don't get scared by numerous references to WinDbg and CDB in relation to memory dump..."

2021-03-01T15:53:27Z

Created page with "You received a crash dump and don't know how to analyze it? That's not a big deal, really. Don't get scared by numerous references to WinDbg and CDB in relation to memory dump..."

New page

You received a crash dump and don't know how to analyze it?
That's not a big deal, really.
Don't get scared by numerous references to WinDbg and CDB in relation to memory dumps!
You can open dumps directly in Visual Studio debugger.

Before starting, you might want to read [https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/user-mode-dump-files Microsoft docs about memory dumps], so that understand what it is and what is included.

===== Learn version =====
First of all, you need to know which exact version of program the memory dump was obtained from. You should learn the exact state of source code, and build platform + configuration.

Here are some options:
* TheDarkMod final release: ask the TDM version, like 2.07.
* TheDarkMod beta release: ask when player has updated TDM last time. Or ask him to attach console dump. Or ask him to open game console and look at the revision number there.
* TheDarkMod developer build: ask developer about source code revision, platform and configuration.

Bitness is not important: you will know it immediately when you open the dump.

===== Get source code =====
Get the matching version of source code. In the case of TDM game or updater, you most likely need to switch to specific revision on the release branch/tag.

===== Get debug symbols =====
This is the hardest part: you need the debug symbols (i.e. the PDB files) for the binaries from which the memory dump was saved.

Every PDB and DMP file contains checksum of executable, and normally you cannot use PDB for memory dump if the checksums don't match, just like you cannot use a PDB to debug a program if it does not match. Merely rebuilding the project is enough to break PDB compatibility.

Here are some options:
* TheDarkMod final release: find PDBs in <tt>devel/release/debugging/X.YZ/</tt> in assets SVN, and unpack them.
* TheDarkMod beta release: ask release manager to upload PDBs somewhere (he usually saves them).
* TheDarkMod developer build: ask developer to upload the matching PDB.

If you cannot get the original PDB but have the original executable, then you can also try to force-load freshly built PDBs. Here are the steps:
# Get exactly matching source code, compiler, target, configuration, whatever else.
# Build new binaries (fresh rebuild advised).
# Find the created PDB files and use [http://www.debuginfo.com/tools/chkmatch.html chkmatch -m] to substitute checksum inside PDB.

Of course, using mismatched PDB can lead to all sort of issues during debugging.

=== Action ===

Open the solution in Visual Studio (instructions written against VS2017).

In menu: "File -> Open -> File...". Choose the memory dump file.

[[File:crashdump_1_open.png|border|700px]]

You will see some general info, most of which is useless. Click <u>Debug with Native Only</u> link in the top-right corner.

[[File:crashdump_2_pdb.png|border|700px]]

Most likely you will see this screen about missing PDB. Click on <u>Browse and find xxx.pdb...</u> link. Find the directory where the PDB file is located.

Visual Studio will accept debug symbols immediately when you open the directory containing them. Make sure the PDB files are unpacked.

[[File:crashdump_3_result.png|border|700px]]

Now you should see where execution stopped, probably exception code also, You can navigate readable call stack, inspect global and local variables.

Everything you can normally do in Visual Studio when program is paused, you can also do with memory dump.

Happy debugging!

[[Category:Coding]]