A Step-By-Step Guide for Handling Crash of Legacy C++ Application

Rui-Tech — Sat, 04 Apr 2026 18:46:32 +0000

You're on call. Production system crashed. No stack trace, no error message — just a dead process and maybe a core file.

Now what?

Do not be panic! Follow this guide step by step. You'll have a methodical approach to finding the root cause, whether you have debug symbols, a stripped binary, or nothing at all.

Let's go.

Decision Tree – Your Quick Reference Flowchart

Do not get lost -- keep the map in hand. Here is the complete decision map.

Path A: Have a Core File

The first question to ask: has a core file already been generated? Follow Step A1 to find out.

Step A1: Locate the Core File

Normally core files reside in the application's working directory and have names like core*.*. Use the following commands to locate the core file.

# Common locations: 
ls -la core*
ls -la /var/core/
find / -name "core*" -type f 2>/dev/null

# Check core pattern to know where cores go, then search again
cat /proc/sys/kernel/core_pattern

If you found a core file → Continue with Step A2
If not found → Go to Path B

Path B: No Core File Generated

If no core file was generated yet, you need to enable core dumps and rerun the crashed app. Follow Steps B1 to B6 to generate a core file.

Step B1: Check Core Dump Settings

Most production systems disable core file generation. That is why you did not find any core file yet.

# 1. Check ulimit
ulimit -c
# If 0 → core dump disabled

# 2. Check core pattern
cat /proc/sys/kernel/core_pattern
# If starts with '|' → core file was piped elsewhere
# If empty or default → cores go to current directory

Step B2: Enable Core Dumps

# Temporary enable core dump (only valid for the current session)
ulimit -c unlimited

# Permanent enable core dump (requires root access)
# A core file will be generated whenever a crash happens
echo "* soft core unlimited" | sudo tee -a /etc/security/limits.conf
echo "* hard core unlimited" | sudo tee -a /etc/security/limits.conf

# For systemd services (add to service file)
[Service]
LimitCORE=infinity

Pro Tips: A core file can be large (e.g., 10 GB). The safer approach is the temporary option – enable core dumps only for the current session.

Step B3: Set Core Location

After core dumps are enabled, you can define where core files go. The easiest method, which works for 80% of applications, is to run the application from its working directory – the core file will be written there.

# Option 1: Local directory
ulimit -c unlimited
cd /path/to/working/dir
./myapp

# Option 2: Custom location (requires root)
# Core files go to /var/core/ with pattern: core.<binary>.<pid>.<timestamp>
echo "/var/core/core.%e.%p.%t" | sudo tee /proc/sys/kernel/core_pattern

# Make permanent (if the app is a systemd service)
echo "kernel.core_pattern=/var/core/core.%e.%p.%t" | sudo tee -a /etc/sysctl.conf
echo "kernel.core_uses_pid=1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Step B4: Check Permissions

The core directory must be writable by the process user.

# Core directory must be writable for the process user
ls -ld /var/core
# If it doesn't exist or has wrong permissions, create and fix it:
sudo mkdir -p /var/core
sudo chmod 1777 /var/core

# Check who runs the process
ps aux | grep myapp
# Ensure that user can write to the core location

Step B5: Test Core Dump

To verify your settings work, run a small program that crashes intentionally, then locate the core file.

# Test with a crashing program
cat > test_crash.cpp << 'EOF'
int main() {
    int* p = nullptr;
    *p = 42;
    return 0;
}
EOF

g++ -o test_crash test_crash.cpp
ulimit -c unlimited
./test_crash
ls -la core*

If core appears → your settings work
If no core → check system settings from Step B1 and B3 again.

Step B6: Rerun the Crashed App to Produce a Core File

Rerun the crashed application. As expected, it will crash again – but this time, a core file will be generated. Then proceed to Step A2.

Step A2: Identify Which Binary Created the Core

Now you have two things: a core file (named core in the example) and your application binary (myapp in the example). Before debugging, verify that this core file actually came from the application myapp.

# Use file command
file core
# Output: core: ELF 64-bit core file, x86-64, version 1, from './myapp'

# Or use gdb
gdb -c core
(gdb) info files
# Shows: Core was generated by `./myapp'.

Question: Did your application binary create this core file?

If No → Repeat Step A1 or go back to Path B until you locate the correct core file.
If Yes → Continue with Step A3

Step A3: Check for Debug Symbols

To debug the binary, you need debug symbols. First, check whether the current binary already includes them.

# Check if the binary has debug symbols
file ./myapp
# If output says "stripped" → no symbols
# If output says "not stripped" → has symbols

# Or use nm
nm ./myapp 2>&1 | head -10
# If "no symbols" → stripped
# If shows function names → has symbols

Decision Question: Does the binary have debug symbols?

If yes → Continue to Path A4: Debug Immediately
If no → Go to Path A5: Find or Create Symbols

Path A4: Debug with Symbols (Happy Path)

You now have a binary with debug symbols and a core file. Debugging can start immediately using GDB.

# Load core with binary
gdb ./myapp core

# Essential commands
(gdb) bt full          # Full backtrace with locals
(gdb) info threads     # All threads
(gdb) thread apply all bt  # Backtrace all threads
(gdb) frame 0          # Select crash frame
(gdb) info locals      # Local variables
(gdb) print variable   # Print specific variable
(gdb) list             # Show source code

Done! You have the crash location and state.

Path A5: Stripped Binary – Get Symbol File

A stripped binary contains no debug symbols. However, your team may have archived symbols in a separate file. Industry practice is to build with debug symbols, strip the binary, store the symbols separately on your artifact servers, then deploy only the stripped binary to production system.

Every binary has a unique Build ID (a fingerprint). You can use this Build ID to find the matching standalone symbol file.

Step A5.1: Get the Build ID from the Core Dump or Binary

# From core dump
gdb -c core -batch -ex "info files" | grep "Build ID"

# Or from binary
readelf -n ./myapp | grep "Build ID"
# Output example: Build ID: a1b2c3d4e5f67890abcdef1234567890

Step A5.2: Find the Symbol File Using That Build ID

Your symbol files should be stored somewhere (CI artifacts, symbol server, or build archive). Search for the file containing this Build ID:

find /symbols -type f -exec grep -l "Build ID: a1b2c3d4e5f67890abcdef1234567890" {} \;

Better practice: Store symbol files by Build ID path so you don't need to grep:

# If stored as /symbols/a1/b2c3d4e5f67890...
ls /symbols/a1/b2c3d4e5f67890abcdef1234567890

Then continue to Step A6.

Step A6: Debug with Separate Symbols

Now you can debug using the stripped binary, symbol file, and core dump. Suppose the symbol file is named myapp.dbg.

# Method 1: Link binary to symbols (if --add-gnu-debuglink was used)
gdb ./myapp core
# GDB auto-loads myapp.dbg if it's in the same directory

# Method 2: Manual symbol loading
gdb -s myapp.dbg -e ./myapp -c core

# Method 3: Recombine into full binary
eu-unstrip ./myapp myapp.dbg -o myapp.full
gdb ./myapp.full core

Path A7: No Symbol File Found – Reproduce with Debug Symbols

When no matching symbol file exists, rebuild the source code with debug symbols enabled. This will produce a binary with debug symbols.

Add these flags to your build:

-g -O0 -fno-omit-frame-pointer

For GCC/Clang: Add to CFLAGS and CXXFLAGS

export CFLAGS="-g -O0 -fno-omit-frame-pointer"
export CXXFLAGS="-g -O0 -fno-omit-frame-pointer"
make clean && make

For CMake: Use Debug build type

cmake -DCMAKE_BUILD_TYPE=Debug .
make

After that, follow Step B6 – rerun the new binary. If the crash occurs again, capture the core file and debug using Path A4.

If you cannot make a debug build work, or you prefer not to reproduce the crash, you can still try Step A8 or Step A9.

Do you have the unstripped original build?

If yes → Continue with Step A8
If no → Go to Step A9

Step A8: Map Addresses from Stripped Binary

If you have the original build tree (unstripped binary) but the crashed binary in production is stripped, the core dump contains raw addresses that you can map back to source lines using the unstripped binary.

The core dump stores raw addresses (e.g., 0x55a3d8e2f4a0). The unstripped binary tells you:

Which function contains 0xf4a0 (offset from base)
Which source code line that address corresponds to

# 1. Get the crash address from core dump
gdb -c core -batch -ex "info registers" | grep "rip"
# Output: rip            0x55a3d8e2f4a0   0x55a3d8e2f4a0

# 2. Find the memory map to calculate offset
gdb -c core -batch -ex "info proc mappings" | grep "r-x"
# Output: 0x55a3d8e00000 0x55a3d8f00000 r-xp /path/to/myapp
# Base address = 0x55a3d8e00000

# 3. Calculate offset within the binary
offset = crash_address - base_address
# offset = 0x55a3d8e2f4a0 - 0x55a3d8e00000 = 0x2f4a0

# 4. Use addr2line with the UNSTRIPPED binary (from your build)
addr2line -e /path/to/unstripped/myapp -f 0x2f4a0
# Output:
# processRequest
# /src/myapp/src/handler.cpp:127

# 5. Get full backtrace using addresses
gdb -c core -batch -ex "bt" | grep "0x" | awk '{print $NF}' | while read addr; do
    offset=$((addr - base_address))
    addr2line -e /path/to/unstripped/myapp -f $offset
done

Step A9: No Symbols, Can't Reproduce – Fallback Analysis

You have nothing: no symbol file, no original build, and you cannot reproduce the crash with a debug build. This is the "desert island" scenario. You can still extract forensic evidence from the core dump.

# 1. Get crash address and registers
gdb -c core -batch -ex "info registers" > crash_info.txt

# 2. Get backtrace (even without names)
gdb -c core -batch -ex "bt" > backtrace.txt

# 3. Examine instructions near crash
gdb -c core -batch -ex "x/20i \$rip-40" > instructions.txt

# 4. Check system library frames
gdb -c core -batch -ex "bt" 2>&1 | grep -E "libc|libstdc"

What you can learn from raw analysis:

gdb -c core
(gdb) info registers
rax = 0x0              → Null pointer dereference
rip = 0x55...          → Crash in your code (not system library)
rsp = 0x7ff...         → Stack pointer valid

(gdb) x/10i $rip-20
0x55...:  mov    %rax, %rdi
0x55...:  test   %rdi, %rdi
0x55...:  je     0x55...    # Jump if null
0x55...:  mov    (%rdi), %rdx   # CRASH HERE

Deduced: Null pointer dereference in a function that takes a pointer parameter and doesn't check it.

Action: Review code for functions that accept pointers and call them without null checks.

Quick Command Reference

Task	Command
Enable cores	`ulimit -c unlimited`
Find cores	`find / -name "core*" 2>/dev/null`
Check binary for symbols	`file ./myapp`
Get build ID	`readelf -n ./myapp
Extract symbols	{% raw %}`objcopy --only-keep-debug myapp myapp.dbg`
Debug with symbols	`gdb -s myapp.dbg -e myapp -c core`
Map address to line	`addr2line -e myapp -f 0x12345`
List function addresses	`nm ./myapp
Check core pattern	{% raw %}`cat /proc/sys/kernel/core_pattern`
Set core location	`echo "/var/core/core.%e.%p" > /proc/sys/kernel/core_pattern`

Pro Tips

Always compile with -g even for release, then strip separately
Store debug symbols with Build ID as filename for easy retrieval
Use -fno-omit-frame-pointer for reliable backtraces
Test core dump generation in your production environment
Document your symbol server location in runbooks
Set up automatic core collection in production

This workflow covers every scenario from "no core file" to "full debug symbols". Bookmark it for your next production crash.

DEV Community: Rui-Tech