Introduction
Ever had your production application freeze mysteriously? No crashes, no errors—just complete silence? You're likely dealing with a deadlock.
Unlike crashes that leave evidence (core dumps, stack traces), deadlocks are silent killers. Your process is alive but completely unresponsive, with threads locked in an eternal embrace.
Today, I'll show you how to use GDB to perform a live autopsy on a deadlocked application and identify the exact lines of code responsible.
💡 Pro Tip: This tutorial assumes basic familiarity with C++ and multithreading. If you're new to threads, check out https://en.cppreference.com/w/cpp/thread.
Deadlock issue
Execute the application in one terminal:
./application
You'll see this output, then the application will freeze:
Creating two threads that will deadlock...
Thread 1: locks mutex1 -> mutex2
Thread 2: locks mutex2 -> mutex1
===============================
Main: Waiting for threads to complete...
Thread 1: Attempting to lock mutex1...
Thread 1: Locked mutex1
Thread 2: Attempting to lock mutex2...
Thread 2: Locked mutex2
Thread 1: Attempting to lock mutex2...
Thread 2: Attempting to lock mutex1...
← HANGS HERE FOREVER
Notice the last two lines—both threads are trying to acquire their second lock. Neither will succeed. The process is now deadlocked. Time to investigate! 🕵️
The Investigation Begins: Attaching GDB
Open a second terminal while leaving the frozen process running.
Find the process ID:
pgrep -a application
This returns something like:
12345 ./application
The number 12345 is your Process ID (PID).
Alternative method:
You can also use ps aux | grep application | grep -v grep
Attach GDB to the running process:
sudo gdb -p 12345
GDB will display something like:
Attaching to process 12345
[New LWP 12346]
[New LWP 12347]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f8a1bef2f3d in __GI___libc_read () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)
The (gdb) prompt means we're in control. Let's investigate!
Finding the Deadlock: Thread Analysis
Quick Thread Overview:
(gdb) info threads
Output:
Id Target Id Frame
* 1 Thread 0x7f8a1c2a3740 (LWP 12345) "deadlock" 0x00007f8a1bef2f3d in __GI___libc_read (...)
2 Thread 0x7f8a1ba92700 (LWP 12346) "deadlock" 0x00007f8a1bee3a4e in __lll_lock_wait ()
3 Thread 0x7f8a1b291700 (LWP 12347) "deadlock" 0x00007f8a1bee3a4e in __lll_lock_wait ()
🚨 Threads 2 and 3 are both in __lll_lock_wait() - this is the low-level mutex wait. When multiple threads are here, you likely have a deadlock.
Investigating Thread 2
Switch to Thread 2 and get its backtrace:
(gdb) thread 2
(gdb) bt
Output:
#0 0x00007f8a1bee3a4e in __lll_lock_wait ()
#1 0x00007f8a1bedc2a3 in __GI___pthread_mutex_lock (mutex=0x555555558060 <mutex2>)
#2 0x00005555555554b2 in __gthread_mutex_lock (__mutex=0x555555558060 <mutex2>)
#3 0x0000555555555687 in std::mutex::lock (this=0x555555558060 <mutex2>)
#4 0x00005555555552f1 in thread1_function () at application.cpp:21
#5 0x0000555555555a7e in std::__invoke_impl<void, void (*)()>
The Critical Information
Frame #4 shows our code: thread1_function() at line 21
Frame #1 shows it's waiting for mutex2 at address 0x555555558060
Jump to the source:
(gdb) frame 4
(gdb) list
Output:
16 std::cout << "Thread 1: Locked mutex1" << std::endl;
18 std::this_thread::sleep_for(std::chrono::milliseconds(100));
19
20 std::cout << "Thread 1: Attempting to lock mutex2..." << std::endl;
21 mutex2.lock(); ← 🔴 STUCK HERE
22 std::cout << "Thread 1: Locked mutex2" << std::endl;
Look at what it already holds:
(gdb) list 10,20
14 mutex1.lock(); ← ✅ THIS SUCCEEDED
15 std::cout << "Thread 1: Locked mutex1" << std::endl;
21 mutex2.lock(); ← 🔴 BLOCKED HERE
Thread 2 Summary:
Holds: mutex1 (line 14)
Waiting for: mutex2 (line 21)
Blocked at: application.cpp:21
Investigating Thread 3
Switch to Thread 3:
(gdb) thread 3
(gdb) bt
Output:
#0 0x00007f8a1bee3a4e in __lll_lock_wait ()
#1 0x00007f8a1bedc2a3 in __GI___pthread_mutex_lock (mutex=0x555555558040 <mutex1>)
#2 0x00005555555554b2 in __gthread_mutex_lock (__mutex=0x555555558040 <mutex1>)
#3 0x0000555555555687 in std::mutex::lock (this=0x555555558040 <mutex1>)
#4 0x000055555555535f in thread2_function () at application.cpp:44
Frame #4 shows: thread2_function() at line 44
Frame #1 shows it's waiting for mutex1 at address 0x555555558040
Check the source:
(gdb) frame 4
(gdb) list
Output:
39 std::cout << "Thread 2: Locked mutex2" << std::endl;
40
41 std::this_thread::sleep_for(std::chrono::milliseconds(100));
42
43 std::cout << "Thread 2: Attempting to lock mutex1..." << std::endl;
44 mutex1.lock(); ← 🔴 STUCK HERE
45 std::cout << "Thread 2: Locked mutex1" << std::endl;
What it already holds:
(gdb) list 33,44
37 mutex2.lock(); ← ✅ THIS SUCCEEDED
38 std::cout << "Thread 2: Locked mutex2" << std::endl;
...
44 mutex1.lock(); ← 🔴 BLOCKED HERE
Thread 3 Summary:
Holds: mutex2 (line 37)
Waiting for: mutex1 (line 44)
Blocked at: application.cpp:44
The Smoking Gun: Circular Dependency
Let's lay out all the evidence we've collected:
Thread 1's Situation:
Location: application.cpp:21
Status: ✅ Successfully holds mutex1
Problem: ❌ Waiting for mutex2 (which Thread 2 has locked)
Can't proceed until: Thread 2 releases mutex2
Thread 2's Situation:
Location: application.cpp:44
Status: ✅ Successfully holds mutex2
Problem: ❌ Waiting for mutex1 (which Thread 1 has locked)
Can't proceed until: Thread 1 releases mutex1
The Deadlock Cycle:
Thread 1 needs mutex2 (held by Thread 2)
↓
Thread 2 needs mutex1 (held by Thread 1)
↓
Thread 1 needs mutex2 (held by Thread 2)
↓
[Infinite loop - DEADLOCK!]
Or visualized as a circle:
Thread 1 → waiting for → mutex2
↓
held by
↓
Thread 2 → waiting for → mutex1
↓
held by
↓
Thread 1
[cycle complete!]
Prevention: Never Let This Happen Again
Solution 1: Global Lock Ordering
The rule: Always acquire locks in the same order across all threads.
Fixed code:
// Both threads now use the same order
void thread1_function()
{
mutex1.lock(); // First
mutex2.lock(); // Second
// ... critical section ...
mutex2.unlock();
mutex1.unlock();
}
void thread2_function()
{
mutex1.lock(); // First (SAME ORDER AS THREAD 1)
mutex2.lock(); // Second (SAME ORDER AS THREAD 1)
mutex2.unlock();
mutex1.unlock();
}
Why this works: If everyone acquires resources in the same sequence, circular dependencies cannot form. No matter how threads interleave, one will successfully acquire both locks, complete
its work, release them, and then the other can proceed.
Solution 2: std::scoped_lock (Recommended for C++17+)
The modern, foolproof approach:
#include <mutex>
void thread1_function()
{
std::scoped_lock lock(mutex1, mutex2);
// Critical section
std::cout << "Thread 1: In critical section" << std::endl;
// Locks automatically released in correct order
}
void thread2_function()
{
std::scoped_lock lock(mutex1, mutex2); // Same call - always safe!
// Critical section
std::cout << "Thread 2: In critical section" << std::endl;
// Locks automatically released in correct order
}
How std::scoped_lock works its magic:
Step 1: Takes multiple mutexes as arguments
Step 2: Internally sorts them by memory address
Step 3: Always acquires them in address order (consistent across all threads)
Step 4: Uses RAII to guarantee release in reverse order
Step 5: Exception-safe—releases locks even if your code throws
Even if you pass the locks in different orders (scoped_lock(mutex2, mutex1) vs scoped_lock(mutex1, mutex2)), it normalizes them internally. Deadlock impossible!
Core Commands for Deadlock Hunting
Attach to a running process:
gdb -p
List all threads:
info threads
Switch to a specific thread:
thread 2
Get backtrace (call stack):
bt
Get backtrace with local variables:
bt full
Get ALL thread backtraces (best for deadlock):
thread apply all bt
Jump to a specific stack frame:
frame 4
Navigate stack frames:
up # Move toward caller
down # Move toward callee
Show source code:
list
list 10,20 # Show lines 10-20
Inspect variables:
print variable_name
print mutex1
info locals
info args
Watch for changes:
watch variable_name
Detach without killing process:
detach
quit
Quick Deadlock Check (One Command)
thread apply all bt | grep -A 5 "lll_lock_wait"
This shows all threads stuck in lock waits with context.
Generate GDB report
gdb -p PID -batch -ex 'thread apply all bt' -ex detach > gdb_bt.txt
This generates a complete report automatically without manual intervention—perfect for production debugging!
Conclusion
We've completed a full deadlock investigation:
✅ Created a deadlock scenario
✅ Attached GDB to the frozen process
✅ Identified threads in wait states
✅ Analyzed backtraces to find lock locations
✅ Discovered the circular dependency
✅ Identified the root cause (lock ordering violation)
✅ Implemented multiple prevention strategies
Resources
GDB Documentation: https://sourceware.org/gdb/documentation/
C++ Concurrency in Action by Anthony Williams - The definitive guide to multithreading in C++
ThreadSanitizer Manual: https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
CPP Reference - Thread Support: https://en.cppreference.com/w/cpp/thread
Top comments (0)