DEV Community: prabhu ponnambalam

Ship Safer Code: The GitHub Actions Patterns That Actually Matter

prabhu ponnambalam — Sun, 15 Mar 2026 22:11:06 +0000

The Problem With Most CI Guides

You search "GitHub Actions unit tests", find a guide, copy the YAML, and it works — until:

A docs-only PR triggers a 20-minute build
Tests pass in CI but crash in production
One flaky test fails randomly and blocks your whole team

These are real problems. Here's how to solve them.

1. Path Filtering — Skip Builds When Nothing Changed

- name: Detect changes
  id: changes
  uses: dorny/paths-filter@v3
  with:
    filters: |
      code:
        - '**/*.cpp'
        - '**/*.h'
        - '**/*.hpp'
        - '**/CMakeLists.txt'

- name: Run Tests
  if: steps.changes.outputs.code == 'true'
  run: make run-tests

Real impact: A typo fix in README.md used to burn 20 minutes of runner time. Now it passes in under 5 seconds.

2. Build and Test Inside Docker

Runner environment drift is the silent killer of reproducibility. The Ubuntu version changes, a system library gets updated, and suddenly your green CI is red for no reason you can explain.

- name: Build Docker image with tests enabled
  run: |
    docker build . -f Dockerfile -t myapp-ci \
      --build-arg WITH_TESTS=true

- name: Run tests inside container
  run: |
    mkdir -p ./test-output
    docker run --rm \
      -v ./test-output:/app/build/Testing/Temporary \
      myapp-ci \
      ctest --test-dir /app/build --output-on-failure

Why mount ./test-output? When tests fail, you need the logs. Without the mount, they disappear when the container exits.

3. Run ASAN and TSAN — Your Code Isn't as Safe as You Think

Most C++ bugs that reach production wouldn't make it past:

AddressSanitizer (ASAN) — buffer overflows, use-after-free, memory leaks
ThreadSanitizer (TSAN) — data races, deadlocks

jobs:
  release:
    uses: ./.github/workflows/test.yml
    with:
      preset: release

  asan:
    uses: ./.github/workflows/test.yml
    with:
      preset: asan

  tsan:
    needs: asan          # stagger — both are CPU-heavy
    uses: ./.github/workflows/test.yml
    with:
      preset: tsan

⚠️ Don't run ASAN and TSAN simultaneously. They compete for the same runner resources and you'll get false failures from starvation, not real bugs. Use needs: to sequence them.

4. Retry Only the Failing Tests — Not the Whole Suite

Flaky tests exist. Denying this costs your team hours per week. The wrong fix is marking tests DISABLED_ or ignoring them. The right fix:

- name: Run Tests
  run: ctest --test-dir build --output-on-failure --parallel $(nproc)

- name: Retry Failed Tests
  if: failure()
  run: |
    ctest --test-dir build \
      --rerun-failed \
      --output-on-failure \
      --repeat until-pass:3

--rerun-failed is a CTest built-in. It reads the last run's results and only executes what failed. A 20-minute full suite retry becomes a 90-second targeted retry.

5. Debug Verbose Output — Without Polluting Normal Runs

Verbose test output in every run is thousands of lines of noise. But when you're debugging at 2am, you need every byte.

- name: Run Tests
  if: runner.debug != '1'
  run: ctest --test-dir build --output-on-failure

- name: Run Tests (verbose)
  if: runner.debug == '1'
  run: ctest --test-dir build --verbose

To enable debug mode: GitHub Actions → Re-run jobs → Enable debug logging. No YAML changes, no new commits. Just flip a switch.

6. Cancel Redundant Runs With One Config Block

You push a fix, realize you forgot something, push again. Now you have two CI runs for the same branch. The first one is waste.

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Two lines at the top of every workflow. The new push kills the old run immediately. Zero cost, zero thought, instant win.

Summary

CI is code. Treat it with the same discipline: no waste, informative failures, easy to maintain.

The Deadlock Detective: GDB Techniques for Solving Thread Crimes in Real-Time

prabhu ponnambalam — Mon, 02 Feb 2026 20:22:33 +0000

Introduction

Ever had your production application freeze mysteriously? No crashes, no errors—just complete silence? You're likely dealing with a deadlock.

Unlike crashes that leave evidence (core dumps, stack traces), deadlocks are silent killers. Your process is alive but completely unresponsive, with threads locked in an eternal embrace.

Today, I'll show you how to use GDB to perform a live autopsy on a deadlocked application and identify the exact lines of code responsible.

💡 Pro Tip: This tutorial assumes basic familiarity with C++ and multithreading. If you're new to threads, check out https://en.cppreference.com/w/cpp/thread.

Deadlock issue

Execute the application in one terminal:

./prabhu_application.out

You'll see this output, then the application will freeze:

Prabhu ~/pgms $ g++ application.cpp -o prabhu_application.out
Prabhu ~/pgms $ ./prabhu_application.out
Creating two threads that will deadlock...                                                                                                                                                   
Thread 1: locks mutex1 -> mutex2                                                                                                                                                             
Thread 2: locks mutex2 -> mutex1                                                                                                                                                             
===============================                                                                                                                                                                                                                                                                                                                                                          
Main: Waiting for threads to complete...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
Thread 1: Attempting to lock mutex1...                                                                                                                                                       
Thread 1: Locked mutex1                                                                                                                                                                      
Thread 2: Attempting to lock mutex2...                                                                                                                                                       
Thread 2: Locked mutex2                                                                                                                                                                      
Thread 1: Attempting to lock mutex2...                                                                                                                                                       
Thread 2: Attempting to lock mutex1...

← HANGS HERE FOREVER

Notice the last two lines—both threads are trying to acquire their second lock. Neither will succeed. The process is now deadlocked. Time to investigate! 🕵️

The Investigation Begins: Attaching GDB

Open a second terminal while leaving the frozen process running.

Find the process ID:

pgrep -a application

This returns something like:

12345 ./application

The number 12345 is your Process ID (PID).

Alternative method:
You can also use ps aux | grep application | grep -v grep

Attach GDB to the running process:

sudo gdb -p 12345

GDB will display something like:

Attaching to process 12345                                                                                                                                                                   
[New LWP 12346]                                                                                                                                                                              
[New LWP 12347]                                                                                                                                                                              
[Thread debugging using libthread_db enabled]                                                                                                                                                
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".                                                                                                                   
0x00007f8a1bef2f3d in __GI___libc_read () from /lib/x86_64-linux-gnu/libc.so.6                                                                                                               
(gdb)

The (gdb) prompt means we're in control. Let's investigate!

Finding the Deadlock: Thread Analysis

Quick Thread Overview:

  (gdb) info threads                                                                                                                                                                           

  Output:                                                                                                                                                                                      

    Id   Target Id                                    Frame                                                                                                                                    
  * 1    Thread 0x7f8a1c2a3740 (LWP 12345) "deadlock" 0x00007f8a1bef2f3d in __GI___libc_read (...)                                                                                             
    2    Thread 0x7f8a1ba92700 (LWP 12346) "deadlock" 0x00007f8a1bee3a4e in __lll_lock_wait ()                                                                                                 
    3    Thread 0x7f8a1b291700 (LWP 12347) "deadlock" 0x00007f8a1bee3a4e in __lll_lock_wait ()                                                                                                 

  🚨 Threads 2 and 3 are both in __lll_lock_wait() - this is the low-level mutex wait. When multiple threads are here, you likely have a deadlock.

Investigating Thread 2

Switch to Thread 2 and get its backtrace:

  (gdb) thread 2                                                                                                                                                                               
  (gdb) bt                                                                                                                                                                                     

  Output:                                                                                                                                                                                      

  #0  0x00007f8a1bee3a4e in __lll_lock_wait ()                                                                                                                                                 
  #1  0x00007f8a1bedc2a3 in __GI___pthread_mutex_lock (mutex=0x555555558060 <mutex2>)                                                                                                          
  #2  0x00005555555554b2 in __gthread_mutex_lock (__mutex=0x555555558060 <mutex2>)                                                                                                             
  #3  0x0000555555555687 in std::mutex::lock (this=0x555555558060 <mutex2>)                                                                                                                    
  #4  0x00005555555552f1 in thread1_function () at application.cpp:21                                                                                                                        
  #5  0x0000555555555a7e in std::__invoke_impl<void, void (*)()>

The Critical Information

Frame #4 shows our code: thread1_function() at line 21

Frame #1 shows it's waiting for mutex2 at address 0x555555558060

Jump to the source:

  (gdb) frame 4                                                                                                                                                                                
  (gdb) list                                                                                                                                                                                   

  Output:                                                                                                                                                                                      

  16      std::cout << "Thread 1: Locked mutex1" << std::endl;                                                                                                                                                                                                                                                                                                                     
  18      std::this_thread::sleep_for(std::chrono::milliseconds(100));                                                                                                                         
  19                                                                                                                                                                                           
  20      std::cout << "Thread 1: Attempting to lock mutex2..." << std::endl;                                                                                                                  
  21      mutex2.lock();  ← 🔴 STUCK HERE                                                                                                                                                      
  22      std::cout << "Thread 1: Locked mutex2" << std::endl;

Look at what it already holds:

  (gdb) list 10,20                                                                                                                                                                             

  14      mutex1.lock();  ← ✅ THIS SUCCEEDED                                                                                                                                                  
  15      std::cout << "Thread 1: Locked mutex1" << std::endl;                                                                                                                                                                                                                                                                                                                          
  21      mutex2.lock();  ← 🔴 BLOCKED HERE

Thread 2 Summary:

Holds: mutex1 (line 14)

Waiting for: mutex2 (line 21)

Blocked at: application.cpp:21

Investigating Thread 3

Switch to Thread 3:

(gdb) thread 3

(gdb) bt

Output:

  #0  0x00007f8a1bee3a4e in __lll_lock_wait ()                                                                                                                                                 
  #1  0x00007f8a1bedc2a3 in __GI___pthread_mutex_lock (mutex=0x555555558040 <mutex1>)                                                                                                          
  #2  0x00005555555554b2 in __gthread_mutex_lock (__mutex=0x555555558040 <mutex1>)                                                                                                             
  #3  0x0000555555555687 in std::mutex::lock (this=0x555555558040 <mutex1>)                                                                                                                    
  #4  0x000055555555535f in thread2_function () at application.cpp:44

Frame #4 shows: thread2_function() at line 44

Frame #1 shows it's waiting for mutex1 at address 0x555555558040

Check the source:

  (gdb) frame 4                                                                                                                                                                                
  (gdb) list                                                                                                                                                                                   

  Output:                                                                                                                                                                                      

  39      std::cout << "Thread 2: Locked mutex2" << std::endl;                                                                                                                                 
  40                                                                                                                                                                                           
  41      std::this_thread::sleep_for(std::chrono::milliseconds(100));                                                                                                                         
  42                                                                                                                                                                                           
  43      std::cout << "Thread 2: Attempting to lock mutex1..." << std::endl;                                                                                                                  
  44      mutex1.lock();  ← 🔴 STUCK HERE                                                                                                                                                      
  45      std::cout << "Thread 2: Locked mutex1" << std::endl;

What it already holds:

(gdb) list 33,44                                                                                                                                                                             

  37      mutex2.lock();  ← ✅ THIS SUCCEEDED                                                                                                                                                  
  38      std::cout << "Thread 2: Locked mutex2" << std::endl;                                                                                                                                 
  ...                                                                                                                                                                                          
  44      mutex1.lock();  ← 🔴 BLOCKED HERE

Thread 3 Summary:

Holds: mutex2 (line 37)

Waiting for: mutex1 (line 44)

Blocked at: application.cpp:44

The Smoking Gun: Circular Dependency

Let's lay out all the evidence we've collected:

Thread 1's Situation:

Location: application.cpp:21

Status: ✅ Successfully holds mutex1

Problem: ❌ Waiting for mutex2 (which Thread 2 has locked)

Can't proceed until: Thread 2 releases mutex2

Thread 2's Situation:

Location: application.cpp:44

Status: ✅ Successfully holds mutex2

Problem: ❌ Waiting for mutex1 (which Thread 1 has locked)

Can't proceed until: Thread 1 releases mutex1

The Deadlock Cycle:

Thread 1 needs mutex2 (held by Thread 2)

↓

Thread 2 needs mutex1 (held by Thread 1)

↓

Thread 1 needs mutex2 (held by Thread 2)

↓

[Infinite loop - DEADLOCK!]

Or visualized as a circle:

Thread 1 → waiting for → mutex2

↓

held by

↓

Thread 2 → waiting for → mutex1

↓

held by

↓

Thread 1

[cycle complete!]

Prevention: Never Let This Happen Again

Solution 1: Global Lock Ordering

The rule: Always acquire locks in the same order across all threads.

Fixed code:

 // Both threads now use the same order                                                                                                                                                       
void thread1_function()                                                                                                                                                                      
{                                                                                                                                                                                            
  mutex1.lock();  // First                                                                                                                                                                   
  mutex2.lock();  // Second                                                                                                                                                                  
  // ... critical section ...                                                                                                                                                                
  mutex2.unlock();                                                                                                                                                                           
  mutex1.unlock();                                                                                                                                                                           
}                                                                                                                                                                                            

void thread2_function()                                                                                                                                                                      
{                                                                                                                                                                                            
   mutex1.lock();  // First (SAME ORDER AS THREAD 1)                                                                                                                                          
   mutex2.lock();  // Second (SAME ORDER AS THREAD 1)                                                                                                                                                                                                                                                                                                                
   mutex2.unlock();
   mutex1.unlock();   

}

Why this works: If everyone acquires resources in the same sequence, circular dependencies cannot form. No matter how threads interleave, one will successfully acquire both locks, complete
its work, release them, and then the other can proceed.

Solution 2: std::scoped_lock (Recommended for C++17+)

The modern, foolproof approach:

   #include <mutex>                                                                                                                                                                             

   void thread1_function()                                                                                                                                                                      
   {                                                                                                                                                                                            
     std::scoped_lock lock(mutex1, mutex2);                                                                                                                                                     
     // Critical section                                                                                                                                                                        
     std::cout << "Thread 1: In critical section" << std::endl;                                                                                                                                 
     // Locks automatically released in correct order                                                                                                                                           
   }                                                                                                                                                                                            


   void thread2_function()                                                                                                                                                                     
   {                                                                                                                                                                                            
     std::scoped_lock lock(mutex1, mutex2);  // Same call - always safe!                                                                                                                        
     // Critical section                                                                                                                                                                        
     std::cout << "Thread 2: In critical section" << std::endl;                                                                                                                                 
     // Locks automatically released in correct order                                                                                                                                           
   }

How std::scoped_lock works its magic:

Step 1: Takes multiple mutexes as arguments

Step 2: Internally sorts them by memory address

Step 3: Always acquires them in address order (consistent across all threads)

Step 4: Uses RAII to guarantee release in reverse order

Step 5: Exception-safe—releases locks even if your code throws

Even if you pass the locks in different orders (scoped_lock(mutex2, mutex1) vs scoped_lock(mutex1, mutex2)), it normalizes them internally. Deadlock impossible!

Core Commands for Deadlock Hunting

Attach to a running process:

gdb -p

List all threads:

info threads

Switch to a specific thread:

thread 2

Get backtrace (call stack):

bt

Get backtrace with local variables:

bt full

Get ALL thread backtraces (best for deadlock):

thread apply all bt

Jump to a specific stack frame:

frame 4

Navigate stack frames:

up # Move toward caller

down # Move toward callee

Show source code:

list

list 10,20 # Show lines 10-20

Inspect variables:

print variable_name

print mutex1

info locals

info args

Watch for changes:

watch variable_name

Detach without killing process:

detach

quit

Quick Deadlock Check (One Command)

thread apply all bt | grep -A 5 "lll_lock_wait"

This shows all threads stuck in lock waits with context.

Generate GDB report

gdb -p PID -batch -ex 'thread apply all bt' -ex detach > gdb_bt.txt

This generates a complete report automatically without manual intervention—perfect for production debugging!

Conclusion

We've completed a full deadlock investigation:

✅ Created a deadlock scenario

✅ Attached GDB to the frozen process

✅ Identified threads in wait states

✅ Analyzed backtraces to find lock locations

✅ Discovered the circular dependency

✅ Identified the root cause (lock ordering violation)

✅ Implemented multiple prevention strategies

Resources

GDB Documentation: https://sourceware.org/gdb/documentation/

C++ Concurrency in Action by Anthony Williams - The definitive guide to multithreading in C++

ThreadSanitizer Manual: https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual

CPP Reference - Thread Support: https://en.cppreference.com/w/cpp/thread