GDB Explained: Understanding How to Debug Applications in HPC

When you write or run an application, things do not always go as expected.

Sometimes the program crashes. Sometimes it produces incorrect results. Sometimes it runs for hours without finishing, even though there appears to be no obvious error.

In these situations, how do you find out what actually went wrong?

One common approach is to add printf() statements throughout the code to display variable values and follow the execution. This can help, but it has limitations. You need to decide in advance what information to print, modify the source code and recompile the application every time you want to investigate something new.

There is a more powerful approach: using a debugger.

One of the most widely used debugging tools in Linux and software development is GDB, the GNU Debugger. In HPC environments, you may also encounter ROCgdb, AMD's version of GDB with additional capabilities for debugging GPU applications.

Let's understand how these tools work and why they are useful.

1. What exactly is GDB?

GDB stands for GNU Debugger.

Think of GDB as a tool that allows you to pause a running application, look inside it and understand what it is doing.

Normally, when you execute a program, it runs from beginning to end without waiting for you.

With GDB, you can take control of that execution.

You can:

1. Pause the application at a particular line of code.
2. Check the values stored in variables.
3. Execute the program one line at a time.
4. Follow which functions called other functions.
5. Investigate exactly where an application crashed.
6. Change certain values or execution behaviour during debugging.

Imagine watching a recorded football match. Normally, you watch the entire match from beginning to end. But with a debugger, you can pause at any moment, go back to a particular point and examine what happened.

That is essentially what GDB allows you to do with a running program.

2. Why is debugging important in HPC?

In HPC, applications are often much more complicated than a simple standalone program.

An application might use multiple MPI processes, CPU threads and GPUs. It might run across several compute nodes and execute for hours or even days.

Consider a scientific simulation that runs across 32 compute nodes.

After several hours, the application crashes.

Without a debugger, you might only see an error message in the job output. You know something went wrong, but you may not know:

• Which function caused the problem?
• Which variable contained an unexpected value?
• Where exactly did the application fail?
• Was the problem caused by an invalid memory access?
• Did the application receive a signal such as SIGSEGV?

A debugger helps investigate these questions by showing what was happening inside the application when the problem occurred.

This is particularly useful when working with C, C++ and Fortran applications commonly found in scientific computing.

3. Before using GDB, prepare your application

There is an important concept to understand before starting a debugger.

When you compile a program, the compiler normally converts your source code into machine instructions that the processor can execute.

However, the compiled application does not automatically retain all the information needed to make debugging easy.

For example, when you look at a running program, you want to understand that the application is currently executing line 25 of simulation.c, and that the variable temperature contains the value 45.

To make this possible, you need to compile the application with debugging information.

Consider a simple C program.

#include <stdio.h>

int main()
{
    int a = 10;
    int b = 20;

    int result = a + b;

    printf("Result = %d\n", result);

    return 0;
}

You can compile it with:

gcc -ggdb -O0 program.c -o program

Two important compiler options are being used here.

-ggdb

This tells the compiler to generate debugging information in a format suitable for GDB.

It helps the debugger understand source code, variable names, functions and line numbers.

-O0

This disables compiler optimisation.

Normally, compilers optimise applications to make them run faster. During optimisation, the compiler might rearrange instructions, remove unnecessary variables or combine calculations.

This can make debugging confusing because the actual machine instructions may no longer correspond directly to the source code.

For beginners, -O0 makes it easier to follow the program line by line.

You might also encounter -g, which generates debugging information in the platform's default format. Particularly for ROCgdb, -ggdb is usually the recommendation, if available.

One important point: optimisation can sometimes expose bugs that do not appear in an unoptimised build. If a problem only happens with -O2 or -O3, you may need to reproduce it using those optimisation settings.

4. Starting GDB and running your application

Once your application has been compiled with debugging information, you can start GDB.

gdb ./program

You will see a debugger prompt similar to:

(gdb)

This is where you enter debugging commands.

Let's look at three useful commands.

run

(gdb) run

This starts your application and allows it to execute normally.

It continues until the application finishes, crashes or reaches a breakpoint.

start

(gdb) start

This starts the application but pauses at the beginning of the main() function.

This is particularly useful for beginners because you can start investigating from the first line of your own application.

continue

(gdb) continue

This tells the debugger to resume execution from wherever the program is currently paused.

Remember the difference:

run starts the application again from the beginning.

continue allows the current execution to carry on.

5. What is a breakpoint?

A breakpoint is one of the most important concepts in debugging.

Imagine your application contains 500 lines of code. You suspect that something goes wrong around line 150.

Instead of watching all 500 lines execute, you can tell GDB to stop when it reaches line 150.

That stopping point is called a breakpoint.

For example:

(gdb) break program.c:15

This tells GDB to stop when execution reaches line 15 of program.c.

You can also set a breakpoint at the beginning of a function:

(gdb) break main

Or:

(gdb) break calculate

The second command tells GDB to stop when the calculate() function is reached.

You can then start the application:

(gdb) run

When the program reaches your breakpoint, execution pauses and GDB gives you control.

Why is this useful?

Imagine a simulation has a function that calculates pressure. You suspect the function receives an incorrect value.

You can set a breakpoint at that function and inspect its input values before it starts calculating.

You do not need to watch the entire application. You can focus on the part you are interested in.

You can also use temporary breakpoints, which are automatically removed after they are reached once.

(gdb) tbreak calculate

This is useful when you only want to stop at a particular location once.

6. Stepping through your code

Now that you understand breakpoints, let's look at another important concept: stepping.

Stepping means executing your application one source line at a time.

Imagine the following program:

#include <stdio.h>

int add(int a, int b)
{
    return a + b;
}

int main()
{
    int x = 10;
    int y = 20;

    int result = add(x, y);

    printf("Result = %d\n", result);

    return 0;
}

Suppose you start GDB and execute:

(gdb) start

The application stops at the beginning of main().

You can now move through the code using different commands.

next

(gdb) next

This executes the current line and moves to the next line.

If the current line calls another function, GDB executes that function completely and stops at the next line in the current function.

Think of it as saying: execute this line, but do not take me inside another function.

step

(gdb) step

This also executes the current line, but if that line calls another function, GDB enters that function.

For example, when execution reaches:

int result = add(x, y);

Using next executes add() and moves to the following line in main().

Using step takes you inside add(), allowing you to inspect how the addition is performed.

This distinction is extremely important when debugging applications containing many functions.

finish

(gdb) finish

Suppose you have entered the add() function and want to return to main().

Instead of stepping through every remaining line, you can use finish.

GDB executes the rest of the current function and stops when it returns to the calling function. It also displays the returned value.

until

(gdb) until

This is useful when debugging loops.

Imagine a loop runs 1,000 times. Stepping through every iteration would be tedious.

The until command can run the loop out to the line after it, rather than making you manually step through every iteration.

Together, next, step, finish and until give you control over how you explore your application's execution.

7. Inspecting variables: What is happening inside the application?

One of the most useful things about GDB is that it allows you to inspect variables while the application is paused.

Consider this example:

int a = 10;
int b = 20;

int result = a + b;

Suppose you stop at the line where result is calculated.

You can ask GDB to display the value of a variable:

(gdb) print a
$1 = 10

Similarly:

(gdb) print b
$2 = 20

You can also inspect the result:

(gdb) print result

The print command allows you to examine variables and expressions without adding extra printf() statements to your source code.

This becomes particularly useful when debugging scientific applications.

Imagine a simulation calculates the temperature of thousands of points. You suspect one of the values is incorrect.

You can pause execution and inspect the relevant variables to understand what values the application is actually using.

Other useful commands include:

(gdb) info locals

Displays local variables in the current function.

(gdb) info args

Displays the arguments passed to the current function.

You can also use:

(gdb) display a

This tells GDB to automatically display the value of a whenever execution stops.

For example, when stepping through a loop, this helps you see how a variable changes without repeatedly typing print a.

8. Understanding the call stack

This is another important concept, especially when investigating application crashes.

A call stack shows the sequence of functions that brought the application to its current execution point.

Imagine your application has three functions:

main()
   |
   v
calculate()
   |
   v
divide()

The main() function calls calculate(), and calculate() calls divide().

Now imagine the application crashes inside divide().

How do you know how it reached that function?

This is where the call stack becomes useful.

You can use:

(gdb) backtrace

Or its shorter form:

(gdb) bt

GDB displays the chain of function calls.

Think of it as a record of the application's journey.

The function at the top of the backtrace is where execution currently stopped. The functions below it show how the application reached that point.

You can also select a particular frame:

(gdb) frame 1

And inspect its variables:

(gdb) info locals

This is particularly useful when a crash occurs deep inside a function and you want to understand which function called it and what values were passed.

9. What happens when an application crashes?

One common problem in C and C++ applications is a segmentation fault.

A segmentation fault, commonly shown as SIGSEGV, occurs when an application attempts an invalid memory access.

For example, consider this code:

#include <stdio.h>

int main()
{
    int *p = NULL;

    printf("%d\n", *p);

    return 0;
}

Here, p is a pointer that contains a NULL value.

The application attempts to access the memory location pointed to by p, but it does not point to a valid object.

As a result, the program may crash with a segmentation fault.

If you run this application through GDB, the debugger can stop at the point where the error occurs.

You might see:

Program received signal SIGSEGV,
Segmentation fault.

You can then run:

(gdb) bt

To understand the function call sequence, and:

(gdb) print p

To inspect the pointer.

In this example, GDB can show that p is NULL.

This is much more useful than simply seeing that the application terminated with a segmentation fault.

You can also tell GDB to stop when a particular signal occurs:

(gdb) catch signal SIGSEGV

This is called a catchpoint. Unlike a normal breakpoint, which stops at a particular source code location, a catchpoint stops when a particular event occurs.

10. Watchpoints: Finding who changed a variable

Sometimes the problem is not that a program crashes. Instead, a variable unexpectedly changes its value.

Imagine a simulation contains a variable called temperature.

You expect it to remain at 100, but somewhere in a large application, another part of the code changes it to 500.

The challenge is finding exactly where that change happens.

You could search through hundreds or thousands of lines of code.

Or you could use a watchpoint.

(gdb) watch temperature

A watchpoint tells GDB to stop when the value of temperature changes.

This is extremely useful when you know which variable is incorrect but do not know which part of the application is modifying it.

There are also other forms:

rwatch stops when an expression is read.

awatch stops when an expression is read or written.

For example:

(gdb) awatch array[2]

This can help investigate unexpected access to a particular array element.

One practical limitation is that hardware watchpoints are limited in number and size. Software watchpoints may be much slower because the debugger has to monitor execution more frequently.

11. GDB versus ROCgdb: What is the difference?

Now let's connect these concepts to GPU computing.

Traditional GDB is designed primarily for debugging CPU applications.

But modern HPC applications increasingly use GPUs for computational workloads.

For example, an application might use the CPU to prepare data and then offload large calculations to an AMD GPU.

This creates a challenge: how do you debug both the CPU and GPU portions of the same application?

This is where ROCgdb comes in.

ROCgdb is AMD's fork of the GNU Debugger. It builds on GDB and adds GPU specific debugging capabilities.

The important point is that many familiar GDB commands remain available.

You can use the same general debugging approach to inspect the CPU side of an application, while ROCgdb adds facilities for investigating the GPU side.

For example, a GPU application might encounter an illegal memory access inside a GPU kernel.

With GPU debugging support, you can investigate the kernel execution and inspect GPU threads rather than looking only at the CPU process.

This is especially relevant when working with AMD GPUs and the ROCm software stack.

12. Why is debugging in HPC different?

There is one final concept worth understanding.

In a normal application, you might run the program directly from a terminal and interact with GDB whenever you want.

HPC applications are often submitted through a scheduler such as Slurm.

For example:

sbatch simulation.sh

Once submitted, the application runs inside a batch job on a compute node. You might not have an interactive terminal attached to it.

This makes debugging more challenging.

Instead of manually entering commands at the GDB prompt, you can prepare a script containing the debugging commands and let the debugger execute them automatically.

For example:

rocgdb -batch \
  -ex "set pagination off" \
  -ex "break main" \
  -ex "run" \
  -ex "bt" \
  ./program

Here is what happens:

• -batch tells ROCgdb to execute the supplied commands and exit.
• -ex supplies a debugger command.
• set pagination off prevents output from stopping at screenfuls.
• break main sets a breakpoint.
• run starts the application.
• bt displays the call stack.

This is useful for automated debugging, testing and HPC environments where interactive sessions are not always practical.

In MPI applications, attaching a debugger to just one rank can also be useful, but it has limitations. Pausing one process can affect timing, hide or introduce race conditions, or cause other processes to wait at an MPI communication point.

This is why debugging parallel applications often requires tools and approaches designed for MPI.

13. The important commands to remember

You do not need to memorise every GDB command when you are starting out. Focus on understanding what the common commands allow you to do.

• run starts the application.
• start starts and pauses at main().
• break sets a breakpoint.
• continue resumes execution.
• next executes the next source line without entering a function.
• step enters a function.
• finish runs until the current function returns.
• until runs out of a loop.
• print displays a variable or expression.
• info locals displays local variables.
• info args displays function arguments.
• backtrace or bt displays the call stack.
• watch stops when a variable changes.
• catch stops when a particular event occurs.

Conclusion

Debugging is not simply about finding the line where an application crashes. It is about understanding what the application was doing, what data it was using and how it reached the point where something went wrong.

GDB provides a way to pause execution, inspect variables, follow function calls and investigate errors without repeatedly modifying the source code.

For HPC developers, these concepts become even more important when applications use MPI, multiple CPU threads and GPUs.

Tools such as ROCgdb extend this debugging approach to AMD GPU applications, helping developers investigate both CPU and GPU execution.

The best way to become comfortable with GDB is to practice with a small application. Compile it with debugging information, set a breakpoint, step through a function and inspect a few variables.

Once you understand how to control a simple program, the same concepts provide a foundation for investigating much more complex HPC applications.

Acknowledgement

This article is inspired by the knowledge and practical insights gained during the course attended at HLRS as well as CCFR EuroHPC.

A big thank you to the teams at HLRS and CCFR EuroHPC, along with the instructors, for sharing their expertise and providing such a valuable learning experience. The course offered an excellent opportunity to explore debugging concepts and tools used in real HPC and (AMD) GPU computing environments.

This article is an effort to share some of that knowledge in a simple and accessible way, especially for those who are beginning their journey into HPC and parallel computing.