DEV Community

Aviral Srivastava
Aviral Srivastava

Posted on

SystemTap and ftrace for debugging

#linux #performance #tooling

SystemTap and ftrace: Powerful Kernel Debugging Tools

Introduction

Debugging kernel code can be a daunting task. Traditional user-space debugging tools often fall short, as they lack the necessary visibility into the kernel's inner workings. This is where kernel tracing tools like SystemTap and ftrace come in. These tools provide developers and system administrators with powerful capabilities to dynamically instrument the kernel, observe its behavior, and diagnose performance bottlenecks or identify root causes of crashes.

SystemTap and ftrace, while serving similar purposes, differ significantly in their approach. SystemTap is a scripting language and infrastructure built for dynamically instrumenting running Linux kernels. It allows you to write scripts that insert probes into the kernel at specific locations, collecting data and performing actions when those probes are hit. ftrace, on the other hand, is a built-in tracing framework within the Linux kernel itself. It provides a collection of tracers and interfaces for examining kernel activity.

This article delves into both SystemTap and ftrace, exploring their functionalities, advantages, disadvantages, and usage scenarios. We will also look at practical examples to illustrate their capabilities.

Prerequisites

Before diving into SystemTap and ftrace, it's essential to have a basic understanding of the following:

  • Linux Kernel: A fundamental understanding of kernel concepts, architecture, and operation is crucial.
  • C Programming: While not strictly mandatory, familiarity with C is helpful for understanding kernel code and SystemTap script syntax.
  • Command Line Interface: Proficiency in using the command line is necessary for interacting with both tools.
  • Root Privileges: Both SystemTap and ftrace typically require root privileges for access to kernel resources.

For SystemTap, you'll also need to install the necessary packages. The specific packages vary depending on your Linux distribution, but they generally include:

  • systemtap: The main SystemTap package.
  • kernel-devel: The kernel header files and build environment required for SystemTap to compile probe modules. It must correspond to the running kernel version.
  • elfutils-devel: Development tools for working with ELF (Executable and Linkable Format) files.
  • dwz: A tool that is part of elfutils which tries to reduce size of debugging info in ELF files

For ftrace, no external installation is usually required as it is integrated into the kernel.

SystemTap

Advantages:

  • Flexibility: SystemTap's scripting language offers a high degree of flexibility in defining probe points, data collection, and analysis logic.
  • High-Level Scripting: The SystemTap language is relatively easy to learn and use, especially for those familiar with C-like syntax.
  • Dynamic Probing: Probes can be inserted and removed while the kernel is running, minimizing disruption.
  • Complex Data Analysis: SystemTap allows for sophisticated data aggregation, filtering, and analysis.
  • User-Defined Functions: You can define custom functions within SystemTap scripts to perform complex operations.
  • Extensibility: SystemTap can be extended with modules and libraries to support new probe types and data analysis techniques.

Disadvantages:

  • Performance Overhead: SystemTap can introduce significant performance overhead, especially when using many probes or complex scripts. Improper probing can even lead to kernel crashes.
  • Kernel Module Compilation: SystemTap compiles scripts into kernel modules, which can take time and require kernel development packages.
  • Kernel Dependency: SystemTap scripts are tied to a specific kernel version, requiring recompilation when the kernel is upgraded.
  • Security Considerations: SystemTap scripts have the potential to compromise system security if not written carefully.

Features:

  • Probe Points: SystemTap supports various probe points, including:
    • kernel.function("function_name"): Probes the entry and exit of a kernel function.
    • kernel.statement("filename:line_number"): Probes a specific line of code in the kernel.
    • kernel.trace("event_name"): Probes a tracepoint defined within the kernel.
    • timer.ms(interval): Probes at a regular interval.
  • Data Collection: SystemTap provides variables to access function arguments, return values, global variables, and other kernel data.
  • Scripting Language: The SystemTap language supports variables, loops, conditional statements, and user-defined functions.
  • Data Aggregation: SystemTap can aggregate data using functions like count(), sum(), avg(), min(), and max().
  • Output Options: SystemTap can print data to the console, write to files, or send data over the network.

Example:

The following SystemTap script traces the time spent in the sys_open system call:

#!/usr/bin/stap

probe begin {
  printf("Starting sys_open tracing...\n");
}

probe kernel.function("sys_open") {
  start_time = gettimeofday_us();
}

probe kernel.function("sys_open").return {
  end_time = gettimeofday_us();
  elapsed_time = end_time - start_time;
  printf("sys_open took %d us to execute\n", elapsed_time);
}

probe end {
  printf("Finished sys_open tracing.\n");
}
Enter fullscreen mode Exit fullscreen mode

To run this script, save it as open_trace.stp and execute:

sudo stap open_trace.stp
Enter fullscreen mode Exit fullscreen mode

This will output the execution time of the sys_open system call each time it is invoked.

ftrace

Advantages:

  • Low Overhead: ftrace is designed for low-overhead tracing, minimizing the impact on system performance.
  • Built-in: ftrace is already integrated into the Linux kernel, eliminating the need for external installations.
  • Boot-time Tracing: ftrace can be configured to start tracing at boot time.
  • Function Tracing: ftrace can trace function calls, providing a call graph of kernel execution.
  • Tracepoints: ftrace supports tracepoints, which are predefined markers in the kernel code for specific events.
  • Filter Options: ftrace provides various filter options to narrow down the trace data to specific processes or functions.
  • Ease of use (for basic tracing): Basic ftrace commands are easy to grasp and use for simple tracing tasks.

Disadvantages:

  • Limited Scripting: ftrace lacks the powerful scripting capabilities of SystemTap.
  • Static Probing: ftrace primarily relies on predefined tracepoints, limiting the ability to probe arbitrary locations in the kernel.
  • Data Analysis: ftrace provides basic data analysis features, but it is less sophisticated than SystemTap's data aggregation capabilities.
  • Interface complexity: Can be more difficult to configure and control for complex scenarios when compared to systemtap.

Features:

  • Function Tracer: Traces the entry and exit of kernel functions.
  • Function Graph Tracer: Provides a call graph of kernel execution.
  • Tracepoints: Traces events at predefined locations in the kernel.
  • kprobes: Allows dynamic probing of kernel functions, similar to SystemTap's kernel.function probe. Can introduce instabilities if incorrectly used.
  • uprobes: Allows probing of user-space applications.
  • Filter Options: Provides filters to narrow down trace data based on PID, function name, and other criteria.
  • Ring Buffer: Stores trace data in a ring buffer, preventing excessive memory usage.

Example:

The following command uses ftrace to trace the sys_open function:

sudo sh -c 'echo function_graph > /sys/kernel/debug/tracing/current_tracer'
sudo sh -c 'echo sys_open > /sys/kernel/debug/tracing/set_graph_function'
sudo sh -c 'echo 1 > /sys/kernel/debug/tracing/tracing_on'
# Execute the program you want to trace
# For example: cat /etc/passwd
sudo sh -c 'echo 0 > /sys/kernel/debug/tracing/tracing_on'
sudo cat /sys/kernel/debug/tracing/trace
Enter fullscreen mode Exit fullscreen mode

This will output the call graph of the sys_open function, showing the functions it calls and the time spent in each function.

A simpler example using tracepoints:

# Enable tracing of the sched_switch tracepoint
sudo sh -c 'echo sched_switch > /sys/kernel/debug/tracing/set_event'
sudo sh -c 'echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable'
sudo sh -c 'echo 1 > /sys/kernel/debug/tracing/tracing_on'

# Run your workload here
# For Example: sleep 2

sudo sh -c 'echo 0 > /sys/kernel/debug/tracing/tracing_on'
sudo cat /sys/kernel/debug/tracing/trace

# Disable the tracepoint
sudo sh -c 'echo 0 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable'
Enter fullscreen mode Exit fullscreen mode

This will print out information about process switches, including the process ID and priority of the old and new processes.

Choosing Between SystemTap and ftrace

The choice between SystemTap and ftrace depends on the specific debugging requirements:

  • SystemTap: Use SystemTap for complex tracing scenarios that require custom data analysis, aggregation, or filtering. Its scripting capability enables very powerful, albeit potentially dangerous, debugging. It's ideal when deep kernel introspection and custom logic are paramount.
  • ftrace: Use ftrace for low-overhead tracing of predefined tracepoints or function calls. It's suitable for performance profiling and identifying bottlenecks without significantly impacting system performance. Its simplicity makes it easier to get started with for basic tracing tasks.

In many cases, SystemTap and ftrace can be used together. ftrace can be used for initial performance analysis to identify areas of interest, and then SystemTap can be used for more in-depth investigation of those specific areas.

Conclusion

SystemTap and ftrace are valuable tools for kernel debugging and performance analysis. SystemTap offers greater flexibility and scripting capabilities, while ftrace provides lower overhead and is built directly into the kernel. Understanding the strengths and weaknesses of each tool allows developers and system administrators to choose the right tool for the task, enabling them to gain insights into the kernel's behavior and effectively troubleshoot problems. By mastering these tools, developers can unlock a deeper understanding of the Linux kernel and improve the performance and stability of their systems. Always remember to use these tools carefully, especially SystemTap, as improper usage can lead to system instability or even crashes.

Top comments (0)