We just got a new super-power! Runtime USDT comes to Linux

Originally posted at Sthima Insights on Oct 30, 2017. Reposted here for preservation purposes. Links have been updated, but the content hasn't.

A new tracing capability has arrived in Linux: the ability to instrument user-level statically defined tracing (USDT) probes defined in dynamic languages such as Python and Node. Prior Linux USDT support has been limited to the static code, such as C or C++. Now, USDT probes can be defined in a Node.js program and used from there, allowing new analysis tools that combine custom Node.js tracepoints along with other library and kernel events. Dtrace-capable operating systems have libusdt, and now Linux has libstapsdt — a library to create USDT probes at runtime. Both have a similar API, as libstapsdt is inspired on libusdt, but they work very different internally.

libstapsdt uses the same structure provided by Systemtap SDT. Thus it should work with any tool able to trace SDT probes from Systemtap (for example, bcc’s tplist and trace tools). It is written in C, which makes it easy to be wrapped by many other languages. If you want to learn more about how libstapsdt works internally, you can look at our documentation here.

In this tutorial, we will show how to use eBPF/bcc to trace probes using Node.js and Python wrappers for libstapsdt. We assume a basic understanding of software instrumentation and tracing tools on this tutorial.

Installing Dependencies

To use either Node.js or Python wrappers, you first need to install libstapsdt on your system. On Ubuntu 16.04, you can install libstapsdt from a PPA. For other distributions, you can build it from source following the instructions here.

sudo add-apt-repository ppa:sthima/oss  
sudo apt-get update  
sudo apt-get install libstapsdt0 libstapsdt-dev

NOTE: the PPA above is no longer maintained or updated. A new PPA might be provided in the future, otherwise users who want the latest version can follow build instructions on GitHub.

In this tutorial, we will use eBPF/bcc (version 0.4.0 or higher) to trace USDT probes. You can install the latest stable version by running:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D4284CDD  
echo "deb https://repo.iovisor.org/apt/xenial xenial main" | sudo tee /etc/apt/sources.list.d/iovisor.list  
sudo apt-get update  
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

Now you can either try the Node.js Example or the Python Example.

Node.js Example

You can install Node.js’s wrapper by running:

npm install usdt

Copy the following code and paste it into a file named index.js:

// Import usdt module
const USDT = require("usdt");

// Create a Provider, which we'll use to register probes
const provider = new USDT.USDTProvider("nodeProvider");

// Register your first probe with two arguments by passing its types
const probe1 = provider.addProbe("firstProbe", "int", "char *");

// Enable provider (needed to fire probes later)
provider.enable();

let countdown = 10;
function waiter() {
  console.log("Trying to fire probe...");
  if(countdown <= 0) {
    console.log("Disable provider");
    provider.disable();
  }
  // Try to fire probe
    probe1.fire(function() {
    // This function will only run if the probe was enabled by an external tool
    console.log("Probe fired!");
    countdown = countdown - 1;
    // Returning values will be passed as arguments to the probe
    return [countdown, "My little string"];
  });
}

setInterval(waiter, 1000);

You can then run it with the following command. It will print “Trying to fire probe…” every 1 second.

node index.js

You can now use some tools to trace your program. For example, if you run the command below in another terminal, it will print the arguments returned on line 26 every time it fires a probe. You can also see that now your Node.js application is printing “Probe fired!” in the terminal.

\# PID is the pid number for "node index.js"  
sudo /usr/share/bcc/tools/trace -p \[PID\] 'u::firstProbe "%d - %s", arg1, arg2'

Now you can use any tool from bcc which uses USDT probes to analyze your Node.js programs with USDT probes!

This example is also available as an ASCIICast:

https://asciinema.org/a/144982

Python Example

You can install Python’s wrapper by running:

pip install stapsdt

Copy the following code and paste it into a file named demo.py:

from time import sleep

import stapsdt

# Create a Provider, which we'll use to register probes
provider = stapsdt.Provider("pythonapp")

# Register your first probe with two arguments by passing its types
probe = provider.add_probe(
    "firstProbe", stapsdt.ArgTypes.uint64, stapsdt.ArgTypes.int32)

# Enable provider (needed to fire probes later)
provider.load()

while True:
    print("Trying to fire probe...")
    # Try to fire probe
    if probe.fire("My little probe", 42):
        # This will only run if the probe was enabled by an external tool
        print("Probe fired!")
    sleep(1)

You can then run it with the following command. It will print “Trying to fire probe…” every 1 second.

python demo.py

You can now use some tools to trace your program. For example, if you run the command below in another terminal, it will print the arguments passed on line 18 every time it fires a probe. You can also see that now your Python application is printing “Probe fired!” in the terminal.

\# PID is the pid number for "python demo.py"  
sudo /usr/share/bcc/tools/trace -p \[PID\]'u::firstProbe "%s - %d", arg1, arg2'

Now you can use any tool from bcc which uses USDT probes to analyze your Python programs with USDT probes!

This example is also available as an ASCIICast:

https://asciinema.org/a/144984

Where to go from here

Instrumentation on Linux is growing fast, especially with recent improvements to eBPF on the Kernel and the fantastic tools present on bcc. With libstapsdt, we’re taking another step by adding the ability to developers to instrument their code in dynamic languages.

There are still some refinements to be done on libstapsdt, but it works for most use cases. If you want to contribute to libstapsdt, you can find us on our Github page. If you decide to write a wrapper, please let us know (we’ll add it to the list of wrappers on the main page)!