Raghu Bharadwaj

Posted on Aug 17 • Originally published at techveda.live

Writing a Minimal sched_ext Scheduler

#linux #kernel #bpf #embedded

A working sched_ext scheduler is a struct sched_ext_ops containing a handful of BPF callbacks. Only the name field is mandatory; the kernel supplies a default for every callback you leave out. Your code does not run tasks directly. It decides which dispatch queue each task belongs in, and the scheduler core takes work from those queues.

Our earlier piece on the sched_ext architecture described the framework in the abstract: four layers, a callback API, and dispatch queues as the point where BPF hands tasks to the kernel. This post goes a level below that and writes the code, loads it, and checks that it took over.

It is written for engineers building or adapting a minimal sched_ext scheduler on a current kernel. It assumes you are comfortable with BPF basics, kernel configuration, and building example programs from a source tree.

Kernel prerequisites

sched_ext is gated by CONFIG_SCHED_CLASS_EXT, and it needs a working BPF stack with BTF. The kernel documentation lists the required set:

CONFIG_BPF=y
CONFIG_SCHED_CLASS_EXT=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y

Check what your running kernel actually has before anything else:

raghu@techveda.org:~$ zgrep -E 'SCHED_CLASS_EXT|DEBUG_INFO_BTF' /proc/config.gz
CONFIG_SCHED_CLASS_EXT=y
CONFIG_DEBUG_INFO_BTF=y

If /proc/config.gz is absent, try /boot/config-$(uname -r).

CONFIG_DEBUG_INFO_BTF is frequently absent on a custom-built embedded kernel, and it is a common reason a scheduler cannot be built or loaded on a target. The BPF program is built against a vmlinux.h generated from the kernel's BTF, and the kernel needs its own BTF present at load time to resolve the program's structure accesses against the running kernel.

Minimal sched_ext scheduler anatomy

A BPF scheduler is a struct sched_ext_ops in a .struct_ops section. Only ops.name is mandatory. Every callback is optional, and the framework supplies a default for each one you leave out. The kernel's scx_simple example defines four callbacks plus the name, which is enough for a complete, working scheduler.

Here is that example assembled into a single file, with the includes and licence declaration a BPF object needs:

#include <scx/common.bpf.h>

char _license[] SEC("license") = "GPL";

/* Set by ops.exit(), read by the user-space loader. */
int exit_type;

s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p,
                   s32 prev_cpu, u64 wake_flags)
{
        s32 cpu;
        /* Need to initialize or the BPF verifier will reject the program */
        bool direct = false;

        cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &direct);

        if (direct)
                scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);

        return cpu;
}

void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
{
        scx_bpf_dsq_insert(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
}

s32 BPF_STRUCT_OPS_SLEEPABLE(simple_init)
{
        return 0;
}

void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
{
        exit_type = ei->type;
}

SEC(".struct_ops")
struct sched_ext_ops simple_ops = {
        .select_cpu             = (void *)simple_select_cpu,
        .enqueue                = (void *)simple_enqueue,
        .init                   = (void *)simple_init,
        .exit                   = (void *)simple_exit,
        .name                   = "simple",
};

Three details in that file are worth stating explicitly.

The comment about initialising direct is not stylistic advice. The verifier tracks whether each variable has been initialised and rejects the program if one can be read before it is written. The variable name itself is arbitrary. It is the initialisation that is mandatory.

scx_bpf_select_cpu_dfl() reports its decision through a bool * out-parameter, named is_idle in the kfunc prototype, while the return value carries the CPU. The example passes a local called direct, which is why the two names differ.

common.bpf.h comes from the scx tree, at tools/sched_ext/include/scx/ in the kernel source. It provides the kfunc declarations and the BPF_STRUCT_OPS macros.

Dispatch queues and task placement

Your BPF scheduler does not run tasks. It influences execution by placing tasks into dispatch queues, and the scheduler core selects runnable tasks from those queues. The path a waking task takes is:

wake-up
   -> ops.select_cpu()      pick a CPU; may insert directly
   -> ops.enqueue()         if not already inserted; choose a DSQ
   -> [ local DSQ -> global DSQ -> ops.dispatch() ]
   -> task runs

When a CPU is ready to schedule, it reads its own local DSQ first, then the global DSQ. If both are empty, ops.dispatch() is invoked so your scheduler can fill the local DSQ.

The targets you can name are SCX_DSQ_LOCAL, SCX_DSQ_LOCAL_ON | cpu for a specific CPU's local queue, SCX_DSQ_GLOBAL, or a custom DSQ identified by an ID below 2^63. Inside ops.dispatch() you may insert up to ops.dispatch_max_batch tasks per invocation.

The slice argument sets how long the task may run, in nanoseconds. SCX_SLICE_DFL is the default. SCX_SLICE_INF means the task never expires, in which case your scheduler must call scx_bpf_kick_cpu() itself to trigger rescheduling. Treat infinite slices as an advanced option: reach for one early and a scheduler that has simply stopped making progress can look like a scheduler that is working.

Building and loading

The example schedulers and the supporting headers live in the sched-ext/scx repository. Its layout separates the BPF and user-space halves: scheds/ holds the scheduler implementations, scheds/include/ holds the shared BPF and C headers including the generated vmlinux.h, and rust/scx_utils/ holds the common Rust support library. Build it following the instructions in the repository README, which track the current toolchain requirements more reliably than any copy of them here would.

Loading a scheduler means running its user-space binary. The program stays in the foreground and holds the scheduler active. Terminating it aborts the BPF scheduler and reverts every task to the default fair-class scheduler, so there is no state to clean up by hand.

Runtime checks under /sys/kernel/sched_ext

With a scheduler loaded:

raghu@techveda.org:~$ cat /sys/kernel/sched_ext/state
enabled
raghu@techveda.org:~$ cat /sys/kernel/sched_ext/root/ops
simple
raghu@techveda.org:~$ cat /sys/kernel/sched_ext/enable_seq
1

Do not read anything into the root/ component of that path. It has been there since the original 6.12 merge and does not indicate a scheduler hierarchy. Hierarchical scheduling arrived later, and is still arriving. The 7.1 merge (tag sched_ext-for-7.1) landed cgroup sub-scheduler groundwork: it made the dispatch path hierarchical while explicitly leaving the enqueue path to a later cycle. Linux 7.2 added most of the remaining infrastructure, including topological CPU IDs and BPF arena integration, and enqueue-path support is still in development. The 7.1 work also introduced per-scheduler directories exposing an events file. Note that CONFIG_EXT_SUB_SCHED is def_bool y wherever SCHED_CLASS_EXT and CGROUPS are enabled, so it is not a switch you turn on yourself.

enable_seq is a counter that increases each time a scheduler is loaded; zero means none has been loaded since boot. That makes it the quickest way to detect a scheduler that was loaded and then dropped while you were not watching. If state reads as disabled but enable_seq has advanced, something loaded and then went away.

Common failure patterns

Failures fall into three groups, and they are distinguishable.

The verifier rejected the program. The load fails immediately and produces a verifier log. Uninitialised variables and unbounded loops are the usual causes. The log identifies the offending instruction, so read it from the bottom up.
The watchdog unloaded it. The scheduler loads, runs, and then disappears. Tasks were not being scheduled within the permitted window. Check /sys/kernel/sched_ext/state and the exit information your ops.exit() callback received.
A kfunc does not exist. The compiler reports a call to an undeclared function, or the loader reports an unknown kfunc. Either the kernel was built without CONFIG_DEBUG_INFO_BTF, or you are working from older source. The next section covers the second case.

A practical order to work through: confirm the two config symbols on the target kernel rather than the build host, load the scheduler and read state, and if the load failed read the verifier output before changing anything else. If it loaded and then disappeared, check enable_seq and the exit information from ops.exit().

Reading verifier output and kernel log evidence is a skill that carries well beyond sched_ext, and it is a large part of what we work on in our Linux Kernel Infrastructure training.

If you are adapting older examples

One thing will catch you out when working from material published before 2026. The kfunc used above to place a task into a dispatch queue has been renamed, and the compatibility path has since closed.

Kernel	What happened
6.12	`scx_bpf_dispatch()` and `scx_bpf_dispatch_vtime()` ship with the initial merge.
6.13	Renamed to `scx_bpf_dsq_insert()` and `scx_bpf_dsq_insert_vtime()`, because "dispatch" was overloaded and confusing. The old names remain as aliases.
6.17	The aliases are removed.
6.19	`scx_bpf_dsq_insert___v2()` is added. It returns `bool` where the original returned nothing.
later	The in-tree `compat.bpf.h` states that the kernel carries the compat variants until v6.23, and that the wrapper is dropped after v6.22.

The table needs two caveats. The 6.17 removal was not limited to the two insert functions; it dropped the whole set of kfuncs marked for deletion, including scx_bpf_consume() and the scx_bpf_dispatch_from_dsq*() family. And the version numbers in the compatibility header predate the move to 7.x numbering, so read them as an intention several releases out rather than a fixed date.

The useful response is not to track kernel versions in your own source. The scx project ships compat.bpf.h, which defines a wrapper named scx_bpf_dsq_insert that resolves to whichever kfunc the target kernel actually provides. Include it, call the modern name, and the header absorbs the difference, including the ___v2 migration when it lands. That is why the retirement dates above matter less than they first appear.

One related detail: the vtime variant now passes its arguments through a struct, as __scx_bpf_dsq_insert_vtime(). The kfunc reference documentation gives the reason as working around BPF's limit of five function arguments. Call the inline wrapper scx_bpf_dsq_insert_vtime() from common.bpf.h rather than the underscore-prefixed kfunc directly.

Key takeaways

Only ops.name is mandatory in struct sched_ext_ops; four callbacks are enough for a complete scheduler.
CONFIG_SCHED_CLASS_EXT and CONFIG_DEBUG_INFO_BTF are the two symbols that most often block a first attempt.
The kernel reads local DSQ, then global DSQ, then calls ops.dispatch(). Your program's job is placement, not selection.
Terminating the user-space loader reverts every task to the default scheduler, so experiments are cheap to abandon.
/sys/kernel/sched_ext/enable_seq tells you whether a scheduler was loaded and then dropped.
When adapting older examples, use compat.bpf.h and the current kfunc names rather than version-checking in your own code.

DEV Community