Mustafa ERBAY

Posted on Sep 9 Originally published at mustafaerbay.com.tr

Swapping the Scheduler While It Runs: sched_ext

#linux #kernel #ebpf #performance

For years there were two parts of Linux you simply did not touch: memory management and the CPU scheduler. Both were places where you shrugged, said "the kernel knows best" and moved on. They had no knobs worth turning and the arguments about them stayed academic. If you genuinely wanted to change the scheduler, the road was clear — patch, build, install, reboot, and most likely regret.

That ended with 6.12. sched_ext landed in mainline in that release and turned the scheduler into a BPF program you can load and unload on a running system. When I first heard about it I wasn't excited; my reflex was a question: what does the machine do when that thing goes wrong?

This article is the answer to that question, and to the more uncomfortable ones that follow it.

First I looked at my own server

$ uname -r
6.8.0-139-generic
$ ls /sys/kernel/sched_ext
ls: cannot access '/sys/kernel/sched_ext': No such file or directory

Ubuntu 24.04.4 LTS, 18 vCPUs. So the machine that carries this blog has no sched_ext at all — the 6.8 kernel never boarded that train. That is the first practical lesson here: a feature being in mainline does not mean it is on your server. The GA kernels of LTS distributions usually trail by a year or two, and between you and the "new Linux feature" you just read about there is often a kernel upgrade standing in the way.

You can learn where you stand in one line: if /sys/kernel/sched_ext/state answers, you're ready; if the file doesn't exist, the discussion is over.

Why the scheduler became a plugin

The upstream overview document groups the motivation into three headings: experimentation, customization, and rapid scheduler deployment — swapping policies out without disrupting production. All three share the same complaint: experimenting with a scheduler inside the kernel is expensive. Understanding the code takes years, and every attempt requires a build, an install and a reboot. The document offers a bitter example: a master's or PhD candidate working in this area spends years ramping up on the codebase, only to graduate right as they become able to contribute.

With sched_ext the scheduler is a BPF program that fills in a struct sched_ext_ops. The only mandatory field is ops.name; every operation is optional, and the default behaviour steps in for whatever you don't define. Tasks travel through queues called DSQs (dispatch queues): by default there is one global FIFO (SCX_DSQ_GLOBAL) and one local queue per CPU (SCX_DSQ_LOCAL), and your policy can create its own with scx_bpf_create_dsq(). A CPU always takes work from its own local queue; if that is empty it tries the global queue, and if that yields nothing either it calls into your ops.dispatch().

Let me also note that this isn't an academic toy: the project repository says both Meta and Google are fully committed to sched_ext, and that Meta is in the process of mass production deployment.

The decision is made at three stops

From the perspective of whoever writes the policy the flow is plain: when a task wakes up, ops.select_cpu() is called first. The documentation attaches a warning here — the CPU this function returns is not binding, only an optimization hint; the real decision is made at the last step of the chain. There is still a small gain in guessing right, and this step wakes the selected CPU if it is idle. The second stop is ops.enqueue(): you either place the task straight into a queue or hold it in your own data structures on the BPF side. The third stop, ops.dispatch(), is called only when both the CPU's local queue and the global queue are empty.

The subtle part is what the documentation calls custody. If you dispatched the task directly to a terminal queue, it has left your hands and ops.dequeue() is never called. If you are holding it in your own queue or your own data structure, the task is your responsibility and ops.dequeue() runs exactly once as it leaves. A policy that misses this distinction gets acquainted with lost tasks and a watchdog that fires every thirty seconds.

There is an encouraging side too: if you only use the built-in queues you don't need to write ops.dispatch() at all — the local and global queues are driven automatically. So the "I'll write my own scheduler" project can start from a far smaller surface than you'd expect. The difficulty isn't in the lines of code; it's in being able to measure which decision you made and why.

What "system integrity is maintained" actually covers

The kernel documentation makes the claim without hedging: system integrity is maintained no matter what the BPF scheduler does, and the default behaviour is restored whenever an error is detected, a runnable task stalls, or the SysRq-S key sequence is invoked.

The "stalled task" part is not a scheduler, it's a watchdog. Every scheduler declares its own threshold with ops.timeout_ms, and the definition in the kernel source says the default and maximum value of that timeout is 30 seconds. So in the worst case, if the policy you wrote fails to run a task for thirty seconds, the kernel steps in, aborts the scheduler and hands every task back to the fair class. On top of that it prints a debug dump; you can read the same dump from the sched_ext_dump tracepoint.

In infrastructure software being fast is easy; having designed the way back before you needed it is rare. Black boxes spend their worst nights without telling anyone, whereas sched_ext announces the night and shuts itself down.

Do you actually hold the emergency exit?

First let's establish how many rungs the exit ladder actually has, because most write-ups collapse it into a single key. In the order the kernel documentation lists them: terminating the scheduler program (Ctrl-C, systemctl stop scx, kill), the SysRq-S key sequence, and any internal error including a stalled task. So in daily life the primary exit is killing the process; SysRq is the safety net for when that path is blocked, and the thirty-second watchdog is for when you can't reach either.

The delicate part is that safety net. Everyone mentions SysRq-S, but nobody asks whether it is enabled on your machine.

In the kernel source, sched_ext registers the S key with the action "Disable sched_ext and revert all tasks to CFS", and the enable mask for that operation is SYSRQ_ENABLE_RTNICE — that is, 0x100. The value I measured on my server is this:

$ sysctl kernel.sysrq
kernel.sysrq = 176

176 means 128 + 32 + 16: the reboot, remount-read-only and sync bits are on. 0x100 is not there. So on this machine, if someone switched to the IPMI console and pressed Alt+SysRq+S, the kernel would print "this sysrq operation is disabled" and move on.

The saving detail is stated plainly in the sysrq documentation: the value of kernel.sysrq influences only invocation via a keyboard, and invocation through /proc/sysrq-trigger is always allowed for a user with admin privileges. The driver code does exactly that, calling the handler with mask checking turned off from the trigger file's write path. In practice:

# echo S > /proc/sysrq-trigger        # works regardless of the mask

One trap worth stating up front: sysrq command keys are case sensitive. Lowercase s means sync — which is precisely what the 176 mask leaves enabled — while uppercase S is the key that disables sched_ext. Someone typing echo s in a panic will flush their filesystems and leave the scheduler exactly where it was. On the console path you need to hold Shift too.

So if you have a root SSH session, your exit is intact. But if SSH itself can't respond because of the scheduler — which is precisely the scenario you fear while testing this feature — what's left is the console keyboard, and that path is closed on the default I measured. If it were me, I would add the 0x100 bit to kernel.sysrq before starting any experiment; this is not a setting you want to remember later.

Whose scheduling changes the moment you load it

This is the most frequently skipped topic. If SCX_OPS_SWITCH_PARTIAL is not set in ops->flags, then the moment your scheduler loads, all SCHED_NORMAL, SCHED_BATCH, SCHED_IDLE and SCHED_EXT tasks move under your policy. The thing you loaded "just to try it on one service" covers everything on the machine.

With that flag set, only tasks that explicitly select the SCHED_EXT policy enter sched_ext and the rest stay in the fair class — which has higher sched class precedence than SCHED_EXT. If you truly want a narrow experiment, this is what you should look for; with a scheduler that doesn't support partial mode there is no such thing as "a small test" on a production machine.

Real-time and deadline classes stay above all of this regardless; they are not touched.

You loaded it — how will you know it helped?

The insidious part of changing schedulers is that the system also looks perfectly fine when it isn't helping. The kernel is generous here — the outputs below are the examples from the kernel documentation, not something I ran, since my own machine has no sched_ext:

# cat /sys/kernel/sched_ext/state
enabled
# cat /sys/kernel/sched_ext/root/ops
simple
# cat /sys/kernel/sched_ext/enable_seq
1

enable_seq counts how many times a scheduler has been loaded since boot; zero means none ever was. The genuinely valuable file is /sys/kernel/sched_ext/root/events. Two of its counters measure your policy's quality directly. SCX_EV_SELECT_CPU_FALLBACK counts the cases where the CPU your policy suggested was unusable and the core scheduler silently picked another one. SCX_EV_REENQ_REPEAT, in the documentation's own words, means the policy keeps re-deciding placements it can't honour — a wheel spinning in place.

SCX_EV_BYPASS_ACTIVATE and SCX_EV_BYPASS_DURATION need separate care. Bypass mode isn't entered only on errors: in the kernel source, loading and unloading a scheduler turns bypass on and off itself, and suspend and hibernate take the same path. So on a perfectly healthy setup this counter is already above zero in the first second. What you read is not the absolute value but the increase over the baseline you take right after loading.

You can also ask directly whether a single task is on sched_ext:

# grep ext /proc/self/sched
ext.enabled                                  :                    1

My advice is this: collect these counters while idle first, then under real load. If you can't read the difference between them, what you're chasing is a gain you never measured.

Which scheduler?

The repository doesn't offer one "good scheduler", it offers a showroom. On the server side, the names that deserve a mention: scx_tickless describes itself as aimed directly at cloud computing, virtualization and high-performance computing workloads — but read the small print, because actually silencing ticks requires the kernel to be booted with nohz_full, which makes it a boot parameter decision. scx_flash builds an EDF policy focused on fairness among tasks and performance predictability; its own document points at latency-sensitive work such as multimedia and real-time audio processing as the main target, while promising consistent behaviour on overcommitted systems too; scx_layered is a highly configurable hybrid that lets you classify tasks into layers and apply a different policy to each; scx_rusty is another hybrid, multi-domain, computing load balancing in user space. scx_lavd measures how latency-critical a task is and reflects that into a virtual deadline; it comes up often on the desktop and gaming side. The default shipped with the upstream systemd service is scx_cosmos, which focuses on preserving task-to-CPU locality.

There is also a scheds/experimental directory. Don't take something from there into production because the name sounded familiar; sitting in a separate directory is exactly the point.

The real bill in production: version matching

The kernel documentation is blunt: the APIs sched_ext provides to BPF schedulers have no stability guarantees and are subject to change without warning between kernel versions. The ledger of that is right there in the open — BREAKING_CHANGES.md in the repository lists breaking changes such as ops.prep_enable() becoming ops.init_task().

This is not a package dependency, it's a maintenance contract. When you upgrade the kernel you must upgrade your scheduler binary too, and if you don't, what you lose is not a feature — it's the machine's scheduler.

The second line item is sneakier and concerns everyone running containers: the cgroup CPU controls. The kernel documentation says cpu.max, cpu.weight and cpu.idle are enforced by the fair class, while for sched_ext tasks those settings are merely communicated to the BPF scheduler through ops.cgroup_init() and the corresponding callbacks. What follows is the critical part: each BPF scheduler is responsible for implementing the semantics of those settings and may choose to ignore them. So on a machine running containers, the wrong scheduler can quietly void your CPU quotas. Read that scheduler's documentation before you load it.

Don't underestimate the first line item either: because the scheduler binary is a component tied to the kernel version, even a routine kernel upgrade arriving with a security patch creates a verification step for you. If automatic updates are on, be prepared for the machine to have quietly reverted to the fair class one night — which, at least, beats quietly getting slower.

The packaging picture I see today looks like this: the Arch repository carries extra/scx-scheds at 1.1.3-2, in line with the upstream v1.1.3 release dated 19 August 2026. On openSUSE Tumbleweed zypper install scx does the job, on Gentoo emerge sys-kernel/scx, on NixOS services.scx.enable = true; on Fedora the route runs not through the official repository but through a COPR the installation document points to.

Debian and Ubuntu sit outside that picture. Nothing named scx shows up among Debian's source packages. When I searched the Ubuntu archive for a file called scx_lavd there were no results for 26.04, and the installation document sends Ubuntu users straight to building from source — compiler, cargo, clang, libbpf, pahole. On the two most crowded distributions in the server world, taking this on means taking on your own packaging and distribution pipeline as well.

The picture felt familiar. I described something similar with MPTCP: the distance between being in the kernel and being usable usually closes outside the kernel.

If you decide to try it

Run kernel 6.12 or newer. CONFIG_SCHED_CLASS_EXT=y alone isn't enough; the documentation lists CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT, CONFIG_DEBUG_INFO_BTF, CONFIG_BPF_JIT_ALWAYS_ON and CONFIG_BPF_JIT_DEFAULT_ON alongside it. The shortcut: if cat /sys/kernel/sched_ext/state answers, you're ready.
Take the measurement first. p99 latency, queue wait time, whatever metric your workload actually cares about — the one that costs you money.
Rehearse the emergency exit in advance: does echo S > /proc/sysrq-trigger work, and is the 0x100 bit set in the kernel.sysrq mask for the console path?
Start on a single machine, with a slice of production traffic.
For a permanent setup, set the SCX_SCHEDULER variable in /etc/default/scx and run systemctl enable --now scx.service. Temporary experiments go through systemctl set-environment SCX_SCHEDULER_OVERRIDE=..., but that alone does nothing — it needs a systemctl restart scx after it, and unset-environment plus another restart to undo.
Alarm on the scheduler dropping out quietly: polling /sys/kernel/sched_ext/state or collecting the sched_ext_dump tracepoint is what saves you from the "we reverted to the fair class overnight" surprise. Write the rollback criterion next to it; "we'll revert if it gets worse" is not a criterion.

Let me also say who this isn't worth it for: general-purpose servers running dozens of different workloads, virtualization hosts, machines without one dominant application. There the gain is uncertain and the maintenance load is certain. Where it is worth it is narrow and clear: when a single workload dominates the machine, when the latency tail genuinely costs you money, or when placement decisions make a measurable difference on large multi-socket machines.

What actually changed

Not the scheduler itself — its ownership. For twenty years CPU scheduling was a decision the kernel didn't need to ask us about; now it is a configuration choice. And like every configuration choice it wants maintenance, version tracking and a rollback plan.

That's why I don't look at sched_ext as "the feature that speeds up my server". It feels more like being handed the right to write your own scheduling policy, together with the invoice for it. Don't take over the right without reading the invoice.

DEV Community