Mustafa ERBAY

Posted on Sep 18 Originally published at mustafaerbay.com.tr

Zero for Root Too: kptr_restrict, dmesg_restrict and perf_event_paranoid

#linux #kernel #security #perf

This morning I took a one-second perf record -a on the server, as root. The report came back like this:

Warning:
Kernel address maps (/proc/{kallsyms,modules}) were restricted.
Check /proc/sys/kernel/kptr_restrict before running 'perf record'.
As no suitable kallsyms nor vmlinux was found, kernel samples
can't be resolved.
     2.24%  [unknown]   [k] 0xffffffffb0e6bd40
     1.71%  [unknown]   [k] 0xffffffffb0e6afe0

I am root; I can open /proc/kallsyms; every address in it is zero. grep " T _text$" /proc/kallsyms → 0000000000000000 T _text. The cause wasn't far away: /etc/sysctl.d/99-security.conf, dated 11 May 2026, written by me. It says kernel.kptr_restrict = 2. The same file had already shown up in the TIME_WAIT post with its tcp_fin_timeout = 15 line; apparently I turned quite a few knobs that evening. Ubuntu's own file (10-kernel-hardening.conf, from the procps package) said 1; on top of it I had written 2, on the "tighter is better" reflex. systemd-sysctl applies files in name order, so 99 wins, and for four months the kernel side of perf on this machine had been blind. Nobody noticed, because for four months nobody had taken a kernel profile.

This post is about those three knobs: kptr_restrict, dmesg_restrict, perf_event_paranoid. They all appear in the same lists under the "kernel hardening" heading, each is documented separately, but reading the source shows they are tied together; one changes what another means. There is also the distribution's own contribution: Ubuntu's perf_event_paranoid value is a number that doesn't exist in the upstream documentation at all. Kernel 6.8 (Ubuntu 24.04, 6.8.0-139-generic), Docker 29.4.3. Every step I took with the sysctls was of the reversible kind and I reverted all of them; the machine is at the values it started with.

kptr_restrict: the docs describe %pK, kallsyms does something else

Documentation/admin-guide/sysctl/kernel.rst defines this knob in terms of the %pK format specifier: at 0 the address is hashed before printing (that is, the same as %p), at 1 pointers printed with %pK are zeroed unless the process holds CAP_SYSLOG and its effective uid/gid equal its real uid/gid, at 2 zeros for everyone. The document also explains the odd "uid equality" condition at 1: the %pK check happens at read() time, not open() time; so that a setuid program that opens the file and then elevates its privileges before reading doesn't leak addresses. In the document's own words this is "a temporary solution only".

But what broke my perf isn't %pK. The code for /proc/kallsyms (kernel/kallsyms.c) prints values with %px and asks a separate function, kallsyms_show_value(), what value to show. That function lives in kernel/ksyms_common.c and looks like this:

bool kallsyms_show_value(const struct cred *cred)
{
    switch (kptr_restrict) {
    case 0:
        if (kallsyms_for_perf())
            return true;
        fallthrough;
    case 1:
        if (security_capable(cred, &init_user_ns, CAP_SYSLOG,
                     CAP_OPT_NOAUDIT) == 0)
            return true;
        fallthrough;
    default:
        return false;
    }
}

Two things stand out. First, case 0 doesn't say "show everyone"; it shows if kallsyms_for_perf() is true, which means sysctl_perf_event_paranoid <= 1. So even if you write kptr_restrict = 0, an unprivileged user still sees zeros in /proc/kallsyms when perf_event_paranoid is 2 or higher. That link is not in the documentation. Second, the check is made at open() time: kallsyms_open() computes the result with file->f_cred and stores it in the iterator; none of %pK's read-time trouble here.

Measuring this is easy. I read the _text symbol's line as root and as nobody:

kptr_restrict	perf_event_paranoid	root	nobody
2	4	`0000000000000000`	`0000000000000000`
1	4	`ffffffffafc00000`	`0000000000000000`
0	4	`ffffffffafc00000`	`0000000000000000`
0	1	`ffffffffafc00000`	`ffffffffafc00000`

The third row is the one a reader of the documentation wouldn't expect: kptr_restrict = 0, yet nobody still sees zero. In the fourth row, lowering perf_event_paranoid to 1 opens the address up. This is why Ubuntu's 4 (more on it shortly) effectively turns a "kptr_restrict = 0" setting into 1 for unprivileged users.

/proc/kallsyms isn't the only caller of kallsyms_show_value(). kernel/module/procfs.c uses the same function to zero the load addresses in /proc/modules (on my box: sch_netem 24576 0 - Live 0x0000000000000000), kernel/kprobes.c does so in the kprobe list under debugfs, kernel/module/sysfs.c in /sys/module/*/sections/, and bpf_dump_raw_ok() in include/linux/filter.h consults the same decision for raw address dumps of BPF programs; the BPF link and kprobe info interfaces (kernel/bpf/syscall.c, kernel/trace/bpf_trace.c) also ask the same function before revealing a kprobe address. One knob, at least five doors.

What perf and bpftrace see at 2

When I recorded as root with kptr_restrict = 2, perf record already warned during the recording: "Couldn't record kernel reference relocation symbol". The report: [unknown]. The notable part: after lowering the knob to 1 and re-reporting the same perf.data file, it stayed [unknown]. If the kernel reference symbol couldn't be captured at record time, the file is permanently unresolvable. Recording again at 1 filled the table: __pv_queued_spin_lock_slowpath 3.61%, smp_call_function_single 2.41%, timerqueue_add 2.38%. So "I'll take the profile and open the knob later to look" doesn't work; the knob has to be open before the recording.

bpftrace's kstack reads from the same source. I asked for a three-frame stack on kprobe:vfs_write; at 2 the result was three raw addresses: 0xffffffffb00f7ce1, 0xffffffffb00f85c9, 0xffffffffafc059ae. No symbols. bpftrace doesn't even error, it silently prints hex; it's entirely possible to see that at midnight and go hunting for a vmlinux wondering whether the symbol table is corrupt.

The question I asked myself: what 2 buys me over 1. At 1 only processes carrying CAP_SYSLOG see addresses; on this machine that amounts to root. 2 stops root from seeing them too; the scenario it defends against is one where root has been compromised but the attacker still needs kernel addresses. For a world in which an attacker who is already root can't write sysctl -w kernel.kptr_restrict=1, you need a MAC policy that closes /proc/sys to root as well, or a read-only /proc/sys; even the lockdown LSM doesn't touch this knob (there is no security_locked_down call in 6.8's kernel/sysctl.c). I don't have that world. On my box, the only concrete effect of 2 was breaking root's perf. My decision is to go back to 1; Ubuntu's own value.

dmesg_restrict: even at 0, not everything is allowed

The documentation is short: no restrictions at 0, CAP_SYSLOG required for dmesg(8) at 1, and CONFIG_SECURITY_DMESG_RESTRICT sets the default. Ubuntu compiles that option in (=y in /boot/config-6.8.0-139-generic), so it is 1 even if you write nothing into sysctl.d. The dmesg_restrict = 1 line in my 99 file re-states what is already the case; harmless, pointless.

The code (kernel/printk/printk.c) is a little finer than the documentation:

static int syslog_action_restricted(int type)
{
    if (dmesg_restrict)
        return 1;
    return type != SYSLOG_ACTION_READ_ALL &&
           type != SYSLOG_ACTION_SIZE_BUFFER;
}

So 0 doesn't mean "no restrictions"; it means "reading and buffer size are free, the rest isn't". Clearing the buffer (dmesg -C) and changing the console level need privilege at 0 too. Measured: with dmesg_restrict = 0, nobody read 4015 lines, but dmesg -C said "clear kernel buffer failed: Operation not permitted". Its caller, check_syslog_permissions(), also leaves a historical door open: without CAP_SYSLOG but with CAP_SYS_ADMIN it lets you through, printing a "deprecated" warning just once via pr_warn_once. CAP_SYSLOG was split from CAP_SYS_ADMIN in 2.6.37; the door is still open.

/dev/kmsg goes through the same check, but inside devkmsg_open(), at open time. head -c 200 /dev/kmsg as nobody → "Operation not permitted". util-linux's dmesg tries /dev/kmsg first and falls back to syslog(2); since both hit the same function, the error text may change but the outcome doesn't.

Who returns EPERM inside a container

The truly instructive part is the container. Docker's default capability list (daemon/pkg/oci/caps/defaults.go) contains neither CAP_SYSLOG nor CAP_PERFMON. But the "klogctl: Operation not permitted" that docker run --rm alpine dmesg gets is not produced by dmesg_restrict. To find who produces it I lowered the knob to zero and tried twice:

dmesg_restrict	seccomp	result
1	default	`Operation not permitted`
1	`unconfined`	`Operation not permitted`
0	default	`Operation not permitted`
0	`unconfined`	3740 lines, the host's kernel log

Docker's default seccomp profile (moby/profiles, seccomp/default.json) allows the syslog syscall when CAP_SYSLOG (or CAP_SYS_ADMIN) is present; likewise perf_event_open requires CAP_PERFMON and bpf requires CAP_BPF, with CAP_SYS_ADMIN sufficing for all three as well. In a container given --cap-add SYS_ADMIN, both seccomp and the kernel's historical door above are open; the value of dmesg_restrict makes no difference. /dev/kmsg doesn't exist in the container's /dev at all. So in an ordinary container, the first wall is seccomp. Where dmesg_restrict comes into play is where that wall is absent: containers running with --security-opt seccomp=unconfined. Plenty of people do that; "turn off seccomp" is standard forum advice for running perf or gdb. That container's root, on a host with dmesg_restrict = 0, reads the host's entire kernel log: which modules are loaded, which disk threw which error, which IP the SYN flood came from. It also reads it with --cap-add SYSLOG (3737 lines), but then you granted it on purpose. (The three line counts were taken in three different minutes, and alpine's busybox dmesg splits the buffer differently from the host's util-linux dmesg; same buffer, three numbers.)

Of the three knobs, this is the one I'll keep at 1 without hesitation. Its cost is near zero: I couldn't find a legitimate workflow in which an unprivileged user needs to see dmesg; journald already runs with CAP_SYSLOG (decoding its CapEff with capsh --decode lists cap_syslog, not cap_perfmon) and hands out read access via journalctl -k through the systemd-journal group.

perf_event_paranoid: the docs end at 2, Ubuntu has a 4

The upstream documentation knows four levels: -1 allows almost everything, from 0 up raw tracepoints and ftrace function tracing are closed, from 1 up CPU-wide events are closed, from 2 up kernel profiling is closed. Default 2. The checks are three small functions in include/linux/perf_event.h: perf_allow_tracepoint() (> -1), perf_allow_cpu() (> 0), perf_allow_kernel() (> 1); all three let you through if perfmon_capable() is true, which is CAP_PERFMON || CAP_SYS_ADMIN. CAP_PERFMON arrived in 5.8; the perf-security.rst page explicitly marks the use of CAP_SYS_ADMIN as "discouraged" and notes that from 5.9 on CAP_SYS_PTRACE isn't needed either.

On my box the value is 4. No sysctl.d file writes it; grep perf_event across /etc/sysctl.d, /usr/lib/sysctl.d, /run/sysctl.d and /etc/sysctl.conf is empty. The source is the kernel itself: Ubuntu carries the CONFIG_SECURITY_PERF_EVENTS_RESTRICT patch that never went upstream (=y in /boot/config), and in the Ubuntu version of the patch the default is this (noble 6.8.0-136 tree, kernel/events/core.c; the official address is git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble, I read it from a GitHub mirror of it):

#ifdef CONFIG_SECURITY_PERF_EVENTS_RESTRICT
int sysctl_perf_event_paranoid __read_mostly = PERF_SECURITY_MAX;
#else
int sysctl_perf_event_paranoid __read_mostly = 2;
#endif

PERF_SECURITY_MAX is 4 in the header, and at the entry of perf_event_open(), if perf_paranoid_any() (i.e. >= 4) is true, everyone without perfmon_capable() gets -EACCES. A small contradiction: the help text in the same tree's security/Kconfig says "the sysctl will be set to 3"; the code says 4. The 2016 LKML original of the patch (Jeff Vander Stoep, for Android) used 3; Ubuntu made PERF_SECURITY_MAX 4 and the help text was left behind. Ubuntu's own perf tool (6.8.12) also only counts up to 2 in its error message; it says "perf_event_paranoid setting is 4" and ends the list with >= 2: Disallow kernel profiling. Neither the documentation nor the tool tells you what 4 is.

I measured what the numbers do, one by one, as nobody; the events were a software counter (page-faults), a user-space hardware counter (cycles:u) and a kernel counter (cycles:k):

paranoid	page-faults	cycles:u	cycles:k
4	denied	denied	denied
3	50	600308	denied
2	51	583120	denied
1	52	551701	2341381

No difference between 3 and 2; 3 has no special meaning in the Ubuntu kernel, the original patch's 3 wasn't carried over. One more detail: at 2 and 3 the page-faults output came back as page-faults:u, at 1 as page-faults. When kernel permission is denied, the perf tool silently retries with exclude_kernel; you learn that your count covers only user space from the letter after the colon.

I also tested whether CAP_PERFMON is enough at 4, because in some derivative trees of the Ubuntu patch (vendor kernels carrying the same patch) that line reads capable(CAP_SYS_ADMIN); if it did here, the entire point of the CAP_PERFMON design would be lost. With setpriv --reuid=nobody --inh-caps=+perfmon --ambient-caps=+perfmon I built a process with only bit 38 set (CapEff: 0000004000000000, capsh --decode → cap_perfmon); at 4 it counted 1955386 cycles:k. In the noble tree the line is perfmon_capable() and CAP_PERFMON alone suffices. Same inside a container: at paranoid = 1, a capability-less alpine couldn't run perf stat (seccomp), with --cap-add PERFMON it counted 33 page faults.

The three together: who sees what

How to read the chain: perf_event_paranoid determines not only perf_event_open() but also what kptr_restrict = 0 means; and dmesg_restrict, whatever the docs say, carries the real protection of %pK, because the most productive place where the kernel leaks addresses is an oops, and that lands in dmesg. The comment in procps's 10-kernel-hardening.conf says as much: "kernel addresses and detailed call traces are frequently found in kernel oops messages".

Who on this machine cares

The ones paying for hardening settings are the unprivileged processes; so I checked how many there are. A scan of /proc/*/status: 721 processes, 461 of them root, 260 non-root. None of the 260 non-root processes has CAP_SYSLOG or CAP_PERFMON; nginx workers, postgres, node, processes inside containers. Of the 461 root ones, 417 have CAP_SYSLOG and 416 have CAP_PERFMON; the remaining 44-45 are daemons that trimmed their own privileges (journald is among those that kept cap_syslog and dropped cap_perfmon).

For this server, that table summarises the three knobs like this: dmesg_restrict = 1 keeps 260 processes away from the host log, and it costs me nothing. perf_event_paranoid = 4 keeps the same 260 away from perf_event_open(); had one of them needed to profile (a Go service's pprof doesn't ask for this, perf does), --cap-add PERFMON is one line. kptr_restrict = 2, on the other hand, gives the 260 nothing beyond what 1 gives, because they don't have CAP_SYSLOG anyway; its only effect is on root. So choosing 2 was a choice against root, not against unprivileged users, and in my threat model there is no defending against root with a sysctl.

That's the arithmetic on a single-admin VPS. On a multi-user build server, or a development box where unprivileged users are expected to profile their own code, the table flips: there you either lower perf_event_paranoid to 1 and keep kptr_restrict at 1, or better, give the perf binary setcap cap_perfmon,cap_sys_ptrace,cap_syslog=ep and open it to a perf_users group; perf-security.rst walks through that recipe step by step. Authorising a group instead of lowering the whole machine to 1 is the narrower form of the same outcome.

Questions to ask your own server

What does sysctl kernel.kptr_restrict kernel.dmesg_restrict kernel.perf_event_paranoid say, and which file wrote each line? grep -rn across /etc/sysctl.d, /usr/lib/sysctl.d, /run/sysctl.d; a value you can't find comes from a kernel build option (DMESG_RESTRICT, PERF_EVENTS_RESTRICT in /boot/config-*).
When you perf record as root, do kernel symbols resolve? If not, every perf.data taken before lowering the knob to 1 is garbage; open it before recording.
Do you have a container running with --security-opt seccomp=unconfined? If so, dmesg_restrict is the only wall between the host log and that container.
If an unprivileged process needs to profile, at which layer is the fix? Machine-wide sysctl, setcap on the binary, or --cap-add PERFMON in the container? Pick the narrowest; CAP_PERFMON suffices in all three, CAP_SYS_ADMIN is not required.
If you're on Ubuntu, do you know that 4 is not the same as 3, and that 3 is the same as 2? Dropping from 4 to 3 "to loosen a little" doesn't open kernel profiling; it opens all user-space counters to unprivileged users.

The one who closed the door was me again

In the fs.protected_regular post I couldn't write to /tmp despite being root; this time I couldn't see the kernel's addresses despite being root, and once again the one who closed the door was me. Both incidents share the same lesson: the documentation of hardening knobs is correct one by one, but their behaviour isn't one by one. You don't know what kptr_restrict = 0 does without knowing perf_event_paranoid; you don't know what dmesg_restrict = 0 means for containers without reading the seccomp profile; and you can't learn what perf_event_paranoid = 4 is from the upstream docs at all.

Before turning a sysctl on the "bigger number is safer" principle, two questions: whom does this number stop, and does that person exist on this machine? If the answer to the second is "only me", you're probably stopping yourself; to find out four months later, on an [unknown] line.

DEV Community