Beyond Auditd: Logster and the Case for Kernel-Level Threat Detection

Auditd isn't bad software. It's software solving a 2004 problem we stopped having. Here's why the filter has to move into the kernel.

The last time auditd actually caught something in my environment, it was 3 a.m. and the alert landed forty minutes after the attacker had already pivoted. Not because the rule was wrong. Because the box was under enough load that the audit backlog filled, records got dropped, and the one execve that mattered never made it to disk.

Auditd isn't bad software. It's software solving a 2004 problem extremely well, and then being asked to solve a 2025 problem it was never designed for. That gap is why kernel-level visibility has quietly gone from a nice-to-have to the baseline everyone eventually arrives at.

Auditd is a queue with a rulebook

Strip away the tooling and auditd is a rule-matching engine in the kernel's audit subsystem that hands matching syscall records to a single userspace daemon over a netlink socket. You write rules: watch execve, watch writes to /etc/passwd, watch connect. The kernel matches them, and auditd drains the queue and writes records to /var/log/audit/audit.log.

That design has two properties that were sensible in 2004 and are load-bearing failures now.

First, the filtering is coarse, and the real decision happens too late. You can match on syscall number, uid, a watched path, a few fields. What you cannot say is "record this openat only if the process isn't one of my forty known-benign scanners," or "dedup identical outbound connects inside a five-second window." Any logic richer than field-equality runs after the record has been formatted and shipped to userspace. You pay the full cost of every event, then decide it was noise.

Second, the transport is a queue, and queues overflow. Every matched record crosses netlink into that one daemon. Run a find / across a large filesystem and you generate hundreds of thousands of openat calls in seconds. The backlog fills, and now failure mode picks your poison: silently drop records, or panic and take the host down. Most shops run drop mode and learn about it during the incident where the record they needed was one of the dropped ones.

I've watched a single sloppy file-access rule produce something like 85 to 90% of all audit volume on a host: find, getcap, and ps walking /proc and eating EACCES at every step. The signal wasn't buried. It was evicted.

The workaround economy

The fair rebuttal is: so tune your rules. Agreed, a disciplined ruleset with tight path watches and aggressive exclusions gets you a long way. I've written those. They hold until the workload changes, and then you're back in the backlog.

The structural problem is that auditd pushes you into a workaround economy. You can't express the filter you want in the kernel, so you ship everything to userspace and filter there, in the SIEM, in Logstash or in a stream processor. By then you've already paid the collection cost, the netlink cost, the serialization cost, and the network cost to move data you're about to discard. You end up running an entire pipeline whose main job is throwing away 90% of what the previous stage worked to produce. That's not detection engineering; it's plumbing built to compensate for a drain in the wrong place.

The filter belongs where the event is born

Here's the whole argument in one move. When a process calls connect(), at that instant, in kernel context, you already have everything: the calling task, its full ancestry, the socket, the destination, the namespace it lives in. If you can run a small, safe program right there, you decide keep/drop/dedup/enrich in place and surface only the fraction that matters. The rest never gets copied, never gets serialized, never touches a queue.

Concretely, compare what you can say in each model. An audit rule watching outbound network activity:

# auditd: match every connect() syscall for uid >= 1000.
# Filtering stops here. "Which destination? dedup? skip my scanners?"
# all happen later, in userspace, after you've paid to ship each one.
-a always,exit -F arch=b64 -S connect -F auid>=1000 -F key=net-connect

The rough eBPF equivalent, where the decision is made in the kernel before anything is emitted:

SEC("tracepoint/syscalls/sys_enter_connect")
int on_connect(struct trace_event_raw_sys_enter *ctx) {
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    if (is_known_scanner(pid))        return 0;   // drop noise in-kernel
    if (seen_recently(pid, daddr))    return 0;   // dedup in-kernel
    emit_enriched(pid, daddr, dport);             // only survivors cross to userspace
    return 0;
}

Same syscall, completely different economics. The audit rule is a wiretap that records everything and sorts it out downstream. The eBPF program is a bouncer at the door. Falco, Tetragon, and Tracee all landed on this model independently, and that convergence matters: three teams with different goals reaching the same architecture is a strong signal it's the shape of the problem, not a fashion.

But we've tried kernel-level before

We have, in two ways, and both taught us something.

Kernel modules give you all the visibility and all the risk. A bug in your out-of-tree module is a kernel panic, a null deref in ring 0, a host that doesn't come back. Nearly every EDR that shipped a kernel driver has, at some point, panicked or blue-screened a fleet on a bad update. Visibility was never the problem there; blast radius was.

ptrace-based approaches (the strace lineage) get you syscall detail without a module, but ptrace stops the tracee twice per syscall and context-switches to the tracer each time. It's a debugging interface, not fleet telemetry. On syscall-heavy workloads that overhead isn't a few percent. strace can slow a program by an order of magnitude, and I don't turn it on in production without a reason and a stopwatch.

So the bar for "detection in the kernel" was never just visibility. It was visibility without a module's panic risk and without ptrace's overhead. That specific combination is the actual news. Not that the kernel sees more (everyone always knew that), but that you can finally run your own logic up there without betting the host on it.

Where auditd still wins

I'd be selling you something if I called auditd dead. There are places I'd still reach for it first.

For a compliance checkbox (CIS, PCI, a control that says "audit logs must exist"), auditd is the blessed, documented, auditor-recognized answer. Rolling your own eBPF collector to satisfy that control is a fine way to fail an assessment for no upside.

On ancient kernels, a fleet frozen on RHEL 7 / 3.10 that nobody will let you touch, the eBPF features you'd actually want aren't there, and auditd is what you've got.

And in an air-gapped, change-frozen environment, "it's in the base OS and the auditors already trust it" is a genuine operational advantage a new agent has to earn, not wave away.

None of that changes the direction of travel. It just makes this a migration, not a conversion.

That's the split we ended up with building Logster, an AI-driven endpoint threat detection platform: it supports both kernel-level collection and auditd, because real Linux fleets have both. The 85 to 90% figure above came from that same work. It's the workaround economy in miniature: we fixed it, but only after the flood had shown us where the rule was too broad.

The tax nobody warns you about

Kernel-level detection isn't free, and the cost isn't compute. It's people. The team that can write and debug a verifier-approved BPF program is smaller than the team that can write an audit rule. When the verifier rejects your program at load time with a message that reads like a compiler venting its frustrations, there's no Stack Overflow answer waiting for you. You'll also chase version drift: what loaded cleanly on 5.15 behaves differently on 6.8, and your CI matrix grows a dimension it didn't have.

That's a real tax. But it's paid by the builders, once, up front. Auditd's tax is paid by the detection itself, every time the host gets busy and the records you needed quietly evaporate. I know which debt I'd rather carry. The threats moved into the space between syscalls years ago; our visibility is only now able to follow them there safely.