DEV Community: Chad Ray

Beyond Auditd: Logster and the Case for Kernel-Level Threat Detection

Chad Ray — Mon, 19 Oct 2026 11:05:33 +0000

Auditd isn't bad software. It's software solving a 2004 problem we stopped having. Here's why the filter has to move into the kernel.

The last time auditd actually caught something in my environment, it was 3 a.m. and the alert landed forty minutes after the attacker had already pivoted. Not because the rule was wrong. Because the box was under enough load that the audit backlog filled, records got dropped, and the one execve that mattered never made it to disk.

Auditd isn't bad software. It's software solving a 2004 problem extremely well, and then being asked to solve a 2025 problem it was never designed for. That gap is why kernel-level visibility has quietly gone from a nice-to-have to the baseline everyone eventually arrives at.

Auditd is a queue with a rulebook

Strip away the tooling and auditd is a rule-matching engine in the kernel's audit subsystem that hands matching syscall records to a single userspace daemon over a netlink socket. You write rules: watch execve, watch writes to /etc/passwd, watch connect. The kernel matches them, and auditd drains the queue and writes records to /var/log/audit/audit.log.

That design has two properties that were sensible in 2004 and are load-bearing failures now.

First, the filtering is coarse, and the real decision happens too late. You can match on syscall number, uid, a watched path, a few fields. What you cannot say is "record this openat only if the process isn't one of my forty known-benign scanners," or "dedup identical outbound connects inside a five-second window." Any logic richer than field-equality runs after the record has been formatted and shipped to userspace. You pay the full cost of every event, then decide it was noise.

Second, the transport is a queue, and queues overflow. Every matched record crosses netlink into that one daemon. Run a find / across a large filesystem and you generate hundreds of thousands of openat calls in seconds. The backlog fills, and now failure mode picks your poison: silently drop records, or panic and take the host down. Most shops run drop mode and learn about it during the incident where the record they needed was one of the dropped ones.

I've watched a single sloppy file-access rule produce something like 85 to 90% of all audit volume on a host: find, getcap, and ps walking /proc and eating EACCES at every step. The signal wasn't buried. It was evicted.

The workaround economy

The fair rebuttal is: so tune your rules. Agreed, a disciplined ruleset with tight path watches and aggressive exclusions gets you a long way. I've written those. They hold until the workload changes, and then you're back in the backlog.

The structural problem is that auditd pushes you into a workaround economy. You can't express the filter you want in the kernel, so you ship everything to userspace and filter there, in the SIEM, in Logstash or in a stream processor. By then you've already paid the collection cost, the netlink cost, the serialization cost, and the network cost to move data you're about to discard. You end up running an entire pipeline whose main job is throwing away 90% of what the previous stage worked to produce. That's not detection engineering; it's plumbing built to compensate for a drain in the wrong place.

The filter belongs where the event is born

Here's the whole argument in one move. When a process calls connect(), at that instant, in kernel context, you already have everything: the calling task, its full ancestry, the socket, the destination, the namespace it lives in. If you can run a small, safe program right there, you decide keep/drop/dedup/enrich in place and surface only the fraction that matters. The rest never gets copied, never gets serialized, never touches a queue.

Concretely, compare what you can say in each model. An audit rule watching outbound network activity:

# auditd: match every connect() syscall for uid >= 1000.
# Filtering stops here. "Which destination? dedup? skip my scanners?"
# all happen later, in userspace, after you've paid to ship each one.
-a always,exit -F arch=b64 -S connect -F auid>=1000 -F key=net-connect

The rough eBPF equivalent, where the decision is made in the kernel before anything is emitted:

SEC("tracepoint/syscalls/sys_enter_connect")
int on_connect(struct trace_event_raw_sys_enter *ctx) {
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    if (is_known_scanner(pid))        return 0;   // drop noise in-kernel
    if (seen_recently(pid, daddr))    return 0;   // dedup in-kernel
    emit_enriched(pid, daddr, dport);             // only survivors cross to userspace
    return 0;
}

Same syscall, completely different economics. The audit rule is a wiretap that records everything and sorts it out downstream. The eBPF program is a bouncer at the door. Falco, Tetragon, and Tracee all landed on this model independently, and that convergence matters: three teams with different goals reaching the same architecture is a strong signal it's the shape of the problem, not a fashion.

But we've tried kernel-level before

We have, in two ways, and both taught us something.

Kernel modules give you all the visibility and all the risk. A bug in your out-of-tree module is a kernel panic, a null deref in ring 0, a host that doesn't come back. Nearly every EDR that shipped a kernel driver has, at some point, panicked or blue-screened a fleet on a bad update. Visibility was never the problem there; blast radius was.

ptrace-based approaches (the strace lineage) get you syscall detail without a module, but ptrace stops the tracee twice per syscall and context-switches to the tracer each time. It's a debugging interface, not fleet telemetry. On syscall-heavy workloads that overhead isn't a few percent. strace can slow a program by an order of magnitude, and I don't turn it on in production without a reason and a stopwatch.

So the bar for "detection in the kernel" was never just visibility. It was visibility without a module's panic risk and without ptrace's overhead. That specific combination is the actual news. Not that the kernel sees more (everyone always knew that), but that you can finally run your own logic up there without betting the host on it.

Where auditd still wins

I'd be selling you something if I called auditd dead. There are places I'd still reach for it first.

For a compliance checkbox (CIS, PCI, a control that says "audit logs must exist"), auditd is the blessed, documented, auditor-recognized answer. Rolling your own eBPF collector to satisfy that control is a fine way to fail an assessment for no upside.

On ancient kernels, a fleet frozen on RHEL 7 / 3.10 that nobody will let you touch, the eBPF features you'd actually want aren't there, and auditd is what you've got.

And in an air-gapped, change-frozen environment, "it's in the base OS and the auditors already trust it" is a genuine operational advantage a new agent has to earn, not wave away.

None of that changes the direction of travel. It just makes this a migration, not a conversion.

That's the split we ended up with building Logster, an AI-driven endpoint threat detection platform: it supports both kernel-level collection and auditd, because real Linux fleets have both. The 85 to 90% figure above came from that same work. It's the workaround economy in miniature: we fixed it, but only after the flood had shown us where the rule was too broad.

The tax nobody warns you about

Kernel-level detection isn't free, and the cost isn't compute. It's people. The team that can write and debug a verifier-approved BPF program is smaller than the team that can write an audit rule. When the verifier rejects your program at load time with a message that reads like a compiler venting its frustrations, there's no Stack Overflow answer waiting for you. You'll also chase version drift: what loaded cleanly on 5.15 behaves differently on 6.8, and your CI matrix grows a dimension it didn't have.

That's a real tax. But it's paid by the builders, once, up front. Auditd's tax is paid by the detection itself, every time the host gets busy and the records you needed quietly evaporate. I know which debt I'd rather carry. The threats moved into the space between syscalls years ago; our visibility is only now able to follow them there safely.

Show Your Work: Logster and Detection That Explains Its Alerts and Its Silences

Chad Ray — Mon, 19 Oct 2026 10:47:23 +0000

Every maths teacher has written the same note in the margin: show your work. A correct answer with no working earns partial credit. A wrong answer with clear working can still be fixed, because the teacher can see where the reasoning went wrong.

Security teams rarely get that courtesy from their tools. An alert arrives with a severity, a technique ID and perhaps a score. The analyst then does the working backwards, reconstructing why the system reached that conclusion before deciding whether they agree with it.

The same problem runs the other way, and there it is quieter and more dangerous. When a detection is tuned to stop firing on something benign, the tuning usually shows even less of its reasoning than the alert did. Nobody reviews an alert that never fires.

This article looks at both halves of the problem: verdicts that can't show their work, and silences that can't either.

Key Takeaways

An alert is only as useful as the evidence an analyst can check behind it; a score alone forces them to reconstruct the reasoning.

Suppressions and exclusions are detection logic too. Broad ones can become places for an attacker to hide.

A good system ties every verdict to specific observed activity and keeps each suppression narrow, visible and testable before it takes effect.

Alerts that can't show their reasoning

Picture two alerts for the same host.

The first reads: Suspicious PowerShell activity. Severity high.

The second reads: A document viewer launched PowerShell with an encoded command. That PowerShell process wrote a script to a temporary folder, and a scheduled task registered a minute later points to that script. Each clause links to the process, the file and the task it refers to.

Both might be correct. Only the second can be checked in a minute. With the first, the analyst still has to open the process tree, find the parent, look for file writes, and search for persistence, which is the investigation the alert was supposed to start.

Pattern-matching rules have always had a partial answer here: the rule itself is the explanation. If a rule fires on "Office application spawns a shell", the analyst knows what it matched. That transparency is one reason rules remain valuable.

Model-driven detection makes the problem harder. A statistical score says how sure the model is, not why. A language model can produce a fluent explanation, but a fluent paragraph is not evidence. The model may describe a connection that isn't in the records. The paragraph and the telemetry need to be tied together, or the analyst is simply trading one kind of reconstruction for another: checking whether the prose is true.

So the useful question isn't "Does the detector explain itself?" It is "Can each step of the explanation be traced to something that was actually observed?"

Suppressions that hide more than they should

Every production detection program tunes. A backup agent that reads thousands of files every night, or a software deployment tool that launches scripts across the fleet, will generate noise until someone suppresses it. That work is necessary; an untuned system buries real findings.

The trouble is the shape suppressions tend to take under pressure. The fastest fix is broad: exclude a directory, a process name or a whole host. Such rules often contain wildcards and are rarely reviewed again.

Attackers understand this. MITRE ATT&CK documents Impair Defenses: Disable or Modify Tools (T1562.001), which covers adversaries modifying or disabling security tooling to avoid detection. (MITRE ATT&CK, T1562.001) Adding an exclusion is one quiet way to do that. One who finds an existing exclusion that is broad enough doesn't need to add anything.

Consider a hypothetical suppression that matches any process whose path contains a trusted vendor's folder name. It was written to silence an updater. It also matches a binary an attacker drops into a subfolder of that directory, and anything that binary launches. The rule matched a name. It had no way to notice that the behaviour around that name had changed.

CISA's joint guidance on living-off-the-land techniques recommends establishing and maintaining baselines of normal network, user and administrative activity, and comparing current activity against them. (CISA and partners, February 2024) A suppression is a baseline written down, and a baseline is only safe if it describes what normal looks like, not merely who is usually present.

What "showing your work" would require

Put the two halves together and a set of requirements emerges. None is exotic on its own. The difficulty is getting all of them in one system.

Requirement	For verdicts	For suppressions
Grounding	Every reasoning step points to observed entities	Every rule matches observed attributes, not substrings of them
Scope	The verdict covers a connected set of activity, not a single event	A match covers only benign activity, not whatever that activity spawned
Visibility	The analyst can see what the conclusion relied on	The team can see what a rule would suppress before it does
Honest failure	"Could not decide" is distinct from "benign"	A rule that stops matching stops suppressing

The common thread is that both the alert and the silence should be decisions about structure: which process did what, to what, and in what order. That is also the information an analyst needs to judge either one.

One way to build it

This is the design behind Logster, an AI-driven endpoint threat detection platform that the team behind these articles builds. Here is what it offers for each requirement, and where the approach has limits.

Logster connects the activity on each Windows and Linux endpoint into a single picture of what happened, so related actions are judged together rather than one event at a time. Each verdict comes with a written explanation of how it was reached.

Grounding the verdict. When an analyst investigates an attack verdict, each step of the AI model's explanation is linked to the specific processes, files and connections it refers to. Clicking a step highlights them. The linking annotates the verdict; it never changes it. The prose now points to evidence, and the analyst can check whether that evidence says what the prose claims.

Scoping the silence. Analysts suppress known-benign activity by approving patterns of behaviour rather than names. A rule matches what it was written for, not everything that happens to resemble it, so a rule written for a vendor's updater doesn't accidentally cover a file dropped beside it. Anything broader has to be written explicitly, where a reviewer can see it.

The subtle part is what happens when approved activity changes. If the trusted updater suddenly launches a shell that nothing approved, that activity stays in view and is still assessed. The exclusion covers the behaviour it was written for and stops there.

Making silence visible. A new rule can be trialled before it takes effect: the team sees what it would have suppressed while nothing is hidden. Once it is live, every verdict records what was suppressed. Before a suppression goes live, and long after it does, the team can see exactly what it is hiding.

Failing honestly. If the AI model can't reach a verdict, Logster reports a failure, not a clean result. A finding set aside by tuning stays on record with the reason, so reviewers can still find it rather than losing it entirely.

Where the working still runs out

Showing your work does not guarantee the answer is right.

Linking an explanation to evidence ties it to real activity, but an explanation can cite real activity and still draw the wrong conclusion from it. The highlighting makes that easier to catch, not impossible to happen. Model confidence is not a calibrated probability until someone measures it against labelled data.

Any verdict also rests only on what was collected. If telemetry is missing or the activity that explains an event happened earlier than what was assessed, the verdict rests on less than the full picture. A narrow suppression rule is still only as good as the analyst's judgement about what counts as benign, and a pattern that is normal today can be abused tomorrow.

Architecture describes intent. Detection quality, false-positive rates and time saved still need to be measured against real attacks and real workloads.

Mark the paper

The next time you review your detection stack, grade both columns.

For a recent alert, ask: could an analyst verify it from the alert alone, or did they have to rebuild the story? Does every claim in the explanation point to a specific event?

For your suppressions, ask harder questions. Which of them match on a substring or wildcard? What would happen if an allowed process launched something new? When did anyone last look at what each one is hiding?

A detection system earns trust the way a student does: by showing work that holds up when someone checks it. That applies to the answers it gives and to the questions it has been told to stop asking.

When Every Event Looks Normal: Logster and the Context Gap in Endpoint Detection

Chad Ray — Mon, 14 Sep 2026 08:39:40 +0000

A PowerShell process starts. A script appears in a temporary directory. A scheduled task is registered. A connection goes out to an unfamiliar address.

An analyst could find a reasonable explanation for each of those actions. Software deployment, perhaps. A maintenance job. An administrator troubleshooting a workstation.

But were they part of the same operation? Did the process that downloaded the script also arrange for it to run at the next login? Was that process launched by the expected management agent, or by something that had no reason to start a shell?

Those questions change the investigation. The challenge is getting enough context to answer them before the activity disappears into another day of logs.

Key takeways

Legitimate tools can perform malicious actions; their names alone rarely settle an investigation.

Relationships between processes, files, and connections help distinguish routine work from suspicious behavior.

Automated analysis is useful when it preserves evidence and reduces the work needed to investigate a finding.

Familiar tools leave ambiguous evidence

Living off the land means abusing tools and capabilities already available in the environment. Their legitimate uses give attackers room to blend into routine administration.

CISA and its partners describe this detection challenge in their joint guidance, Identifying and Mitigating Living Off the Land Techniques. They emphasize detailed logging, established baselines, and correlation to distinguish malicious activity from legitimate operations. Recognizing an executable is only one part of understanding its use. (CISA and partners, February 2024)

The wider threat data reinforces why signatures alone are insufficient. CrowdStrike's 2026 Global Threat Report states that 82% of detections in 2025 were malware-free. That describes CrowdStrike's observed detections, not all attacks worldwide, and malware-free activity is broader than living off the land. It does, however, show why detection cannot depend entirely on finding a malicious file. (CrowdStrike, 2026 Global Threat Report)

Consider this hypothetical Windows sequence:

A process launches PowerShell with an encoded command.
PowerShell writes a script into a temporary directory.
A child process registers a scheduled task that references that script.
The same process tree contacts an external destination and queries domain accounts.

This is an illustration, not a captured incident. It also isn't proof of compromise: administration software can perform similar operations. Some of the individual actions may already trigger good detections.

What makes the sequence worth investigating is the connection between the actions. A newly written script, a persistence mechanism pointing to it, and related discovery activity give an analyst a more specific question than “Was PowerShell running?”

Collecting the events is only the beginning

A searchable log store can contain every relevant record and still leave considerable work for the analyst.

The process creation event may live in one view, the file write in another, and the network connection in a third. Before making a judgment, someone has to establish which process performed each action, reconstruct its ancestry, and align the timestamps. Missing identifiers or incomplete collection make that harder.

Existing tools already address parts of this problem. Event correlation is an established capability: Elastic's EQL rules, for example, can match ordered sequences and join events through shared fields such as process or host identifiers. It would be inaccurate to say that rules only understand isolated events. (Elastic, Event correlation rules)

The practical challenge is coverage. A correlation rule describes a relationship its author anticipated. Detection engineers still have to choose which sequences matter, tune them against legitimate operations, and maintain them as the environment changes.

A useful additional capability would preserve the surrounding behavior even when it doesn't match a prewritten sequence. That gives both automated analysis and the person reviewing its result more evidence to work with.

A process tree helps, but the activity extends beyond it

A behavior graph connects the entities involved in an operation. Processes, files, and network destinations become nodes; actions such as spawning, writing, and connecting become edges. Timestamps help retain the order in which those actions occurred.

The value becomes clearer when you compare the questions available at each level:

Recorded activity	Context that helps explain it
PowerShell started	Which parent launched it, with what command line and account?
A script was written	Which process wrote it, and was it subsequently executed?
A scheduled task was registered	What does it run, and how does that target relate to earlier activity?
An outbound connection occurred	Which process opened it, and what else did that process do?

A graph does not make these actions malicious. It makes their relationships available for analysis. A management agent carrying out an expected deployment and an unexpected shell arranging persistence may use overlapping tools while producing different surrounding behavior.

Nor does proximity prove causation. Two events on the same host within a minute are not automatically related. Reliable process identity, explicit file references, and observed parent-child relationships matter more than simply drawing lines between nearby timestamps. This also connects detection quality to collection quality.

Making the context available before an analyst opens the alert

Once activity has been organized into a graph, a detector can assess a connected set of actions. Rules, graph models, and language models offer different ways to do that; the representation and the method used to evaluate it are separate choices.

A language model can receive a textual representation of the graph alongside instructions about suspicious behavior and legitimate operating patterns. The intended benefit is contextual assessment: examining what the connected actions suggest, then returning a structured result that the surrounding system can process.

Logster is one implementation of this approach. It is an endpoint threat detection platform for SOC teams, with ingestion paths for Windows Sysmon and PowerShell logs and Linux auditd and eBPF telemetry. Its pipeline normalizes incoming records and builds activity graphs over per-endpoint windows.

In the current implementation, an LLM classifier evaluates the serialized graph and supplies the verdict, subject to configured thresholds. The platform also provides alerts, graph views, and process trees for investigation. This brings the connection between collection, detection, and investigation into the same workflow: the relationships assembled for analysis can also help the analyst understand the finding.

That is the useful idea here. Instead of beginning every investigation by reconstructing how disconnected records fit together, the analyst can start with an assembled account of the observed activity and test whether the conclusion is supported.

The architecture alone does not establish detection accuracy or time saved. Those outcomes need to be measured against representative attacks and legitimate workloads.

The context still has limits

Any verdict is bounded by the evidence the system collected and retained.

A short activity window may miss an earlier action that explains what happened next. An endpoint graph cannot, by itself, establish the full story of an operation spanning several hosts. Missing script content or incomplete process ancestry can leave crucial questions unanswered.

Model input limits add another constraint. Logster's serializer includes mechanisms to compact and truncate oversized graphs to fit the available input budget. A model may therefore receive less detail than the complete collected window. Evaluating what survives that reduction matters as much as evaluating the model's answer.

A language model can also produce a plausible explanation that the records do not support. Its reported confidence should not be treated as a calibrated probability of correctness without validation. An analyst needs a way to inspect the evidence behind the conclusion.

Operational failures deserve equally clear treatment. In Logster's current classifier resolution path, a failed model call or missing verdict is reported as an error. Separately, model attack verdicts that fall below the configured probability or confidence thresholds are demoted to benign, with a suppression reason recorded. Both behaviors matter when measuring coverage: an unavailable decision and a suppressed finding need to be visible during evaluation.

These constraints leave plenty of room for conventional controls. Known malicious artifacts and well-understood behaviors still warrant direct detection. Contextual analysis adds value when the ambiguity lies in how otherwise familiar actions fit together.

Evaluate the investigation, not just the alert

A practical evaluation should pair a suspicious sequence with a legitimate workflow that uses similar tools. A scripted software deployment is a useful comparison for a sequence involving script execution, downloads, and scheduled tasks.

Ask the system to show what distinguishes them. Then inspect the evidence:

Does it preserve the process ancestry and relevant command lines?
Can the reviewer follow a file from creation to later use?
Are the claimed relationships observed, or inferred only from timing?
What happens when activity crosses a window boundary or telemetry is missing?
How often does it flag legitimate work, miss the test attack, or fail to produce a verdict?

Measure time to a supported decision alongside detection results. An alert that names a technique but leaves the analyst to reconstruct everything may offer less operational value than its demo suggests.

The next time a detection fires on a familiar administrative tool, follow the actions around it. Check who launched it, what it changed, and what followed. A useful endpoint detection system should make that investigation easier to begin and easier to verify.

Tags: Endpoint Detection, Living off the Land, Security Operations, Behavioral Analysis, Logster