You have just switched Tetragon to enforce mode on a production cluster. A few minutes later your phone lights up. Four alerts, on a single payment service.
A gcc compiler ran inside the container. The service mesh sidecar spawned processes and opened connections. A nightly migration job started a shell script. And an on-call engineer opened a kubectl exec to understand an incident.
Four alerts, all legitimate in some sense. The natural reflex, allowlist everything for some peace, is the worst thing to do for three of them.
We are taught to detect, not to triage
The available Tetragon content teaches you to cast a wide net. Install, write a first TracingPolicy, capture every exec, replay a container escape demo, switch to enforce. It is useful and it is already very well covered.
The problem is that it stops there. Nobody teaches triage. False positives are treated as folklore you swap with peers, not as an engineering problem. You endure them, you comment on them, you do not model them. And a noisy detection always ends the same way, ignored then disabled. A disabled rule detects nothing.
The flip
Here is the idea that changes everything. A detection rule is a hypothesis about what is normal. A false positive is therefore not a flaw in the tool, it is proof that your hypothesis of normal is wrong or incomplete.
Noise is not noise. It is information about your model. The right question is not how to filter this alert, it is what the alert tells you about what you believed was normal.
One symptom, four opposite treatments
Back to the payment service. The four alerts look like the same symptom, an unexpected exec. They have nothing in common.
The gcc should not be there. It is an image defect, a missing multi-stage build. You fix it at the source, you never allowlist it. Allowlisting a compiler in prod hands a tool to an attacker.
The mesh sidecar is doing exactly its job. Stable and permanent behavior, to be described as normal, by workload class and not by pod.
The migration job is legitimate but transient. You allow it within its lifecycle window, not in steady state.
The kubectl exec is legitimate but human and unpredictable. You never allowlist an interactive shell, that leaves the door wide open. You attribute the action to a person and you audit it.
Same symptom, four opposite treatments. Triage is not a detail, it is the job.
Stop guessing
The good news, you do not have to imagine the list of normal behaviors. You can observe it.
Tetragon runs in observation by default, a policy only blocks if you add an action to it. There is no magic switch between seeing and blocking. So you let it run in observation over a representative window, two weeks to catch the weekly cycles and the batch jobs, then you generate the allowlist from the behavior actually observed, aggregated by workload class.
Then you climb the rungs one at a time. Observe, alert only, permissive, narrow block. One workload class at a time, never all at once. It is the same logic as seccomp or AppArmor profile generation, record then permissive then enforce, carried over to runtime security.
What baselining does not tell you
Two traps, which I detail in the full module.
Observed is not approved. Automatic generation can perfectly well capture a behavior you absolutely do not want to make permanent, the gcc for example. A human validates before freezing it.
And baselining an already compromised environment bakes the attacker into normal. The observation window assumes a healthy environment, a hypothesis to verify, not to presume.
In the end the allowlist becomes an audit artifact, dated and justified. Because an auditor does not only ask what you detect, they ask why you allow what you allow.
What comes next
This article is a taste. It belongs to a series on the blind spots of Tetragon training, the topics the beginner content does not cover. The ground of install, first policies, escape demo, observe then enforce is saturated. The real questions start after.
Each angle has its talk then its lab, on the same rhythm. This one, signal engineering, treats false positives as a measurable discipline, with a taxonomy, baselining and signal quality indicators.
Which false positive has already cost you a night and how did you handle it?
Top comments (0)