DEV Community: Eliot Ferstl

One False Alert and Zero False Autonomous Actions

Eliot Ferstl — Mon, 31 Aug 2026 20:27:35 +0000

Last week, we published the scoring rules for this soak before we had any results. The run has closed. Here are the numbers, graded against that post.

7 days, 94 protected pods, 14 namespaces, 0 attacks deployed. 1 signed evidence record. 0 detector trips, 0 isolations, 0 terminations. Against the pre-registered bars: drift-plan evidence at 0.0000637 per pod-hour against a ceiling of 0.1, isolations and terminations at true zero, not rounded. All four bars passed, the evidence bar with roughly 1,570x margin.

The denominator is 15,690 online pod-hours out of 15,792 gross. We threw away 101 pod-hours of cold start and post-churn relearn, exactly as promised, because counting hours in which the sidecar deliberately cannot act inflates the denominator and flatters the rate. Churn was scored by observed effect rather than job success: pods recreated, pods killed, sidecars restarted on a 12-hour rotation, confirmed by counting distinct container ids instead of trusting that the cron reported healthy.

The single detection is worth reproducing. A redis pod tripped cumulative drift at 0.0792 against a learned threshold of 0.0351. It stayed inside the long-scale envelope, the ML plane returned normal, and the event was capped at alert: evidence emitted, no response chain, no trip counted. That is the two-factor gate doing on benign traffic exactly what it exists for.

Cohorts stayed separate, as promised. The out-of-the-box group logged 8,346 pod-hours with zero trips, the integrity-armed group 7,178, also zero. Of the six armed workloads, three needed a uid0 and SYS_PTRACE grant some customers will not make, so “armed” should not be read as “default”. The wrong-zero check also came back clean: every trip was claimed by a named event class and the residual was zero, meaning no events were hiding from the count under an unrecognized label.

One finding constrains all of it. Only 23 of 94 pods carried all five process-level signals live, and in the out-of-the-box configuration the executable-link signal was dark on every pod. Fewer live signals means fewer opportunities to fire, so reduced coverage biases the false positive rate downward. The zero is real, but it was measured on a fleet where most pods could not see much at the process layer.

Provenance is the same story in stronger form: it contributed zero because it was blind, not because it was quiet. Manifest presence and maps readability both read zero on all 14 workloads for the entire window, so that plane cannot be cited as a clean result at all. And as flagged in advance, the terminate zero is partly structural, since statistical events are capped below termination by design.

Two more things we said we would not bury. Sidecar CPU ran a median near 110 millicores per pod with memory around 5 MiB. And the one evidence record we produced has no durable storage anywhere in the product today: it was captured by hand before teardown, and its Kubernetes event aged out within the hour. A campaign that emits evidence it cannot keep has a numerator it cannot audit. That is now a work item.

The next run closes these in order: signal coverage first, then evidence durability. The numbers above are what a strict denominator and a partly blind fleet produced, and the definitions they were scored against were published before anyone here had seen them.

The Rules We Use To Define False Positives

Eliot Ferstl — Wed, 26 Aug 2026 19:09:36 +0000

pandocore.io

A false positive rate without an event definition, a denominator, and a labeling method is marketing. We wrote that in our first post. This is the other half: our largest soak to date is running right now. 94 protected pods across 14 namespaces, seven days, mostly stock upstream images (nginx, Postgres, Mongo, Kafka, Elasticsearch, RabbitMQ, plus JVM, Go, Python, and Node services). The results aren't in. So we're publishing the scoring rules first. Whatever the numbers turn out to be, they get graded against what's written here, in a post that went out before we saw them.

The run deploys zero attacks. That's the entire labeling method: every trip in this window is by definition a false positive. No adjudication, no “we reviewed each alert and determined.” Nothing gets subtracted.

Four counters, never blended: a detector fired; a signed evidence record was produced; an isolation was actually applied; a pod was actually terminated. Those are different failures with different costs, so the pass bars sit on different counters. Alerting is judged on evidence, response is judged on actions. The bars: zero false terminations, at most 0.1 false evidence records per pod-hour on the statistical plane, at most 0.01 false isolations per pod-hour. One caveat stated up front: zero false terminations is partly structural. Statistical events are capped below termination by design, so part of that zero is architecture, not model quality.

The denominator is where FP numbers usually get flattered, so ours is strict: we count only pod-hours where the detection ensemble was actually online. Cold-start hours are excluded. The sidecar deliberately can't take action while warming, and counting hours where the measured event is impossible inflates the denominator. Churn is part of the campaign, not an excuse: every 12 hours the run recreates pods, kills pods, and restarts sidecars on a rotation, and every post-churn relearn window is excluded too. Both exclusions shrink the denominator. Both make our reported rate worse. That's the direction the arithmetic should err.

Also stated now, because it would be conspicuous later: part of the fleet runs out-of-the-box config and part runs with integrity baselining armed. Of the armed group, half needed a privilege grant most customers won't make, so those cohorts get reported separately. “Armed” doesn't mean “default.” And the measured build is the released chart plus a staging-signed sidecar carrying the same detector code as the release. Same code, different artifact.

The failure mode we worry about most on a clean run is a wrong zero. Deploy no attacks and a broken counter looks identical to a good result: mislabel one event class and it reads 0.000 forever, which looks like success. So the analyzer refuses unclaimed events. Every trip must be claimed by a named event class, and any remainder means the taxonomy has a hole, not that the product is clean. We have caught real mislabelings exactly this way. A zero we can't cross-check doesn't get published.

The numbers go out when the run completes, graded against this post. Per-workload spread included, ugly parts included. Until then: if a vendor hands you a false positive rate, ask for their event definition, their denominator, and how they'd know if their zero was wrong.

Why We Ship Our Security Webhook Fail-Open

Eliot Ferstl — Thu, 20 Aug 2026 16:22:44 +0000

pandocore.io

Anything that installs an admission webhook into your cluster puts itself in the critical path of pod creation: service meshes, secret injectors, policy engines, security agents. One field in that config, failurePolicy, decides what happens when the vendor's code goes down. Most install guides don't mention it.

The textbook answer is fail-closed: failurePolicy: Fail, opt-in namespaces, an objectSelector so only labeled pods ever touch the webhook. PandoCore ships the opposite: Ignore, watching every namespace you haven't excluded. On purpose.

Here's what Fail actually does. The webhook goes unreachable, crashed, evicted, mid-upgrade, slow past its timeout, and the API server rejects every matched pod creation until it's back. That doesn't look like a security incident. It looks like your cluster breaking: ReplicaSets throwing FailedCreate, deploys hanging, the autoscaler stuck, drained nodes with nowhere to put workloads. A protection tool that can freeze pod scheduling is a bigger operational risk than most of what it protects you from. If we're down, your cluster keeps working.

The cost of Ignore is real: during an outage, pods get admitted without the sidecar. Silently. For a security product, that's the worst failure there is; workloads look protected and aren't. A mesh pod missing its sidecar breaks loudly: no mTLS, no routing. A security sidecar missing is invisible: the workload runs perfectly and nothing pages. Fail-open is survivable for a mesh; here it's dangerous unless you engineer the alarm yourself. So the actual engineering isn't picking the field value. It's making the silent failure loud. That's why there's no objectSelector: our webhook sees every pod in a protected namespace, not just labeled ones, so it can flag the pods that should be protected and aren't. Each one raises a Kubernetes warning event and increments a Prometheus counter you can alert on. Scope the webhook down to labeled pods and that signal goes blind. The wide match is the detection.

Blast radius gets bounded at a different layer instead. kube-system and friends are excluded by default. Our own namespace is excluded unconditionally (you can't remove it in values), so combined with Ignore, the webhook can never block recreation of its own pods. Two replicas, a PodDisruptionBudget, anti-affinity across nodes, an explicit timeout. One gap, stated plainly: default-on alerting for when the webhook itself is down. The case where it can't count what it never saw isn't shipped yet. A single Prometheus absence rule on pando_webhook_admissions_total closes it.

Whatever vendor you're evaluating, ask them: which failurePolicy do you ship, and where's the reasoning written down? What limits the damage when your webhook misbehaves? Can it block its own recovery? How do I find out you're skipping pods? How do I find out you're gone? We'd rather you ask us.

Three Silent Failures Between You and Sidecar Injection

Eliot Ferstl — Tue, 18 Aug 2026 20:40:03 +0000

Running kubectl label deployment my-app pandocore.io/inject=enabled modifies the Deployment's top-level metadata.labels, not spec.template.metadata.labels. Those are two different things, and the difference is the bug.

Sidecar injection works through a mutating admission webhook that fires when a pod is created. The webhook matches on the pod's labels. Pods inherit their labels from exactly one place: the Deployment's pod template. Labels on the Deployment object itself never propagate down to pods. They exist so you can select the Deployment with things like kubectl get deploy -l team=payments.

So when you label the Deployment, three things go wrong at once, all silently. First, kubectl prints deployment.apps/my-app labeled. Success. Second, because the pod template didn't change, the Deployment controller sees nothing to do, so no rollout happens and no new pods are created. Third, even if pods were recreated, they wouldn't carry the label, so the webhook would never match them. There is no error at any layer. You check the Deployment, the label is right there, and every pod is still running 1/1 with no sidecar in sight.

We hit this while reviewing our own onboarding docs. The instruction looked correct, the command succeeded, and injection never happened. It's an easy mistake to ship because nothing anywhere tells you it didn't work.

The fix is to put the label where pods are actually born: the pod template.

kubectl patch deployment my-app --type merge \
  -p '{"spec":{"template":{"metadata":{"labels":{"pandocore.io/inject":"enabled"}}}}}'

Because this changes the pod template, it triggers a rollout on its own. The controller creates a new ReplicaSet, and every new pod comes up with the label and passes through the webhook. If you set the label some other way and aren't sure the template has changed, kubectl rollout restart deployment my-app forces fresh pods. Verify with kubectl get pods -l pandocore.io/inject=enabled. The containers column should now read 2/2.

Label the template, roll the pods, check the count. The Deployment's own labels were never going to do it.

Running kubectl label deployment my-app pandocore.io/inject=enabled modifies the Deployment's top-level metadata.labels, not spec.template.metadata.labels. Those are two different things, and the difference is the bug.

So when you label the Deployment, three things go wrong at once, all silently. First, kubectl prints deployment.apps/my-app labeled. Success. Second, because the pod template didn't change, the Deployment controller sees nothing to do, so no rollout happens and no new pods are created. Third, even if pods were recreated, they wouldn't carry the label, so the webhook would never match them. There is no error at any layer. You check the Deployment, the label is right there, and every pod is still running 1/1 with no sidecar in sight.

The fix is to put the label where pods are actually born: the pod template.

kubectl patch deployment my-app --type merge \
-p '{"spec":{"template":{"metadata":{"labels":{"pandocore.io/inject":"enabled"}}}}}'
Because this changes the pod template, it triggers a rollout on its own. The controller creates a new ReplicaSet, and every new pod comes up with the label and passes through the webhook. If you set the label some other way and aren't sure the template has changed, kubectl rollout restart deployment my-app forces fresh pods. Verify with kubectl get pods -l pandocore.io/inject=enabled. The containers column should now read 2/2.

Label the template, roll the pods, check the count. The Deployment's own labels were never going to do it.

pandocore.io