DEV Community

Cover image for Your evidence folder is complete. Meanwhile, someone just opened a shell in your container.
Arif Kurnaz
Arif Kurnaz

Posted on

Your evidence folder is complete. Meanwhile, someone just opened a shell in your container.

Since 11 September 2026, the EU Cyber Resilience Act asks you to report an actively exploited vulnerability within 24 hours. Twenty-four hours is enough to take away the comfort of "we'll collect the evidence before the audit."

I have watched a lot of teams in the same spot. Evidence collection runs like a project: the audit gets close, and out come the screenshots, the exports, the folder. A security incident does not wait for any of that. While the folder is being filled, something happens in the cluster, and the two never hear about each other.

This post is about why I see collecting evidence and catching a breach as two sides of the same job, and how that shaped the Falco integration in Attestkeep.

The frameworks already ask for this

In SOC 2, most of the technical evidence for a Kubernetes environment lands in CC6, CC7 and CC8. CC7 is system operations: detecting vulnerabilities, spotting anomalies, handling incidents. NIS2 Article 21 lists incident handling as a duty of its own. CRA puts a clock on reporting.

In an earlier post I wrote that all of these frameworks ask your cluster the same five questions:

  1. What is running?
  2. Where did it come from?
  3. What is wrong with it?
  4. Who allowed the exceptions?
  5. Can you prove all of the above for a period you didn't choose?

Those five end at the door. When something happens inside, two more join them:

  1. What did it do while it was running?
  2. What did you do about it?

The 3 a.m. alert

Falco sends an alert: a terminal shell was opened in a running container. The alert gives you the pod, the namespace and the rule that fired.

Then the questions start. Which image is this? Was it scanned? Was it signed? Did it come in clean, or under a warning? Who granted the exception? That information usually lives in another tool, on another screen, and sometimes nowhere at all.

Falco is very good at seeing what happens at runtime. Remembering how a workload got there is not its job. Admission remembers exactly that. As long as the two don't talk, the alert stays a symptom.

What I did not build

I did not write my own runtime detection. Falco has done that well for years. Attestkeep does not install Falco, does not configure it and does not write to it.

This is how I think about the split: Attestkeep is the gate, Falco is the patrol. The gate records who came in and on what grounds. The patrol sees what happens inside. The integration has one job, which is to tie what the patrol sees to the gate's record.

How it works

From a Falco alert to evidence: Falco sends the event over mutual TLS, Attestkeep checks the sender, ties the event to the image's admission record, matches a policy, then notifies, opens an issue or stops the pod, and everything lands in the evidence pack

It starts with two lines in the chart:

integrations:
  falco:
    enabled: true
    connection: mtls   # or: sidekick
Enter fullscreen mode Exit fullscreen mode

After that, every event goes through five steps:

  1. Falco sends the event, either straight from its own HTTP output or through Falcosidekick.
  2. Attestkeep checks the sender. The connection is mutual TLS, and Attestkeep issues the client certificate from its own CA. A sender without that certificate is turned away before its message is read. There is no shared token, so there is no token to leak.
  3. The event is tied to an image. Attestkeep reads the image from the event. If the event carries no digest, it uses the digest it last recorded for that name and tag. The alert now comes with its history: the scan result, the signature, the admission decision.
  4. A policy decides what happens. Send a notification, open an issue in GitHub, GitLab or Jira, or stop the pod.
  5. Everything is recorded: the event, the policy that matched, the action taken.

Every event is stored, not only the high-priority ones. An event that looks harmless at night can turn out to be the first link of a chain three weeks later.

Seeing is not enough

A runtime policy answers three questions: which Falco rules, in which namespaces, and what should happen. That is the whole form.

The interesting part is that a policy can be narrowed by the image's admission history. It can act only on unsigned images, or only on images that did not pass admission checks. The same shell alert deserves a notification on a signed, clean image. On an image that only got in under a warning, it deserves a stopped pod.

"Stop the pod" is an eviction, not a blunt delete. Pod disruption budgets are respected. You choose between stopping the pod at once and giving the application its own shutdown time. The permission exists only if you set rbac.allowEnforce: true in the chart. Attestkeep never stops pods in its own namespace. Every attempt, whether it succeeds or is refused, is written to the audit log with the pod's ID.

One thing I left out on purpose: nothing that happens at runtime goes back and changes an admission decision. The gate keeps its own rules, and the patrol adds context. Mixing the two produces a control that is hard to explain to an auditor.

New policies start switched off, so nothing surprising happens on the day you install.

The guard that never sleeps

This is the part I like most. Once a policy is written, it does not wait for anyone to wake up. If the shell Falco sees at 3 a.m. matches your policy, the pod is stopped right then, and the attempt is on the record. The guard does not sleep, does not get tired and does not take holidays.

It does not think either. It does exactly what Falco reports and what your policy says, nothing more. Two things I want to be clear about:

  • If the stopped pod belongs to a Deployment or another controller, Kubernetes creates a new one, and the new pod goes through admission again. Since runtime does not change admission decisions, the same image is let in again. Stopping the pod cuts off the session the attacker opened. Fixing or refusing the image is still your job.
  • The guard's eyes are Falco's. What Falco does not see, the guard does not see.

Evidence as a by-product

Attestkeep's evidence pack already carried admission decisions. It now has a runtime section with:

  • the policy set in force for the period, and its hash,
  • the decisions the policies made,
  • audit entries, including pod stop attempts,
  • optionally, the raw events.

None of this was prepared for the audit. It exists because the control ran. When the auditor asks "what did you do about this incident?", the answer comes out as one story: this image, with this signature, admitted under this policy; then this event, this policy matched, the pod was stopped.

Writing Falco rules in the console (beta)

Good policies need good rules, so the console has a rule editor. You write the rule, and Attestkeep validates it against the Falco version already running in your cluster. Publish pushes it to your own registry as an artifact that falcoctl can follow. Your Falco configuration is still left alone.

This part is beta, and it is labelled that way.

Limits

  • No Falco, no runtime visibility. Attestkeep holds the gate, and Falco sees inside.
  • The supported version is Falco 0.45.0 (Falco chart 9.2.0). I make no claims about older versions.
  • The console form is limited to rules, namespaces and actions. Finer conditions on image, priority or tags are API-only for now.
  • This integration is not an incident response process. Who gets woken up, who gets told, and whose signature goes on the CRA notification are your process. Attestkeep gives you the record you need in the first minute of it.

Closing

Whatever you use, make evidence a by-product of the control running instead of a project you run before the audit. The same goes for incidents: the moment you see the breach is the moment your evidence is made.

I build Attestkeep, a Kubernetes admission controller that scans images, verifies signatures, enforces policy at admission and keeps signed evidence that the control ran. Everything in this post is in Attestkeep 1.3.2, the current release. The Falco integration is in every edition, Community included. The setup guide is on docs.attestkeep.com, under "Runtime events (Falco)".

It is early. If you run Falco in production and have had to carry its events to an auditor, and something above is wrong, I would really like to hear it.

Top comments (0)