Falco on Kubernetes: install, tune, and route alerts

You scan images in CI, sign them before deploy, and gate manifests at admission. Every one of those controls answers the same question: should this workload be allowed to start? None of them can tell you what it did at 02:00 once it was running. Someone opening a shell in a production pod, a binary that was never in the image suddenly executing, a process reading /etc/shadow for no good reason. Those are runtime events, and runtime is where Falco lives.

The catch is that Falco tutorials age faster than most. The project has dropped its legacy eBPF probe, its gVisor engine and its gRPC output in recent releases, deprecated the append: true rule syntax in favour of override:, and moved rule distribution onto OCI artifacts pulled by a sidecar. A lot of the snippets still ranking on the first page will not load on a current install. So this walkthrough sticks to what the current chart and docs actually do: install, trigger one stock alert, write one custom rule, and then spend most of the time on tuning, which is the part that decides whether Falco is still switched on in six months.

Where runtime detection fits

A Kubernetes security stack has layers, and each one answers a different question:

• image scanning: what went into the artefact
• signing and provenance: whether you trust where it came from
• admission policy: whether the cluster should accept it
• runtime detection: what the workload is doing now

The last layer is the one teams skip until something odd is already running in production. Falco fills it by watching syscalls from the kernel, matching them against a rules engine in userspace, and emitting an alert when a rule fires. If you want the eBPF side of that story in more depth, the Cilium networking post covers how the same machinery is used for the network path.

How Falco collects events, and where it differs from Tetragon

Search for "Falco vs eBPF" and you will find a lot of confused threads. Falco is not an alternative to eBPF. It uses eBPF as one way of collecting kernel events, and there are two supported drivers today:

• modern_ebpf, a CO-RE probe embedded in the binary, which is what Falco itself defaults to
• kmod, the kernel module, still supported and still the right answer on kernels the modern probe cannot run on

The Helm chart adds a third value, auto, which is its default. It tries the modern probe first and falls back to the kernel module. The older ebpf and gvisor driver kinds are gone: the chart will not even render if you pass either, which is a kinder failure than a pod that starts and silently sees nothing.

Tetragon, from the Cilium project, makes a different trade-off. It leans into in-kernel enforcement: a tracing policy can override a kernel function's return value or send SIGKILL to the offending process before it gets any further. Falco keeps the decision in userspace, with a mature rules engine, a maintained default ruleset, and a lot of flexibility around outputs and tuning. If your first question is "what happened inside that container?", Falco is the more natural place to start. If it is "stop that from ever completing", look at Tetragon.

Install Falco with the Helm chart

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
  --create-namespace \
  --namespace falco \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true \
  --set collectors.kubernetes.enabled=true

Those three flags are worth understanding rather than copying:

• falcosidekick.enabled=true deploys Falcosidekick and switches Falco to JSON output over HTTP, so alerts can be forwarded to systems people already watch
• falcosidekick.webui.enabled=true adds the web UI and a Redis instance behind it, which is handy while you are testing and worth turning off once alerts flow somewhere durable
• collectors.kubernetes.enabled=true deploys the metadata collector and the k8smeta plugin, which adds the owning Deployment, ReplicaSet and Service to each alert. Without it you still get the pod name, namespace and labels, because Falco reads those from the container runtime directly

The chart also deploys two containers you did not ask for by name: a falcoctl init container that pulls the default rules as an OCI artifact, and a falcoctl sidecar that checks for a newer artifact once a week and swaps it in. That has consequences for tuning, which we will come back to.

If you need to be explicit about the driver, use one of the two current values:

--set driver.kind=modern_ebpf   # or driver.kind=kmod

Then watch the Falco container start. The pod has several containers now, so name the one you want:

kubectl -n falco logs ds/falco -c falco

You are looking for a clean driver load and a Loading rules from file line for each rules file. If the driver is wrong for the kernel underneath, this is where you find out.

Trigger one stock alert first

Before writing anything custom, prove the default path works. Start a throwaway pod:

kubectl run runtime-test --image=busybox --restart=Never -- sleep 3600

Then exec into it with a terminal attached:

kubectl exec -it runtime-test -- sh

That fires the stock Terminal shell in container rule. The -t is doing real work here: the rule checks that the shell has a TTY, so kubectl exec runtime-test -- sh -c 'id' without one will not trigger it. Check the Falco logs or the Falcosidekick UI and you should see the alert with container_id and container_name appended to the end of the output line. Those fields are not in the rule's own output string; the container plugin suggests them and Falco appends suggested fields automatically.

This is also the moment to reset expectations about volume. The stock falco_rules.yaml is a small, conservative file: 25 stable rules at the time of writing, every one enabled, none of them experimental. The noisy reputation comes from the incubating and sandbox rulesets, which the chart does not load by default, and from teams tuning carelessly once they do.

Write one custom rule

The simplest useful rule is one you can explain in a sentence: tell me if a miner-like binary starts inside a container. With the chart, custom rules go in the customRules value, and each key becomes a file under /etc/falco/rules.d/:

customRules:
  miners.yaml: |-
    - rule: Crypto miner process in container
      desc: Detect common miner binaries starting inside a container
      condition: >
        spawned_process and container and
        proc.name = anyof (xmrig, minerd, cryptominer)
      output: >
        Crypto miner process in container | user=%user.name
        process=%proc.name command=%proc.cmdline
      priority: WARNING
      tags: [container, malware, crypto-miner]

Two things about that condition. spawned_process and container are macros from the stock file, so you get exec events inside containers for free. And anyof is a comparator modifier from the current rule language: one field compared against a short list, without a chain of or clauses. It sits after the operator, with a space either side and the values in parentheses.

One trap that is not in the tutorials: proc.name is the kernel's process name, and the kernel truncates it to 15 characters. The stock rules file depends on this, which is why you will find entries like mysql_install_d in its lists. If your binary has a long name, match on proc.exepath or container.image.repository instead.

Load order matters. The default rules_files list reads falco_rules.yaml, then falco_rules.local.yaml, then everything in rules.d, and a file that overrides a stock rule has to load after it. Give the file a .yaml or .yml extension too, because anything else in that directory is ignored.

Before you apply, validate:

falco -V miners.yaml

That parses and compiles the rules file and exits, and the easiest place to run it is the same Falco image tag you deploy, so the engine version matches. It is worth the thirty seconds, because a rules error is fatal. Falco stops at the first file that fails to load, prints the reason, and exits non-zero, which on Kubernetes means the whole DaemonSet goes into CrashLoopBackOff over one bad line in one file. Warnings are logged and tolerated. Errors are not.

Tuning is where most Falco deployments succeed or fail

Install is the easy part. Tuning decides whether people trust the alerts next month, and the stock rules file is built for it in a specific way that most tutorials, including the official exceptions examples, obscure.

The fact the docs bury is that none of the stable stock rules ships an exceptions: block. The docs' canonical example appends values to an exception on Write below binary dir, and that rule lives in the sandbox ruleset, not the one you just installed. Copy it against a stable rule and the loader rejects it, because there is no existing exception to inherit fields from.

What the stable rules give you instead is a set of hooks with names starting user_, each defined as (never_true), plus empty image lists. Take Contact K8S API Server From Container, which fires the first time an operator or controller you run outside kube-system talks to the API server. Its condition ends with and not user_known_contact_k8s_api_server_activities, and that macro is yours to replace:

- macro: user_known_contact_k8s_api_server_activities
  condition: >
    (k8s.ns.name = argocd and
     container.image.repository = quay.io/argoproj/argocd)
  override:
    condition: replace

The same shape works for the other stable rules. Terminal shell in container has user_expected_terminal_shell_in_container_conditions. Read sensitive file untrusted has a read_sensitive_file_images list you can append to:

- list: read_sensitive_file_images
  items: [quay.io/argoproj/argocd]
  override:
    items: append

If you would rather use exceptions, you can. Appending a new named exception to a stable rule is supported as long as you supply fields alongside values:

- rule: Contact K8S API Server From Container
  exceptions:
    - name: argocd_controllers
      fields: [container.image.repository, k8s.ns.name]
      values:
        - [quay.io/argoproj/argocd, argocd]
  override:
    exceptions: append

Falco adds and not (container.image.repository = ... and k8s.ns.name = ...) to the condition for you. Either approach beats disabling the rule, because the rule is still right for every other workload.

When a rule really does not fit your threat model, turn it off explicitly:

- rule: Drop and execute new binary in container
  enabled: false
  override:
    enabled: replace

The override: block is the part older posts get wrong. It states, per key, whether you are appending to or replacing the stock definition. append: true still parses for now but is deprecated, so translate any snippet built around it before you paste.

One more thing the file already did for you: the API-server rule's k8s_containers macro exempts everything in kube-system. If it is firing, the workload is somewhere else.

Route alerts somewhere people already look

An alert that only exists in one DaemonSet's log stream changes nothing. Falcosidekick takes Falco's JSON output and fans it out to Slack, Loki, Elasticsearch, Splunk, Datadog, Alertmanager and a long list beyond that. The chart values map directly to its outputs:

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ
      minimumpriority: warning
    loki:
      hostport: http://loki-gateway.monitoring:80

There is no global minimum priority in Falcosidekick. Each output has its own minimumpriority, and its severity ladder spells the level informational rather than Falco's info, which catches people out when a filter silently matches nothing.

The pattern that holds up on platform teams is a split by urgency: Slack or Teams for the handful of rules that need a human now, a log backend such as Loki for searchable history, and the rest filtered out at the output. If Grafana is already your front door, the Grafana monitoring post covers getting Grafana itself stood up and the first dashboards in place.

Operating it after week one

The first three sections get Falco running. These are the three things that go wrong once it has been running for a while.

Your rules are floating. The falcoctl sidecar follows a major-version tag of the rules artifact and installs whatever is newest every 168 hours, into an emptyDir mounted over /etc/falco. Two consequences: an edited falco_rules.yaml inside the pod is overwritten on the next restart or the next follow, and a rule you tuned by name can change underneath you on a Tuesday. Keep every override in customRules, and pin falcoctl.config.artifact.install.refs and follow.refs to an exact artifact tag or an @sha256: digest once you are past the exploring stage.

You are dropping events and nobody knows. Falco emits an internal alert named Falco internal: syscall event drop when the kernel ring buffer overflows, but it is a debug-priority alert rate-limited to one message every 30 seconds. Raise Falco's minimum priority to cut noise, or set minimumpriority: warning on your Slack output, and it vanishes. The durable answer is metrics:

--set metrics.enabled=true \
--set serviceMonitor.create=true

That turns on Falco's Prometheus endpoint and a ServiceMonitor for it. The two series to graph against each other are falcosecurity_scap_n_drops_total and falcosecurity_scap_n_evts_total. If drops climb with load, the documented remedies are a bigger ring buffer via engine.modern_ebpf.buf_size_preset, fewer CPUs per buffer via engine.modern_ebpf.cpus_for_each_buffer, and base_syscalls to narrow what the driver captures. Which of those helps depends on your syscall mix, which is also why there is no universal "Falco overhead" figure worth quoting: it tracks what your workloads do, not a percentage.

You want to act, not just alert. Falco itself only detects. Falco Talon is the incubating response engine in the same ecosystem: it receives events from Falcosidekick's talon output and applies rules such as terminating the pod or labelling it for quarantine. Its Kubernetes actions need k8s.pod.name and k8s.ns.name among the event's output fields, which the stock rule outputs do not carry, so expect to append those to any rule you want it to act on. It is receiving commits but has had long gaps between tagged releases, so treat it as something to evaluate rather than a default, and keep the enforcement question in mind when you weigh it against Tetragon.

The failure you will hit first

Almost every first Falco tuning session ends one of two ways. Either the DaemonSet is crash-looping, in which case the Falco container's log has the offending file and line near the top and falco -V would have told you first. Or the pods are healthy and the override seems to do nothing. For the second, check three things before you touch the rule. Confirm the file landed in /etc/falco/rules.d/ with a .yaml extension and appears in the Loading rules from file lines. Check the rule name you are overriding still exists with that exact spelling in the ruleset the falcoctl sidecar just installed. Then check whether the workload is already exempt, the way kube-system is for the API-server rule. Usually the rule was fine and the plumbing was not.