DEV Community

SelimBF
SelimBF

Posted on Originally published at getruptura.dev

Predict Kubernetes OOM kills 30 minutes ahead with one PromQL alert

#kubernetes #prometheus #sre #devops

Your memory alert fires at 90%. The pod is killed 60 seconds later. You read the alert after the restart.

The problem is not the number. A static threshold answers "is memory high?", when the question you care about is "will this container hit its limit, and when?" Prometheus can already answer a first version of that question with one function: predict_linear.

This post gives you a copy-paste alert that warns before an OOM kill, explains every part of it, and is honest about where it breaks.

The alert 

groups:
- name: memory-forecast
  rules:
  - alert: ContainerMemoryForecastOOM
    expr: |
      max by (namespace, pod, container) (
        predict_linear(container_memory_working_set_bytes{container!="", container!="POD"}[30m], 30 * 60)
      )
      >
      max by (namespace, pod, container) (
        kube_pod_container_resource_limits{resource="memory"}
      )
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is forecast to reach its memory limit within 30 minutes"
Enter fullscreen mode Exit fullscreen mode

It works on any cluster that runs cAdvisor metrics and kube-state-metrics, which includes the OpenShift monitoring stack out of the box.

How it works

container_memory_working_set_bytes is the memory the kernel actually counts against the container's cgroup limit, the number the OOM killer looks at. container!="" and container!="POD" drop the pod-level and pause-container series.

predict_linear(...[30m], 30 * 60) fits a straight line through the last 30 minutes of samples and extrapolates it 30 minutes (1800 seconds) into the future.

kube_pod_container_resource_limits{resource="memory"} is the configured limit, exposed by kube-state-metrics.

max by (namespace, pod, container) on both sides gives each side exactly one series per container, so the comparison matches cleanly and does not fail on extra labels.

for: 10m asks for the forecast to stay above the limit for 10 minutes before firing, which filters out short spikes.

Read the result as: "if memory keeps growing at the pace of the last 30 minutes, this container hits its limit within the next 30 minutes."

Where it breaks

This rule is a real improvement over a 90% threshold. It also has blind spots you should know before trusting it.

  1. No limit, no alert. A container without a memory limit has no right-hand side, so it is silently excluded.
  2. Restarts reset the line. After an OOM kill, memory drops to its baseline and the fit starts over. A leak that kills every 40 minutes can stay just below the 30-minute window forever.
  3. Garbage collectors look like leaks. JVM, Go and Node heaps grow in a sawtooth. Over a short window, the rising edge of a sawtooth extrapolates straight into the limit. A longer window helps, but makes real leaks slower to detect.
  4. One horizon for everything. A cache that should sit near its limit and a stateless API that should not are judged with the same 30 minutes and the same straight line.
  5. It does not tell you why, or who is affected. Was it the new release? Is it one replica or all of them? Which services call this pod and will fail with it? The alert cannot say.
  6. No confidence. A single number, with no range, hides how stable the trend actually is.

Going further

Those blind spots are why I built Ruptura, an open-source operator for Kubernetes and OpenShift. It runs on top of the same Prometheus metrics and, for each workload:

  • forecasts the time left with a range, against the workload's own history rather than one global window;
  • separates predicted from detected failures and reports what it cannot see, such as a container without a limit;
  • names the one replica that leaks while its siblings stay flat, and flags a release that leaks faster than the one it replaced;
  • lists which callers a failing service can take down;
  • sends one structured result per failure to Alertmanager, including the OpenShift cluster Alertmanager.

It is early and not yet recommended for production. If you try either the rule above or Ruptura, I would like to hear what you find: GitHub Discussions.

A 4-minute overview of the idea:

Top comments (0)