DEV Community

Cover image for Kubernetes events disappear after an hour
Abhishek Jaiswal
Abhishek Jaiswal

Posted on Originally published at ownkube.io

Kubernetes events disappear after an hour

#observability #devops #monitoring #kubernetes

kubectl get events is the first place most people look when a pod misbehaves. Kubernetes events tell you exactly what's happening in your cluster (pod scheduling failures, image pull errors, OOM kills), but by default they disappear from etcd after 60 minutes.

This guide walks through why that's a problem and how to export events to the tools your team already uses.

What are Kubernetes events?

Every time something happens in your cluster, Kubernetes creates an Event object. A pod gets scheduled, a container crashes, a volume fails to mount. Events are first-class API resources, same as Pods or Deployments.

Here's what one looks like:

apiVersion: v1
kind: Event
metadata:
  name: my-app-pod.17a6f8e2c
  namespace: production
reason: BackOff
message: "Back-off restarting failed container"
type: Warning
involvedObject:
  kind: Pod
  name: my-app-pod
  namespace: production
count: 15
firstTimestamp: "2026-03-07T10:00:00Z"
lastTimestamp: "2026-03-07T10:15:00Z"
Enter fullscreen mode Exit fullscreen mode

The useful fields:

  • Reason: the machine-readable cause (BackOff, FailedScheduling, Unhealthy, Pulling, Killing)
  • Type: either Normal or Warning
  • Message: a human-readable description of what happened
  • InvolvedObject: the resource this event is about
  • Count: how many times this event has occurred

Why the defaults are a problem

Three things make stock Kubernetes events nearly useless for debugging:

Events expire after 1 hour

The API server retains events for 60 minutes, so the record of a pod crash at 2 AM is gone by 9 AM.

No search or aggregation

kubectl get events gives you a flat, unsorted list with no cross-namespace search, severity filtering or time correlation.

No alerting

Kubernetes won't tell you when something goes wrong. A FailedScheduling event could fire hundreds of times before anyone notices, unless someone happens to be watching.

Export events to external systems

The fix is an event exporter: a lightweight component that watches the Kubernetes event stream via the API server and forwards events to external destinations (sinks) like Elasticsearch, Slack, Loki, Kafka, or webhooks.

The pipeline:

Kubernetes event exporter pipeline: the API server streams events to a SharedInformer-backed watcher, which passes them through routing rules that filter, match, and drop, then fans out to sinks like Slack, Elasticsearch, Loki, and Kafka/webhooks.

The exporter uses a SharedInformer to watch events without polling, applies configurable routing rules, and delivers them to one or more sinks.

Setting up the exporter

The Ownkube Kubernetes Events Exporter is a fork we maintain. The upstream project by Opsgenie (later Resmo) has been dormant since 2023 and has some bugs that cause silent event loss, covered in the last section.

Install with Helm 

helm install event-exporter \
  oci://ghcr.io/ownkube/charts/kubernetes-events-exporter \
  --namespace monitoring \
  --create-namespace
Enter fullscreen mode Exit fullscreen mode

Basic configuration

The exporter takes a YAML config (deployed as a ConfigMap). A minimal config that sends everything to stdout:

logLevel: info
route:
  routes:
    - match:
        - receiver: "stdout"
receivers:
  - name: "stdout"
    stdout: {}
Enter fullscreen mode Exit fullscreen mode

Routing

You can filter and direct events to different sinks based on any event property: namespace, event type, reason, involved object kind, etc.

Route warning events to Slack 

route:
  routes:
    - match:
        - receiver: "slack-warnings"
      match:
        kind: ".*"
        type: "Warning"
    - match:
        - receiver: "elasticsearch"
receivers:
  - name: "slack-warnings"
    slack:
      token: "${SLACK_TOKEN}"
      channel: "#k8s-alerts"
      message: |
        *{{ .Reason }}* in {{ .InvolvedObject.Namespace }}/{{ .InvolvedObject.Name }}
        ```
{% endraw %}
{{ .Message }}
{% raw %}
Enter fullscreen mode Exit fullscreen mode

Drop noisy events

Some events are just noise. Drop them before they hit any sink:

route:
  drop:
    - type: "Normal"
      reason: "LeaderElection"
    - namespace: "kube-system"
      reason: "ScalingReplicaSet"
  routes:
    - match:
        - receiver: "elasticsearch"
Enter fullscreen mode Exit fullscreen mode

Route by namespace

This sends production warnings to PagerDuty, staging events to Slack, and everything to Elasticsearch.

route:
  routes:
    - match:
        - receiver: "pagerduty-webhook"
      match:
        namespace: "production"
        type: "Warning"
    - match:
        - receiver: "slack-staging"
      match:
        namespace: "staging"
    - match:
        - receiver: "elasticsearch"
Enter fullscreen mode Exit fullscreen mode

Sink examples

The exporter supports 30+ sinks. Here are the ones most people use.

Elasticsearch

Store all events for long-term analysis and Kibana dashboards:

receivers:
  - name: "elasticsearch"
    elasticsearch:
      hosts:
        - "https://elasticsearch.monitoring:9200"
      index: "k8s-events"
      username: "${ES_USER}"
      password: "${ES_PASSWORD}"
      useEventID: true
Enter fullscreen mode Exit fullscreen mode

Loki

Feed events into Grafana Loki for log-style querying:

receivers:
  - name: "loki"
    loki:
      url: "http://loki.monitoring:3100/loki/api/v1/push"
      streamLabels:
        app: "kubernetes-events"
      basicAuth:
        username: "${LOKI_USER}"
        password: "${LOKI_PASSWORD}"
Enter fullscreen mode Exit fullscreen mode

Slack 

receivers:
  - name: "slack"
    slack:
      token: "${SLACK_TOKEN}"
      channel: "#k8s-events"
      message: |
        *{{ .Type }}* event on *{{ .InvolvedObject.Kind }}* `{{ .InvolvedObject.Name }}`
        Namespace: `{{ .InvolvedObject.Namespace }}`
        Reason: `{{ .Reason }}`
        Message: {{ .Message }}
Enter fullscreen mode Exit fullscreen mode

Kafka 

receivers:
  - name: "kafka"
    kafka:
      brokers:
        - "kafka-broker:9092"
      topic: "kubernetes-events"
      tls:
        enable: true
Enter fullscreen mode Exit fullscreen mode

Webhook

Send events to any HTTP endpoint:

receivers:
  - name: "webhook"
    webhook:
      endpoint: "https://your-api.example.com/k8s-events"
      headers:
        Authorization: "Bearer ${WEBHOOK_TOKEN}"
        Content-Type: "application/json"
Enter fullscreen mode Exit fullscreen mode

Prometheus

Turn events into Prometheus metrics for alerting with Alertmanager:

receivers:
  - name: "prometheus"
    prometheus:
      labels:
        - "reason"
        - "type"
        - "involvedObject.kind"
        - "involvedObject.namespace"
Enter fullscreen mode Exit fullscreen mode

Events worth watching

These are the events that usually point to real problems.

Pod lifecycle issues

Reason Type What it means
BackOff Warning Container keeps crashing and restarting
Unhealthy Warning Liveness or readiness probe failed
OOMKilling Warning Container exceeded memory limits
FailedMount Warning Volume couldn't be mounted
ErrImagePull Warning Can't pull the container image

Scheduling problems

Reason Type What it means
FailedScheduling Warning No node has enough resources
Preempting Normal Pod is being preempted for a higher-priority pod
NotTriggerScaleUp Warning Cluster autoscaler can't add nodes

Node issues

Reason Type What it means
NodeNotReady Warning Node is unhealthy
EvictionThresholdMet Warning Node is running low on resources
Rebooted Warning Node was rebooted

Practical tips

Drop noise early

Leader election events, successful image pulls, and routine scaling events generate a lot of volume. Drop them at the top of your routing tree to keep both storage costs and Slack noise down.

Don't hardcode secrets

The exporter supports ${ENV_VAR} syntax. Use it with Kubernetes Secrets:

env:
  - name: SLACK_TOKEN
    valueFrom:
      secretKeyRef:
        name: event-exporter-secrets
        key: slack-token
Enter fullscreen mode Exit fullscreen mode

Separate sinks for separate jobs

  • Elasticsearch or Loki for historical analysis
  • Slack for real-time awareness of warnings
  • Webhook or PagerDuty for on-call alerting
  • Prometheus for dashboards and SLO tracking

Run with leader election

If you're running multiple replicas, enable leader election so only one instance processes events:

leaderElection:
  enabled: true
  leaderElectionID: "event-exporter-leader"
Enter fullscreen mode Exit fullscreen mode

Monitor the exporter itself

It exposes Prometheus metrics at /metrics. Set up alerts for high error rates on sink delivery, event processing lag, and exporter pod restarts.

Why we maintain this fork

The original kubernetes-event-exporter by Opsgenie has been dormant since 2023. We ran into bugs in production and started fixing them. The fork now includes:

  • A fix for silent event loss. The upstream only handled add and delete operations, missing updates entirely.
  • Accurate event age calculation using max(EventTime, FirstTimestamp, LastTimestamp) instead of just LastTimestamp, which was causing stale events to get re-exported.
  • Loki improvements: basic auth, stream label templating, TLS transport fixes.
  • A Prometheus sink for converting events directly into metrics.
  • Elasticsearch v8 compatibility (the v8 API broke the upstream).
  • SNS FIFO support with proper message group IDs.

The full code and Helm chart are on GitHub.

Getting started

  1. Install the exporter via Helm into your monitoring namespace
  2. Start with stdout + one sink (Slack or Elasticsearch) to validate routing works
  3. Add drop rules for the noisy stuff
  4. Expand sinks as you need them

With that in place, events stay searchable, filterable, and alertable long after the 60-minute window.

If you'd rather skip the plumbing

If you're running a single-instance cluster or EKS in your own AWS account, Ownkube gives every app logs, metrics and alerts out of the box, including an alert when a container keeps restarting, without installing a chart or writing a routing YAML. It runs on your cluster in your AWS account, so that data stays inside your perimeter.

Related reading

See Ownkube pricing for the per-minute wallet model, from $5 a month. Deploy my first app and event monitoring is one of the things you stop thinking about.

Top comments (0)