DEV Community

James Lee
James Lee

Posted on

client-go Deep Dive: EventBroadcaster — How Kubernetes Records and Distributes Cluster Events

#architecture #go #kubernetes #tutorial

In the previous article we covered WorkQueue — the retry and rate-limiting backbone of controllers. In this article we look at a different but equally important mechanism: EventBroadcaster. It's how Kubernetes components record what they're doing, and how you can do the same in your own controllers.

Source: k8s.io/client-go/tools/record/

1. What Is a Kubernetes Event?

A Kubernetes Event is a first-class resource object — stored in etcd, queryable via kubectl, and subject to the same API machinery as Pods or Deployments.

# View recent events in a namespace
kubectl get events -n default

# View events for a specific Pod (shown in describe output)
kubectl describe pod nginx-deployment-7d8ff89d87-25kb4
Enter fullscreen mode Exit fullscreen mode 
LAST SEEN   TYPE      REASON      OBJECT                          MESSAGE
2m          Normal    Scheduled   pod/nginx-7d8ff89d87-25kb4      Successfully assigned default/nginx to node-1
2m          Normal    Pulling     pod/nginx-7d8ff89d87-25kb4      Pulling image "nginx:1.21"
1m          Normal    Pulled      pod/nginx-7d8ff89d87-25kb4      Successfully pulled image in 3.2s
1m          Normal    Created     pod/nginx-7d8ff89d87-25kb4      Created container nginx
1m          Normal    Started     pod/nginx-7d8ff89d87-25kb4      Started container nginx
Enter fullscreen mode Exit fullscreen mode

⚠️ Important distinction: Kubernetes Events are resource objects managed by the API Server. They are NOT the same as the etcd watch callback events used internally by Informer/Reflector. Don't confuse the two.

2. The Event Resource Structure 

// k8s.io/api/core/v1/types.go
type Event struct {
    metav1.TypeMeta
    metav1.ObjectMeta

    // The object this event is about (e.g. the Pod that was scheduled)
    InvolvedObject ObjectReference

    // Short machine-readable reason string (e.g. "Scheduled", "Pulling", "Failed")
    Reason  string

    // Human-readable description of what happened
    Message string

    // Component that generated this event
    Source EventSource   // {Component: "scheduler", Host: "master-1"}

    // Timing
    FirstTimestamp metav1.Time      // when this event first occurred
    LastTimestamp  metav1.Time      // most recent occurrence
    Count          int32            // how many times this event has occurred

    // "Normal" or "Warning"
    Type string

    // High-precision timestamp (for newer event API)
    EventTime metav1.MicroTime

    // For aggregated/series events
    Series *EventSeries

    // What action was taken
    Action string

    // Secondary object involved (e.g. the Node a Pod was scheduled to)
    Related *ObjectReference

    // For audit: which controller generated this event
    ReportingController string
    ReportingInstance   string
}
Enter fullscreen mode Exit fullscreen mode

Key fields explained

Field Example value Purpose
InvolvedObject {Kind:"Pod", Name:"nginx-xxx"} The resource this event describes
Reason "Scheduled", "Pulling", "BackOff" Machine-readable event category
Message "Successfully assigned to node-1" Human-readable detail
Type "Normal" or "Warning" Severity indicator
Count 5 Deduplication counter — same event aggregated
FirstTimestamp / LastTimestamp timestamps When it first/last occurred
Source.Component "kubelet", "scheduler" Which Kubernetes component fired it

3. Event Retention Policy

Since Events are stored in etcd, unchecked growth would fill disk space. Kubernetes enforces a strict retention policy:

┌─────────────────────────────────────────────────────────┐
│               Event Retention Rules                     │
│                                                         │
│  • Default TTL: 1 hour after last occurrence            │
│  • kubectl get events shows only last-hour events       │
│  • Identical events are AGGREGATED (Count++ instead     │
│    of creating duplicate objects)                       │
│  • etcd stores events in a separate keyspace to         │
│    prevent them from crowding out other resources       │
└─────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

This is why kubectl describe pod shows "Events: " for Pods that have been running stably for more than an hour — the events existed, but have been garbage collected.

4. The EventBroadcaster Pipeline

EventBroadcaster is the mechanism that takes an event generated by a component and delivers it to one or more sinks (e.g. the API Server, a log file, stdout).

┌──────────────────────────────────────────────────────────────────┐
│                  EventBroadcaster Pipeline                       │
│                                                                  │
│  Your Controller                                                 │
│       │                                                          │
│       │  recorder.Eventf(obj, type, reason, message)             │
│       ▼                                                          │
│  ┌──────────────┐                                                │
│  │EventRecorder │  generates Event object                        │
│  └──────┬───────┘                                                │
│         │                                                        │
│         ▼                                                        │
│  ┌──────────────────┐                                            │
│  │ EventBroadcaster │  fan-out to multiple sinks                 │
│  └──────┬───────────┘                                            │
│         │                                                        │
│    ┌────┴──────────────────────────┐                             │
│    ▼                               ▼                             │
│  ┌──────────────────┐   ┌──────────────────────────┐            │
│  │  API Server Sink │   │  Logging Sink             │            │
│  │  (writes Event   │   │  (prints to stdout/log)   │            │
│  │   to etcd)       │   │                           │            │
│  └──────────────────┘   └──────────────────────────┘            │
└──────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

The three actors

Actor Role
EventRecorder The interface your code calls — Event(), Eventf(), AnnotatedEventf()
EventBroadcaster Receives events from recorders, fans out to registered sinks
EventSink The destination — API Server (etcd), log output, or custom handler

5. Using EventBroadcaster in a Custom Controller

Here's the complete setup pattern used in real Kubernetes controllers:

Step 1 — Create the broadcaster and recorder 

import (
    "k8s.io/client-go/tools/record"
    "k8s.io/client-go/kubernetes/scheme"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/runtime"
)

func NewController(clientset kubernetes.Interface, ...) *Controller {
    // Create the broadcaster
    eventBroadcaster := record.NewBroadcaster()

    // Sink 1: write events to the API Server (persisted in etcd)
    eventBroadcaster.StartRecordingToSink(&typedcorev1.EventSinkImpl{
        Interface: clientset.CoreV1().Events(""),
    })

    // Sink 2: also log events to stdout (useful for debugging)
    eventBroadcaster.StartEventWatcher(func(e *corev1.Event) {
        klog.V(4).Infof("Event: %s %s %s", e.Type, e.Reason, e.Message)
    })

    // Create a recorder scoped to this controller
    // "my-canary-controller" will appear in Event.Source.Component
    recorder := eventBroadcaster.NewRecorder(
        scheme.Scheme,
        corev1.EventSource{Component: "my-canary-controller"},
    )

    return &Controller{
        recorder: recorder,
        ...
    }
}
Enter fullscreen mode Exit fullscreen mode

Step 2 — Emit events during reconciliation 

func (c *Controller) reconcile(key string) error {
    namespace, name, _ := cache.SplitMetaNamespaceKey(key)

    // Fetch the resource from local cache
    canary, err := c.canaryLister.Canaries(namespace).Get(name)
    if err != nil {
        return err
    }

    // --- Normal event: record a successful action ---
    c.recorder.Event(
        canary,           // InvolvedObject
        corev1.EventTypeNormal,
        "TrafficShifted", // Reason
        "Shifted 10% traffic to canary deployment",
    )

    // --- Warning event: record a problem ---
    if canary.Status.ErrorRate > threshold {
        c.recorder.Eventf(
            canary,
            corev1.EventTypeWarning,
            "HighErrorRate",
            "Canary error rate %.2f%% exceeds threshold %.2f%%, rolling back",
            canary.Status.ErrorRate,
            threshold,
        )
        return c.rollback(canary)
    }

    return nil
}
Enter fullscreen mode Exit fullscreen mode

What the emitted Event looks like in kubectl 

LAST SEEN   TYPE      REASON           OBJECT                  MESSAGE
30s         Normal    TrafficShifted   canary/my-app-canary    Shifted 10% traffic to canary deployment
10s         Warning   HighErrorRate    canary/my-app-canary    Canary error rate 5.23% exceeds threshold 1.00%, rolling back
Enter fullscreen mode Exit fullscreen mode

6. Pod Lifecycle Events: The Most Important Events in Practice

Since Kubernetes is Pod-centric — Deployments, StatefulSets, DaemonSets, CronJobs all ultimately create Pods — Pod events are the most frequently used events in production operations.

Pod lifecycle → key events:

┌─────────────────────────────────────────────────────────────────┐
│  Phase          │ Reason           │ Type    │ Source           │
├─────────────────┼──────────────────┼─────────┼──────────────────┤
│  Scheduling     │ Scheduled        │ Normal  │ scheduler        │
│                 │ FailedScheduling │ Warning │ scheduler        │
├─────────────────┼──────────────────┼─────────┼──────────────────┤
│  Image pull     │ Pulling          │ Normal  │ kubelet          │
│                 │ Pulled           │ Normal  │ kubelet          │
│                 │ Failed           │ Warning │ kubelet          │
│                 │ ErrImagePull     │ Warning │ kubelet          │
├─────────────────┼──────────────────┼─────────┼──────────────────┤
│  Container      │ Created          │ Normal  │ kubelet          │
│  lifecycle      │ Started          │ Normal  │ kubelet          │
│                 │ Killing          │ Normal  │ kubelet          │
│                 │ BackOff          │ Warning │ kubelet          │
├─────────────────┼──────────────────┼─────────┼──────────────────┤
│  Node           │ Evicted          │ Warning │ kubelet          │
│                 │ OOMKilling       │ Warning │ kernel/kubelet   │
└─────────────────┴──────────────────┴─────────┴──────────────────┘
Enter fullscreen mode Exit fullscreen mode

Practical use cases:

  • Slow startup diagnosis: Compare Pulling → Pulled timestamps to measure image pull time
  • Crash analysis: BackOff events with increasing intervals signal CrashLoopBackOff
  • Scheduling failures: FailedScheduling with reason message explains why a Pod is Pending
  • Eviction tracking: Evicted events identify nodes under memory pressure

7. Summary 

EventBroadcaster flow:

recorder.Eventf(obj, type, reason, msg)
     │
     ▼
EventBroadcaster.ActionOrDrop(event)
     │
     ├── StartRecordingToSink()  → clientset.CoreV1().Events().Create/Patch
     │                             (aggregates duplicate events via Count++)
     │
     └── StartEventWatcher()    → custom handler (logging, metrics, alerting)
Enter fullscreen mode Exit fullscreen mode
Concept Detail
Event A core API resource — stored in etcd, queryable via kubectl
Type Normal (informational) or Warning (something needs attention)
Reason Short machine-readable string — used for filtering and alerting
Count Identical events are aggregated — prevents etcd flooding
Retention Events are deleted 1 hour after last occurrence
EventRecorder Your controller's interface for emitting events
EventBroadcaster Fan-out hub — delivers events to all registered sinks
EventSink Destination: API Server (etcd), stdout log, or custom handler

EventBroadcaster closes the observability loop for Kubernetes controllers. By emitting structured events at key decision points in your reconciliation logic, you give operators the same visibility into your custom controller that they have for built-in Kubernetes components — making your controller production-ready and debuggable.

Next in this series: Controller: The Central Hub of the Informer Framework (Part 7)

Follow the series for more deep dives into Kubernetes development.

Top comments (0)