Talking to Kubernetes from Go: a practical client-go guide

This is part of the Platform Engineering with Go series. This post builds on the minikube cluster set up in post 2. Read post 2 first if you haven't yet.

---

kubectl is great for humans. But what about programs?

When you run kubectl get pods, something has to talk to the Kubernetes API server, deserialize the JSON response, and format it for your terminal. That something is client-go, the official Go client library for Kubernetes.

Every tool in the Kubernetes ecosystem is built on it. kubectl itself, helm, ArgoCD, controller-runtime, Tekton. If it talks to a Kubernetes cluster and it's written in Go, it's using client-go.

In this post, we'll learn how client-go works by building something immediately useful: a cluster health reporter. One Go program that connects to your minikube cluster, scans every pod across every namespace, and prints a complete health summary, the kind of view that usually takes four or five kubectl commands to assemble.

Cluster Health Report
Generated: 2026-06-24 14:32:00 UTC

NAMESPACE     TOTAL   RUNNING   FAILING   PENDING
──────────────────────────────────────────────────
default           3         3         0         0
kube-system       6         6         0         0

✓ All pods healthy

And if something is wrong:

Cluster Health Report
Generated: 2026-06-24 14:32:00 UTC

NAMESPACE     TOTAL   RUNNING   FAILING   PENDING
──────────────────────────────────────────────────
default           3         1         2         0
kube-system       6         6         0         0

FAILING PODS
──────────────────────────────────────────────────
default   go-api-7d6b9f8c4-mn9rt   CrashLoopBackOff   14 restarts
default   go-api-7d6b9f8c4-p8wvz   OOMKilled            3 restarts

What you'll learn

• What client-go is and how it relates to the Kubernetes API.

• The four key packages and what each one contains.

• How to connect to a cluster, out-of-cluster and in-cluster.

• How the Clientset is organized, API groups explained.

• How to list, filter, and inspect Kubernetes resources from Go.

• How to build the cluster health reporter step by step.

• How to test client-go code, two approaches with clear rules for when to use each.

Prerequisites

• minikube running with the go-api deployment from post 2

---

What client-go is

The Kubernetes API is a REST API. When you run kubectl get pods -n default, kubectl makes an HTTP GET request to something like:

GET https://kubernetes/api/v1/namespaces/default/pods

The API server returns JSON. kubectl deserializes it and formats it for your terminal.

client-go is the library that handles all of this in Go: the HTTP calls, TLS, authentication, rate limiting, and deserialization into typed Go structs. Instead of working with raw JSON, you get a *corev1.PodList with typed fields you can navigate in your editor with autocomplete.

Your Go program
      │
      │  client-go
      │  ├── kubeconfig loading
      │  ├── TLS + authentication
      │  ├── HTTP client + rate limiting
      │  └── JSON → typed Go structs
      │
      ▼
Kubernetes API server
      │
      ▼
etcd (the cluster's data store)

Everything you can do with kubectl, you can do with client-go, and more. kubectl is itself a client-go program.

---

Project structure

Here is the Go project structure we are going to build; remember, you can see how to organize the code for your own projects here.

You can find the entire project here.

go-k8s-health-reporter/
├── go.mod
├── main.go                  ← entry point — wires dependencies, calls reporter
└── internal/
    ├── reporter/
    │   ├── types.go         ← data types — PodStatus, NamespaceSummary, Report
    │   ├── reporter.go      ← Reporter struct, Generate()
    │   ├── inspector.go     ← pod health detection — inspectPod(), pendingReason()
    │   └── printer.go       ← output formatting — Print(), tabwriter tables
    └── kubernetes/
        └── client.go        ← KubeClient interface + real implementation

Installing the packages

Before writing any code, initialize the module and install the dependencies. The version numbers must match your Kubernetes cluster; since we're running Kubernetes 1.36.2, we use v0.36.2 for all three packages. Mixing versions causes compilation errors because the types won't match across packages. Remember to rename the module's name to your own module name.

# Initialize the Go module
mkdir go-k8s-health-reporter && cd go-k8s-health-reporter
go mod init github.com/FerRiosCosta/go-k8s-health-reporter

# Install the four packages pinned to Kubernetes 1.36.2
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.36.2
go get k8s.io/apimachinery@v0.36.2

# Tidy up — removes unused dependencies and ensures versions are consistent
go mod tidy

If you see a missing go.sum entry error after installing the packages, it means some transitive dependencies, packages that client-go depends on, like k8s.io/klog/v2, weren't added to go.sum automatically. Fix it with:

bash

go get ./...
go mod tidy

go get ./... scans all packages in your module and fetches every dependency recursively, including transitive ones. go mod tidy then writes the complete go.sum.

Why pin to v0.36.2 instead of @latest? The client-go versioning scheme directly mirrors Kubernetes releases; v0.36.2 corresponds to Kubernetes v1.36.2. Always match your client-go version to your cluster version. If you run @latest you might get a newer client-go version than your cluster supports, which can cause unexpected API behavior.

You can verify your cluster version at any time:

kubectl version

# Client Version: v1.36.2
# Server Version: v1.36.2

Both should match (ideally). If they don't, pin your packages to match the server version.

Note: k8s.io/metrics is not needed for the health reporter in this post, but you will need it for the next posts.

go get k8s.io/metrics@v0.36.2
go mod tidy

Here's the go.mod:

module github.com/FerRiosCosta/go-k8s-health-reporter

go 1.26

require (
    k8s.io/api          v0.36.2
    k8s.io/apimachinery v0.36.2
    k8s.io/client-go    v0.36.2
)

---

The four key packages

Before writing any code, let's understand the four packages client-go code almost always imports. This is the section that makes every future client-go import immediately readable.

k8s.io/api: the Go type definitions

import (
    corev1  "k8s.io/api/core/v1"   // Pod, Service, ConfigMap, Secret, Namespace, Node
    appsv1  "k8s.io/api/apps/v1"   // Deployment, StatefulSet, DaemonSet, ReplicaSet
    batchv1 "k8s.io/api/batch/v1"  // Job, CronJob
)

This package contains the Go struct definitions for every Kubernetes resource. When you write corev1.Pod{} or appsv1.Deployment{}, you're using this package. The aliases (corev1, appsv1) are conventions, the actual package names are just v1, which is too generic to use unaliased.

k8s.io/apimachinery: the shared building blocks

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" // ObjectMeta, ListOptions, Time
    "k8s.io/apimachinery/pkg/api/resource"         // resource.Quantity (CPU, memory)
)

This package contains types shared across all Kubernetes API groups. metav1.ObjectMeta is the block every resource has: name, namespace, labels, annotations, creation timestamp. metav1.ListOptions lets you filter API calls by label selectors and field selectors. You'll import something from this package in virtually every file that touches the Kubernetes API.

k8s.io/client-go: the actual client

import (
    "k8s.io/client-go/kubernetes"        // the Clientset — entry point to all APIs
    "k8s.io/client-go/rest"              // rest.Config — holds connection details
    "k8s.io/client-go/tools/clientcmd"   // reads and parses kubeconfig files
)

This is the main package. kubernetes.NewForConfig() creates the Clientset, your entry point to every Kubernetes API. clientcmd reads the ~/.kube/config file that minikube wrote in post 2. rest.Config holds the connection details: API server URL, TLS certificates, auth token.

k8s.io/metrics: the metrics-server API

import (
    metrics "k8s.io/metrics/pkg/client/clientset/versioned" // metrics Clientset
)

This is a separate package because metrics-server is an extension API, not part of the core Kubernetes API.

# Install in post 4 — pinned to the same version as the rest
go get k8s.io/metrics@v0.36.2
go mod tidy

---

Connecting to the cluster

Every client-go program starts the same way: build a rest.Config, then create a Clientset from it.

// internal/kubernetes/client.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
)

// KubeClient defines the Kubernetes operations the health reporter uses.
// Wrapping the Clientset behind an interface makes the reporter testable
// without a real cluster, the same pattern used throughout this series.
type KubeClient interface {
    // ListPods returns all pods in the given namespace.
    // An empty namespace string returns pods from all namespaces.
    ListPods(ctx context.Context, namespace string) (*corev1.PodList, error)
}

// client is the real implementation backed by the official Kubernetes Clientset.
// It is unexported, callers always use the KubeClient interface.
type client struct {
    clientset kubernetes.Interface
}

// NewClient creates a KubeClient connected to the cluster in kubeconfig.
func NewClient(kubeconfigPath string) (KubeClient, error) {
    config, err := buildConfig(kubeconfigPath)
    if err != nil {
        return nil, fmt.Errorf("build config: %w", err)
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create clientset: %w", err)
    }

    return &client{clientset: clientset}, nil
}

// buildConfig tries in-cluster config first, then falls back to kubeconfig.
// This means the same binary works both locally (reads ~/.kube/config)
// and when deployed as a pod inside a cluster (reads the mounted service account token).
func buildConfig(kubeconfigPath string) (*rest.Config, error) {
    // in-cluster config: reads /var/run/secrets/kubernetes.io/serviceaccount/token
    // automatically mounted inside every pod by Kubernetes.
    if config, err := rest.InClusterConfig(); err == nil {
        return config, nil
    }

    // out-of-cluster config: reads the kubeconfig file from disk.
    // This is the file minikube wrote to ~/.kube/config in post 2.
    return clientcmd.BuildConfigFromFlags("", kubeconfigPath)
}

// ListPods retrieves all pods in the given namespace from the Kubernetes API.
// Passing an empty string returns pods from all namespaces.
func (c *client) ListPods(ctx context.Context, namespace string) (*corev1.PodList, error) {
    pods, err := c.clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pods namespace=%q: %w", namespace, err)
    }
    return pods, nil
}

The Clientset and API groups

c.clientset.CoreV1().Pods(namespace).List(ctx, opts), let's break this chain down because you'll see it everywhere in client-go code:

c.clientset           // the Clientset, entry point to all Kubernetes APIs
    .CoreV1()         // select the "core" API group, version 1
    .Pods(namespace)  // select the Pods resource in the given namespace
    .List(ctx, opts)  // call the List operation with options

Kubernetes organizes resources into API groups. Each group contains related resources:

CoreV1()  → Pods, Services, ConfigMaps, Secrets, Nodes, Namespaces
AppsV1()  → Deployments, StatefulSets, DaemonSets, ReplicaSets
BatchV1() → Jobs, CronJobs

The pattern is always the same: clientset.APIGroup().ResourceType(namespace).Operation(). Once you know this pattern, any client-go call is readable.

---

Building the cluster health reporter

Now let's build the reporter. Instead of putting everything in one file, we'll split it across four focused files from the start, each with one clear responsibility.

internal/reporter/
├── types.go      ← data types only — PodStatus, NamespaceSummary, Report
├── reporter.go   ← Reporter struct, constructor, Generate()
├── inspector.go  ← pod inspection logic — detects failing and pending states
└── printer.go    ← output formatting — tabwriter tables

Each file has one reason to change:

• New data field → types.go

• New data source → reporter.go

• New failure reason to detect → inspector.go

• New output format (JSON, CSV) → printer.go

types.go, data types only

// internal/reporter/types.go
package reporter

import "time"

// PodStatus represents the health state of a single pod.
type PodStatus struct {
    Namespace string
    Name      string
    Phase     string // Running, Pending, Failed, Succeeded
    Reason    string // CrashLoopBackOff, OOMKilled, etc. — more specific than Phase
    Restarts  int32  // total restarts across all containers
    IsFailing bool   // true if the pod is in a non-running problem state
    IsPending bool   // true if the pod is waiting to be scheduled or started
}

// NamespaceSummary holds the pod counts for a single namespace.
type NamespaceSummary struct {
    Namespace string
    Total     int
    Running   int
    Failing   int
    Pending   int
}

// Report is the complete cluster health picture.
type Report struct {
    GeneratedAt time.Time
    Namespaces  []NamespaceSummary
    Failing     []PodStatus
    Pending     []PodStatus
}

Thirty lines. No imports beyond time. If you need to add a field to a pod's health state, you open this file and only this file.

inspector.go, pod health detection

// internal/reporter/inspector.go
package reporter

import corev1 "k8s.io/api/core/v1"

// inspectPod examines a pod's status and returns a detailed PodStatus.
// It checks container states first, a pod can be in "Running" phase
// while individual containers are in CrashLoopBackOff or OOMKilled.
func inspectPod(pod *corev1.Pod) PodStatus {
    status := PodStatus{
        Namespace: pod.Namespace,
        Name:      pod.Name,
        Phase:     string(pod.Status.Phase),
    }

    // Sum restart counts and check container states.
    // Must check containers before pod phase, a pod can be "Running"
    // while a container inside it is waiting to restart.
    for _, cs := range pod.Status.ContainerStatuses {
        status.Restarts += cs.RestartCount

        if cs.State.Waiting != nil {
            switch cs.State.Waiting.Reason {
            case "CrashLoopBackOff", "OOMKilled", "ImagePullBackOff",
                "ErrImagePull", "CreateContainerConfigError":
                status.Reason = cs.State.Waiting.Reason
                status.IsFailing = true
                return status
            }
        }

        if cs.State.Terminated != nil {
            reason := cs.State.Terminated.Reason
            if reason == "OOMKilled" || cs.State.Terminated.ExitCode != 0 {
                status.Reason = reason
                status.IsFailing = true
                return status
            }
        }
    }

    // Check pod-level phase for broader failure states.
    switch pod.Status.Phase {
    case corev1.PodFailed:
        status.Reason = pod.Status.Reason
        status.IsFailing = true
    case corev1.PodPending:
        status.Reason = pendingReason(pod)
        status.IsPending = true
    }

    return status
}

// pendingReason extracts why a pod is pending from its conditions.
// The PodScheduled condition contains the human-readable scheduling
// failure message; e.g. "0/1 nodes are available: 1 Insufficient memory."
func pendingReason(pod *corev1.Pod) string {
    for _, condition := range pod.Status.Conditions {
        if condition.Type == corev1.PodScheduled &&
            condition.Status == corev1.ConditionFalse {
            return condition.Message
        }
    }
    return string(pod.Status.Phase)
}

Understanding container states

The inspectPod function works by reading pod.Status.ContainerStatuses, a slice that contains one entry per container in the pod. Each entry has a State field that holds exactly one of three possible states at any given moment:

Waiting: the container is not yet running. This is where most failure reasons live:

Reason	What it means
ContainerCreating	Image is being pulled, or volumes are being mounted
PodInitializing	Init containers are still running, main container hasn't started
CrashLoopBackOff	Container keeps crashing, Kubernetes is backing off before restarting
ImagePullBackOff	Image pull failed, Kubernetes is backing off before retrying
ErrImagePull	The immediate image pull error, before backoff kicks in
CreateContainerConfigError	Bad config, missing ConfigMap, Secret, or invalid env var
OOMKilled	Killed by the OOM killer, waiting to restart

Running: the container process is alive. Note that this doesn't mean the application inside it is healthy, it just means the process is executing. Your app could still be returning 500s or stuck in a deadlock.

Terminated: the container has finished. The most important fields are:

Field	What it means
ExitCode	0 = clean exit, anything else = failure
Reason	Completed (clean) or OOMKilled (killed by OOM killer)

Common exit codes worth knowing:

Exit code	Meaning
0	Clean exit, completed successfully
1	Application error, the process itself returned an error
137	SIGKILL, OOM killer fired, or forcefully killed
143	SIGTERM, graceful shutdown (normal during pod termination)

This is exactly why inspectPod checks cs.State.Waiting and cs.State.Terminated separately; each state carries different information in different fields.

Init containers, the hidden failure

There's one more status field worth knowing about:

pod.Status.ContainerStatuses      // current state of main containers
pod.Status.InitContainerStatuses  // current state of init containers

Init containers run before your main containers and go through the same three states. If an init container is stuck in CrashLoopBackOff, your main container will never start, and pod.Status.ContainerStatuses will be empty, giving you no indication of what's wrong.

The current inspectPod implementation only checks main container statuses. A more complete version would check init containers first:

// A more complete inspectPod that also catches stuck init containers.
// This is left as an exercise — the pattern is identical to the main
// container check, just reading from InitContainerStatuses instead.
func inspectPod(pod *corev1.Pod) PodStatus {
    status := PodStatus{
        Namespace: pod.Namespace,
        Name:      pod.Name,
        Phase:     string(pod.Status.Phase),
    }

    // Check init containers first — if one is stuck, main containers
    // will never start and ContainerStatuses will be empty.
    for _, cs := range pod.Status.InitContainerStatuses {
        status.Restarts += cs.RestartCount
        if cs.State.Waiting != nil {
            switch cs.State.Waiting.Reason {
            case "CrashLoopBackOff", "OOMKilled", "ImagePullBackOff",
                "ErrImagePull", "CreateContainerConfigError":
                status.Reason = "init: " + cs.State.Waiting.Reason
                status.IsFailing = true
                return status
            }
        }
    }

    // Then check main containers as before.
    for _, cs := range pod.Status.ContainerStatuses {
        // ... same logic as above
    }

    // ... rest of the function unchanged
}

Prefixing the reason with "init: " makes it immediately clear in the report that it's an init container failing, not the main application.

You can add this code block before checking the main container in your project if you want.

printer.go, output formatting

// internal/reporter/printer.go
package reporter

import (
    "fmt"
    "io"
    "strings"
    "text/tabwriter"
)

// Print writes the health report as a formatted table to w.
// Accepting io.Writer instead of os.Stdout directly keeps this
// testable — callers can pass a bytes.Buffer to inspect the output.
func (r *Reporter) Print(w io.Writer, report *Report) {
    fmt.Fprintf(w, "Cluster Health Report\n")
    fmt.Fprintf(w, "Generated: %s\n\n",
        report.GeneratedAt.Format("2006-01-02 15:04:05 UTC"),
    )

    printNamespaceSummary(w, report.Namespaces)

    if len(report.Failing) > 0 {
        printFailingPods(w, report.Failing)
        return
    }

    if len(report.Pending) > 0 {
        printPendingPods(w, report.Pending)
        return
    }

    fmt.Fprintf(w, "\n✓ All pods healthy\n")
}

// printNamespaceSummary writes the namespace totals table.
func printNamespaceSummary(w io.Writer, namespaces []NamespaceSummary) {
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    fmt.Fprintln(tw, "NAMESPACE\tTOTAL\tRUNNING\tFAILING\tPENDING")
    fmt.Fprintln(tw, strings.Repeat("─", 52))
    for _, ns := range namespaces {
        fmt.Fprintf(tw, "%s\t%d\t%d\t%d\t%d\n",
            ns.Namespace, ns.Total, ns.Running, ns.Failing, ns.Pending,
        )
    }
    tw.Flush()
}

// printFailingPods writes the failing pods detail section.
func printFailingPods(w io.Writer, pods []PodStatus) {
    fmt.Fprintf(w, "\nFAILING PODS\n")
    fmt.Fprintln(w, strings.Repeat("─", 52))
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    for _, p := range pods {
        fmt.Fprintf(tw, "%s\t%s\t%s\t%d restarts\n",
            p.Namespace, p.Name, p.Reason, p.Restarts,
        )
    }
    tw.Flush()
}

// printPendingPods writes the pending pods detail section.
func printPendingPods(w io.Writer, pods []PodStatus) {
    fmt.Fprintf(w, "\nPENDING PODS\n")
    fmt.Fprintln(w, strings.Repeat("─", 52))
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    for _, p := range pods {
        fmt.Fprintf(tw, "%s\t%s\t%s\n",
            p.Namespace, p.Name, p.Reason,
        )
    }
    tw.Flush()
}

reporter.go, orchestration only

// internal/reporter/reporter.go
package reporter

import (
    "context"
    "fmt"
    "time"

    k8s "github.com/FerRiosCosta/go-k8s-health-reporter/internal/kubernetes"
)

// Reporter builds health reports from Kubernetes data.
// It depends on the KubeClient interface, never on a concrete
// client-go type, so it works with both real and fake clients.
type Reporter struct {
    kube k8s.KubeClient
}

// New creates a Reporter with the given KubeClient injected.
func New(kube k8s.KubeClient) *Reporter {
    return &Reporter{kube: kube}
}

// Generate fetches all pods from the cluster and builds a health report.
func (r *Reporter) Generate(ctx context.Context) (*Report, error) {
    // Fetch all pods across all namespaces.
    // Passing "" to ListPods returns pods from every namespace.
    pods, err := r.kube.ListPods(ctx, "")
    if err != nil {
        return nil, fmt.Errorf("fetch pods: %w", err)
    }

    report := &Report{
        GeneratedAt: time.Now().UTC(),
    }

    // Group pods by namespace using a map.
    summaries := make(map[string]*NamespaceSummary)

    for _, pod := range pods.Items {
        // getOrCreate avoids the boilerplate of checking if the
        // namespace key exists before writing to the map.
        summary := getOrCreate(summaries, pod.Namespace)
        summary.Total++

        // inspectPod is defined in inspector.go, same package,
        // no import needed.
        status := inspectPod(&pod)

        switch {
        case status.IsFailing:
            summary.Failing++
            report.Failing = append(report.Failing, status)
        case status.IsPending:
            summary.Pending++
            report.Pending = append(report.Pending, status)
        default:
            summary.Running++
        }
    }

    for _, summary := range summaries {
        report.Namespaces = append(report.Namespaces, *summary)
    }

    return report, nil
}

// getOrCreate returns the existing NamespaceSummary for the given namespace
// or creates a new one and registers it in the map.
// Extracted into its own function to keep Generate() readable.
func getOrCreate(m map[string]*NamespaceSummary, namespace string) *NamespaceSummary {
    if s, ok := m[namespace]; ok {
        return s
    }
    s := &NamespaceSummary{Namespace: namespace}
    m[namespace] = s
    return s
}

Notice that reporter.go now has no output imports (tabwriter, strings, io) and no Kubernetes type imports beyond what it needs for ListPods. The corev1 import moved entirely to inspector.go where it belongs. The reporter package is still one package, Go compiles all four files together, but each file is now independently readable.

Wire it together in main.go

// main.go
package main

import (
    "context"
    "fmt"
    "os"

    k8s "github.com/FerRiosCosta/go-k8s-health-reporter/internal/kubernetes"
    "github.com/FerRiosCosta/go-k8s-health-reporter/internal/reporter"
)

func main() {
    // Build the Kubernetes client using the kubeconfig minikube wrote in post 2.
    // This is the same file kubectl uses, no extra setup needed.
    kubeconfigPath := os.Getenv("HOME") + "/.kube/config"
    kube, err := k8s.NewClient(kubeconfigPath)
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to connect to cluster: %v\n", err)
        os.Exit(1)
    }

    // Build the reporter with the client injected.
    r := reporter.New(kube)

    // Generate and print the report.
    ctx := context.Background()
    report, err := r.Generate(ctx)
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to generate report: %v\n", err)
        os.Exit(1)
    }

    r.Print(os.Stdout, report)
}

---

Verifying against minikube

The best way to verify the health reporter works is to run it against your real minikube cluster. No fakes, no mocks, just your actual cluster from post 2.

Healthy cluster

Make sure minikube is running and the go-api deployment from post 2 is up:

kubectl get pods
# NAME                      READY   STATUS    RESTARTS   AGE
# go-api-7d6b9f8c4-xk2pq   1/1     Running   0          2d
# go-api-7d6b9f8c4-mn9rt   1/1     Running   0          2d
# go-api-7d6b9f8c4-p8wvz   1/1     Running   0          2d

Run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-06-29 15:58:43 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
default                3   3   0   0
kube-system            8   8   0   0
kubernetes-dashboard   2   2   0   0

✓ All pods healthy

Simulating a failing pod

To verify the failing pod detection works we need a pod that reliably OOMKills. Our go-api binary turns out to be too lean to reliably trigger OOMKill at the Docker minimum, the multi-stage Dockerfile produced an extremely efficient binary. That's good engineering, but it makes for a tricky simulation.

Instead, let's deploy a pod that deliberately holds memory using Python's bytearray. Unlike shell tools like dd which stream data through memory and release it immediately, bytearray allocates a contiguous block and holds it, guaranteeing an OOMKill with a tight limit.

Create a file called memory-hog.yaml:

# memory-hog.yaml
apiVersion: v1
kind: Pod
metadata:
  name: memory-hog
  namespace: default
spec:
  # restartPolicy: Always ensures Kubernetes keeps restarting after each
  # OOMKill, driving the pod into CrashLoopBackOff.
  restartPolicy: Always
  containers:
  - name: hog
    image: python:3.12-alpine
    command: ["python3", "-c"]
    args:
    - |
      import time
      # bytearray allocates 100MB and holds it in physical memory.
      # The moment this runs, usage exceeds the 10Mi limit
      # and the OOM killer fires immediately.
      data = bytearray(100 * 1024 * 1024)
      print("allocated 100MB, holding...", flush=True)
      time.sleep(3600)
    resources:
      requests:
        memory: "10Mi"
      limits:
        memory: "10Mi"

Apply it:

kubectl apply -f memory-hog.yaml

Watch the pod:

kubectl get pods -w

# NAME          READY   STATUS             RESTARTS   AGE
# memory-hog   0/1     ContainerCreating  0          1s
# memory-hog   0/1     OOMKilled          0          4s   ← transient
# memory-hog   0/1     CrashLoopBackOff   1          10s  ← stable, run the reporter now

Once you see CrashLoopBackOff, run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-07-01 21:37:47 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
kubernetes-dashboard   2   2   0   0
default                4   3   1   0
kube-system            8   8   0   0

FAILING PODS
────────────────────────────────────────────────────
default   memory-hog   CrashLoopBackOff   13 restarts

The reporter correctly identifies the failing pod, the reason, and the restart count.

Clean up when done:

kubectl delete pod memory-hog

Simulating a pending pod

To verify pending detection, set a memory request higher than your node can provide. Again, both requests and limits must be set together:

kubectl set resources deployment/go-api \
  --requests=memory=100Gi \
  --limits=memory=100Gi \
  -n default

The pods can't be scheduled, no node has 100Gi available. Run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-07-01 22:03:51 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
default                4   3   0   1
kube-system            8   8   0   0
kubernetes-dashboard   2   2   0   0

PENDING PODS
────────────────────────────────────────────────────
default   go-api-6ff6496cb5-dnq6m   0/1 nodes are available: 1 Insufficient memory. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

The reason message comes directly from the pod's scheduling condition, exactly what the pendingReason() function extracts. Restore the correct values when done:

kubectl set resources deployment/go-api \
  --requests=memory=32Mi \
  --limits=memory=64Mi \
  -n default

Note on testing with fakes: In a future post we'll cover how to write unit tests for this reporter using interface fakes and client-go's built-in fake clientset, so the test suite runs without a real cluster. For now, minikube is our test environment.

---

Common gotchas

Wrong kubeconfig context

The most common mistake when starting out with client-go. If you have multiple clusters in your kubeconfig, always verify before running Go code:

kubectl config current-context
# Should print: minikube

If it shows a production cluster, switch back:

kubectl config use-context minikube

Passing "" vs "default" to namespace-scoped resources

// Returns pods from ALL namespaces
clientset.CoreV1().Pods("").List(ctx, metav1.ListOptions{})

// Returns pods from the "default" namespace only
clientset.CoreV1().Pods("default").List(ctx, metav1.ListOptions{})

These behave completely differently. The empty string means "all namespaces", not "the default namespace." This is one of the most common sources of confusion in client-go code.

Pod phase vs container state, the big one

A pod in Running phase can have containers in CrashLoopBackOff. The phase is the pod-level state; the container state is what's actually happening inside. Always check both:

// WRONG: misses CrashLoopBackOff pods
if pod.Status.Phase != corev1.PodRunning {
    // handle failing pod
}

// RIGHT: catches CrashLoopBackOff even when phase is Running
for _, cs := range pod.Status.ContainerStatuses {
    if cs.State.Waiting != nil && cs.State.Waiting.Reason == "CrashLoopBackOff" {
        // handle failing container
    }
}

Creating a new Clientset per function call

client-go's Clientset maintains an internal HTTP connection pool and rate limiter. Creating a new one on every call throws away the connection pool and can trigger API server rate limits. Always create the Clientset once at startup and reuse it:

// WRONG: new Clientset every call, no connection reuse
func listPods() {
    clientset, _ := kubernetes.NewForConfig(config)
    clientset.CoreV1().Pods("").List(...)
}

// RIGHT: Clientset created once, reused everywhere
var clientset kubernetes.Interface

func init() {
    clientset, _ = kubernetes.NewForConfig(config)
}

func listPods() {
    clientset.CoreV1().Pods("").List(...)
}

API version mismatches, always pin all three packages together

All three packages, k8s.io/api, k8s.io/apimachinery, and k8s.io/client-go, must be on the same minor version. They are developed and released together as part of each Kubernetes release cycle. Mixing versions causes compile errors because types from one package won't match types expected by another.

The version number maps directly to your Kubernetes cluster: v0.36.2 = Kubernetes v1.36.2. Since we're running Kubernetes 1.36.2 in minikube, v0.36.2 is the right choice throughout this series:

# WRONG: mixing minor versions causes type mismatch errors at compile time
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.35.0          # different minor version — will break
go get k8s.io/apimachinery@v0.34.0  # different minor version — will break

# RIGHT: all three pinned to match Kubernetes 1.36.2
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.36.2
go get k8s.io/apimachinery@v0.36.2
go mod tidy

Always verify your cluster version before pinning:

kubectl version 

# Client Version: v1.36.2
# Server Version: v1.36.2

If your client and server versions differ, for example Client Version: v1.36.2 and Server Version: v1.35.1, that's completely fine. Kubernetes has an official version skew policy: kubectl can be up to one minor version ahead or behind the server. Pin your client-go packages to match whichever version you're running in the cluster.

---

Summary

client-go is the foundation everything in the Kubernetes Go ecosystem is built on. Four packages, one Clientset, typed access to every resource the API server exposes.

Three things to take away:

• The Clientset pattern: clientset.APIGroup().Resource(namespace).Operation(), is consistent across every resource type. Learn it once, read any client-go code.

• Always check container states, not just pod phase, a "Running" pod can have failing containers.

• Wrap the Clientset behind an interface from the start, it costs nothing and makes every function that uses it instantly testable.

What's next

The health reporter we built here is a real tool. Run it on any cluster, and it gives you immediate operational value. It's also the direct foundation for a new tool we are going to build in the later chapter:ferctl top: a full CLI with Cobra, flags, tabwriter output, and near-limit warnings. The health reporter becomes a subcommand. The KubeClient interface gets extended with metrics.

Every pattern, every import, and every interface in this post appears in the next post.

---

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

• LinkedIn

• GitHub

• Twitter / X

Building from Asunción, Paraguay 🇵🇾