DEV Community: Fer Rios

Building Go CLIs with Cobra: a practical guide

Fer Rios — Tue, 21 Jul 2026 13:19:01 +0000

Series: Platform engineering with Go | Topics: Go, Cobra, CLI, Platform Engineering

This is part of the Platform Engineering with Go series. This post builds on the client-go patterns from post 3. Read post 3 first if you haven't yet.

kubectl, helm, and the GitHub CLI all have something in common

They're all built with Cobra.

Cobra is the Go library for building command-line tools. It handles subcommands, flags, help text, shell completions, and argument validation, everything you need to turn a Go program into a proper CLI that behaves the way engineers expect a CLI to behave.

If you've used kubectl get pods, helm install, or gh pr create, you've already used a Cobra CLI without knowing it. In this post, you'll learn how to build one.

By the end, you'll have a working CLI called k8s-info with two subcommands; version and cluster.

Prerequisites

Read posts 1–3 of this series (helpful but not required for this post)

Project structure

As always, we start our project by creating the structure. If you don't know how and why this structure is created, you can learn it here.

k8s-info/
├── go.mod
├── Makefile
├── cmd/
│   └── k8s-info/
│       └── main.go              ← 3 lines — just calls Execute()
└── internal/
    └── cmd/
        ├── root.go              ← root command + persistent flags
        ├── version.go           ← "k8s-info version" subcommand
        └── cluster.go           ← "k8s-info cluster" subcommand

This structure scales. When we build our next command in the next post, we add more files under internal/cmd/, the root command never changes.

You can find the whole project code here

Initialize (with your own module name), and install:

mkdir k8s-info && cd k8s-info
go mod init github.com/yourname/k8s-info

## Install Cobra — the CLI framework
go get github.com/spf13/cobra@v1.10.2

# Install the Kubernetes packages needed for the cluster subcommand
# All three must be pinned to the same version matching your server version
# Since our minikube server is v1.35.1, we use v0.35.1
go get k8s.io/client-go@v0.35.1
go get k8s.io/api@v0.35.1
go get k8s.io/apimachinery@v0.35.1

# Fetch all transitive dependencies and clean up
go get ./...
go mod tidy

What Cobra actually is

Before writing any code, it helps to understand how Cobra thinks about a CLI.

Every Cobra CLI is a tree of commands:

k8s-info                     ← root command
├── version                  ← subcommand
└── cluster                  ← subcommand
    ├── --namespace           ← local flag (only for cluster)
    └── --output              ← local flag (only for cluster)

The root command is the entry point, it's what runs when you type just k8s-info. Subcommands are registered on the root with AddCommand(). Flags can be attached to any command, and persistent flags (attached to the root) are inherited by every subcommand.

Cobra handles everything else automatically: --help, error messages, usage text, and shell completion stubs.

The entry point

// cmd/k8s-info/main.go
package main

import "github.com/FerRiosCosta/k8s-info/internal/cmd"

// main is intentionally minimal, all command logic lives in internal/cmd/.
// This keeps the entry point clean and the business logic testable.
func main() {
    cmd.Execute()
}

Three lines. If your main.go is longer than this, something belongs in a package.

The root command

// internal/cmd/root.go
package cmd

import (
    "fmt"
    "os"

    "github.com/spf13/cobra"
)

// kubeconfigPath holds the value of the --kubeconfig persistent flag.
// Declared at package level so every subcommand can read it.
var kubeconfigPath string

// rootCmd is the base command, the one that runs when you type just "k8s-info".
// Every subcommand is registered on it via AddCommand().
var rootCmd = &cobra.Command{
    // Use is the one-line usage shown in help text.
    // The first word must be the binary name.
    Use: "k8s-info",

    // Short is the brief description shown in parent command help.
    Short: "A CLI for inspecting your Kubernetes cluster",

    // Long is the full description shown when you run "k8s-info --help".
    Long: `k8s-info is a platform engineering CLI for inspecting
Kubernetes cluster state. It connects to whatever cluster
your kubeconfig is currently pointing at.`,

    // SilenceUsage prevents Cobra from printing the full usage text
    // on every error, which gets noisy fast. Errors are still shown.
    SilenceUsage: true,
}

func init() {
    // PersistentFlags are inherited by every subcommand automatically.
    // --kubeconfig is something every subcommand might need, so it
    // belongs on the root as a persistent flag.
    rootCmd.PersistentFlags().StringVar(
        &kubeconfigPath,
        "kubeconfig",
        os.Getenv("HOME")+"/.kube/config", // default: ~/.kube/config
        "path to the kubeconfig file",
    )
}

// Execute is the single public entry point called from main.go.
// It parses os.Args, finds the right command, and runs it.
func Execute() {
    if err := rootCmd.Execute(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

Why SilenceUsage matters

By default, Cobra prints the full usage text every time an error occurs. This is often more confusing than helpful; the user sees a wall of text when what they really need is just the error message. SilenceUsage: true on the root command suppresses usage on errors for the entire CLI.

Adding subcommands

version subcommand

// internal/cmd/version.go
package cmd

import (
    "fmt"

    "github.com/spf13/cobra"
)

// Version is set at build time using ldflags.
// Default to "dev" so it's always meaningful, even without a build.
var Version = "dev"

// versionCmd prints the current version of k8s-info.
var versionCmd = &cobra.Command{
    Use:   "version",
    Short: "Print the k8s-info version",

    // Run is used when the command cannot fail.
    // If your command can return an error, use RunE instead (shown below).
    Run: func(cmd *cobra.Command, args []string) {
        fmt.Printf("k8s-info version %s\n", Version)
    },
}

func init() {
    // Register versionCmd on the root command.
    // After this, "k8s-info version" is a valid command.
    rootCmd.AddCommand(versionCmd)
}

cluster subcommand

// internal/cmd/cluster.go
package cmd

import (
    "context"
    "fmt"
    "os"

    "github.com/spf13/cobra"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

// namespace is a local flag, only available on the cluster subcommand,
// not on version or any future subcommand.
var namespace string

// clusterCmd shows a summary of what's running in the cluster.
var clusterCmd = &cobra.Command{
    Use:   "cluster",
    Short: "Show a summary of running pods in your cluster",
    Example: `  k8s-info cluster
  k8s-info cluster -n kube-system
  k8s-info cluster --namespace default`,

    // RunE is the idiomatic choice when a command can fail.
    // It returns an error, Cobra prints it and exits with code 1.
    // Never call os.Exit inside RunE, let Cobra handle it.
    RunE: func(cmd *cobra.Command, args []string) error {
        // Build the Kubernetes config from the kubeconfig flag.
        // kubeconfigPath is inherited from the root persistent flag.
        config, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
        if err != nil {
            return fmt.Errorf("connect to cluster: %w", err)
        }

        clientset, err := kubernetes.NewForConfig(config)
        if err != nil {
            return fmt.Errorf("create client: %w", err)
        }

        // List pods in the given namespace.
        pods, err := clientset.CoreV1().Pods(namespace).List(
            context.Background(),
            metav1.ListOptions{},
        )
        if err != nil {
            return fmt.Errorf("list pods namespace=%q: %w", namespace, err)
        }

        // Count pods by phase.
        var running, pending, failed int
        for _, pod := range pods.Items {
            switch pod.Status.Phase {
            case "Running":
                running++
            case "Pending":
                pending++
            case "Failed":
                failed++
            }
        }

        fmt.Printf("Namespace: %s\n", namespace)
        fmt.Printf("Total pods: %d\n", len(pods.Items))
        fmt.Printf("  Running: %d\n", running)
        fmt.Printf("  Pending: %d\n", pending)
        fmt.Printf("  Failed:  %d\n", failed)

        return nil
    },
}

func init() {
    // Local flags only apply to this subcommand.
    // --namespace with -n shorthand, defaults to "default".
    clusterCmd.Flags().StringVarP(
        &namespace,
        "namespace", "n",
        "default",
        "namespace to inspect",
    )

    rootCmd.AddCommand(clusterCmd)
}

Persistent flags vs local flags

This is one of the most important concepts in Cobra and worth understanding clearly before your CLI grows.

Do you always have to type --kubeconfig?

No, and this is the part that trips people up. Because --kubeconfig has a default value set:

rootCmd.PersistentFlags().StringVar(
    &kubeconfigPath,
    "kubeconfig",
    os.Getenv("HOME")+"/.kube/config", // ← default is already set here
    "path to the kubeconfig file",
)

...kubeconfigPath is automatically populated with ~/.kube/config every time any command runs. You never need to type --kubeconfig unless you want to point at a different file:

# Default kubeconfig, flag not needed at all
k8s-info version
k8s-info cluster -n default

# Non-default kubeconfig, only specify when you need a different path
k8s-info --kubeconfig /path/to/other/config version
k8s-info --kubeconfig /path/to/other/config cluster

The real difference between persistent and local

The distinction that actually matters is this:

Persistent flag, declared on rootCmd with PersistentFlags()
    ↓ inherited by ALL subcommands, available everywhere in the CLI
    k8s-info --kubeconfig /other/config version     ✓ works
    k8s-info --kubeconfig /other/config cluster     ✓ works
    k8s-info version --kubeconfig /other/config     ✓ also works (order flexible)

Local flag, declared on a specific command with Flags()
    ↓ only available on THAT specific subcommand
    k8s-info cluster -n kube-system                 ✓ works
    k8s-info version -n kube-system                 ✗ unknown flag: -n

The last two lines are what matters. -n is a local flag on cluster, it doesn't exist on version. If a user tries to pass it to version, Cobra rejects it with a clear error.

The rule

Simple: if every subcommand needs the flag (like --kubeconfig), make it persistent on the root. If only one subcommand needs it (like --namespace for cluster), make it local. When in doubt, start local; you can always promote a flag to persistent later, but removing a persistent flag is a breaking change for your users.

Run vs RunE

Cobra gives you two options for the command function:

// Run, use when the command cannot fail
Run: func(cmd *cobra.Command, args []string) {
    fmt.Println("hello")
},

// RunE, use when the command can return an error
RunE: func(cmd *cobra.Command, args []string) error {
    if err := doSomething(); err != nil {
        return fmt.Errorf("something failed: %w", err)
    }
    return nil
},

Always prefer RunE for anything that does I/O, file reads, network calls, Kubernetes API calls. The error is printed automatically by Cobra and the process exits with code 1. Never call os.Exit inside RunE; that bypasses any deferred cleanup and makes testing harder.

The Makefile

A Makefile is a build automation file; it defines named commands called targets that you run with make <target>. Instead of typing long go build commands with flags every time, you type make build and the Makefile handles the rest.

Every target follows the same basic anatomy:

target_name: optional_dependency
    command to run

The dependency means "run this target first before mine." If install depends on build, running make install automatically runs make build first, you never have to remember the order.

Two rules that trip up everyone writing their first Makefile:

Commands must be indented with a real tab character: spaces look identical but Make rejects them with a cryptic error
.PHONY tells Make these are command names, not filenames: without it, Make looks for a file called build or install and gets confused when it doesn't find one

Variables work like this:

BINARY := ferctl        # declare a variable
$(BINARY)               # reference it, expands to "ferctl"

The @ prefix on a command suppresses echoing that line to the terminal, only the output is shown, not the command itself.

Here's the k8s-info Makefile with every line explained:

# BINARY is the name of the output binary, change this and everything updates.
BINARY  := k8s-info

# VERSION reads the current git tag automatically at build time.
# git describe returns something like "v1.2.0-3-gabcdef" for tagged commits.
# If you're not in a git repo, it falls back to "dev".
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")

# .PHONY declares these as command names, not file names.
# Without this, Make would look for files called "build", "install", etc.
.PHONY: build install clean

# build compiles the binary for the current platform.
# -ldflags="-s -w" strips debug symbols, reduces binary size by ~30%.
# -X sets a Go package-level variable at build time (explained below).
# -o $(BINARY) names the output file.
build:
    go build \
      -ldflags="-s -w -X github.com/FerRiosCosta/k8s-info/internal/cmd.Version=$(VERSION)" \
      -o $(BINARY) \
      ./cmd/k8s-info/...

# install depends on build, "make install" runs "make build" first automatically.
# Then copies the binary to /usr/local/bin so it's available system-wide.
install: build
    cp $(BINARY) /usr/local/bin/$(BINARY)
    @echo "installed $(BINARY) to /usr/local/bin"
    # @ suppresses printing the echo command, only the output is shown.

# clean removes the compiled binary so you can start fresh.
clean:
    rm -f $(BINARY)

The -X ldflag is new here compared to previous posts. It sets a Go package-level variable at build time; in this case cmd.Version. When you run make build in a git repository, git describe automatically fills in the version tag. When you run it outside git, it falls back to "dev".

The targets work together in a natural workflow:

make build    # compile the binary
make install  # compile + install to /usr/local/bin (runs build first)
make clean    # remove the binary and start fresh

Understanding versions and git describe

The VERSION variable in the Makefile uses git describe to automatically read a version from your git history. To understand how this works, it helps to understand what a version actually is in a git project.

What is a version?

A version is a named point in your git history, a label that says "this is the code that shipped as v1.0.0". In git, this is called a tag. A tag is just a pointer to a specific commit, with a human-readable name.

Versions typically follow semantic versioning (semver):

v1.2.3
│ │ └─── patch: bug fixes, no new features
│ └───── minor: new features, backward compatible
└─────── major: breaking changes

Examples of what each bump means in practice:

v1.0.0  → first public release
v1.0.1  → fixed a bug in the cluster command
v1.1.0  → added a new "nodes" subcommand
v2.0.0  → renamed flags, changed output format (breaking)

How to create a version in git

First make sure your code is committed:

git add .
git commit -m "feat: add cluster subcommand"

Then create a tag:

# Create a tag pointing at the current commit
git tag v1.0.0

# Create a tag with an annotation message (recommended)
git tag -a v1.0.0 -m "First stable release of k8s-info"

# Push the tag to GitHub (tags dont push automatically with git push)
git push origin v1.0.0

# Or push all tags at once
git push origin --tags

How git describe uses your tags

git describe reads the most recent tag in your git history and builds a version string from it:

git describe --tags --always --dirty

The output depends on where you are in your commit history:

v1.0.0                    ← exactly on the tagged commit
v1.0.0-3-gabcdef          ← 3 commits after v1.0.0, current hash is abcdef
v1.0.0-3-gabcdef-dirty    ← same, but you have uncommitted changes
abcdef                    ← no tags exist yet, falls back to commit hash

The flags we use:

Flag	What it does
`--tags`	Considers all tags, not just annotated ones
`--always`	Falls back to the commit hash if no tags exist
`--dirty`	Appends `-dirty` if there are uncommitted changes

And 2>/dev/null || echo "dev" handles the case where git isn't available at all (like in a CI environment without git installed); it silently swallows the error and falls back to "dev".

What this means for your binary

When you run make build with a tagged commit:

git tag -a v1.0.0 -m "first release"
make build
./k8s-info version
# k8s-info version v1.0.0

When you run it after adding more commits:

git commit -m "fix: handle empty namespace"
make build
./k8s-info version
# k8s-info version v1.0.0-1-gabcdef

This tells you exactly which commit the binary was built from, invaluable when debugging a production issue and trying to figure out what version is actually running.

A practical workflow for releasing

# 1. Make sure everything is committed and tested
git status              # should show nothing uncommitted
make test               # all tests pass

# 2. Create the version tag
git tag -a v1.0.0 -m "Initial release of k8s-info"

# 3. Build the binary, version is embedded automatically
make build
./k8s-info version
# k8s-info version v1.0.0

# 4. Install it
make install

# 5. Push the tag to GitHub
git push origin v1.0.0

From this point on, anyone who clones your repo and runs make build on that commit gets a binary that reports v1.0.0. No manual version files to maintain, no risk of forgetting to update a constant; git is the source of truth.

Running the CLI

Build and install:

make install

Try the commands:

# Show help — generated automatically by Cobra
k8s-info --help

# Show version
k8s-info version
# k8s-info version dev

# Show cluster summary for the default namespace
k8s-info cluster

# Namespace: default
# Total pods: 3
# Running: 3
# Pending: 0
# Failed:  0

# Inspect kube-system
k8s-info cluster -n kube-system

# Namespace: kube-system
# Total pods: 7
# Running: 7
# Pending: 0
# Failed:  0

# Show help for the cluster subcommand
k8s-info cluster --help

What Cobra generates automatically

Run k8s-info --help and notice what you didn't have to write:

A CLI for inspecting your Kubernetes cluster

Usage:
  k8s-info [command]

Available Commands:
  cluster     Show a summary of running pods in your cluster
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  version     Print the k8s-info version

Flags:
  -h, --help                help for k8s-info
      --kubeconfig string   path to the kubeconfig file (default "/Users/yourname/.kube/config")

Use "k8s-info [command] --help" for more information about a command.

The completion and help subcommands, the --help flag, the usage text, the default values, all generated for free. That's what using Cobra buys you.

How init() wires everything together

You may have noticed that every file calls rootCmd.AddCommand() inside an init() function. This is Go's package initialization mechanism; init() runs automatically when the package is imported, before main().

main() calls cmd.Execute()
      │
      ▼
Go initializes the cmd package
      │
      ├── root.go   init() runs → registers --kubeconfig flag
      ├── version.go init() runs → registers versionCmd on root
      └── cluster.go init() runs → registers clusterCmd on root
      │
      ▼
rootCmd.Execute() runs with all subcommands registered

This is why you can add a new subcommand by creating a new file and calling rootCmd.AddCommand() in its init(), no other file needs to change. The root command picks it up automatically.

Common gotchas

Using Run instead of RunE for fallible commands

If your command calls a Kubernetes API or reads a file and you use Run instead of RunE, you have to handle errors yourself, usually with fmt.Fprintln(os.Stderr, err); os.Exit(1). This is verbose, inconsistent, and bypasses Cobra's error handling. Always use RunE for anything that can fail.

Forgetting SilenceUsage

Without SilenceUsage: true, every error prints the full usage text, which is almost never what you want when a flag value is wrong or a Kubernetes call fails. Add it to the root command and save your users the confusion.

Persistent flags vs local flags confusion

If you declare a flag with Flags() on the root command, it's still a local flag, only available directly on the root, not inherited by subcommands. Use PersistentFlags() on the root for flags that should be available everywhere.

Flag order matters to users

Cobra parses flags after the command name. This means:

k8s-info --kubeconfig ~/.kube/config cluster -n default   ✓ works
k8s-info cluster --kubeconfig ~/.kube/config -n default   ✓ also works
k8s-info cluster -n default --kubeconfig ~/.kube/config   ✓ also works

All three are equivalent; Cobra is flexible about flag position. But document a consistent order in your examples to avoid confusing users.

init() order is not guaranteed across files

Go guarantees that init() functions within a single file run in order, but the order between files in the same package is not guaranteed. This is fine for Cobra; AddCommand() calls are independent of each other. But don't write init() functions that depend on each other across files.

Summary

Cobra turns a Go program into a proper CLI in three concepts:

rootCmd: the entry point, owns persistent flags, registers subcommands
AddCommand(): how subcommands attach to the tree
RunE: the function that runs when a command is invoked, returns an error or nil

The init() pattern means adding a new subcommand is always: create a file, write the command, call rootCmd.AddCommand() in init(). Nothing else changes.

This is exactly the structure the next command uses in post 5; the root command, the persistent --kubeconfig flag, the top subcommand registered in its own file, all wired together by init().

What's next

In nex post we take this Cobra foundation and build ferctl top on top of it, adding the metrics-server client, resource threshold warnings, and tabwriter output. The k8s-info cluster command becomes the starting point for the richer ferctl top command.

Let's connect

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, platform engineering, jazz, or jiu-jitsu. I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Talking to Kubernetes from Go: a practical client-go guide

Fer Rios — Mon, 13 Jul 2026 13:34:48 +0000

This is part of the Platform Engineering with Go series. This post builds on the minikube cluster set up in post 2. Read post 2 first if you haven't yet.

kubectl is great for humans. But what about programs?

When you run kubectl get pods, something has to talk to the Kubernetes API server, deserialize the JSON response, and format it for your terminal. That something is client-go, the official Go client library for Kubernetes.

Every tool in the Kubernetes ecosystem is built on it. kubectl itself, helm, ArgoCD, controller-runtime, Tekton. If it talks to a Kubernetes cluster and it's written in Go, it's using client-go.

In this post, we'll learn how client-go works by building something immediately useful: a cluster health reporter. One Go program that connects to your minikube cluster, scans every pod across every namespace, and prints a complete health summary, the kind of view that usually takes four or five kubectl commands to assemble.

Cluster Health Report
Generated: 2026-06-24 14:32:00 UTC

NAMESPACE     TOTAL   RUNNING   FAILING   PENDING
──────────────────────────────────────────────────
default           3         3         0         0
kube-system       6         6         0         0

✓ All pods healthy

And if something is wrong:

Cluster Health Report
Generated: 2026-06-24 14:32:00 UTC

NAMESPACE     TOTAL   RUNNING   FAILING   PENDING
──────────────────────────────────────────────────
default           3         1         2         0
kube-system       6         6         0         0

FAILING PODS
──────────────────────────────────────────────────
default   go-api-7d6b9f8c4-mn9rt   CrashLoopBackOff   14 restarts
default   go-api-7d6b9f8c4-p8wvz   OOMKilled            3 restarts

What you'll learn

What client-go is and how it relates to the Kubernetes API.
The four key packages and what each one contains.
How to connect to a cluster, out-of-cluster and in-cluster.
How the Clientset is organized, API groups explained.
How to list, filter, and inspect Kubernetes resources from Go.
How to build the cluster health reporter step by step.
How to test client-go code, two approaches with clear rules for when to use each.

Prerequisites

minikube running with the go-api deployment from post 2

What client-go is

The Kubernetes API is a REST API. When you run kubectl get pods -n default, kubectl makes an HTTP GET request to something like:

GET https://kubernetes/api/v1/namespaces/default/pods

The API server returns JSON. kubectl deserializes it and formats it for your terminal.

client-go is the library that handles all of this in Go: the HTTP calls, TLS, authentication, rate limiting, and deserialization into typed Go structs. Instead of working with raw JSON, you get a *corev1.PodList with typed fields you can navigate in your editor with autocomplete.

Your Go program
      │
      │  client-go
      │  ├── kubeconfig loading
      │  ├── TLS + authentication
      │  ├── HTTP client + rate limiting
      │  └── JSON → typed Go structs
      │
      ▼
Kubernetes API server
      │
      ▼
etcd (the cluster's data store)

Everything you can do with kubectl, you can do with client-go, and more. kubectl is itself a client-go program.

Project structure

Here is the Go project structure we are going to build; remember, you can see how to organize the code for your own projects here.

You can find the entire project here.

go-k8s-health-reporter/
├── go.mod
├── main.go                  ← entry point — wires dependencies, calls reporter
└── internal/
    ├── reporter/
    │   ├── types.go         ← data types — PodStatus, NamespaceSummary, Report
    │   ├── reporter.go      ← Reporter struct, Generate()
    │   ├── inspector.go     ← pod health detection — inspectPod(), pendingReason()
    │   └── printer.go       ← output formatting — Print(), tabwriter tables
    └── kubernetes/
        └── client.go        ← KubeClient interface + real implementation

Installing the packages

Before writing any code, initialize the module and install the dependencies. The version numbers must match your Kubernetes cluster; since we're running Kubernetes 1.36.2, we use v0.36.2 for all three packages. Mixing versions causes compilation errors because the types won't match across packages. Remember to rename the module's name to your own module name.

# Initialize the Go module
mkdir go-k8s-health-reporter && cd go-k8s-health-reporter
go mod init github.com/FerRiosCosta/go-k8s-health-reporter

# Install the four packages pinned to Kubernetes 1.36.2
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.36.2
go get k8s.io/apimachinery@v0.36.2

# Tidy up — removes unused dependencies and ensures versions are consistent
go mod tidy

If you see a missing go.sum entry error after installing the packages, it means some transitive dependencies, packages that client-go depends on, like k8s.io/klog/v2, weren't added to go.sum automatically. Fix it with:

bash
go get ./...
go mod tidy
go get ./... scans all packages in your module and fetches every dependency recursively, including transitive ones. go mod tidy then writes the complete go.sum.

Why pin to v0.36.2 instead of @latest? The client-go versioning scheme directly mirrors Kubernetes releases; v0.36.2 corresponds to Kubernetes v1.36.2. Always match your client-go version to your cluster version. If you run @latest you might get a newer client-go version than your cluster supports, which can cause unexpected API behavior.

You can verify your cluster version at any time:

kubectl version

# Client Version: v1.36.2
# Server Version: v1.36.2

Both should match (ideally). If they don't, pin your packages to match the server version.

Note: k8s.io/metrics is not needed for the health reporter in this post, but you will need it for the next posts.
go get k8s.io/metrics@v0.36.2
go mod tidy

Here's the go.mod:

module github.com/FerRiosCosta/go-k8s-health-reporter

go 1.26

require (
    k8s.io/api          v0.36.2
    k8s.io/apimachinery v0.36.2
    k8s.io/client-go    v0.36.2
)

The four key packages

Before writing any code, let's understand the four packages client-go code almost always imports. This is the section that makes every future client-go import immediately readable.

k8s.io/api: the Go type definitions

import (
    corev1  "k8s.io/api/core/v1"   // Pod, Service, ConfigMap, Secret, Namespace, Node
    appsv1  "k8s.io/api/apps/v1"   // Deployment, StatefulSet, DaemonSet, ReplicaSet
    batchv1 "k8s.io/api/batch/v1"  // Job, CronJob
)

This package contains the Go struct definitions for every Kubernetes resource. When you write corev1.Pod{} or appsv1.Deployment{}, you're using this package. The aliases (corev1, appsv1) are conventions, the actual package names are just v1, which is too generic to use unaliased.

k8s.io/apimachinery: the shared building blocks

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" // ObjectMeta, ListOptions, Time
    "k8s.io/apimachinery/pkg/api/resource"         // resource.Quantity (CPU, memory)
)

This package contains types shared across all Kubernetes API groups. metav1.ObjectMeta is the block every resource has: name, namespace, labels, annotations, creation timestamp. metav1.ListOptions lets you filter API calls by label selectors and field selectors. You'll import something from this package in virtually every file that touches the Kubernetes API.

k8s.io/client-go: the actual client

import (
    "k8s.io/client-go/kubernetes"        // the Clientset — entry point to all APIs
    "k8s.io/client-go/rest"              // rest.Config — holds connection details
    "k8s.io/client-go/tools/clientcmd"   // reads and parses kubeconfig files
)

This is the main package. kubernetes.NewForConfig() creates the Clientset, your entry point to every Kubernetes API. clientcmd reads the ~/.kube/config file that minikube wrote in post 2. rest.Config holds the connection details: API server URL, TLS certificates, auth token.

k8s.io/metrics: the metrics-server API

import (
    metrics "k8s.io/metrics/pkg/client/clientset/versioned" // metrics Clientset
)

This is a separate package because metrics-server is an extension API, not part of the core Kubernetes API.

# Install in post 4 — pinned to the same version as the rest
go get k8s.io/metrics@v0.36.2
go mod tidy

Connecting to the cluster

Every client-go program starts the same way: build a rest.Config, then create a Clientset from it.

// internal/kubernetes/client.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
)

// KubeClient defines the Kubernetes operations the health reporter uses.
// Wrapping the Clientset behind an interface makes the reporter testable
// without a real cluster, the same pattern used throughout this series.
type KubeClient interface {
    // ListPods returns all pods in the given namespace.
    // An empty namespace string returns pods from all namespaces.
    ListPods(ctx context.Context, namespace string) (*corev1.PodList, error)
}

// client is the real implementation backed by the official Kubernetes Clientset.
// It is unexported, callers always use the KubeClient interface.
type client struct {
    clientset kubernetes.Interface
}

// NewClient creates a KubeClient connected to the cluster in kubeconfig.
func NewClient(kubeconfigPath string) (KubeClient, error) {
    config, err := buildConfig(kubeconfigPath)
    if err != nil {
        return nil, fmt.Errorf("build config: %w", err)
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create clientset: %w", err)
    }

    return &client{clientset: clientset}, nil
}

// buildConfig tries in-cluster config first, then falls back to kubeconfig.
// This means the same binary works both locally (reads ~/.kube/config)
// and when deployed as a pod inside a cluster (reads the mounted service account token).
func buildConfig(kubeconfigPath string) (*rest.Config, error) {
    // in-cluster config: reads /var/run/secrets/kubernetes.io/serviceaccount/token
    // automatically mounted inside every pod by Kubernetes.
    if config, err := rest.InClusterConfig(); err == nil {
        return config, nil
    }

    // out-of-cluster config: reads the kubeconfig file from disk.
    // This is the file minikube wrote to ~/.kube/config in post 2.
    return clientcmd.BuildConfigFromFlags("", kubeconfigPath)
}

// ListPods retrieves all pods in the given namespace from the Kubernetes API.
// Passing an empty string returns pods from all namespaces.
func (c *client) ListPods(ctx context.Context, namespace string) (*corev1.PodList, error) {
    pods, err := c.clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pods namespace=%q: %w", namespace, err)
    }
    return pods, nil
}

The Clientset and API groups

c.clientset.CoreV1().Pods(namespace).List(ctx, opts), let's break this chain down because you'll see it everywhere in client-go code:

c.clientset           // the Clientset, entry point to all Kubernetes APIs
    .CoreV1()         // select the "core" API group, version 1
    .Pods(namespace)  // select the Pods resource in the given namespace
    .List(ctx, opts)  // call the List operation with options

Kubernetes organizes resources into API groups. Each group contains related resources:

CoreV1()  → Pods, Services, ConfigMaps, Secrets, Nodes, Namespaces
AppsV1()  → Deployments, StatefulSets, DaemonSets, ReplicaSets
BatchV1() → Jobs, CronJobs

The pattern is always the same: clientset.APIGroup().ResourceType(namespace).Operation(). Once you know this pattern, any client-go call is readable.

Building the cluster health reporter

Now let's build the reporter. Instead of putting everything in one file, we'll split it across four focused files from the start, each with one clear responsibility.

internal/reporter/
├── types.go      ← data types only — PodStatus, NamespaceSummary, Report
├── reporter.go   ← Reporter struct, constructor, Generate()
├── inspector.go  ← pod inspection logic — detects failing and pending states
└── printer.go    ← output formatting — tabwriter tables

Each file has one reason to change:

New data field → types.go
New data source → reporter.go
New failure reason to detect → inspector.go
New output format (JSON, CSV) → printer.go

types.go, data types only

// internal/reporter/types.go
package reporter

import "time"

// PodStatus represents the health state of a single pod.
type PodStatus struct {
    Namespace string
    Name      string
    Phase     string // Running, Pending, Failed, Succeeded
    Reason    string // CrashLoopBackOff, OOMKilled, etc. — more specific than Phase
    Restarts  int32  // total restarts across all containers
    IsFailing bool   // true if the pod is in a non-running problem state
    IsPending bool   // true if the pod is waiting to be scheduled or started
}

// NamespaceSummary holds the pod counts for a single namespace.
type NamespaceSummary struct {
    Namespace string
    Total     int
    Running   int
    Failing   int
    Pending   int
}

// Report is the complete cluster health picture.
type Report struct {
    GeneratedAt time.Time
    Namespaces  []NamespaceSummary
    Failing     []PodStatus
    Pending     []PodStatus
}

Thirty lines. No imports beyond time. If you need to add a field to a pod's health state, you open this file and only this file.

inspector.go, pod health detection

// internal/reporter/inspector.go
package reporter

import corev1 "k8s.io/api/core/v1"

// inspectPod examines a pod's status and returns a detailed PodStatus.
// It checks container states first, a pod can be in "Running" phase
// while individual containers are in CrashLoopBackOff or OOMKilled.
func inspectPod(pod *corev1.Pod) PodStatus {
    status := PodStatus{
        Namespace: pod.Namespace,
        Name:      pod.Name,
        Phase:     string(pod.Status.Phase),
    }

    // Sum restart counts and check container states.
    // Must check containers before pod phase, a pod can be "Running"
    // while a container inside it is waiting to restart.
    for _, cs := range pod.Status.ContainerStatuses {
        status.Restarts += cs.RestartCount

        if cs.State.Waiting != nil {
            switch cs.State.Waiting.Reason {
            case "CrashLoopBackOff", "OOMKilled", "ImagePullBackOff",
                "ErrImagePull", "CreateContainerConfigError":
                status.Reason = cs.State.Waiting.Reason
                status.IsFailing = true
                return status
            }
        }

        if cs.State.Terminated != nil {
            reason := cs.State.Terminated.Reason
            if reason == "OOMKilled" || cs.State.Terminated.ExitCode != 0 {
                status.Reason = reason
                status.IsFailing = true
                return status
            }
        }
    }

    // Check pod-level phase for broader failure states.
    switch pod.Status.Phase {
    case corev1.PodFailed:
        status.Reason = pod.Status.Reason
        status.IsFailing = true
    case corev1.PodPending:
        status.Reason = pendingReason(pod)
        status.IsPending = true
    }

    return status
}

// pendingReason extracts why a pod is pending from its conditions.
// The PodScheduled condition contains the human-readable scheduling
// failure message; e.g. "0/1 nodes are available: 1 Insufficient memory."
func pendingReason(pod *corev1.Pod) string {
    for _, condition := range pod.Status.Conditions {
        if condition.Type == corev1.PodScheduled &&
            condition.Status == corev1.ConditionFalse {
            return condition.Message
        }
    }
    return string(pod.Status.Phase)
}

Understanding container states

The inspectPod function works by reading pod.Status.ContainerStatuses, a slice that contains one entry per container in the pod. Each entry has a State field that holds exactly one of three possible states at any given moment:

Waiting: the container is not yet running. This is where most failure reasons live:

Reason	What it means
`ContainerCreating`	Image is being pulled, or volumes are being mounted
`PodInitializing`	Init containers are still running, main container hasn't started
`CrashLoopBackOff`	Container keeps crashing, Kubernetes is backing off before restarting
`ImagePullBackOff`	Image pull failed, Kubernetes is backing off before retrying
`ErrImagePull`	The immediate image pull error, before backoff kicks in
`CreateContainerConfigError`	Bad config, missing ConfigMap, Secret, or invalid env var
`OOMKilled`	Killed by the OOM killer, waiting to restart

Running: the container process is alive. Note that this doesn't mean the application inside it is healthy, it just means the process is executing. Your app could still be returning 500s or stuck in a deadlock.

Terminated: the container has finished. The most important fields are:

Field	What it means
`ExitCode`	`0` = clean exit, anything else = failure
`Reason`	`Completed` (clean) or `OOMKilled` (killed by OOM killer)

Common exit codes worth knowing:

Exit code	Meaning
`0`	Clean exit, completed successfully
`1`	Application error, the process itself returned an error
`137`	`SIGKILL`, OOM killer fired, or forcefully killed
`143`	`SIGTERM`, graceful shutdown (normal during pod termination)

This is exactly why inspectPod checks cs.State.Waiting and cs.State.Terminated separately; each state carries different information in different fields.

Init containers, the hidden failure

There's one more status field worth knowing about:

pod.Status.ContainerStatuses      // current state of main containers
pod.Status.InitContainerStatuses  // current state of init containers

Init containers run before your main containers and go through the same three states. If an init container is stuck in CrashLoopBackOff, your main container will never start, and pod.Status.ContainerStatuses will be empty, giving you no indication of what's wrong.

The current inspectPod implementation only checks main container statuses. A more complete version would check init containers first:

// A more complete inspectPod that also catches stuck init containers.
// This is left as an exercise — the pattern is identical to the main
// container check, just reading from InitContainerStatuses instead.
func inspectPod(pod *corev1.Pod) PodStatus {
    status := PodStatus{
        Namespace: pod.Namespace,
        Name:      pod.Name,
        Phase:     string(pod.Status.Phase),
    }

    // Check init containers first — if one is stuck, main containers
    // will never start and ContainerStatuses will be empty.
    for _, cs := range pod.Status.InitContainerStatuses {
        status.Restarts += cs.RestartCount
        if cs.State.Waiting != nil {
            switch cs.State.Waiting.Reason {
            case "CrashLoopBackOff", "OOMKilled", "ImagePullBackOff",
                "ErrImagePull", "CreateContainerConfigError":
                status.Reason = "init: " + cs.State.Waiting.Reason
                status.IsFailing = true
                return status
            }
        }
    }

    // Then check main containers as before.
    for _, cs := range pod.Status.ContainerStatuses {
        // ... same logic as above
    }

    // ... rest of the function unchanged
}

Prefixing the reason with "init: " makes it immediately clear in the report that it's an init container failing, not the main application.

You can add this code block before checking the main container in your project if you want.

printer.go, output formatting

// internal/reporter/printer.go
package reporter

import (
    "fmt"
    "io"
    "strings"
    "text/tabwriter"
)

// Print writes the health report as a formatted table to w.
// Accepting io.Writer instead of os.Stdout directly keeps this
// testable — callers can pass a bytes.Buffer to inspect the output.
func (r *Reporter) Print(w io.Writer, report *Report) {
    fmt.Fprintf(w, "Cluster Health Report\n")
    fmt.Fprintf(w, "Generated: %s\n\n",
        report.GeneratedAt.Format("2006-01-02 15:04:05 UTC"),
    )

    printNamespaceSummary(w, report.Namespaces)

    if len(report.Failing) > 0 {
        printFailingPods(w, report.Failing)
        return
    }

    if len(report.Pending) > 0 {
        printPendingPods(w, report.Pending)
        return
    }

    fmt.Fprintf(w, "\n✓ All pods healthy\n")
}

// printNamespaceSummary writes the namespace totals table.
func printNamespaceSummary(w io.Writer, namespaces []NamespaceSummary) {
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    fmt.Fprintln(tw, "NAMESPACE\tTOTAL\tRUNNING\tFAILING\tPENDING")
    fmt.Fprintln(tw, strings.Repeat("─", 52))
    for _, ns := range namespaces {
        fmt.Fprintf(tw, "%s\t%d\t%d\t%d\t%d\n",
            ns.Namespace, ns.Total, ns.Running, ns.Failing, ns.Pending,
        )
    }
    tw.Flush()
}

// printFailingPods writes the failing pods detail section.
func printFailingPods(w io.Writer, pods []PodStatus) {
    fmt.Fprintf(w, "\nFAILING PODS\n")
    fmt.Fprintln(w, strings.Repeat("─", 52))
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    for _, p := range pods {
        fmt.Fprintf(tw, "%s\t%s\t%s\t%d restarts\n",
            p.Namespace, p.Name, p.Reason, p.Restarts,
        )
    }
    tw.Flush()
}

// printPendingPods writes the pending pods detail section.
func printPendingPods(w io.Writer, pods []PodStatus) {
    fmt.Fprintf(w, "\nPENDING PODS\n")
    fmt.Fprintln(w, strings.Repeat("─", 52))
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    for _, p := range pods {
        fmt.Fprintf(tw, "%s\t%s\t%s\n",
            p.Namespace, p.Name, p.Reason,
        )
    }
    tw.Flush()
}

reporter.go, orchestration only

// internal/reporter/reporter.go
package reporter

import (
    "context"
    "fmt"
    "time"

    k8s "github.com/FerRiosCosta/go-k8s-health-reporter/internal/kubernetes"
)

// Reporter builds health reports from Kubernetes data.
// It depends on the KubeClient interface, never on a concrete
// client-go type, so it works with both real and fake clients.
type Reporter struct {
    kube k8s.KubeClient
}

// New creates a Reporter with the given KubeClient injected.
func New(kube k8s.KubeClient) *Reporter {
    return &Reporter{kube: kube}
}

// Generate fetches all pods from the cluster and builds a health report.
func (r *Reporter) Generate(ctx context.Context) (*Report, error) {
    // Fetch all pods across all namespaces.
    // Passing "" to ListPods returns pods from every namespace.
    pods, err := r.kube.ListPods(ctx, "")
    if err != nil {
        return nil, fmt.Errorf("fetch pods: %w", err)
    }

    report := &Report{
        GeneratedAt: time.Now().UTC(),
    }

    // Group pods by namespace using a map.
    summaries := make(map[string]*NamespaceSummary)

    for _, pod := range pods.Items {
        // getOrCreate avoids the boilerplate of checking if the
        // namespace key exists before writing to the map.
        summary := getOrCreate(summaries, pod.Namespace)
        summary.Total++

        // inspectPod is defined in inspector.go, same package,
        // no import needed.
        status := inspectPod(&pod)

        switch {
        case status.IsFailing:
            summary.Failing++
            report.Failing = append(report.Failing, status)
        case status.IsPending:
            summary.Pending++
            report.Pending = append(report.Pending, status)
        default:
            summary.Running++
        }
    }

    for _, summary := range summaries {
        report.Namespaces = append(report.Namespaces, *summary)
    }

    return report, nil
}

// getOrCreate returns the existing NamespaceSummary for the given namespace
// or creates a new one and registers it in the map.
// Extracted into its own function to keep Generate() readable.
func getOrCreate(m map[string]*NamespaceSummary, namespace string) *NamespaceSummary {
    if s, ok := m[namespace]; ok {
        return s
    }
    s := &NamespaceSummary{Namespace: namespace}
    m[namespace] = s
    return s
}

Notice that reporter.go now has no output imports (tabwriter, strings, io) and no Kubernetes type imports beyond what it needs for ListPods. The corev1 import moved entirely to inspector.go where it belongs. The reporter package is still one package, Go compiles all four files together, but each file is now independently readable.

Wire it together in main.go

// main.go
package main

import (
    "context"
    "fmt"
    "os"

    k8s "github.com/FerRiosCosta/go-k8s-health-reporter/internal/kubernetes"
    "github.com/FerRiosCosta/go-k8s-health-reporter/internal/reporter"
)

func main() {
    // Build the Kubernetes client using the kubeconfig minikube wrote in post 2.
    // This is the same file kubectl uses, no extra setup needed.
    kubeconfigPath := os.Getenv("HOME") + "/.kube/config"
    kube, err := k8s.NewClient(kubeconfigPath)
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to connect to cluster: %v\n", err)
        os.Exit(1)
    }

    // Build the reporter with the client injected.
    r := reporter.New(kube)

    // Generate and print the report.
    ctx := context.Background()
    report, err := r.Generate(ctx)
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to generate report: %v\n", err)
        os.Exit(1)
    }

    r.Print(os.Stdout, report)
}

Verifying against minikube

The best way to verify the health reporter works is to run it against your real minikube cluster. No fakes, no mocks, just your actual cluster from post 2.

Healthy cluster

Make sure minikube is running and the go-api deployment from post 2 is up:

kubectl get pods
# NAME                      READY   STATUS    RESTARTS   AGE
# go-api-7d6b9f8c4-xk2pq   1/1     Running   0          2d
# go-api-7d6b9f8c4-mn9rt   1/1     Running   0          2d
# go-api-7d6b9f8c4-p8wvz   1/1     Running   0          2d

Run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-06-29 15:58:43 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
default                3   3   0   0
kube-system            8   8   0   0
kubernetes-dashboard   2   2   0   0

✓ All pods healthy

Simulating a failing pod

To verify the failing pod detection works we need a pod that reliably OOMKills. Our go-api binary turns out to be too lean to reliably trigger OOMKill at the Docker minimum, the multi-stage Dockerfile produced an extremely efficient binary. That's good engineering, but it makes for a tricky simulation.

Instead, let's deploy a pod that deliberately holds memory using Python's bytearray. Unlike shell tools like dd which stream data through memory and release it immediately, bytearray allocates a contiguous block and holds it, guaranteeing an OOMKill with a tight limit.

Create a file called memory-hog.yaml:

# memory-hog.yaml
apiVersion: v1
kind: Pod
metadata:
  name: memory-hog
  namespace: default
spec:
  # restartPolicy: Always ensures Kubernetes keeps restarting after each
  # OOMKill, driving the pod into CrashLoopBackOff.
  restartPolicy: Always
  containers:
  - name: hog
    image: python:3.12-alpine
    command: ["python3", "-c"]
    args:
    - |
      import time
      # bytearray allocates 100MB and holds it in physical memory.
      # The moment this runs, usage exceeds the 10Mi limit
      # and the OOM killer fires immediately.
      data = bytearray(100 * 1024 * 1024)
      print("allocated 100MB, holding...", flush=True)
      time.sleep(3600)
    resources:
      requests:
        memory: "10Mi"
      limits:
        memory: "10Mi"

Apply it:

kubectl apply -f memory-hog.yaml

Watch the pod:

kubectl get pods -w

# NAME          READY   STATUS             RESTARTS   AGE
# memory-hog   0/1     ContainerCreating  0          1s
# memory-hog   0/1     OOMKilled          0          4s   ← transient
# memory-hog   0/1     CrashLoopBackOff   1          10s  ← stable, run the reporter now

Once you see CrashLoopBackOff, run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-07-01 21:37:47 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
kubernetes-dashboard   2   2   0   0
default                4   3   1   0
kube-system            8   8   0   0

FAILING PODS
────────────────────────────────────────────────────
default   memory-hog   CrashLoopBackOff   13 restarts

The reporter correctly identifies the failing pod, the reason, and the restart count.

Clean up when done:

kubectl delete pod memory-hog

Simulating a pending pod

To verify pending detection, set a memory request higher than your node can provide. Again, both requests and limits must be set together:

kubectl set resources deployment/go-api \
  --requests=memory=100Gi \
  --limits=memory=100Gi \
  -n default

The pods can't be scheduled, no node has 100Gi available. Run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-07-01 22:03:51 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
default                4   3   0   1
kube-system            8   8   0   0
kubernetes-dashboard   2   2   0   0

PENDING PODS
────────────────────────────────────────────────────
default   go-api-6ff6496cb5-dnq6m   0/1 nodes are available: 1 Insufficient memory. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

The reason message comes directly from the pod's scheduling condition, exactly what the pendingReason() function extracts. Restore the correct values when done:

kubectl set resources deployment/go-api \
  --requests=memory=32Mi \
  --limits=memory=64Mi \
  -n default

Note on testing with fakes: In a future post we'll cover how to write unit tests for this reporter using interface fakes and client-go's built-in fake clientset, so the test suite runs without a real cluster. For now, minikube is our test environment.

Common gotchas

Wrong kubeconfig context

The most common mistake when starting out with client-go. If you have multiple clusters in your kubeconfig, always verify before running Go code:

kubectl config current-context
# Should print: minikube

If it shows a production cluster, switch back:

kubectl config use-context minikube

Passing "" vs "default" to namespace-scoped resources

// Returns pods from ALL namespaces
clientset.CoreV1().Pods("").List(ctx, metav1.ListOptions{})

// Returns pods from the "default" namespace only
clientset.CoreV1().Pods("default").List(ctx, metav1.ListOptions{})

These behave completely differently. The empty string means "all namespaces", not "the default namespace." This is one of the most common sources of confusion in client-go code.

Pod phase vs container state, the big one

A pod in Running phase can have containers in CrashLoopBackOff. The phase is the pod-level state; the container state is what's actually happening inside. Always check both:

// WRONG: misses CrashLoopBackOff pods
if pod.Status.Phase != corev1.PodRunning {
    // handle failing pod
}

// RIGHT: catches CrashLoopBackOff even when phase is Running
for _, cs := range pod.Status.ContainerStatuses {
    if cs.State.Waiting != nil && cs.State.Waiting.Reason == "CrashLoopBackOff" {
        // handle failing container
    }
}

Creating a new Clientset per function call

client-go's Clientset maintains an internal HTTP connection pool and rate limiter. Creating a new one on every call throws away the connection pool and can trigger API server rate limits. Always create the Clientset once at startup and reuse it:

// WRONG: new Clientset every call, no connection reuse
func listPods() {
    clientset, _ := kubernetes.NewForConfig(config)
    clientset.CoreV1().Pods("").List(...)
}

// RIGHT: Clientset created once, reused everywhere
var clientset kubernetes.Interface

func init() {
    clientset, _ = kubernetes.NewForConfig(config)
}

func listPods() {
    clientset.CoreV1().Pods("").List(...)
}

API version mismatches, always pin all three packages together

All three packages, k8s.io/api, k8s.io/apimachinery, and k8s.io/client-go, must be on the same minor version. They are developed and released together as part of each Kubernetes release cycle. Mixing versions causes compile errors because types from one package won't match types expected by another.

The version number maps directly to your Kubernetes cluster: v0.36.2 = Kubernetes v1.36.2. Since we're running Kubernetes 1.36.2 in minikube, v0.36.2 is the right choice throughout this series:

# WRONG: mixing minor versions causes type mismatch errors at compile time
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.35.0          # different minor version — will break
go get k8s.io/apimachinery@v0.34.0  # different minor version — will break

# RIGHT: all three pinned to match Kubernetes 1.36.2
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.36.2
go get k8s.io/apimachinery@v0.36.2
go mod tidy

Always verify your cluster version before pinning:

kubectl version 

# Client Version: v1.36.2
# Server Version: v1.36.2

If your client and server versions differ, for example Client Version: v1.36.2 and Server Version: v1.35.1, that's completely fine. Kubernetes has an official version skew policy: kubectl can be up to one minor version ahead or behind the server. Pin your client-go packages to match whichever version you're running in the cluster.

Summary

client-go is the foundation everything in the Kubernetes Go ecosystem is built on. Four packages, one Clientset, typed access to every resource the API server exposes.

Three things to take away:

The Clientset pattern: clientset.APIGroup().Resource(namespace).Operation(), is consistent across every resource type. Learn it once, read any client-go code.
Always check container states, not just pod phase, a "Running" pod can have failing containers.
Wrap the Clientset behind an interface from the start, it costs nothing and makes every function that uses it instantly testable.

What's next

The health reporter we built here is a real tool. Run it on any cluster, and it gives you immediate operational value. It's also the direct foundation for a new tool we are going to build in the later chapter:ferctl top: a full CLI with Cobra, flags, tabwriter output, and near-limit warnings. The health reporter becomes a subcommand. The KubeClient interface gets extended with metrics.

Every pattern, every import, and every interface in this post appears in the next post.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Setting up a local Kubernetes cluster with minikube

Fer Rios — Mon, 06 Jul 2026 15:07:43 +0000

This is part of the Platform Engineering with Go series. View all posts in the series

Every post in this series assumes you have a Kubernetes cluster

And that's a problem, because most engineers don't have one sitting around they can freely experiment on. Spinning up an EKS cluster just to follow a tutorial costs money, requires an AWS account, and takes longer than you want. Using your production cluster is obviously off the table.

minikube fixes this. It gives you a real, full Kubernetes cluster running on your laptop in under 5 minutes. Same API as production. Free. No cloud account needed. When you're done for the day, you stop it. When things get messy, you delete it and start fresh.

By the end of this post you'll have a local cluster running with metrics-server enabled and a Go app deployed inside it.

What you'll learn

What minikube is and how it relates to a real production cluster
How to install minikube on Mac and Linux
How to start, stop, and manage a local cluster
How minikube connects to kubectl and your kubeconfig
How to enable the metrics-server addon
How to deploy a real Go app to your local cluster

Prerequisites

Mac or Linux, this post does not cover Windows
Docker installed and running, minikube uses it as the default driver

We'll install kubectl as part of this post, so you don't need it yet.

What minikube actually is

Before installing anything, it's worth understanding what minikube does, and what it doesn't do.

minikube runs a single-node Kubernetes cluster locally, inside a Docker container (or VM) on your machine. That cluster runs the exact same Kubernetes API as the production clusters you'd find on EKS, GKE, or AKS. The same kubectl commands work. The same YAML manifests work. The same Go code works, which is exactly why it's the right tool for this series.

What minikube is not: a production cluster. It's a single node with no high availability, no cloud integrations, and no autoscaling. It's a development and learning environment, not something you'd run workloads on.

How it compares to alternatives

There are several tools for running Kubernetes locally. Here's how they compare so you understand why we're using minikube:

Tool	Best for	Why we chose minikube
minikube	Learning and development	Addons system, excellent docs, most widely used
kind	CI pipelines	Lighter than minikube, less convenient for local dev
k3s	Lightweight production	More complex to set up locally
Docker Desktop K8s	Quick start	Heavier, less control over the cluster

The addons system is the main reason we use minikube here. Enabling metrics-server is a single command with minikube. Other tools require manual installation and configuration.

Installing minikube

Mac

# Install minikube via Homebrew
brew install minikube

# Verify the install
minikube version
# minikube version: v1.38.1

Linux

# Download the minikube binary
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

# Make it executable and move it to your PATH
chmod +x minikube-linux-amd64
sudo mv minikube-linux-amd64 /usr/local/bin/minikube

# Verify the install
minikube version
# minikube version: v1.32.0

Installing kubectl

kubectl is the command-line tool for interacting with Kubernetes clusters. minikube configures it automatically, but you need to install it separately.

Mac

brew install kubectl

# Verify
kubectl version --client
# Client Version: v1.36.2

Linux

# Download the latest stable kubectl binary
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Make it executable and move to PATH
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl

# Verify
kubectl version --client
# Client Version: v1.29.0

Starting your first cluster

With minikube and kubectl installed, starting a cluster is one command:

minikube start

You'll see output like this as minikube works through the startup sequence:

😄  minikube v1.38.1 on Darwin 26.5.1 (arm64)
✨  Automatically selected the docker driver
❗  Starting v1.39.0, minikube will default to "containerd" container runtime. See #21973 for more info.
📌  Using Docker Desktop driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.50 ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.35.1 on Docker 29.2.1 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

What happened in those few seconds:

minikube pulled a Docker image containing a full Kubernetes cluster.
It started a Docker container running that image.
It generated TLS certificates for the API server.
It configured kubectl to talk to this new cluster by writing a new context to ~/.kube/config

Configuring resources

The default CPUs and memory are sometimes not enough, especially once you have several pods running. We recommend starting with a bit more:

# Delete any existing cluster first, then start with more resources
minikube delete
minikube start --cpus=4 --memory=4096

Pinning the Kubernetes version

If you want to match your production cluster's Kubernetes version exactly, (I'm not going to run this; I want the last version 1.35.1):

minikube start --kubernetes-version=v1.28.3

Verifying the cluster is running

# Check the node is Ready
kubectl get nodes

# NAME       STATUS   ROLES           AGE     VERSION
# minikube   Ready    control-plane   4m59s   v1.35.1

# Check the system pods are all running
kubectl get pods -A

# NAMESPACE     NAME                               READY   STATUS    RESTARTS        AGE
# kube-system   coredns-7d764666f9-pzqlt           1/1     Running   0               5m33s
# kube-system   etcd-minikube                      1/1     Running   0               5m39s
# kube-system   kube-apiserver-minikube            1/1     Running   0               5m39s
# kube-system   kube-controller-manager-minikube   1/1     Running   0               5m39s
# kube-system   kube-proxy-2l7h5                   1/1     Running   0               5m33s
# kube-system   kube-scheduler-minikube            1/1     Running   0               5m39s
# kube-system   storage-provisioner                1/1     Running   1 (5m32s ago)   5m38s

If all pods show Running, your cluster is healthy.

You can also see the docker container running on behalf our minikube cluster

docker ps -a                                                                                                                                                     
# CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS          PORTS                                                                                                                                  NAMES
# de39b5440365   gcr.io/k8s-minikube/kicbase:v0.0.50   "/usr/local/bin/entr…"   10 minutes ago   Up 10 minutes   127.0.0.1:60921->22/tcp, 127.0.0.1:60923->2376/tcp, 127.0.0.1:60922->5000/tcp, 127.0.0.1:60925->8443/tcp, 127.0.0.1:60924->32443/tcp   minikube

Managing your cluster

You'll use these commands constantly throughout the series:

# Pause the cluster — preserves all state, frees up resources
minikube stop

# Resume where you left off
minikube start

# Delete everything and start completely fresh
# Use this when things get into a bad state
minikube delete

# Check what's currently running
minikube status

# Open the Kubernetes dashboard in your browser
# Useful for getting a visual overview of your cluster
minikube dashboard

# SSH into the minikube node
# Useful for low-level debugging
minikube ssh

A practical workflow: minikube stop at the end of every session, minikube start when you come back. minikube delete && minikube start When something is broken, and you want a clean slate.

What is metrics-server?

Before we enable it, it's worth understanding what metrics-server actually is and where it fits in the Kubernetes ecosystem.

metrics-server is a lightweight, in-memory component that runs inside your Kubernetes cluster and collects resource usage data, including CPU and memory, from every node and pod. It does this by talking to the kubelet (the agent that runs on each node) and scraping the resource stats it exposes.

┌─────────────────────────────────────────┐
│           Kubernetes cluster            │
│                                         │
│  ┌──────────┐     scrapes    ┌────────┐ │
│  │  kubelet │ ◄────────────  │metrics │ │
│  │ (node 1) │               │ server │ │
│  └──────────┘               └───┬────┘ │
│                                 │      │
│  ┌──────────┐                   │ exposes
│  │  kubelet │ ◄─────────────────┘      │
│  │ (node 2) │      Metrics API         │
│  └──────────┘           ▲             │
│                          │             │
└──────────────────────────┼─────────────┘
                           │
              kubectl top / your Go code

The data metrics-server collects is exposed through the Metrics API, a standard Kubernetes API endpoint that both kubectl top and, client-go can query. When you run kubectl top pods, kubectl is calling the Metrics API.

What metrics-server is not

It's worth being clear about what metrics-server doesn't do, because it's a common source of confusion:

It does not store historical data. It only keeps the most recent sample in memory. If you want trends over time, you need Prometheus.
It does not replace Prometheus or Grafana. It's a lightweight source of current usage data, not a full monitoring stack.
It does not run by default on most clusters. On EKS, GKE, and AKS it's usually pre-installed, but on self-managed clusters, and on minikube, you have to enable it yourself.

Think of it as the simplest possible answer to the question: "what is my cluster using right now?" For anything more sophisticated, alerting, historical analysis, dashboards, you'd reach for Prometheus.

Enabling metrics-server

minikube has an add-ons system, a curated list of optional components you can enable with a single command. metrics-server is one of them:

# Enable metrics-server
minikube addons enable metrics-server

# Verify it's running
kubectl get deployment metrics-server -n kube-system

# NAME             READY   UP-TO-DATE   AVAILABLE
# metrics-server   1/1     1            1

Wait about 60 seconds, then verify it's collecting data:

# Check node resource usage
kubectl top nodes

# NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# minikube   185m         4%     1042Mi          26%

# Check pod resource usage across all namespaces
kubectl top pods -A

If kubectl top returns error: metrics not available yet, just wait another 30 seconds and try again. metrics-server takes a moment to scrape its first round of data.

Deploying a real Go app to your local cluster

An empty cluster is fine for testing connections, but what we actually want is a cluster with real pods running, pods we can query with the health reporter.

Let's deploy a simple Go HTTP server properly, built with a multi-stage Dockerfile, loaded into minikube, and deployed with real resource requests and limits.

Why we use a Dockerfile

You might wonder why we don't just use go run inside the container for simplicity. The answer ties directly back to post 1: resource management.

go run compiles your code at container start, and the Go compiler needs over 1GB of memory just to compile a hello world app. That forces you to set absurdly high memory limits for a tiny service, which defeats the entire purpose of a post about resource management. It also causes OOMKill on any reasonable memory limit, and makes startup slow.

The right approach, and the one every real Go service uses, is to compile the binary at image build time. The running container only needs the binary, not the compiler. Memory usage drops from 1GB to under 20MB:

go run approach:     limits.memory: 1024Mi  ← embarrassing for a hello world
Dockerfile approach: limits.memory: 128Mi    ← honest and production-realistic

The project structure

Create a new directory for this example:

mkdir go-api && cd go-api

Create three files:

go-api/
├── main.go        ← the Go HTTP server
├── Dockerfile     ← multi-stage build
└── deployment.yaml ← Kubernetes manifests

The application

// main.go
package main

import (
    "fmt"
    "log/slog"
    "net/http"
    "os"
)

func main() {
    // Handle the root endpoint
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "hello from Go on Kubernetes!")
    })

    // Health endpoint — used by readiness and liveness probes.
    // Always returns 200 OK as long as the server is running.
    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })

    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }

    slog.Info("server starting", "port", port)
    if err := http.ListenAndServe(":"+port, nil); err != nil {
        slog.Error("server failed", "error", err)
        os.Exit(1)
    }
}

The Dockerfile

This is a multi-stage build, one stage compiles the binary, a second minimal stage runs it. The final image contains only the binary and nothing else, no Go toolchain, no compiler, no source code:

# Stage 1: builder
# Uses the full Go image to compile the binary.
# This stage is discarded after the build — it never ships.
FROM golang:1.26-alpine AS builder

WORKDIR /app

# Copy the source code
COPY main.go .

# Initialize a Go module — needed for go build
RUN go mod init github.com/yourname/go-api

# Build a statically linked binary.
# CGO_ENABLED=0 produces a fully static binary with no external dependencies.
# -ldflags="-s -w" strips debug symbols to reduce binary size.
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o server .

# Stage 2: runtime
# Uses a minimal Alpine image — no Go toolchain, no compiler.
# The final image is typically under 10MB.
FROM alpine:3.24

# Run as a non-root user — a basic security best practice.
RUN addgroup -S app && adduser -S app -G app
USER app

WORKDIR /app

# Copy only the compiled binary from the builder stage.
COPY --from=builder /app/server .

EXPOSE 8080

CMD ["./server"]

Build the image and load it into minikube

minikube runs its own Docker daemon, separate from your laptop's Docker. To use a locally built image in minikube, you need to build it inside minikube's Docker environment:

# Point your terminal's Docker CLI at minikube's Docker daemon
eval $(minikube docker-env)

# Build the image — this builds directly inside minikube
docker build -t go-api:latest .

# Verify the image is available inside minikube
docker images | grep go-api
# go-api   latest   a1b2c3d4e5f6   10 seconds ago   14.3MB

Notice the image size, 14.3MB. Compare that to the golang:1.26-alpine base image which is ~300MB. Multi-stage builds make a real difference.

When you're done working and want to restore your terminal's Docker CLI to your laptop's Docker:

eval $(minikube docker-env -u)

The deployment manifest

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-api
  namespace: default
  labels:
    app: go-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: go-api
  template:
    metadata:
      labels:
        app: go-api
    spec:
      containers:
      - name: server
        image: go-api:latest

        # imagePullPolicy: Never tells Kubernetes to use the local image
        # and never try to pull it from a registry.
        # Required when using images built inside minikube's Docker daemon.
        imagePullPolicy: Never

        ports:
        - containerPort: 8080

        # Resource requests and limits — from post 1.
        # These are now realistic for a pre-compiled Go binary.
        # A running Go binary uses ~10-20Mi — we give it 64Mi to be safe.
        resources:
          requests:
            cpu: "50m"
            memory: "32Mi"
          limits:
            cpu: "100m"
            memory: "64Mi"

        # Readiness probe — Kubernetes only sends traffic once this passes.
        # initialDelaySeconds is short because a pre-compiled binary
        # starts in milliseconds — no compilation step.
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

        # Liveness probe — Kubernetes restarts the container if this fails.
        # Always set higher than readinessProbe.initialDelaySeconds.
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: go-api
  namespace: default
spec:
  # ClusterIP is the default service type.
  # It gives the service a stable internal IP address only reachable
  # from inside the cluster. This is correct for internal services —
  # we access it locally using kubectl port-forward, not via a public IP.
  type: ClusterIP
  selector:
    app: go-api
  ports:
  - port: 80
    targetPort: 8080

Deploy it

# Apply both the Deployment and Service
kubectl apply -f deployment.yaml

# Watch the pods come up — should be fast since the binary starts instantly
kubectl get pods -w

# NAME                      READY   STATUS    RESTARTS   AGE
# go-api-7d6b9f8c4-xk2pq   1/1     Running   0          8s
# go-api-7d6b9f8c4-mn9rt   1/1     Running   0          8s
# go-api-7d6b9f8c4-p8wvz   1/1     Running   0          9s

No ContainerCreating delays. No OOMKill. The binary starts in milliseconds.

Verify it's working

# Check resource usage — these are the honest numbers for a Go binary
kubectl top pods

# NAME                      CPU(cores)   MEMORY(bytes)
# go-api-7d6b9f8c4-xk2pq   2m           11Mi
# go-api-7d6b9f8c4-mn9rt   2m           10Mi
# go-api-7d6b9f8c4-p8wvz   2m           11Mi

To hit the service from your terminal, use kubectl port-forward, it works the same way on both Mac and Linux regardless of which minikube driver you're using:

# Forward local port 8080 to the go-api service
# Keep this running in one terminal
kubectl port-forward service/go-api 8080:80

# Forwarding from 127.0.0.1:8080 -> 8080
# Forwarding from [::1]:8080 -> 8080

In a second terminal:

curl http://localhost:8080
# hello from Go on Kubernetes!

You should see hello from Go on Kubernetes!, confirming the service is running and routing traffic to your pods correctly. You now have three running pods with real, honest resource usage data available from metrics-server.

How Go connects to this cluster

This section previews what happens in the next post, so that when you see the first line of Go code, it already makes sense.

When minikube start ran, it wrote a new entry to your ~/.kube/config file. Here's the relevant section it added:

# ~/.kube/config (relevant section)
contexts:
- context:
    cluster: minikube
    namespace: default
    user: minikube
  name: minikube       # ← the context name

current-context: minikube  # ← currently active context

This is the same file kubectl reads when you run any kubectl command. It's also the file our Go code will read using BuildConfigFromFlags:

// This is how our Go program will connect to minikube
// in the next post — reads the same ~/.kube/config file
config, err := clientcmd.BuildConfigFromFlags("", os.Getenv("HOME")+"/.kube/config")

No extra setup. No credentials to manage. The Go program connects to exactly the same cluster kubectl connects to, because both read the same kubeconfig.

Verifying the context before running Go code

Before running any Go code that talks to Kubernetes, always verify which cluster you're pointing at:

# Which context is currently active?
kubectl config current-context
# minikube

# List all available contexts (useful if you have multiple clusters)
kubectl config get-contexts

# CURRENT   NAME       CLUSTER    AUTHINFO   NAMESPACE
# *         minikube   minikube   minikube   default
#           prod-eks   eks-prod   aws-user   production

The * marks the active context. If you ever accidentally point at a production cluster while running your Go code, the get-contexts output makes it obvious.

To switch between contexts:

# Switch to a different context
kubectl config use-context prod-eks

# Switch back to minikube
kubectl config use-context minikube

Common gotchas

minikube service tunnel on Mac with Docker driver

On Mac, minikube runs inside a Docker container with its own internal network. Running minikube service go-api --url creates a temporary tunnel and warns:

Services [default/go-api] have type "ClusterIP" not meant to be exposed,
however for local development minikube allows you to access this!
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

This isn't an error, but the tunnel only works while that terminal stays open. Use kubectl port-forward instead, it's more portable, works the same on Mac and Linux, and doesn't need a dedicated terminal window.

Docker not running

minikube uses Docker as its default driver. If Docker isn't running when you call minikube start, you'll see:

X Exiting due to PROVIDER_DOCKER_NOT_RUNNING: Docker Desktop is not running

Start Docker first, then retry minikube start.

Not enough resources: pods stuck in Pending

If your pods stay in Pending state, the most common cause is insufficient resources. Check with:

kubectl describe pod <pod-name> | grep -A5 Events
# Warning  FailedScheduling  Insufficient memory

Fix it by giving minikube more resources:

minikube delete
minikube start --cpus=4 --memory=8192

minikube IP changes on restart

Every time you minikube delete && minikube start, minikube may get a different IP address for the API server. If you ever hardcode the API server URL anywhere in your Go code, it will break after a restart. Always use BuildConfigFromFlags with the kubeconfig path, it reads the current IP dynamically from the config file.

Summary

You now have everything you need to follow the rest of this series:

A local Kubernetes cluster running minikube
metrics-server enabled and collecting data
A Go app deployed with 3 replicas and proper resource configuration
kubectl configured and pointing at your local cluster

Three things to remember as you work through the series:

minikube stop at the end of every session, don't leave it consuming resources in the background
minikube delete when things get into a broken state, a clean cluster takes less than 2 minutes to rebuild
kubectl config current-context before running any Go code, always know which cluster you're talking to

What's next

In the next post we write our first Go program that connects to this cluster using client-go, the official Kubernetes Go client. We'll query the pods we just deployed, detect failing ones, and build something immediately useful: a cluster health reporter that gives you a full picture of your cluster's state in a single command.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Kubernetes resource requests and limits explained: scheduling, throttling, and OOMKill

Fer Rios — Thu, 02 Jul 2026 01:11:20 +0000

This is part of the Platform engineering with Go series: a growing collection of posts on Kubernetes, Go tooling, and infrastructure automation. View all posts in the series

The 3am incident nobody talks about

It's 3am. Your on-call phone goes off. A service is down in production. You log in, check the pods, and see this:

kubectl get pods -n production

NAME                    READY   STATUS             RESTARTS   AGE
api-7d6b9f8c4-xk2pq    0/1     OOMKilled          14         2d
api-7d6b9f8c4-mn9rt    0/1     OOMKilled          14         2d
api-7d6b9f8c4-p8wvz    0/1     OOMKilled          14         2d

You restart the pods. They come back. Five minutes later, they die again.

The problem could be several things: limits set too low for the actual workload, no limits set at all so the pod consumed memory freely until the node ran out, a traffic spike the pod wasn't provisioned for, or yes, a memory leak in the application itself. In all cases, the result is the same: memory consumption exceeded the allowed ceiling, the OOM killer fired, the pod died, and Kubernetes restarted it into the same situation. Fourteen times.

This is one of the most common production incidents in Kubernetes, and one of the most preventable. But preventing it requires understanding what requests and limits actually do, what happens when memory consumption exceeds them, and how to configure them correctly for your actual workload.

That's what this post is about.

No Go knowledge required: this post has zero code

Requests vs limits, two completely different things

Before anything else, the most important concept to internalize: requests and limits are not two versions of the same thing. They serve entirely different purposes and are enforced by entirely different components of Kubernetes.

# A pod spec with both requests and limits set
resources:
  requests:
    cpu: "250m"     # 250 millicores = 0.25 CPU cores
    memory: "256Mi" # 256 mebibytes
  limits:
    cpu: "500m"     # 500 millicores = 0.5 CPU cores
    memory: "512Mi" # 512 mebibytes

Here's what each one actually does:

Requests are a promise to the scheduler. When you set requests.memory: 256Mi, you're telling Kubernetes: "I need at least 256Mi of memory reserved for this pod on whatever node it runs on.. The scheduler uses this value to decide which node to place the pod on. Once scheduled, that 256Mi is considered reserved on that node, even if the pod is only using 50Mi at the moment.

Limits are a ceiling enforced by the runtime. When you set limits.memory: 512Mi, you're telling Kubernetes: "This pod is never allowed to use more than 512Mi of memory." If it tries to exceed that ceiling, the kernel kills it. No warning. No graceful shutdown. Just gone.

The key insight: requests affect scheduling; limits affect runtime behavior. They are read by different components at different times for entirely different reasons.

How the scheduler uses requests to place pods

The Kubernetes scheduler's job is to find a node for each new pod. It does this by looking at each node's allocatable resources and comparing them against the sum of all pod requests already scheduled there.

A node's allocatable resources are not the same as its total capacity. Some resources are always reserved for the operating system and Kubernetes system components:

kubectl describe node my-node

# Look for the Allocatable section:
Allocatable:
  cpu:     3800m   # 3.8 cores available to pods (out of 4 total)
  memory:  7Gi     # 7Gi available to pods (out of 8Gi total)

The scheduler adds up the requests of all pods already on a node and compares that sum against the allocatable resources. If the remaining capacity is less than the new pod's request, the node is skipped.

Here's the critical subtlety: the scheduler cares about requests, not actual usage.

Node: 4 CPU allocatable

Pod A: requests 1 CPU → actual usage: 0.2 CPU
Pod B: requests 1 CPU → actual usage: 0.1 CPU
Pod C: requests 1 CPU → actual usage: 0.8 CPU

From the scheduler's perspective: 3 out of 4 CPU are "used"
From the kernel's perspective: only 1.1 CPU are actually being consumed

New pod requesting 1.5 CPU → scheduler says NO (only 1 CPU remaining)

This creates an important tension: if your requests are set too high relative to actual usage, your nodes look full when they're actually mostly idle, and new pods can't be scheduled. This is called poor bin packing, and it wastes money.

On the other hand, if your requests are too low, too many pods get scheduled onto the same node. When they all start consuming resources simultaneously, the node becomes overloaded, and Kubernetes starts evicting pods to relieve the pressure.

Getting requests right is a balancing act between cost efficiency and stability.

CPU limits and throttling: the silent killer

CPU is what Kubernetes calls a compressible resource. If a pod tries to use more CPU than its limit allows, the Linux kernel doesn't kill it, it throttles it. The pod keeps running, but it gets fewer CPU cycles, so everything it does takes longer.

The throttling mechanism is the Linux CFS (Completely Fair Scheduler).

Understanding CPU cycles and scheduling periods

Before we talk about how throttling works, it helps to understand two concepts that are invisible in day-to-day operations but fundamental to what's happening under the hood.

What is a CPU cycle?

Your server's CPU is constantly doing work, executing instructions, processing data, running code. A CPU cycle is the smallest unit of that work. A modern CPU completes billions of cycles per second (gigahertz, that's what the "3.2GHz" on a server spec means: 3.2 billion cycles per second).

Think of CPU cycles like minutes of attention from a very fast worker. Your pod's processes need a certain number of those "minutes" to do their job, handle a request, run a query, process a message. The more cycles your pod gets, the faster it runs. The fewer it gets, the slower it runs.

When Kubernetes talks about CPU in millicores (250m, 500m, 1000m), it's describing what fraction of one CPU core's cycles your pod gets access to:

1000m = 1 full CPU core = 100% of one core's cycles
 500m = 0.5 CPU core   = 50% of one core's cycles
 250m = 0.25 CPU core  = 25% of one core's cycles

What is a scheduling period?

A CPU doesn't serve one process at a time from start to finish. It slices time into tiny windows and gives each process a turn. This is called time-sharing, and the windows are called scheduling periods.

Think of it like a teacher in a classroom. Instead of helping one student for the entire class, the teacher spends 5 minutes with each student in rotation. Every student gets attention, but no single student monopolizes the teacher's time.

The Linux CFS uses scheduling periods of 100 milliseconds by default. In every 100ms window, the CPU divides its time among all the processes competing for it.

100ms scheduling period
│
├── 0ms  - 25ms  → Pod A gets its turn  (250m limit = 25% of 100ms = 25ms)
├── 25ms - 75ms  → Pod B gets its turn  (500m limit = 50% of 100ms = 50ms)
├── 75ms - 100ms → Pod C gets its turn  (250m limit = 25% of 100ms = 25ms)
│
└── (next 100ms period starts)

Each pod's CPU limit determines how many milliseconds of that 100ms window it's allowed to use:

CPU limit of 250m → 25ms of CPU time per 100ms period
CPU limit of 500m → 50ms of CPU time per 100ms period
CPU limit of 1000m (1 full core) → 100ms of CPU time per 100ms period

What happens when a pod hits its limit mid-period?

Here's where throttling kicks in. If a pod uses up its entire allocation before the 100ms period ends, the CFS puts it in a throttled state for the rest of that period; it gets zero CPU cycles until the next period starts, regardless of whether other pods are idle.

Pod with 250m CPU limit → 25ms of allowed CPU time per 100ms period

Period 1 (0ms to 100ms):
  ├── 0ms:  Pod starts processing a request
  ├── 25ms: Pod has used its full 25ms allocation ← throttled here
  ├── 25ms to 100ms: Pod sits idle, gets zero CPU cycles
  └── 100ms: New period starts, pod gets another 25ms

What the user experiences:
  ├── Request arrives at 0ms
  ├── Pod processes half the request, then waits 75ms doing nothing
  └── Response arrives much later than it should

The pod didn't crash. It didn't log an error. It just stopped making progress for 75ms out of every 100ms, which is why a throttled service feels sluggish rather than broken. Everything works, just much slower than it should.

A concrete analogy

Imagine you're writing a report and your manager says you can only use the shared laptop for 15 minutes every hour. You start writing, but at the 15-minute mark your access is cut off, even if you're mid-sentence. You sit and wait for the next hour to start before you can type another word.

That's exactly what the CFS does to a throttled pod. It doesn't care that you were in the middle of something important. When the quota is up, the process waits, and whatever request it was handling has to wait too.

Why this is hard to detect

The reason CPU throttling causes so much confusion is that it's invisible in all the usual places:

# This shows current usage — looks fine
kubectl top pods -n production
NAME                 CPU(cores)   MEMORY(bytes)
api-6d8f9b7c-xk2pq   240m         180Mi

# But the pod might be throttled 80% of the time
# kubectl top shows average usage, not whether that usage caused throttling

A pod using 240m CPU on average can still be heavily throttled if it regularly bursts above its limit within a single 100ms period. The average looks healthy; the latency tells a different story.

So, this is why this is dangerous: CPU throttling is completely silent.

There's no error. No log line. No Kubernetes event. Your pod just slows down. Requests take longer. Latency increases. Users notice something is wrong, but nothing in your logs explains why.

The only reliable way to detect throttling is through the container_cpu_cfs_throttled_periods_total metric in Prometheus, which counts how many scheduling periods a container was throttled in. If that number is climbing, throttling is happening regardless of what kubectl top shows.

To detect throttling, you can use the Kubernetes metrics server:

# See current CPU usage vs requests
kubectl top pods -n production

NAME                    CPU(cores)   MEMORY(bytes)
api-7d6b9f8c4-xk2pq    480m         210Mi
api-7d6b9f8c4-mn9rt    498m         198Mi

If you see pods consistently near their CPU limit, throttling is likely happening, especially if you're seeing latency spikes that don't correlate with error rates.

The CPU limits controversy

There's an ongoing debate in the Kubernetes community about CPU limits. Some platform teams remove CPU limits entirely for latency-sensitive services, allowing pods to burst freely as long as there's spare capacity on the node.

The argument for removing CPU limits:

Eliminates throttling completely.
Workloads use spare node capacity efficiently.
Latency becomes predictable because pods are never artificially slowed.

The argument for keeping CPU limits:

A noisy neighbor pod can consume all spare CPU and starve other pods.
Without limits, a bug in one service can degrade the entire node.

The right answer depends on your workload. For latency-sensitive APIs, consider removing CPU limits and relying on requests alone. For batch workloads, CPU limits are fine. For anything in between, measure throttling first before deciding.

Memory limits and OOMKill: the dangerous one

Memory is what Kubernetes calls an incompressible resource. Unlike CPU, the kernel cannot throttle memory access, it can't say "you only get 80% of the memory reads you asked for." Memory is binary: either the process has it, or it doesn't.

When a pod's memory usage exceeds its limit, the Linux OOM (Out of Memory) killer terminates one of its processes immediately. No warning. No graceful shutdown. The process is gone. Kubernetes then sees that a container has exited unexpectedly and restarts it, which is where CrashLoopBackOff comes from.

# Detect OOMKill in pod description
kubectl describe pod api-7d6b9f8c4-xk2pq -n production

# Look for this in the output:
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 21 Jun 2026 02:14:32 +0000
  Finished:     Mon, 21 Jun 2026 02:14:33 +0000

Exit code 137 means the process was killed by signal 9 (SIGKILL) from the OOM killer. If you see this, your memory limit is too low for your actual workload.

OOMKill at the node level

OOMKill can also happen at the node level, independently of your pod limits. If total memory consumption across all pods on a node approaches the node's total capacity, the Linux kernel's node-level OOM killer activates.

In this case, Kubernetes doesn't wait for the OOM killer. It has its own eviction mechanism, if available memory on a node drops below a configured threshold, Kubernetes starts evicting pods proactively. Which pods get evicted first is determined by QoS class, which we'll cover next.

QoS classes: who dies first under pressure

Kubernetes automatically assigns every pod a Quality of Service (QoS) class based on how its requests and limits are configured. You don't set this manually; it's derived. Under node pressure, Kubernetes evicts pods in order from lowest to highest QoS class.

There are three classes:

BestEffort: lowest priority

A pod is BestEffort when it has no requests or limits set at all:

# BestEffort, no resources section at all
resources: {}

BestEffort pods are the first to be evicted under node pressure. They get whatever resources happen to be available, and nothing is guaranteed. Never run production workloads as BestEffort.

Burstable: middle priority

A pod is Burstable when it has requests set, but limits are either not set or higher than requests:

# Burstable, requests set, limits higher than requests
resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Most production workloads should be Burstable. The pod has guaranteed minimum resources (the requests) but can burst above them when capacity is available. Under eviction pressure, Burstable pods are evicted after BestEffort but before Guaranteed.

Guaranteed: highest priority

A pod is Guaranteed when its requests equal its limits for all containers:

# Guaranteed, requests == limits
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "500m"     # same as request
    memory: "512Mi" # same as request

Guaranteed pods are the last to be evicted. Kubernetes will sacrifice BestEffort and Burstable pods before touching a Guaranteed pod. Use this for your most critical services, databases, core APIs, and anything where an eviction is catastrophic.

To check a pod's QoS class:

kubectl get pod api-7d6b9f8c4-xk2pq -n production \
  -o jsonpath='{.status.qosClass}'

# Output: Burstable

A practical framework for setting requests and limits

Knowing the theory is one thing. Knowing what numbers to put in your YAML is another. Here's a step-by-step framework that works in production:

Step 1: observe before you configure

Deploy your workload without limits first (or with very high limits that won't be hit). Let it run under realistic traffic for several days, including peak hours. Collect metrics with kubectl top pods or Prometheus.

# Watch resource usage over time
watch kubectl top pods -n production -l app=api

Step 2: set requests at p50 (median) usage

Your request should reflect typical usage, what the pod uses most often. The p50 (50th percentile) of observed CPU and memory usage is a good starting point.

If your service typically uses 150m CPU and 200Mi memory:

requests:
  cpu: "150m"
  memory: "200Mi"

Step 3: set memory limits at p99 plus a buffer

Your memory limit should handle traffic spikes without triggering OOMKill. The p99 (99th percentile) plus a 20-30% buffer is a safe starting point:

limits:
  memory: "350Mi"  # p99 was ~280Mi, plus 25% buffer

Step 4: handle CPU limits carefully

Start at 2-4x your CPU request. Monitor for throttling. If you see consistent throttling in container_cpu_throttled_seconds_total, either raise the limit or remove it for that service.

limits:
  cpu: "500m"  # 3x the 150m request, gives room to burst

Step 5: use LimitRange to enforce defaults at the namespace level

As a platform engineer, you don't want to rely on every developer remembering to set resources. Use a LimitRange to provide defaults for pods that don't specify them:

# Enforce default requests and limits for all pods in a namespace
apiVersion: v1
kind: LimitRange
metadata:
  name: default-resource-limits
  namespace: production
spec:
  limits:
  - type: Container
    default:           # applied when no limits are specified
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:    # applied when no requests are specified
      cpu: "100m"
      memory: "128Mi"
    max:               # no pod can exceed these
      cpu: "2"
      memory: "2Gi"
    min:               # no pod can go below these
      cpu: "50m"
      memory: "64Mi"

Step 6: use ResourceQuota to cap total namespace consumption

ResourceQuota limits the total resources that can be consumed across all pods in a namespace, useful for multi-tenant clusters and chargeback:

# Cap total resource consumption for the production namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "10"      # total CPU requests across all pods
    requests.memory: "20Gi" # total memory requests across all pods
    limits.cpu: "20"        # total CPU limits across all pods
    limits.memory: "40Gi"   # total memory limits across all pods
    pods: "50"              # maximum number of pods

Common mistakes

Running without any resource configuration (BestEffort)

This is the most dangerous mistake. With no requests, the scheduler has no visibility into what your pod actually needs. Under node pressure, your pods are the first to be evicted, regardless of how critical your service is.

Setting limits without requests

When you set a limit without a request, Kubernetes automatically sets the request equal to the limit, which makes the pod Guaranteed. This isn't always wrong, but it means you're reserving 100% of your limit on the scheduler even if typical usage is much lower. Over time this leads to poor bin packing and wasted capacity.

Copying the same resource values for every service

A stateless Go API, a JVM-based service, and a batch data processor have completely different memory profiles. A Go binary might be happy with 128Mi. A JVM service might need 1Gi just for the heap. Tune per workload, not per deployment template.

Forgetting init containers

Init containers run before your main container and have their own resource requirements. If you set tight limits on init containers, pod initialization can fail, and Kubernetes will keep retrying. Always check the init container resource usage too:

initContainers:
- name: db-migrate
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "200m"
      memory: "256Mi"

Setting memory requests too low relative to actual usage

If your pod's actual memory usage regularly exceeds its request (even if it stays below the limit), it becomes a candidate for eviction under node pressure. The request is not just a scheduling hint, it also affects eviction priority within the Burstable class. Pods whose usage exceeds their request are evicted before pods whose usage stays within their request.

Summary

Requests and limits are the foundation of stable Kubernetes workloads. Get them wrong, and you'll spend nights chasing OOM kills and throttling-induced latency spikes. Get them right, and your cluster runs efficiently with predictable, stable workloads.

Three things to take away:

Requests tell the scheduler where to place your pod. Always set them, never run BestEffort in production.
Memory limits tell the kernel when to kill your pod. Set them generously enough to handle traffic spikes, and monitor for OOMKill with kubectl describe pod.
CPU limits are more nuanced. Start with 2-4x your CPU request and remove them for latency-sensitive services if throttling is a problem.

Understanding this is the prerequisite for everything else in platform engineering, autoscaling, capacity planning, chargeback, and multi-tenancy, all of which depend on the resource configuration being correct.

P.S.: Writing this post from Asunción, Paraguay 🇵🇾 where it's hard to concentrate right now. Paraguay just knocked Germany out of the 2026 World Cup on penalties. For the first time in history, Germany lost a penalty shootout. Goalkeeper Orlando Gil was a hero. We face France next. Forgive me if there are any bugs or mistakes in this post. I may have been screaming at a television.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Go HTTP middleware explained: what it is, how it works, and how to build your own

Fer Rios — Mon, 22 Jun 2026 14:31:52 +0000

This is part 2 of 2 in the series Go HTTP middleware from scratch. If you're new to Go types, → read part 1 first.

The same code keeps showing up everywhere

You're building an HTTP server. Every request needs a unique ID for tracing. Every request needs to check authentication. Every request needs its duration logged. The naive approach: copy that code into every handler.

// Without middleware — same boilerplate in every single handler
func getUserHandler(w http.ResponseWriter, r *http.Request) {
    // Auth check — copy/pasted from every other handler
    token := r.Header.Get("Authorization")
    if token == "" {
        http.Error(w, "unauthorized", http.StatusUnauthorized)
        return
    }

    // Request ID — copy/pasted from every other handler
    requestID := uuid.New().String()
    log.Printf("requestID=%s handling getUser", requestID)

    // actual handler logic finally starts here...
}

Do this across 20 handlers, and you have a maintenance nightmare. Change the auth logic once, and you have to update 20 files.

Middleware is the Go way to solve this. Write the cross-cutting logic once, wrap it around your handlers, and never think about it again.

What is this post about

What middleware is and how Go's HTTP model makes it natural.
How the http.Handler interface works, the foundation everything builds on.
How to write your own middleware from scratch.
How to chain multiple middleware together.
How to pass data through the chain using context.
Tips for advanced usage: ordering, third-party routers, and common pitfalls.

The foundation: the http.Handler interface

As I mentioned in the first part of this serie, everything in Go's HTTP stack is built on one tiny interface:

// This is the entire http.Handler interface — just one method.
// Any type that has a ServeHTTP method satisfies it automatically.
type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

If you've read the previous posts in this series, you'll recognize this pattern. Go interfaces are just method signatures. Any Go type with ServeHTTP(ResponseWriter, *Request) is automatically an HTTP handler. No registration, no declaration. If you remember, http.HandlerFunc is a convenience type that lets you turn a plain function into an http.Handler:

// HandlerFunc is a function type that also implements http.Handler.
// This is how you write handlers without creating a struct.
type HandlerFunc func(ResponseWriter, *Request)

func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
    f(w, r)
}

This means these two ways of writing a handler are equivalent:

// Option A: using http.HandleFunc, to convert a plain function 
// to a HandlerFunc type
http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "hello")
})

// Option B: a struct implementing http.Handler
type HelloHandler struct{}

func (h HelloHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "hello")
}

Most people use option A for simple handlers. The struct approach becomes useful when your handler needs to hold state (like a database client).

What is middleware?

Middleware is a function that:

Accepts an http.Handler.
Returns a new http.Handler.
Does something before and/or after calling the original handler.

// This is the shape of every middleware in Go.
// It wraps a handler and returns a new one.
func MyMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Do something BEFORE the handler runs
        // (auth check, logging, adding context...)

        next.ServeHTTP(w, r) // call the actual handler

        // Do something AFTER the handler runs
        // (log the response, record timing...)
    })
}

The next parameter is the handler this middleware wraps. Calling next.ServeHTTP(w, r) passes control forward, like handing a baton in a relay race.

The relay race explained

Here's the mental model that makes middleware click:

Request comes in
      │
      ▼
┌─────────────────┐
│   Middleware 1    ← does its work (e.g. logs start time)
│       │         
│       ▼         
│  Middleware 2     ← does its work (e.g. checks auth)
│       │         
│       ▼         
│    Handler        ← does the actual business logic
│       │         
│       ▼         
│  Middleware 2     ← does its work after (e.g. checks result)
│       │         
│       ▼         
│  Middleware 1     ← does its work after (e.g. logs duration)
└─────────────────┘
      │
      ▼
Response goes out

Each middleware wraps the next layer. Code before next.ServeHTTP runs on the way in. Code after runs on the way out. This is sometimes called the "onion model."

Building three real middleware functions

Let's build three practical middleware functions and chain them together. You can find the repo here.

But before writing any code, here's how the project is laid out. You will get a clearer understanding of how a Golang project should be structured if you read this post first about Go Packages. Each middleware gets its own file inside a shared middleware package, and cmd/server/main.go wires everything together.

go-middleware-examples/
├── go.mod
├── cmd/
│   └── server/
│       └── main.go            ← entry point — registers routes and chains middleware
└── internal/
    └── middleware/
        ├── requestid.go       ← RequestID middleware + context key + getter
        ├── logger.go          ← Logger middleware + responseWriter wrapper
        ├── auth.go            ← Auth middleware
        └── chain.go           ← Chain helper for readable middleware composition

A few decisions worth noting:

All middleware lives under internal/middleware/, it's private to this module. If you later wanted to publish this middleware as a reusable library for other projects, it would move to pkg/middleware/ instead.
Each middleware gets its own file even though they all declare package middleware. This keeps each one focused and easy to find, requestid.go only ever deals with request IDs, auth.go only ever deals with authentication.
The context key (RequestIDKey) and its getter (GetRequestID) live in the same file as the middleware that sets it (requestid.go). Anything that reads the request ID imports the middleware package and calls middleware.GetRequestID(ctx), never reaching into the context directly. You can know more about Go contexts here.
cmd/server/main.go stays thin, it only registers handlers and chains middleware together. No middleware logic lives there.

From the root directory (go-middleware-examples), run the following commands to create your Go project, and install the uuid package:

go mod init github.com/<your-username>/go-middleware-examples
go get github.com/google/uuid

You can name your project as you want. I usually name my projects according to my GitHub app repo name.

Here's the go.mod for reference:

module github.com/FerRiosCosta/go-middleware-examples

go 1.26.3

require github.com/google/uuid v1.6.0 // indirect

Now let's build each file.

Request ID middleware

Please consider the first comment line, which specifies where the file should be created. This middleware attaches a unique ID to every request. Useful for tracing a single request through your logs.

// internal/middleware/requestid.go
package middleware

import (
    "context"
    "net/http"

    "github.com/google/uuid"
)

// key is an unexported type for context keys in this package.
// Using a custom type prevents collisions with keys from other packages.
// (Two packages both using the string "requestID" would overwrite each other.)
type key string

// RequestIDKey is the context key for the request ID.
// Exported so handlers in other packages can retrieve it.
const RequestIDKey key = "requestID"

// RequestID generates a unique ID for every incoming request,
// stores it in the context, and adds it to the response headers.
func RequestID(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Generate a new UUID for this request.
        id := uuid.New().String()

        // Store the ID in a new context derived from the request's context.
        // r.Context() returns the existing context, we build on top of it,
        // not replace it, so any previously set values are preserved.
        ctx := context.WithValue(r.Context(), RequestIDKey, id)

        // Add the ID to the response header.
        // This lets clients (and load balancers) correlate their logs with yours.
        w.Header().Set("X-Request-ID", id)

        // This is the line from the previous explanation:
        // create a copy of the request with the new context attached,
        // then pass it to the next handler in the chain.
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

// GetRequestID is a helper to retrieve the request ID from a context.
// Handlers call this instead of reaching into the context directly.
func GetRequestID(ctx context.Context) string {
    id, _ := ctx.Value(RequestIDKey).(string)
    return id
}

Logger middleware

Logs every request: method, path, duration, and status code. But, before looking at the full code, let's first try to understand how this logger middleware works under the hood.

What is http.ResponseWriter?

Before looking at the code, it helps to understand the type we're about to wrap.

http.ResponseWriter is an interface, every handler you write receives one as its first argument. It's how your handler sends a response back to the client: the status code, the headers, and the body.

// http.ResponseWriter is an interface with three methods:
type ResponseWriter interface {
    Header() http.Header         // get/set response headers
    Write([]byte) (int, error)   // write the response body
    WriteHeader(statusCode int)  // write the status code (200, 404, 500...)
}

A typical handler uses it like this:

func myHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)        // write the status code
    w.Write([]byte("hello!"))            // write the body
}

The problem: you can't read the status code back

Here's the catch. http.ResponseWriter lets you write a status code, but it has no method to read it back afterward. There's no w.StatusCode() you can call. Once you write it, it's gone, sent straight to the client over the network.

This is a problem for our Logger middleware. Remember the relay race model: Logger calls next.ServeHTTP(w, r), waits for the handler to finish, and only then logs the result. But by the time the handler finishes, it has already called w.WriteHeader(404) (or whatever status it chose), and Logger has no way to find out what that status was. The information is gone.

The solution: wrap it in our own struct

To solve this, we create our own type that:

Embeds the original http.ResponseWriter, so it still behaves exactly the same way for the handler
Adds a field to remember the status code
Intercepts the one method we care about (WriteHeader) to save the value before passing it through.

// responseWriter wraps http.ResponseWriter to capture the status code.
// http.ResponseWriter doesn't expose the status code after it's written,
// so we intercept WriteHeader to save it.
type responseWriter struct {
    http.ResponseWriter            // embed the original, we get all its methods for free
    statusCode         int         // the status code the handler wrote
}

The embedded http.ResponseWriter is the key part. In Go, embedding a type inside a struct means the outer struct automatically gets all of the embedded type's methods, for free, with no extra code. So our responseWriter already has Header(), Write(), and WriteHeader() just by embedding http.ResponseWriter. As far as the handler is concerned, our responseWriter looks and behaves exactly like a normal http.ResponseWriter, because it satisfies the same interface.

The only thing left to do is override WriteHeader, the one method we actually need to intercept:

// WriteHeader intercepts the status code before passing it through.
func (rw *responseWriter) WriteHeader(code int) {
    rw.statusCode = code               // save it for logging
    rw.ResponseWriter.WriteHeader(code) // pass it through to the real writer
}

This is the trick: when our responseWriter defines its own WriteHeader method, that one overrides the embedded version. So when a handler calls w.WriteHeader(404), it actually calls our version, which saves 404 into rw.statusCode, then calls the original ResponseWriter.WriteHeader(404) so the client still receives the real response. The handler never notices anything different happened.

Now Logger can read wrapped.statusCode after the handler finishes, something that was impossible with a plain http.ResponseWriter.

Here's the full middleware:

// internal/middleware/logger.go
package middleware

import (
    "log/slog"
    "net/http"
    "time"
)

// responseWriter wraps http.ResponseWriter to capture the status code.
// http.ResponseWriter doesn't expose the status code after it's written,
// so we intercept WriteHeader to save it.
type responseWriter struct {
    http.ResponseWriter            // embed the original — we get all its methods for free
    statusCode         int         // the status code the handler wrote
}

// WriteHeader intercepts the status code before passing it through.
func (rw *responseWriter) WriteHeader(code int) {
    rw.statusCode = code               // save it for logging
    rw.ResponseWriter.WriteHeader(code) // pass it through to the real writer
}

// Logger logs each request's method, path, status code, duration,
// and request ID (if set by the RequestID middleware).
func Logger(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Record when the request arrived.
        start := time.Now()

        // Wrap the ResponseWriter so we can capture the status code.
        wrapped := &responseWriter{
            ResponseWriter: w,
            statusCode:     http.StatusOK, // default to 200 if WriteHeader is never called
        }

        // Call the next handler — this is where the actual work happens.
        next.ServeHTTP(wrapped, r)

        // By the time we reach here, the handler has finished.
        // We can now log everything we know about this request.
        slog.Info("request",
            "method",     r.Method,
            "path",       r.URL.Path,
            "status",     wrapped.statusCode,
            "duration",   time.Since(start).String(),
            "requestID",  GetRequestID(r.Context()), // set by RequestID middleware
        )
    })
}

Auth middleware

Checks for a valid Bearer token before allowing the request through.

This one looks more complicated than RequestID and Logger at first glance, it's a function that returns a function that returns a function. Let's slow down and unpack why, before looking at the full code.

Why Auth needs an extra layer

RequestID and Logger both have this shape:

func RequestID(next http.Handler) http.Handler {
    // ...
}

One input (next), one output (http.Handler). That's the standard middleware shape we covered earlier.

But Auth needs something extra: a configurable token. Every other middleware works the same way no matter what, but Auth needs to know which token counts as valid, and that's different for every app (and usually comes from an environment variable, not a hardcoded value).

So Auth can't just be a middleware, it needs to be a function that produces a middleware, once you tell it which token to check:

// Auth is NOT middleware itself.
// It's a function that BUILDS a middleware, once you give it a token.
func Auth(validToken string) func(http.Handler) http.Handler {
    // ...
}

Breaking down the three layers

Here's the same function with each layer labeled:

//        ┌─── Layer 1: takes your config (the token)
//        │
func Auth(validToken string) func(http.Handler) http.Handler {
    //                        └─── Layer 2: this is what gets returned —
    //                             a real middleware function, with the
    //                             standard func(http.Handler) http.Handler shape

    return func(next http.Handler) http.Handler {
        //     └─── Layer 2 starts here. This is now a normal middleware,
        //          identical in shape to RequestID and Logger.

        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            //   └─── Layer 3: the actual handler logic that runs per-request.
            //        This is the same as the inner function in every middleware
            //        you've already seen.

            // ... auth check logic goes here ...
        })
    }
}

Three layers, three different jobs:

Layer	Runs when	Job
1 — `Auth(validToken)`	Once, at startup	Receives your config (the token)
2 — `func(next http.Handler) http.Handler`	Once, when you build your middleware chain	Receives the next handler to wrap
3 — `func(w, r)`	On every single request	Does the actual work — checks the header

Why this pattern, and how you use it

The payoff is that calling Auth looks almost identical to using RequestID or Logger, you just add one extra set of parentheses to pass in the token:

// RequestID and Logger take no config, pass them directly
middleware.RequestID
middleware.Logger

// Auth takes config, call it first to get back a real middleware
middleware.Auth("my-secret-token")   // ← this call returns a middleware function

That's it. Auth("my-secret-token") runs Layer 1 immediately and hands you back Layer 2, a normal func(http.Handler) http.Handler, ready to be used exactly like any other middleware in your Chain:

Chain(mux,
    middleware.RequestID,                // sets the request ID first
    middleware.Logger,                   // then logs — sees the ID
    middleware.Auth("my-secret-token"),  // called with () — returns a middleware
)

If this still feels unfamiliar, it helps to compare it to something simpler, a function that builds a greeting function:

// Same shape, much simpler example.
// MakeGreeter takes a name and returns a NEW function
// that already "remembers" that name.
func MakeGreeter(name string) func() string {
    return func() string {
        return "Hello, " + name + "!"
    }
}

sayHiToAlice := MakeGreeter("Alice") // returns a function
fmt.Println(sayHiToAlice())          // "Hello, Alice!"

Auth does the exact same thing, it just returns a middleware function instead of a greeting function. Once you see MakeGreeter, Auth(validToken) stops looking complicated; it's the same "function that builds and returns another function" pattern, just doing more useful work inside.

Now here's the full code with that context in mind:

// internal/middleware/auth.go
package middleware

import (
    "net/http"
    "strings"
)

// Auth checks for a valid Bearer token in the Authorization header.
// If the token is missing or invalid, it returns 401 and stops the chain —
// next.ServeHTTP is never called, so the handler never runs.
func Auth(validToken string) func(http.Handler) http.Handler {
    // Return a middleware function — this pattern lets us configure
    // middleware with parameters (the token) while keeping the standard shape.
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            // Read the Authorization header.
            authHeader := r.Header.Get("Authorization")

            // Check the format: "Bearer <token>"
            parts := strings.SplitN(authHeader, " ", 2)
            if len(parts) != 2 || parts[0] != "Bearer" {
                // Respond with 401 and stop. next.ServeHTTP is never called.
                http.Error(w, "missing or malformed token", http.StatusUnauthorized)
                return
            }

            // Validate the token.
            // In a real app you'd verify a JWT or look up the token in a database.
            if parts[1] != validToken {
                http.Error(w, "invalid token", http.StatusUnauthorized)
                return
            }

            // Token is valid — pass the request to the next handler.
            next.ServeHTTP(w, r)
        })
    }
}

Chaining middleware together

Now let's wire everything up. But first, create your own secret token and paste it into your main.go file, you can create it from here. Remember to also include that token in your request. You can see how to do it in the section about testing the server.

The standard Go way to chain middleware is to nest the calls:

// cmd/server/main.go
package main

import (
    "fmt"
    "net/http"

    "github.com/yourname/myapp/internal/middleware"
)

// userHandler is the actual business logic.
// It doesn't know about auth, logging, or request IDs —
// all of that is handled by middleware before it runs.
func userHandler(w http.ResponseWriter, r *http.Request) {
    // Retrieve the request ID that was set by RequestID middleware.
    id := middleware.GetRequestID(r.Context())
    fmt.Fprintf(w, "hello! your request ID is %s\n", id)
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/user", userHandler)

    // Chain middleware by wrapping from the outside in.
    // The outermost middleware runs first on the way IN,
    // and last on the way OUT.
    //
    // RequestID must wrap Logger — not the other way around.
    // Logger only ever sees the *r it directly receives. If RequestID
    // runs AFTER Logger, Logger is holding the original request,
    // not the enriched one RequestID produces — so the ID would be empty.
    //
    // Execution order for a request:
    //   RequestID → Logger → Auth → userHandler
    //               (then back out in reverse)
    handler := middleware.RequestID(
        middleware.Logger(
            middleware.Auth("your-secret-token")(
                mux,
            ),
        ),
    )

    http.ListenAndServe(":8080", handler)
}

Testing the server

# Start the server
go run ./cmd/server

# Valid request — should return 200 with a request ID
curl -H "Authorization: Bearer your-secret-token" http://localhost:8080/user

# Missing token — should return 401
curl http://localhost:8080/user

# Invalid token — should return 401
curl -H "Authorization: Bearer wrong-token" http://localhost:8080/user

Server logs:

2026/06/21 20:46:13 INFO request method=GET path=/user status=200 duration=7.208µs requestID=e667de2b-310a-4038-9891-2d948613e93b
2026/06/21 20:46:15 INFO request method=GET path=/user status=200 duration=90.667µs requestID=23a61730-4dd2-4b01-8dc5-986fbcb46436

Every request gets logged. The request ID ties your server log to the X-Request-ID header in the response, if a client reports a bug, they can give you their request ID and you can find exactly that request in your logs.

Best practices

Order matters, RequestID before Logger, Logger before Auth

The outermost middleware runs first on the way in and last on the way out. Two rules drive the order here:

RequestID must be outermost (or at least before Logger), Logger can only read context values from the exact *http.Request it was handed. context.WithValue and r.WithContext don't mutate the original request, they return a new one. If RequestID runs after Logger, Logger is stuck holding the original, un-enriched request, GetRequestID will always come back empty. If you are new to Go contexts, you can check this post about it; it explains everything you need to know about Go contexts.
Logger should wrap Auth, so that failed (401) requests still get logged, not just successful ones.

// Good order
Chain(mux,
    middleware.RequestID, // outermost: enriches the request first
    middleware.Logger,    // sees the enriched request — requestID shows up in logs
    middleware.Auth(...), // innermost: blocks invalid requests, but they're still logged
)

// Bad order — this is the bug you'll hit if you swap these two
Chain(mux,
    middleware.Logger,    // ← runs first, but hasn't seen RequestID's context yet
    middleware.RequestID, // ← enriches the request, but only for handlers AFTER this point
    middleware.Auth(...),
)
// Logger's slog.Info call reads r.Context() from its OWN copy of r —
// which was captured before RequestID ever ran. requestID="" every time.

The general rule: any middleware that reads a context value must be positioned after (further in than) the middleware that sets it.

Never call next.ServeHTTP more than once

Calling it twice sends a double response, which causes a panic in Go's HTTP stack. Always return after an early exit:

// Good — returns after writing the error
if !valid {
    http.Error(w, "unauthorized", http.StatusUnauthorized)
    return // ← critical: stops execution here
}
next.ServeHTTP(w, r)

// Bad — falls through and calls next anyway
if !valid {
    http.Error(w, "unauthorized", http.StatusUnauthorized)
    // missing return — next.ServeHTTP still runs!
}
next.ServeHTTP(w, r)

Use unexported context keys

Always define context keys as a custom unexported type, not a plain string. This prevents key collisions between packages:

// Bad — any package using the string "userID" will collide
ctx = context.WithValue(ctx, "userID", id)

// Good — only this package can access this key
type key string
const userIDKey key = "userID"
ctx = context.WithValue(ctx, userIDKey, id)

Tips for advanced users

Tip 1: Use a third-party router for cleaner middleware

The standard library's http.ServeMux applies middleware to all routes at once. If you need per-route middleware (auth on some routes, not others), a router like chi or gorilla/mux makes this clean:

r := chi.NewRouter()

r.Use(middleware.Logger)    // applies to all routes
r.Use(middleware.RequestID) // applies to all routes

r.Group(func(r chi.Router) {
    r.Use(middleware.Auth("secret")) // applies only to routes in this group
    r.Get("/admin", adminHandler)
    r.Get("/users", usersHandler)
})

r.Get("/health", healthHandler) // no auth required

Tip 2: Capture response body size

Extend responseWriter to also track bytes written — useful for monitoring:

type responseWriter struct {
    http.ResponseWriter
    statusCode int
    bytes      int
}

func (rw *responseWriter) Write(b []byte) (int, error) {
    n, err := rw.ResponseWriter.Write(b)
    rw.bytes += n // track how many bytes were written
    return n, err
}

Tip 3: Recover from panics

A panic in any handler crashes the goroutine handling that request. A recovery middleware catches it and returns a 500 instead of crashing:

func Recovery(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        defer func() {
            if err := recover(); err != nil {
                slog.Error("panic recovered", "error", err, "requestID", GetRequestID(r.Context()))
                http.Error(w, "internal server error", http.StatusInternalServerError)
            }
        }()
        next.ServeHTTP(w, r)
    })
}

Add this as the outermost middleware so it catches panics from every layer:

Chain(mux,
    middleware.Recovery,  // outermost — catches panics from every layer below
    middleware.RequestID, // sets the request ID next
    middleware.Logger,    // sees the enriched request — requestID shows up in logs
    middleware.Auth("secret"),
)

Summary

Middleware is just a function that wraps a handler and returns a new one. Everything else follows from that.

Three things to take away:

next.ServeHTTP(w, r.WithContext(ctx)), this is the line that passes the request forward with an enriched context. Understanding it unlocks the whole pattern.
Order matters, outermost middleware runs first on the way in, last on the way out. Anything that sets a context value (like RequestID) must come before anything that reads it (like Logger), or the value will be empty.
Always return after early exits, calling next.ServeHTTP after writing an error response causes a double response panic.

The three middleware functions in this post, RequestID, Logger, and Auth, are production-ready. Add Recovery and you have a solid foundation for any Go HTTP server.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Go types explained: the foundation you need before writing HTTP handlers

Fer Rios — Mon, 22 Jun 2026 14:22:54 +0000

A quick introduction to Go types

According to https://go.dev/ref/spec#Types, in Go, a type specifies a set of values, along with operations and methods specific to those values.

This is something fundamental about Go: you can define your own types based on existing ones, and those types can have methods attached to them.

You've probably seen this with structs:

// A struct type — groups fields together
type User struct {
    Name  string
    Email string
}

// A method attached to the User type
func (u User) Greet() string {
    return "Hello, " + u.Name
}

But Go lets you do the same thing with any type, including functions.

Function types

In Go, functions are values. You can assign them to variables, pass them as arguments, and — most importantly here, define a named type based on a function signature:

// This defines a new type called "Greeter"
// whose underlying type is a function that takes a string and returns a string.
type Greeter func(name string) string

Now Greeter is a type, just like int or string or User. Any function with the right signature (func(string) string) is automatically a Greeter.

// This function matches the Greeter signature
func sayHello(name string) string {
    return "Hello, " + name
}

// So we can assign it to a Greeter variable
var g Greeter = sayHello

fmt.Println(g("Alice")) // Hello, Alice

Attaching methods to a function type

Here's where it gets interesting. Because Greeter is now a named type, you can attach methods to it, exactly like you would with a struct:

// A method on a function type — this is valid Go
func (g Greeter) Shout(name string) string {
    return strings.ToUpper(g(name)) // call the function, then shout it
}

fmt.Println(g.Shout("Alice")) // HELLO, ALICE

This might look strange at first. A function with a method? But it makes perfect sense in Go. Greeter is just a type, and any type can have methods.

Why this matters for HTTP handlers

Now that you understand function types, let's look at how Go's HTTP package uses this exact pattern to make your life easier.

Go's HTTP package has two things that are easy to confuse at first: http.Handler and http.HandlerFunc. They sound almost identical, but they play different roles: one is an interface, and one is a type.

http.Handler — the interface

An HTTP handler in GO is an object or function responsible for processing an incoming HTTP request and constructing an HTTP response.

http.Handler is the interface that represents anything that can handle an HTTP request. It has exactly one method:

// http.Handler is an interface — a contract.
// It says: "I don't care what you are.
// As long as you have a ServeHTTP method, you can handle HTTP requests."
type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

http.ResponseWriter: An interface used to construct and send the HTTP response (setting headers, status codes, and writing the body).
*http.Request: A pointer to a struct containing all incoming data (headers, query parameters, URL path, and request body).

So, to be recognized as a handler, a type must implement the http.Handler interface by defining a ServeHTTP.

This is what Go's HTTP server actually works with internally. When you call http.ListenAndServe, the second argument must be an http.Handler:

// The second argument is an http.Handler —
// Go's server calls ServeHTTP on it for every incoming request.
http.ListenAndServe(":8080", myHandler)

Any type that has a ServeHTTP method satisfies this interface automatically. A struct, a function type, anything, as long as it has that one method.

http.HandlerFunc — the type

Here's the problem: writing a full struct with a ServeHTTP method for every single handler in your app is verbose. Most of the time, you just want to write a plain function:

// This is clean and simple, just a function.
// But it's NOT an http.Handler because it has no ServeHTTP method.
func greetUser(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello!")
}

This function can't be passed to http.ListenAndServe directly, it's not an http.Handler. It has no ServeHTTP method.

This is exactly the problem http.HandlerFunc solves. It's a named function type with a ServeHTTP method attached, which makes it an http.Handler:

// HandlerFunc is a named type based on a function signature.
// Any function that matches func(ResponseWriter, *Request)
// is automatically compatible with this type.
type HandlerFunc func(ResponseWriter, *Request)

// ServeHTTP is attached to HandlerFunc.
// It just calls the function itself — f is the function, so f(w, r) runs it.
// This single method is what makes HandlerFunc satisfy http.Handler.
func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
    f(w, r)
}

Now Go can convert your plain function into an http.HandlerFunc, which has ServeHTTP, which satisfies http.Handler.

You can imagine that this is what Go does under the hood:

var fn http.HandlerFunc = greetUser  // convert the function to HandlerFunc
var h  http.Handler    = fn          // HandlerFunc satisfies http.Handler

But, in practice, you never write these steps manually; you use the http.HandleFunc function.

http.HandleFunc

// Registers the "/hello" route on the default mux.
// http.HandleFunc converts greetUser to HandlerFunc internally —
// you never see that conversion happen.
http.HandleFunc("/hello", greetUser)

http.HandleFunc register a route on the default server mux. It takes a path and a handler function, converts the function to a HandlerFunc internally, and registers it.

http.HandleFunc actually uses http.HandlerFunc internally, it's just hiding it from you:

// This is roughly what http.HandleFunc does under the hood:
func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
    // converts your function to HandlerFunc, then registers it
    DefaultServeMux.Handle(pattern, HandlerFunc(handler))
    //                                ^^^^^^^^^^^^^
    //                                HandlerFunc TYPE doing the conversion
}

So http.HandleFunc is convenience on top of http.HandlerFunc. When you call http.HandleFunc("/hello", greetUser), Go silently wraps greetUser in an http.HandlerFunc and registers it. You only need to use http.HandlerFunc directly when you're building something lower-level, like middleware, where you need to return an http.Handler explicitly.

Putting it all together

Here's the full picture in one example:

// A plain function — clean and simple to write
func greetUser(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello!")
}

// A struct handler — useful when you need to hold state like a DB client
type UserHandler struct {
    db *sql.DB // injected dependency
}

func (h *UserHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // can use h.db here
    fmt.Fprintln(w, "Hello from UserHandler!")
}

func main() {
    mux := http.NewServeMux()

    // http.HandleFunc converts greetUser → HandlerFunc → Handler automatically
    mux.HandleFunc("/hello", greetUser)

    // UserHandler already has ServeHTTP — it satisfies http.Handler directly
    mux.Handle("/user", &UserHandler{db: myDB})

    // Both routes are registered on the same mux —
    // the server calls ServeHTTP on whichever one matches the request path.
    http.ListenAndServe(":8080", mux)
}

TIP: Use a plain function (via HandlerFunc) for simple handlers. Use a struct for handlers that need dependencies like a database client or a config object. The server doesn't care which one you chose — it just calls ServeHTTP.

Summary

Think of it this way:

http.Handler is the job description: "must be able to handle HTTP requests"
http.HandlerFunc is one way to fulfill that job description: "a plain function that handles HTTP requests"
Your struct with a ServeHTTP method is another way to fulfill it

The server only knows about http.Handler. It never knows whether you used a struct or a HandlerFunc — and it doesn't need to. That's the power of the interface.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Go Context Package

Fer Rios — Wed, 17 Jun 2026 00:08:53 +0000

What is context

Every Go developer encounters context.Context early. It shows up in function signatures everywhere, HTTP handlers, database calls, AWS SDK methods, gRPC stubs. Most beginners copy the pattern without fully understanding it, passing context.Background() everywhere and moving on.

That works until it doesn't. A Lambda that ignores context hangs past its timeout. A database query that ignores context keeps running after the HTTP client disconnected.

Context is a built-in package in the Go standard library that enables the propagation of cancellation signals, deadlines, and values across API boundaries and between processes.

Why Context Exists

Before context was added to the standard library in Go 1.7, there was no standard way to answer these questions across a call chain:

Should this operation still be running, or has the caller given up?
Is there a deadline this operation must finish by?
What request-scoped data (trace ID, user ID, auth token) should flow with this call?

What is really a Context?

The context.Context type is an interface with exactly four methods. Every context implements all four.

type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key any) any
}

Here's what each one does.

Deadline() — does this context have an expiry?

deadline, ok := ctx.Deadline()

Returns the time when this context will automatically cancel, and a boolean indicating whether a deadline was set at all. If you created the context with context.Background() or context.WithCancel(), there's no deadline — ok will be false. If you used context.WithTimeout() or context.WithDeadline(), ok is true and deadline tells you exactly when it expires.

You don't call this often in day-to-day code, but it's useful when you need to decide whether there's enough time left to start an expensive operation.

Done() — tell me when to stop

case <-ctx.Done():

Returns a channel that gets closed the moment the context is cancelled or times out. You listen to this channel, typically inside a select statement, to know when to abandon work. Until the context ends, the channel blocks. When it closes, your code unblocks and can clean up.

This is the method you'll use most often in loops and long-running operations.

Err() — why did we stop?

err := ctx.Err()

Returns nil while the context is still alive. Once the context ends, it returns one of two sentinel errors:

context.DeadlineExceeded — the timeout fired automatically
context.Canceled — someone called cancel() manually

This is how you distinguish between "we ran out of time" and "something upstream decided to abort". Both mean stop, but they mean different things for logging and debugging.

Value() — carry data across function calls

id := ctx.Value(myKey)

Retrieves a value stored in the context by key. This is used to pass request-scoped data, like a request ID, a user ID, or an auth token, through a chain of function calls without adding extra parameters to every function signature.

One important rule: use a custom unexported type for your keys (not a plain string), otherwise different packages might accidentally overwrite each other's values.

Context's creation

With contexts, there is a tree structure. The parent context is located at the root of the context tree, and the newly derived context from the parent context is called the child context. Through the parent context, four types of contexts can be derived using the four With methods, and each new context can continue to derive child contexts, making the whole structure look like a tree.

Parent Context

There are two ways to create the parent context: an empty context still with no functionality.

context.Background()

ctx := context.Background()

Use this at the top of your program — in main(), in init(), or as the starting point when creating a server. It's the foundation everything else derives from.

context.TODO()

Semantically identical to context.Background() at runtime, but signals intent: "I know a context should go here, I haven't figured out which one yet."

ctx := context.TODO() // placeholder — replace with a real context later

Use it when you're refactoring code to add context support and haven't wired up the full chain yet. It's a marker for future work, not a production pattern.

Child Contexts

To make the context useful, we use one of the following child contexts:

func WithCancel(parent Context) (ctx Context, cancel CancelFunc)
func WithDeadline(parent Context, deadline time.Time) (Context, CancelFunc)
func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc)
func WithValue(parent Context, key, val interface{}) Context

context.WithCancel()

Returns a copy of the parent context with a new Done channel, plus a cancel function. Calling cancel closes the Done channel and marks the context as canceled.

ctx, cancel := context.WithCancel(context.Background())
defer cancel() // always call cancel

Always call cancel. If you don't, the context and everything derived from it leaks until the parent is canceled or the program exits. defer cancel() immediately after creation is the idiomatic pattern.

context.WithTimeout() and context.WithDeadline()

These set an automatic cancellation time. WithTimeout takes a duration; WithDeadline takes an absolute time.Time.

// Cancel after 5 seconds
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

// Cancel at a specific moment
deadline := time.Now().Add(5 * time.Second)
ctx, cancel := context.WithDeadline(context.Background(), deadline)
defer cancel()

WithTimeout(parent, d) is shorthand for WithDeadline(parent, time.Now().Add(d)). They behave identically. When the deadline passes, the context's Done channel is closed automatically and Err() returns context.DeadlineExceeded.

Still call defer cancel() even with a timeout — if your operation finishes before the deadline, calling cancel releases the timer resources immediately rather than waiting for it to fire.

context.WithValue

Creates a child context from the parent context for value passing. Used for passing context information, such as the unique request ID and trace ID, for link tracing and configuration passing.

ctx := context.WithValue(context.Background(), authToken, "XYZ_123")
fmt.Println(ctx.Value(authToken))

How Cancellation Propagates

Remember, contexts form a tree. When a parent context is canceled, all contexts derived from it are canceled too — automatically, immediately, without any extra code.

parent, cancelParent := context.WithCancel(context.Background())

child1, cancelChild1 := context.WithCancel(parent)
child2, cancelTimeout := context.WithTimeout(parent, 10*time.Second)

// Canceling the parent cancels both children
cancelParent()

// child1.Err() == context.Canceled
// child2.Err() == context.Canceled

This is the mechanism that makes context useful across a call tree. An HTTP handler creates a context with a timeout. It passes that context to a database call, which passes it to a connection pool, which passes it to a network read. When the HTTP client disconnects, the handler's context is canceled, and that cancellation flows down through every layer automatically.

Real-World Use Cases

HTTP Servers: Respecting Client Disconnects

Go's net/http package attaches a context to every incoming request. The context is canceled when the client disconnects or the server's write deadline fires. Your handler receives it via r.Context().

The following is a fully runnable example. It simulates a slow db.Search using a select with time.After, shows what happens when the client disconnects mid-query, and adds a middleware that attaches a request ID to the context, so every layer of the call chain can log it without it being passed as an explicit argument. You can find the repo here:

package main

import (
    "context"
    "errors"
    "fmt"
    "log/slog"
    "net/http"
    "os"
    "time"

    "github.com/google/uuid"
)

// contextKey is an unexported type for context keys in this package.
// Using a custom type prevents collisions with keys from other packages
// that might also store something under a plain string like "request_id".
type contextKey string

const keyRequestID contextKey = "request_id"

// withRequestID attaches a request ID to the context.
func withRequestID(ctx context.Context, id string) context.Context {
    return context.WithValue(ctx, keyRequestID, id)
}

// requestIDFromContext retrieves the request ID from the context.
// Returns an empty string if no ID was set — callers should handle that gracefully.
func requestIDFromContext(ctx context.Context) string {
    id, _ := ctx.Value(keyRequestID).(string)
    return id
}

var logger = slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
    ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
        if a.Key == slog.TimeKey {
            return slog.Attr{}
        }
        return a
    },
}))

// requestIDMiddleware generates a unique ID for every incoming request,
// attaches it to the context, and sends it back in the response header.
// Every function below the handler can read it from ctx for logging —
// without it being passed as a function argument anywhere.
func requestIDMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        requestID := uuid.NewString()

        ctx := withRequestID(r.Context(), requestID)
        w.Header().Set("X-Request-ID", requestID)

        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

// Search simulates a slow database query that respects context cancellation.
// It reads the request ID directly from ctx — no extra parameter needed.
func Search(ctx context.Context, query string, delay time.Duration) ([]string, error) {
    requestID := requestIDFromContext(ctx)

    logger.Info("db.Search started",
        "request_id", requestID,
        "query", query,
        "simulated_delay", delay,
    )

    select {
    case <-time.After(delay):
        logger.Info("db.Search completed", "request_id", requestID, "query", query)
        return []string{"result-1", "result-2", "result-3"}, nil

    case <-ctx.Done():
        logger.Warn("db.Search aborted",
            "request_id", requestID,
            "query", query,
            "reason", ctx.Err(),
        )
        return nil, ctx.Err()
    }
}

func searchHandler(w http.ResponseWriter, r *http.Request) {
    requestID := requestIDFromContext(r.Context())

    query := r.URL.Query().Get("q")
    if query == "" {
        query = "gophers"
    }

    delay := 5 * time.Second
    if d := r.URL.Query().Get("delay"); d != "" {
        if parsed, err := time.ParseDuration(d); err == nil {
            delay = parsed
        }
    }

    logger.Info("handler started",
        "request_id", requestID,
        "query", query,
        "client", r.RemoteAddr,
    )

    // r.Context() carries both the cancellation signal AND the request ID value.
    // Search receives one ctx and gets everything it needs from it.
    results, err := Search(r.Context(), query, delay)
    if err != nil {
        if errors.Is(err, context.Canceled) {
            logger.Warn("client disconnected before query finished, no response sent",
                "request_id", requestID,
            )
            return
        }
        logger.Error("unexpected search error", "request_id", requestID, "error", err)
        http.Error(w, "search failed", http.StatusInternalServerError)
        return
    }

    logger.Info("handler finished — sending response",
        "request_id", requestID,
        "result_count", len(results),
    )
    fmt.Fprintf(w, "results: %v\n", results)
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/search", searchHandler)

    // Wrap the mux — every request gets a request ID attached to its context
    handler := requestIDMiddleware(mux)

    logger.Info("server listening on :8080")
    logger.Info("try: curl -v 'http://localhost:8080/search?q=gophers&delay=5s'")
    logger.Info("cancel with Ctrl+C before 5s to trigger client disconnect")

    if err := http.ListenAndServe(":8080", handler); err != nil {
        logger.Error("server error", "error", err)
        os.Exit(1)
    }
}

Before running, initialize the module and fetch the uuid package, which is in charge of creating random uuids for the incoming requests:

go mod init example/context-demo
go get github.com/google/uuid

To test the cancellation path:

# Terminal 1 — start the server
go run main.go

# Terminal 2 — send a request with a 5s delay, then hit Ctrl+C before it finishes
curl 'http://localhost:8080/search?q=gophers&delay=5s'
# Press Ctrl+C after ~2 seconds

You should see in the server logs:

level=INFO msg="handler started" request_id=4a1f9c2e-... query=gophers client=127.0.0.1:PORT
level=INFO msg="db.Search started" request_id=4a1f9c2e-... query=gophers simulated_delay=5s
level=WARN msg="db.Search aborted" request_id=4a1f9c2e-... query=gophers reason="context canceled"
level=WARN msg="client disconnected before query finished, no response sent" request_id=4a1f9c2e-...

Notice the same request_id appears in every log line, from the middleware, through the handler, all the way into Search, without it ever being passed as a function argument. It traveled through the context.

You can also verify the ID was sent back in the response header:

curl -v 'http://localhost:8080/search?q=gophers&delay=5s'| grep X-Request-ID
# X-Request-ID: 4a1f9c2e-...

To test the happy path

Let the query finish before disconnecting:

curl 'http://localhost:8080/search?q=gophers&delay=2s'
# Wait for it to complete

level=INFO msg="handler started" request_id=9b3d7f1a-... query=gophers client=127.0.0.1:PORT
level=INFO msg="db.Search started" request_id=9b3d7f1a-... query=gophers simulated_delay=2s
level=INFO msg="db.Search completed" request_id=9b3d7f1a-... query=gophers
level=INFO msg="handler finished — sending response" request_id=9b3d7f1a-... result_count=3

The key is the select inside Search. It blocks on two channels simultaneously: time.After(delay) represents the slow query, and ctx.Done() represents the client. Whichever fires first wins. If the client disconnects, ctx.Done() closes and Search returns context.Canceled immediately, without waiting for the timer.

This example shows both things context is good for working together in one flow: the cancellation signal (ctx.Done()) stops the query when the client leaves, and the value (request_id) flows through every layer for correlated logging, all through the same single ctx argument.

Database Calls: Enforcing Query Timeouts

Most database drivers in Go accept a context. Pass one with a timeout to prevent slow queries from holding connections indefinitely.

func getUserByID(ctx context.Context, db *sql.DB, userID string) (*User, error) {
    // Give the query 3 seconds — no matter what the caller's deadline is
    queryCtx, cancel := context.WithTimeout(ctx, 3*time.Second)
    defer cancel()

    var user User
    err := db.QueryRowContext(queryCtx,
        "SELECT id, name, email FROM users WHERE id = $1", userID,
    ).Scan(&user.ID, &user.Name, &user.Email)
    if err != nil {
        if errors.Is(err, context.DeadlineExceeded) {
            return nil, fmt.Errorf("user query timed out after 3s: %w", err)
        }
        return nil, fmt.Errorf("querying user %s: %w", userID, err)
    }

    return &user, nil
}

Notice that queryCtx derives from ctx — the caller's context. If the caller's context is already canceled (client disconnected, Lambda timed out), queryCtx is canceled immediately regardless of the 3-second timeout. The child inherits the parent's cancellation, but can set a tighter deadline on top of it.

Checking for Cancellation in Long Loops

If your function does significant work in a loop, check ctx.Err() periodically so it can exit early when canceled.

func processFiles(ctx context.Context, files []string) error {
    for _, file := range files {
        // Check at the top of each iteration
        if err := ctx.Err(); err != nil {
            return fmt.Errorf("processing canceled after %s: %w", file, err)
        }

        if err := processFile(ctx, file); err != nil {
            return fmt.Errorf("processing %s: %w", file, err)
        }
    }
    return nil
}

Without this check, a canceled context doesn't stop the loop — it only stops operations that accept and check the context themselves. If processFile takes 500ms and you have 1000 files, a cancellation mid-loop still runs all remaining iterations.

Common Mistakes and Gotchas

Passing context.Background() everywhere. It works, but it opts every operation out of cancellation and timeout propagation. If your HTTP handler times out, your database queries keep running. Pass the incoming context through your call chain — that's the whole point.

Not calling cancel(). Every WithCancel, WithTimeout, and WithDeadline must have its cancel called. The Go runtime cannot garbage-collect a context until cancel is called or the parent is canceled. defer cancel() right after creation is non-negotiable.

Storing contexts in structs. The Go documentation is explicit: do not store contexts in structs. A context is for a single operation, not for the lifetime of an object. Pass it as the first argument to every function that needs it.

// Wrong
type Service struct {
    ctx context.Context // don't do this
}

// Right
func (s *Service) DoWork(ctx context.Context) error { ... }

Using context.WithValue for function dependencies. If you're pulling a database client out of a context, something has gone wrong. Use dependency injection or package-level variables for clients and config. Context values are for request-scoped metadata.

Ignoring ctx.Err() in loops. Passing a context to a function that calls ctx.Err() isn't enough if your own loop never checks it. Long-running loops need explicit cancellation checks.

Wrapping errors without checking for cancellation. When a context-aware call fails, check whether the cause is cancellation before deciding how to handle it:

if err != nil {
    if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
        // The operation was stopped — not necessarily a bug
        return err
    }
    // Real error — log, alert, or return
    return fmt.Errorf("unexpected error: %w", err)
}

Summary

The context package gives every Go operation a standard way to answer three questions: should I keep running, when must I finish by, and what metadata flows with this request?

The key things to take away:

context.Background() is the root — use it at program entry points.
WithCancel, WithTimeout, and WithDeadline create derived contexts — always defer cancel().
Cancellation propagates from parent to child automatically — wire it through your call chain and it works for free.
Pass context as the first argument to every function that does I/O — HTTP, database, AWS SDK, gRPC.
WithValue is for cross-cutting metadata like trace IDs, not for dependencies.
Check ctx.Err() explicitly in long-running loops.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Go Packages and Modules explained

Fer Rios — Thu, 11 Jun 2026 00:37:56 +0000

What is a package?

In Go, every Go program is made up of packages. A package is a directory of .go files that share the same package declaration. The primary purpose of packages is to help you isolate and reuse code.

myapp/
├── main.go       ← package main
└── math/
    ├── add.go    ← package math
    └── sub.go    ← package math

Both add.go and sub.go declare package math. They can call each other's functions directly, no import needed within the same package.

Inside a package, every .go file should begin with a package {name} statement which indicates the name of the package that the file is a part of. Every exported identifier (capitalized name) in that directory is accessible to anyone who imports the package.

Here's what that looks like in practice:

// math/add.go
package math

// pi is an unexported variable.
var pi = 3.14159

// Add returns the sum of two integers.
// Exported — starts with a capital letter.
func Add(a, b int) int {
    return a + b
}

// math/sub.go
package math

// Exported — starts with a capital letter.
func Subtract(a, b int) int {

    return a-b
}

// main.go
package main

import (
    "fmt"
    "github.com/yourname/myapp/math"
)

func main() {
    fmt.Println(math.Add(3, 4))      // 7
    fmt.Println(math.Subtract(10, 3)) // 7
    // fmt.Println(math.pi) — compile error: unexported
}

Two rules to remember:

Capital letter = exported (public). Lowercase = unexported (private to the package).
One package per directory. One directory per package.

What is a module?

If a package is a folder, a module is the whole project, a tree of packages with a name, a Go version requirement, and a list of external dependencies.

When you start a Go project, you create a module, and inside that module, there will be packages.

Every Go project has exactly one go.mod file at its root. That file defines the module. Here's what a real one looks like:

module github.com/yourname/weather-cli

go 1.21

require (
    github.com/aws/aws-sdk-go-v2 v1.24.0
    github.com/aws/aws-sdk-go-v2/service/s3 v1.47.0
    gopkg.in/yaml.v3 v3.0.1
)

Three things in every go.mod:

module — the module path. This is the base import path for every package in your project. It's usually your GitHub URL, but it can be anything.
go — the minimum Go version your code requires.
require — the external dependencies your module needs, each pinned to an exact version.

go.sum — the lockfile

Alongside go.mod lives go.sum. You never edit this by hand. It contains cryptographic checksums for every dependency (and their dependencies) your module uses:

github.com/aws/aws-sdk-go-v2 v1.24.0 h1:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=
github.com/aws/aws-sdk-go-v2 v1.24.0/go.mod h1:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=

go.sum guarantees that the code you download is bit-for-bit identical to what was there when you first added the dependency. It's your protection against supply chain attacks and "works on my machine" problems. Always commit both go.mod and go.sum.

Working with modules day to day

Creating a new module

So, let's say you want to start a new project: you create the folder first, then you create your go.mod file, passing your module path to it (github.com/yourname/myapp).

mkdir myapp && cd myapp
go mod init github.com/yourname/myapp

This creates go.mod with your module path. That's it — you're ready to write Go.

Adding a dependency

go get github.com/aws/aws-sdk-go-v2/service/s3@latest

go get does three things: downloads the package, adds it to go.mod, and updates go.sum. After running it your go.mod will have a new require entry.

You can also just write the import in your code and run go mod tidy — it figures out what's missing and adds it:

go mod tidy

go mod tidy is the command you'll run most often. It adds missing dependencies and removes unused ones, keeping go.mod clean.

Upgrading a dependency

# Upgrade to the latest minor/patch version
go get github.com/aws/aws-sdk-go-v2@latest

# Upgrade to a specific version
go get github.com/aws/aws-sdk-go-v2@v1.25.0

Removing a dependency

Remove the import from your code, then run:

go mod tidy

go mod tidy will remove the require entry automatically if nothing in your code imports it anymore.

Viewing your dependency tree

go mod graph

This prints the full dependency graph — your dependencies, their dependencies, and so on. Useful for debugging version conflicts.

Vendoring dependencies

For environments without internet access (some CI setups, air-gapped servers), you can vendor all dependencies into a local vendor/ directory:

go mod vendor

After vendoring, go build uses vendor/ instead of the module cache. Commit vendor/ to your repo and your build never needs to call the internet.

How modules and packages connect

Let's make the relationship tangible. Say you run:

go get github.com/spf13/cobra@v1.8.0

Your go.mod now contains:

require (
    github.com/spf13/cobra v1.8.0
)

Cobra is a module at github.com/spf13/cobra. Inside that module, there are multiple packages:

github.com/spf13/cobra          ← the root package
github.com/spf13/cobra/doc      ← generates documentation
github.com/spf13/cobra/completions ← shell completions

When you import github.com/spf13/cobra in your code, you're importing one specific package from that module. The module is what gets versioned and downloaded; the package is what you actually use in your code.

import (
    "github.com/spf13/cobra"     // imports the root package of the cobra module
)

One require in go.mod, many importable packages available. That's the module/package split.

Package main is special

Every Go program needs exactly one package main with exactly one main() function. That's the entry point. Everything else is a library package — it exports functions and types but can't be run directly.

// This is a runnable program
package main

import "fmt"

func main() {
    fmt.Println("hello from main")
}

// This is a library — can't be run, only imported
package greet

import "fmt"

func Hello(name string) string {
    return fmt.Sprintf("Hello, %s!", name)
}

You can have multiple package main files in a project — but each one must live in its own directory. This is exactly what the cmd/ convention is for, which we'll get to shortly.

How imports work

The import path is always: module path (from go.mod) + directory path from the module root.

module github.com/yourname/myapp   ← module path

myapp/
├── go.mod
└── internal/
    └── config/
        └── config.go              ← import path: github.com/yourname/myapp/internal/config

// Importing your own packages — always use the full module path
import "github.com/yourname/myapp/internal/config"

// Importing from the standard library — no module path needed
import "fmt"
import "net/http"
import "encoding/json"

// Importing a third-party package — use its full module path + sub-path
import "github.com/spf13/cobra"
import "github.com/aws/aws-sdk-go-v2/service/s3"

Aliasing imports

When two packages have the same name, or when a name is too long, you can alias the import:

import (
    "fmt"

    // Alias to avoid conflict with standard library math
    gomath "github.com/yourname/myapp/math"

    // Alias for readability
    yaml "gopkg.in/yaml.v3"
)

func main() {
    fmt.Println(gomath.Add(1, 2))
    _ = yaml.Marshal
}

The blank identifier import

Sometimes you import a package only for its side effects — database drivers are the classic example. The _ alias tells Go "import this but don't use its name":

import (
    "database/sql"
    _ "github.com/lib/pq" // registers the postgres driver via init()
)

Without the _, Go's compiler would complain about an unused import and refuse to compile.

A practical project structure

Here's a real structure that scales from small to large. This is what most production Go projects converge on:

myapp/
├── go.mod
├── go.sum
├── Makefile
│
├── cmd/
│   ├── server/
│   │   └── main.go     ← the HTTP server entry point
│   └── worker/
│       └── main.go     ← the background worker entry point
│
├── internal/
│   ├── handler/
│   │   ├── handler.go
│   │   └── handler_test.go
│   ├── store/
│   │   ├── postgres.go
│   │   └── store.go
│   └── config/
│       └── config.go
│
└── pkg/
    └── validate/
        ├── validate.go
        └── validate_test.go

Let's walk through each directory.

cmd/ — entry points

One subdirectory per runnable binary. Each contains a single main.go. The main.go should do almost nothing except wire dependencies together and start the program.

// cmd/server/main.go
package main

import (
    "log/slog"
    "net/http"
    "os"

    "github.com/yourname/myapp/internal/config"
    "github.com/yourname/myapp/internal/handler"
    "github.com/yourname/myapp/internal/store"
)

func main() {
    cfg := config.Load()

    db, err := store.Connect(cfg.DatabaseURL)
    if err != nil {
        slog.Error("failed to connect to database", "error", err)
        os.Exit(1)
    }

    h := handler.New(db)

    slog.Info("starting server", "port", cfg.Port)
    if err := http.ListenAndServe(":"+cfg.Port, h); err != nil {
        slog.Error("server failed", "error", err)
        os.Exit(1)
    }
}

💡No business logic in main.go. Ever. It's a wiring diagram, not an application.

internal/ — private application code

The internal/ directory is enforced by the Go toolchain. Packages inside internal/ can only be imported by code in the parent directory tree. Code outside your module cannot import them — not even if they find your repo on the internet.

This is how you keep implementation details private:

// internal/config/config.go
package config

import "os"

type Config struct {
    Port        string
    DatabaseURL string
    LogLevel    string
}

// Load reads config from environment variables.
// This is internal — callers outside this module can't import it.
func Load() Config {
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    return Config{
        Port:        port,
        DatabaseURL: os.Getenv("DATABASE_URL"),
        LogLevel:    os.Getenv("LOG_LEVEL"),
    }
}

// internal/store/store.go
package store

import (
    "context"
    "database/sql"
    "fmt"

    _ "github.com/lib/pq"
)

// Store handles database operations.
type Store struct {
    db *sql.DB
}

func Connect(dsn string) (*Store, error) {
    db, err := sql.Open("postgres", dsn)
    if err != nil {
        return nil, fmt.Errorf("open db: %w", err)
    }
    if err := db.Ping(); err != nil {
        return nil, fmt.Errorf("ping db: %w", err)
    }
    return &Store{db: db}, nil
}

// GetUser fetches a user by ID.
func (s *Store) GetUser(ctx context.Context, id string) (string, error) {
    var name string
    err := s.db.QueryRowContext(ctx, "SELECT name FROM users WHERE id = $1", id).Scan(&name)
    if err != nil {
        return "", fmt.Errorf("get user %s: %w", id, err)
    }
    return name, nil
}

pkg/ — reusable public code

pkg/ contains packages that are safe to import by external projects. Put things here that are genuinely reusable across projects — validation logic, utility functions, shared types. If in doubt, use internal/ instead.

// pkg/validate/validate.go
package validate

import (
    "errors"
    "strings"
)

// Email returns an error if the given string is not a valid email address.
// Simple check — not RFC 5322 compliant, but good enough for most cases.
func Email(email string) error {
    if email == "" {
        return errors.New("email is required")
    }
    if !strings.Contains(email, "@") || !strings.Contains(email, ".") {
        return errors.New("email is invalid")
    }
    return nil
}

// Required returns an error if the string is empty or whitespace only.
func Required(field, value string) error {
    if strings.TrimSpace(value) == "" {
        return fmt.Errorf("%s is required", field)
    }
    return nil
}

A practical example: a CLI tool

GitHub Repository: https://github.com/FerRiosCosta/weather-cli

Let's make this concrete with a small but complete project — a CLI that fetches weather for a city. It shows package structure, separation of concerns, and how packages talk to each other. Let's create the following directory structure:

weather-cli/
├── go.mod
├── cmd/
│   └── weather/
│       └── main.go
└── internal/
    ├── api/
    │   └── api.go        ← HTTP client for weather API
    └── display/
        └── display.go    ← formats output for the terminal

Open your terminal and create the following directories and files:

mkdir weather-cli
cd weather-cli
mkdir -p cmd/weather
touch cmd/weather/main.go
mkdir -p internal/api
touch internal/api/api.go
mkdir -p internal/display
touch internal/display/display.go

Open VSCode while you are in the weather-cli root directory:

code .

Go to the Openweathermap portal and create an account if you don't have one. Once you have created it, your API key will be enabled after a couple of hours.

Open a new terminal from VScode since we are going to run our program from here, but first we need to export our API KEY.

export WEATHER_API_KEY=<your-api-key>

Now, initialize your module (or project) by running the following command:

go mod init

You should see your new go.mod file created now:

module github.com/FerRiosCosta/weather-cli

go 1.26.3

Start filling the files you created with the code below:

// internal/api/api.go
package api

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
)

// WeatherResponse holds the data we care about from the API.
type WeatherResponse struct {
    City        string  `json:"name"`
    Temperature float64 `json:"main.temp"`
    Description string  `json:"weather.0.description"`
}

// apiResponse mirrors the actual nested JSON structure from OpenWeatherMap.
// Go's JSON decoder doesn't support dot notation like `json:"main.temp"` —
// nested fields require nested structs.
//
// The API returns something like:
//
//  {
//    "name": "Asuncion",
//    "main": { "temp": 28.4 },
//    "weather": [{ "description": "partly cloudy" }]
//  }
type apiResponse struct {
    Name string `json:"name"`
    Main struct {
        Temp float64 `json:"temp"`
    } `json:"main"`
    Weather []struct {
        Description string `json:"description"`
    } `json:"weather"`
}

// Client makes requests to the weather API.
type Client struct {
    apiKey     string
    httpClient *http.Client
}

// NewClient creates and returns a new Client configured with the given API key.
// This is a constructor function — Go doesn't have classes or `new` keywords,
// so by convention we define a function named New<TypeName> to build a struct.
func NewClient(apiKey string) *Client {
    return &Client{
        // Store the API key so every request this client makes can use it.
        // It's set once here and reused — no need to pass it on every call.
        apiKey: apiKey,
        // Create a default HTTP client for making requests.
        // We initialize it here rather than inside GetWeather so it's
        // reused across calls — http.Client maintains a connection pool
        // internally, so reusing it is more efficient than creating a new
        // one each time.
        httpClient: &http.Client{},
    }
}

// GetWeather fetches current weather for the given city.
func (c *Client) GetWeather(ctx context.Context, city string) (*WeatherResponse, error) {
    url := fmt.Sprintf(
        "https://api.openweathermap.org/data/2.5/weather?q=%s&appid=%s&units=metric",
        city, c.apiKey,
    )

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return nil, fmt.Errorf("create request: %w", err)
    }

    resp, err := c.httpClient.Do(req)
    if err != nil {
        return nil, fmt.Errorf("fetch weather: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return nil, fmt.Errorf("unexpected status: %d", resp.StatusCode)
    }

    // Decode into the nested API struct first.
    var raw apiResponse
    if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil {
        return nil, fmt.Errorf("decode response: %w", err)
    }

    // Guard against an empty weather array — the API should always return
    // at least one entry, but defensive code is good code.
    description := ""
    if len(raw.Weather) > 0 {
        description = raw.Weather[0].Description
    }

    // Return the clean, flat struct callers actually care about.
    return &WeatherResponse{
        City:        raw.Name,
        Temperature: raw.Main.Temp,
        Description: description,
    }, nil
}

// internal/display/display.go
package display

import (
    "fmt"
    "io"

    "github.com/yourname/weather-cli/internal/api"
)

// Weather prints a formatted weather summary to w.
func Weather(w io.Writer, data *api.WeatherResponse) {
    fmt.Fprintf(w, "City:        %s\n", data.City)
    fmt.Fprintf(w, "Temperature: %.1f°C\n", data.Temperature)
    fmt.Fprintf(w, "Conditions:  %s\n", data.Description)
}

// cmd/weather/main.go
package main

import (
    "context"
    "log/slog"
    "os"

    "github.com/yourname/weather-cli/internal/api"
    "github.com/yourname/weather-cli/internal/display"
)

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintln(os.Stderr, "usage: weather <city>")
        os.Exit(1)
    }

    city := os.Args[1]
    apiKey := os.Getenv("WEATHER_API_KEY")
    if apiKey == "" {
        slog.Error("WEATHER_API_KEY environment variable is required")
        os.Exit(1)
    }

    client := api.NewClient(apiKey)

    weather, err := client.GetWeather(context.Background(), city)
    if err != nil {
        slog.Error("failed to get weather", "city", city, "error", err)
        os.Exit(1)
    }

    display.Weather(os.Stdout, weather)
}

Run it:

go run ./cmd/weather Asuncion


City:        Asuncion
Temperature: 28.4°C
Conditions:  partly cloudy

Notice how each package has one job:

api — knows how to talk to the weather API. Nothing else.
display — knows how to format output. Nothing else.
cmd/weather/main.go — wires them together. Nothing else.

Adding a second output format (JSON, CSV) means a new file in display/. Adding a second weather provider means a new file in api/. Neither change touches the other.

Common gotchas

1. Circular imports — Go forbids them

Package A cannot import package B if package B imports package A. Go will refuse to compile.

// This will not compile
package a imports package b
package b imports package a  ← circular — error

The fix: extract the shared type into a third package that neither A nor B imports from each other.

2. Package name vs directory name

The directory name and the package name don't have to match — but they should. The one common exception is main: the directory is usually named after the binary (server, worker, weather), but the package is always package main.

// Directory: cmd/server/
// File: main.go
package main  // always "main", not "server"

3. Trying to have multiple packages in one directory

Every .go file in a directory must declare the same package name (with the exception of _test.go files, which can use package foo_test for black-box testing). This is a compile error:

myapp/
├── server.go    ← package server
└── handler.go   ← package handler  ← ERROR: can't mix packages in one directory

4. Exporting by accident

If you capitalize a function name, it's exported — immediately accessible to anyone who imports the package. This isn't always what you want, especially for internal helpers. Default to lowercase until you know something needs to be public.

Summary

Go's package and module system is built on a few simple rules that compound into something powerful.

Four things to take away:

A package is a folder — everything in it shares a namespace, capital letters are exported.
A module is a versioned collection of packages — defined by go.mod, locked by go.sum.
Use cmd/ for entry points, internal/ for private code, pkg/ for reusable public code.
go mod tidy is your best friend — run it whenever you add, remove, or change dependencies.

The project structure in this post is the same one I'll use for every project on this blog. Every Go + Lambda, Go + Kubernetes, and Go + AWS post builds on it. Once it's familiar, you'll recognize it instantly in any Go project you open on GitHub.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾