DEV Community: Fer Rios

Build an MCP Server in Go (Part 3): How IRSA and EKS Pod Identity Actually Work

Fer Rios — Thu, 27 Aug 2026 16:14:57 +0000

Everything the MCP server could check through Part 2 lives inside the cluster. But "my pod can't reach S3" is very often not a Kubernetes problem at all, it's a workload identity problem, and that identity lives partly on the Kubernetes side (the ServiceAccount) and partly on the AWS side (IAM, and now EKS Pod Identity). Before building a tool that diagnoses it, it's worth actually understanding how it works; most of the confusion people run into comes from treating IRSA and Pod Identity as interchangeable, when they solve the same problem through genuinely different mechanisms.

How pod permissions actually work on EKS

Before touching the diagnostic tool, it's worth stopping and explaining this properly, because "how does a pod get AWS permissions" trips up almost everyone the first time, and the two mechanisms work in genuinely different ways under the hood.

The problem neither mechanism existed to solve

An EC2 instance gets AWS credentials through an instance profile, no secrets anywhere, the instance just asks a local metadata endpoint (IMDS) and gets temporary credentials back, automatically. A pod is not an instance. Dozens of pods can run on the same EC2 node, each one potentially needing different AWS permissions; the checkout-service pod should be able to write to one S3 bucket, the image-resizer pod to a different one, and neither should be able to touch what the other can. If every pod on a node just inherited the node's IAM role, that isolation is gone; any pod could do anything the node itself could do.

Before:

Both IRSA and Pod Identity exist to solve exactly this: giving individual pods their own, scoped-down AWS identity, distinct from the node they happen to be running on and from every other pod sharing that node.

Now:

What a Kubernetes ServiceAccount actually is

Both mechanisms above lean on something Kubernetes already has, so it's worth pinning down before going further: a ServiceAccount is Kubernetes's own concept of "an identity a pod runs as," entirely separate from AWS and predating both IRSA and Pod Identity by years. It has nothing to do with AWS by default, its original job is purely internal to the cluster.

Every pod runs as some ServiceAccount, whether you set one explicitly or not, if you don't specify one, the pod gets the default ServiceAccount for its namespace automatically. A ServiceAccount is a real Kubernetes object, small and unremarkable on its own:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout-service
  namespace: production

Its native purpose is to authenticate to the Kubernetes API server itself, not AWS. Kubernetes automatically mounts a token for the pod's ServiceAccount into the container filesystem, and that token is what lets, say, a controller running inside the cluster call kubectl-equivalent API operations as "the checkout-service identity", with whatever RBAC permissions (Role/RoleBinding or ClusterRole/ClusterRoleBinding) have been granted to that ServiceAccount. This is the exact same object check_endpoints and describe_pod implicitly rely on: the RBAC section back in Part 2 that scoped this MCP server's own ServiceAccount to get/list/watch; that's this same mechanism, just describing what the server itself is allowed to do inside the cluster.

What both IRSA and Pod Identity do is extend that existing identity outward, to AWS, instead of inventing a new one. That's why IRSA works by annotating a ServiceAccount rather than creating some new AWS-specific object, and why Pod Identity's association is keyed on a namespace/ServiceAccount pair rather than anything else.

A ServiceAccount was already sitting there as "the identity of this pod" in Kubernetes' own model, and both mechanisms are answers to the same follow-up question: how do I get that identity recognized by AWS too?

RBAC: what a ServiceAccount is actually allowed to do

A ServiceAccount by itself is just an identity, a name Kubernetes can authenticate. It comes with zero permissions attached. What decides what that identity is actually allowed to do inside the cluster is a separate system: RBAC, Role-Based Access Control.

RBAC works through four object types, and the relationship between them is the whole model:

Role (or ClusterRole, the cluster-wide version), a set of permissions, expressed as verbs on resources. Something like "can get and list pods in the production namespace." A Role on its own grants nothing to anyone, it's a description of a permission set, sitting unused until something is bound to it.
RoleBinding (or ClusterRoleBinding), the thing that actually connects a Role to an identity. This is where a ServiceAccount (or a user, or a group) gets named and paired with aRole, and only once that binding exists does the permission become real.

Put together: a ServiceAccount is who, a Role is what's allowed, and a RoleBinding is the because I said so connecting the two. Kubernetes checks all three together on every single API request, "is the ServiceAccount making this call bound, via some RoleBinding, to a Role that permits this specific verb on this specific resource", and denies anything that doesn't match.

This is exactly what Part 2's "RBAC scope the service account, not just the interface" practice was describing, now with the actual mechanism behind it named: this MCP server runs under its own ServiceAccount, which needs a ClusterRole granting only get/list/watch (never create/update/delete/patch), connected by a ClusterRoleBinding. The read-only guarantee built into KubeClient's Go interface in Part 1 and the read-only guarantee enforced by RBAC here are two independent layers checking the same thing from different directions; the Go code structurally can't call a write operation because the interface has none, and even if it somehow did, RBAC would reject the request at the API server before it touched anything. Neither one depends on the other holding; both have to fail at once for something to go wrong.

It's also worth being precise about where RBAC's authority ends, since that boundary is exactly why this post exists: RBAC only governs the Kubernetes API. It has no opinion at all about what a pod is allowed to do once it starts making calls to AWS instead of tokube-apiserver, a ServiceAccount can be tightly scoped under RBAC and still hold an overly broad IAM role through IRSA or Pod Identity, because those are a completely separate permission system, checked by AWS, not by Kubernetes. That gap, RBAC stopping exactly at the cluster's edge, is the reason diagnose_workload_identity has to exist as its own tool rather than just being another RBAC check.

One ServiceAccount, two independent extensions. RBAC (left) is entirely a Kubernetes-side concern, it never leaves the cluster. IRSA and Pod Identity (right) both start from that same ServiceAccount but end up at AWS through completely different trust mechanisms, which is exactly why diagnosing one has nothing in common with diagnosing the other. The two sections below walk through each AWS-side path in detail.

IRSA: a pod proves who it is directly to AWS

IRSA (IAM Roles for Service Accounts) works by turning a Kubernetes ServiceAccount into something AWS can verify cryptographically, using OpenID Connect (OIDC).

OIDC (OpenID Connect) itself is worth grounding first, since everything else in this section builds on it. It's an identity protocol built on top of OAuth 2.0, and it solves a specific problem: how does party B trust a claim about identity made by party A, without B having to call A up and ask "is this real?" every single time? OIDC's answer is the JWT (JSON Web Token), a compact, signed piece of text containing claims like "this is ServiceAccount X, in namespace Y, issued at time Z". The token is signed with a private key only the issuer holds, and anyone who has the issuer's corresponding public key can verify that signature offline, no network call back to the issuer required, no shared secret sitting in a config file waiting to leak. That's the whole trick: proving something is genuine by checking a signature, not by trusting whoever happens to be presenting it. It's the same reason OIDC has become the backbone of "log in with X" buttons everywhere, the token itself carries proof of who issued it.

Before the rest of the flow makes sense, it's also worth being clear on what an OIDC issuer actually is, and, just as importantly, where it actually runs, since "the cluster's own issuer" is easy to misread as something running alongside your workloads. In OIDC terms, the issuer is the party that signs and stands behind tokens; Google, in the "Sign in with Google" example, or in this case, EKS itself.

Every EKS cluster gets its own OIDC issuer: a public HTTPS endpoint, unique to that cluster, that serves the OIDC discovery document and the JWKS (the public keys needed to verify token signatures). That endpoint is part of the EKS control plane, hosted and managed by AWS, the same way kube-apiserver itself is, not a pod running on your nodes. The actual signing of ServiceAccount tokens happens inside kube-apiserver using a private key the control plane holds; the issuer endpoint is just where the corresponding public key gets published, so AWS STS can fetch it and verify signatures later without ever calling back into your cluster.

Registering that issuer with AWS IAM (step 1 below) is what lets IAM trust tokens signed by your cluster specifically; an OIDC provider registered in one AWS account only vouches for tokens from the one EKS cluster it points at, which is exactly why the trust policy's Federated principal has to reference this cluster's specific issuer URL, not some generic "any EKS cluster" identity.

Here's the flow, step by step:

When you set up IRSA, your EKS cluster's OIDC issuer gets registered with AWS IAM as a trusted identity provider. This is a one-time step per cluster.
You annotate a ServiceAccount with eks.amazonaws.com/role-arn: arn:aws:iam::...:role/my-role. An annotation here just means ordinary Kubernetes metadata, a key-value pair stored under metadata.annotations on the object, the same mechanism used for all sorts of unrelated things across the ecosystem. Unlike labels, which Kubernetes uses to select and group objects, annotations aren't used for selection; they're just data a controller or webhook can read to change its behavior. In this case:

yaml

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: checkout-service
     namespace: production
     annotations:
       eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/checkout-s3-reader

That specific eks.amazonaws.com/role-arn key is what the mutating webhook in the next step is watching for; its presence is the trigger, and its value is the role the webhook tells the pod to assume.

3. When a pod using that ServiceAccount starts, a mutating webhook (installed as part of the EKS setup) injects a JWT, a signed token proving "this pod belongs to ServiceAccount X in namespace Y, in this specific cluster", into the pod as a file, along with environment variables (AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE) telling the AWS SDK where to find it.

A mutating webhook is part of Kubernetes' admission control pipeline, the sequence of checks every API request passes through between "the request is authenticated" and "the object actually gets written to etcd". Here's the full sequence for something like kubectl apply -f pod.yaml:

Authentication: who is making this request?
Authorization: RBAC check: is this identity allowed to do this?
Mutating admission webhooks: a chance for registered external services to modify the object before it's persisted.
Object schema validation: is the (possibly now-modified) object well-formed?
Validating admission webhooks: a chance for external services to approve or reject the final object, no more modifications allowed.
Persisted to etcd: the object is now real.

A mutating webhook is registered via a MutatingWebhookConfiguration pointing at an HTTPS endpoint, usually a small service running as its own Deployment in the cluster. When a matching request comes in (here, Pod CREATE), kube-apiserver pauses, sends the object to that endpoint, and applies whatever JSON patch comes back before continuing.

For IRSA, this is the Amazon EKS Pod Identity Webhook (an IRSA-era name that predates EKS Pod Identity the newer mechanism, which is a frequent source of confusion). It looks up the pod's spec.serviceAccountName, checks that ServiceAccount for the eks.amazonaws.com/role-arn annotation from step 2, and if present, patches in the projected token volume and the two environment variables above. None of this appears anywhere in the pod spec as written, it's injected transparently at admission time, which is exactly why IRSA "just works" once the annotation is set, with nothing else to configure on the pod itself.

4. When your Go code makes its first AWS API call, the SDK automatically reads that token and calls sts:AssumeRoleWithWebIdentity, presenting the JWT directly to AWS Security Token Service.

5. STS verifies the JWT's signature against the cluster's registered OIDC provider, checks the IAM role's trust policy for a matching condition (the specific sub claim, system:serviceaccount:<namespace>:<serviceaccount>), and if everything lines up, hands back temporary credentials.

The important thing to notice: the pod talks to AWS STS directly. No extra agent runs in the cluster to make this work, the SDK, the token file, and STS are the whole path. That also means the trust policy is where all the real security logic lives: it's the only thing standing between "I have a valid JWT" and "I get to assume this specific role."

Same five steps as above, made concrete: a pod running as the checkout-service ServiceAccount wants to read an object from S3. Its AWS SDK component presents the JWT sitting in its JWT Token file to AWS STS (1), STS checks that token and the requested role against AWS IAM (3), IAM confirms the trust relationship checks out against the registered OIDC identity provider and the role's attached policies (3–4), STS hands back temporary credentials (5), and only then does the pod call S3 directly (6), S3 itself never sees the JWT at all, only the temporary credentials STS issued.

Pod Identity: an in-cluster agent brokers the exchange

EKS Pod Identity, AWS's newer mechanism, moves that broker inside the cluster instead. Different flow:

Instead of an OIDC trust relationship, you create an association, a record, stored in AWS via the EKS API, saying "ServiceAccount X in namespace Y on cluster Z maps to IAM role R." No OIDC provider, no cluster-specific string embedded in an IAM trust policy.
A DaemonSet called the EKS Pod Identity Agent runs on every node, alongside your workloads.
The same kind of mutating webhook sets environment variables on the pod, but this time pointing the AWS SDK at the local agent (AWS_CONTAINER_CREDENTIALS_FULL_URI, a standard mechanism the SDK already supports for container environments, the same one ECS tasks use), not at AWS directly.
When your Go code makes its first AWS call, the SDK requests credentials from that local endpoint instead of STS. The Pod Identity Agent receives that request, looks up the association for this pod's namespace/ServiceAccount, and calls a dedicated API (eks-auth:AssumeRoleForPodIdentity) on the pod's behalf to get temporary credentials, then hands them back to the SDK.

It's worth being precise about where STS actually sits in this chain, since it's easy to assume Pod Identity skips STS entirely, it doesn't, it just moves further away from the pod. The pod's SDK never talks to STS or IAM directly; it only ever talks to the local agent.

Credentials flow back up the same four steps in reverse: STS to the EKS Auth API, to the agent, to the pod's SDK, so the diagram only needs to show one direction to be complete. The practical difference from IRSA's diagram earlier: there, the pod called STS itself, in one hop. Here, there are two extra hops and a different AWS API in the middle, which is exactly why Pod Identity failures can look different from IRSA failures even though both ultimately end at STS.

The trust policy here looks completely different too: instead of trusting an OIDC federated principal, it trusts the AWS service pods.eks.amazonaws.com directly. The association, which pod maps to which role, lives entirely in AWS, not encoded into the trust policy's conditions at all.

The EKS Auth API

The agent calls eks-auth:AssumeRoleForPodIdentity, and it's worth being precise about what that actually is, too: Amazon EKS Auth is its own standalone AWS service, with its own service ID (eks-auth), its own regional endpoint, and its own SDK client in every language, separate from both the general eks API and from iam/sts. It's part of the broader EKS product family, but it isn't the Kubernetes control plane (kube-apiserver, etcd, and friends, Kubernetes-specific infrastructure) and it isn't IAM's API either (IAM is global, not regional, with its own iam:* action namespace). Architecturally, EKS Auth sits as a peer to IAM and STS, not a part of either, a purpose-built bridge service that receives the pod's own projected token (audience pods.eks.amazonaws.com, different from IRSA's sts.amazonaws.com token), checks whether a Pod Identity association exists for this namespace/ServiceAccount, and whether the target role's trust policy allows it. Only then, internally, an implementation detail your code and the agent never see directly, does it call STS to actually mint the credentials, and hand them back down the chain.

One practical consequence of EKS Auth being its own regional service: a private EKS cluster with no outbound internet access needs its own VPC endpoint specifically for eks-auth, distinct from the endpoints for eks, sts, and everything else. It's newer and less well-known than the others, which makes it an easy one to miss when locking down a private cluster's networking, Pod Identity will simply fail to issue credentials with no obvious explanation until that endpoint exists.

Why this distinction matters for diagnosis

This is exactly why diagnose_workload_identity has to check two completely different places rather than one: an IRSA misconfiguration is almost always a trust policy problem (wrong OIDC provider, wrong sub condition, wrong namespace/ServiceAccount string), diagnosable by reading the role. A Pod Identity misconfiguration can be an association problem, no association exists at all for this ServiceAccount, something no amount of staring at an IAM role will ever reveal, since the role's trust policy for Pod Identity looks identical regardless of which ServiceAccounts happen to be associated with it. You have to ask AWS's EKS API directly, which is exactly what FindPodIdentityRole does.

And since both can be configured on the same ServiceAccount at once, very common mid-migration, and Pod Identity silently wins when both are present, a pod can have a perfectly correct-looking IRSA setup that's doing nothing at all, because an association nobody remembers creating is quietly overriding it. That's not a rare edge case to handle defensively; it's routine enough right now, with teams actively migrating from IRSA to Pod Identity, that it deserves to be the first thing the tool checks.

Summary: one rule that matters most

A pod gets AWS credentials one of two ways on EKS: IRSA (OIDC-based, checked via the ServiceAccount annotation and the IAM role's trust policy) or EKS Pod Identity (association-based, checked via the EKS API). Both can be configured at once during a migration, and when they are, Pod Identity wins, the mutating webhook prefers it whenever both are present. A role that looks unused from the Kubernetes side might still be the one actually granting access, or the one silently being ignored, and that precedence rule is the first thing any diagnostic tooling built on top of this needs to check.

Three things worth carrying forward:

A ServiceAccount is Kubernetes's own native identity concept, and RBAC governs it only inside the cluster, neither has any authority once a pod starts talking to AWS instead of kube-apiserver. IRSA and Pod Identity both extend that same ServiceAccount identity outward to AWS, but through completely different mechanisms: IRSA is a direct, OIDC-verified call from the pod to STS; Pod Identity routes through an in-cluster agent and an AWS-side association that has no visible trace on the Kubernetes side at all. Diagnosing a failure means checking two unrelated surfaces, an IAM role's trust policy for IRSA, and the EKS API's association records for Pod Identity, because nothing about looking at one tells you anything about the other.

Part 4 builds the actual diagnostic tool on top of everything explained here, a new AWS-facing Go client, a diagnose_workload_identity MCP tool that checks both mechanisms and applies the precedence rule above, and an honest look at what it means for this server to start needing AWS IAM permissions, not just Kubernetes RBAC.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

LeetCode 151: Reverse Words in a String in Go

Fer Rios — Wed, 19 Aug 2026 15:48:17 +0000

The problem that looks simple until the whitespace shows up

You open LeetCode 151 — Reverse Words in a String expecting a variation on reversing a string, and mostly, that's right. Given "the sky is blue", you need "blue is sky the", the words reverse order, not the letters inside them.

Then you notice the real complication buried in the constraints: the input can have leading spaces, trailing spaces, and multiple spaces between words (" the sky is blue "), and none of that garbage is allowed to survive into your output.

So this problem is really two problems stacked together: reversing word order, and cleaning up whitespace. This post walks through two ways to solve it in Go, and along the way clears up a common misconception about what "O(1) extra space" actually means once your language doesn't let you mutate strings directly.

Approach 1: split, reverse, join

The simplest mental model: let the standard library handle the messy whitespace work, then reuse the same two-pointer swap pattern from reversing vowels, one level up, swapping whole words instead of characters.

Step 1: Split, with whitespace handled for free

words := strings.Fields(s)

strings.Fields splits on any run of whitespace and silently discards empty pieces. That one call solves leading spaces, trailing spaces, and multiple spaces between words, all at once:

strings.Fields("  the sky  is blue  ")
// → []string{"the", "sky", "is", "blue"}

Compare that to strings.Split(s, " "), which would instead give you ["", "", "the", "sky", "", "is", "blue", "", ""]; every extra space becomes an empty string you'd have to filter out yourself. Picking the right stdlib function does half the problem for you here.

Step 2: Reverse the slice of words

Same collision two-pointer pattern as before, left and right start at opposite ends and swap toward the middle. The only difference from reversing vowels is that every element gets swapped unconditionally; there's no "is this a vowel" check gating the swap:

left, right := 0, len(words)-1
for left < right {
    words[left], words[right] = words[right], words[left]
    left++
    right--
}

Step 3: Join back with single spaces

strings.Join(words, " ")

Full Solution

func reverseWords(s string) string {
    words := strings.Fields(s)

    left, right := 0, len(words)-1
    for left < right {
        words[left], words[right] = words[right], words[left]
        left++
        right--
    }

    return strings.Join(words, " ")
}

Trace

" the sky is blue "

strings.Fields → ["the", "sky", "is", "blue"]
Reverse → ["blue", "is", "sky", "the"]
strings.Join → "blue is sky the" ✅

Three lines of real logic, and every whitespace edge case handled without writing a single line for it. The cost: strings.Fields allocates a slice of strings (one per word), and strings.Join allocates another buffer for the final result. Fine for most purposes, but not the leanest possible solution, which is exactly what Approach 2 addresses.

Approach 2: reverse-then-reverse, in place

This is the classic interview follow-up answer: reverse the entire string first, then reverse each word individually back to normal. It reuses an idea worth remembering, since it reappears in other problems too: reverse the whole thing, then reverse the parts.

Why "reverse, then reverse" works

Take "the sky". Reverse it completely, character by character:

"the sky" → "yks eht"

The word order is now correct (sky's letters come first, then the's), but each word's internal letter order got scrambled as a side effect of reversing everything at once. So a second pass fixes just that: scan for each word's boundaries and reverse only that sub-range, back to normal:

"yks eht" → "sky eht" → "sky the"

Result: "sky the"; word order fixed by the full reversal, letters fixed locally afterward, without disturbing the word order that's already correct.

The part that's actually new: whitespace compaction, in place

Since we can't call strings.Fields here without giving up the "in place" goal, whitespace has to be compacted manually, using a pattern we haven't needed until now: the read/write pointer technique (two pointers moving in the same direction, at different speeds, as opposed to the collision pattern, which moves them toward each other).

Think of it like re-typing a messy sentence onto the same sheet of paper, one pen, no extra paper. You read through with one pointer; every time you find something worth keeping, you write it at the next available spot with a second pointer, which is always at or behind where you're reading, so you never overwrite something you still need.

write := 0
read := 0
for read < n {
    for read < n && b[read] == ' ' {
        read++ // skip spaces entirely
    }
    if read == n {
        break // trailing spaces exhausted
    }
    if write != 0 {
        b[write] = ' ' // exactly one separator, except before the first word
        write++
    }
    for read < n && b[read] != ' ' {
        b[write] = b[read] // copy the word
        write++
        read++
    }
}
b = b[:write]
n = write

Why this collapses multiple spaces into one: the inner "skip spaces" loop consumes an entire run of spaces before write ever moves; one space or ten, doesn't matter. Why it drops leading/trailing spaces: the separator is only written if write != 0, so nothing goes in before the first word, and once the last word is copied, there's no new word left to trigger writing a trailing separator.

Reverse the whole compacted string

reverse(b, 0, n-1)

using a small unconditional helper:

func reverse(b []byte, left, right int) {
    for left < right {
        b[left], b[right] = b[right], b[left]
        left++
        right--
    }
}

Reverse each word back to normal

start := 0
for i := 0; i <= n; i++ {
    if i == n || b[i] == ' ' {
        reverse(b, start, i-1)
        start = i + 1
    }
}

The loop runs to i <= n, one past the last index, specifically so the final word, which has no trailing space to mark its end, still gets caught and reversed.

Full Solution

func reverseWords(s string) string {
    b := []byte(s)
    n := len(b)

    write := 0
    read := 0
    for read < n {
        for read < n && b[read] == ' ' {
            read++
        }
        if read == n {
            break
        }
        if write != 0 {
            b[write] = ' '
            write++
        }
        for read < n && b[read] != ' ' {
            b[write] = b[read]
            write++
            read++
        }
    }
    b = b[:write]
    n = write

    reverse(b, 0, n-1)

    start := 0
    for i := 0; i <= n; i++ {
        if i == n || b[i] == ' ' {
            reverse(b, start, i-1)
            start = i + 1
        }
    }

    return string(b)
}

func reverse(b []byte, left, right int) {
    for left < right {
        b[left], b[right] = b[right], b[left]
        left++
        right--
    }
}

Three passes, three applications of the same two-pointer family: read/write for compaction, collision for the full reversal, collision again (on sub-ranges) for the per-word fix.

What "extra space" (auxiliary space) means

When people analyze space complexity, they usually split memory usage into two buckets:

Input space: memory used to store the input itself (the string you were given). This doesn't count against you; you didn't choose to receive a big input.
Extra / auxiliary space: any additional memory your algorithm allocates beyond the input, in order to compute the answer.

"O(1) extra space" means: no matter how large the input n gets, the amount of additional memory your algorithm needs stays constant; a fixed handful of variables (a few ints for pointers/indices, maybe one temp variable), and nothing that grows with n.

What "truly O(1)" looks like, concretely

Take the classic version of this problem in C or C++, where a string is literally a mutable array of characters (char[] or char*) sitting in memory that you already own.

In that world, the reverse-then-reverse-words algorithm works like this:

You're handed the character array directly; this is your input space, already allocated.
You do the whitespace compaction, the full reversal, and the per-word reversal directly inside that same array, using only a few integer index variables (read, write, left, right, start).
When you're done, that same array, now holding the answer, is returned or printed; no new array was ever created.

The only extra memory used across the entire algorithm is a small, fixed number of integers; regardless of whether the string is 10 characters or 10 million. That's genuinely O(1) extra space: the auxiliary memory footprint never grows with n.

So which one is actually "O(1) extra space"? Neither, really, and here's why that matters

This is worth being precise about, because "O(1) space" gets thrown around loosely.

"Extra" or "auxiliary" space means memory used beyond the input itself; a fixed handful of index variables counts as O(1) extra space; anything that grows with the input size n doesn't.

In C or C++, a string is a mutable char[] you already own. The reverse-then-reverse algorithm edits that array directly and hands the same array back; no new array is ever created. The only extra memory used is a few integers (read, write, left, right), regardless of whether the string is 10 characters or 10 million. That's genuinely O(1) extra space.

Go doesn't let you do that, because string is immutable by design (this exists so that substrings can safely share underlying memory without one mutation corrupting another). So:

b := []byte(s)      // copies the input, O(n) new memory, before any logic runs
// ... in-place work on b ...
return string(b)    // copies again, another O(n) new memory

Both of those copies are proportional to n. So strictly speaking, Approach 2's Go implementation is O(n) extra space overall, not O(1), the in-place logic between those two conversions is O(1) extra beyond the []byte copy, but the copy itself isn't optional in Go.

Then what's actually different between the two approaches?

Since neither is truly O(1) in Go, the real distinction is in how much intermediate garbage gets created along the way, constant factors that big-O notation deliberately hides but that matter in practice:

	Approach 1	Approach 2
Time	O(n)	O(n)
Space (Go-accurate)	O(n)	O(n)
Allocations	One string per word (from `strings.Fields`) + one buffer for `strings.Join`, roughly O(number of words) + 1	One `[]byte` copy of the input, mutated in place, one final `string()` copy, a constant 2, regardless of word count
Practical memory pressure	Higher, many small allocations, more garbage-collector work	Lower, few allocations, less GC pressure

So when this problem's classic follow-up asks "can you do it with O(1) extra space," the honest answer in Go is: not truly, since string immutability forces at least one input-sized copy in and one out;

but Approach 2 is still the intended and meaningfully leaner answer, because it avoids all the additional per-word allocations Approach 1 introduces on top of that unavoidable baseline.

Summary

Approach 1 (strings.Fields → reverse slice → strings.Join) is the simplest correct solution: whitespace cleanup comes free from the standard library, and the reversal is the same collision two-pointer pattern from earlier problems, just applied to whole words.
Approach 2 (compact → reverse all → reverse each word) is the leaner, interview-favorite solution, introducing the read/write pointer pattern for in-place compaction alongside the familiar collision pattern for reversal; two flavors of "two pointers," composed across three passes.
"O(1) extra space" is baseline-dependent. In C/C++, it means genuinely no new allocations. In Go, string immutability forces an input-sized copy in and out regardless of approach, so the honest framing is: Approach 2 minimizes allocations beyond that unavoidable baseline, rather than achieving true O(1).

That last point is the one worth carrying forward to every future string problem in Go: read space-complexity claims with the question "O(1) relative to what baseline?" in mind, especially in a language where the built-in string type won't let you mutate in place at all.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Build an MCP Server in Go (Part 2): Building the MCP tool layer

Fer Rios — Fri, 14 Aug 2026 14:15:57 +0000

Part 1 designs the KubeClient interface this post builds on. If you haven't read it, the short version: a read-only, domain-composed Kubernetes client, pods, events, workloads, nodes, network, config, storage, metrics, with every unbounded query capped before an agent ever touches it.

With that client in hand, this post does the rest: wrap it as MCP tools, wire it to an agent, and run the whole thing end-to-end against a real cluster.

What MCP actually is

Strip away the hype, and MCP is a small, well-defined protocol: a server exposes a list of tools, each with a name, a description, and a JSON Schema describing its input. A client, usually an LLM-powered agent, asks the server "what tools do you have," decides which one is relevant to the user's question, and calls it with structured arguments. The server runs the tool and returns a structured result. Repeat, as many times as the agent needs, until it has enough information to answer.

Agent (Claude, etc.)
      │
      │  "list your tools"          →  tools/list
      │  ← [list_pods, get_pod_logs, describe_pod, ...]
      │
      │  "call get_pod_logs"        →  tools/call
      │  ← { "logs": "panic: ..." }
      │
      ▼
Your MCP server
      │
      │  client-go
      ▼
Kubernetes API server

That's the whole mental model. The interesting engineering problem wasn't the protocol, it was the KubeClient interface from part 1. What's left is wrapping it correctly.

Two ways tools and calls actually travel: stdio vs. HTTP

The diagram above shows what gets exchanged. It says nothing about how those messages physically move between client and server, that's a separate choice, called the transport, and MCP defines two of them.

Stdio transport is what this post builds. The client, Claude Desktop, VS Code, spawns your compiled binary as a local child process and talks to it over that process's stdin and stdout. Every tools/list and tools/call message is a line of JSON written to the child's stdin; every response is a line of JSON written back to stdout. There's no network involved, no port to open, no auth to configure: the trust boundary is just "whoever can run this binary already has access to whatever it can reach," which for this server means your kubeconfig. This is exactly why main.go calls server.Run(ctx, &mcp.StdioTransport{}), the server's entire life cycle is tied to one client process spawning it, talking to it, and eventually killing it.

Streamable HTTP transport is the other option, and it's a genuinely different deployment shape, not just a different function call. The server runs as a long-lived process, think a pod in a cluster, listening on a port, and any number of clients connect to it over the network instead of spawning it. That unlocks things stdio structurally can't do: multiple agents sharing one server, a server that outlives any single client's session, a server nowhere near the machine that's using it. It also introduces problems stdio never has to solve: authenticating requests from strangers on the network, and keeping session state consistent when a client's calls might land on different backend replicas instead of the one process that remembers the conversation so far.

Neither transport is strictly better, they fit different situations. Stdio is the right choice for exactly what this post is: a tool one engineer runs locally, pointed at whatever cluster their kubeconfig currently targets. HTTP is what you'd reach for to run this as shared infrastructure other people's agents connect to, and that jump, with the session handling and auth it requires, is real enough to be its own post rather than a paragraph here.

Project structure

go-k8s-mcp-server/
├── go.mod
├── cmd/
│   └── k8s-mcp-server/
│       └── main.go             ← entry point — wires client, tools, transport
└── internal/
    ├── kubernetes/              ← the KubeClient from part 1, unchanged
    │   ├── client.go
    │   ├── pods.go
    │   ├── workloads.go
    │   ├── nodes.go
    │   ├── events.go
    │   ├── network.go
    │   ├── ingress.go
    │   ├── gateway.go
    │   ├── config.go
    │   ├── storage.go
    │   └── metrics.go
    └── tools/                   ← new in this post
        ├── list_pods.go
        ├── pod_logs.go
        ├── describe_pod.go
        ├── get_events.go
        ├── check_endpoints.go
        ├── describe_ingress.go
        ├── describe_httproute.go
        ├── list_configmap_keys.go
        └── pod_metrics.go

internal/kubernetes/ is exactly what part 1 built, nothing changes there. Everything new in this post lives in cmd/k8s-mcp-server/ and internal/tools/, following the same cmd/ + internal/ convention from Go packages and modules explained.

The tool layer

KubeClient answers "what can this program do to a cluster." The tool layer answers a different question: "what is the agent allowed to ask for, and in what shape." These are not the same interface. A KubeClient method might take a LogOptions struct; an MCP tool needs a JSON Schema an LLM can read and a handler that translates one into the other.

We'll use the official Go SDK, github.com/modelcontextprotocol/go-sdk/mcp. It infers JSON Schema directly from Go struct tags and gives you typed handlers. func(ctx, req, In) (*mcp.CallToolResult, Out, error), so there's no manual schema-writing and no untyped map[string]any argument parsing.

go get github.com/modelcontextprotocol/go-sdk

Each tool gets its own file, same "one reason to change" rule as the reporter package in the client-go post: a new tool is a new file, not a new branch in a shared switch statement.

Anatomy of a tool file

Every tool in this project follows the same four-part shape. Before looking at a real one, it's worth naming each piece, since the same pattern repeats seven times below and it's easier to recognize once you know what you're looking at:

*mcp.Tool: the tool's identity as far as the agent is concerned: a Name the agent calls it by ("list_pods"), and a Description the agent reads to decide when to call it. This is metadata only, it holds no logic and touches no Kubernetes code.
In (the input struct), what the agent is allowed to send as arguments, expressed as a normal Go struct with json and jsonschema tags. The SDK turns this struct's shape into the JSON Schema the agent actually sees; you never write schema by hand.
The handler function: the actual logic, with the signature func(ctx context.Context, req *mcp.CallToolRequest, in In) (*mcp.CallToolResult, Out, error). This is where the input gets validated, KubeClient gets called, and the result gets shaped into Out. It's an ordinary Go function; nothing about it is MCP-specific beyond its signature.
Out (the output struct): what the agent receives back, again a plain Go struct. Returning a typed struct instead of a preformatted string matters: the agent gets structured JSON it can reason over programmatically (checking a Phase field, summing a Restarts count), rather than a paragraph of text it has to parse itself.
The New*Tool constructor: a small function that builds and returns the (*mcp.Tool, handler) pair together, closing over a KubeClient so the handler has something to call. This is what makes each tool independently testable: pass in a fake KubeClient, call the returned handler directly, assert on Out, no MCP machinery involved at all.

(*mcp.Tool, handler) is the pair every New*Tool function returns, and the pair mcp.AddTool expects at registration time in server.go, worth keeping that shape in mind, since it's also exactly where the multi-value spread gotcha from the "Common gotchas" section comes from.

`list_pods`

The pieces above, in a real file: tool below is the *mcp.Tool identity, ListPodsInput is In, the closure returned second is the handler, and ListPodsOutput is Out.

// internal/tools/list_pods.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

// ListPodsInput is inferred into JSON Schema from these struct tags,
// the agent sees "namespace" as an optional string with this description,
// nothing more to write by hand.
type ListPodsInput struct {
    Namespace string `json:"namespace" jsonschema:"Kubernetes namespace to list pods from. Leave empty to list pods across all namespaces."`
}

type PodSummary struct {
    Namespace string `json:"namespace"`
    Name      string `json:"name"`
    Phase     string `json:"phase"`
    Ready     string `json:"ready"`
    Restarts  int32  `json:"restarts"`
}

type ListPodsOutput struct {
    Pods []PodSummary `json:"pods"`
}

// NewListPodsTool returns the tool definition and its handler, bound to
// a concrete KubeClient. Keeping the constructor take the client as a
// parameter (rather than a package-level global) is what keeps this
// testable with a fake client.
func NewListPodsTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, ListPodsInput) (*mcp.CallToolResult, ListPodsOutput, error)) {
    tool := &mcp.Tool{
        Name:        "list_pods",
        Description: "List pods in a namespace, or across all namespaces, with phase, readiness, and restart count.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in ListPodsInput) (*mcp.CallToolResult, ListPodsOutput, error) {
        pods, err := kube.ListPods(ctx, in.Namespace)
        if err != nil {
            return nil, ListPodsOutput{}, fmt.Errorf("list_pods: %w", err)
        }

        out := ListPodsOutput{Pods: make([]PodSummary, 0, len(pods.Items))}
        for _, pod := range pods.Items {
            ready, total := 0, len(pod.Status.ContainerStatuses)
            var restarts int32
            for _, cs := range pod.Status.ContainerStatuses {
                if cs.Ready {
                    ready++
                }
                restarts += cs.RestartCount
            }
            out.Pods = append(out.Pods, PodSummary{
                Namespace: pod.Namespace,
                Name:      pod.Name,
                Phase:     string(pod.Status.Phase),
                Ready:     fmt.Sprintf("%d/%d", ready, total),
                Restarts:  restarts,
            })
        }
        return nil, out, nil
    }

    return tool, handler
}

Two things worth calling out. First, the tool returns a typed ListPodsOutput, not a formatted string, the SDK turns that into structured JSON the agent can reason over, the same way it would parse an API response. Second, the description on the tool itself matters as much as the code: it's the only thing the agent has to decide when to call this versus another tool. Vague descriptions produce agents that call the wrong tool, or the right tool with the wrong arguments, worth testing with real prompts once you've built a few tools, not just checking that the code compiles.

`get_pod_logs`

// internal/tools/pod_logs.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

type GetPodLogsInput struct {
    Namespace string `json:"namespace" jsonschema:"Kubernetes namespace of the pod."`
    Name      string `json:"name" jsonschema:"Name of the pod."`
    Container string `json:"container" jsonschema:"Container name. Leave empty for the pod's first container."`
    Previous  bool   `json:"previous" jsonschema:"If true, fetch logs from the previous terminated instance of the container. Use this for CrashLoopBackOff pods — the current instance often has no logs yet."`
    TailLines int64  `json:"tailLines" jsonschema:"Number of lines to return from the end of the log, capped at 500. Defaults to 200."`
}

type GetPodLogsOutput struct {
    Logs string `json:"logs"`
}

const maxLogTailLines = 500

func NewGetPodLogsTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, GetPodLogsInput) (*mcp.CallToolResult, GetPodLogsOutput, error)) {
    tool := &mcp.Tool{
        Name:        "get_pod_logs",
        Description: "Fetch recent logs for a pod's container. Set previous=true to diagnose CrashLoopBackOff.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in GetPodLogsInput) (*mcp.CallToolResult, GetPodLogsOutput, error) {
        tail := in.TailLines
        if tail <= 0 || tail > maxLogTailLines {
            tail = maxLogTailLines
        }

        logs, err := kube.GetPodLogs(ctx, in.Namespace, in.Name, kubernetes.LogOptions{
            Container: in.Container,
            Previous:  in.Previous,
            TailLines: tail,
        })
        if err != nil {
            return nil, GetPodLogsOutput{}, fmt.Errorf("get_pod_logs: %w", err)
        }
        return nil, GetPodLogsOutput{Logs: logs}, nil
    }

    return tool, handler
}

Notice the cap is enforced again here, on top of the default already living in LogOptions. That's not redundant, the schema description tells the agent "capped at 500," but nothing stops a misbehaving or adversarial caller from sending tailLines: 50000 anyway. The tool handler is the last line of defense before the request reaches the cluster, so it re-validates rather than trusting that the caller respected the description.

`describe_pod`

This one composes three KubeClient calls into a single tool; deliberately, it mirrors what kubectl describe pod does under the hood, giving the agent one call instead of three for a question it will ask constantly.

// internal/tools/describe_pod.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"
    corev1 "k8s.io/api/core/v1"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

type DescribePodInput struct {
    Namespace string `json:"namespace" jsonschema:"Kubernetes namespace of the pod."`
    Name      string `json:"name" jsonschema:"Name of the pod."`
}

type ContainerState struct {
    Name     string `json:"name"`
    State    string `json:"state"`    // Waiting, Running, Terminated
    Reason   string `json:"reason"`   // CrashLoopBackOff, OOMKilled, etc.
    Restarts int32  `json:"restarts"`
}

type DescribePodOutput struct {
    Phase      string           `json:"phase"`
    Containers []ContainerState `json:"containers"`
    Events     []string         `json:"recentEvents"`
}

func NewDescribePodTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, DescribePodInput) (*mcp.CallToolResult, DescribePodOutput, error)) {
    tool := &mcp.Tool{
        Name:        "describe_pod",
        Description: "Get a pod's phase, per-container state and restart reasons, and its most recent events — the fastest way to understand why a pod is unhealthy.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in DescribePodInput) (*mcp.CallToolResult, DescribePodOutput, error) {
        pod, err := kube.GetPod(ctx, in.Namespace, in.Name)
        if err != nil {
            return nil, DescribePodOutput{}, fmt.Errorf("describe_pod: %w", err)
        }

        events, err := kube.GetEvents(ctx, in.Namespace, in.Name)
        if err != nil {
            return nil, DescribePodOutput{}, fmt.Errorf("describe_pod: %w", err)
        }

        out := DescribePodOutput{Phase: string(pod.Status.Phase)}
        for _, cs := range pod.Status.ContainerStatuses {
            state, reason := containerState(cs)
            out.Containers = append(out.Containers, ContainerState{
                Name: cs.Name, State: state, Reason: reason, Restarts: cs.RestartCount,
            })
        }
        for _, e := range events {
            out.Events = append(out.Events, fmt.Sprintf("[%s] %s", e.Reason, e.Message))
        }

        return nil, out, nil
    }

    return tool, handler
}

func containerState(cs corev1.ContainerStatus) (state, reason string) {
    switch {
    case cs.State.Waiting != nil:
        return "Waiting", cs.State.Waiting.Reason
    case cs.State.Terminated != nil:
        return "Terminated", cs.State.Terminated.Reason
    default:
        return "Running", ""
    }
}

If you've read the client-go post, containerState will look familiar, it's the same waiting/terminated distinction inspectPod made in the cluster health reporter, because it's the same underlying problem: a pod's Phase alone doesn't tell you the truth; you have to read ContainerStatuses.

`get_events`

Standalone from describe_pod because an agent sometimes wants events for a whole namespace, not one pod, a burst of FailedScheduling events across several pods is a cluster-capacity story, not a single-pod story, and describe_pod can't surface that.

// internal/tools/get_events.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

type GetEventsInput struct {
    Namespace          string `json:"namespace" jsonschema:"Kubernetes namespace to fetch events from."`
    InvolvedObjectName string `json:"involvedObjectName" jsonschema:"Optional. Filter events to those involving this object name, e.g. a Pod or Deployment. Leave empty for all events in the namespace."`
}

type EventSummary struct {
    Type    string `json:"type"`   // Normal or Warning
    Reason  string `json:"reason"` // FailedScheduling, BackOff, Pulled, etc.
    Object  string `json:"object"`
    Message string `json:"message"`
    Count   int32  `json:"count"`
}

type GetEventsOutput struct {
    Events []EventSummary `json:"events"`
}

const maxEventsReturned = 50

func NewGetEventsTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, GetEventsInput) (*mcp.CallToolResult, GetEventsOutput, error)) {
    tool := &mcp.Tool{
        Name:        "get_events",
        Description: "List recent Kubernetes events in a namespace, optionally filtered to a specific object. A burst of Warning events across multiple objects usually points to a cluster-level problem, not a single-resource one.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in GetEventsInput) (*mcp.CallToolResult, GetEventsOutput, error) {
        events, err := kube.GetEvents(ctx, in.Namespace, in.InvolvedObjectName)
        if err != nil {
            return nil, GetEventsOutput{}, fmt.Errorf("get_events: %w", err)
        }

        // Cap the response the same way get_pod_logs caps its tail,
        // a noisy namespace can return thousands of events, and the
        // agent needs the most relevant slice, not all of it.
        if len(events) > maxEventsReturned {
            events = events[len(events)-maxEventsReturned:]
        }

        out := GetEventsOutput{Events: make([]EventSummary, 0, len(events))}
        for _, e := range events {
            out.Events = append(out.Events, EventSummary{
                Type:    e.Type,
                Reason:  e.Reason,
                Object:  e.InvolvedObject.Name,
                Message: e.Message,
                Count:   e.Count,
            })
        }
        return nil, out, nil
    }

    return tool, handler
}

`check_endpoints`

This is the tool that answers "the Service exists, so why can't anything reach it", a question list_pods and describe_pod genuinely can't answer, since a healthy-looking pod can still be missing from a Service's endpoints entirely.

// internal/tools/check_endpoints.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

type CheckEndpointsInput struct {
    Namespace   string `json:"namespace" jsonschema:"Kubernetes namespace of the service."`
    ServiceName string `json:"serviceName" jsonschema:"Name of the service to check."`
}

type CheckEndpointsOutput struct {
    ServiceSelector map[string]string `json:"serviceSelector"`
    ReadyAddresses  int               `json:"readyAddresses"`
    NotReady        int               `json:"notReadyAddresses"`
    Healthy         bool              `json:"healthy"`
}

func NewCheckEndpointsTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, CheckEndpointsInput) (*mcp.CallToolResult, CheckEndpointsOutput, error)) {
    tool := &mcp.Tool{
        Name:        "check_endpoints",
        Description: "Check whether a Service has healthy endpoints backing it. Zero ready addresses despite matching pods usually means a label selector mismatch or a failing readiness probe.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in CheckEndpointsInput) (*mcp.CallToolResult, CheckEndpointsOutput, error) {
        svc, err := kube.GetService(ctx, in.Namespace, in.ServiceName)
        if err != nil {
            return nil, CheckEndpointsOutput{}, fmt.Errorf("check_endpoints: %w", err)
        }

        eps, err := kube.GetEndpoints(ctx, in.Namespace, in.ServiceName)
        if err != nil {
            return nil, CheckEndpointsOutput{}, fmt.Errorf("check_endpoints: %w", err)
        }

        ready, notReady := 0, 0
        for _, subset := range eps.Subsets {
            ready += len(subset.Addresses)
            notReady += len(subset.NotReadyAddresses)
        }

        return nil, CheckEndpointsOutput{
            ServiceSelector: svc.Spec.Selector,
            ReadyAddresses:  ready,
            NotReady:        notReady,
            Healthy:         ready > 0,
        }, nil
    }

    return tool, handler
}

Returning ServiceSelector alongside the counts is deliberate: when ReadyAddresses is zero, the very next thing an agent needs is the selector to cross-reference against pod labels via list_pods, handing it back here saves a tool call the agent would otherwise have to make anyway.

`describe_ingress`

Composes GetIngress, ListIngressClasses, and GetEvents, the same three-source pattern describe_pod used, applied to "is the ALB even provisioned" instead of "is the pod healthy."

// internal/tools/describe_ingress.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

type DescribeIngressInput struct {
    Namespace string `json:"namespace" jsonschema:"Kubernetes namespace of the Ingress."`
    Name      string `json:"name" jsonschema:"Name of the Ingress."`
}

type DescribeIngressOutput struct {
    IngressClassName  string   `json:"ingressClassName"`
    IngressClassFound bool     `json:"ingressClassFound"`
    LoadBalancerHost  string   `json:"loadBalancerHost"` // empty if not yet provisioned
    RecentEvents      []string `json:"recentEvents"`
}

func NewDescribeIngressTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, DescribeIngressInput) (*mcp.CallToolResult, DescribeIngressOutput, error)) {
    tool := &mcp.Tool{
        Name:        "describe_ingress",
        Description: "Diagnose an Ingress on EKS: whether its ingressClassName resolves to a real IngressClass, whether the ALB has finished provisioning, and recent reconciliation events from the AWS Load Balancer Controller.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in DescribeIngressInput) (*mcp.CallToolResult, DescribeIngressOutput, error) {
        ing, err := kube.GetIngress(ctx, in.Namespace, in.Name)
        if err != nil {
            return nil, DescribeIngressOutput{}, fmt.Errorf("describe_ingress: %w", err)
        }

        out := DescribeIngressOutput{}
        if ing.Spec.IngressClassName != nil {
            out.IngressClassName = *ing.Spec.IngressClassName
        }
        if len(ing.Status.LoadBalancer.Ingress) > 0 {
            out.LoadBalancerHost = ing.Status.LoadBalancer.Ingress[0].Hostname
        }

        classes, err := kube.ListIngressClasses(ctx)
        if err != nil {
            return nil, DescribeIngressOutput{}, fmt.Errorf("describe_ingress: %w", err)
        }
        for _, ic := range classes.Items {
            if ic.Name == out.IngressClassName {
                out.IngressClassFound = true
                break
            }
        }

        events, err := kube.GetEvents(ctx, in.Namespace, in.Name)
        if err != nil {
            return nil, DescribeIngressOutput{}, fmt.Errorf("describe_ingress: %w", err)
        }
        for _, e := range events {
            out.RecentEvents = append(out.RecentEvents, fmt.Sprintf("[%s] %s", e.Reason, e.Message))
        }

        return nil, out, nil
    }

    return tool, handler
}

IngressClassFound deliberately gets computed here rather than left for the agent to figure out from a raw class list, an empty or mismatched ingressClassName with no error anywhere else in the object is exactly the kind of silent failure an agent won't reliably catch by eyeballing JSON, so the tool does the comparison itself and hands back a plain boolean.

`describe_httproute`

Gateway API's structured status.conditions make this tool simpler than describe_ingress in one respect, no cross-referencing an IngressClass list. but it has to check two objects instead of one, since a route's health depends on the Gateway underneath it.

// internal/tools/describe_httproute.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

type DescribeHTTPRouteInput struct {
    Namespace string `json:"namespace" jsonschema:"Kubernetes namespace of the HTTPRoute."`
    Name      string `json:"name" jsonschema:"Name of the HTTPRoute."`
    // GatewayName lets the tool also check the parent Gateway's status,
    // since a route can never work if its Gateway isn't Programmed,
    // optional because a route can have multiple parents.
    GatewayName string `json:"gatewayName" jsonschema:"Optional. Name of the parent Gateway to check alongside the route. Leave empty to skip the Gateway-level check."`
}

type ConditionSummary struct {
    Type    string `json:"type"`   // Accepted, ResolvedRefs, Programmed, etc.
    Status  string `json:"status"` // True, False, Unknown
    Reason  string `json:"reason"`
    Message string `json:"message"`
}

type DescribeHTTPRouteOutput struct {
    RouteConditions   []ConditionSummary `json:"routeConditions"`
    GatewayConditions []ConditionSummary `json:"gatewayConditions,omitempty"`
}

func NewDescribeHTTPRouteTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, DescribeHTTPRouteInput) (*mcp.CallToolResult, DescribeHTTPRouteOutput, error)) {
    tool := &mcp.Tool{
        Name:        "describe_httproute",
        Description: "Diagnose an HTTPRoute's Accepted/ResolvedRefs conditions, and optionally its parent Gateway's Accepted/Programmed conditions. Check the Gateway conditions first — a route cannot work if its Gateway isn't Programmed.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in DescribeHTTPRouteInput) (*mcp.CallToolResult, DescribeHTTPRouteOutput, error) {
        route, err := kube.GetHTTPRoute(ctx, in.Namespace, in.Name)
        if err != nil {
            return nil, DescribeHTTPRouteOutput{}, fmt.Errorf("describe_httproute: %w", err)
        }

        out := DescribeHTTPRouteOutput{}
        for _, parent := range route.Status.Parents {
            out.RouteConditions = append(out.RouteConditions, summarizeConditions(parent.Conditions)...)
        }

        if in.GatewayName != "" {
            gw, err := kube.GetGateway(ctx, in.Namespace, in.GatewayName)
            if err != nil {
                return nil, DescribeHTTPRouteOutput{}, fmt.Errorf("describe_httproute: %w", err)
            }
            out.GatewayConditions = summarizeConditions(gw.Status.Conditions)
        }

        return nil, out, nil
    }

    return tool, handler
}

func summarizeConditions(conditions []metav1.Condition) []ConditionSummary {
    summaries := make([]ConditionSummary, 0, len(conditions))
    for _, c := range conditions {
        summaries = append(summaries, ConditionSummary{
            Type:    c.Type,
            Status:  string(c.Status),
            Reason:  c.Reason,
            Message: c.Message,
        })
    }
    return summaries
}

summarizeConditions exists as its own small function because both route.Status.Parents[].Conditions and gw.Status.Conditions are the same []metav1.Condition shape; Gateway API standardizes on metav1.Condition throughout, which is exactly what lets one helper cover both without a type switch.

`list_configmap_keys`

The read-only, keys-only boundary from ConfigClient carries straight through to the tool layer, there's no value field anywhere in this output, by design.

// internal/tools/list_configmap_keys.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

type ListConfigMapKeysInput struct {
    Namespace string `json:"namespace" jsonschema:"Kubernetes namespace of the ConfigMap."`
    Name      string `json:"name" jsonschema:"Name of the ConfigMap."`
}

type ListConfigMapKeysOutput struct {
    Keys []string `json:"keys"`
}

func NewListConfigMapKeysTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, ListConfigMapKeysInput) (*mcp.CallToolResult, ListConfigMapKeysOutput, error)) {
    tool := &mcp.Tool{
        Name:        "list_configmap_keys",
        Description: "List the key names present in a ConfigMap. Returns keys only, never values — use this to check whether an expected config key exists.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in ListConfigMapKeysInput) (*mcp.CallToolResult, ListConfigMapKeysOutput, error) {
        keys, err := kube.GetConfigMapKeys(ctx, in.Namespace, in.Name)
        if err != nil {
            return nil, ListConfigMapKeysOutput{}, fmt.Errorf("list_configmap_keys: %w", err)
        }
        return nil, ListConfigMapKeysOutput{Keys: keys}, nil
    }

    return tool, handler
}

There's no list_secret_keys tool shown separately here, it's the identical shape calling kube.GetSecretKeys instead, so I'm not pasting a near-duplicate file. If you're building this yourself, add it exactly like this one.

`pod_metrics`

The one tool that depends on metrics-server actually being installed in the cluster, worth surfacing that as a real error, not a generic 404, since it's a common reason for this specific tool to fail in a fresh cluster.

// internal/tools/pod_metrics.go
package tools

import (
    "context"
    "fmt"

    "github.com/modelcontextprotocol/go-sdk/mcp"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
)

type GetPodMetricsInput struct {
    Namespace string `json:"namespace" jsonschema:"Kubernetes namespace of the pod."`
    Name      string `json:"name" jsonschema:"Name of the pod."`
}

type ContainerUsage struct {
    Name   string `json:"name"`
    CPU    string `json:"cpu"`    // e.g. "12m"
    Memory string `json:"memory"` // e.g. "84Mi"
}

type GetPodMetricsOutput struct {
    Containers []ContainerUsage `json:"containers"`
}

func NewGetPodMetricsTool(kube kubernetes.KubeClient) (*mcp.Tool, func(context.Context, *mcp.CallToolRequest, GetPodMetricsInput) (*mcp.CallToolResult, GetPodMetricsOutput, error)) {
    tool := &mcp.Tool{
        Name:        "pod_metrics",
        Description: "Get current CPU and memory usage per container in a pod, from metrics-server. Requires metrics-server to be installed in the cluster.",
    }

    handler := func(ctx context.Context, _ *mcp.CallToolRequest, in GetPodMetricsInput) (*mcp.CallToolResult, GetPodMetricsOutput, error) {
        metrics, err := kube.GetPodMetrics(ctx, in.Namespace, in.Name)
        if err != nil {
            // metrics-server missing or not yet scraped this pod shows up
            // as a normal API error here, wrap it with the likely cause
            // rather than letting the agent see a bare "not found."
            return nil, GetPodMetricsOutput{}, fmt.Errorf("pod_metrics: %w (is metrics-server installed?)", err)
        }

        out := GetPodMetricsOutput{Containers: make([]ContainerUsage, 0, len(metrics.Containers))}
        for _, c := range metrics.Containers {
            out.Containers = append(out.Containers, ContainerUsage{
                Name:   c.Name,
                CPU:    c.Usage.Cpu().String(),
                Memory: c.Usage.Memory().String(),
            })
        }
        return nil, out, nil
    }

    return tool, handler
}

That's twelve KubeClient methods reachable through nine tools (describe_pod, describe_ingress, and describe_httproute each composing more than one). Every tool file follows the identical shape: typed input, typed output, a handler that calls one or more KubeClient methods and translates the result, which is exactly the point of designing the client interface first, the way this post did.

Wiring it together

// internal/mcpserver/server.go
package mcpserver

import (
    "github.com/modelcontextprotocol/go-sdk/mcp"

    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/tools"
)

// New builds an MCP server with every diagnostic tool registered against
// the given KubeClient. Adding a tool means adding one pair of lines
// here, the server itself has no Kubernetes-specific logic, it only
// wires things up.
func New(kube kubernetes.KubeClient) *mcp.Server {
    server := mcp.NewServer(&mcp.Implementation{
        Name:    "k8s-diagnostics",
        Version: "0.1.0",
    }, nil)

    // Each New*Tool constructor returns two values, (*mcp.Tool, handler).
    // Go's multi-value spread only applies when that call is the sole
    // argument to the outer call, mcp.AddTool also takes server, so the
    // pair has to be captured into named variables first, tool by tool,
    // rather than nested directly inside AddTool.
    listPodsTool, listPodsHandler := tools.NewListPodsTool(kube)
    mcp.AddTool(server, listPodsTool, listPodsHandler)

    getPodLogsTool, getPodLogsHandler := tools.NewGetPodLogsTool(kube)
    mcp.AddTool(server, getPodLogsTool, getPodLogsHandler)

    describePodTool, describePodHandler := tools.NewDescribePodTool(kube)
    mcp.AddTool(server, describePodTool, describePodHandler)

    getEventsTool, getEventsHandler := tools.NewGetEventsTool(kube)
    mcp.AddTool(server, getEventsTool, getEventsHandler)

    checkEndpointsTool, checkEndpointsHandler := tools.NewCheckEndpointsTool(kube)
    mcp.AddTool(server, checkEndpointsTool, checkEndpointsHandler)

    describeIngressTool, describeIngressHandler := tools.NewDescribeIngressTool(kube)
    mcp.AddTool(server, describeIngressTool, describeIngressHandler)

    describeHTTPRouteTool, describeHTTPRouteHandler := tools.NewDescribeHTTPRouteTool(kube)
    mcp.AddTool(server, describeHTTPRouteTool, describeHTTPRouteHandler)

    listConfigMapKeysTool, listConfigMapKeysHandler := tools.NewListConfigMapKeysTool(kube)
    mcp.AddTool(server, listConfigMapKeysTool, listConfigMapKeysHandler)

    getPodMetricsTool, getPodMetricsHandler := tools.NewGetPodMetricsTool(kube)
    mcp.AddTool(server, getPodMetricsTool, getPodMetricsHandler)

    return server
}

// cmd/k8s-mcp-server/main.go
package main

import (
    "context"
    "fmt"
    "os"

    "github.com/modelcontextprotocol/go-sdk/mcp"

    k8s "github.com/FerRiosCosta/go-k8s-mcp-server/internal/kubernetes"
    "github.com/FerRiosCosta/go-k8s-mcp-server/internal/mcpserver"
)

func main() {
    kube, err := k8s.NewClient()
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to connect to cluster: %v\n", err)
        os.Exit(1)
    }

    server := mcpserver.New(kube)

    // Stdio is the right transport for a local agent (Claude Desktop, an
    // editor extension, or an agent CLI) running on the same machine as
    // the server. A remote, multi-tenant deployment needs a different
    // transport and a different set of tradeoffs entirely, that's its
    // own post.
    if err := server.Run(context.Background(), &mcp.StdioTransport{}); err != nil {
        fmt.Fprintf(os.Stderr, "server error: %v\n", err)
        os.Exit(1)
    }
}

k8s.NewClient no longer takes a path, it resolves config the same way kubectl does: in-cluster if running inside a pod, otherwise $KUBECONFIG if set, otherwise ~/.kube/config, using whatever context is currently active. That means switching clusters with kubectl config use-context changes what this server talks to as well, without touching any config file for the MCP client itself.

Running it end-to-end

Build the binary first:

go build -o k8s-mcp-server ./cmd/k8s-mcp-server

Then point an MCP-compatible client at it. Since NewClient resolves config the same way kubectl does, no KUBECONFIG needs to be passed through the client config at all, whatever context is currently active is the one the server talks to.

Claude Desktop, edit claude_desktop_config.json (~/Library/Application Support/Claude/ on macOS, %APPDATA%\Claude\ on Windows):

{
  "mcpServers": {
    "k8s-diagnostics": {
      "command": "/absolute/path/to/k8s-mcp-server"
    }
  }
}

Restart Claude Desktop, it only reads this file on startup, not live.

VS Code (GitHub Copilot), add to .vscode/mcp.json in the project, or via Command Palette → MCP: Open User Configuration. Note the root key is servers, not mcpServers:

{
  "servers": {
    "k8s-diagnostics": {
      "type": "stdio",
      "command": "/absolute/path/to/k8s-mcp-server"
    }
  }
}

MCP tools only show up once Copilot Chat's mode dropdown is switched to Agent, the default chat mode won't call tools at all.

If Copilot still reports the tools as unavailable even after MCP: List Servers shows k8s-diagnostics as Running and every tool is checked in the tools picker, start a brand new chat rather than continuing the existing one. Copilot fetches a session's tool list once, at the start of that conversation, enabling a server or checking its tools after the chat is already open doesn't retroactively update it. A fresh chat re-fetches the list and picks them up immediately. This is a quick thing to rule out before assuming the server itself is broken.

With minikube running and the go-api deployment from the earlier posts still there, recreate the memory-hog pod from the client-go post's OOMKill simulation:

kubectl apply -f memory-hog.yaml
# wait for CrashLoopBackOff

Then, in the agent, ask the question you'd normally type into a terminal:

"Why is memory-hog failing in the default namespace?"

Behind the scenes, the agent calls list_pods, sees memory-hog with a non-zero restart count, calls describe_pod, sees the container's waiting reason is CrashLoopBackOff, and calls get_pod_logs with previous: true since it knows the current instance may not have logged anything yet. It comes back with something like: "memory-hog is crash-looping with 13 restarts. The container is being OOMKilled, it's allocating 100MB but its memory limit is 10Mi." That's the same conclusion the reporter in the client-go post prints as a table row, arrived at through conversation instead of a report you have to read and interpret yourself.

kubectl delete pod memory-hog

Engineering practices that matter here

Read-only by default. Covered above at the interface level, worth restating: nothing in this server can mutate the cluster. If you extend it later with write operations, put them behind an explicit flag and treat them as a different trust boundary, not an addition to this interface.

Bound every unbounded query. Logs are the obvious one, but the same instinct applies to ListPods with no namespace on a thousand-pod cluster, or GetEvents on a noisy namespace. An agent doesn't know your cluster's scale before it asks; the server has to protect itself.

Never construct a new Clientset per tool call. Same gotcha as the client-go post, just resurfacing in a new context: build the Clientset once in main.go, inject it into every tool through KubeClient, and let client-go's connection pooling and rate limiting actually do their job across the lifetime of the process.

Log every tool invocation, structured. Not for debugging your Go code, for auditability: which tool was called, with what arguments, and what it returned, is the trail you'll want if an agent's actions ever need to be reconstructed after the fact. A single slog call wrapping each handler is enough to start.

RBAC scope the service account, not just the interface. The KubeClient interface being read-only is a code-level guarantee. Pair it with a Kubernetes-level one: run the server with a ServiceAccount bound to a ClusterRole that only grants get, list, and watch verbs. Two independent layers that both have to agree before anything unexpected can happen.

Common gotchas

Vague tool descriptions produce wrong tool calls. "Get pod info" and "Get a pod's phase, per-container state and restart reasons, and its most recent events" are the difference between an agent guessing and an agent knowing exactly when to reach for this tool over list_pods. Write descriptions for the agent the way you'd write a docstring for a teammate who's never seen your codebase.

Agents default to broad queries. Ask "what's wrong with my cluster" and an agent will happily call list_pods with an empty namespace before narrowing down. That's fine, as long as the tool is built to handle it cheaply, it's a reason the tool layer, not just the agent's judgment, needs to enforce limits.

Silent schema mismatches are worse than loud errors. If a struct field's jsonschema tag doesn't match what you documented in the tool's Description, the agent gets a schema that contradicts your prose and picks unpredictably. Keep the two in sync, and when in doubt, let the inferred schema be the source of truth.

Namespace scoping mistakes are common on the agent side. An agent asked to check "the checkout service" may not know which namespace it lives in and will guess default. Consider a tool that resolves a name to its namespace first (a find_resource style tool), rather than assuming every downstream tool call carries the right namespace.

mcp.AddTool(server, tools.NewXTool(kube)) doesn't compile. Every New*Tool constructor returns two values, (*mcp.Tool, handler). Go only spreads a multi-value call into another call's parameters when that call is the sole argument — mcp.AddTool also takes server as a first argument, so the pair has to be captured into named variables first: tool, handler := tools.NewListPodsTool(kube), then mcp.AddTool(server, tool, handler). Easy to miss since the single-argument version of this pattern (a bare f(g())) does compile, which makes the two-argument failure feel inconsistent the first time you hit it.

describe_httproute erroring on every call means Gateway API isn't installed. gatewayClientset.NewForConfig succeeds even against a cluster with zero Gateway API CRDs — it's just building a REST client for a set of API paths, not checking they exist. The failure only shows up once GetHTTPRoute or GetGateway actually runs, as a NotFound-shaped error on the CRD's group/version rather than a clear "Gateway API isn't installed" message. Worth wrapping that specific error with a clearer message in the tool handler, the same way pod_metrics wraps a missing metrics-server.

Summary

The protocol part of MCP is almost incidental, tools, JSON Schema, call and response. What actually made this server useful was already built in part 1: a KubeClient interface scoped to real diagnostic needs, read-only by construction. This post's job was just to wrap it correctly.

Three things worth taking away:

Every tool follows the same four-part shape, *mcp.Tool for identity, a typed In, a handler, a typed Out, which is what makes nine tools feel like one pattern instead of nine separate designs.
Return structured Out types, not preformatted strings. The agent reasons over JSON fields, not text it has to parse back apart.
Every boundary that matters, bounded log tails, capped event counts, secret values never leaving the cluster, gets enforced again at the tool layer, not just assumed from the client underneath it. An agent (or a misbehaving caller) can send arguments that ignore a schema's stated limits; the handler is the last line of defense.

What's next for this server is running it as shared infrastructure instead of a local stdio process someone points their laptop at, a different transport, session handling under concurrent agents, and the AWS-specific pieces of running it on EKS. That's a real jump in complexity, and it deserves its own post rather than a rushed section at the end of this one.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Build an MCP Server in Go (Part 1): Designing a diagnostic-grade Kubernetes client

Fer Rios — Fri, 14 Aug 2026 14:12:02 +0000

This post designs the Kubernetes client. The next post wraps it as an MCP server and wires it to an agent.

Asking instead of running

There's a familiar rhythm to debugging a bad deploy: kubectl get pods, spot the CrashLoopBackOff, kubectl logs --previous, still not obvious, kubectl describe pod, scroll to Events, cross-reference with kubectl get events, maybe check if the Service even has endpoints. Five or six commands, in your head, in a specific order, because you've done this enough times to know the order.

That order is exactly what an AI agent can execute for you, if you give it the right tools, ask "why is checkout-service failing?" and have it chain through the same commands you would have, arriving at an actual answer instead of a wall of YAML. The protocol that makes an agent capable of calling tools like that is the Model Context Protocol (MCP), and building that server is part 2 of this series.

This post is about what comes first and matters more: the Kubernetes client those tools will eventually sit on top of. An agent is only as good as what it's allowed to ask for. Hand it a client with one ListPods method and it can list pods, nothing else. Hand it a client that mirrors what a senior engineer actually checks when something's broken, pod state, events, endpoints, node capacity, rollout history, and it can genuinely diagnose. That client is a real piece of engineering on its own, independent of whether an LLM ever touches it, which is why it gets its own post before MCP enters the picture at all.

`ferctl` vs. an MCP server: two answers to the same problem

ferctl was my first attempt at this: a Cobra CLI, backed by client-go, that wraps common troubleshooting checks into subcommands: ferctl top, ferctl logs, that shape. It's fast, deterministic, and scriptable. Run ferctl describe-pod checkout-service, get the same structured output every time, pipe it intojq, drop it into a CI step, no ambiguity about what ran or why. That predictability is exactly what a CLI is good at.

What it doesn't do is investigate. ferctl runs the one command you gave it, you're still the one who has to know that a CrashLoopBackOff means "check previous logs, then check events", and you're still typing each step by hand. It's a faster way to run the commands you already know, not a way to skip learning them.

An MCP server flips that. You ask "why is checkout-service failing" once, and the agent decides the sequence, list pods, notices the restart count, pulls previous logs, cross-references events, the way a ferctl invocation never will, because no single subcommand can adapt its next step to what the last one returned. That's the shift this series is really about: from "a faster way to run known commands" to "something that can figure out which commands to run." It's also, frankly, the more current way to build this kind of tooling, a fixed CLI surface is a fine interface for a human who already knows the shape of the problem, but an LLM-driven agent chaining calls dynamically is a better fit for the actual shape of debugging, which rarely follows a script.

None of that makes ferctl obsolete. A CLI is still the right tool when you want guaranteed, repeatable output, a CI health check, a pre-deploy gate, anything where non-determinism is a bug, not a feature. An agent is the right tool when the problem is exploratory, and you don't yet know which three commands you'll need. They're not really competitors; they're two different interfaces that answer "is my cluster healthy" in two different situations. And notably, both could sit on top of the exact same KubeClient this post builds, the interface doesn't care whether its caller is a Cobra command or an MCP tool handler, which is itself a small argument for designing the client first, independent of either.

Project structure

go-k8s-mcp-server/
├── go.mod
└── internal/
    └── kubernetes/
        ├── client.go           ← KubeClient interface, constructor
        ├── pods.go             ← PodClient implementation
        ├── workloads.go        ← WorkloadClient implementation
        ├── nodes.go            ← NodeClient implementation
        ├── events.go           ← EventClient implementation
        ├── network.go          ← NetworkClient implementation
        ├── ingress.go          ← IngressClient implementation
        ├── gateway.go          ← GatewayClient implementation
        ├── config.go           ← ConfigClient implementation
        ├── storage.go          ← StorageClient implementation
        └── metrics.go          ← MetricsClient implementation

This is only the client half of the project, next post adds cmd/k8s-mcp-server/ and internal/tools/ on top of this same internal/kubernetes/ package. The internal/ boundary and one-package-per-domain layout follow the same convention laid out in Go packages and modules explained: internal/ for everything the Go toolchain should keep private to this module, one focused file per concern rather than one large one.

If you haven't set up a client-go connection before, this post assumes that groundwork. I covered kubeconfig loading, the Clientset, and how API groups work in Talking to Kubernetes from Go: a practical client-go guide. Everything below builds directly on that same connection pattern, same rest.Config setup, same in-cluster/out-of-cluster fallback, same "wrap the Clientset behind an interface" philosophy. What's new here is the shape of the interface itself.

Designing the `KubeClient` interface

A tempting first move is one fat interface with every method the agent might ever need. Resist it. A twenty-method interface is hard to test, hard to fake, and it hides the fact that these methods address genuinely different diagnostic concerns: some are about pods, some are about scheduling, some are about networking. Go rewards small interfaces, so we compose one from several.

If you're starting this project fresh rather than following along from the client-go post, pin the packages to your cluster's version, same as before, mixing minor versions across them breaks at compile time. k8s.io/metrics and sigs.k8s.io/gateway-api are included here too, even though neither is used until their respective sub-clients further down, because the client struct and NewClient constructor right below already need both:

mkdir go-k8s-mcp-server && cd go-k8s-mcp-server
go mod init github.com/FerRiosCosta/go-k8s-mcp-server

go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.36.2
go get k8s.io/apimachinery@v0.36.2
go get k8s.io/metrics@v0.36.2
go get sigs.k8s.io/gateway-api@v1.3.0
go mod tidy

// internal/kubernetes/client.go
package kubernetes

import (
    "context"
    "fmt"

    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
    metricsclientset "k8s.io/metrics/pkg/client/clientset/versioned"
    gatewayclientset "sigs.k8s.io/gateway-api/pkg/client/clientset/versioned"
)

// KubeClient is the full set of read-only diagnostic operations the
// MCP server exposes as tools. It's composed from smaller, domain-specific
// interfaces so each one stays independently testable and each
// implementation file has exactly one reason to change.
//
// Every method here is read-only by design. There is no Delete, Scale,
// or Patch anywhere in this interface, that's not an accident, it's
// the security boundary. See "Read-only by construction" below.
type KubeClient interface {
    PodClient
    WorkloadClient
    NodeClient
    EventClient
    NetworkClient
    IngressClient
    GatewayClient
    ConfigClient
    StorageClient
    MetricsClient
}

// client is the concrete implementation every sub-interface method
// below is defined on. It holds three Clientsets: the core Clientset
// for everything except metrics and Gateway API, a metrics-server
// Clientset (the split the client-go post flagged as needed once
// k8s.io/metrics entered the picture), and a Gateway API Clientset,
// Gateway API is CRD-based, so it isn't reachable through the core
// Clientset the way Ingress is.
type client struct {
    clientset        kubernetes.Interface
    metricsClientset metricsclientset.Interface
    gatewayClientset gatewayclientset.Interface
}

// NewClient builds a KubeClient the same way kubectl resolves its
// config: in-cluster config if running inside a pod, otherwise
// $KUBECONFIG if set, otherwise ~/.kube/config, and whichever
// context is marked current-context in that file, automatically.
// There's no path parameter here on purpose: hardcoding a path means
// this server silently ignores context switches made with
// `kubectl config use-context`, which is exactly the surprise you
// don't want from a diagnostic tool.
func NewClient() (KubeClient, error) {
    config, err := buildConfig()
    if err != nil {
        return nil, fmt.Errorf("build config: %w", err)
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create clientset: %w", err)
    }

    metricsClientset, err := metricsclientset.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create metrics clientset: %w", err)
    }

    gatewayClientset, err := gatewayclientset.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create gateway clientset: %w", err)
    }

    return &client{
        clientset:        clientset,
        metricsClientset: metricsClientset,
        gatewayClientset: gatewayClientset,
    }, nil
}

func buildConfig() (*rest.Config, error) {
    if config, err := rest.InClusterConfig(); err == nil {
        return config, nil
    }

    // NewDefaultClientConfigLoadingRules checks $KUBECONFIG first
    // (colon-separated on Linux/macOS, semicolon on Windows, merging
    // multiple files if listed), then falls back to ~/.kube/config.
    // ConfigOverrides{} is empty on purpose, an empty overrides
    // struct means "use whatever current-context is set," the same
    // default clientcmd.BuildConfigFromFlags gave us before, just
    // without hardcoding the path ourselves.
    loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
    overrides := &clientcmd.ConfigOverrides{}
    return clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, overrides).ClientConfig()
}

Same packages as the client-go post's client.go, plus two additions: metricsclientset from k8s.io/metrics, and gatewayclientset from sigs.k8s.io/gateway-api, both alongside the core Clientset. gatewayClientset will show NewForConfig succeed even against a cluster with no Gateway API CRDs installed; it's just a REST client pointed at a set of API paths, and only fails once you actually call it against a cluster that doesn't serve them. That failure mode gets called out concretely in the Gateway API section below.

Every method in the sections below is defined on this same *client type, one file per domain.

Pods, the core of "why is my app broken"

// internal/kubernetes/pods.go
package kubernetes

import (
    "context"
    "fmt"
    "io"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// PodClient covers the operations that answer "what's wrong with this pod."
type PodClient interface {
    // ListPods returns pods in namespace. An empty namespace returns
    // pods from every namespace, same convention as ListPods in the
    // client-go guide.
    ListPods(ctx context.Context, namespace string) (*corev1.PodList, error)

    // GetPod returns a single pod by name, used once the agent has
    // narrowed down which pod it cares about.
    GetPod(ctx context.Context, namespace, name string) (*corev1.Pod, error)

    // GetPodLogs returns log output for one container in a pod.
    GetPodLogs(ctx context.Context, namespace, name string, opts LogOptions) (string, error)
}

// LogOptions bounds a log request. This type exists specifically so an
// agent can never accidentally pull megabytes of logs in a single tool
// call, TailLines and SinceSeconds are the caps, not suggestions.
type LogOptions struct {
    Container    string // empty selects the pod's first container
    Previous     bool   // true fetches the last terminated instance's logs,
                         // essential for CrashLoopBackOff, since the *current*
                         // instance often hasn't logged the failure yet
    TailLines    int64  // 0 falls back to a safe default, never "unbounded"
    SinceSeconds int64  // 0 means no time bound
}

const defaultLogTailLines = 200

func (c *client) ListPods(ctx context.Context, namespace string) (*corev1.PodList, error) {
    pods, err := c.clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pods namespace=%q: %w", namespace, err)
    }
    return pods, nil
}

func (c *client) GetPod(ctx context.Context, namespace, name string) (*corev1.Pod, error) {
    pod, err := c.clientset.CoreV1().Pods(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get pod namespace=%q name=%q: %w", namespace, name, err)
    }
    return pod, nil
}

func (c *client) GetPodLogs(ctx context.Context, namespace, name string, opts LogOptions) (string, error) {
    tail := opts.TailLines
    if tail <= 0 {
        tail = defaultLogTailLines
    }

    podLogOpts := &corev1.PodLogOptions{
        Container: opts.Container,
        Previous:  opts.Previous,
        TailLines: &tail,
    }
    if opts.SinceSeconds > 0 {
        podLogOpts.SinceSeconds = &opts.SinceSeconds
    }

    req := c.clientset.CoreV1().Pods(namespace).GetLogs(name, podLogOpts)
    stream, err := req.Stream(ctx)
    if err != nil {
        return "", fmt.Errorf("stream logs namespace=%q pod=%q: %w", namespace, name, err)
    }
    defer stream.Close()

    data, err := io.ReadAll(stream)
    if err != nil {
        return "", fmt.Errorf("read logs namespace=%q pod=%q: %w", namespace, name, err)
    }
    return string(data), nil
}

Note that TailLines defaults to 200, not 0 meaning unbounded. That single line is the difference between a tool an agent can call freely and a tool that occasionally hands it 40,000 lines of stack traces and blows past its context window on one call.

Events, often the fastest path to a root cause

// internal/kubernetes/events.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EventClient covers the Events API, which frequently surfaces a root
// cause, a failed scheduling attempt, an image pull error, before
// anything else does. It's queried by involved object, not by name,
// which is different enough from PodClient to earn its own file.
type EventClient interface {
    // GetEvents returns events in namespace, optionally filtered to
    // those involving a specific object name (e.g. a Pod).
    // An empty involvedObjectName returns all events in the namespace.
    GetEvents(ctx context.Context, namespace, involvedObjectName string) ([]corev1.Event, error)
}

func (c *client) GetEvents(ctx context.Context, namespace, involvedObjectName string) ([]corev1.Event, error) {
    opts := metav1.ListOptions{}
    if involvedObjectName != "" {
        opts.FieldSelector = "involvedObject.name=" + involvedObjectName
    }

    events, err := c.clientset.CoreV1().Events(namespace).List(ctx, opts)
    if err != nil {
        return nil, fmt.Errorf("list events namespace=%q object=%q: %w", namespace, involvedObjectName, err)
    }
    return events.Items, nil
}

Workloads, rollout and scheduling problems above the pod

Not every failure lives at the pod level. A deployment that's stuck rolling out, or one whose pods belong to two different generations at once, needs a different query shape entirely: you're not asking about one pod anymore, you're asking about the relationship between a Deployment and the ReplicaSets it owns.

// internal/kubernetes/workloads.go
package kubernetes

import (
    "context"
    "fmt"

    appsv1 "k8s.io/api/apps/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// WorkloadClient covers Deployments and the ReplicaSets they own,
// the layer above individual pods, where rollout and scheduling
// problems that span multiple pods actually live.
type WorkloadClient interface {
    // GetDeployment returns a Deployment's spec and status, including
    // desired vs. available replica counts and rollout conditions.
    GetDeployment(ctx context.Context, namespace, name string) (*appsv1.Deployment, error)

    // ListReplicaSets finds every ReplicaSet matching labelSelector in
    // a namespace. This is what surfaces orphaned or stuck ReplicaSets,
    // the classic "Deployment says 3/3 but two of those pods belong to
    // the old ReplicaSet" failed-rollout scenario, which GetDeployment
    // alone won't show you.
    ListReplicaSets(ctx context.Context, namespace, labelSelector string) (*appsv1.ReplicaSetList, error)
}

func (c *client) GetDeployment(ctx context.Context, namespace, name string) (*appsv1.Deployment, error) {
    deploy, err := c.clientset.AppsV1().Deployments(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get deployment namespace=%q name=%q: %w", namespace, name, err)
    }
    return deploy, nil
}

func (c *client) ListReplicaSets(ctx context.Context, namespace, labelSelector string) (*appsv1.ReplicaSetList, error) {
    rs, err := c.clientset.AppsV1().ReplicaSets(namespace).List(ctx, metav1.ListOptions{
        LabelSelector: labelSelector,
    })
    if err != nil {
        return nil, fmt.Errorf("list replicasets namespace=%q selector=%q: %w", namespace, labelSelector, err)
    }
    return rs, nil
}

ListReplicaSets takes a label selector rather than a Deployment name because that's what the ReplicaSets API actually indexes on, a Deployment doesn't "contain" its ReplicaSets, it selects them by label, the same way a Service selects its pods. To go from "diagnose this Deployment" to "here are its ReplicaSets," a tool built on this interface pulls the selector off the Deployment's spec first (deploy.Spec.Selector), then passes that into ListReplicaSets. That two-step shape is exactly why this stays two methods instead of one convenience method that hides the relationship, the tool layer is where composing them into a single diagnose_rollout-style tool belongs, not the client.

Nodes, capacity and scheduling context

// internal/kubernetes/nodes.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// NodeClient covers cluster capacity, what you need to explain a
// "0/3 nodes are available" scheduling failure, which is a node-level
// question, not a pod-level one.
type NodeClient interface {
    ListNodes(ctx context.Context) (*corev1.NodeList, error)

    // GetNode returns conditions (MemoryPressure, DiskPressure,
    // PIDPressure), taints, and allocatable resources for one node.
    GetNode(ctx context.Context, name string) (*corev1.Node, error)
}

func (c *client) ListNodes(ctx context.Context) (*corev1.NodeList, error) {
    nodes, err := c.clientset.CoreV1().Nodes().List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list nodes: %w", err)
    }
    return nodes, nil
}

func (c *client) GetNode(ctx context.Context, name string) (*corev1.Node, error) {
    node, err := c.clientset.CoreV1().Nodes().Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get node name=%q: %w", name, err)
    }
    return node, nil
}

Network, is anything actually answering

// internal/kubernetes/network.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// NetworkClient covers Services and Endpoints. A Service existing
// doesn't mean anything is listening behind it, that's exactly the
// gap GetEndpoints is here to close.
type NetworkClient interface {
    GetService(ctx context.Context, namespace, name string) (*corev1.Service, error)

    // GetEndpoints checks whether a Service actually has healthy pods
    // backing it. Zero endpoints despite matching pods usually means a
    // label selector mismatch or a failing readiness probe, one of the
    // most common "works locally, broken in cluster" bugs.
    GetEndpoints(ctx context.Context, namespace, serviceName string) (*corev1.Endpoints, error)
}

func (c *client) GetService(ctx context.Context, namespace, name string) (*corev1.Service, error) {
    svc, err := c.clientset.CoreV1().Services(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get service namespace=%q name=%q: %w", namespace, name, err)
    }
    return svc, nil
}

func (c *client) GetEndpoints(ctx context.Context, namespace, serviceName string) (*corev1.Endpoints, error) {
    // The Endpoints object shares its name with the Service it backs,
    // Kubernetes creates and keeps it in sync automatically, so no
    // separate lookup by label is needed here.
    eps, err := c.clientset.CoreV1().Endpoints(namespace).Get(ctx, serviceName, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get endpoints namespace=%q service=%q: %w", namespace, serviceName, err)
    }
    return eps, nil
}

Ingress on EKS, is the ALB even pointing here

A healthy Service with healthy endpoints still means nothing to an external caller if the layer in front of it, the Ingress, and on EKS almost always the AWS Load Balancer Controller (LBC) behind it, never finished provisioning. This is a different failure class from anything NetworkClient covers: the Service can be perfectly correct and traffic still never arrives, because the ALB was never created, was created against the wrong subnets, or is pointing at the wrong target group.

// internal/kubernetes/ingress.go
package kubernetes

import (
    "context"
    "fmt"

    networkingv1 "k8s.io/api/networking/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// IngressClient covers classic Ingress resources, on EKS, almost
// always reconciled by the AWS Load Balancer Controller into an ALB.
// No AWS API calls happen here; everything comes from the Kubernetes
// object the controller writes status and events back onto, which is
// usually enough to tell you where reconciliation stalled.
type IngressClient interface {
    GetIngress(ctx context.Context, namespace, name string) (*networkingv1.Ingress, error)

    // ListIngressClasses lets a diagnosis confirm the Ingress's
    // ingressClassName actually exists and matches a real controller.
    // A typo'd or missing class is a common reason an Ingress just
    // sits there, the LBC never picks it up, and nothing in the
    // Ingress object itself says why.
    ListIngressClasses(ctx context.Context) (*networkingv1.IngressClassList, error)
}

func (c *client) GetIngress(ctx context.Context, namespace, name string) (*networkingv1.Ingress, error) {
    ing, err := c.clientset.NetworkingV1().Ingresses(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get ingress namespace=%q name=%q: %w", namespace, name, err)
    }
    return ing, nil
}

func (c *client) ListIngressClasses(ctx context.Context) (*networkingv1.IngressClassList, error) {
    classes, err := c.clientset.NetworkingV1().IngressClasses().List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list ingress classes: %w", err)
    }
    return classes, nil
}

Three things matter when reading an Ingress object for diagnosis, worth knowing even before the tool layer wraps this in part 2:

ingress.Status.LoadBalancer.Ingress: empty means the LBC either hasn't reconciled yet or is stuck. A populated hostname (the ALB's DNS name) means AWS-side provisioning succeeded; an empty one after more than a minute or two almost always means the LBC is failing, not just slow.
ingress.Spec.IngressClassName: cross-reference this against ListIngressClasses output. A nil or misspelled class is the single most common reason an Ingress is silently ignored, no error on the object itself, the LBC just never claims it.
Events on the Ingress object: this is where EventClient.GetEvents, already built above, becomes directly useful here without any new code: the LBC writes events like SuccessfullyReconciled on success, and specific failure reasons (invalid target group, subnet tagging problems, certificate ARN not found) as Warning events when reconciliation fails. Composing GetIngress with GetEvents is exactly the same pattern describe_pod used for pods, applied one layer up the stack.

Gateway API, the newer, cluster-portable entry point

Gateway API is the follow-on to Ingress: instead of one annotation-heavy resource, traffic routing splits across a Gateway (the listener; where traffic enters, which addresses/ports it accepts) and route resources like HTTPRoute (how it's matched and where it goes). EKS supports it the same way it supports Ingress, the AWS Load Balancer Controller reconciles Gateway/HTTPRoute into an ALB, the same way it reconciles Ingress.

The client-side difference is real, not cosmetic: Gateway API types are CRDs, not part of the core Kubernetes API, so they need their own Clientset, the gatewayClientset already added to the client struct above.

// internal/kubernetes/gateway.go
package kubernetes

import (
    "context"
    "fmt"

    gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// GatewayClient covers Gateway API's two core resources. Both come
// back with rich status.conditions, Gateway API standardizes on
// Accepted/Programmed for Gateways and Accepted/ResolvedRefs for
// routes, which is a more structured diagnostic signal than Ingress
// ever gave you: no need to infer state from events alone.
type GatewayClient interface {
    GetGateway(ctx context.Context, namespace, name string) (*gatewayv1.Gateway, error)

    // GetHTTPRoute returns the route's status, including
    // per-parent-Gateway conditions. ResolvedRefs=False on a route
    // almost always means a backendRef points at a Service or port
    // that doesn't exist, check it before assuming the problem is
    // upstream at the Gateway.
    GetHTTPRoute(ctx context.Context, namespace, name string) (*gatewayv1.HTTPRoute, error)
}

func (c *client) GetGateway(ctx context.Context, namespace, name string) (*gatewayv1.Gateway, error) {
    gw, err := c.gatewayClientset.GatewayV1().Gateways(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get gateway namespace=%q name=%q: %w", namespace, name, err)
    }
    return gw, nil
}

func (c *client) GetHTTPRoute(ctx context.Context, namespace, name string) (*gatewayv1.HTTPRoute, error) {
    route, err := c.gatewayClientset.GatewayV1().HTTPRoutes(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get httproute namespace=%q name=%q: %w", namespace, name, err)
    }
    return route, nil
}

Diagnosis here is a two-level check, and skipping the first level is the most common mistake:

gateway.Status.Conditions: check Accepted and Programmed first. If the Gateway itself isn't Programmed, no HTTPRoute attached to it can possibly work, no matter how correct that route is. This is the equivalent of checking describe_pod's container state before chasing application-level logs, start at the layer closest to the infrastructure.
httproute.Status.Parents[].Conditions: a route can reference multiple Gateways, so status is reported per-parent, not once. ResolvedRefs: False specifically means a backendRef, the Service and port the route sends traffic to, doesn't resolve. That's diagnosable the same way check_endpoints diagnoses a plain Service: the route's backendRefs[].name and .port are exactly what GetService/GetEndpoints from earlier need to cross-check.
A cluster without Gateway API CRDs installed: surfaces as a real error from GetGateway/GetHTTPRoute, a NotFound-shaped error on the CRD's group/version, not on the specific object. Worth catching and wrapping with a clearer message at the tool layer in part 2, the same way pod_metrics wraps a missing metrics-server as a clearer error than the bare API response gives you.

Config, keys only, never values

// internal/kubernetes/config.go
package kubernetes

import (
    "context"
    "fmt"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ConfigClient deliberately returns key names only, for both
// ConfigMaps and Secrets. An agent diagnosing a missing env var needs
// to know a key exists, never what it contains, see "Read-only by
// construction" below for why that boundary lives here and not in a
// tool description.
type ConfigClient interface {
    GetConfigMapKeys(ctx context.Context, namespace, name string) ([]string, error)
    GetSecretKeys(ctx context.Context, namespace, name string) ([]string, error)
}

func (c *client) GetConfigMapKeys(ctx context.Context, namespace, name string) ([]string, error) {
    cm, err := c.clientset.CoreV1().ConfigMaps(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get configmap namespace=%q name=%q: %w", namespace, name, err)
    }
    keys := make([]string, 0, len(cm.Data))
    for k := range cm.Data {
        keys = append(keys, k)
    }
    return keys, nil
}

func (c *client) GetSecretKeys(ctx context.Context, namespace, name string) ([]string, error) {
    secret, err := c.clientset.CoreV1().Secrets(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get secret namespace=%q name=%q: %w", namespace, name, err)
    }
    // secret.Data is map[string][]byte, the values are deliberately
    // never read here, only the key names.
    keys := make([]string, 0, len(secret.Data))
    for k := range secret.Data {
        keys = append(keys, k)
    }
    return keys, nil
}

Storage, volumes that can't bind

// internal/kubernetes/storage.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// StorageClient surfaces the "pod stuck Pending because its volume
// can't bind" class of failure, a Pending PVC explains a Pending pod
// that otherwise looks like a scheduling mystery.
type StorageClient interface {
    ListPVCs(ctx context.Context, namespace string) (*corev1.PersistentVolumeClaimList, error)
}

func (c *client) ListPVCs(ctx context.Context, namespace string) (*corev1.PersistentVolumeClaimList, error) {
    pvcs, err := c.clientset.CoreV1().PersistentVolumeClaims(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pvcs namespace=%q: %w", namespace, err)
    }
    return pvcs, nil
}

Metrics, the same extension the client-go post already teed up

k8s.io/metrics was already installed above alongside the three core packages, since NewClient needed metricsclientset before this section even started. This is where it actually gets used:

// internal/kubernetes/metrics.go
package kubernetes

import (
    "context"
    "fmt"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    metricsv1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1"
)

// MetricsClient reads from the metrics-server extension API, a
// separate Clientset from the core one, same as it was flagged in the
// client-go post's "what's next" for the ferctl top CLI.
type MetricsClient interface {
    GetPodMetrics(ctx context.Context, namespace, name string) (*metricsv1beta1.PodMetrics, error)
    GetNodeMetrics(ctx context.Context, name string) (*metricsv1beta1.NodeMetrics, error)
}

func (c *client) GetPodMetrics(ctx context.Context, namespace, name string) (*metricsv1beta1.PodMetrics, error) {
    m, err := c.metricsClientset.MetricsV1beta1().PodMetricses(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get pod metrics namespace=%q name=%q: %w", namespace, name, err)
    }
    return m, nil
}

func (c *client) GetNodeMetrics(ctx context.Context, name string) (*metricsv1beta1.NodeMetrics, error) {
    m, err := c.metricsClientset.MetricsV1beta1().NodeMetricses().Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get node metrics name=%q: %w", name, err)
    }
    return m, nil
}

GetPodMetrics and GetNodeMetrics reach through a second Clientset, c.metricsClientset, built from k8s.io/metrics alongside the core one in NewClient, the same two-Clientset shape the client-go post flagged as coming in a future post, now realized here instead.

GetConfigMapKeys and GetSecretKeys are worth pausing on. It would be less code to return the whole object, map[string]string and all. Returning only the keys is a security decision made at the interface boundary, before a single tool or prompt exists, an agent that can only ever see "this Secret has a key called DATABASE_URL" cannot leak what that key contains, no matter how it's prompted. That's a much stronger guarantee than "the tool description tells the agent not to print secret values."

Read-only by construction

Summary

Every method on KubeClient is List, Get, or Stream, nothing that mutates the cluster. That boundary, along with the read-only Secret/ConfigMap key-only design and the bounded LogOptions, all live in the Go types themselves rather than in documentation someone has to remember to follow. Three things worth taking away:

Compose small, domain-specific interfaces (PodClient, EventClient, NetworkClient, ...) instead of one large KubeClient with twenty methods on it, easier to test, easier to fake, and each file has one reason to change.
Design the client before you design anything that calls it. Whether the caller ends up being a CLI, a report generator, or an AI agent, the interface should already answer "what's actually useful to ask a cluster" independent of who's asking.
Every safety boundary that matters, read-only access, bounded queries, secret values never leaving the cluster, belongs in the interface's types and method signatures, not in a comment asking future callers to be careful.

Part 2 picks up exactly here: wrapping this KubeClient as MCP tools, wiring it to an agent, and running the whole thing end-to-end against a real cluster. Building a Kubernetes-aware AI agent with a Go MCP server →

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

LeetCode #345 in Go: reverse vowels of a string, and how strings, bytes, and runes work in Go

Fer Rios — Wed, 05 Aug 2026 13:33:25 +0000

Why I started this series

A while back, I went through a job interview that included a coding challenge. It looked pretty easy, so I finished in like 90 seconds, and all tests passed! Then, the (non-technical) interviewer said to me:

Interviewer: Why did you choose Golang? Do you actually know how to code in Golang?
Me: I said, yes, that challenge was easy, and all tests passed!.
Interviewer: Well, you only got 43% on that challenge.

I'm going to be honest with you.

I've been writing about Go for a while: Lambda functions, Kubernetes tooling, HTTP middleware, client-go. But there's a difference between using Go and actually understanding Go.

I decided to fix that by doing LeetCode challenges. Not to grind interview prep. Not to get a FAANG job. Just to force myself to use the language in ways I wouldn't naturally reach for on a normal workday as a Senior SRE.

And something unexpected happened: every challenge I struggled with exposed a Go concept I hadn't fully understood. This is the first one.

The challenge

LeetCode #345

https://leetcode.com/problems/reverse-vowels-of-a-string/description/

The problem requires us to reverse the vowels in a string. Note that letter case matters.

Input: "IceCreAm"

Output: "AceCreIm"

Solution Approach

The solution approach for this problem is to use 2 pointers, the two-pointer collision approach, to continuously swap the elements at the beginning and end.

The solution approach was extrated from this [amazing book](http:// https://books.halfrost.com/leetcode/en/ ), please consider reading this book if you want to improve your Golang coding skills like me.

The core idea: two pointers moving toward each other

First, let's try to understand how the solution approach works. Imagine the string as a row of people standing in a line, numbered from left to right. You place one finger at the very first person (index left = 0) and another finger at the very last person (index right = len - 1). Then you move both fingers toward the middle, one step at a time, until they meet or cross.

This is different from two pointers moving in the same direction (like for finding duplicates in a sorted array). Here they move toward each other, which is why it's called the "collision" pattern; they're on a collision course toward the center.

Why this pattern fits "reverse vowels"

Think about what "reversing only the vowels" really means: the vowels that appear in the string should end up in reverse order, but every consonant should stay exactly where it is.

The trick is: you don't need to know the positions of all the vowels ahead of time. You can discover them as you scan from both ends, because:

The leftmost vowel needs to be swapped with the rightmost vowel.
Then the second leftmost vowel needs to be swapped with the second rightmost vowel.
And so on, working inward.

That's exactly what the collision pointers give you for free, if you use them correctly.

Step-by-step logic

Set left at the start of the string, right at the end.
Move left forward until it lands on a vowel (skip consonants).
Move right backward until it lands on a vowel (skip consonants).
Once both left and right are sitting on vowels, swap them.
Move left one step further right, and right one step further left (since those two are now "done").
Repeat until left >= right (the pointers meet or cross).

Notice the elegance: consonants are never touched or moved; the pointers just glide past them. Only vowels get swapped, and because you're always taking the current outermost unprocessed vowel from each side, they naturally end up reversed relative to each other.

Walking through an example

Let's use "leetcode".

l  e  e  t  c  o  d  e
0  1  2  3  4  5  6  7

left = 0 → 'l' is not a vowel → move left to 1.
Index 1 → 'e' is a vowel → stop left here.
right = 7 → 'e' is a vowel → stop right here.
Both are vowels → swap index 1 and index 7. Since both are 'e', string looks unchanged, but conceptually a swap happened.
Move left to 2, right to 6.
Index 2 → 'e' is a vowel → stop left.
Index 6 → 'd' is not a vowel → move right to 5.
Index 5 → 'o' is a vowel → stop right.
Swap index 2 ('e') and index 5 ('o') → now index 2 is 'o', index 5 is 'e'.
Move left to 3, right to 4. Now left (3) < right (4) still true, but index 3 is 't' (not vowel) → move left to 4. Now left == right, loop stops.

Result: "leotcede". That's the correct answer; the vowels e, e, o, e have been reversed to e, o, e, e while consonants l, t, c, d stayed in place.

A second, trickier example: mixed case

Try "Aa", both are vowels ('A' and 'a', the problem says vowels can be uppercase or lowercase). left=0 is 'A', right=1 is 'a', both vowels immediately, so you swap. Result: "aA". This tests that your vowel-check needs to handle both cases, not just lowercase.

Checklist for how you'd think about implementing it

When you're ready to write this yourself, you'll want to think through:

How do I quickly check whether a character is a vowel?: Hint: a small lookup set/map is cleaner than a long chain of if statements comparing against a, e, i, o, u, A, E, I, O, U.
Can I swap characters directly in a Go string?: This is an important Go-specific gotcha: strings in Go are immutable, so you cannot do s[i], s[j] = s[j], s[i] directly on a string. You'll need to convert it into something mutable first.
[]byte vs []rune: Since this problem only deals with ASCII vowels, a []byte conversion works fine and is efficient. (If the problem involved non-ASCII/multi-byte characters, you'd need []rune instead.)
Loop condition: When do you stop? (left < right)
Two separate inner loops or one combined check?: You need to advance left past consonants and advance right past consonants, independently, before doing the swap.

Let's dig into the Go-specific mechanics that matter for this problem.

Why Go strings are immutable

In Go, a string is internally a small struct; conceptually, it looks like this:

type stringHeader struct {
    data uintptr // pointer to the underlying byte array
    len  int     // length in bytes
}

It's a read-only view into a sequence of bytes. The language spec deliberately does not let you write to that underlying array through a string variable. So this:

s := "leetcode"
s[0] = 'L' // compile error!

...will not even compile. Go gives you: cannot assign to s[0] (neither addressable nor a map index expression).

This immutability exists for good reasons; it lets multiple strings safely share the same underlying byte array (e.g., substring operations like s[2:5] don't copy memory; they just create a new header pointing into the same data). If strings were mutable, that sharing would be dangerous; modifying one substring could silently corrupt another.

How to get mutability: convert to `[]byte`

Since you can mutate a slice, the standard trick is to convert the string into a []byte:

b := []byte(s)

This conversion copies the string's bytes into a brand-new, independent slice. Now you're free to do:

b[0] = 'L'

...because b is a slice, a mutable, addressable view over its own memory. When you're done modifying, you convert back:

result := string(b)

That conversion also copies the bytes back into a new immutable string. So the full round trip (string → []byte → string) does involve two copies total, but for a single-pass algorithm like this one, that's totally acceptable and is the idiomatic way to do in-place-style string manipulation in Go.

`[]byte` vs `[]rune`: why it matters

This is one of the most important Go concepts for string problems, so let's be precise about it.

A byte is Go's alias for uint8, one 8-bit unit. []byte(s) gives you the raw UTF-8 encoded bytes of the string.
A rune is Go's alias for int32, it represents a single Unicode code point, which might be encoded using 1 to 4 bytes in UTF-8.

For plain ASCII characters, like the letters a-z, A-Z that this problem deals with, each character is exactly one byte. So []byte gives you a clean one-index-per-character view, which is exactly what the two-pointer approach needs (b[left] and b[right] each represent one whole letter).

If the string contained non-ASCII characters (like é, 日, or emoji), a single character could span multiple bytes, and indexing into []byte at arbitrary positions could land you in the middle of a multi-byte character, corrupting it. In that scenario, you'd need []rune(s) instead, which decodes the UTF-8 bytes into a slice where each element is one full code point, safe to index individually, at the cost of an extra decoding pass.

For this exercise, the constraints guarantee the input consists only of printable ASCII characters, so []byte is the correct and more efficient choice. Good habit to build: always check the problem's constraints for character set before deciding between []byte and []rune.

Checking "is this a vowel" idiomatically

A few ways this is commonly done in Go, from least to most idiomatic for this case:

Option A: long boolean chain (works, but verbose and easy to mistype):

c == 'a' || c == 'e' || c == 'i' || c == 'o' || c == 'u' || c == 'A' || ...

Option B: a map used as a set:

vowels := map[byte]bool{
    'a': true, 'e': true, 'i': true, 'o': true, 'u': true,
    'A': true, 'E': true, 'I': true, 'O': true, 'U': true,
}
// then: vowels[c]

This is very readable. Map lookups are O(1) on average, so it doesn't hurt performance meaningfully for such a small, fixed set.

Option C: strings.ContainsRune:

strings.ContainsRune("aeiouAEIOU", rune(c))

Short and clean, though technically does a linear scan over 10 characters each time; negligible here, but worth knowing it's not a true O(1) lookup like the map.

For a problem like this with a tiny, fixed alphabet, either B or C is considered clean, idiomatic Go. The map is generally preferred when you want to signal "this is a set membership check" clearly and efficiently.

Bringing it together conceptually (still no full code)

So your mental model in Go terms becomes:

Convert s to b := []byte(s) to get a mutable working copy.
Run your left/right collision pointers over b, using your vowel-check function/map on b[left] and b[right].
When both point to vowels, do a byte swap: b[left], b[right] = b[right], b[left]; this works fine because b is a slice, and Go allows this kind of tuple-assignment swap directly, no temp variable needed.
After the loop finishes, convert back: string(b).

A quick introduction to Big-O notation, and why this matter

Good, let's tackle Big-O intuition first since it'll come up constantly in LeetCode, according to wikipedia:

Big O notation is a mathematical framework used in computer science to describe how an algorithm's execution time or memory usage scales as the input size grows.

What "O(1)" actually means

Big-O notation describes how the cost of an operation grows as the input size grows. The letter usually used for input size is n.

O(n) means: if you double the input size, the work roughly doubles too. A linear scan through an array is O(n), checking every element once.
O(1) means: the amount of work stays constant, no matter how big the input is. It doesn't mean "one operation" literally, it means "a fixed, bounded number of operations that doesn't depend on n."

Think of it like this analogy: imagine you have a giant phone book with a million names.

O(n) approach: You start at page 1 and flip through every single page looking for "Smith." If the book had 10 names, that'd be fast. With a million names, it's painfully slow; the work scales with the size of the book.
O(1) approach: Instead, imagine you have a magic index card that instantly tells you "Smith is on page 4213", you jump straight there. Whether the book has 10 names or 10 million, that lookup takes the same tiny amount of time.

Why map lookups are O(1)

A Go map is implemented as a hash table. When you do vowels[c], Go doesn't loop through every key checking for a match. Instead:

It runs the key (c) through a hash function; a formula that converts the key into a number very quickly.
That number tells it almost directly which "bucket" in memory to look in.
It checks that bucket (usually just one item, sometimes a couple if there's a hash collision) and returns the result.

This process takes roughly the same amount of time whether your map has 5 entries or 5 million, that's why it's called O(1) on average (technically "amortized average case," since hash collisions can rarely cause it to degrade, but for practical purposes, treat it as constant time).

Compare that to strings.ContainsRune("aeiouAEIOU", rune(c)); internally, this does loop through the characters of that 10-character string comparing one by one until it finds a match or exhausts the string. It's technically O(k) where k is the length of that fixed string (10). Since 10 never changes, people often casually call it "O(1) too" because it's a bounded constant; but it's doing more actual comparisons than a map lookup, just not enough to matter for a 10-character set.

The key exam-style distinction to remember: O(1) doesn't mean "instant" or "zero cost"; it means "cost that doesn't grow with your input size n."

Now, the tuple swap

Go has a really elegant built-in feature for swapping: multiple assignment. When you write:

b[left], b[right] = b[right], b[left]

Here's what happens under the hood, step by step:

Go first evaluates the entire right-hand side of the assignment, in order, before assigning anything. So it reads the current value of b[right] and the current value of b[left], and holds both of them temporarily (essentially in a tuple, like (valueOfRight, valueOfLeft)).
Then it assigns those held values to the left-hand side, in order: b[left] gets the first held value (old b[right]), and b[right] gets the second held value (old b[left]).

Because Go evaluates the whole right-hand side first, there's no risk of accidentally overwriting b[left] before you've read its original value to place into b[right]. This is exactly why you don't need a temporary variable, unlike languages such as C, where a swap traditionally requires:

temp = b[left];
b[left] = b[right];
b[right] = temp;

In Go, the language does that temporary-holding for you internally, and lets you express it in one clean line.

Remember: this works because slice indexing (b[left]) is addressable, meaning you can actually assign to that memory location. Recall this is exactly why we converted our string to a []byte earlier; string indexing (s[left]) is not addressable, so s[left], s[right] = s[right], s[left] would fail to compile, but b[left], b[right] = b[right], b[left] works perfectly because b is a slice.

Writing the full solution

You now have all the conceptual pieces: two-pointer collision logic, why/how to convert string ↔ []byte, []byte vs []rune, O(1) vowel checks, and how Go's swap syntax works.

func reverseVowels(s string) string {
    vowels := map[byte]bool{
        'a': true, 'e': true, 'i': true, 'o': true, 'u': true,
        'A': true, 'E': true, 'I': true, 'O': true, 'U': true, 
    }

    b := []byte(s)
    left, right := 0, len(b) - 1

    for left < right {
        if !vowels[b[left]] {
            left++
            continue
        }
        if !vowels[b[right]] {
            right--
            continue
        }
        b[left], b[right] = b[right], b[left]
        left++
        right--

    }
    return string(b)

}

Summary

Three things worth keeping from this problem:

Two-pointer collision is the pattern to reach for whenever a problem wants you to work from both ends of a sequence toward the middle, reversing, palindrome checks, and partitioning problems all lean on this same idea.
Go strings are immutable. Any in-place-style string manipulation goes through a string → []byte → string round trip.
A map is your O(1) toolbox for set membership, and Go's tuple assignment (a, b = b, a) gives you a clean swap for free, as long as both sides are addressable.

This problem is small, but it's a solid vehicle for internalizing patterns that show up constantly in string and array problems on LeetCode.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Building ferctl top: Kubernetes resource usage vs requests and limits

Fer Rios — Mon, 03 Aug 2026 15:36:32 +0000

Series: Platform engineering with Go | Topics: Go, Kubernetes, Cobra, client-go, metrics-server, Platform Engineering

This is part of the Platform Engineering with Go series. This post builds on the Cobra CLI patterns from post 4 and client-go from post 3. Read post 4 first if you haven't yet.

kubectl top tells you what's happening. It doesn't tell you how close to the edge you are.

In post 3 and post 4, we built a health reporter and learned how to structure a Go CLI with Cobra. Now we put both together into something with real operational value.

kubectl top pods -n production

NAME                      CPU(cores)   MEMORY(bytes)
go-api-7d6b9f8c4-xk2pq   240m         490Mi
go-api-7d6b9f8c4-mn9rt   180m         210Mi
go-api-7d6b9f8c4-p8wvz   200m         198Mi

That first pod is using 490Mi of memory. Is that fine or is that a problem? Without knowing the limit, you can't tell. You'd have to run kubectl describe pod go-api-7d6b9f8c4-xk2pq, find the resources section, do the mental arithmetic, and repeat for every pod you care about.

ferctl top does all of that in one command:

ferctl top -n production

NAMESPACE    NAME                      CPU USE  CPU REQ  CPU LIM  CPU%  MEM USE  MEM REQ  MEM LIM  MEM%  STATUS
production   go-api-7d6b9f8c4-xk2pq   240m     250m     500m     48%   490Mi    256Mi    512Mi    95%   !! CRITICAL
production   go-api-7d6b9f8c4-mn9rt   180m     250m     500m     36%   210Mi    256Mi    512Mi    41%      OK
production   go-api-7d6b9f8c4-p8wvz   200m     250m     500m     40%   198Mi    256Mi    512Mi    38%      OK

One pod is at 95% of its memory limit. In production, that's a page waiting to happen. ferctl top catches it before it becomes an incident.

What you'll learn

How to extend the Cobra CLI structure from post 4 with a real subcommand
How to query the metrics-server API using k8s.io/metrics
How to correlate live metrics with pod specs to show usage vs limits
How to implement configurable near-limit warnings
How to format clean aligned output with tabwriter
How to verify the tool against your real minikube cluster

Prerequisites

Posts 1–4 read; client-go patterns from post 3, Cobra CLI structure from post 4
minikube running with metrics-server enabled (from post 2)
Understand what limits and requests are in Kubernetes (from post 1)
Go 1.26.3 installed
The go-api deployment from post 2 running in your cluster

Project structure

As always, go to this post if you don't know how to create a structure for your Golang project.

Also, you can find the completed repo here

ferctl/
├── go.mod
├── Makefile
├── cmd/
│   └── ferctl/
│       └── main.go                  ← 3 lines — just calls Execute()
└── internal/
    ├── kubernetes/
    │   ├── client.go                ← KubeClient interface + real implementation
    │   └── metrics.go               ← MetricsClient interface + real implementation
    ├── top/
    │   ├── command.go               ← Cobra command — flags and wiring only
    │   ├── runner.go                ← business logic — fetches and correlates data
    │   ├── output.go                ← tabwriter formatting
    │   └── types.go                 ← PodRow and shared types
    └── threshold/
        └── threshold.go             ← near-limit warning logic — isolated and reusable

Each file has one reason to change, a design decision that will pay off in the next post when we add a namespace summary and JSON output without touching any of the existing files.

Here's the go.mod:

module github.com/FerRiosCosta/ferctl

go 1.26

require (
    github.com/spf13/cobra      v1.10.2
    k8s.io/api                  v0.35.1
    k8s.io/apimachinery         v0.35.1
    k8s.io/client-go            v0.35.1
    k8s.io/metrics              v0.35.1
)

Installing the dependencies

Before writing any code, initialize the module and install all dependencies. Run these commands inside your project directory:

mkdir ferctl && cd ferctl
go mod init github.com/FerRiosCosta/ferctl

# Install Cobra, the CLI framework
go get github.com/spf13/cobra@v1.10.2

# Install the Kubernetes packages, all pinned to match your server version
go get k8s.io/client-go@v0.35.1
go get k8s.io/api@v0.35.1
go get k8s.io/apimachinery@v0.35.1

# Install the metrics package, separate from client-go, explained below
go get k8s.io/metrics@v0.35.1

# Fetch all transitive dependencies and clean up
go get ./...
go mod tidy

What is go get ./...?

You'll notice we run go get ./... after installing the individual packages. Here's what it does and why it matters.

./... means "every package in this module, recursively", . is the current directory, and ... expands to all subdirectories.

go get ./...
      │
      ▼
Scans every .go file in your module
      │
      ▼
Finds every import, including transitive ones
(packages that your packages depend on, that you never imported directly)
      │
      ▼
Downloads missing packages and adds them to go.mod and go.sum

When you run individual go get commands, Go downloads those specific packages, but doesn't always fetch everything they depend on internally. k8s.io/client-go itself imports k8s.io/klog/v2, github.com/spf13/pflag, golang.org/x/net, and many others. Those transitive dependencies might not be in your go.sum yet.

Without go get ./... you might see errors like:

missing go.sum entry for module providing package k8s.io/klog/v2

That's k8s.io/klog/v2, a logging package that k8s.io/client-go uses internally. You never imported it directly, but Go needs its checksum in go.sum to verify the build. go get ./... catches it automatically.

What is go mod tidy?

go mod tidy and go get ./... are often confused because they're usually run together. They do different things:

go get ./...   → ADDS missing dependencies
               → fetches everything your code needs

go mod tidy    → CLEANS UP
               → removes dependencies that are no longer used
               → verifies go.sum is complete and consistent

Think of it this way:

go get ./... — "make sure everything I need is downloaded"
go mod tidy — "remove anything I don't need and make sure go.sum is complete"

Running both together is the safest way to ensure your module is in a clean, consistent state. Always run go mod tidy last; it's the final cleanup step.

You don't need to run go mod tidy immediately; better to have all your code written and then run it.

Why is k8s.io/metrics a separate package?

The k8s.io/metrics package is not part of k8s.io/client-go. It's a separate package because metrics-server is an extension API, not a core Kubernetes API. The Kubernetes API server doesn't collect resource usage itself; it delegates that to metrics-server, which exposes its data through a separate API endpoint. That's why it needs its own client package.

Before writing any code, verify metrics-server is running and collecting data:

kubectl top pods -n default

# NAME                      CPU(cores)   MEMORY(bytes)
# go-api-7d6b9f8c4-xk2pq   2m           11Mi

If you see error: Metrics API not available, metrics-server isn't ready yet. Wait 60 seconds and try again. If it's not installed, enable it:

minikube addons enable metrics-server

The entry point

// cmd/ferctl/main.go
package main

import "github.com/FerRiosCosta/ferctl/internal/top"

// main is intentionally minimal, all wiring happens in internal packages.
func main() {
    top.Execute()
}

The Kubernetes client

The KubeClient interface is the same pattern from post 3. We need one additional method, ListPods to fetch resource requests and limits from pod specs:

// internal/kubernetes/client.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
)

// KubeClient defines the Kubernetes operations ferctl uses.
// Wrapping the Clientset behind an interface keeps ferctl testable
// without a real cluster.
type KubeClient interface {
    // ListPods returns all pods in the given namespace.
    // An empty namespace string returns pods from all namespaces.
    ListPods(ctx context.Context, namespace string) (*corev1.PodList, error)
}

// client is the real implementation backed by the official Kubernetes Clientset.
// Unexported, callers always use the KubeClient interface.
type client struct {
    clientset kubernetes.Interface
}

// NewClient creates a KubeClient from the kubeconfig at the given path.
// Tries in-cluster config first, then falls back to kubeconfig file.
func NewClient(kubeconfigPath string) (KubeClient, error) {
    config, err := buildConfig(kubeconfigPath)
    if err != nil {
        return nil, fmt.Errorf("build kubeconfig: %w", err)
    }
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create clientset: %w", err)
    }
    return &client{clientset: clientset}, nil
}

// buildConfig tries in-cluster config first (inside a pod), then
// falls back to the kubeconfig file on disk (local development).
func buildConfig(kubeconfigPath string) (*rest.Config, error) {
    if config, err := rest.InClusterConfig(); err == nil {
        return config, nil
    }
    return clientcmd.BuildConfigFromFlags("", kubeconfigPath)
}

func (c *client) ListPods(ctx context.Context, namespace string) (*corev1.PodList, error) {
    pods, err := c.clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pods namespace=%q: %w", namespace, err)
    }
    return pods, nil
}

The metrics client

The metrics client queries the metrics-server API for real-time CPU and memory usage. We define our own PodMetrics type rather than exposing the raw metrics API type, this decouples the rest of ferctl from the metrics API's exact structure:

// internal/kubernetes/metrics.go
package kubernetes

import (
    "context"
    "fmt"

    "k8s.io/apimachinery/pkg/api/resource"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    metrics "k8s.io/metrics/pkg/client/clientset/versioned"
)

// PodMetrics holds real-time resource usage for a single pod.
// We define our own type here rather than exposing the raw metrics API
// type, this decouples the rest of ferctl from the metrics API's
// exact structure and makes the fake trivial to build.
type PodMetrics struct {
    Name      string
    Namespace string
    CPU       resource.Quantity // current CPU usage in millicores
    Memory    resource.Quantity // current memory usage in bytes
}

// MetricsClient defines the metrics-server operations ferctl uses.
type MetricsClient interface {
    // ListPodMetrics returns current CPU and memory usage for all pods
    // in the given namespace. An empty namespace returns all namespaces.
    ListPodMetrics(ctx context.Context, namespace string) ([]PodMetrics, error)
}

// metricsClient is the real implementation backed by the metrics-server API.
type metricsClient struct {
    clientset metrics.Interface
}

// NewMetricsClient creates a MetricsClient connected to the metrics-server.
func NewMetricsClient(kubeconfigPath string) (MetricsClient, error) {
    config, err := buildConfig(kubeconfigPath)
    if err != nil {
        return nil, fmt.Errorf("build kubeconfig for metrics: %w", err)
    }
    clientset, err := metrics.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create metrics clientset: %w", err)
    }
    return &metricsClient{clientset: clientset}, nil
}

// ListPodMetrics fetches current resource usage from the metrics-server
// and maps it to our own PodMetrics type.
func (m *metricsClient) ListPodMetrics(ctx context.Context, namespace string) ([]PodMetrics, error) {
    list, err := m.clientset.MetricsV1beta1().PodMetricses(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pod metrics namespace=%q: %w", namespace, err)
    }

    result := make([]PodMetrics, 0, len(list.Items))
    for _, item := range list.Items {
        // Sum CPU and memory across all containers in the pod.
        // A pod can have multiple containers, sidecars, init containers,
        // and we want the total resource usage, not just the main container.
        var totalCPU, totalMemory resource.Quantity
        for _, container := range item.Containers {
            totalCPU.Add(*container.Usage.Cpu())
            totalMemory.Add(*container.Usage.Memory())
        }

        result = append(result, PodMetrics{
            Name:      item.Name,
            Namespace: item.Namespace,
            CPU:       totalCPU,
            Memory:    totalMemory,
        })
    }

    return result, nil
}

The threshold package

Warning logic lives in its own package; isolated, independently testable, and reusable across any future ferctl command that needs near-limit detection:

// internal/threshold/threshold.go
package threshold

import "k8s.io/apimachinery/pkg/api/resource"

// DefaultWarningPercent is the usage percentage above which a pod
// is considered near its limit and should be flagged.
const DefaultWarningPercent = 80

// Status represents how close a pod's usage is to its configured limit.
type Status int

const (
    StatusOK       Status = iota // usage is comfortably below the limit
    StatusWarning                // usage is above the warning threshold
    StatusCritical               // usage is at or above 95% of the limit
)

// String returns a human-readable label for display in the output table.
func (s Status) String() string {
    switch s {
    case StatusWarning:
        return "!  WARNING"
    case StatusCritical:
        return "!! CRITICAL"
    default:
        return "   OK"
    }
}

// Check returns the Status of a resource given its current usage and limit.
// If the limit is zero (not set), returns StatusOK, we can't warn without
// a limit to compare against.
func Check(usage, limit resource.Quantity, warningPercent int) Status {
    if limit.IsZero() {
        return StatusOK
    }

    usageVal := usage.MilliValue()
    limitVal := limit.MilliValue()

    if limitVal == 0 {
        return StatusOK
    }

    percent := (usageVal * 100) / limitVal

    switch {
    case percent >= 95:
        return StatusCritical
    case percent >= int64(warningPercent):
        return StatusWarning
    default:
        return StatusOK
    }
}

// Percent returns usage as a percentage of the limit.
// Returns 0 if the limit is zero; avoids division by zero.
func Percent(usage, limit resource.Quantity) int64 {
    if limit.IsZero() {
        return 0
    }

    usageVal := usage.MilliValue()
    limitVal := limit.MilliValue()

    if limitVal == 0 {
        return 0
    }

    return (usageVal * 100) / limitVal
}

The shared types

// internal/top/types.go
package top

import (
    "k8s.io/apimachinery/pkg/api/resource"

    "github.com/FerRiosCosta/ferctl/internal/threshold"
)

// PodRow represents one row in the ferctl top output table.
// It's the intermediate data type between data fetching (runner.go)
// and formatting (output.go). Keeping it separate means the output
// layer can be swapped, tabwriter today, JSON in post 6, without
// touching the runner at all.
type PodRow struct {
    Namespace  string
    Name       string
    CPUUsage   resource.Quantity
    CPURequest resource.Quantity
    CPULimit   resource.Quantity
    CPUPercent int64
    CPUStatus  threshold.Status
    MemUsage   resource.Quantity
    MemRequest resource.Quantity
    MemLimit   resource.Quantity
    MemPercent int64
    MemStatus  threshold.Status
}

The runner, correlating pods and metrics

The runner is the business logic layer. It fetches pod specs and metrics independently, correlates them by namespace and name, and returns a slice of PodRow ready for display. It knows nothing about Cobra or tabwriter, just data in, data out:

// internal/top/runner.go
package top

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/resource"

    k8s "github.com/FerRiosCosta/ferctl/internal/kubernetes"
    "github.com/FerRiosCosta/ferctl/internal/threshold"
)

// Config holds everything the runner needs to execute.
// Using a struct instead of individual parameters makes it easy
// to add new options in the future without changing the function signature.
type Config struct {
    Namespace      string
    AllNamespaces  bool
    WarningPercent int
}

// Runner fetches and correlates Kubernetes resource data.
// It depends on interfaces, never on concrete client-go types,
// so it works identically with real clients and fakes.
type Runner struct {
    kube    k8s.KubeClient
    metrics k8s.MetricsClient
}

// NewRunner creates a Runner with its dependencies injected.
func NewRunner(kube k8s.KubeClient, metrics k8s.MetricsClient) *Runner {
    return &Runner{kube: kube, metrics: metrics}
}

// Run fetches pod specs and metrics, correlates them, and returns
// a slice of PodRows ready for the output formatter.
func (r *Runner) Run(ctx context.Context, cfg Config) ([]PodRow, error) {
    namespace := cfg.Namespace
    if cfg.AllNamespaces {
        // Empty string means all namespaces in the Kubernetes API.
        namespace = ""
    }

    // Fetch pod specs, gives us requests and limits.
    pods, err := r.kube.ListPods(ctx, namespace)
    if err != nil {
        return nil, fmt.Errorf("fetch pods: %w", err)
    }

    // Fetch live resource usage from metrics-server.
    podMetrics, err := r.metrics.ListPodMetrics(ctx, namespace)
    if err != nil {
        return nil, fmt.Errorf("fetch metrics: %w", err)
    }

    // Build a lookup map from pod metrics for O(1) correlation.
    // Key format: "namespace/name", uniquely identifies a pod in the cluster.
    metricsMap := make(map[string]k8s.PodMetrics, len(podMetrics))
    for _, m := range podMetrics {
        key := m.Namespace + "/" + m.Name
        metricsMap[key] = m
    }

    rows := make([]PodRow, 0, len(pods.Items))
    for _, pod := range pods.Items {
        // Skip non-running pods, they won't have metrics from metrics-server.
        if pod.Status.Phase != corev1.PodRunning {
            continue
        }

        // Aggregate requests and limits across all containers in the pod.
        cpuReq, cpuLim, memReq, memLim := aggregateResources(&pod)

        row := PodRow{
            Namespace:  pod.Namespace,
            Name:       pod.Name,
            CPURequest: cpuReq,
            CPULimit:   cpuLim,
            MemRequest: memReq,
            MemLimit:   memLim,
        }

        // Look up live metrics for this pod.
        // Newly started pods may not have metrics yet, handle gracefully
        // by leaving usage fields at their zero values.
        key := pod.Namespace + "/" + pod.Name
        if m, ok := metricsMap[key]; ok {
            row.CPUUsage   = m.CPU
            row.CPUPercent = threshold.Percent(m.CPU, cpuLim)
            row.CPUStatus  = threshold.Check(m.CPU, cpuLim, cfg.WarningPercent)
            row.MemUsage   = m.Memory
            row.MemPercent = threshold.Percent(m.Memory, memLim)
            row.MemStatus  = threshold.Check(m.Memory, memLim, cfg.WarningPercent)
        }

        rows = append(rows, row)
    }

    return rows, nil
}

// aggregateResources sums CPU and memory requests and limits across
// all containers in a pod, including sidecars and init containers.
// This matches how Kubernetes accounts for pod resource usage.
func aggregateResources(pod *corev1.Pod) (cpuReq, cpuLim, memReq, memLim resource.Quantity) {
    for _, container := range pod.Spec.Containers {
        res := container.Resources
        if req := res.Requests.Cpu(); req != nil {
            cpuReq.Add(*req)
        }
        if lim := res.Limits.Cpu(); lim != nil {
            cpuLim.Add(*lim)
        }
        if req := res.Requests.Memory(); req != nil {
            memReq.Add(*req)
        }
        if lim := res.Limits.Memory(); lim != nil {
            memLim.Add(*lim)
        }
    }
    return
}

The output formatter

// internal/top/output.go
package top

import (
    "fmt"
    "io"
    "text/tabwriter"

    "k8s.io/apimachinery/pkg/api/resource"

    "github.com/FerRiosCosta/ferctl/internal/threshold"
)

// PrintTable writes the pod rows as an aligned table to w.
// Accepting io.Writer instead of os.Stdout directly means this
// function is testable, tests can pass a bytes.Buffer and inspect output.
func PrintTable(w io.Writer, rows []PodRow) {
    // tabwriter aligns columns by padding with spaces.
    // Arguments: output, minwidth, tabwidth, padding, padchar, flags.
    tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
    defer tw.Flush()

    // Print the header row, \t is the column separator tabwriter uses.
    fmt.Fprintln(tw, "NAMESPACE\tNAME\tCPU USE\tCPU REQ\tCPU LIM\tCPU%\tMEM USE\tMEM REQ\tMEM LIM\tMEM%\tSTATUS")

    for _, row := range rows {
        // The overall row status is the worse of CPU and memory status.
        // If either is CRITICAL, the whole row is CRITICAL.
        status := worstStatus(row.CPUStatus, row.MemStatus)

        fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%d%%\t%s\t%s\t%s\t%d%%\t%s\n",
            row.Namespace,
            row.Name,
            formatQuantity(row.CPUUsage, "m"),    // CPU in millicores
            formatQuantity(row.CPURequest, "m"),
            formatQuantity(row.CPULimit, "m"),
            row.CPUPercent,
            formatQuantity(row.MemUsage, "Mi"),   // memory in mebibytes
            formatQuantity(row.MemRequest, "Mi"),
            formatQuantity(row.MemLimit, "Mi"),
            row.MemPercent,
            status.String(),
        )
    }
}

// worstStatus returns the more severe of two threshold statuses.
// CRITICAL > WARNING > OK, we always surface the worst condition.
func worstStatus(a, b threshold.Status) threshold.Status {
    if a > b {
        return a
    }
    return b
}

// formatQuantity converts a resource.Quantity to a readable string.
// unit controls the suffix: "m" for millicores, "Mi" for mebibytes.
func formatQuantity(q resource.Quantity, unit string) string {
    switch unit {
    case "m":
        return fmt.Sprintf("%dm", q.MilliValue())
    case "Mi":
        // Convert bytes to mebibytes (1 MiB = 1024 * 1024 bytes)
        return fmt.Sprintf("%dMi", q.Value()/(1024*1024))
    default:
        return q.String()
    }
}

The Cobra command

In post 4 we built the full Cobra foundation, root command, persistent flags, RunE, SilenceUsage, and the init() wiring pattern. This file follows exactly the same structure. If anything here looks unfamiliar, post 4 covers it in depth.

The key difference from the k8s-info example in post 4 is that ferctl has two clients to initialize (Kubernetes and metrics) and passes them into a dedicated runner rather than doing the work inline in RunE. The Cobra layer stays thin; it only handles flags and wiring.

// internal/top/command.go
package top

import (
    "context"
    "fmt"
    "os"

    "github.com/spf13/cobra"

    k8s "github.com/FerRiosCosta/ferctl/internal/kubernetes"
    "github.com/FerRiosCosta/ferctl/internal/threshold"
)

// flags holds the parsed flag values across all subcommands.
// Using a struct keeps flag handling tidy as the CLI grows.
type flags struct {
    namespace      string
    allNamespaces  bool
    kubeconfig     string
    warningPercent int
}

var f flags

// rootCmd is the base command, every subcommand is registered on it.
var rootCmd = &cobra.Command{
    Use:   "ferctl",
    Short: "Platform tooling for your Kubernetes cluster",
    Long:  "ferctl wraps common Kubernetes workflows into team-friendly commands.",
    // SilenceUsage prevents Cobra from printing the full usage text on
    // every error, covered in post 4. Add it to every root command.
    SilenceUsage: true,
}

// topCmd is the "ferctl top" subcommand.
var topCmd = &cobra.Command{
    Use:   "top",
    Short: "Show resource usage vs requests and limits",
    Long: `Show CPU and memory usage alongside configured requests and limits
for every running pod. Flags pods approaching their limits.`,
    Example: `  ferctl top -n default
  ferctl top --all-namespaces
  ferctl top -n default --warning-percent 70`,
    RunE: func(cmd *cobra.Command, args []string) error {
        ctx := context.Background()

        // Build the Kubernetes client, real implementation using kubeconfig.
        kube, err := k8s.NewClient(f.kubeconfig)
        if err != nil {
            return fmt.Errorf("connect to cluster: %w", err)
        }

        // Build the metrics client, connects to metrics-server.
        metricsClient, err := k8s.NewMetricsClient(f.kubeconfig)
        if err != nil {
            return fmt.Errorf("connect to metrics-server: %w", err)
        }

        // Inject both clients into the runner, business logic is
        // completely decoupled from how clients are created.
        runner := NewRunner(kube, metricsClient)

        rows, err := runner.Run(ctx, Config{
            Namespace:      f.namespace,
            AllNamespaces:  f.allNamespaces,
            WarningPercent: f.warningPercent,
        })
        if err != nil {
            return err
        }

        if len(rows) == 0 {
            fmt.Println("no running pods found")
            return nil
        }

        PrintTable(os.Stdout, rows)
        return nil
    },
}

func init() {
    // --kubeconfig is a persistent flag, inherited by all subcommands.
    // Defaults to the standard kubectl location (~/.kube/config).
    // You almost never need to type it, the default handles it automatically.
    rootCmd.PersistentFlags().StringVar(
        &f.kubeconfig,
        "kubeconfig",
        os.Getenv("HOME")+"/.kube/config",
        "path to kubeconfig file",
    )

    // Flags specific to the top subcommand, local flags, not persistent.
    topCmd.Flags().StringVarP(&f.namespace, "namespace", "n", "default", "namespace to query")
    topCmd.Flags().BoolVar(&f.allNamespaces, "all-namespaces", false, "query all namespaces")
    topCmd.Flags().IntVar(
        &f.warningPercent,
        "warning-percent",
        threshold.DefaultWarningPercent,
        "usage % above which to show a warning (default 80)",
    )

    rootCmd.AddCommand(topCmd)
}

// Execute is the public entry point called from main.go.
func Execute() {
    if err := rootCmd.Execute(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

The Makefile

Post 4 covered what a Makefile is, how targets and dependencies work, and the tab character rule. This Makefile follows the same pattern, the only addition is a test target and the git version embedding introduced in post 4's Makefile:

BINARY  := ferctl
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")

.PHONY: build install test clean

# -X embeds the git version into the binary at build time, same pattern as post 4.
build:
    go build \
      -ldflags="-s -w -X github.com/FerRiosCosta/ferctl/internal/top.Version=$(VERSION)" \
      -o $(BINARY) \
      ./cmd/ferctl/...

# install depends on build — runs build first automatically.
install: build
    cp $(BINARY) /usr/local/bin/$(BINARY)
    @echo "installed $(BINARY) to /usr/local/bin"

# test runs the full test suite.
# -v: verbose output  -count=1: no caching  -race: detect race conditions.
test:
    go test ./... -v -count=1 -race

clean:
    rm -f $(BINARY)

Build and install:

make install

# Verify it's available
ferctl --help

Verifying against minikube

Healthy cluster

ferctl top -n default

NAMESPACE   NAME                      CPU USE  CPU REQ  CPU LIM  CPU%  MEM USE  MEM REQ  MEM LIM  MEM%  STATUS
default     go-api-7d6b9f8c4-xk2pq   2m       50m      100m     2%    11Mi     32Mi     64Mi     17%      OK
default     go-api-7d6b9f8c4-mn9rt   2m       50m      100m     2%    10Mi     32Mi     64Mi     15%      OK
default     go-api-7d6b9f8c4-p8wvz   2m       50m      100m     2%    11Mi     32Mi     64Mi     17%      OK

All namespaces

ferctl top --all-namespaces

NAMESPACE     NAME                                CPU USE  CPU REQ  CPU LIM  CPU%  MEM USE  MEM REQ  MEM LIM  MEM%  STATUS
default       go-api-7d6b9f8c4-xk2pq             2m       50m      100m     2%    11Mi     32Mi     64Mi     17%      OK
kube-system   coredns-5dd5756b68-7wvzp            4m       100m     0m       0%    14Mi     70Mi     170Mi    8%       OK
kube-system   metrics-server-7746f77f97-xswk9     4m       100m     0m       0%    20Mi     200Mi    0Mi      0%       OK

Triggering a warning

Use the memory-hog.yaml from post 4 to create a pod that's under memory pressure:

kubectl apply -f memory-hog.yaml

# Wait for CrashLoopBackOff
kubectl get pods -w

Lower the warning threshold to catch it:

ferctl top -n default --warning-percent 50

NAMESPACE   NAME                      CPU USE  CPU REQ  CPU LIM  CPU%  MEM USE  MEM REQ  MEM LIM  MEM%  STATUS
default     go-api-7d6b9f8c4-xk2pq   2m       50m      100m     2%    11Mi     32Mi     64Mi     17%      OK
default     memory-hog                8m       0m       10Mi     0%    9Mi      10Mi     10Mi     90%   !! CRITICAL

memory-hog is at 90% of its memory limit, ferctl top catches it immediately as CRITICAL. The go-api pods show as OK.

Clean up:

kubectl delete pod memory-hog

Common gotchas

metrics-server not running

If metrics-server isn't available, ferctl top returns a clear error immediately:

Error: fetch metrics: list pod metrics namespace="default":
the server is currently unable to handle the request

Enable it on minikube and wait 60 seconds:

minikube addons enable metrics-server

Newly started pods have no metrics

metrics-server scrapes data on a ~60 second interval. A pod that started less than a minute ago won't appear in metrics output. ferctl top handles this gracefully, it shows the pod with its requests and limits but leaves CPU% and MEM% at 0% rather than panicking on a missing key.

Metrics lag under burst traffic

metrics-server shows a point-in-time snapshot, not an average. A pod that just spiked and recovered may look fine by the time you check. For trending data over time, Prometheus with container_memory_working_set_bytes is more reliable. ferctl top is for current state visibility, not historical analysis.

Multi-container pods, sidecars count

ferctl top sums CPU and memory across all containers in a pod, including sidecars like logging agents or service mesh proxies. If a sidecar is consuming unexpected resources, it shows up in the totals. Use kubectl top pod <name> --containers to see per-container breakdown.

Pods without limits show 0%

If a pod has no memory limit set (limits.memory not configured), ferctl top shows 0Mi for MEM LIM and 0% for MEM%, not a division error. This is intentional: we can't calculate a percentage without a known limit. This is also a signal, pods without limits set are BestEffort QoS class and are evicted first under node pressure (as covered in post 1).

Summary

ferctl top is what the series has been building toward; the Kubernetes theory from post 1, the local cluster from post 2, the client-go patterns from post 3, and the Cobra CLI foundation from post 4, all combined into a single tool with real operational value.

Three things to take away:

The PodRow intermediate type is what decouples the runner from the output layer, swapping tabwriter for JSON in post 6 is a 20-line change because of this design decision
The threshold package is isolated for a reason, reusable across any future ferctl command that needs near-limit detection
Pods without limits show 0%, which is itself a useful signal that resource configuration is incomplete

What's next

In post 6 we extend ferctl with two new features: a namespace-level resource summary for capacity planning, and --output=json for CI pipelines and scripting. The project structure we built here handles both additions without touching any existing files.

Let's connect

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, platform engineering, jazz, or jiu-jitsu. I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Building Go CLIs with Cobra: a practical guide

Fer Rios — Tue, 21 Jul 2026 13:19:01 +0000

Series: Platform engineering with Go | Topics: Go, Cobra, CLI, Platform Engineering

This is part of the Platform Engineering with Go series. This post builds on the client-go patterns from post 3. Read post 3 first if you haven't yet.

kubectl, helm, and the GitHub CLI all have something in common

They're all built with Cobra.

Cobra is the Go library for building command-line tools. It handles subcommands, flags, help text, shell completions, and argument validation, everything you need to turn a Go program into a proper CLI that behaves the way engineers expect a CLI to behave.

If you've used kubectl get pods, helm install, or gh pr create, you've already used a Cobra CLI without knowing it. In this post, you'll learn how to build one.

By the end, you'll have a working CLI called k8s-info with two subcommands; version and cluster.

Prerequisites

Read posts 1–3 of this series (helpful but not required for this post)

Project structure

As always, we start our project by creating the structure. If you don't know how and why this structure is created, you can learn it here.

k8s-info/
├── go.mod
├── Makefile
├── cmd/
│   └── k8s-info/
│       └── main.go              ← 3 lines — just calls Execute()
└── internal/
    └── cmd/
        ├── root.go              ← root command + persistent flags
        ├── version.go           ← "k8s-info version" subcommand
        └── cluster.go           ← "k8s-info cluster" subcommand

This structure scales. When we build our next command in the next post, we add more files under internal/cmd/, the root command never changes.

You can find the whole project code here

Initialize (with your own module name), and install:

mkdir k8s-info && cd k8s-info
go mod init github.com/yourname/k8s-info

## Install Cobra — the CLI framework
go get github.com/spf13/cobra@v1.10.2

# Install the Kubernetes packages needed for the cluster subcommand
# All three must be pinned to the same version matching your server version
# Since our minikube server is v1.35.1, we use v0.35.1
go get k8s.io/client-go@v0.35.1
go get k8s.io/api@v0.35.1
go get k8s.io/apimachinery@v0.35.1

# Fetch all transitive dependencies and clean up
go get ./...
go mod tidy

What Cobra actually is

Before writing any code, it helps to understand how Cobra thinks about a CLI.

Every Cobra CLI is a tree of commands:

k8s-info                     ← root command
├── version                  ← subcommand
└── cluster                  ← subcommand
    ├── --namespace           ← local flag (only for cluster)
    └── --output              ← local flag (only for cluster)

The root command is the entry point, it's what runs when you type just k8s-info. Subcommands are registered on the root with AddCommand(). Flags can be attached to any command, and persistent flags (attached to the root) are inherited by every subcommand.

Cobra handles everything else automatically: --help, error messages, usage text, and shell completion stubs.

The entry point

// cmd/k8s-info/main.go
package main

import "github.com/FerRiosCosta/k8s-info/internal/cmd"

// main is intentionally minimal, all command logic lives in internal/cmd/.
// This keeps the entry point clean and the business logic testable.
func main() {
    cmd.Execute()
}

Three lines. If your main.go is longer than this, something belongs in a package.

The root command

// internal/cmd/root.go
package cmd

import (
    "fmt"
    "os"

    "github.com/spf13/cobra"
)

// kubeconfigPath holds the value of the --kubeconfig persistent flag.
// Declared at package level so every subcommand can read it.
var kubeconfigPath string

// rootCmd is the base command, the one that runs when you type just "k8s-info".
// Every subcommand is registered on it via AddCommand().
var rootCmd = &cobra.Command{
    // Use is the one-line usage shown in help text.
    // The first word must be the binary name.
    Use: "k8s-info",

    // Short is the brief description shown in parent command help.
    Short: "A CLI for inspecting your Kubernetes cluster",

    // Long is the full description shown when you run "k8s-info --help".
    Long: `k8s-info is a platform engineering CLI for inspecting
Kubernetes cluster state. It connects to whatever cluster
your kubeconfig is currently pointing at.`,

    // SilenceUsage prevents Cobra from printing the full usage text
    // on every error, which gets noisy fast. Errors are still shown.
    SilenceUsage: true,
}

func init() {
    // PersistentFlags are inherited by every subcommand automatically.
    // --kubeconfig is something every subcommand might need, so it
    // belongs on the root as a persistent flag.
    rootCmd.PersistentFlags().StringVar(
        &kubeconfigPath,
        "kubeconfig",
        os.Getenv("HOME")+"/.kube/config", // default: ~/.kube/config
        "path to the kubeconfig file",
    )
}

// Execute is the single public entry point called from main.go.
// It parses os.Args, finds the right command, and runs it.
func Execute() {
    if err := rootCmd.Execute(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

Why SilenceUsage matters

By default, Cobra prints the full usage text every time an error occurs. This is often more confusing than helpful; the user sees a wall of text when what they really need is just the error message. SilenceUsage: true on the root command suppresses usage on errors for the entire CLI.

Adding subcommands

version subcommand

// internal/cmd/version.go
package cmd

import (
    "fmt"

    "github.com/spf13/cobra"
)

// Version is set at build time using ldflags.
// Default to "dev" so it's always meaningful, even without a build.
var Version = "dev"

// versionCmd prints the current version of k8s-info.
var versionCmd = &cobra.Command{
    Use:   "version",
    Short: "Print the k8s-info version",

    // Run is used when the command cannot fail.
    // If your command can return an error, use RunE instead (shown below).
    Run: func(cmd *cobra.Command, args []string) {
        fmt.Printf("k8s-info version %s\n", Version)
    },
}

func init() {
    // Register versionCmd on the root command.
    // After this, "k8s-info version" is a valid command.
    rootCmd.AddCommand(versionCmd)
}

cluster subcommand

// internal/cmd/cluster.go
package cmd

import (
    "context"
    "fmt"
    "os"

    "github.com/spf13/cobra"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

// namespace is a local flag, only available on the cluster subcommand,
// not on version or any future subcommand.
var namespace string

// clusterCmd shows a summary of what's running in the cluster.
var clusterCmd = &cobra.Command{
    Use:   "cluster",
    Short: "Show a summary of running pods in your cluster",
    Example: `  k8s-info cluster
  k8s-info cluster -n kube-system
  k8s-info cluster --namespace default`,

    // RunE is the idiomatic choice when a command can fail.
    // It returns an error, Cobra prints it and exits with code 1.
    // Never call os.Exit inside RunE, let Cobra handle it.
    RunE: func(cmd *cobra.Command, args []string) error {
        // Build the Kubernetes config from the kubeconfig flag.
        // kubeconfigPath is inherited from the root persistent flag.
        config, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
        if err != nil {
            return fmt.Errorf("connect to cluster: %w", err)
        }

        clientset, err := kubernetes.NewForConfig(config)
        if err != nil {
            return fmt.Errorf("create client: %w", err)
        }

        // List pods in the given namespace.
        pods, err := clientset.CoreV1().Pods(namespace).List(
            context.Background(),
            metav1.ListOptions{},
        )
        if err != nil {
            return fmt.Errorf("list pods namespace=%q: %w", namespace, err)
        }

        // Count pods by phase.
        var running, pending, failed int
        for _, pod := range pods.Items {
            switch pod.Status.Phase {
            case "Running":
                running++
            case "Pending":
                pending++
            case "Failed":
                failed++
            }
        }

        fmt.Printf("Namespace: %s\n", namespace)
        fmt.Printf("Total pods: %d\n", len(pods.Items))
        fmt.Printf("  Running: %d\n", running)
        fmt.Printf("  Pending: %d\n", pending)
        fmt.Printf("  Failed:  %d\n", failed)

        return nil
    },
}

func init() {
    // Local flags only apply to this subcommand.
    // --namespace with -n shorthand, defaults to "default".
    clusterCmd.Flags().StringVarP(
        &namespace,
        "namespace", "n",
        "default",
        "namespace to inspect",
    )

    rootCmd.AddCommand(clusterCmd)
}

Persistent flags vs local flags

This is one of the most important concepts in Cobra and worth understanding clearly before your CLI grows.

Do you always have to type --kubeconfig?

No, and this is the part that trips people up. Because --kubeconfig has a default value set:

rootCmd.PersistentFlags().StringVar(
    &kubeconfigPath,
    "kubeconfig",
    os.Getenv("HOME")+"/.kube/config", // ← default is already set here
    "path to the kubeconfig file",
)

...kubeconfigPath is automatically populated with ~/.kube/config every time any command runs. You never need to type --kubeconfig unless you want to point at a different file:

# Default kubeconfig, flag not needed at all
k8s-info version
k8s-info cluster -n default

# Non-default kubeconfig, only specify when you need a different path
k8s-info --kubeconfig /path/to/other/config version
k8s-info --kubeconfig /path/to/other/config cluster

The real difference between persistent and local

The distinction that actually matters is this:

Persistent flag, declared on rootCmd with PersistentFlags()
    ↓ inherited by ALL subcommands, available everywhere in the CLI
    k8s-info --kubeconfig /other/config version     ✓ works
    k8s-info --kubeconfig /other/config cluster     ✓ works
    k8s-info version --kubeconfig /other/config     ✓ also works (order flexible)

Local flag, declared on a specific command with Flags()
    ↓ only available on THAT specific subcommand
    k8s-info cluster -n kube-system                 ✓ works
    k8s-info version -n kube-system                 ✗ unknown flag: -n

The last two lines are what matters. -n is a local flag on cluster, it doesn't exist on version. If a user tries to pass it to version, Cobra rejects it with a clear error.

The rule

Simple: if every subcommand needs the flag (like --kubeconfig), make it persistent on the root. If only one subcommand needs it (like --namespace for cluster), make it local. When in doubt, start local; you can always promote a flag to persistent later, but removing a persistent flag is a breaking change for your users.

Run vs RunE

Cobra gives you two options for the command function:

// Run, use when the command cannot fail
Run: func(cmd *cobra.Command, args []string) {
    fmt.Println("hello")
},

// RunE, use when the command can return an error
RunE: func(cmd *cobra.Command, args []string) error {
    if err := doSomething(); err != nil {
        return fmt.Errorf("something failed: %w", err)
    }
    return nil
},

Always prefer RunE for anything that does I/O, file reads, network calls, Kubernetes API calls. The error is printed automatically by Cobra and the process exits with code 1. Never call os.Exit inside RunE; that bypasses any deferred cleanup and makes testing harder.

The Makefile

A Makefile is a build automation file; it defines named commands called targets that you run with make <target>. Instead of typing long go build commands with flags every time, you type make build and the Makefile handles the rest.

Every target follows the same basic anatomy:

target_name: optional_dependency
    command to run

The dependency means "run this target first before mine." If install depends on build, running make install automatically runs make build first, you never have to remember the order.

Two rules that trip up everyone writing their first Makefile:

Commands must be indented with a real tab character: spaces look identical but Make rejects them with a cryptic error
.PHONY tells Make these are command names, not filenames: without it, Make looks for a file called build or install and gets confused when it doesn't find one

Variables work like this:

BINARY := ferctl        # declare a variable
$(BINARY)               # reference it, expands to "ferctl"

The @ prefix on a command suppresses echoing that line to the terminal, only the output is shown, not the command itself.

Here's the k8s-info Makefile with every line explained:

# BINARY is the name of the output binary, change this and everything updates.
BINARY  := k8s-info

# VERSION reads the current git tag automatically at build time.
# git describe returns something like "v1.2.0-3-gabcdef" for tagged commits.
# If you're not in a git repo, it falls back to "dev".
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")

# .PHONY declares these as command names, not file names.
# Without this, Make would look for files called "build", "install", etc.
.PHONY: build install clean

# build compiles the binary for the current platform.
# -ldflags="-s -w" strips debug symbols, reduces binary size by ~30%.
# -X sets a Go package-level variable at build time (explained below).
# -o $(BINARY) names the output file.
build:
    go build \
      -ldflags="-s -w -X github.com/FerRiosCosta/k8s-info/internal/cmd.Version=$(VERSION)" \
      -o $(BINARY) \
      ./cmd/k8s-info/...

# install depends on build, "make install" runs "make build" first automatically.
# Then copies the binary to /usr/local/bin so it's available system-wide.
install: build
    cp $(BINARY) /usr/local/bin/$(BINARY)
    @echo "installed $(BINARY) to /usr/local/bin"
    # @ suppresses printing the echo command, only the output is shown.

# clean removes the compiled binary so you can start fresh.
clean:
    rm -f $(BINARY)

The -X ldflag is new here compared to previous posts. It sets a Go package-level variable at build time; in this case cmd.Version. When you run make build in a git repository, git describe automatically fills in the version tag. When you run it outside git, it falls back to "dev".

The targets work together in a natural workflow:

make build    # compile the binary
make install  # compile + install to /usr/local/bin (runs build first)
make clean    # remove the binary and start fresh

Understanding versions and git describe

The VERSION variable in the Makefile uses git describe to automatically read a version from your git history. To understand how this works, it helps to understand what a version actually is in a git project.

What is a version?

A version is a named point in your git history, a label that says "this is the code that shipped as v1.0.0". In git, this is called a tag. A tag is just a pointer to a specific commit, with a human-readable name.

Versions typically follow semantic versioning (semver):

v1.2.3
│ │ └─── patch: bug fixes, no new features
│ └───── minor: new features, backward compatible
└─────── major: breaking changes

Examples of what each bump means in practice:

v1.0.0  → first public release
v1.0.1  → fixed a bug in the cluster command
v1.1.0  → added a new "nodes" subcommand
v2.0.0  → renamed flags, changed output format (breaking)

How to create a version in git

First make sure your code is committed:

git add .
git commit -m "feat: add cluster subcommand"

Then create a tag:

# Create a tag pointing at the current commit
git tag v1.0.0

# Create a tag with an annotation message (recommended)
git tag -a v1.0.0 -m "First stable release of k8s-info"

# Push the tag to GitHub (tags dont push automatically with git push)
git push origin v1.0.0

# Or push all tags at once
git push origin --tags

How git describe uses your tags

git describe reads the most recent tag in your git history and builds a version string from it:

git describe --tags --always --dirty

The output depends on where you are in your commit history:

v1.0.0                    ← exactly on the tagged commit
v1.0.0-3-gabcdef          ← 3 commits after v1.0.0, current hash is abcdef
v1.0.0-3-gabcdef-dirty    ← same, but you have uncommitted changes
abcdef                    ← no tags exist yet, falls back to commit hash

The flags we use:

Flag	What it does
`--tags`	Considers all tags, not just annotated ones
`--always`	Falls back to the commit hash if no tags exist
`--dirty`	Appends `-dirty` if there are uncommitted changes

And 2>/dev/null || echo "dev" handles the case where git isn't available at all (like in a CI environment without git installed); it silently swallows the error and falls back to "dev".

What this means for your binary

When you run make build with a tagged commit:

git tag -a v1.0.0 -m "first release"
make build
./k8s-info version
# k8s-info version v1.0.0

When you run it after adding more commits:

git commit -m "fix: handle empty namespace"
make build
./k8s-info version
# k8s-info version v1.0.0-1-gabcdef

This tells you exactly which commit the binary was built from, invaluable when debugging a production issue and trying to figure out what version is actually running.

A practical workflow for releasing

# 1. Make sure everything is committed and tested
git status              # should show nothing uncommitted
make test               # all tests pass

# 2. Create the version tag
git tag -a v1.0.0 -m "Initial release of k8s-info"

# 3. Build the binary, version is embedded automatically
make build
./k8s-info version
# k8s-info version v1.0.0

# 4. Install it
make install

# 5. Push the tag to GitHub
git push origin v1.0.0

From this point on, anyone who clones your repo and runs make build on that commit gets a binary that reports v1.0.0. No manual version files to maintain, no risk of forgetting to update a constant; git is the source of truth.

Running the CLI

Build and install:

make install

Try the commands:

# Show help — generated automatically by Cobra
k8s-info --help

# Show version
k8s-info version
# k8s-info version dev

# Show cluster summary for the default namespace
k8s-info cluster

# Namespace: default
# Total pods: 3
# Running: 3
# Pending: 0
# Failed:  0

# Inspect kube-system
k8s-info cluster -n kube-system

# Namespace: kube-system
# Total pods: 7
# Running: 7
# Pending: 0
# Failed:  0

# Show help for the cluster subcommand
k8s-info cluster --help

What Cobra generates automatically

Run k8s-info --help and notice what you didn't have to write:

A CLI for inspecting your Kubernetes cluster

Usage:
  k8s-info [command]

Available Commands:
  cluster     Show a summary of running pods in your cluster
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  version     Print the k8s-info version

Flags:
  -h, --help                help for k8s-info
      --kubeconfig string   path to the kubeconfig file (default "/Users/yourname/.kube/config")

Use "k8s-info [command] --help" for more information about a command.

The completion and help subcommands, the --help flag, the usage text, the default values, all generated for free. That's what using Cobra buys you.

How init() wires everything together

You may have noticed that every file calls rootCmd.AddCommand() inside an init() function. This is Go's package initialization mechanism; init() runs automatically when the package is imported, before main().

main() calls cmd.Execute()
      │
      ▼
Go initializes the cmd package
      │
      ├── root.go   init() runs → registers --kubeconfig flag
      ├── version.go init() runs → registers versionCmd on root
      └── cluster.go init() runs → registers clusterCmd on root
      │
      ▼
rootCmd.Execute() runs with all subcommands registered

This is why you can add a new subcommand by creating a new file and calling rootCmd.AddCommand() in its init(), no other file needs to change. The root command picks it up automatically.

Common gotchas

Using Run instead of RunE for fallible commands

If your command calls a Kubernetes API or reads a file and you use Run instead of RunE, you have to handle errors yourself, usually with fmt.Fprintln(os.Stderr, err); os.Exit(1). This is verbose, inconsistent, and bypasses Cobra's error handling. Always use RunE for anything that can fail.

Forgetting SilenceUsage

Without SilenceUsage: true, every error prints the full usage text, which is almost never what you want when a flag value is wrong or a Kubernetes call fails. Add it to the root command and save your users the confusion.

Persistent flags vs local flags confusion

If you declare a flag with Flags() on the root command, it's still a local flag, only available directly on the root, not inherited by subcommands. Use PersistentFlags() on the root for flags that should be available everywhere.

Flag order matters to users

Cobra parses flags after the command name. This means:

k8s-info --kubeconfig ~/.kube/config cluster -n default   ✓ works
k8s-info cluster --kubeconfig ~/.kube/config -n default   ✓ also works
k8s-info cluster -n default --kubeconfig ~/.kube/config   ✓ also works

All three are equivalent; Cobra is flexible about flag position. But document a consistent order in your examples to avoid confusing users.

init() order is not guaranteed across files

Go guarantees that init() functions within a single file run in order, but the order between files in the same package is not guaranteed. This is fine for Cobra; AddCommand() calls are independent of each other. But don't write init() functions that depend on each other across files.

Summary

Cobra turns a Go program into a proper CLI in three concepts:

rootCmd: the entry point, owns persistent flags, registers subcommands
AddCommand(): how subcommands attach to the tree
RunE: the function that runs when a command is invoked, returns an error or nil

The init() pattern means adding a new subcommand is always: create a file, write the command, call rootCmd.AddCommand() in init(). Nothing else changes.

This is exactly the structure the next command uses in post 5; the root command, the persistent --kubeconfig flag, the top subcommand registered in its own file, all wired together by init().

What's next

In nex post we take this Cobra foundation and build ferctl top on top of it, adding the metrics-server client, resource threshold warnings, and tabwriter output. The k8s-info cluster command becomes the starting point for the richer ferctl top command.

Let's connect

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, platform engineering, jazz, or jiu-jitsu. I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Talking to Kubernetes from Go: a practical client-go guide

Fer Rios — Mon, 13 Jul 2026 13:34:48 +0000

This is part of the Platform Engineering with Go series. This post builds on the minikube cluster set up in post 2. Read post 2 first if you haven't yet.

kubectl is great for humans. But what about programs?

When you run kubectl get pods, something has to talk to the Kubernetes API server, deserialize the JSON response, and format it for your terminal. That something is client-go, the official Go client library for Kubernetes.

Every tool in the Kubernetes ecosystem is built on it. kubectl itself, helm, ArgoCD, controller-runtime, Tekton. If it talks to a Kubernetes cluster and it's written in Go, it's using client-go.

In this post, we'll learn how client-go works by building something immediately useful: a cluster health reporter. One Go program that connects to your minikube cluster, scans every pod across every namespace, and prints a complete health summary, the kind of view that usually takes four or five kubectl commands to assemble.

Cluster Health Report
Generated: 2026-06-24 14:32:00 UTC

NAMESPACE     TOTAL   RUNNING   FAILING   PENDING
──────────────────────────────────────────────────
default           3         3         0         0
kube-system       6         6         0         0

✓ All pods healthy

And if something is wrong:

Cluster Health Report
Generated: 2026-06-24 14:32:00 UTC

NAMESPACE     TOTAL   RUNNING   FAILING   PENDING
──────────────────────────────────────────────────
default           3         1         2         0
kube-system       6         6         0         0

FAILING PODS
──────────────────────────────────────────────────
default   go-api-7d6b9f8c4-mn9rt   CrashLoopBackOff   14 restarts
default   go-api-7d6b9f8c4-p8wvz   OOMKilled            3 restarts

What you'll learn

What client-go is and how it relates to the Kubernetes API.
The four key packages and what each one contains.
How to connect to a cluster, out-of-cluster and in-cluster.
How the Clientset is organized, API groups explained.
How to list, filter, and inspect Kubernetes resources from Go.
How to build the cluster health reporter step by step.
How to test client-go code, two approaches with clear rules for when to use each.

Prerequisites

minikube running with the go-api deployment from post 2

What client-go is

The Kubernetes API is a REST API. When you run kubectl get pods -n default, kubectl makes an HTTP GET request to something like:

GET https://kubernetes/api/v1/namespaces/default/pods

The API server returns JSON. kubectl deserializes it and formats it for your terminal.

client-go is the library that handles all of this in Go: the HTTP calls, TLS, authentication, rate limiting, and deserialization into typed Go structs. Instead of working with raw JSON, you get a *corev1.PodList with typed fields you can navigate in your editor with autocomplete.

Your Go program
      │
      │  client-go
      │  ├── kubeconfig loading
      │  ├── TLS + authentication
      │  ├── HTTP client + rate limiting
      │  └── JSON → typed Go structs
      │
      ▼
Kubernetes API server
      │
      ▼
etcd (the cluster's data store)

Everything you can do with kubectl, you can do with client-go, and more. kubectl is itself a client-go program.

Project structure

Here is the Go project structure we are going to build; remember, you can see how to organize the code for your own projects here.

You can find the entire project here.

go-k8s-health-reporter/
├── go.mod
├── main.go                  ← entry point — wires dependencies, calls reporter
└── internal/
    ├── reporter/
    │   ├── types.go         ← data types — PodStatus, NamespaceSummary, Report
    │   ├── reporter.go      ← Reporter struct, Generate()
    │   ├── inspector.go     ← pod health detection — inspectPod(), pendingReason()
    │   └── printer.go       ← output formatting — Print(), tabwriter tables
    └── kubernetes/
        └── client.go        ← KubeClient interface + real implementation

Installing the packages

Before writing any code, initialize the module and install the dependencies. The version numbers must match your Kubernetes cluster; since we're running Kubernetes 1.36.2, we use v0.36.2 for all three packages. Mixing versions causes compilation errors because the types won't match across packages. Remember to rename the module's name to your own module name.

# Initialize the Go module
mkdir go-k8s-health-reporter && cd go-k8s-health-reporter
go mod init github.com/FerRiosCosta/go-k8s-health-reporter

# Install the four packages pinned to Kubernetes 1.36.2
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.36.2
go get k8s.io/apimachinery@v0.36.2

# Tidy up — removes unused dependencies and ensures versions are consistent
go mod tidy

If you see a missing go.sum entry error after installing the packages, it means some transitive dependencies, packages that client-go depends on, like k8s.io/klog/v2, weren't added to go.sum automatically. Fix it with:

bash
go get ./...
go mod tidy
go get ./... scans all packages in your module and fetches every dependency recursively, including transitive ones. go mod tidy then writes the complete go.sum.

Why pin to v0.36.2 instead of @latest? The client-go versioning scheme directly mirrors Kubernetes releases; v0.36.2 corresponds to Kubernetes v1.36.2. Always match your client-go version to your cluster version. If you run @latest you might get a newer client-go version than your cluster supports, which can cause unexpected API behavior.

You can verify your cluster version at any time:

kubectl version

# Client Version: v1.36.2
# Server Version: v1.36.2

Both should match (ideally). If they don't, pin your packages to match the server version.

Note: k8s.io/metrics is not needed for the health reporter in this post, but you will need it for the next posts.
go get k8s.io/metrics@v0.36.2
go mod tidy

Here's the go.mod:

module github.com/FerRiosCosta/go-k8s-health-reporter

go 1.26

require (
    k8s.io/api          v0.36.2
    k8s.io/apimachinery v0.36.2
    k8s.io/client-go    v0.36.2
)

The four key packages

Before writing any code, let's understand the four packages client-go code almost always imports. This is the section that makes every future client-go import immediately readable.

k8s.io/api: the Go type definitions

import (
    corev1  "k8s.io/api/core/v1"   // Pod, Service, ConfigMap, Secret, Namespace, Node
    appsv1  "k8s.io/api/apps/v1"   // Deployment, StatefulSet, DaemonSet, ReplicaSet
    batchv1 "k8s.io/api/batch/v1"  // Job, CronJob
)

This package contains the Go struct definitions for every Kubernetes resource. When you write corev1.Pod{} or appsv1.Deployment{}, you're using this package. The aliases (corev1, appsv1) are conventions, the actual package names are just v1, which is too generic to use unaliased.

k8s.io/apimachinery: the shared building blocks

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" // ObjectMeta, ListOptions, Time
    "k8s.io/apimachinery/pkg/api/resource"         // resource.Quantity (CPU, memory)
)

This package contains types shared across all Kubernetes API groups. metav1.ObjectMeta is the block every resource has: name, namespace, labels, annotations, creation timestamp. metav1.ListOptions lets you filter API calls by label selectors and field selectors. You'll import something from this package in virtually every file that touches the Kubernetes API.

k8s.io/client-go: the actual client

import (
    "k8s.io/client-go/kubernetes"        // the Clientset — entry point to all APIs
    "k8s.io/client-go/rest"              // rest.Config — holds connection details
    "k8s.io/client-go/tools/clientcmd"   // reads and parses kubeconfig files
)

This is the main package. kubernetes.NewForConfig() creates the Clientset, your entry point to every Kubernetes API. clientcmd reads the ~/.kube/config file that minikube wrote in post 2. rest.Config holds the connection details: API server URL, TLS certificates, auth token.

k8s.io/metrics: the metrics-server API

import (
    metrics "k8s.io/metrics/pkg/client/clientset/versioned" // metrics Clientset
)

This is a separate package because metrics-server is an extension API, not part of the core Kubernetes API.

# Install in post 4 — pinned to the same version as the rest
go get k8s.io/metrics@v0.36.2
go mod tidy

Connecting to the cluster

Every client-go program starts the same way: build a rest.Config, then create a Clientset from it.

// internal/kubernetes/client.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
)

// KubeClient defines the Kubernetes operations the health reporter uses.
// Wrapping the Clientset behind an interface makes the reporter testable
// without a real cluster, the same pattern used throughout this series.
type KubeClient interface {
    // ListPods returns all pods in the given namespace.
    // An empty namespace string returns pods from all namespaces.
    ListPods(ctx context.Context, namespace string) (*corev1.PodList, error)
}

// client is the real implementation backed by the official Kubernetes Clientset.
// It is unexported, callers always use the KubeClient interface.
type client struct {
    clientset kubernetes.Interface
}

// NewClient creates a KubeClient connected to the cluster in kubeconfig.
func NewClient(kubeconfigPath string) (KubeClient, error) {
    config, err := buildConfig(kubeconfigPath)
    if err != nil {
        return nil, fmt.Errorf("build config: %w", err)
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create clientset: %w", err)
    }

    return &client{clientset: clientset}, nil
}

// buildConfig tries in-cluster config first, then falls back to kubeconfig.
// This means the same binary works both locally (reads ~/.kube/config)
// and when deployed as a pod inside a cluster (reads the mounted service account token).
func buildConfig(kubeconfigPath string) (*rest.Config, error) {
    // in-cluster config: reads /var/run/secrets/kubernetes.io/serviceaccount/token
    // automatically mounted inside every pod by Kubernetes.
    if config, err := rest.InClusterConfig(); err == nil {
        return config, nil
    }

    // out-of-cluster config: reads the kubeconfig file from disk.
    // This is the file minikube wrote to ~/.kube/config in post 2.
    return clientcmd.BuildConfigFromFlags("", kubeconfigPath)
}

// ListPods retrieves all pods in the given namespace from the Kubernetes API.
// Passing an empty string returns pods from all namespaces.
func (c *client) ListPods(ctx context.Context, namespace string) (*corev1.PodList, error) {
    pods, err := c.clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pods namespace=%q: %w", namespace, err)
    }
    return pods, nil
}

The Clientset and API groups

c.clientset.CoreV1().Pods(namespace).List(ctx, opts), let's break this chain down because you'll see it everywhere in client-go code:

c.clientset           // the Clientset, entry point to all Kubernetes APIs
    .CoreV1()         // select the "core" API group, version 1
    .Pods(namespace)  // select the Pods resource in the given namespace
    .List(ctx, opts)  // call the List operation with options

Kubernetes organizes resources into API groups. Each group contains related resources:

CoreV1()  → Pods, Services, ConfigMaps, Secrets, Nodes, Namespaces
AppsV1()  → Deployments, StatefulSets, DaemonSets, ReplicaSets
BatchV1() → Jobs, CronJobs

The pattern is always the same: clientset.APIGroup().ResourceType(namespace).Operation(). Once you know this pattern, any client-go call is readable.

Building the cluster health reporter

Now let's build the reporter. Instead of putting everything in one file, we'll split it across four focused files from the start, each with one clear responsibility.

internal/reporter/
├── types.go      ← data types only — PodStatus, NamespaceSummary, Report
├── reporter.go   ← Reporter struct, constructor, Generate()
├── inspector.go  ← pod inspection logic — detects failing and pending states
└── printer.go    ← output formatting — tabwriter tables

Each file has one reason to change:

New data field → types.go
New data source → reporter.go
New failure reason to detect → inspector.go
New output format (JSON, CSV) → printer.go

types.go, data types only

// internal/reporter/types.go
package reporter

import "time"

// PodStatus represents the health state of a single pod.
type PodStatus struct {
    Namespace string
    Name      string
    Phase     string // Running, Pending, Failed, Succeeded
    Reason    string // CrashLoopBackOff, OOMKilled, etc. — more specific than Phase
    Restarts  int32  // total restarts across all containers
    IsFailing bool   // true if the pod is in a non-running problem state
    IsPending bool   // true if the pod is waiting to be scheduled or started
}

// NamespaceSummary holds the pod counts for a single namespace.
type NamespaceSummary struct {
    Namespace string
    Total     int
    Running   int
    Failing   int
    Pending   int
}

// Report is the complete cluster health picture.
type Report struct {
    GeneratedAt time.Time
    Namespaces  []NamespaceSummary
    Failing     []PodStatus
    Pending     []PodStatus
}

Thirty lines. No imports beyond time. If you need to add a field to a pod's health state, you open this file and only this file.

inspector.go, pod health detection

// internal/reporter/inspector.go
package reporter

import corev1 "k8s.io/api/core/v1"

// inspectPod examines a pod's status and returns a detailed PodStatus.
// It checks container states first, a pod can be in "Running" phase
// while individual containers are in CrashLoopBackOff or OOMKilled.
func inspectPod(pod *corev1.Pod) PodStatus {
    status := PodStatus{
        Namespace: pod.Namespace,
        Name:      pod.Name,
        Phase:     string(pod.Status.Phase),
    }

    // Sum restart counts and check container states.
    // Must check containers before pod phase, a pod can be "Running"
    // while a container inside it is waiting to restart.
    for _, cs := range pod.Status.ContainerStatuses {
        status.Restarts += cs.RestartCount

        if cs.State.Waiting != nil {
            switch cs.State.Waiting.Reason {
            case "CrashLoopBackOff", "OOMKilled", "ImagePullBackOff",
                "ErrImagePull", "CreateContainerConfigError":
                status.Reason = cs.State.Waiting.Reason
                status.IsFailing = true
                return status
            }
        }

        if cs.State.Terminated != nil {
            reason := cs.State.Terminated.Reason
            if reason == "OOMKilled" || cs.State.Terminated.ExitCode != 0 {
                status.Reason = reason
                status.IsFailing = true
                return status
            }
        }
    }

    // Check pod-level phase for broader failure states.
    switch pod.Status.Phase {
    case corev1.PodFailed:
        status.Reason = pod.Status.Reason
        status.IsFailing = true
    case corev1.PodPending:
        status.Reason = pendingReason(pod)
        status.IsPending = true
    }

    return status
}

// pendingReason extracts why a pod is pending from its conditions.
// The PodScheduled condition contains the human-readable scheduling
// failure message; e.g. "0/1 nodes are available: 1 Insufficient memory."
func pendingReason(pod *corev1.Pod) string {
    for _, condition := range pod.Status.Conditions {
        if condition.Type == corev1.PodScheduled &&
            condition.Status == corev1.ConditionFalse {
            return condition.Message
        }
    }
    return string(pod.Status.Phase)
}

Understanding container states

The inspectPod function works by reading pod.Status.ContainerStatuses, a slice that contains one entry per container in the pod. Each entry has a State field that holds exactly one of three possible states at any given moment:

Waiting: the container is not yet running. This is where most failure reasons live:

Reason	What it means
`ContainerCreating`	Image is being pulled, or volumes are being mounted
`PodInitializing`	Init containers are still running, main container hasn't started
`CrashLoopBackOff`	Container keeps crashing, Kubernetes is backing off before restarting
`ImagePullBackOff`	Image pull failed, Kubernetes is backing off before retrying
`ErrImagePull`	The immediate image pull error, before backoff kicks in
`CreateContainerConfigError`	Bad config, missing ConfigMap, Secret, or invalid env var
`OOMKilled`	Killed by the OOM killer, waiting to restart

Running: the container process is alive. Note that this doesn't mean the application inside it is healthy, it just means the process is executing. Your app could still be returning 500s or stuck in a deadlock.

Terminated: the container has finished. The most important fields are:

Field	What it means
`ExitCode`	`0` = clean exit, anything else = failure
`Reason`	`Completed` (clean) or `OOMKilled` (killed by OOM killer)

Common exit codes worth knowing:

Exit code	Meaning
`0`	Clean exit, completed successfully
`1`	Application error, the process itself returned an error
`137`	`SIGKILL`, OOM killer fired, or forcefully killed
`143`	`SIGTERM`, graceful shutdown (normal during pod termination)

This is exactly why inspectPod checks cs.State.Waiting and cs.State.Terminated separately; each state carries different information in different fields.

Init containers, the hidden failure

There's one more status field worth knowing about:

pod.Status.ContainerStatuses      // current state of main containers
pod.Status.InitContainerStatuses  // current state of init containers

Init containers run before your main containers and go through the same three states. If an init container is stuck in CrashLoopBackOff, your main container will never start, and pod.Status.ContainerStatuses will be empty, giving you no indication of what's wrong.

The current inspectPod implementation only checks main container statuses. A more complete version would check init containers first:

// A more complete inspectPod that also catches stuck init containers.
// This is left as an exercise — the pattern is identical to the main
// container check, just reading from InitContainerStatuses instead.
func inspectPod(pod *corev1.Pod) PodStatus {
    status := PodStatus{
        Namespace: pod.Namespace,
        Name:      pod.Name,
        Phase:     string(pod.Status.Phase),
    }

    // Check init containers first — if one is stuck, main containers
    // will never start and ContainerStatuses will be empty.
    for _, cs := range pod.Status.InitContainerStatuses {
        status.Restarts += cs.RestartCount
        if cs.State.Waiting != nil {
            switch cs.State.Waiting.Reason {
            case "CrashLoopBackOff", "OOMKilled", "ImagePullBackOff",
                "ErrImagePull", "CreateContainerConfigError":
                status.Reason = "init: " + cs.State.Waiting.Reason
                status.IsFailing = true
                return status
            }
        }
    }

    // Then check main containers as before.
    for _, cs := range pod.Status.ContainerStatuses {
        // ... same logic as above
    }

    // ... rest of the function unchanged
}

Prefixing the reason with "init: " makes it immediately clear in the report that it's an init container failing, not the main application.

You can add this code block before checking the main container in your project if you want.

printer.go, output formatting

// internal/reporter/printer.go
package reporter

import (
    "fmt"
    "io"
    "strings"
    "text/tabwriter"
)

// Print writes the health report as a formatted table to w.
// Accepting io.Writer instead of os.Stdout directly keeps this
// testable — callers can pass a bytes.Buffer to inspect the output.
func (r *Reporter) Print(w io.Writer, report *Report) {
    fmt.Fprintf(w, "Cluster Health Report\n")
    fmt.Fprintf(w, "Generated: %s\n\n",
        report.GeneratedAt.Format("2006-01-02 15:04:05 UTC"),
    )

    printNamespaceSummary(w, report.Namespaces)

    if len(report.Failing) > 0 {
        printFailingPods(w, report.Failing)
        return
    }

    if len(report.Pending) > 0 {
        printPendingPods(w, report.Pending)
        return
    }

    fmt.Fprintf(w, "\n✓ All pods healthy\n")
}

// printNamespaceSummary writes the namespace totals table.
func printNamespaceSummary(w io.Writer, namespaces []NamespaceSummary) {
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    fmt.Fprintln(tw, "NAMESPACE\tTOTAL\tRUNNING\tFAILING\tPENDING")
    fmt.Fprintln(tw, strings.Repeat("─", 52))
    for _, ns := range namespaces {
        fmt.Fprintf(tw, "%s\t%d\t%d\t%d\t%d\n",
            ns.Namespace, ns.Total, ns.Running, ns.Failing, ns.Pending,
        )
    }
    tw.Flush()
}

// printFailingPods writes the failing pods detail section.
func printFailingPods(w io.Writer, pods []PodStatus) {
    fmt.Fprintf(w, "\nFAILING PODS\n")
    fmt.Fprintln(w, strings.Repeat("─", 52))
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    for _, p := range pods {
        fmt.Fprintf(tw, "%s\t%s\t%s\t%d restarts\n",
            p.Namespace, p.Name, p.Reason, p.Restarts,
        )
    }
    tw.Flush()
}

// printPendingPods writes the pending pods detail section.
func printPendingPods(w io.Writer, pods []PodStatus) {
    fmt.Fprintf(w, "\nPENDING PODS\n")
    fmt.Fprintln(w, strings.Repeat("─", 52))
    tw := tabwriter.NewWriter(w, 0, 0, 3, ' ', 0)
    for _, p := range pods {
        fmt.Fprintf(tw, "%s\t%s\t%s\n",
            p.Namespace, p.Name, p.Reason,
        )
    }
    tw.Flush()
}

reporter.go, orchestration only

// internal/reporter/reporter.go
package reporter

import (
    "context"
    "fmt"
    "time"

    k8s "github.com/FerRiosCosta/go-k8s-health-reporter/internal/kubernetes"
)

// Reporter builds health reports from Kubernetes data.
// It depends on the KubeClient interface, never on a concrete
// client-go type, so it works with both real and fake clients.
type Reporter struct {
    kube k8s.KubeClient
}

// New creates a Reporter with the given KubeClient injected.
func New(kube k8s.KubeClient) *Reporter {
    return &Reporter{kube: kube}
}

// Generate fetches all pods from the cluster and builds a health report.
func (r *Reporter) Generate(ctx context.Context) (*Report, error) {
    // Fetch all pods across all namespaces.
    // Passing "" to ListPods returns pods from every namespace.
    pods, err := r.kube.ListPods(ctx, "")
    if err != nil {
        return nil, fmt.Errorf("fetch pods: %w", err)
    }

    report := &Report{
        GeneratedAt: time.Now().UTC(),
    }

    // Group pods by namespace using a map.
    summaries := make(map[string]*NamespaceSummary)

    for _, pod := range pods.Items {
        // getOrCreate avoids the boilerplate of checking if the
        // namespace key exists before writing to the map.
        summary := getOrCreate(summaries, pod.Namespace)
        summary.Total++

        // inspectPod is defined in inspector.go, same package,
        // no import needed.
        status := inspectPod(&pod)

        switch {
        case status.IsFailing:
            summary.Failing++
            report.Failing = append(report.Failing, status)
        case status.IsPending:
            summary.Pending++
            report.Pending = append(report.Pending, status)
        default:
            summary.Running++
        }
    }

    for _, summary := range summaries {
        report.Namespaces = append(report.Namespaces, *summary)
    }

    return report, nil
}

// getOrCreate returns the existing NamespaceSummary for the given namespace
// or creates a new one and registers it in the map.
// Extracted into its own function to keep Generate() readable.
func getOrCreate(m map[string]*NamespaceSummary, namespace string) *NamespaceSummary {
    if s, ok := m[namespace]; ok {
        return s
    }
    s := &NamespaceSummary{Namespace: namespace}
    m[namespace] = s
    return s
}

Notice that reporter.go now has no output imports (tabwriter, strings, io) and no Kubernetes type imports beyond what it needs for ListPods. The corev1 import moved entirely to inspector.go where it belongs. The reporter package is still one package, Go compiles all four files together, but each file is now independently readable.

Wire it together in main.go

// main.go
package main

import (
    "context"
    "fmt"
    "os"

    k8s "github.com/FerRiosCosta/go-k8s-health-reporter/internal/kubernetes"
    "github.com/FerRiosCosta/go-k8s-health-reporter/internal/reporter"
)

func main() {
    // Build the Kubernetes client using the kubeconfig minikube wrote in post 2.
    // This is the same file kubectl uses, no extra setup needed.
    kubeconfigPath := os.Getenv("HOME") + "/.kube/config"
    kube, err := k8s.NewClient(kubeconfigPath)
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to connect to cluster: %v\n", err)
        os.Exit(1)
    }

    // Build the reporter with the client injected.
    r := reporter.New(kube)

    // Generate and print the report.
    ctx := context.Background()
    report, err := r.Generate(ctx)
    if err != nil {
        fmt.Fprintf(os.Stderr, "failed to generate report: %v\n", err)
        os.Exit(1)
    }

    r.Print(os.Stdout, report)
}

Verifying against minikube

The best way to verify the health reporter works is to run it against your real minikube cluster. No fakes, no mocks, just your actual cluster from post 2.

Healthy cluster

Make sure minikube is running and the go-api deployment from post 2 is up:

kubectl get pods
# NAME                      READY   STATUS    RESTARTS   AGE
# go-api-7d6b9f8c4-xk2pq   1/1     Running   0          2d
# go-api-7d6b9f8c4-mn9rt   1/1     Running   0          2d
# go-api-7d6b9f8c4-p8wvz   1/1     Running   0          2d

Run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-06-29 15:58:43 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
default                3   3   0   0
kube-system            8   8   0   0
kubernetes-dashboard   2   2   0   0

✓ All pods healthy

Simulating a failing pod

To verify the failing pod detection works we need a pod that reliably OOMKills. Our go-api binary turns out to be too lean to reliably trigger OOMKill at the Docker minimum, the multi-stage Dockerfile produced an extremely efficient binary. That's good engineering, but it makes for a tricky simulation.

Instead, let's deploy a pod that deliberately holds memory using Python's bytearray. Unlike shell tools like dd which stream data through memory and release it immediately, bytearray allocates a contiguous block and holds it, guaranteeing an OOMKill with a tight limit.

Create a file called memory-hog.yaml:

# memory-hog.yaml
apiVersion: v1
kind: Pod
metadata:
  name: memory-hog
  namespace: default
spec:
  # restartPolicy: Always ensures Kubernetes keeps restarting after each
  # OOMKill, driving the pod into CrashLoopBackOff.
  restartPolicy: Always
  containers:
  - name: hog
    image: python:3.12-alpine
    command: ["python3", "-c"]
    args:
    - |
      import time
      # bytearray allocates 100MB and holds it in physical memory.
      # The moment this runs, usage exceeds the 10Mi limit
      # and the OOM killer fires immediately.
      data = bytearray(100 * 1024 * 1024)
      print("allocated 100MB, holding...", flush=True)
      time.sleep(3600)
    resources:
      requests:
        memory: "10Mi"
      limits:
        memory: "10Mi"

Apply it:

kubectl apply -f memory-hog.yaml

Watch the pod:

kubectl get pods -w

# NAME          READY   STATUS             RESTARTS   AGE
# memory-hog   0/1     ContainerCreating  0          1s
# memory-hog   0/1     OOMKilled          0          4s   ← transient
# memory-hog   0/1     CrashLoopBackOff   1          10s  ← stable, run the reporter now

Once you see CrashLoopBackOff, run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-07-01 21:37:47 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
kubernetes-dashboard   2   2   0   0
default                4   3   1   0
kube-system            8   8   0   0

FAILING PODS
────────────────────────────────────────────────────
default   memory-hog   CrashLoopBackOff   13 restarts

The reporter correctly identifies the failing pod, the reason, and the restart count.

Clean up when done:

kubectl delete pod memory-hog

Simulating a pending pod

To verify pending detection, set a memory request higher than your node can provide. Again, both requests and limits must be set together:

kubectl set resources deployment/go-api \
  --requests=memory=100Gi \
  --limits=memory=100Gi \
  -n default

The pods can't be scheduled, no node has 100Gi available. Run the reporter:

go run ./main.go

Cluster Health Report
Generated: 2026-07-01 22:03:51 UTC

NAMESPACE   TOTAL   RUNNING   FAILING   PENDING
────────────────────────────────────────────────────
default                4   3   0   1
kube-system            8   8   0   0
kubernetes-dashboard   2   2   0   0

PENDING PODS
────────────────────────────────────────────────────
default   go-api-6ff6496cb5-dnq6m   0/1 nodes are available: 1 Insufficient memory. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

The reason message comes directly from the pod's scheduling condition, exactly what the pendingReason() function extracts. Restore the correct values when done:

kubectl set resources deployment/go-api \
  --requests=memory=32Mi \
  --limits=memory=64Mi \
  -n default

Note on testing with fakes: In a future post we'll cover how to write unit tests for this reporter using interface fakes and client-go's built-in fake clientset, so the test suite runs without a real cluster. For now, minikube is our test environment.

Common gotchas

Wrong kubeconfig context

The most common mistake when starting out with client-go. If you have multiple clusters in your kubeconfig, always verify before running Go code:

kubectl config current-context
# Should print: minikube

If it shows a production cluster, switch back:

kubectl config use-context minikube

Passing "" vs "default" to namespace-scoped resources

// Returns pods from ALL namespaces
clientset.CoreV1().Pods("").List(ctx, metav1.ListOptions{})

// Returns pods from the "default" namespace only
clientset.CoreV1().Pods("default").List(ctx, metav1.ListOptions{})

These behave completely differently. The empty string means "all namespaces", not "the default namespace." This is one of the most common sources of confusion in client-go code.

Pod phase vs container state, the big one

A pod in Running phase can have containers in CrashLoopBackOff. The phase is the pod-level state; the container state is what's actually happening inside. Always check both:

// WRONG: misses CrashLoopBackOff pods
if pod.Status.Phase != corev1.PodRunning {
    // handle failing pod
}

// RIGHT: catches CrashLoopBackOff even when phase is Running
for _, cs := range pod.Status.ContainerStatuses {
    if cs.State.Waiting != nil && cs.State.Waiting.Reason == "CrashLoopBackOff" {
        // handle failing container
    }
}

Creating a new Clientset per function call

client-go's Clientset maintains an internal HTTP connection pool and rate limiter. Creating a new one on every call throws away the connection pool and can trigger API server rate limits. Always create the Clientset once at startup and reuse it:

// WRONG: new Clientset every call, no connection reuse
func listPods() {
    clientset, _ := kubernetes.NewForConfig(config)
    clientset.CoreV1().Pods("").List(...)
}

// RIGHT: Clientset created once, reused everywhere
var clientset kubernetes.Interface

func init() {
    clientset, _ = kubernetes.NewForConfig(config)
}

func listPods() {
    clientset.CoreV1().Pods("").List(...)
}

API version mismatches, always pin all three packages together

All three packages, k8s.io/api, k8s.io/apimachinery, and k8s.io/client-go, must be on the same minor version. They are developed and released together as part of each Kubernetes release cycle. Mixing versions causes compile errors because types from one package won't match types expected by another.

The version number maps directly to your Kubernetes cluster: v0.36.2 = Kubernetes v1.36.2. Since we're running Kubernetes 1.36.2 in minikube, v0.36.2 is the right choice throughout this series:

# WRONG: mixing minor versions causes type mismatch errors at compile time
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.35.0          # different minor version — will break
go get k8s.io/apimachinery@v0.34.0  # different minor version — will break

# RIGHT: all three pinned to match Kubernetes 1.36.2
go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.36.2
go get k8s.io/apimachinery@v0.36.2
go mod tidy

Always verify your cluster version before pinning:

kubectl version 

# Client Version: v1.36.2
# Server Version: v1.36.2

If your client and server versions differ, for example Client Version: v1.36.2 and Server Version: v1.35.1, that's completely fine. Kubernetes has an official version skew policy: kubectl can be up to one minor version ahead or behind the server. Pin your client-go packages to match whichever version you're running in the cluster.

Summary

client-go is the foundation everything in the Kubernetes Go ecosystem is built on. Four packages, one Clientset, typed access to every resource the API server exposes.

Three things to take away:

The Clientset pattern: clientset.APIGroup().Resource(namespace).Operation(), is consistent across every resource type. Learn it once, read any client-go code.
Always check container states, not just pod phase, a "Running" pod can have failing containers.
Wrap the Clientset behind an interface from the start, it costs nothing and makes every function that uses it instantly testable.

What's next

The health reporter we built here is a real tool. Run it on any cluster, and it gives you immediate operational value. It's also the direct foundation for a new tool we are going to build in the later chapter:ferctl top: a full CLI with Cobra, flags, tabwriter output, and near-limit warnings. The health reporter becomes a subcommand. The KubeClient interface gets extended with metrics.

Every pattern, every import, and every interface in this post appears in the next post.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Setting up a local Kubernetes cluster with minikube

Fer Rios — Mon, 06 Jul 2026 15:07:43 +0000

This is part of the Platform Engineering with Go series. View all posts in the series

Every post in this series assumes you have a Kubernetes cluster

And that's a problem, because most engineers don't have one sitting around they can freely experiment on. Spinning up an EKS cluster just to follow a tutorial costs money, requires an AWS account, and takes longer than you want. Using your production cluster is obviously off the table.

minikube fixes this. It gives you a real, full Kubernetes cluster running on your laptop in under 5 minutes. Same API as production. Free. No cloud account needed. When you're done for the day, you stop it. When things get messy, you delete it and start fresh.

By the end of this post you'll have a local cluster running with metrics-server enabled and a Go app deployed inside it.

What you'll learn

What minikube is and how it relates to a real production cluster
How to install minikube on Mac and Linux
How to start, stop, and manage a local cluster
How minikube connects to kubectl and your kubeconfig
How to enable the metrics-server addon
How to deploy a real Go app to your local cluster

Prerequisites

Mac or Linux, this post does not cover Windows
Docker installed and running, minikube uses it as the default driver

We'll install kubectl as part of this post, so you don't need it yet.

What minikube actually is

Before installing anything, it's worth understanding what minikube does, and what it doesn't do.

minikube runs a single-node Kubernetes cluster locally, inside a Docker container (or VM) on your machine. That cluster runs the exact same Kubernetes API as the production clusters you'd find on EKS, GKE, or AKS. The same kubectl commands work. The same YAML manifests work. The same Go code works, which is exactly why it's the right tool for this series.

What minikube is not: a production cluster. It's a single node with no high availability, no cloud integrations, and no autoscaling. It's a development and learning environment, not something you'd run workloads on.

How it compares to alternatives

There are several tools for running Kubernetes locally. Here's how they compare so you understand why we're using minikube:

Tool	Best for	Why we chose minikube
minikube	Learning and development	Addons system, excellent docs, most widely used
kind	CI pipelines	Lighter than minikube, less convenient for local dev
k3s	Lightweight production	More complex to set up locally
Docker Desktop K8s	Quick start	Heavier, less control over the cluster

The addons system is the main reason we use minikube here. Enabling metrics-server is a single command with minikube. Other tools require manual installation and configuration.

Installing minikube

Mac

# Install minikube via Homebrew
brew install minikube

# Verify the install
minikube version
# minikube version: v1.38.1

Linux

# Download the minikube binary
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

# Make it executable and move it to your PATH
chmod +x minikube-linux-amd64
sudo mv minikube-linux-amd64 /usr/local/bin/minikube

# Verify the install
minikube version
# minikube version: v1.32.0

Installing kubectl

kubectl is the command-line tool for interacting with Kubernetes clusters. minikube configures it automatically, but you need to install it separately.

Mac

brew install kubectl

# Verify
kubectl version --client
# Client Version: v1.36.2

Linux

# Download the latest stable kubectl binary
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Make it executable and move to PATH
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl

# Verify
kubectl version --client
# Client Version: v1.29.0

Starting your first cluster

With minikube and kubectl installed, starting a cluster is one command:

minikube start

You'll see output like this as minikube works through the startup sequence:

😄  minikube v1.38.1 on Darwin 26.5.1 (arm64)
✨  Automatically selected the docker driver
❗  Starting v1.39.0, minikube will default to "containerd" container runtime. See #21973 for more info.
📌  Using Docker Desktop driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.50 ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.35.1 on Docker 29.2.1 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

What happened in those few seconds:

minikube pulled a Docker image containing a full Kubernetes cluster.
It started a Docker container running that image.
It generated TLS certificates for the API server.
It configured kubectl to talk to this new cluster by writing a new context to ~/.kube/config

Configuring resources

The default CPUs and memory are sometimes not enough, especially once you have several pods running. We recommend starting with a bit more:

# Delete any existing cluster first, then start with more resources
minikube delete
minikube start --cpus=4 --memory=4096

Pinning the Kubernetes version

If you want to match your production cluster's Kubernetes version exactly, (I'm not going to run this; I want the last version 1.35.1):

minikube start --kubernetes-version=v1.28.3

Verifying the cluster is running

# Check the node is Ready
kubectl get nodes

# NAME       STATUS   ROLES           AGE     VERSION
# minikube   Ready    control-plane   4m59s   v1.35.1

# Check the system pods are all running
kubectl get pods -A

# NAMESPACE     NAME                               READY   STATUS    RESTARTS        AGE
# kube-system   coredns-7d764666f9-pzqlt           1/1     Running   0               5m33s
# kube-system   etcd-minikube                      1/1     Running   0               5m39s
# kube-system   kube-apiserver-minikube            1/1     Running   0               5m39s
# kube-system   kube-controller-manager-minikube   1/1     Running   0               5m39s
# kube-system   kube-proxy-2l7h5                   1/1     Running   0               5m33s
# kube-system   kube-scheduler-minikube            1/1     Running   0               5m39s
# kube-system   storage-provisioner                1/1     Running   1 (5m32s ago)   5m38s

If all pods show Running, your cluster is healthy.

You can also see the docker container running on behalf our minikube cluster

docker ps -a                                                                                                                                                     
# CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS          PORTS                                                                                                                                  NAMES
# de39b5440365   gcr.io/k8s-minikube/kicbase:v0.0.50   "/usr/local/bin/entr…"   10 minutes ago   Up 10 minutes   127.0.0.1:60921->22/tcp, 127.0.0.1:60923->2376/tcp, 127.0.0.1:60922->5000/tcp, 127.0.0.1:60925->8443/tcp, 127.0.0.1:60924->32443/tcp   minikube

Managing your cluster

You'll use these commands constantly throughout the series:

# Pause the cluster — preserves all state, frees up resources
minikube stop

# Resume where you left off
minikube start

# Delete everything and start completely fresh
# Use this when things get into a bad state
minikube delete

# Check what's currently running
minikube status

# Open the Kubernetes dashboard in your browser
# Useful for getting a visual overview of your cluster
minikube dashboard

# SSH into the minikube node
# Useful for low-level debugging
minikube ssh

A practical workflow: minikube stop at the end of every session, minikube start when you come back. minikube delete && minikube start When something is broken, and you want a clean slate.

What is metrics-server?

Before we enable it, it's worth understanding what metrics-server actually is and where it fits in the Kubernetes ecosystem.

metrics-server is a lightweight, in-memory component that runs inside your Kubernetes cluster and collects resource usage data, including CPU and memory, from every node and pod. It does this by talking to the kubelet (the agent that runs on each node) and scraping the resource stats it exposes.

┌─────────────────────────────────────────┐
│           Kubernetes cluster            │
│                                         │
│  ┌──────────┐     scrapes    ┌────────┐ │
│  │  kubelet │ ◄────────────  │metrics │ │
│  │ (node 1) │               │ server │ │
│  └──────────┘               └───┬────┘ │
│                                 │      │
│  ┌──────────┐                   │ exposes
│  │  kubelet │ ◄─────────────────┘      │
│  │ (node 2) │      Metrics API         │
│  └──────────┘           ▲             │
│                          │             │
└──────────────────────────┼─────────────┘
                           │
              kubectl top / your Go code

The data metrics-server collects is exposed through the Metrics API, a standard Kubernetes API endpoint that both kubectl top and, client-go can query. When you run kubectl top pods, kubectl is calling the Metrics API.

What metrics-server is not

It's worth being clear about what metrics-server doesn't do, because it's a common source of confusion:

It does not store historical data. It only keeps the most recent sample in memory. If you want trends over time, you need Prometheus.
It does not replace Prometheus or Grafana. It's a lightweight source of current usage data, not a full monitoring stack.
It does not run by default on most clusters. On EKS, GKE, and AKS it's usually pre-installed, but on self-managed clusters, and on minikube, you have to enable it yourself.

Think of it as the simplest possible answer to the question: "what is my cluster using right now?" For anything more sophisticated, alerting, historical analysis, dashboards, you'd reach for Prometheus.

Enabling metrics-server

minikube has an add-ons system, a curated list of optional components you can enable with a single command. metrics-server is one of them:

# Enable metrics-server
minikube addons enable metrics-server

# Verify it's running
kubectl get deployment metrics-server -n kube-system

# NAME             READY   UP-TO-DATE   AVAILABLE
# metrics-server   1/1     1            1

Wait about 60 seconds, then verify it's collecting data:

# Check node resource usage
kubectl top nodes

# NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# minikube   185m         4%     1042Mi          26%

# Check pod resource usage across all namespaces
kubectl top pods -A

If kubectl top returns error: metrics not available yet, just wait another 30 seconds and try again. metrics-server takes a moment to scrape its first round of data.

Deploying a real Go app to your local cluster

An empty cluster is fine for testing connections, but what we actually want is a cluster with real pods running, pods we can query with the health reporter.

Let's deploy a simple Go HTTP server properly, built with a multi-stage Dockerfile, loaded into minikube, and deployed with real resource requests and limits.

Why we use a Dockerfile

You might wonder why we don't just use go run inside the container for simplicity. The answer ties directly back to post 1: resource management.

go run compiles your code at container start, and the Go compiler needs over 1GB of memory just to compile a hello world app. That forces you to set absurdly high memory limits for a tiny service, which defeats the entire purpose of a post about resource management. It also causes OOMKill on any reasonable memory limit, and makes startup slow.

The right approach, and the one every real Go service uses, is to compile the binary at image build time. The running container only needs the binary, not the compiler. Memory usage drops from 1GB to under 20MB:

go run approach:     limits.memory: 1024Mi  ← embarrassing for a hello world
Dockerfile approach: limits.memory: 128Mi    ← honest and production-realistic

The project structure

Create a new directory for this example:

mkdir go-api && cd go-api

Create three files:

go-api/
├── main.go        ← the Go HTTP server
├── Dockerfile     ← multi-stage build
└── deployment.yaml ← Kubernetes manifests

The application

// main.go
package main

import (
    "fmt"
    "log/slog"
    "net/http"
    "os"
)

func main() {
    // Handle the root endpoint
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "hello from Go on Kubernetes!")
    })

    // Health endpoint — used by readiness and liveness probes.
    // Always returns 200 OK as long as the server is running.
    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })

    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }

    slog.Info("server starting", "port", port)
    if err := http.ListenAndServe(":"+port, nil); err != nil {
        slog.Error("server failed", "error", err)
        os.Exit(1)
    }
}

The Dockerfile

This is a multi-stage build, one stage compiles the binary, a second minimal stage runs it. The final image contains only the binary and nothing else, no Go toolchain, no compiler, no source code:

# Stage 1: builder
# Uses the full Go image to compile the binary.
# This stage is discarded after the build — it never ships.
FROM golang:1.26-alpine AS builder

WORKDIR /app

# Copy the source code
COPY main.go .

# Initialize a Go module — needed for go build
RUN go mod init github.com/yourname/go-api

# Build a statically linked binary.
# CGO_ENABLED=0 produces a fully static binary with no external dependencies.
# -ldflags="-s -w" strips debug symbols to reduce binary size.
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o server .

# Stage 2: runtime
# Uses a minimal Alpine image — no Go toolchain, no compiler.
# The final image is typically under 10MB.
FROM alpine:3.24

# Run as a non-root user — a basic security best practice.
RUN addgroup -S app && adduser -S app -G app
USER app

WORKDIR /app

# Copy only the compiled binary from the builder stage.
COPY --from=builder /app/server .

EXPOSE 8080

CMD ["./server"]

Build the image and load it into minikube

minikube runs its own Docker daemon, separate from your laptop's Docker. To use a locally built image in minikube, you need to build it inside minikube's Docker environment:

# Point your terminal's Docker CLI at minikube's Docker daemon
eval $(minikube docker-env)

# Build the image — this builds directly inside minikube
docker build -t go-api:latest .

# Verify the image is available inside minikube
docker images | grep go-api
# go-api   latest   a1b2c3d4e5f6   10 seconds ago   14.3MB

Notice the image size, 14.3MB. Compare that to the golang:1.26-alpine base image which is ~300MB. Multi-stage builds make a real difference.

When you're done working and want to restore your terminal's Docker CLI to your laptop's Docker:

eval $(minikube docker-env -u)

The deployment manifest

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-api
  namespace: default
  labels:
    app: go-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: go-api
  template:
    metadata:
      labels:
        app: go-api
    spec:
      containers:
      - name: server
        image: go-api:latest

        # imagePullPolicy: Never tells Kubernetes to use the local image
        # and never try to pull it from a registry.
        # Required when using images built inside minikube's Docker daemon.
        imagePullPolicy: Never

        ports:
        - containerPort: 8080

        # Resource requests and limits — from post 1.
        # These are now realistic for a pre-compiled Go binary.
        # A running Go binary uses ~10-20Mi — we give it 64Mi to be safe.
        resources:
          requests:
            cpu: "50m"
            memory: "32Mi"
          limits:
            cpu: "100m"
            memory: "64Mi"

        # Readiness probe — Kubernetes only sends traffic once this passes.
        # initialDelaySeconds is short because a pre-compiled binary
        # starts in milliseconds — no compilation step.
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

        # Liveness probe — Kubernetes restarts the container if this fails.
        # Always set higher than readinessProbe.initialDelaySeconds.
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: go-api
  namespace: default
spec:
  # ClusterIP is the default service type.
  # It gives the service a stable internal IP address only reachable
  # from inside the cluster. This is correct for internal services —
  # we access it locally using kubectl port-forward, not via a public IP.
  type: ClusterIP
  selector:
    app: go-api
  ports:
  - port: 80
    targetPort: 8080

Deploy it

# Apply both the Deployment and Service
kubectl apply -f deployment.yaml

# Watch the pods come up — should be fast since the binary starts instantly
kubectl get pods -w

# NAME                      READY   STATUS    RESTARTS   AGE
# go-api-7d6b9f8c4-xk2pq   1/1     Running   0          8s
# go-api-7d6b9f8c4-mn9rt   1/1     Running   0          8s
# go-api-7d6b9f8c4-p8wvz   1/1     Running   0          9s

No ContainerCreating delays. No OOMKill. The binary starts in milliseconds.

Verify it's working

# Check resource usage — these are the honest numbers for a Go binary
kubectl top pods

# NAME                      CPU(cores)   MEMORY(bytes)
# go-api-7d6b9f8c4-xk2pq   2m           11Mi
# go-api-7d6b9f8c4-mn9rt   2m           10Mi
# go-api-7d6b9f8c4-p8wvz   2m           11Mi

To hit the service from your terminal, use kubectl port-forward, it works the same way on both Mac and Linux regardless of which minikube driver you're using:

# Forward local port 8080 to the go-api service
# Keep this running in one terminal
kubectl port-forward service/go-api 8080:80

# Forwarding from 127.0.0.1:8080 -> 8080
# Forwarding from [::1]:8080 -> 8080

In a second terminal:

curl http://localhost:8080
# hello from Go on Kubernetes!

You should see hello from Go on Kubernetes!, confirming the service is running and routing traffic to your pods correctly. You now have three running pods with real, honest resource usage data available from metrics-server.

How Go connects to this cluster

This section previews what happens in the next post, so that when you see the first line of Go code, it already makes sense.

When minikube start ran, it wrote a new entry to your ~/.kube/config file. Here's the relevant section it added:

# ~/.kube/config (relevant section)
contexts:
- context:
    cluster: minikube
    namespace: default
    user: minikube
  name: minikube       # ← the context name

current-context: minikube  # ← currently active context

This is the same file kubectl reads when you run any kubectl command. It's also the file our Go code will read using BuildConfigFromFlags:

// This is how our Go program will connect to minikube
// in the next post — reads the same ~/.kube/config file
config, err := clientcmd.BuildConfigFromFlags("", os.Getenv("HOME")+"/.kube/config")

No extra setup. No credentials to manage. The Go program connects to exactly the same cluster kubectl connects to, because both read the same kubeconfig.

Verifying the context before running Go code

Before running any Go code that talks to Kubernetes, always verify which cluster you're pointing at:

# Which context is currently active?
kubectl config current-context
# minikube

# List all available contexts (useful if you have multiple clusters)
kubectl config get-contexts

# CURRENT   NAME       CLUSTER    AUTHINFO   NAMESPACE
# *         minikube   minikube   minikube   default
#           prod-eks   eks-prod   aws-user   production

The * marks the active context. If you ever accidentally point at a production cluster while running your Go code, the get-contexts output makes it obvious.

To switch between contexts:

# Switch to a different context
kubectl config use-context prod-eks

# Switch back to minikube
kubectl config use-context minikube

Common gotchas

minikube service tunnel on Mac with Docker driver

On Mac, minikube runs inside a Docker container with its own internal network. Running minikube service go-api --url creates a temporary tunnel and warns:

Services [default/go-api] have type "ClusterIP" not meant to be exposed,
however for local development minikube allows you to access this!
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

This isn't an error, but the tunnel only works while that terminal stays open. Use kubectl port-forward instead, it's more portable, works the same on Mac and Linux, and doesn't need a dedicated terminal window.

Docker not running

minikube uses Docker as its default driver. If Docker isn't running when you call minikube start, you'll see:

X Exiting due to PROVIDER_DOCKER_NOT_RUNNING: Docker Desktop is not running

Start Docker first, then retry minikube start.

Not enough resources: pods stuck in Pending

If your pods stay in Pending state, the most common cause is insufficient resources. Check with:

kubectl describe pod <pod-name> | grep -A5 Events
# Warning  FailedScheduling  Insufficient memory

Fix it by giving minikube more resources:

minikube delete
minikube start --cpus=4 --memory=8192

minikube IP changes on restart

Every time you minikube delete && minikube start, minikube may get a different IP address for the API server. If you ever hardcode the API server URL anywhere in your Go code, it will break after a restart. Always use BuildConfigFromFlags with the kubeconfig path, it reads the current IP dynamically from the config file.

Summary

You now have everything you need to follow the rest of this series:

A local Kubernetes cluster running minikube
metrics-server enabled and collecting data
A Go app deployed with 3 replicas and proper resource configuration
kubectl configured and pointing at your local cluster

Three things to remember as you work through the series:

minikube stop at the end of every session, don't leave it consuming resources in the background
minikube delete when things get into a broken state, a clean cluster takes less than 2 minutes to rebuild
kubectl config current-context before running any Go code, always know which cluster you're talking to

What's next

In the next post we write our first Go program that connects to this cluster using client-go, the official Kubernetes Go client. We'll query the pods we just deployed, detect failing ones, and build something immediately useful: a cluster health reporter that gives you a full picture of your cluster's state in a single command.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Kubernetes resource requests and limits explained: scheduling, throttling, and OOMKill

Fer Rios — Thu, 02 Jul 2026 01:11:20 +0000

This is part of the Platform engineering with Go series: a growing collection of posts on Kubernetes, Go tooling, and infrastructure automation. View all posts in the series

The 3am incident nobody talks about

It's 3am. Your on-call phone goes off. A service is down in production. You log in, check the pods, and see this:

kubectl get pods -n production

NAME                    READY   STATUS             RESTARTS   AGE
api-7d6b9f8c4-xk2pq    0/1     OOMKilled          14         2d
api-7d6b9f8c4-mn9rt    0/1     OOMKilled          14         2d
api-7d6b9f8c4-p8wvz    0/1     OOMKilled          14         2d

You restart the pods. They come back. Five minutes later, they die again.

The problem could be several things: limits set too low for the actual workload, no limits set at all so the pod consumed memory freely until the node ran out, a traffic spike the pod wasn't provisioned for, or yes, a memory leak in the application itself. In all cases, the result is the same: memory consumption exceeded the allowed ceiling, the OOM killer fired, the pod died, and Kubernetes restarted it into the same situation. Fourteen times.

This is one of the most common production incidents in Kubernetes, and one of the most preventable. But preventing it requires understanding what requests and limits actually do, what happens when memory consumption exceeds them, and how to configure them correctly for your actual workload.

That's what this post is about.

No Go knowledge required: this post has zero code

Requests vs limits, two completely different things

Before anything else, the most important concept to internalize: requests and limits are not two versions of the same thing. They serve entirely different purposes and are enforced by entirely different components of Kubernetes.

# A pod spec with both requests and limits set
resources:
  requests:
    cpu: "250m"     # 250 millicores = 0.25 CPU cores
    memory: "256Mi" # 256 mebibytes
  limits:
    cpu: "500m"     # 500 millicores = 0.5 CPU cores
    memory: "512Mi" # 512 mebibytes

Here's what each one actually does:

Requests are a promise to the scheduler. When you set requests.memory: 256Mi, you're telling Kubernetes: "I need at least 256Mi of memory reserved for this pod on whatever node it runs on.. The scheduler uses this value to decide which node to place the pod on. Once scheduled, that 256Mi is considered reserved on that node, even if the pod is only using 50Mi at the moment.

Limits are a ceiling enforced by the runtime. When you set limits.memory: 512Mi, you're telling Kubernetes: "This pod is never allowed to use more than 512Mi of memory." If it tries to exceed that ceiling, the kernel kills it. No warning. No graceful shutdown. Just gone.

The key insight: requests affect scheduling; limits affect runtime behavior. They are read by different components at different times for entirely different reasons.

How the scheduler uses requests to place pods

The Kubernetes scheduler's job is to find a node for each new pod. It does this by looking at each node's allocatable resources and comparing them against the sum of all pod requests already scheduled there.

A node's allocatable resources are not the same as its total capacity. Some resources are always reserved for the operating system and Kubernetes system components:

kubectl describe node my-node

# Look for the Allocatable section:
Allocatable:
  cpu:     3800m   # 3.8 cores available to pods (out of 4 total)
  memory:  7Gi     # 7Gi available to pods (out of 8Gi total)

The scheduler adds up the requests of all pods already on a node and compares that sum against the allocatable resources. If the remaining capacity is less than the new pod's request, the node is skipped.

Here's the critical subtlety: the scheduler cares about requests, not actual usage.

Node: 4 CPU allocatable

Pod A: requests 1 CPU → actual usage: 0.2 CPU
Pod B: requests 1 CPU → actual usage: 0.1 CPU
Pod C: requests 1 CPU → actual usage: 0.8 CPU

From the scheduler's perspective: 3 out of 4 CPU are "used"
From the kernel's perspective: only 1.1 CPU are actually being consumed

New pod requesting 1.5 CPU → scheduler says NO (only 1 CPU remaining)

This creates an important tension: if your requests are set too high relative to actual usage, your nodes look full when they're actually mostly idle, and new pods can't be scheduled. This is called poor bin packing, and it wastes money.

On the other hand, if your requests are too low, too many pods get scheduled onto the same node. When they all start consuming resources simultaneously, the node becomes overloaded, and Kubernetes starts evicting pods to relieve the pressure.

Getting requests right is a balancing act between cost efficiency and stability.

CPU limits and throttling: the silent killer

CPU is what Kubernetes calls a compressible resource. If a pod tries to use more CPU than its limit allows, the Linux kernel doesn't kill it, it throttles it. The pod keeps running, but it gets fewer CPU cycles, so everything it does takes longer.

The throttling mechanism is the Linux CFS (Completely Fair Scheduler).

Understanding CPU cycles and scheduling periods

Before we talk about how throttling works, it helps to understand two concepts that are invisible in day-to-day operations but fundamental to what's happening under the hood.

What is a CPU cycle?

Your server's CPU is constantly doing work, executing instructions, processing data, running code. A CPU cycle is the smallest unit of that work. A modern CPU completes billions of cycles per second (gigahertz, that's what the "3.2GHz" on a server spec means: 3.2 billion cycles per second).

Think of CPU cycles like minutes of attention from a very fast worker. Your pod's processes need a certain number of those "minutes" to do their job, handle a request, run a query, process a message. The more cycles your pod gets, the faster it runs. The fewer it gets, the slower it runs.

When Kubernetes talks about CPU in millicores (250m, 500m, 1000m), it's describing what fraction of one CPU core's cycles your pod gets access to:

1000m = 1 full CPU core = 100% of one core's cycles
 500m = 0.5 CPU core   = 50% of one core's cycles
 250m = 0.25 CPU core  = 25% of one core's cycles

What is a scheduling period?

A CPU doesn't serve one process at a time from start to finish. It slices time into tiny windows and gives each process a turn. This is called time-sharing, and the windows are called scheduling periods.

Think of it like a teacher in a classroom. Instead of helping one student for the entire class, the teacher spends 5 minutes with each student in rotation. Every student gets attention, but no single student monopolizes the teacher's time.

The Linux CFS uses scheduling periods of 100 milliseconds by default. In every 100ms window, the CPU divides its time among all the processes competing for it.

100ms scheduling period
│
├── 0ms  - 25ms  → Pod A gets its turn  (250m limit = 25% of 100ms = 25ms)
├── 25ms - 75ms  → Pod B gets its turn  (500m limit = 50% of 100ms = 50ms)
├── 75ms - 100ms → Pod C gets its turn  (250m limit = 25% of 100ms = 25ms)
│
└── (next 100ms period starts)

Each pod's CPU limit determines how many milliseconds of that 100ms window it's allowed to use:

CPU limit of 250m → 25ms of CPU time per 100ms period
CPU limit of 500m → 50ms of CPU time per 100ms period
CPU limit of 1000m (1 full core) → 100ms of CPU time per 100ms period

What happens when a pod hits its limit mid-period?

Here's where throttling kicks in. If a pod uses up its entire allocation before the 100ms period ends, the CFS puts it in a throttled state for the rest of that period; it gets zero CPU cycles until the next period starts, regardless of whether other pods are idle.

Pod with 250m CPU limit → 25ms of allowed CPU time per 100ms period

Period 1 (0ms to 100ms):
  ├── 0ms:  Pod starts processing a request
  ├── 25ms: Pod has used its full 25ms allocation ← throttled here
  ├── 25ms to 100ms: Pod sits idle, gets zero CPU cycles
  └── 100ms: New period starts, pod gets another 25ms

What the user experiences:
  ├── Request arrives at 0ms
  ├── Pod processes half the request, then waits 75ms doing nothing
  └── Response arrives much later than it should

The pod didn't crash. It didn't log an error. It just stopped making progress for 75ms out of every 100ms, which is why a throttled service feels sluggish rather than broken. Everything works, just much slower than it should.

A concrete analogy

Imagine you're writing a report and your manager says you can only use the shared laptop for 15 minutes every hour. You start writing, but at the 15-minute mark your access is cut off, even if you're mid-sentence. You sit and wait for the next hour to start before you can type another word.

That's exactly what the CFS does to a throttled pod. It doesn't care that you were in the middle of something important. When the quota is up, the process waits, and whatever request it was handling has to wait too.

Why this is hard to detect

The reason CPU throttling causes so much confusion is that it's invisible in all the usual places:

# This shows current usage — looks fine
kubectl top pods -n production
NAME                 CPU(cores)   MEMORY(bytes)
api-6d8f9b7c-xk2pq   240m         180Mi

# But the pod might be throttled 80% of the time
# kubectl top shows average usage, not whether that usage caused throttling

A pod using 240m CPU on average can still be heavily throttled if it regularly bursts above its limit within a single 100ms period. The average looks healthy; the latency tells a different story.

So, this is why this is dangerous: CPU throttling is completely silent.

There's no error. No log line. No Kubernetes event. Your pod just slows down. Requests take longer. Latency increases. Users notice something is wrong, but nothing in your logs explains why.

The only reliable way to detect throttling is through the container_cpu_cfs_throttled_periods_total metric in Prometheus, which counts how many scheduling periods a container was throttled in. If that number is climbing, throttling is happening regardless of what kubectl top shows.

To detect throttling, you can use the Kubernetes metrics server:

# See current CPU usage vs requests
kubectl top pods -n production

NAME                    CPU(cores)   MEMORY(bytes)
api-7d6b9f8c4-xk2pq    480m         210Mi
api-7d6b9f8c4-mn9rt    498m         198Mi

If you see pods consistently near their CPU limit, throttling is likely happening, especially if you're seeing latency spikes that don't correlate with error rates.

The CPU limits controversy

There's an ongoing debate in the Kubernetes community about CPU limits. Some platform teams remove CPU limits entirely for latency-sensitive services, allowing pods to burst freely as long as there's spare capacity on the node.

The argument for removing CPU limits:

Eliminates throttling completely.
Workloads use spare node capacity efficiently.
Latency becomes predictable because pods are never artificially slowed.

The argument for keeping CPU limits:

A noisy neighbor pod can consume all spare CPU and starve other pods.
Without limits, a bug in one service can degrade the entire node.

The right answer depends on your workload. For latency-sensitive APIs, consider removing CPU limits and relying on requests alone. For batch workloads, CPU limits are fine. For anything in between, measure throttling first before deciding.

Memory limits and OOMKill: the dangerous one

Memory is what Kubernetes calls an incompressible resource. Unlike CPU, the kernel cannot throttle memory access, it can't say "you only get 80% of the memory reads you asked for." Memory is binary: either the process has it, or it doesn't.

When a pod's memory usage exceeds its limit, the Linux OOM (Out of Memory) killer terminates one of its processes immediately. No warning. No graceful shutdown. The process is gone. Kubernetes then sees that a container has exited unexpectedly and restarts it, which is where CrashLoopBackOff comes from.

# Detect OOMKill in pod description
kubectl describe pod api-7d6b9f8c4-xk2pq -n production

# Look for this in the output:
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 21 Jun 2026 02:14:32 +0000
  Finished:     Mon, 21 Jun 2026 02:14:33 +0000

Exit code 137 means the process was killed by signal 9 (SIGKILL) from the OOM killer. If you see this, your memory limit is too low for your actual workload.

OOMKill at the node level

OOMKill can also happen at the node level, independently of your pod limits. If total memory consumption across all pods on a node approaches the node's total capacity, the Linux kernel's node-level OOM killer activates.

In this case, Kubernetes doesn't wait for the OOM killer. It has its own eviction mechanism, if available memory on a node drops below a configured threshold, Kubernetes starts evicting pods proactively. Which pods get evicted first is determined by QoS class, which we'll cover next.

QoS classes: who dies first under pressure

Kubernetes automatically assigns every pod a Quality of Service (QoS) class based on how its requests and limits are configured. You don't set this manually; it's derived. Under node pressure, Kubernetes evicts pods in order from lowest to highest QoS class.

There are three classes:

BestEffort: lowest priority

A pod is BestEffort when it has no requests or limits set at all:

# BestEffort, no resources section at all
resources: {}

BestEffort pods are the first to be evicted under node pressure. They get whatever resources happen to be available, and nothing is guaranteed. Never run production workloads as BestEffort.

Burstable: middle priority

A pod is Burstable when it has requests set, but limits are either not set or higher than requests:

# Burstable, requests set, limits higher than requests
resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Most production workloads should be Burstable. The pod has guaranteed minimum resources (the requests) but can burst above them when capacity is available. Under eviction pressure, Burstable pods are evicted after BestEffort but before Guaranteed.

Guaranteed: highest priority

A pod is Guaranteed when its requests equal its limits for all containers:

# Guaranteed, requests == limits
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "500m"     # same as request
    memory: "512Mi" # same as request

Guaranteed pods are the last to be evicted. Kubernetes will sacrifice BestEffort and Burstable pods before touching a Guaranteed pod. Use this for your most critical services, databases, core APIs, and anything where an eviction is catastrophic.

To check a pod's QoS class:

kubectl get pod api-7d6b9f8c4-xk2pq -n production \
  -o jsonpath='{.status.qosClass}'

# Output: Burstable

A practical framework for setting requests and limits

Knowing the theory is one thing. Knowing what numbers to put in your YAML is another. Here's a step-by-step framework that works in production:

Step 1: observe before you configure

Deploy your workload without limits first (or with very high limits that won't be hit). Let it run under realistic traffic for several days, including peak hours. Collect metrics with kubectl top pods or Prometheus.

# Watch resource usage over time
watch kubectl top pods -n production -l app=api

Step 2: set requests at p50 (median) usage

Your request should reflect typical usage, what the pod uses most often. The p50 (50th percentile) of observed CPU and memory usage is a good starting point.

If your service typically uses 150m CPU and 200Mi memory:

requests:
  cpu: "150m"
  memory: "200Mi"

Step 3: set memory limits at p99 plus a buffer

Your memory limit should handle traffic spikes without triggering OOMKill. The p99 (99th percentile) plus a 20-30% buffer is a safe starting point:

limits:
  memory: "350Mi"  # p99 was ~280Mi, plus 25% buffer

Step 4: handle CPU limits carefully

Start at 2-4x your CPU request. Monitor for throttling. If you see consistent throttling in container_cpu_throttled_seconds_total, either raise the limit or remove it for that service.

limits:
  cpu: "500m"  # 3x the 150m request, gives room to burst

Step 5: use LimitRange to enforce defaults at the namespace level

As a platform engineer, you don't want to rely on every developer remembering to set resources. Use a LimitRange to provide defaults for pods that don't specify them:

# Enforce default requests and limits for all pods in a namespace
apiVersion: v1
kind: LimitRange
metadata:
  name: default-resource-limits
  namespace: production
spec:
  limits:
  - type: Container
    default:           # applied when no limits are specified
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:    # applied when no requests are specified
      cpu: "100m"
      memory: "128Mi"
    max:               # no pod can exceed these
      cpu: "2"
      memory: "2Gi"
    min:               # no pod can go below these
      cpu: "50m"
      memory: "64Mi"

Step 6: use ResourceQuota to cap total namespace consumption

ResourceQuota limits the total resources that can be consumed across all pods in a namespace, useful for multi-tenant clusters and chargeback:

# Cap total resource consumption for the production namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "10"      # total CPU requests across all pods
    requests.memory: "20Gi" # total memory requests across all pods
    limits.cpu: "20"        # total CPU limits across all pods
    limits.memory: "40Gi"   # total memory limits across all pods
    pods: "50"              # maximum number of pods

Common mistakes

Running without any resource configuration (BestEffort)

This is the most dangerous mistake. With no requests, the scheduler has no visibility into what your pod actually needs. Under node pressure, your pods are the first to be evicted, regardless of how critical your service is.

Setting limits without requests

When you set a limit without a request, Kubernetes automatically sets the request equal to the limit, which makes the pod Guaranteed. This isn't always wrong, but it means you're reserving 100% of your limit on the scheduler even if typical usage is much lower. Over time this leads to poor bin packing and wasted capacity.

Copying the same resource values for every service

A stateless Go API, a JVM-based service, and a batch data processor have completely different memory profiles. A Go binary might be happy with 128Mi. A JVM service might need 1Gi just for the heap. Tune per workload, not per deployment template.

Forgetting init containers

Init containers run before your main container and have their own resource requirements. If you set tight limits on init containers, pod initialization can fail, and Kubernetes will keep retrying. Always check the init container resource usage too:

initContainers:
- name: db-migrate
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "200m"
      memory: "256Mi"

Setting memory requests too low relative to actual usage

If your pod's actual memory usage regularly exceeds its request (even if it stays below the limit), it becomes a candidate for eviction under node pressure. The request is not just a scheduling hint, it also affects eviction priority within the Burstable class. Pods whose usage exceeds their request are evicted before pods whose usage stays within their request.

Summary

Requests and limits are the foundation of stable Kubernetes workloads. Get them wrong, and you'll spend nights chasing OOM kills and throttling-induced latency spikes. Get them right, and your cluster runs efficiently with predictable, stable workloads.

Three things to take away:

Requests tell the scheduler where to place your pod. Always set them, never run BestEffort in production.
Memory limits tell the kernel when to kill your pod. Set them generously enough to handle traffic spikes, and monitor for OOMKill with kubectl describe pod.
CPU limits are more nuanced. Start with 2-4x your CPU request and remove them for latency-sensitive services if throttling is a problem.

Understanding this is the prerequisite for everything else in platform engineering, autoscaling, capacity planning, chargeback, and multi-tenancy, all of which depend on the resource configuration being correct.

P.S.: Writing this post from Asunción, Paraguay 🇵🇾 where it's hard to concentrate right now. Paraguay just knocked Germany out of the 2026 World Cup on penalties. For the first time in history, Germany lost a penalty shootout. Goalkeeper Orlando Gil was a hero. We face France next. Forgive me if there are any bugs or mistakes in this post. I may have been screaming at a television.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Go HTTP middleware explained: what it is, how it works, and how to build your own

Fer Rios — Mon, 22 Jun 2026 14:31:52 +0000

This is part 2 of 2 in the series Go HTTP middleware from scratch. If you're new to Go types, → read part 1 first.

The same code keeps showing up everywhere

You're building an HTTP server. Every request needs a unique ID for tracing. Every request needs to check authentication. Every request needs its duration logged. The naive approach: copy that code into every handler.

// Without middleware — same boilerplate in every single handler
func getUserHandler(w http.ResponseWriter, r *http.Request) {
    // Auth check — copy/pasted from every other handler
    token := r.Header.Get("Authorization")
    if token == "" {
        http.Error(w, "unauthorized", http.StatusUnauthorized)
        return
    }

    // Request ID — copy/pasted from every other handler
    requestID := uuid.New().String()
    log.Printf("requestID=%s handling getUser", requestID)

    // actual handler logic finally starts here...
}

Do this across 20 handlers, and you have a maintenance nightmare. Change the auth logic once, and you have to update 20 files.

Middleware is the Go way to solve this. Write the cross-cutting logic once, wrap it around your handlers, and never think about it again.

What is this post about

What middleware is and how Go's HTTP model makes it natural.
How the http.Handler interface works, the foundation everything builds on.
How to write your own middleware from scratch.
How to chain multiple middleware together.
How to pass data through the chain using context.
Tips for advanced usage: ordering, third-party routers, and common pitfalls.

The foundation: the http.Handler interface

As I mentioned in the first part of this serie, everything in Go's HTTP stack is built on one tiny interface:

// This is the entire http.Handler interface — just one method.
// Any type that has a ServeHTTP method satisfies it automatically.
type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

If you've read the previous posts in this series, you'll recognize this pattern. Go interfaces are just method signatures. Any Go type with ServeHTTP(ResponseWriter, *Request) is automatically an HTTP handler. No registration, no declaration. If you remember, http.HandlerFunc is a convenience type that lets you turn a plain function into an http.Handler:

// HandlerFunc is a function type that also implements http.Handler.
// This is how you write handlers without creating a struct.
type HandlerFunc func(ResponseWriter, *Request)

func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
    f(w, r)
}

This means these two ways of writing a handler are equivalent:

// Option A: using http.HandleFunc, to convert a plain function 
// to a HandlerFunc type
http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "hello")
})

// Option B: a struct implementing http.Handler
type HelloHandler struct{}

func (h HelloHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "hello")
}

Most people use option A for simple handlers. The struct approach becomes useful when your handler needs to hold state (like a database client).

What is middleware?

Middleware is a function that:

Accepts an http.Handler.
Returns a new http.Handler.
Does something before and/or after calling the original handler.

// This is the shape of every middleware in Go.
// It wraps a handler and returns a new one.
func MyMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Do something BEFORE the handler runs
        // (auth check, logging, adding context...)

        next.ServeHTTP(w, r) // call the actual handler

        // Do something AFTER the handler runs
        // (log the response, record timing...)
    })
}

The next parameter is the handler this middleware wraps. Calling next.ServeHTTP(w, r) passes control forward, like handing a baton in a relay race.

The relay race explained

Here's the mental model that makes middleware click:

Request comes in
      │
      ▼
┌─────────────────┐
│   Middleware 1    ← does its work (e.g. logs start time)
│       │         
│       ▼         
│  Middleware 2     ← does its work (e.g. checks auth)
│       │         
│       ▼         
│    Handler        ← does the actual business logic
│       │         
│       ▼         
│  Middleware 2     ← does its work after (e.g. checks result)
│       │         
│       ▼         
│  Middleware 1     ← does its work after (e.g. logs duration)
└─────────────────┘
      │
      ▼
Response goes out

Each middleware wraps the next layer. Code before next.ServeHTTP runs on the way in. Code after runs on the way out. This is sometimes called the "onion model."

Building three real middleware functions

Let's build three practical middleware functions and chain them together. You can find the repo here.

But before writing any code, here's how the project is laid out. You will get a clearer understanding of how a Golang project should be structured if you read this post first about Go Packages. Each middleware gets its own file inside a shared middleware package, and cmd/server/main.go wires everything together.

go-middleware-examples/
├── go.mod
├── cmd/
│   └── server/
│       └── main.go            ← entry point — registers routes and chains middleware
└── internal/
    └── middleware/
        ├── requestid.go       ← RequestID middleware + context key + getter
        ├── logger.go          ← Logger middleware + responseWriter wrapper
        ├── auth.go            ← Auth middleware
        └── chain.go           ← Chain helper for readable middleware composition

A few decisions worth noting:

All middleware lives under internal/middleware/, it's private to this module. If you later wanted to publish this middleware as a reusable library for other projects, it would move to pkg/middleware/ instead.
Each middleware gets its own file even though they all declare package middleware. This keeps each one focused and easy to find, requestid.go only ever deals with request IDs, auth.go only ever deals with authentication.
The context key (RequestIDKey) and its getter (GetRequestID) live in the same file as the middleware that sets it (requestid.go). Anything that reads the request ID imports the middleware package and calls middleware.GetRequestID(ctx), never reaching into the context directly. You can know more about Go contexts here.
cmd/server/main.go stays thin, it only registers handlers and chains middleware together. No middleware logic lives there.

From the root directory (go-middleware-examples), run the following commands to create your Go project, and install the uuid package:

go mod init github.com/<your-username>/go-middleware-examples
go get github.com/google/uuid

You can name your project as you want. I usually name my projects according to my GitHub app repo name.

Here's the go.mod for reference:

module github.com/FerRiosCosta/go-middleware-examples

go 1.26.3

require github.com/google/uuid v1.6.0 // indirect

Now let's build each file.

Request ID middleware

Please consider the first comment line, which specifies where the file should be created. This middleware attaches a unique ID to every request. Useful for tracing a single request through your logs.

// internal/middleware/requestid.go
package middleware

import (
    "context"
    "net/http"

    "github.com/google/uuid"
)

// key is an unexported type for context keys in this package.
// Using a custom type prevents collisions with keys from other packages.
// (Two packages both using the string "requestID" would overwrite each other.)
type key string

// RequestIDKey is the context key for the request ID.
// Exported so handlers in other packages can retrieve it.
const RequestIDKey key = "requestID"

// RequestID generates a unique ID for every incoming request,
// stores it in the context, and adds it to the response headers.
func RequestID(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Generate a new UUID for this request.
        id := uuid.New().String()

        // Store the ID in a new context derived from the request's context.
        // r.Context() returns the existing context, we build on top of it,
        // not replace it, so any previously set values are preserved.
        ctx := context.WithValue(r.Context(), RequestIDKey, id)

        // Add the ID to the response header.
        // This lets clients (and load balancers) correlate their logs with yours.
        w.Header().Set("X-Request-ID", id)

        // This is the line from the previous explanation:
        // create a copy of the request with the new context attached,
        // then pass it to the next handler in the chain.
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

// GetRequestID is a helper to retrieve the request ID from a context.
// Handlers call this instead of reaching into the context directly.
func GetRequestID(ctx context.Context) string {
    id, _ := ctx.Value(RequestIDKey).(string)
    return id
}

Logger middleware

Logs every request: method, path, duration, and status code. But, before looking at the full code, let's first try to understand how this logger middleware works under the hood.

What is http.ResponseWriter?

Before looking at the code, it helps to understand the type we're about to wrap.

http.ResponseWriter is an interface, every handler you write receives one as its first argument. It's how your handler sends a response back to the client: the status code, the headers, and the body.

// http.ResponseWriter is an interface with three methods:
type ResponseWriter interface {
    Header() http.Header         // get/set response headers
    Write([]byte) (int, error)   // write the response body
    WriteHeader(statusCode int)  // write the status code (200, 404, 500...)
}

A typical handler uses it like this:

func myHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)        // write the status code
    w.Write([]byte("hello!"))            // write the body
}

The problem: you can't read the status code back

Here's the catch. http.ResponseWriter lets you write a status code, but it has no method to read it back afterward. There's no w.StatusCode() you can call. Once you write it, it's gone, sent straight to the client over the network.

This is a problem for our Logger middleware. Remember the relay race model: Logger calls next.ServeHTTP(w, r), waits for the handler to finish, and only then logs the result. But by the time the handler finishes, it has already called w.WriteHeader(404) (or whatever status it chose), and Logger has no way to find out what that status was. The information is gone.

The solution: wrap it in our own struct

To solve this, we create our own type that:

Embeds the original http.ResponseWriter, so it still behaves exactly the same way for the handler
Adds a field to remember the status code
Intercepts the one method we care about (WriteHeader) to save the value before passing it through.

// responseWriter wraps http.ResponseWriter to capture the status code.
// http.ResponseWriter doesn't expose the status code after it's written,
// so we intercept WriteHeader to save it.
type responseWriter struct {
    http.ResponseWriter            // embed the original, we get all its methods for free
    statusCode         int         // the status code the handler wrote
}

The embedded http.ResponseWriter is the key part. In Go, embedding a type inside a struct means the outer struct automatically gets all of the embedded type's methods, for free, with no extra code. So our responseWriter already has Header(), Write(), and WriteHeader() just by embedding http.ResponseWriter. As far as the handler is concerned, our responseWriter looks and behaves exactly like a normal http.ResponseWriter, because it satisfies the same interface.

The only thing left to do is override WriteHeader, the one method we actually need to intercept:

// WriteHeader intercepts the status code before passing it through.
func (rw *responseWriter) WriteHeader(code int) {
    rw.statusCode = code               // save it for logging
    rw.ResponseWriter.WriteHeader(code) // pass it through to the real writer
}

This is the trick: when our responseWriter defines its own WriteHeader method, that one overrides the embedded version. So when a handler calls w.WriteHeader(404), it actually calls our version, which saves 404 into rw.statusCode, then calls the original ResponseWriter.WriteHeader(404) so the client still receives the real response. The handler never notices anything different happened.

Now Logger can read wrapped.statusCode after the handler finishes, something that was impossible with a plain http.ResponseWriter.

Here's the full middleware:

// internal/middleware/logger.go
package middleware

import (
    "log/slog"
    "net/http"
    "time"
)

// responseWriter wraps http.ResponseWriter to capture the status code.
// http.ResponseWriter doesn't expose the status code after it's written,
// so we intercept WriteHeader to save it.
type responseWriter struct {
    http.ResponseWriter            // embed the original — we get all its methods for free
    statusCode         int         // the status code the handler wrote
}

// WriteHeader intercepts the status code before passing it through.
func (rw *responseWriter) WriteHeader(code int) {
    rw.statusCode = code               // save it for logging
    rw.ResponseWriter.WriteHeader(code) // pass it through to the real writer
}

// Logger logs each request's method, path, status code, duration,
// and request ID (if set by the RequestID middleware).
func Logger(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Record when the request arrived.
        start := time.Now()

        // Wrap the ResponseWriter so we can capture the status code.
        wrapped := &responseWriter{
            ResponseWriter: w,
            statusCode:     http.StatusOK, // default to 200 if WriteHeader is never called
        }

        // Call the next handler — this is where the actual work happens.
        next.ServeHTTP(wrapped, r)

        // By the time we reach here, the handler has finished.
        // We can now log everything we know about this request.
        slog.Info("request",
            "method",     r.Method,
            "path",       r.URL.Path,
            "status",     wrapped.statusCode,
            "duration",   time.Since(start).String(),
            "requestID",  GetRequestID(r.Context()), // set by RequestID middleware
        )
    })
}

Auth middleware

Checks for a valid Bearer token before allowing the request through.

This one looks more complicated than RequestID and Logger at first glance, it's a function that returns a function that returns a function. Let's slow down and unpack why, before looking at the full code.

Why Auth needs an extra layer

RequestID and Logger both have this shape:

func RequestID(next http.Handler) http.Handler {
    // ...
}

One input (next), one output (http.Handler). That's the standard middleware shape we covered earlier.

But Auth needs something extra: a configurable token. Every other middleware works the same way no matter what, but Auth needs to know which token counts as valid, and that's different for every app (and usually comes from an environment variable, not a hardcoded value).

So Auth can't just be a middleware, it needs to be a function that produces a middleware, once you tell it which token to check:

// Auth is NOT middleware itself.
// It's a function that BUILDS a middleware, once you give it a token.
func Auth(validToken string) func(http.Handler) http.Handler {
    // ...
}

Breaking down the three layers

Here's the same function with each layer labeled:

//        ┌─── Layer 1: takes your config (the token)
//        │
func Auth(validToken string) func(http.Handler) http.Handler {
    //                        └─── Layer 2: this is what gets returned —
    //                             a real middleware function, with the
    //                             standard func(http.Handler) http.Handler shape

    return func(next http.Handler) http.Handler {
        //     └─── Layer 2 starts here. This is now a normal middleware,
        //          identical in shape to RequestID and Logger.

        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            //   └─── Layer 3: the actual handler logic that runs per-request.
            //        This is the same as the inner function in every middleware
            //        you've already seen.

            // ... auth check logic goes here ...
        })
    }
}

Three layers, three different jobs:

Layer	Runs when	Job
1 — `Auth(validToken)`	Once, at startup	Receives your config (the token)
2 — `func(next http.Handler) http.Handler`	Once, when you build your middleware chain	Receives the next handler to wrap
3 — `func(w, r)`	On every single request	Does the actual work — checks the header

Why this pattern, and how you use it

The payoff is that calling Auth looks almost identical to using RequestID or Logger, you just add one extra set of parentheses to pass in the token:

// RequestID and Logger take no config, pass them directly
middleware.RequestID
middleware.Logger

// Auth takes config, call it first to get back a real middleware
middleware.Auth("my-secret-token")   // ← this call returns a middleware function

That's it. Auth("my-secret-token") runs Layer 1 immediately and hands you back Layer 2, a normal func(http.Handler) http.Handler, ready to be used exactly like any other middleware in your Chain:

Chain(mux,
    middleware.RequestID,                // sets the request ID first
    middleware.Logger,                   // then logs — sees the ID
    middleware.Auth("my-secret-token"),  // called with () — returns a middleware
)

If this still feels unfamiliar, it helps to compare it to something simpler, a function that builds a greeting function:

// Same shape, much simpler example.
// MakeGreeter takes a name and returns a NEW function
// that already "remembers" that name.
func MakeGreeter(name string) func() string {
    return func() string {
        return "Hello, " + name + "!"
    }
}

sayHiToAlice := MakeGreeter("Alice") // returns a function
fmt.Println(sayHiToAlice())          // "Hello, Alice!"

Auth does the exact same thing, it just returns a middleware function instead of a greeting function. Once you see MakeGreeter, Auth(validToken) stops looking complicated; it's the same "function that builds and returns another function" pattern, just doing more useful work inside.

Now here's the full code with that context in mind:

// internal/middleware/auth.go
package middleware

import (
    "net/http"
    "strings"
)

// Auth checks for a valid Bearer token in the Authorization header.
// If the token is missing or invalid, it returns 401 and stops the chain —
// next.ServeHTTP is never called, so the handler never runs.
func Auth(validToken string) func(http.Handler) http.Handler {
    // Return a middleware function — this pattern lets us configure
    // middleware with parameters (the token) while keeping the standard shape.
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            // Read the Authorization header.
            authHeader := r.Header.Get("Authorization")

            // Check the format: "Bearer <token>"
            parts := strings.SplitN(authHeader, " ", 2)
            if len(parts) != 2 || parts[0] != "Bearer" {
                // Respond with 401 and stop. next.ServeHTTP is never called.
                http.Error(w, "missing or malformed token", http.StatusUnauthorized)
                return
            }

            // Validate the token.
            // In a real app you'd verify a JWT or look up the token in a database.
            if parts[1] != validToken {
                http.Error(w, "invalid token", http.StatusUnauthorized)
                return
            }

            // Token is valid — pass the request to the next handler.
            next.ServeHTTP(w, r)
        })
    }
}

Chaining middleware together

Now let's wire everything up. But first, create your own secret token and paste it into your main.go file, you can create it from here. Remember to also include that token in your request. You can see how to do it in the section about testing the server.

The standard Go way to chain middleware is to nest the calls:

// cmd/server/main.go
package main

import (
    "fmt"
    "net/http"

    "github.com/yourname/myapp/internal/middleware"
)

// userHandler is the actual business logic.
// It doesn't know about auth, logging, or request IDs —
// all of that is handled by middleware before it runs.
func userHandler(w http.ResponseWriter, r *http.Request) {
    // Retrieve the request ID that was set by RequestID middleware.
    id := middleware.GetRequestID(r.Context())
    fmt.Fprintf(w, "hello! your request ID is %s\n", id)
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/user", userHandler)

    // Chain middleware by wrapping from the outside in.
    // The outermost middleware runs first on the way IN,
    // and last on the way OUT.
    //
    // RequestID must wrap Logger — not the other way around.
    // Logger only ever sees the *r it directly receives. If RequestID
    // runs AFTER Logger, Logger is holding the original request,
    // not the enriched one RequestID produces — so the ID would be empty.
    //
    // Execution order for a request:
    //   RequestID → Logger → Auth → userHandler
    //               (then back out in reverse)
    handler := middleware.RequestID(
        middleware.Logger(
            middleware.Auth("your-secret-token")(
                mux,
            ),
        ),
    )

    http.ListenAndServe(":8080", handler)
}

Testing the server

# Start the server
go run ./cmd/server

# Valid request — should return 200 with a request ID
curl -H "Authorization: Bearer your-secret-token" http://localhost:8080/user

# Missing token — should return 401
curl http://localhost:8080/user

# Invalid token — should return 401
curl -H "Authorization: Bearer wrong-token" http://localhost:8080/user

Server logs:

2026/06/21 20:46:13 INFO request method=GET path=/user status=200 duration=7.208µs requestID=e667de2b-310a-4038-9891-2d948613e93b
2026/06/21 20:46:15 INFO request method=GET path=/user status=200 duration=90.667µs requestID=23a61730-4dd2-4b01-8dc5-986fbcb46436

Every request gets logged. The request ID ties your server log to the X-Request-ID header in the response, if a client reports a bug, they can give you their request ID and you can find exactly that request in your logs.

Best practices

Order matters, RequestID before Logger, Logger before Auth

The outermost middleware runs first on the way in and last on the way out. Two rules drive the order here:

RequestID must be outermost (or at least before Logger), Logger can only read context values from the exact *http.Request it was handed. context.WithValue and r.WithContext don't mutate the original request, they return a new one. If RequestID runs after Logger, Logger is stuck holding the original, un-enriched request, GetRequestID will always come back empty. If you are new to Go contexts, you can check this post about it; it explains everything you need to know about Go contexts.
Logger should wrap Auth, so that failed (401) requests still get logged, not just successful ones.

// Good order
Chain(mux,
    middleware.RequestID, // outermost: enriches the request first
    middleware.Logger,    // sees the enriched request — requestID shows up in logs
    middleware.Auth(...), // innermost: blocks invalid requests, but they're still logged
)

// Bad order — this is the bug you'll hit if you swap these two
Chain(mux,
    middleware.Logger,    // ← runs first, but hasn't seen RequestID's context yet
    middleware.RequestID, // ← enriches the request, but only for handlers AFTER this point
    middleware.Auth(...),
)
// Logger's slog.Info call reads r.Context() from its OWN copy of r —
// which was captured before RequestID ever ran. requestID="" every time.

The general rule: any middleware that reads a context value must be positioned after (further in than) the middleware that sets it.

Never call next.ServeHTTP more than once

Calling it twice sends a double response, which causes a panic in Go's HTTP stack. Always return after an early exit:

// Good — returns after writing the error
if !valid {
    http.Error(w, "unauthorized", http.StatusUnauthorized)
    return // ← critical: stops execution here
}
next.ServeHTTP(w, r)

// Bad — falls through and calls next anyway
if !valid {
    http.Error(w, "unauthorized", http.StatusUnauthorized)
    // missing return — next.ServeHTTP still runs!
}
next.ServeHTTP(w, r)

Use unexported context keys

Always define context keys as a custom unexported type, not a plain string. This prevents key collisions between packages:

// Bad — any package using the string "userID" will collide
ctx = context.WithValue(ctx, "userID", id)

// Good — only this package can access this key
type key string
const userIDKey key = "userID"
ctx = context.WithValue(ctx, userIDKey, id)

Tips for advanced users

Tip 1: Use a third-party router for cleaner middleware

The standard library's http.ServeMux applies middleware to all routes at once. If you need per-route middleware (auth on some routes, not others), a router like chi or gorilla/mux makes this clean:

r := chi.NewRouter()

r.Use(middleware.Logger)    // applies to all routes
r.Use(middleware.RequestID) // applies to all routes

r.Group(func(r chi.Router) {
    r.Use(middleware.Auth("secret")) // applies only to routes in this group
    r.Get("/admin", adminHandler)
    r.Get("/users", usersHandler)
})

r.Get("/health", healthHandler) // no auth required

Tip 2: Capture response body size

Extend responseWriter to also track bytes written — useful for monitoring:

type responseWriter struct {
    http.ResponseWriter
    statusCode int
    bytes      int
}

func (rw *responseWriter) Write(b []byte) (int, error) {
    n, err := rw.ResponseWriter.Write(b)
    rw.bytes += n // track how many bytes were written
    return n, err
}

Tip 3: Recover from panics

A panic in any handler crashes the goroutine handling that request. A recovery middleware catches it and returns a 500 instead of crashing:

func Recovery(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        defer func() {
            if err := recover(); err != nil {
                slog.Error("panic recovered", "error", err, "requestID", GetRequestID(r.Context()))
                http.Error(w, "internal server error", http.StatusInternalServerError)
            }
        }()
        next.ServeHTTP(w, r)
    })
}

Add this as the outermost middleware so it catches panics from every layer:

Chain(mux,
    middleware.Recovery,  // outermost — catches panics from every layer below
    middleware.RequestID, // sets the request ID next
    middleware.Logger,    // sees the enriched request — requestID shows up in logs
    middleware.Auth("secret"),
)

Summary

Middleware is just a function that wraps a handler and returns a new one. Everything else follows from that.

Three things to take away:

next.ServeHTTP(w, r.WithContext(ctx)), this is the line that passes the request forward with an enriched context. Understanding it unlocks the whole pattern.
Order matters, outermost middleware runs first on the way in, last on the way out. Anything that sets a context value (like RequestID) must come before anything that reads it (like Logger), or the value will be empty.
Always return after early exits, calling next.ServeHTTP after writing an error response causes a double response panic.

The three middleware functions in this post, RequestID, Logger, and Auth, are production-ready. Add Recovery and you have a solid foundation for any Go HTTP server.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

Go types explained: the foundation you need before writing HTTP handlers

Fer Rios — Mon, 22 Jun 2026 14:22:54 +0000

A quick introduction to Go types

According to https://go.dev/ref/spec#Types, in Go, a type specifies a set of values, along with operations and methods specific to those values.

This is something fundamental about Go: you can define your own types based on existing ones, and those types can have methods attached to them.

You've probably seen this with structs:

// A struct type — groups fields together
type User struct {
    Name  string
    Email string
}

// A method attached to the User type
func (u User) Greet() string {
    return "Hello, " + u.Name
}

But Go lets you do the same thing with any type, including functions.

Function types

In Go, functions are values. You can assign them to variables, pass them as arguments, and — most importantly here, define a named type based on a function signature:

// This defines a new type called "Greeter"
// whose underlying type is a function that takes a string and returns a string.
type Greeter func(name string) string

Now Greeter is a type, just like int or string or User. Any function with the right signature (func(string) string) is automatically a Greeter.

// This function matches the Greeter signature
func sayHello(name string) string {
    return "Hello, " + name
}

// So we can assign it to a Greeter variable
var g Greeter = sayHello

fmt.Println(g("Alice")) // Hello, Alice

Attaching methods to a function type

Here's where it gets interesting. Because Greeter is now a named type, you can attach methods to it, exactly like you would with a struct:

// A method on a function type — this is valid Go
func (g Greeter) Shout(name string) string {
    return strings.ToUpper(g(name)) // call the function, then shout it
}

fmt.Println(g.Shout("Alice")) // HELLO, ALICE

This might look strange at first. A function with a method? But it makes perfect sense in Go. Greeter is just a type, and any type can have methods.

Why this matters for HTTP handlers

Now that you understand function types, let's look at how Go's HTTP package uses this exact pattern to make your life easier.

Go's HTTP package has two things that are easy to confuse at first: http.Handler and http.HandlerFunc. They sound almost identical, but they play different roles: one is an interface, and one is a type.

http.Handler — the interface

An HTTP handler in GO is an object or function responsible for processing an incoming HTTP request and constructing an HTTP response.

http.Handler is the interface that represents anything that can handle an HTTP request. It has exactly one method:

// http.Handler is an interface — a contract.
// It says: "I don't care what you are.
// As long as you have a ServeHTTP method, you can handle HTTP requests."
type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

http.ResponseWriter: An interface used to construct and send the HTTP response (setting headers, status codes, and writing the body).
*http.Request: A pointer to a struct containing all incoming data (headers, query parameters, URL path, and request body).

So, to be recognized as a handler, a type must implement the http.Handler interface by defining a ServeHTTP.

This is what Go's HTTP server actually works with internally. When you call http.ListenAndServe, the second argument must be an http.Handler:

// The second argument is an http.Handler —
// Go's server calls ServeHTTP on it for every incoming request.
http.ListenAndServe(":8080", myHandler)

Any type that has a ServeHTTP method satisfies this interface automatically. A struct, a function type, anything, as long as it has that one method.

http.HandlerFunc — the type

Here's the problem: writing a full struct with a ServeHTTP method for every single handler in your app is verbose. Most of the time, you just want to write a plain function:

// This is clean and simple, just a function.
// But it's NOT an http.Handler because it has no ServeHTTP method.
func greetUser(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello!")
}

This function can't be passed to http.ListenAndServe directly, it's not an http.Handler. It has no ServeHTTP method.

This is exactly the problem http.HandlerFunc solves. It's a named function type with a ServeHTTP method attached, which makes it an http.Handler:

// HandlerFunc is a named type based on a function signature.
// Any function that matches func(ResponseWriter, *Request)
// is automatically compatible with this type.
type HandlerFunc func(ResponseWriter, *Request)

// ServeHTTP is attached to HandlerFunc.
// It just calls the function itself — f is the function, so f(w, r) runs it.
// This single method is what makes HandlerFunc satisfy http.Handler.
func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
    f(w, r)
}

Now Go can convert your plain function into an http.HandlerFunc, which has ServeHTTP, which satisfies http.Handler.

You can imagine that this is what Go does under the hood:

var fn http.HandlerFunc = greetUser  // convert the function to HandlerFunc
var h  http.Handler    = fn          // HandlerFunc satisfies http.Handler

But, in practice, you never write these steps manually; you use the http.HandleFunc function.

http.HandleFunc

// Registers the "/hello" route on the default mux.
// http.HandleFunc converts greetUser to HandlerFunc internally —
// you never see that conversion happen.
http.HandleFunc("/hello", greetUser)

http.HandleFunc register a route on the default server mux. It takes a path and a handler function, converts the function to a HandlerFunc internally, and registers it.

http.HandleFunc actually uses http.HandlerFunc internally, it's just hiding it from you:

// This is roughly what http.HandleFunc does under the hood:
func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
    // converts your function to HandlerFunc, then registers it
    DefaultServeMux.Handle(pattern, HandlerFunc(handler))
    //                                ^^^^^^^^^^^^^
    //                                HandlerFunc TYPE doing the conversion
}

So http.HandleFunc is convenience on top of http.HandlerFunc. When you call http.HandleFunc("/hello", greetUser), Go silently wraps greetUser in an http.HandlerFunc and registers it. You only need to use http.HandlerFunc directly when you're building something lower-level, like middleware, where you need to return an http.Handler explicitly.

Putting it all together

Here's the full picture in one example:

// A plain function — clean and simple to write
func greetUser(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello!")
}

// A struct handler — useful when you need to hold state like a DB client
type UserHandler struct {
    db *sql.DB // injected dependency
}

func (h *UserHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // can use h.db here
    fmt.Fprintln(w, "Hello from UserHandler!")
}

func main() {
    mux := http.NewServeMux()

    // http.HandleFunc converts greetUser → HandlerFunc → Handler automatically
    mux.HandleFunc("/hello", greetUser)

    // UserHandler already has ServeHTTP — it satisfies http.Handler directly
    mux.Handle("/user", &UserHandler{db: myDB})

    // Both routes are registered on the same mux —
    // the server calls ServeHTTP on whichever one matches the request path.
    http.ListenAndServe(":8080", mux)
}

TIP: Use a plain function (via HandlerFunc) for simple handlers. Use a struct for handlers that need dependencies like a database client or a config object. The server doesn't care which one you chose — it just calls ServeHTTP.

Summary

Think of it this way:

http.Handler is the job description: "must be able to handle HTTP requests"
http.HandlerFunc is one way to fulfill that job description: "a plain function that handles HTTP requests"
Your struct with a ServeHTTP method is another way to fulfill it

The server only knows about http.Handler. It never knows whether you used a struct or a HandlerFunc — and it doesn't need to. That's the power of the interface.

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

Building from Asunción, Paraguay 🇵🇾

DEV Community: Fer Rios

Build an MCP Server in Go (Part 3): How IRSA and EKS Pod Identity Actually Work

How pod permissions actually work on EKS

The problem neither mechanism existed to solve

What a Kubernetes ServiceAccount actually is

RBAC: what a ServiceAccount is actually allowed to do

IRSA: a pod proves who it is directly to AWS

Pod Identity: an in-cluster agent brokers the exchange

The EKS Auth API

Why this distinction matters for diagnosis

Summary: one rule that matters most

Three things worth carrying forward:

Let's connect!

LeetCode 151: Reverse Words in a String in Go

The problem that looks simple until the whitespace shows up

Approach 1: split, reverse, join

Step 1: Split, with whitespace handled for free

Step 2: Reverse the slice of words

Step 3: Join back with single spaces

Full Solution

Trace

Approach 2: reverse-then-reverse, in place

Why "reverse, then reverse" works

The part that's actually new: whitespace compaction, in place

Reverse the whole compacted string

Reverse each word back to normal

Full Solution

What "extra space" (auxiliary space) means

What "truly O(1)" looks like, concretely

So which one is actually "O(1) extra space"? Neither, really, and here's why that matters

Then what's actually different between the two approaches?

Summary

Let's connect!

Build an MCP Server in Go (Part 2): Building the MCP tool layer

What MCP actually is

Two ways tools and calls actually travel: stdio vs. HTTP

Project structure

The tool layer

Anatomy of a tool file

list_pods

get_pod_logs

describe_pod

get_events

check_endpoints

describe_ingress

describe_httproute

list_configmap_keys

pod_metrics

Wiring it together

Running it end-to-end

Engineering practices that matter here

Common gotchas

Summary

Let's connect!

Build an MCP Server in Go (Part 1): Designing a diagnostic-grade Kubernetes client

Asking instead of running

ferctl vs. an MCP server: two answers to the same problem

Project structure

Designing the KubeClient interface

Pods, the core of "why is my app broken"

Events, often the fastest path to a root cause

Workloads, rollout and scheduling problems above the pod

Nodes, capacity and scheduling context

Network, is anything actually answering

Ingress on EKS, is the ALB even pointing here

Gateway API, the newer, cluster-portable entry point

Config, keys only, never values

Storage, volumes that can't bind

Metrics, the same extension the client-go post already teed up

Read-only by construction

Summary

Let's connect!

LeetCode #345 in Go: reverse vowels of a string, and how strings, bytes, and runes work in Go

Why I started this series

The challenge

LeetCode #345

Solution Approach

The core idea: two pointers moving toward each other

Why this pattern fits "reverse vowels"

Step-by-step logic

`list_pods`

`get_pod_logs`

`describe_pod`

`get_events`

`check_endpoints`

`describe_ingress`

`describe_httproute`

`list_configmap_keys`

`pod_metrics`

`ferctl` vs. an MCP server: two answers to the same problem

Designing the `KubeClient` interface

How to get mutability: convert to `[]byte`

`[]byte` vs `[]rune`: why it matters