Build an MCP Server in Go (Part 1): Designing a diagnostic-grade Kubernetes client

This post designs the Kubernetes client. The next post wraps it as an MCP server and wires it to an agent.

Asking instead of running

There's a familiar rhythm to debugging a bad deploy: kubectl get pods, spot the CrashLoopBackOff, kubectl logs --previous, still not obvious, kubectl describe pod, scroll to Events, cross-reference with kubectl get events, maybe check if the Service even has endpoints. Five or six commands, in your head, in a specific order, because you've done this enough times to know the order.

That order is exactly what an AI agent can execute for you, if you give it the right tools, ask "why is checkout-service failing?" and have it chain through the same commands you would have, arriving at an actual answer instead of a wall of YAML. The protocol that makes an agent capable of calling tools like that is the Model Context Protocol (MCP), and building that server is part 2 of this series.

This post is about what comes first and matters more: the Kubernetes client those tools will eventually sit on top of. An agent is only as good as what it's allowed to ask for. Hand it a client with one ListPods method and it can list pods, nothing else. Hand it a client that mirrors what a senior engineer actually checks when something's broken, pod state, events, endpoints, node capacity, rollout history, and it can genuinely diagnose. That client is a real piece of engineering on its own, independent of whether an LLM ever touches it, which is why it gets its own post before MCP enters the picture at all.

ferctl vs. an MCP server: two answers to the same problem

ferctl was my first attempt at this: a Cobra CLI, backed by client-go, that wraps common troubleshooting checks into subcommands: ferctl top, ferctl logs, that shape. It's fast, deterministic, and scriptable. Run ferctl describe-pod checkout-service, get the same structured output every time, pipe it intojq, drop it into a CI step, no ambiguity about what ran or why. That predictability is exactly what a CLI is good at.

What it doesn't do is investigate. ferctl runs the one command you gave it, you're still the one who has to know that a CrashLoopBackOff means "check previous logs, then check events", and you're still typing each step by hand. It's a faster way to run the commands you already know, not a way to skip learning them.

An MCP server flips that. You ask "why is checkout-service failing" once, and the agent decides the sequence, list pods, notices the restart count, pulls previous logs, cross-references events, the way a ferctl invocation never will, because no single subcommand can adapt its next step to what the last one returned. That's the shift this series is really about: from "a faster way to run known commands" to "something that can figure out which commands to run." It's also, frankly, the more current way to build this kind of tooling, a fixed CLI surface is a fine interface for a human who already knows the shape of the problem, but an LLM-driven agent chaining calls dynamically is a better fit for the actual shape of debugging, which rarely follows a script.

None of that makes ferctl obsolete. A CLI is still the right tool when you want guaranteed, repeatable output, a CI health check, a pre-deploy gate, anything where non-determinism is a bug, not a feature. An agent is the right tool when the problem is exploratory, and you don't yet know which three commands you'll need. They're not really competitors; they're two different interfaces that answer "is my cluster healthy" in two different situations. And notably, both could sit on top of the exact same KubeClient this post builds, the interface doesn't care whether its caller is a Cobra command or an MCP tool handler, which is itself a small argument for designing the client first, independent of either.

---

Project structure

go-k8s-mcp-server/
├── go.mod
└── internal/
    └── kubernetes/
        ├── client.go           ← KubeClient interface, constructor
        ├── pods.go             ← PodClient implementation
        ├── workloads.go        ← WorkloadClient implementation
        ├── nodes.go            ← NodeClient implementation
        ├── events.go           ← EventClient implementation
        ├── network.go          ← NetworkClient implementation
        ├── ingress.go          ← IngressClient implementation
        ├── gateway.go          ← GatewayClient implementation
        ├── config.go           ← ConfigClient implementation
        ├── storage.go          ← StorageClient implementation
        └── metrics.go          ← MetricsClient implementation

This is only the client half of the project, next post adds cmd/k8s-mcp-server/ and internal/tools/ on top of this same internal/kubernetes/ package. The internal/ boundary and one-package-per-domain layout follow the same convention laid out in Go packages and modules explained: internal/ for everything the Go toolchain should keep private to this module, one focused file per concern rather than one large one.

If you haven't set up a client-go connection before, this post assumes that groundwork. I covered kubeconfig loading, the Clientset, and how API groups work in Talking to Kubernetes from Go: a practical client-go guide. Everything below builds directly on that same connection pattern, same rest.Config setup, same in-cluster/out-of-cluster fallback, same "wrap the Clientset behind an interface" philosophy. What's new here is the shape of the interface itself.

---

Designing the KubeClient interface

A tempting first move is one fat interface with every method the agent might ever need. Resist it. A twenty-method interface is hard to test, hard to fake, and it hides the fact that these methods address genuinely different diagnostic concerns: some are about pods, some are about scheduling, some are about networking. Go rewards small interfaces, so we compose one from several.

If you're starting this project fresh rather than following along from the client-go post, pin the packages to your cluster's version, same as before, mixing minor versions across them breaks at compile time. k8s.io/metrics and sigs.k8s.io/gateway-api are included here too, even though neither is used until their respective sub-clients further down, because the client struct and NewClient constructor right below already need both:

mkdir go-k8s-mcp-server && cd go-k8s-mcp-server
go mod init github.com/FerRiosCosta/go-k8s-mcp-server

go get k8s.io/client-go@v0.36.2
go get k8s.io/api@v0.36.2
go get k8s.io/apimachinery@v0.36.2
go get k8s.io/metrics@v0.36.2
go get sigs.k8s.io/gateway-api@v1.3.0
go mod tidy

// internal/kubernetes/client.go
package kubernetes

import (
    "context"
    "fmt"

    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
    metricsclientset "k8s.io/metrics/pkg/client/clientset/versioned"
    gatewayclientset "sigs.k8s.io/gateway-api/pkg/client/clientset/versioned"
)

// KubeClient is the full set of read-only diagnostic operations the
// MCP server exposes as tools. It's composed from smaller, domain-specific
// interfaces so each one stays independently testable and each
// implementation file has exactly one reason to change.
//
// Every method here is read-only by design. There is no Delete, Scale,
// or Patch anywhere in this interface, that's not an accident, it's
// the security boundary. See "Read-only by construction" below.
type KubeClient interface {
    PodClient
    WorkloadClient
    NodeClient
    EventClient
    NetworkClient
    IngressClient
    GatewayClient
    ConfigClient
    StorageClient
    MetricsClient
}

// client is the concrete implementation every sub-interface method
// below is defined on. It holds three Clientsets: the core Clientset
// for everything except metrics and Gateway API, a metrics-server
// Clientset (the split the client-go post flagged as needed once
// k8s.io/metrics entered the picture), and a Gateway API Clientset,
// Gateway API is CRD-based, so it isn't reachable through the core
// Clientset the way Ingress is.
type client struct {
    clientset        kubernetes.Interface
    metricsClientset metricsclientset.Interface
    gatewayClientset gatewayclientset.Interface
}

// NewClient builds a KubeClient the same way kubectl resolves its
// config: in-cluster config if running inside a pod, otherwise
// $KUBECONFIG if set, otherwise ~/.kube/config, and whichever
// context is marked current-context in that file, automatically.
// There's no path parameter here on purpose: hardcoding a path means
// this server silently ignores context switches made with
// `kubectl config use-context`, which is exactly the surprise you
// don't want from a diagnostic tool.
func NewClient() (KubeClient, error) {
    config, err := buildConfig()
    if err != nil {
        return nil, fmt.Errorf("build config: %w", err)
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create clientset: %w", err)
    }

    metricsClientset, err := metricsclientset.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create metrics clientset: %w", err)
    }

    gatewayClientset, err := gatewayclientset.NewForConfig(config)
    if err != nil {
        return nil, fmt.Errorf("create gateway clientset: %w", err)
    }

    return &client{
        clientset:        clientset,
        metricsClientset: metricsClientset,
        gatewayClientset: gatewayClientset,
    }, nil
}

func buildConfig() (*rest.Config, error) {
    if config, err := rest.InClusterConfig(); err == nil {
        return config, nil
    }

    // NewDefaultClientConfigLoadingRules checks $KUBECONFIG first
    // (colon-separated on Linux/macOS, semicolon on Windows, merging
    // multiple files if listed), then falls back to ~/.kube/config.
    // ConfigOverrides{} is empty on purpose, an empty overrides
    // struct means "use whatever current-context is set," the same
    // default clientcmd.BuildConfigFromFlags gave us before, just
    // without hardcoding the path ourselves.
    loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
    overrides := &clientcmd.ConfigOverrides{}
    return clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, overrides).ClientConfig()
}

Same packages as the client-go post's client.go, plus two additions: metricsclientset from k8s.io/metrics, and gatewayclientset from sigs.k8s.io/gateway-api, both alongside the core Clientset. gatewayClientset will show NewForConfig succeed even against a cluster with no Gateway API CRDs installed; it's just a REST client pointed at a set of API paths, and only fails once you actually call it against a cluster that doesn't serve them. That failure mode gets called out concretely in the Gateway API section below.

Every method in the sections below is defined on this same *client type, one file per domain.

Pods, the core of "why is my app broken"

// internal/kubernetes/pods.go
package kubernetes

import (
    "context"
    "fmt"
    "io"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// PodClient covers the operations that answer "what's wrong with this pod."
type PodClient interface {
    // ListPods returns pods in namespace. An empty namespace returns
    // pods from every namespace, same convention as ListPods in the
    // client-go guide.
    ListPods(ctx context.Context, namespace string) (*corev1.PodList, error)

    // GetPod returns a single pod by name, used once the agent has
    // narrowed down which pod it cares about.
    GetPod(ctx context.Context, namespace, name string) (*corev1.Pod, error)

    // GetPodLogs returns log output for one container in a pod.
    GetPodLogs(ctx context.Context, namespace, name string, opts LogOptions) (string, error)
}

// LogOptions bounds a log request. This type exists specifically so an
// agent can never accidentally pull megabytes of logs in a single tool
// call, TailLines and SinceSeconds are the caps, not suggestions.
type LogOptions struct {
    Container    string // empty selects the pod's first container
    Previous     bool   // true fetches the last terminated instance's logs,
                         // essential for CrashLoopBackOff, since the *current*
                         // instance often hasn't logged the failure yet
    TailLines    int64  // 0 falls back to a safe default, never "unbounded"
    SinceSeconds int64  // 0 means no time bound
}

const defaultLogTailLines = 200

func (c *client) ListPods(ctx context.Context, namespace string) (*corev1.PodList, error) {
    pods, err := c.clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pods namespace=%q: %w", namespace, err)
    }
    return pods, nil
}

func (c *client) GetPod(ctx context.Context, namespace, name string) (*corev1.Pod, error) {
    pod, err := c.clientset.CoreV1().Pods(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get pod namespace=%q name=%q: %w", namespace, name, err)
    }
    return pod, nil
}

func (c *client) GetPodLogs(ctx context.Context, namespace, name string, opts LogOptions) (string, error) {
    tail := opts.TailLines
    if tail <= 0 {
        tail = defaultLogTailLines
    }

    podLogOpts := &corev1.PodLogOptions{
        Container: opts.Container,
        Previous:  opts.Previous,
        TailLines: &tail,
    }
    if opts.SinceSeconds > 0 {
        podLogOpts.SinceSeconds = &opts.SinceSeconds
    }

    req := c.clientset.CoreV1().Pods(namespace).GetLogs(name, podLogOpts)
    stream, err := req.Stream(ctx)
    if err != nil {
        return "", fmt.Errorf("stream logs namespace=%q pod=%q: %w", namespace, name, err)
    }
    defer stream.Close()

    data, err := io.ReadAll(stream)
    if err != nil {
        return "", fmt.Errorf("read logs namespace=%q pod=%q: %w", namespace, name, err)
    }
    return string(data), nil
}

Note that TailLines defaults to 200, not 0 meaning unbounded. That single line is the difference between a tool an agent can call freely and a tool that occasionally hands it 40,000 lines of stack traces and blows past its context window on one call.

Events, often the fastest path to a root cause

// internal/kubernetes/events.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EventClient covers the Events API, which frequently surfaces a root
// cause, a failed scheduling attempt, an image pull error, before
// anything else does. It's queried by involved object, not by name,
// which is different enough from PodClient to earn its own file.
type EventClient interface {
    // GetEvents returns events in namespace, optionally filtered to
    // those involving a specific object name (e.g. a Pod).
    // An empty involvedObjectName returns all events in the namespace.
    GetEvents(ctx context.Context, namespace, involvedObjectName string) ([]corev1.Event, error)
}

func (c *client) GetEvents(ctx context.Context, namespace, involvedObjectName string) ([]corev1.Event, error) {
    opts := metav1.ListOptions{}
    if involvedObjectName != "" {
        opts.FieldSelector = "involvedObject.name=" + involvedObjectName
    }

    events, err := c.clientset.CoreV1().Events(namespace).List(ctx, opts)
    if err != nil {
        return nil, fmt.Errorf("list events namespace=%q object=%q: %w", namespace, involvedObjectName, err)
    }
    return events.Items, nil
}

Workloads, rollout and scheduling problems above the pod

Not every failure lives at the pod level. A deployment that's stuck rolling out, or one whose pods belong to two different generations at once, needs a different query shape entirely: you're not asking about one pod anymore, you're asking about the relationship between a Deployment and the ReplicaSets it owns.

// internal/kubernetes/workloads.go
package kubernetes

import (
    "context"
    "fmt"

    appsv1 "k8s.io/api/apps/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// WorkloadClient covers Deployments and the ReplicaSets they own,
// the layer above individual pods, where rollout and scheduling
// problems that span multiple pods actually live.
type WorkloadClient interface {
    // GetDeployment returns a Deployment's spec and status, including
    // desired vs. available replica counts and rollout conditions.
    GetDeployment(ctx context.Context, namespace, name string) (*appsv1.Deployment, error)

    // ListReplicaSets finds every ReplicaSet matching labelSelector in
    // a namespace. This is what surfaces orphaned or stuck ReplicaSets,
    // the classic "Deployment says 3/3 but two of those pods belong to
    // the old ReplicaSet" failed-rollout scenario, which GetDeployment
    // alone won't show you.
    ListReplicaSets(ctx context.Context, namespace, labelSelector string) (*appsv1.ReplicaSetList, error)
}

func (c *client) GetDeployment(ctx context.Context, namespace, name string) (*appsv1.Deployment, error) {
    deploy, err := c.clientset.AppsV1().Deployments(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get deployment namespace=%q name=%q: %w", namespace, name, err)
    }
    return deploy, nil
}

func (c *client) ListReplicaSets(ctx context.Context, namespace, labelSelector string) (*appsv1.ReplicaSetList, error) {
    rs, err := c.clientset.AppsV1().ReplicaSets(namespace).List(ctx, metav1.ListOptions{
        LabelSelector: labelSelector,
    })
    if err != nil {
        return nil, fmt.Errorf("list replicasets namespace=%q selector=%q: %w", namespace, labelSelector, err)
    }
    return rs, nil
}

ListReplicaSets takes a label selector rather than a Deployment name because that's what the ReplicaSets API actually indexes on, a Deployment doesn't "contain" its ReplicaSets, it selects them by label, the same way a Service selects its pods. To go from "diagnose this Deployment" to "here are its ReplicaSets," a tool built on this interface pulls the selector off the Deployment's spec first (deploy.Spec.Selector), then passes that into ListReplicaSets. That two-step shape is exactly why this stays two methods instead of one convenience method that hides the relationship, the tool layer is where composing them into a single diagnose_rollout-style tool belongs, not the client.

Nodes, capacity and scheduling context

// internal/kubernetes/nodes.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// NodeClient covers cluster capacity, what you need to explain a
// "0/3 nodes are available" scheduling failure, which is a node-level
// question, not a pod-level one.
type NodeClient interface {
    ListNodes(ctx context.Context) (*corev1.NodeList, error)

    // GetNode returns conditions (MemoryPressure, DiskPressure,
    // PIDPressure), taints, and allocatable resources for one node.
    GetNode(ctx context.Context, name string) (*corev1.Node, error)
}

func (c *client) ListNodes(ctx context.Context) (*corev1.NodeList, error) {
    nodes, err := c.clientset.CoreV1().Nodes().List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list nodes: %w", err)
    }
    return nodes, nil
}

func (c *client) GetNode(ctx context.Context, name string) (*corev1.Node, error) {
    node, err := c.clientset.CoreV1().Nodes().Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get node name=%q: %w", name, err)
    }
    return node, nil
}

Network, is anything actually answering

// internal/kubernetes/network.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// NetworkClient covers Services and Endpoints. A Service existing
// doesn't mean anything is listening behind it, that's exactly the
// gap GetEndpoints is here to close.
type NetworkClient interface {
    GetService(ctx context.Context, namespace, name string) (*corev1.Service, error)

    // GetEndpoints checks whether a Service actually has healthy pods
    // backing it. Zero endpoints despite matching pods usually means a
    // label selector mismatch or a failing readiness probe, one of the
    // most common "works locally, broken in cluster" bugs.
    GetEndpoints(ctx context.Context, namespace, serviceName string) (*corev1.Endpoints, error)
}

func (c *client) GetService(ctx context.Context, namespace, name string) (*corev1.Service, error) {
    svc, err := c.clientset.CoreV1().Services(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get service namespace=%q name=%q: %w", namespace, name, err)
    }
    return svc, nil
}

func (c *client) GetEndpoints(ctx context.Context, namespace, serviceName string) (*corev1.Endpoints, error) {
    // The Endpoints object shares its name with the Service it backs,
    // Kubernetes creates and keeps it in sync automatically, so no
    // separate lookup by label is needed here.
    eps, err := c.clientset.CoreV1().Endpoints(namespace).Get(ctx, serviceName, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get endpoints namespace=%q service=%q: %w", namespace, serviceName, err)
    }
    return eps, nil
}

Ingress on EKS, is the ALB even pointing here

A healthy Service with healthy endpoints still means nothing to an external caller if the layer in front of it, the Ingress, and on EKS almost always the AWS Load Balancer Controller (LBC) behind it, never finished provisioning. This is a different failure class from anything NetworkClient covers: the Service can be perfectly correct and traffic still never arrives, because the ALB was never created, was created against the wrong subnets, or is pointing at the wrong target group.

// internal/kubernetes/ingress.go
package kubernetes

import (
    "context"
    "fmt"

    networkingv1 "k8s.io/api/networking/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// IngressClient covers classic Ingress resources, on EKS, almost
// always reconciled by the AWS Load Balancer Controller into an ALB.
// No AWS API calls happen here; everything comes from the Kubernetes
// object the controller writes status and events back onto, which is
// usually enough to tell you where reconciliation stalled.
type IngressClient interface {
    GetIngress(ctx context.Context, namespace, name string) (*networkingv1.Ingress, error)

    // ListIngressClasses lets a diagnosis confirm the Ingress's
    // ingressClassName actually exists and matches a real controller.
    // A typo'd or missing class is a common reason an Ingress just
    // sits there, the LBC never picks it up, and nothing in the
    // Ingress object itself says why.
    ListIngressClasses(ctx context.Context) (*networkingv1.IngressClassList, error)
}

func (c *client) GetIngress(ctx context.Context, namespace, name string) (*networkingv1.Ingress, error) {
    ing, err := c.clientset.NetworkingV1().Ingresses(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get ingress namespace=%q name=%q: %w", namespace, name, err)
    }
    return ing, nil
}

func (c *client) ListIngressClasses(ctx context.Context) (*networkingv1.IngressClassList, error) {
    classes, err := c.clientset.NetworkingV1().IngressClasses().List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list ingress classes: %w", err)
    }
    return classes, nil
}

Three things matter when reading an Ingress object for diagnosis, worth knowing even before the tool layer wraps this in part 2:

• ingress.Status.LoadBalancer.Ingress: empty means the LBC either hasn't reconciled yet or is stuck. A populated hostname (the ALB's DNS name) means AWS-side provisioning succeeded; an empty one after more than a minute or two almost always means the LBC is failing, not just slow.

• ingress.Spec.IngressClassName: cross-reference this against ListIngressClasses output. A nil or misspelled class is the single most common reason an Ingress is silently ignored, no error on the object itself, the LBC just never claims it.

• Events on the Ingress object: this is where EventClient.GetEvents, already built above, becomes directly useful here without any new code: the LBC writes events like SuccessfullyReconciled on success, and specific failure reasons (invalid target group, subnet tagging problems, certificate ARN not found) as Warning events when reconciliation fails. Composing GetIngress with GetEvents is exactly the same pattern describe_pod used for pods, applied one layer up the stack.

Gateway API, the newer, cluster-portable entry point

Gateway API is the follow-on to Ingress: instead of one annotation-heavy resource, traffic routing splits across a Gateway (the listener; where traffic enters, which addresses/ports it accepts) and route resources like HTTPRoute (how it's matched and where it goes). EKS supports it the same way it supports Ingress, the AWS Load Balancer Controller reconciles Gateway/HTTPRoute into an ALB, the same way it reconciles Ingress.

The client-side difference is real, not cosmetic: Gateway API types are CRDs, not part of the core Kubernetes API, so they need their own Clientset, the gatewayClientset already added to the client struct above.

// internal/kubernetes/gateway.go
package kubernetes

import (
    "context"
    "fmt"

    gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// GatewayClient covers Gateway API's two core resources. Both come
// back with rich status.conditions, Gateway API standardizes on
// Accepted/Programmed for Gateways and Accepted/ResolvedRefs for
// routes, which is a more structured diagnostic signal than Ingress
// ever gave you: no need to infer state from events alone.
type GatewayClient interface {
    GetGateway(ctx context.Context, namespace, name string) (*gatewayv1.Gateway, error)

    // GetHTTPRoute returns the route's status, including
    // per-parent-Gateway conditions. ResolvedRefs=False on a route
    // almost always means a backendRef points at a Service or port
    // that doesn't exist, check it before assuming the problem is
    // upstream at the Gateway.
    GetHTTPRoute(ctx context.Context, namespace, name string) (*gatewayv1.HTTPRoute, error)
}

func (c *client) GetGateway(ctx context.Context, namespace, name string) (*gatewayv1.Gateway, error) {
    gw, err := c.gatewayClientset.GatewayV1().Gateways(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get gateway namespace=%q name=%q: %w", namespace, name, err)
    }
    return gw, nil
}

func (c *client) GetHTTPRoute(ctx context.Context, namespace, name string) (*gatewayv1.HTTPRoute, error) {
    route, err := c.gatewayClientset.GatewayV1().HTTPRoutes(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get httproute namespace=%q name=%q: %w", namespace, name, err)
    }
    return route, nil
}

Diagnosis here is a two-level check, and skipping the first level is the most common mistake:

• gateway.Status.Conditions: check Accepted and Programmed first. If the Gateway itself isn't Programmed, no HTTPRoute attached to it can possibly work, no matter how correct that route is. This is the equivalent of checking describe_pod's container state before chasing application-level logs, start at the layer closest to the infrastructure.

• httproute.Status.Parents[].Conditions: a route can reference multiple Gateways, so status is reported per-parent, not once. ResolvedRefs: False specifically means a backendRef, the Service and port the route sends traffic to, doesn't resolve. That's diagnosable the same way check_endpoints diagnoses a plain Service: the route's backendRefs[].name and .port are exactly what GetService/GetEndpoints from earlier need to cross-check.

• A cluster without Gateway API CRDs installed: surfaces as a real error from GetGateway/GetHTTPRoute, a NotFound-shaped error on the CRD's group/version, not on the specific object. Worth catching and wrapping with a clearer message at the tool layer in part 2, the same way pod_metrics wraps a missing metrics-server as a clearer error than the bare API response gives you.

Config, keys only, never values

// internal/kubernetes/config.go
package kubernetes

import (
    "context"
    "fmt"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ConfigClient deliberately returns key names only, for both
// ConfigMaps and Secrets. An agent diagnosing a missing env var needs
// to know a key exists, never what it contains, see "Read-only by
// construction" below for why that boundary lives here and not in a
// tool description.
type ConfigClient interface {
    GetConfigMapKeys(ctx context.Context, namespace, name string) ([]string, error)
    GetSecretKeys(ctx context.Context, namespace, name string) ([]string, error)
}

func (c *client) GetConfigMapKeys(ctx context.Context, namespace, name string) ([]string, error) {
    cm, err := c.clientset.CoreV1().ConfigMaps(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get configmap namespace=%q name=%q: %w", namespace, name, err)
    }
    keys := make([]string, 0, len(cm.Data))
    for k := range cm.Data {
        keys = append(keys, k)
    }
    return keys, nil
}

func (c *client) GetSecretKeys(ctx context.Context, namespace, name string) ([]string, error) {
    secret, err := c.clientset.CoreV1().Secrets(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get secret namespace=%q name=%q: %w", namespace, name, err)
    }
    // secret.Data is map[string][]byte, the values are deliberately
    // never read here, only the key names.
    keys := make([]string, 0, len(secret.Data))
    for k := range secret.Data {
        keys = append(keys, k)
    }
    return keys, nil
}

Storage, volumes that can't bind

// internal/kubernetes/storage.go
package kubernetes

import (
    "context"
    "fmt"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// StorageClient surfaces the "pod stuck Pending because its volume
// can't bind" class of failure, a Pending PVC explains a Pending pod
// that otherwise looks like a scheduling mystery.
type StorageClient interface {
    ListPVCs(ctx context.Context, namespace string) (*corev1.PersistentVolumeClaimList, error)
}

func (c *client) ListPVCs(ctx context.Context, namespace string) (*corev1.PersistentVolumeClaimList, error) {
    pvcs, err := c.clientset.CoreV1().PersistentVolumeClaims(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("list pvcs namespace=%q: %w", namespace, err)
    }
    return pvcs, nil
}

Metrics, the same extension the client-go post already teed up

k8s.io/metrics was already installed above alongside the three core packages, since NewClient needed metricsclientset before this section even started. This is where it actually gets used:

// internal/kubernetes/metrics.go
package kubernetes

import (
    "context"
    "fmt"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    metricsv1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1"
)

// MetricsClient reads from the metrics-server extension API, a
// separate Clientset from the core one, same as it was flagged in the
// client-go post's "what's next" for the ferctl top CLI.
type MetricsClient interface {
    GetPodMetrics(ctx context.Context, namespace, name string) (*metricsv1beta1.PodMetrics, error)
    GetNodeMetrics(ctx context.Context, name string) (*metricsv1beta1.NodeMetrics, error)
}

func (c *client) GetPodMetrics(ctx context.Context, namespace, name string) (*metricsv1beta1.PodMetrics, error) {
    m, err := c.metricsClientset.MetricsV1beta1().PodMetricses(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get pod metrics namespace=%q name=%q: %w", namespace, name, err)
    }
    return m, nil
}

func (c *client) GetNodeMetrics(ctx context.Context, name string) (*metricsv1beta1.NodeMetrics, error) {
    m, err := c.metricsClientset.MetricsV1beta1().NodeMetricses().Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("get node metrics name=%q: %w", name, err)
    }
    return m, nil
}

GetPodMetrics and GetNodeMetrics reach through a second Clientset, c.metricsClientset, built from k8s.io/metrics alongside the core one in NewClient, the same two-Clientset shape the client-go post flagged as coming in a future post, now realized here instead.

GetConfigMapKeys and GetSecretKeys are worth pausing on. It would be less code to return the whole object, map[string]string and all. Returning only the keys is a security decision made at the interface boundary, before a single tool or prompt exists, an agent that can only ever see "this Secret has a key called DATABASE_URL" cannot leak what that key contains, no matter how it's prompted. That's a much stronger guarantee than "the tool description tells the agent not to print secret values."

Read-only by construction

Summary

Every method on KubeClient is List, Get, or Stream, nothing that mutates the cluster. That boundary, along with the read-only Secret/ConfigMap key-only design and the bounded LogOptions, all live in the Go types themselves rather than in documentation someone has to remember to follow. Three things worth taking away:

• Compose small, domain-specific interfaces (PodClient, EventClient, NetworkClient, ...) instead of one large KubeClient with twenty methods on it, easier to test, easier to fake, and each file has one reason to change.

• Design the client before you design anything that calls it. Whether the caller ends up being a CLI, a report generator, or an AI agent, the interface should already answer "what's actually useful to ask a cluster" independent of who's asking.

• Every safety boundary that matters, read-only access, bounded queries, secret values never leaving the cluster, belongs in the interface's types and method signatures, not in a comment asking future callers to be careful.

Part 2 picks up exactly here: wrapping this KubeClient as MCP tools, wiring it to an agent, and running the whole thing end-to-end against a real cluster. Building a Kubernetes-aware AI agent with a Go MCP server →

---

Let's connect!

One of the best parts of writing in public is the people you meet along the way, engineers at different stages of their journey, working on similar problems from completely different angles.

If something in this post resonated, if you spotted a bug, or if you just want to talk Go, Kubernetes, Platform Engineering, DevOps, or whatever, I'm always happy to hear from you.

• LinkedIn

• GitHub

• Twitter / X

Building from Asunción, Paraguay 🇵🇾