DEV Community: Jesica Kearney

I operate Karpenter every day. I wanted to see how its brain works, so I built a controller.

Jesica Kearney — Fri, 02 Oct 2026 11:38:41 +0000

I've been running EKS with Karpenter in production for a while. I can debug a node that won't provision, I know what a NodeClaim is, and I've written the Terraform that installs the whole thing.

What I'd never done is look at the machinery. There's a gap between "pod is unschedulable" and "EC2 instance appears" where something is watching, deciding and acting, and I only knew it from the outside.

This was pure curiosity. I wanted to see how the brain works.

So I spent a couple of evenings building a controller with client-go — the official Go library every Kubernetes component is built on, kubectl and Karpenter included. No controller-runtime, no Kubebuilder, nowhere to hide. About 250 lines, running against a local kind cluster.

It works. But the three things that went wrong taught me more than the working version did, and none of them are in the tutorials.

What I built

A controller that watches nodes for a disruption taint and reports which pods on them won't come back if the node dies.

Not all pods are equal when a node goes away:

Owned by a ReplicaSet, StatefulSet or Job → rescheduled elsewhere, fine
Owned by a DaemonSet → goes with the node, expected
No controller owner → gone, permanently

That last case is what kubectl drain refuses to evict without --force. I wanted to surface it before the drain, which is when it's actually useful.

The shape is the same as every controller:

   API server
       │  WATCH (opened once, held open)
       ▼
  Node informer ────► local cache
       │
       │ AddFunc / UpdateFunc / DeleteFunc
       ▼
   enqueue(key)         "clientgo-control-plane"
       │
       ▼
  ┌─────────────┐
  │  workqueue  │  dedupes · rate limits · tracks in-flight
  └─────────────┘
       │
       ▼
  worker goroutine
       │
       ▼
  reconcile(key)

Everything left of reconcile is plumbing. All the decisions live in about 40 lines inside it.

Right — the bugs.

1. Four pods said their owner was the node

First working version. I tainted the node and got every pod on it with its owner. Most were what you'd expect — ReplicaSets, DaemonSets. Then these:

kube-system/etcd-clientgo-control-plane — owned by Node clientgo-control-plane
kube-system/kube-apiserver-clientgo-control-plane — owned by Node clientgo-control-plane
kube-system/kube-scheduler-clientgo-control-plane — owned by Node clientgo-control-plane
kube-system/kube-controller-manager-clientgo-control-plane — owned by Node clientgo-control-plane

A Node isn't a workload controller. Nothing creates pods from a Node. I'd expected either a real owner or none at all.

They're static pods. The kubelet reads their manifests from /etc/kubernetes/manifests/ on local disk and runs them directly, with the API server not involved at all — which it has to, because you can't create the API server through the API server.

The kubelet then creates a read-only mirror pod in the API so the thing appears in kubectl get pods, and sets the Node as its owner. The mirror pod is a reflection. Delete it and the kubelet puts it straight back.

So my controller was about to judge whether something would be recreated, using an ownership model that doesn't describe how these pods live at all. They'd survive a kubelet restart. They wouldn't survive the node going away. Neither fact is visible in ownerReferences.

I filtered them on the annotation the kubelet sets on every mirror pod:

if _, isMirror := pod.Annotations["kubernetes.io/config.mirror"]; isMirror {
    continue
}

What I actually learned: there are two lifecycles running on every node, and the API only shows you one. I'd read about static pods before. I'd never had a reason to care about the difference until my own code tried to reason about them with the wrong model.

2. My controller could make a claim but not retract it

Second version annotated at-risk pods so the finding outlived my terminal:

node-drain-controller/at-risk: clientgo-control-plane

Worked nicely. Then I removed the taint, and the annotation stayed. Forever.

The pod was now carrying a note saying it was at risk on a node that was completely healthy. Nothing was ever going to remove it.

Here's the line that caused it:

if taint == nil {
    return nil        // "no taint, nothing to do"
}

I'd treated "this node has no taint" as nothing to do. It isn't. It's information: this node is healthy. And if it's healthy, anything on it claiming otherwise is wrong and needs correcting.

if taint == nil {
    return c.clearAnnotations(name)
}

What I actually learned: I'd written half a reconcile loop. A reconcile function isn't "do the thing when the trigger fires" — that's an event handler. It's make reality match what it should be, whatever reality currently is. And that has to work in both directions, or the state you write drifts away from the truth and nobody can tell which annotations still mean anything.

There's a subtlety in the cleanup too. It keys off the annotation I left behind, not the conditions I used when leaving it:

if pod.Annotations[riskAnnotation] == "" {
    continue        // I never annotated this one
}

If I later change which pods get annotated, cleanup still finds everything I've ever written. Re-deriving the original condition would orphan state I could no longer locate — which is how controllers leave debris nobody can trace back to them.

3. Making it configurable broke state written under the old config

Last addition: a CRD so the taint key wasn't hardcoded.

apiVersion: jk.io/v1alpha1
kind: DisruptionPolicy
metadata:
  name: default
spec:
  taintKey: spot-interruption

Changed it to maintenance with kubectl patch, restarted, and got:

loaded policy: taintKey=maintenance annotate=true
caches synced — starting workers
cleared annotation on default/orphan — node no longer tainted

The node still had spot-interruption on it. The pod was still annotated under that key. But the controller was now looking for maintenance, found none, concluded the node was fine, and cleaned up.

Which is correct, and I only got away with it because the bidirectional cleanup from bug 2 existed and the controller restarted and re-evaluated everything. If I'd implemented a live policy watch instead of reading at startup, the same change would have left stale annotations with no path back.

What I actually learned: changing a controller's configuration invalidates state it wrote under the old configuration, and that's a design question, not an implementation detail. "Make the taint key configurable" sounds like a one-line change. It isn't. It means deciding what happens to everything you've already written.

I left it reading at startup and wrote the limitation down, which I think is the right call — a half-built live watch would have been worse than an honest restart requirement.

The one I didn't hit

Worth mentioning because it's in a lot of tutorials.

In the worker loop, this placement is wrong:

func (c *Controller) runWorker() {
    for {
        key, _ := c.queue.Get()
        defer c.queue.Done(key)   // ← never fires
        c.reconcile(key)
    }
}

defer is function-scoped, not block-scoped. runWorker loops forever, so the defer doesn't run until shutdown. Every key stays marked in-flight permanently, the queue stops redelivering them, and the controller silently stops reconciling. No error. No log.

Splitting the loop so each item gets its own function is what makes it fire:

func (c *Controller) runWorker() {
    for c.processNextItem() {
    }
}

func (c *Controller) processNextItem() bool {
    key, shutdown := c.queue.Get()
    if shutdown {
        return false
    }
    defer c.queue.Done(key)       // ← fires per item
    ...
}

I avoided this one because a commenter on the tutorial I was following had already caught it. I'd have written the broken version otherwise, and I don't think I'd have found it quickly — a controller that processes each object exactly once and then goes quiet looks a lot like a controller with nothing to do.

What changed about how I debug Karpenter

This is the part I actually wanted.

Memory sizing stopped being arbitrary. Our Karpenter controller requests 2Gi. That's the informer caches — every Pod, Node, NodeClaim and NodePool held in RAM. When I size a controller now I'm sizing its caches, not guessing.

The restart delay makes sense. After a Karpenter restart there's a window where nothing provisions, plus a spike of API server load. That's every informer doing its initial LIST, and WaitForCacheSync holding the workers back until it's done — because a controller acting on a half-populated cache is worse than one that's briefly idle.

The "why is it doing nothing" question has a better answer. Controllers are level-triggered: they react to what's true now, not to which event fired. A controller sitting silent is usually correct, not stuck. Mine runs reconcile several times a minute and almost always decides there's nothing to do.

Our Terraform's ugliest code has a reason. We have a destroy-time provisioner with a bash loop — sleep 10, a counter, give up after 30 — waiting for NodeClaims to drain, and eventually force-stripping finalizers. I'd always read that as a hack. It is, but it's a necessary one: Karpenter puts a finalizer on every Node so it can drain pods and terminate the EC2 instance before the object disappears, and Terraform has no way to wait on a watch. The workqueue does that properly — exponential backoff, per-key, non-blocking. The bash loop is the same idea with none of the machinery.

Would I use raw client-go again?

No. For anything real I'd use controller-runtime — SetupWithManager does five things in one line that took me two evenings, and it does them better.

But I can now say what those five things are, and that's worth more to me than the two evenings. When something in Karpenter behaves oddly I'm no longer guessing at a black box; I'm reasoning about an informer, a queue and a reconcile loop I've written myself.

If you operate Kubernetes and haven't written a controller, I'd recommend it on exactly those grounds. Not because you'll ship it. Because the next time you debug one, you'll know what's inside.

I went in expecting something that remembers and reacts. What's actually there is something that forgets everything, looks at the world fresh, and fixes the difference. That's the whole trick.

Code is on GitHub. The README goes into the design decisions and the limitations in more detail.

What actually happens when a TLS certificate expires (and how I built a Bash guardrail for it)

Jesica Kearney — Sat, 12 Sep 2026 17:51:20 +0000

Every few months, somewhere, a website goes down for the most avoidable reason in infrastructure: a TLS certificate expired and nobody was watching. Browsers throw a full-screen red warning, API clients refuse to connect, and an on-call engineer gets paged for something a calendar reminder could have prevented.

I've dealt with enough certificate issues in production to want a lightweight guardrail I fully control — so I built one in Bash, rather than reaching for a managed service that hides the mechanics. It's a small tool, but it's a good excuse to walk through what a certificate actually is and why expiry is only half the problem.

A certificate is a public identity document

A website's certificate is public by design. When a client connects over HTTPS, the server presents its certificate to anyone who asks — before any encryption begins — because the certificate is how the server proves it is who it claims to be.

You can see this yourself. This one command opens a TLS connection and reads the certificate's expiry date:

echo | openssl s_client -servername github.com -connect github.com:443 2>/dev/null \
  | openssl x509 -noout -enddate

No login, no API key. The certificate carries only public metadata: the domain it was issued for, the issuing authority, the public key, and the validity window. The private key never leaves the server. That split is the whole basis of TLS — anyone can verify the server's identity, but only the server can prove it.

The handshake, step by step

Checking a certificate is just performing the opening moves of a normal HTTPS connection:

TCP connect to the host on port 443, exactly as a browser would.
ClientHello + SNI. The client signals it wants to speak TLS, and the -servername flag sets SNI (Server Name Indication) — because many sites share one IP address, SNI tells the server which site's certificate to present. Omit it and you can be handed the wrong certificate entirely.
The server presents its certificate in the clear.
Parse the notAfter date and compute days remaining.

A browser would continue from here into an encrypted session; the check just reads the date and disconnects.

Expiry is only one of the checks

Expiry is the obvious failure mode, but it's only one of several independent checks a browser runs. A certificate can be perfectly in-date and still be rejected — because it was issued for the wrong hostname.

The clearest example is a wildcard-scope mismatch. A *.badssl.com certificate is valid, but wildcards match only one level, so it does not cover wrong.host.badssl.com. Other common causes in the wild:

Shared IP / wrong default cert — a missing SNI default hands you another site's valid certificate.
Reused certificate — a cert deployed onto the wrong host by mistake.
CDN / load-balancer gaps — a new subdomain pointed at infrastructure whose certificate doesn't include it yet. In every case the browser hard-blocks the connection, because a valid certificate belonging to someone else is exactly what a man-in-the-middle attack looks like. Expiry checks miss all of these, so hostname matching has to be a separate check:

bash
openssl s_client -servername "$domain" -connect "$domain":443 \
  -verify_hostname "$domain"

This returns Verify return code: 0 (ok) on a match, or 62 (hostname mismatch) when the certificate is real but wrong for that host.

Turning a script into infrastructure

A script that prints text is a toy. What makes it usable by real systems is exit codes. This one follows the Nagios convention:

Code    Meaning
0   All certificates OK
1   Warning window, or a hostname mismatch
2   Critical, expired, or unreachable

That single design choice is what lets a scheduler act on the result without parsing any text. Wired to a GitHub Actions cron, it needs no extra tooling: because the script exits non-zero on trouble, a failing certificate turns the workflow run red — and that red status is the alert.

Why build what you could buy

Some companies offer certificate monitoring as a checkbox, so building it by hand isn't about saving money. The value is owning every layer under that checkbox — the handshake, SNI, the public/private key split, why hostname verification is separate from expiry, and how exit codes turn a script into something a monitoring system can consume. When the managed version misbehaves, that's the difference between debugging it and filing a support ticket.

The full tool — including a companion process/inode inspector — is on GitHub:[https://github.com/JayKearney/linux-toolbox]. Feedback and questions welcome.