DEV Community: Metronom

Every MCP revision, and what each one took back

Metronom — Thu, 20 Aug 2026 08:55:34 +0000

MCP is twenty months old and has had five specification revisions. I put them all in one table because the shape of the list says more than any single entry does.

Read the removals, not the additions. Batching arrived in March 2025 and was gone by June, before it was three months old, which tells you it was never load-bearing: nothing had been built on it, so the specification was free to admit the mistake. Sessions and the opening handshake went on 28 July 2026, and that is a different kind of news entirely. A capability deleted in month seven is a correction. A rework of the foundations in month twenty means the foundations were wrong.

Revision	Added	Taken back
2024-11-05	JSON-RPC; stdio and HTTP+SSE; tools, resources, prompts	—
2025-03-26	OAuth 2.1; Streamable HTTP; batching	HTTP+SSE
2025-06-18	Structured output	Batching, under three months old
2025-11-25	Icons; schema defaults; JSON Schema 2020-12	Nothing the revision list records
2026-07-28	Capability-listing call; "input required"; response cache hints; routing headers	Sessions; the handshake; stream resumption. Deprecated with a year of grace: server-to-model calls, folder access, protocol-level logging, dynamic client registration

The 2026-07-28 row is the one worth arguing about. Removing sessions made MCP stateless, which is Fielding's second REST constraint, adopted twenty months late by a protocol that had started from the opposite premise. Everything else in that revision follows from it: response cache hints, and headers carrying enough routing information that a proxy can direct a request without parsing the body. These are the features you add when your protocol has to survive contact with ordinary infrastructure.

My reading, and you are welcome to disagree with it: the reversal is late and right. Sessions in a protocol living between two processes owned by different people were a mistake from the beginning. The two ends are deployed, restarted, scaled and upgraded on separate schedules, and a shared memory between them is a liability one side always ends up holding.

Nothing broke on 28 July. The revision came out as a Release Candidate, the SDKs are in beta, and everything deprecated has a year of grace. The practical consequence is not "wait", it is "budget for rewriting".

Source: modelcontextprotocol.io, revisions 2024-11-05, 2025-03-26, 2025-06-18, 2025-11-25 and 2026-07-28, plus the official changelog for the last of them. I keep this table current as revisions land, with per-revision anchors so you can link to a row rather than to a page: https://dorokhovich.com/blog/mcp-revision-history

One real question: is there a row here you would have filled in differently? The 2025-11-25 line in particular says "nothing the revision list records", and that is a claim about a changelog rather than a line-by-line diff. If you have diffed it, tell me what I missed.

Hacker News + dev.to (a concrete '2-5 minutes to a few seconds' DX win is a proven front-page hook)

Metronom — Wed, 19 Aug 2026 08:59:47 +0000

We got the Kubernetes inner loop from 2–5 minutes back to seconds with Tilt

Chapter 8 of a local-Kubernetes series — the payoff that closes the loop chapter 1 opened. Tilt watches files, rebuilds, deploys, and with Live Update syncs code into a running container in seconds.

Key takeaways

The pain, quantified: by hand, every edit is docker build → push → kubectl rollout restart → wait → check logs — a minute or two each. Ten edits = half an hour gone.
A Tiltfile is a program, not YAML — it's Starlark (a Python dialect). tilt up executes it top to bottom; edit the Tiltfile and Tilt re-executes it, nothing to restart.
The three pillars: docker_build('tag','ctx') (how to build), k8s_yaml('file') (what to deploy), k8s_resource('name', port_forwards=...) (fine-tuning). Tilt matches built images to manifest images by tag and swaps in the fresh build.
Live Update is the star: instead of rebuild+redeploy, Tilt copies changed files straight into the running container. Steps run in a strict order — fall_back_on → sync → run → restart_container — and the order can't be broken.
Two newbie traps: sync paths must live inside the same docker_build context ("if Tilt is watching it, you can sync it"); run() can't come before sync(). Also: the first deploy is always full — Live Update needs a running container.
Hot reload, two scenarios: (A) framework reloads itself — uvicorn --reload --reload-dir app, sync is enough; (B) no reload — use the restart_process extension (docker_build_with_restart), the modern replacement for the deprecated restart_container() on k8s.
Dashboard at localhost:10350: per-resource update + runtime status, logs with filtering, endpoints as links, manual Trigger Update, optional TRIGGER_MODE_MANUAL.
vs Skaffold/DevSpace: Tilt = UI-first + Starlark + Live Update; Skaffold/DevSpace = CLI-first + YAML. Fair caveat: pick by team taste.

Full article: https://dorokhovich.com/blog/local-k8s-tilt-fast-dev-loop?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-tilt-fast-dev-loop

Local Kubernetes Dev — Part 8: Tilt — a fast development loop

Metronom — Mon, 17 Aug 2026 09:41:15 +0000

Remember the loop from chapter one that stretched from seconds to 2–5 minutes on every change? This is the chapter where we bring it back down to seconds.

Part eight of the series is the emotional climax. Tilt orchestrates your inner loop: it watches files, builds the image for you, deploys to the cluster for you, and — crucially — can update code right inside the running container in seconds. Plus a web dashboard where all your services, statuses, and logs live on one screen, no juggling ten terminals.

The three pillars of any Tiltfile: docker_build (how to build), k8s_yaml (what to deploy), k8s_resource (fine-tuning — port forwarding). The magic is that Tilt matches images from docker_build to images in the manifests by tag and substitutes the fresh build. The star of the show is Live Update: instead of "rebuild image → redeploy pod," Tilt copies the changed files straight into the live container. With FastAPI it pairs beautifully with uvicorn --reload.

I break down the strict order of live_update steps (fall_back_on → sync → run → restart), two beginner traps (sync only within the build context; run can't come before sync), the no-hot-reload path via restart_process, and an honest comparison of Tilt vs Skaffold vs DevSpace. https://dorokhovich.com/blog/local-k8s-tilt-fast-dev-loop?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-tilt-fast-dev-loop

What GitHub's MCP Server Costs You Before You Ask It Anything

Metronom — Thu, 13 Aug 2026 14:02:21 +0000

TL;DR
GitHub's official MCP server spends roughly 17,600 tokens on tool descriptions before you ask it anything, with every toolset switched on.
The same operation through MCP costs 4 to 32 times what the equivalent command-line call costs.
If you can write the list of calls in advance, write it, and keep REST underneath.

Attach GitHub's official MCP server to an assistant, every toolset switched on, and the conversation opens with a bill of roughly 17,600 tokens. Nobody has asked anything yet. That is the cost of the tool descriptions alone — the names, the sentence explaining what each tool does, the shape of the arguments it expects — and it is paid again on every request for as long as the server stays attached. The figure is a count of the server's own tool schemas, published by people who counted them; it is not a vendor's number.

Written 5 August 2026. Every figure below was measured between late 2025 and July 2026, before the revision of 28 July changed some of the rules they were measured under. The prices move; the arithmetic does not. Measure on your own workload before you plan around them.

I want to be careful with that number, because it belongs to a configuration rather than to a product. The server groups its tools into toolsets you can switch off, and switching the unused ones off cuts the figure several times over. Which is exactly the point: nobody can honestly quote it without saying which toolsets were on. In the measurement above, all of them were.

The "MCP versus REST" argument is almost always conducted as though it were a matter of taste, settled by whichever side sounds more modern. It is not a matter of taste. There is a price list, the numbers on it are fairly harsh, and once you have seen them the choice stops being about architecture and starts being about whether the thing you are building actually needs a model to decide anything.

Where the numbers come from, before anyone asks in the comments: public analyses of the GitHub MCP Server and survey write-ups of context overhead from late 2025 and early 2026, plus the schema-overhead discussion in issue #2808 of the specification repository. Specification revisions used: 2025-06-18 and 2026-07-28. The truncation incident later in this piece is from my own publishing server, audited on 25 July 2026 — that part is not a citation, it is my own damage.

FULL ARTICLE TEXT — syndicate from https://dorokhovich.com/blog/mcp-token-cost

Everything from the section "Start with the bill" to the end of "How to cut the MCP token bill" goes across unchanged. Forem renders GFM, so nothing needs converting. The truncation incident is now its own section, "The cost that appears on no invoice", second in the article rather than buried three headings down — do not move it back.

CLOSING (replaces the blog's series-navigation block)

The three articles this comes from are on my own site: where MCP and REST came from, this one on what MCP costs in tokens, and what an MCP server is actually made of.

One real question, and I want the disagreement rather than the agreement: have you measured what your own tool catalogue costs per request? Not the vendor's number for their server — yours, counted on your own descriptions. I have not published mine, and that is a fair thing to hold against this article: I run a server with twenty tools and I have quoted somebody else's figure rather than my own. If you have counted yours, I would like the number and the method, because the method is the part nobody publishes.

ADAPTATION NOTES

Keep the GFM tables. Forem renders them; this is the platform where the comparison table earns its place.
The dated callout does not go at the top here. Forem builds the feed and share card from the first ~140 characters, so a callout in that position puts a caveat where the hook belongs. It is already placed above, immediately after the opening paragraph, as an ordinary blockquote; dev.to renders it fine there.
Images: the three PNGs need re-uploading to Forem or referencing by absolute URL; relative /images/blog/... paths will 404 on dev.to. Check every one after publishing rather than assuming.
After publishing, spend the following 48 hours in the comments under my own name. Since mutual claps and cross-profile upvoting are off the table as manipulation, replying in comments is the only distribution channel this campaign has that costs nothing and is honest.

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

Metronom — Wed, 12 Aug 2026 12:32:08 +0000

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

Most Kubernetes ImagePullBackOff / CrashLoopBackOff troubleshooting is not deep. The statuses read like spaceship error codes, but nearly all of them fall out of three commands and a short list of causes. This is the playbook I run on autopilot, plus the one k3d trap that cost me a morning.

The golden set

Whatever the status, start here. Same describe / logs / exec loop the Kubernetes 'Debug Running Pods' guide recommends.

# 1. State, Reason, and the Events block at the bottom — ALWAYS first
kubectl describe pod <pod> -n myapp

# 2. App logs, and the CRASHED container's logs via --previous
kubectl logs <pod> -n myapp
kubectl logs <pod> -n myapp --previous

# 3. Namespace timeline
kubectl get events -n myapp --sort-by=.lastTimestamp

describe hands you three fields: State, Reason, and Events. That's the loop. The rest is pattern-matching. There's a solid field guide to these same statuses at this troubleshooting writeup.

ImagePullBackOff — and the k3d containerd trap

Symptom

ErrImagePull first, then ImagePullBackOff after backoff. Pod stuck Waiting. Events line usually says it outright: not found / unauthorized / no such host.

Root cause

The boring causes are enumerated in the Kubernetes images docs: typo in name or tag, private registry with no imagePullSecrets, Docker Hub rate limit (toomanyrequests), bad registry address.

The trap that isn't boring: k3d nodes run on containerd, isolated from your Docker daemon. The image you just built with docker build — visible right there in docker images — the cluster cannot see. Docker has it; containerd doesn't. ImagePullBackOff, even though "the image is obviously built."

The fix

Two correct ways to get the image into the cluster, both in the k3d image-import docs:

# Option 1: import into the nodes directly
docker build -t myapp:dev .
k3d image import myapp:dev -c dev

# Option 2 (recommended): a local registry created with the cluster
k3d cluster create dev --registry-create k3d-registry.localhost:5000

# then the FULL name matters: address + port + tag
image: k3d-registry.localhost:5000/myapp:dev

The classic follow-up mistake is an incomplete image name — myapp:dev instead of k3d-registry.localhost:5000/myapp:dev. The cluster looks in the wrong place and you're back in backoff. If you use Tilt, docker_build handles delivery, but the Tiltfile image name must match the manifest exactly or you get the same error.

CrashLoopBackOff — the status that hides its own cause

Symptom

Container starts, crashes, kubelet restarts it, it crashes again — with exponential backoff, roughly 10s → 20s → 40s, capped at 5 minutes. The status itself tells you nothing.

The fix

The command I wish I'd known on day one:

kubectl logs <pod> -n myapp --previous

Without --previous you get the freshly-started container, which usually hasn't written anything. --previous pulls the stack trace from the instance that actually died. Also read Last State: Terminated and its Exit Code in describe.

Causes I check in order:

Startup bug — exception, non-zero exit. For a FastAPI service, missing DB_HOST / DB_PASSWORD crashes it on connect.
Missing env/config — a ConfigMap/Secret isn't mounted.
Dependency not up — Postgres hasn't started and the app doesn't wait.
OOM — out of memory (below).
Liveness too strict — kills the app before it warms up. Fix with initialDelaySeconds or a startup probe.
Exit code 0 — for a long-running service, "exited successfully" is still a crash loop. Usually a wrong entrypoint that runs and exits instead of starting uvicorn.

When the container dies instantly and logs are empty, keep it alive and climb inside:

# temporary, in the Deployment
command: ["sleep", "infinity"]

kubectl exec -it <pod> -n myapp -- sh
env | grep DB_
uvicorn app.main:app --host 0.0.0.0 --port 8080   # run by hand, read the real error

Pending — the scheduler couldn't place it

Symptom

Pod never leaves Pending. describe → Events → FailedScheduling, usually naming the cause: Insufficient cpu / Insufficient memory.

Root cause

On a small local cluster this is the most common one. I once set requests: 4 CPUs on a 2-CPU k3d VM and the pod simply never scheduled. Other causes: nodeSelector/affinity mismatch, taints without tolerations, an unbound PVC, a taken hostPort.

The fix

Lower requests, fix the selector/PVC, or grow the cluster:

k3d node create extra --cluster dev --role agent

For a local FastAPI service, 100m CPU / 128Mi memory is fine. Don't paste production numbers into a laptop cluster.

OOMKilled — the exit-137 signature

Symptom

Process blew past its memory limit and the kernel SIGKILL'd it. Recognize it by exit code 137 (128 + 9, SIGKILL) and Reason: OOMKilled in describe. That Reason field distinguishes a memory kill from other SIGKILLs, and OOM is often the hidden cause behind a CrashLoopBackOff.

Root cause

Two flavors:

Container-level — exceeded resources.limits.memory. Raise the limit or fix a leak.
Node-level (sneaky on k3d) — the whole Docker VM ran out of memory, so pods get OOM-killed even while each app is within its own limit. The fix isn't Kubernetes limits; it's giving the Docker VM more memory, or not overcommitting — the sum of all pods' limits shouldn't exceed the VM's memory.

The Service that's green but silent

Symptom

Pods Running, everything looks fine, requests never reach the app. Almost always one of three breaks in Service → endpoints → Pod.

The fix

Check endpoints first:

kubectl get endpointslices -n myapp -l kubernetes.io/service-name=myapp

Empty means nothing is behind the Service. Three causes:

Selector mismatch — labels are case-sensitive, app: Web ≠ app: web.
Port mismatch — targetPort must equal the real containerPort.
Readiness not passing — pod shows 0/1 Ready, Kubernetes pulls it from endpoints so no traffic flows, but the container doesn't restart and still reads Running.

Localize by hitting the pod directly, bypassing the Service:

kubectl port-forward <pod> -n myapp 8080:8080
curl http://localhost:8080/healthz   # works? then it's selector/endpoints/readiness, not the app

Before / after

Status	Old reflex	The move
ImagePullBackOff	re-`docker build`, panic	check k3d containerd/registry gotcha; full image name
CrashLoopBackOff	stare at empty logs	`logs --previous` + exit code
Pending	assume cluster broken	read `FailedScheduling`; shrink requests
OOMKilled	confused restarts	spot exit 137 + `Reason: OOMKilled`
Service silent	blame the app	check endpoints/selector/readiness

Guardrail

The whole thing is one algorithm: get pods shows what → describe explains why in Reason/Events → logs --previous gives the crash detail. Ninety percent of cases are on the list above.
Watch disk creep. Kubelet image GC is lazy — it won't clean until the node's disk crosses its high-threshold, 85% by default — so images pile up on purpose. docker system df to see it, prune carefully; docker system prune -a can delete images the cluster still needs.
Keep the golden set in muscle memory. It's faster than any search bar.

What I'd do differently

Stop treating a red status as a mystery. Every one of them resolves to a Reason field and a short cause list. The dread was never warranted — the statuses just have unfriendly names.

Bottom line: memorize describe + logs --previous + get events, learn the five common statuses, and remember that in k3d "the image is built" and "the cluster can see it" are two different facts.

Sources

docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

Metronom — Tue, 11 Aug 2026 19:26:02 +0000

docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

The service passed every local test in docker-compose. The first Kubernetes rollout fell over inside an hour on things Compose had never modeled: readiness, Secrets, startup order, service discovery. This is the docker-compose vs kubernetes dev/prod parity gap, and it cost us an outage before I admitted it existed.

Symptom

First prod rollout. Pod Running, but the Service sent it zero traffic.

$ kubectl get pod -l app=api
NAME                   READY   STATUS    RESTARTS   AGE
api-6c9f8b7d4-2xk9p    0/1     Running   3          2m

$ kubectl describe pod api-6c9f8b7d4-2xk9p | tail
  Warning  Unhealthy  readiness probe failed: connection refused
  ...
$ kubectl logs api-6c9f8b7d4-2xk9p --previous
  sqlalchemy.exc.OperationalError: could not connect to server: Connection refused
  Is the server running on host "postgres" (10.43.x.x) and accepting connections?

Three separate failures, none of which any green Compose test had touched:

No readiness probe existed, so I'd never configured one — Compose has no such concept.
A plaintext env var in Compose was supposed to be a Secret in k8s.
The app crashed reaching the DB before the DB was up. Compose's depends_on had papered over that for years.

Root cause

I'd assumed a Compose file and a set of k8s manifests were the same idea at different fidelity. They aren't. The Twelve-Factor App's tenth factor, dev/prod parity, names the trap: keep the gaps between dev and prod small — especially the tools gap, where "tiny incompatibilities crop up, causing code that worked and passed tests in development to fail in production."

Two different problems live under that one heading, and I'd conflated them:

Backing-service parity. The canonical sin is SQLite locally, Postgres in prod. Sooner or later a behavioral difference bites at the worst moment. Same type, same version, both sides. No substitutions.
Manifest-level bugs. A whole class of failure lives only at the Kubernetes object layer: misconfigured probes, thin RBAC, service discovery by in-cluster DNS, missing resource limits. Compose has none of these concepts, so it can surface none of these bugs.

That second bucket is the one that got me. A single Compose service is not one Kubernetes object — it's several, each carrying its own slice of production behavior:

a Deployment — how the pods run
a Service — how they're reached
a ConfigMap + Secret — config and secrets
an Ingress — external access

Every one of those is a place to misconfigure something, and a place Compose never made me think about. There's a solid breakdown of what "production-like local" actually means at this writeup on production-like local environments.

The fix that didn't work: Kompose

First instinct was to cheat and auto-convert. Kompose is a real project under the Kubernetes org, so I ran it:

# a starting point, NOT a prod manifest
kompose convert -f compose.yaml

Kompose's own authors are honest: "our conversions are not always 1-1... but will get you 99% of the way there." Production lives in that last 1%. What the conversion silently drops or mangles:

depends_on is ignored. Kubernetes has no "start B after A." You use init containers or, better, app-level retries — reconnect to Postgres until it answers.
build: doesn't build. Kubernetes can't build from source. The image must already be pushed to a registry.
network_mode: host and custom networks map poorly or not at all.
Bind mounts vanish. You're expected to use a ConfigMap/Secret.

The kicker: default Kompose output ships without requests/limits, without probes, with env vars in plaintext instead of Secrets. The automation strips out exactly the production-like properties you were trying to gain. A draft, not a deploy.

The fix that worked: a production-like k3d cluster

I switched to running the service in a real local cluster with k3d (k3s in Docker). The discipline that mattered was deciding, on purpose, what to reproduce and what to simplify.

Reproduce, no exceptions:

# pin the SAME minor as prod. Docker tags can't use '+',
# so it's 'v1.31.5-k3s1', not 'v1.30.2+k3s1'
k3d cluster create dev --image rancher/k3s:v1.31.5-k3s1

Same backing services by type and version — Postgres, never SQLite.
Real requests/limits, so scheduler and eviction behavior is realistic.
The exact probe pair that would've saved the first rollout — liveness and readiness: /healthz liveness (process is up), /ready readiness (checks the Postgres connection; no traffic while the DB is unreachable, but no restart either).
ConfigMap + Secret instead of hardcoding; Ingress + in-cluster DNS so routing matches prod.

Same real manifests, deployed the same declarative way as prod:

kubectl apply -k k8s/overlays/dev
kubectl rollout status deploy/api

Deliberately simplify, and write down why: prod scale and topology, managed cloud services (Postgres in a container instead of RDS), and load testing (doesn't belong on the box running the app it tests).

Proving the three bugs are dead, locally

The point of parity is that the failures now reproduce on the laptop. I ran each one on purpose before trusting the rollout:

# 1. startup order: kill the DB, restart the app, watch it retry not crashloop
kubectl delete pod -l app=postgres
kubectl logs -f deploy/api        # "waiting for postgres... retry 3" then ready

# 2. readiness gates traffic: app is up but DB down -> 0/1, no endpoint
kubectl get endpointslices -l kubernetes.io/service-name=api

# 3. secret is wired, not baked into the image
kubectl exec deploy/api -- printenv DATABASE_URL   # from Secret, not Dockerfile

All three passed at my desk. The next prod rollout was uneventful — the first one in that project that ever had been.

Before / after

	docker-compose only	Production-like k3d
Backing service	often SQLite/stand-in	same Postgres + version as prod
K8s version	not modeled	pinned to match prod
Probes	none	liveness + readiness, real
Secrets	plaintext env	ConfigMap + Secret
Service discovery	Compose networking	real CoreDNS / Ingress
`depends_on`	hides startup-order bugs	forces real retry logic
First prod rollout	collapsed on unmodeled objects	boring, matched local

Guardrail

"Works in Compose" validates the app, not the manifests. Manifest bugs only appear after a real rollout — so test in k8s.
Production-like is not "everything like prod." It's an environment where you know exactly what matches, what doesn't, and why. The list of documented simplifications is the most underrated line item on the whole checklist.
You don't need the whole prod zoo. Bring up only the namespace and services you're touching this session. Parity isn't recreating everything at once.
Don't try to copy prod exactly. Copying prod onto a laptop is impossible and pointless. The goal is narrower: shrink the gap between how the service behaves at your desk and how it behaves in prod, so packaging and deployment bugs get caught before the rollout.

What I'd do differently

Keep Compose for the earliest, fastest hacking loop — it still earns its place there. Just never again mistake a green Compose run for validation of a Kubernetes deploy. The moment a service is bound for k8s, it gets tested in a local cluster on pinned versions and the same manifests as prod. My rollouts have since gotten boring, which after that first outage is the highest praise I have.

Bottom line: a Compose file is structurally not equivalent to k8s manifests — pin the same versions and run the same manifests in a local cluster, or the manifest-layer bugs wait for prod to find them.

Sources

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

Metronom — Mon, 10 Aug 2026 08:15:50 +0000

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

Nobody deployed. Nobody ran kubectl apply. At 02:00 the orders API started throwing errors we'd fixed three weeks earlier, and it stayed broken for 40 minutes. The kubernetes latest tag problem is not theoretical. This is the incident that made me delete :latest from every manifest we own.

Symptom

On-call paged. Errors that were fixed weeks ago, live again.

2025-... POST /orders 500  InvalidStateError: order already settled
2025-... POST /orders 500
2025-... POST /orders 500

Deploy history: empty. No release. No apply. No human near the cluster. Running code had changed anyway. First reflex was to blame a rogue actor with cluster creds. kubectl get events and the audit log both said no — nobody had written to the API server in hours. The image had moved without a single Kubernetes write.

Root cause

The Deployment referenced a mutable tag.

containers:
  - name: orders
    image: orders:latest

Timeline, reconstructed from the kubelet events and registry push log:

Someone pushed a mid-refactor build and tagged it latest. Never promoted it.
Hours later a pod OOM'd and restarted.
On restart the kubelet pulled latest again — now pointing at the broken build.
Pod came up on code nobody chose to deploy. Prod changed its own version while we slept.

Three separate properties of latest conspired here, and the Kubernetes image docs warn about all of them:

latest is mutable. It points at a different digest tomorrow. You cannot answer "what is running in prod?"
Re-running apply is a no-op. Kubernetes rolls out only when the pod template changes. The literal string image: orders:latest never changes, so the cluster sees nothing to do even when the registry digest is completely different.
A restart re-pulls. imagePullPolicy defaults to Always for latest (and IfNotPresent for a pinned tag). So latest maximizes the odds of a surprise pull on every restart. The inverse trap is just as ugly: a pinned tag with IfNotPresent can stick on a stale local layer and never pull the fix you just pushed. Both are the same disease — the running image and the name you typed have drifted apart.

There's a solid breakdown of the full local-to-CI path that this class of bug forced on us at this writeup on preparing a local setup for deployment and CI. I'll stick to the fixes.

The fix

1. Immutable tags, generated in CI

Rule: every image is tagged by something that resolves to exactly one build. Short SHA, semver, or a digest pin for maximum determinism.

image: ghcr.io/acme/orders@sha256:45b23dee...

Don't hand-write tags. docker/metadata-action emits them from the git context and writes OCI labels like org.opencontainers.image.revision. "What's in prod?" collapses from an investigation to one label read:

docker inspect ghcr.io/acme/orders:sha-abc1234 \
  --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}'

2. One base, kill the copy-pasted `k8s-prod/`

The other sin: a k8s/ folder copied to k8s-prod/ months earlier. Two copies drift. Ours had, and prod's memory limit no longer matched what we'd tested. Configuration drift, cured by a single base with only the diffs layered on top.

Kustomize — built into kubectl, plain YAML, no templating engine to reason about:

# overlays/prod/kustomization.yaml
resources:
  - ../../base
namePrefix: prod-
images:
  - name: orders
    newName: ghcr.io/acme/orders
    newTag: sha-abc1234    # swap the image without touching the Deployment
patches:
  - path: replicas-patch.yaml

# render and eyeball it before it touches the cluster
kustomize build overlays/prod | kubectl apply -f -

The images field is the quiet win: CI rewrites the tag without editing the Deployment YAML. Helm with a per-env values-prod.yaml lands the same place — just mind -f ordering, last file wins, and swapping the order silently changes your prod replica count.

3. A pipeline that physically cannot ship `:latest`

Push-based and deliberately dumb: commit → build → push → update manifest to the immutable tag → apply. GitHub Actions skeleton, using Docker's build-push-action:

name: build-and-deploy
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/acme/orders
          tags: |
            type=sha
            type=semver,pattern={{version}}
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
      - name: Deploy
        run: |
          echo "${{ secrets.KUBECONFIG }}" > kubeconfig
          export KUBECONFIG=kubeconfig
          kubectl set image deployment/orders \
            orders=ghcr.io/acme/orders:sha-${GITHUB_SHA::7}

The deploy step sets an immutable tag, never latest. Now the pod template genuinely differs, the rollout is honest, and a restart can't re-pull a mystery image.

Reproducing the outage on k3d first

I didn't ship any of this blind. I reproduced the exact failure on a laptop k3d cluster, then confirmed the fix killed it. The whole loop is a few minutes and costs nothing.

k3d cluster create latest-repro
# push two builds to the same mutable tag
docker build -t k3d-registry:5000/orders:latest ./good && docker push k3d-registry:5000/orders:latest
kubectl apply -k overlays/local           # Deployment references :latest
kubectl rollout status deploy/orders

# now poison latest and force a restart, no apply involved
docker build -t k3d-registry:5000/orders:latest ./broken && docker push k3d-registry:5000/orders:latest
kubectl delete pod -l app=orders          # simulate the 02:00 crash
kubectl get pod -l app=orders -o jsonpath='{.items[0].spec.containers[0].image}'

The pod came back on the broken image with an empty deploy history — the outage, on demand. Re-run the same script against the SHA-pinned overlay and the delete/restart brings the same image back every time. That's the whole proof. Once it held, I trusted it in prod.

Before / after

	Before	After
Image tags	`:latest`	immutable `sha-…` / digest
"What's in prod?"	an investigation	one label read
Environments	copy-pasted `k8s-prod/`	one base + overlays
Drift	silent, unbounded	the diff is the only diff
Restart	may pull broken `latest`	pinned, deterministic
Rollout on redeploy	sometimes a no-op	always honest

Guardrail

Grep CI and manifests for :latest in a pre-merge check. No merge if it hits.
Most local Kubernetes work carried straight to prod: same Dockerfile, same Kustomize base, same probes. Prod parity is the payoff. Prod only adds — registry, Secrets, TLS, HPA, PodDisruptionBudget.
Some things must never cross the line: --reload, file-sync dev accelerators, exposed debug ports. The prod overlay is the mechanism that keeps them out.
Pin a digest, not just a tag, for anything you truly can't afford to move — @sha256:... is the only reference in Kubernetes that a registry cannot repoint under you.

What I'd do differently

Move off the push pipeline sooner. It has three built-in weaknesses I already know will bite:

CI holds the kubeconfig. CI access equals prod access.
Drift is invisible. Someone runs kubectl edit by hand and CI never notices.
No automatic rollback. Ship broken, revert by hand, at 02:00.

Those three are exactly what GitOps removes. Per the CNCF OpenGitOps principles, Git is the source of truth and an in-cluster agent (Argo CD or Flux) pulls and reconciles drift. CI's job ends at pushing the image and committing the tag. That's the next rung: CI builds an immutable image → a bot bumps the tag in Git → the agent applies it.

Bottom line: the outage wasn't a Kubernetes flaw, it was a mutable tag plus a copy-pasted folder — pin an immutable SHA and collapse onto one base before your own 02:00 page.

Sources

k3d Ingress Localhost Setup: Fixing "curl: could not resolve host" and the Dead Ingress

Metronom — Sun, 09 Aug 2026 08:50:44 +0000

k3d Ingress Localhost Setup: Fixing "curl: could not resolve host" and the Dead Ingress

Worked in Chrome. Died in curl. Safari showed a blank page. The Ingress existed and routed to a void. I burned two days on a k3d ingress localhost setup, and not one of the three failures was a bug. Every one was a documented layer I didn't understand.

What broke

Moving a checkout service off docker-compose into local k3d, because staging kept surfacing bugs the laptops never saw. FastAPI on 8080, Postgres behind it. Pods came up green.

$ curl http://localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused

Expected — the cluster has its own network. What I didn't grasp was the three layers between my laptop and a pod. I faceplanted on all three. There's a solid breakdown of the whole path in this local networking and Ingress writeup. Here's the compressed field version.

Symptom 1: the Ingress existed and did nothing

Reasonable manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: checkout
  namespace: checkout
spec:
  rules:
  - host: checkout.localhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: checkout
            port:
              number: 80

kubectl apply, no errors. kubectl get ingress showed it. No route, no response, no error.

Root cause

The line from the Ingress docs:

You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.

An Ingress manifest is a declaration of rules. A controller has to be running to read them and accept traffic. I assumed "Ingress" was something Kubernetes did. It's something a controller does.

k3d is built on k3s, and k3s ships Traefik out of the box plus a ServiceLB so LoadBalancer services don't hang pending — the k3d bundled-k3s docs cover both. So the controller was there. My real problem was one layer down.

Symptom 2: no route from the host

Traefik ran inside the cluster on 80/443. Nothing on my host forwarded to it.

The fix

k3d needs a port mapping to its loadbalancer node, set at creation:

k3d cluster create dev --api-port 6550 -p "8081:80@loadbalancer" --agents 2

-p "8081:80@loadbalancer" maps host 8081 to Traefik's port 80 — the k3d exposing-services guide documents the HOST:CONTAINER@loadbalancer syntax.

The gotcha that cost the most: you can only set this at cluster creation. There is no k3d cluster edit that adds a port to a running cluster. I hunted for one for an hour. Delete and recreate:

k3d cluster delete dev
k3d cluster create dev -p "8081:80@loadbalancer" --agents 2

With that in place the chain is the real prod routing shape, just *.localhost instead of a domain and k3d instead of a cloud LB:

curl -> host :8081 -> Traefik :80 -> rule host: checkout.localhost -> Service -> Pod

How I confirmed each hop rather than guessing. Traefik has the mapping the moment the LB container exposes the port:

$ docker ps --format '{{.Names}}\t{{.Ports}}' | grep serverlb
k3d-dev-serverlb   0.0.0.0:8081->80/tcp, ...

Then the Service actually has endpoints — an empty endpoints list here means the selector doesn't match any pod, which is its own silent failure mode:

$ kubectl get endpoints checkout -n checkout
NAME       ENDPOINTS           AGE
checkout   10.42.0.14:8080     3m

Empty ENDPOINTS? The Ingress and port mapping are irrelevant — fix the Service selector first.

Symptom 3: Chrome worked, curl and Safari didn't

After the recreate, http://checkout.localhost:8081 opened in Chrome. Four minutes of feeling clever, then QA on Safari said blank, and the integration test failed:

curl: (6) Could not resolve host: checkout.localhost

Same URL, same cluster, opposite results by client. I blamed the cluster.

Root cause

The .localhost TLD is reserved by RFC 6761 for loopback, and resolvers are supposed to return 127.0.0.1 without hitting DNS. "Supposed to" does the heavy lifting:

Chrome and Firefox resolve any *.localhost to loopback themselves. Zero config. That's why it worked.
Safari on macOS does not — it defers to the OS resolver and finds nothing.
curl and every non-browser HTTP client treat *.localhost as an ordinary domain. No entry, could not resolve host.

The fix

One line, now in our onboarding README:

echo '127.0.0.1  checkout.localhost' | sudo tee -a /etc/hosts

(Windows: C:\Windows\System32\drivers\etc\hosts, as admin.) After that, curl, Safari, and CI all behaved. Rule: Chrome/Firefox get *.localhost free; curl, Safari, and any custom domain need an /etc/hosts entry.

The DNS trap that bit a week later

Saving you the trip. Inside the cluster you reach services by name via CoreDNS. Within one namespace, checkout hit Postgres as postgres. I moved Postgres to a shared data namespace and the short name broke:

$ nslookup postgres
** server can't find postgres: NXDOMAIN

From another namespace you need postgres.data or the full FQDN postgres.data.svc.cluster.local — the <service>.<namespace>.svc.cluster.local form in the DNS for Services and Pods spec. Test resolution from a throwaway pod:

kubectl run -it --rm dnstest --image=busybox --restart=Never -- \
  nslookup checkout.checkout.svc.cluster.local

Rule of thumb: hard-code the short name in your config and you've hard-coded "same namespace forever." I now use the two-part <service>.<namespace> form everywhere, so moving a dependency across namespaces is a one-line env change, not a debugging session.

Runbook: prod-like local routing from scratch

What I do now, every new cluster, in order:

Create with the mapping baked in: k3d cluster create dev -p "8081:80@loadbalancer" --agents 2.
Confirm the controller exists: kubectl -n kube-system get pods | grep traefik.
Apply Deployment + Service, confirm endpoints are non-empty: kubectl get endpoints <svc> -n <ns>.
Apply the Ingress with host: <app>.localhost.
Add the hosts entry so curl and CI match Chrome: echo '127.0.0.1 <app>.localhost' | sudo tee -a /etc/hosts.
Smoke test with curl, not the browser: curl -sS http://<app>.localhost:8081/healthz.

Step 6 with curl and not Chrome is deliberate — curl fails loudly on the DNS gap, Chrome hides it. Test with the strict client first.

When I reach for what

Port-forward, NodePort, and Ingress aren't competitors. Different tools:

Ingress (Traefik + *.localhost) — default. Reproduces the prod path: host/path/TLS through a controller, human-friendly domains, load-balances across replicas. Configure once, forget it.
kubectl port-forward — a debug tunnel to a single pod. Dies when the pod is recreated. It's a probe, not access.
NodePort — the lazy fallback on 30000-32767. Least prod-like. I basically never use it; k3d docs warn wide NodePort ranges can hang the system on iptables churn.

# everyday debug reflex — straight into a pod, bypassing the Service
kubectl port-forward -n checkout deploy/checkout 8080:8080
curl http://localhost:8080/healthz

Guardrail

When local networking "flakes," check three things in order before touching the cluster:

Is a controller actually running? A bare Ingress does nothing on its own.
Was the -p "HOST:80@loadbalancer" mapping set at creation? It's creation-time only — recreate if not.
What client are you testing with? Chrome hides the DNS gap; curl and Safari expose it. Add the /etc/hosts line.

What I'd do differently

Put the port mapping and the /etc/hosts line in the cluster bootstrap script from day one, so nobody recreates a cluster five times to learn what a controller is. And test with curl first, not Chrome — Chrome's free *.localhost resolution papers over a gap CI will hit anyway.

Bottom line: none of these were bugs — controller, creation-time port mapping, and client-side .localhost resolution are three separate layers, and knowing which one you're on turns two lost days into two minutes.

Sources

Zero-Downtime Kubernetes Deployment: Killing Deploy 5xx With k3d Before Prod

Metronom — Sat, 08 Aug 2026 17:16:59 +0000

Zero-Downtime Kubernetes Deployment: Killing Deploy 5xx With k3d Before Prod

Every deploy of our payments API leaked 5xx. A dozen requests, every time, right at the rollout. They retried and succeeded, so nobody paged. We process money. "A dozen dropped requests, they retried" is not a sentence I say to a customer.

This is a zero-downtime kubernetes deployment story. It has no service mesh in it. The fix was four boring things — readiness probes, real SIGTERM handling, a preStop sleep, and maxUnavailable: 0 — and I proved every one of them on a laptop k3d cluster before prod ever saw the change.

Symptom

billing-api. Python, FastAPI, Postgres. Two replicas. Green on every dashboard: pods Running, CPU flat, error rate a rounding error.

Then the synthetic checks:

2025-... GET /charge 200
2025-... GET /charge 503   <-- rollout starts
2025-... GET /charge 502
2025-... GET /charge 503
2025-... GET /charge 200   <-- rollout ends

Every deploy. A burst of 502/503 that lined up exactly with the new ReplicaSet coming up. For a year I filed it under "cost of rolling updates." Wrong on every count.

Root cause

I read our Deployment against a production-readiness checklist. It was embarrassing.

spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: billing-api
          image: ghcr.io/acme/billing-api:sha-9f2c1a
          # no probes
          # no resources
          # no graceful shutdown

Four independent bugs, each producing 5xx on its own:

No readiness probe. Kubernetes routed traffic to pods before FastAPI had opened its DB pool.
The container launched via sh -c "uvicorn ...". PID 1 was the shell. The app never received SIGTERM.
Endpoint removal and SIGTERM happen in parallel on pod deletion. Traffic hit pods that had already started shutting down.
Default RollingUpdate allowed 25% unavailable — a real capacity dip on two replicas.

There is a solid, end-to-end breakdown of why each of these matters at this writeup on making a local cluster production-like; I won't re-derive the theory here. I'll show the fixes and how I tested them in k3d — CNCF's k3s in Docker — before shipping.

The fix

1. Readiness is the gate

Split health into two endpoints. Liveness answers "is the process alive." Readiness answers "can I serve a request right now."

@app.get("/healthz")
def healthz():
    # liveness: process is up. Nothing else.
    return {"status": "ok"}

@app.get("/ready")
def ready():
    # readiness: DB pool reachable? If not, 503.
    # Pod leaves endpoints. It is NOT restarted.
    ...
    return {"status": "ready"}

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5

The model: liveness restarts, readiness reroutes. A readiness failure pulls the pod out of the Service endpoints without killing it — the Kubernetes probe docs say so directly. Do not put a DB check in liveness. Postgres flickers, liveness fails, Kubernetes restarts a healthy app, and now you have a restart storm on top of a DB blip. Dependency checks go in readiness. Always.

Verified on k3d in about ten seconds:

kubectl get endpointslices -n billing -w

Kill the DB connection. Watch the pod drop out of endpoints. Restore it. Watch it come back. No restart. That one experiment taught me more than a year of prod Grafana.

2. SIGTERM, PID 1, and the shell that ate the signal

CMD sh -c "uvicorn app.main:app ..." makes sh PID 1. sh does not forward SIGTERM. The app ran until the 30s grace period elapsed, then got SIGKILL'd mid-request.

One line. Exec form. uvicorn becomes PID 1.

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]

Modern uvicorn drains cleanly on SIGTERM: stop accepting, finish in-flight, close the pool, exit — but only if it receives the signal. The Pod lifecycle docs spell out the sequence: kubelet runs preStop, sends SIGTERM to PID 1, waits terminationGracePeriodSeconds, then SIGKILLs. A shell at PID 1 eats step two.

Check what your PID 1 actually is:

kubectl exec -it deploy/billing-api -n billing -- ps -o pid,comm
#   PID COMMAND
#     1 uvicorn      <- good
#     1 sh           <- your signal is going nowhere

3. The endpoints race, and the preStop sleep

Here is the one nobody tells you. On pod deletion, endpoint removal and SIGTERM fire in parallel. For a short window kube-proxy and the ingress still route to a pod that has already begun shutdown. Those requests die. That was the rest of our 5xx. The CNCF pod-termination walkthrough spells out the race and why a short preStop sleep is the standard fix.

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: billing-api
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 15"]

The preStop sleep buys time for routing to converge before the app tears down.

Gotcha that nearly bit me: preStop counts against the grace period. sleep 30 with a 30s grace budget leaves zero seconds to drain — SIGKILL again. Keep the sleep well under budget. sleep 15, grace 30.

4. maxUnavailable: 0

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  minReadySeconds: 5

maxUnavailable: 0 means the ready-pod count never dips during a rollout — the rolling-update guide documents the surge-then-retire dance. maxSurge: 1 brings up a new pod, waits for its readiness, retires an old one, loops. minReadySeconds guards against pods that report ready and immediately crash. None of it works without the readiness probe from fix 1 — the rollout waits on it.

Verifying zero downtime locally

Runbook I ran on k3d before touching prod:

Spin the cluster: k3d cluster create billing --agents 2.
Apply the manifests, wait for kubectl rollout status deployment/billing-api -n billing.
Hammer it in one terminal:

while true; do curl -s -o /dev/null -w "%{http_code}\n" http://billing.localhost:8081/; done

Trigger a rollout in another: kubectl set image deploy/billing-api billing-api=...:sha-new -n billing.
Watch the curl loop. Any non-200 is a bug still in the manifest.

The first few passes still threw 5xx — which told me my readiness-plus-preStop combo wasn't done. I iterated on the laptop, for free, on a Tuesday, instead of debugging it in prod on a Friday night. When it held zero errors through a dozen rollouts locally, I shipped. Prod matched: zero, sustained.

Guardrail

Memory limit too low = death, not slowdown. CPU is compressible (throttled). Memory is not — OOMKilled, exit code 137. Set the request near P99 + ~20% headroom.
limit without request sets request = limit silently. Inflates scheduler demand until pods sit Pending.
replicas: 1 is downtime by definition. Any restart, drain, or rollout is an outage.
Slow starters want a startup probe, not a giant initialDelaySeconds. The startup probe holds off liveness/readiness until boot completes, so liveness can stay strict.
The production-readiness checklist is now a required review item on every Deployment PR. No probes, no shutdown story, no merge.

What I'd do differently

Fold all of this into the Helm chart as defaults so no service can ship without probes and a drain story. And stop treating "it retries" as "it works." A retried 5xx is still a 5xx that reached a customer.

If your deploys leak errors and the team shrugs it off as normal — it isn't. Grab a local cluster and break your own rollout until it stops breaking.

Bottom line: zero-downtime rollouts aren't a mesh feature, they're four manifest details, and a laptop k3d cluster catches all four before prod does.

Sources

Real Kubernetes on your laptop in ~2 minutes with k3d (and you can throw it away for free)

Metronom — Fri, 07 Aug 2026 08:53:15 +0000

Chapter 5 of a local-Kubernetes series: the payoff chapter where you spin up an actual cluster. k3d gives you a real Kubernetes API, not an emulation, and it's so cheap to create and destroy that recreating from scratch is routine.

Key takeaways

k3s vs k3d: k3s is a certified, minimalist Kubernetes distro (whole control plane in one binary); k3d is a "lightweight wrapper to run k3s in Docker." Nodes are Docker containers.
Why it's fast: k3s defaults to embedded SQLite instead of etcd — no etcd cluster to stand up, so fast start and low memory. (Multiple server nodes auto-switch to embedded etcd.)
The one command to keep:

k3d cluster create dev \
  --servers 1 --agents 2 \
  --registry-create k3d-registry.localhost:5000 \
  -p "8081:80@loadbalancer"

Built-in registry (the fix for chapter 1's ImagePullBackOff-from-local-Docker) + a forwarded port, in one line.

Context gotcha: k3d names the context k3d-dev, not dev. Always kubectl config current-context before anything destructive.
Registry naming: k3d prepends k3d-, so k3d registry create registry.localhost is referenced as k3d-registry.localhost:5000 in --registry-use and in image: fields. Push from host via localhost:5000.
Port-forwarding via serverlb with @loadbalancer (k3s ships Traefik) — but port forwards are fixed at creation time; to add one you recreate the cluster (cheap in k3d).
Honest "when to use kind or minikube instead": kind for strict upstream parity / conformance; minikube for learning + addons + VM isolation.

Full article: https://dorokhovich.com/blog/local-k8s-cluster-with-k3d?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-cluster-with-k3d

Kubernetes Service 'Endpoints: ': The One Mismatched Label That Drops All Traffic

Metronom — Fri, 07 Aug 2026 08:52:38 +0000

The service was deployed. The Pod was Running. The Service existed. Zero traffic reached it. When a Kubernetes Service shows Endpoints: <none>, it is not broken — it's matching nothing, because its selector doesn't match the pod labels. The fix was one hyphen in one YAML file. Understanding why changed how I read every manifest since.

Symptom

Three manifests, the ones everyone starts with: a Namespace, a Deployment, a Service. Applied clean. Then a curl through the Service:

$ kubectl apply -f ./k8s/
namespace/myapp created
deployment.apps/myapp created
service/myapp created

$ curl http://myapp.myapp.svc.cluster.local
curl: (7) Failed to connect: Connection refused

No error on apply. Pod healthy. Logs clean. Nothing to grab onto. I'd copied fragments from three tutorials and somewhere in the stitching a label drifted. There's a solid breakdown of the minimal skeleton and how labels glue it together at this manifests walkthrough.

Root cause

The thing nobody had made click: nothing connects these objects with hard references. No "Service, here are the pod IDs." It's all labels — arbitrary key/value tags — and label selectors matching against them, per the Kubernetes labels-and-selectors concept. Three separate things reference the same app: myapp label:

The Deployment stamps app: myapp onto every pod it creates via spec.template.metadata.labels.
The same Deployment uses spec.selector.matchLabels: {app: myapp} to recognize those pods as its own.
The Service uses its own spec.selector: {app: myapp} to find pods, collect their addresses into an EndpointSlice, and balance traffic — the Service docs describe the controller continuously scanning for matching pods.

Two selectors, same pod labels, two problems, and no schema anywhere enforcing that they agree — the string just has to match, character for character. If the Service's selector points at a label no pod has, the Service binds to nothing. It exists, it has an IP, traffic goes nowhere.

The fix

Diagnosis was instant:

$ kubectl describe svc myapp -n myapp
...
Endpoints:   <none>

The Deployment stamped app: myapp; I'd typed app: my-app in the Service selector. One hyphen. Endpoints: <none> means the pod label and the Service selector didn't match — full stop. Fixed the selector, re-applied, endpoints populated immediately:

Endpoints:   10.42.0.14:8080,10.42.0.15:8080

A related rule that would've caught me later, and the Deployment docs make it hard: spec.selector.matchLabels must match spec.template.metadata.labels, or Kubernetes rejects the manifest at apply time. Different failure (loud, not silent), same root cause — labels that don't line up.

The runbook I use now for "Service, but no traffic"

Whenever a Service exists and nothing reaches it, I run this in order. It localizes the break in under a minute, every time:

# 1. Does the Service have any endpoints at all?
kubectl get endpointslices -n myapp -l kubernetes.io/service-name=myapp
#    empty -> selector matches nothing. Stop here, fix labels.

# 2. What selector is the Service actually using?
kubectl get svc myapp -n myapp -o jsonpath='{.spec.selector}'; echo

# 3. What labels do the pods actually carry?
kubectl get pods -n myapp --show-labels

# 4. Do they intersect? Ask the API directly with the Service's selector.
kubectl get pods -n myapp -l app=myapp
#    no pods listed -> confirmed mismatch between (2) and (3)

Steps 2 and 3 side by side make the mismatch obvious — my-app next to myapp jumps out once you're looking at both. Before I had this list I'd start by tcpdumping the pod network, which is exactly the wrong end of the problem.

The other trap: ports that don't mean what you think

While in there, I finally understood the port fields I'd been cargo-culting:

apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: myapp
spec:
  selector:
    app: myapp
  ports:
    - protocol: TCP
      port: 80          # the port the Service listens on
      targetPort: 8080  # the port on the pod it forwards to

port is where cluster clients connect (80). targetPort is the pod's actual port (8080). Callers hit myapp on 80; the app only ever listens on 8080; the Service translates. Also why kubectl port-forward svc/myapp 8080:80 and port-forward pod/... 8080:8080 use different numbers — one targets the Service port, the other the container port.

Before / after

	Before	After
Symptom	Service exists, no traffic, no error	traffic flows
Root cause	selector `my-app` vs label `myapp`	labels aligned
Diagnosis	hours of guessing	`describe svc` → `Endpoints: <none>`
Mental model	"objects are wired together"	"labels are the only glue"
Manifests	pasted from 3 tutorials	one reviewed `k8s/` folder in git

The namespace traps hiding underneath

Once labels made sense, a second layer did too — all about namespaces. Objects live in a dedicated myapp namespace, not default — namespaces make it easier to delete everything at once, attach limits, and avoid colliding with other teams. But namespaces shape DNS. Services get a name of the form <service>.<namespace>.svc.cluster.local, per the DNS-for-Services spec. From the same namespace the short name myapp works; from a different namespace you need the FQDN myapp.myapp.svc.cluster.local. A job in another namespace "couldn't reach" the service purely because it used the short name — looks like a networking failure, is really a DNS-scope misunderstanding. The tell is that the error is DNS resolution, not connection refused: could not translate host name "myapp" means the name never resolved, so the packet never left. nslookup myapp.myapp.svc.cluster.local from inside a debug pod confirms which form the cluster actually answers.

Guardrail

Endpoints: <none> = label/selector mismatch. First thing to check when a Service "works" but nothing reaches it.
Deployment selector must equal template labels, or apply is rejected outright.
Forgetting -n myapp makes kubectl get look in default and report nothing — the "where did everything go?" panic. Pin it once: kubectl config set-context --current --namespace=myapp.
port vs targetPort are not interchangeable. Backwards and traffic dies at the pod.
Two selector syntaxes, same meaning. The Service writes flat selector: {app: myapp} (equality-based); the Deployment uses selector.matchLabels (which can also do matchExpressions). Different syntax generations, both compare labels.
kubectl diff -f deployment.yaml before every apply. apply is declarative and idempotent, easy to run blind; diff catches an accidental label or replica edit before the cluster does.
A green apply proves nothing about routing. The three objects can each be valid and still not connect. Endpoints are the only proof that the wiring worked.

What I'd do differently

The deeper problem wasn't the hyphen — it was why I had one. My manifests were fragments on a laptop, applied from the desktop, no history, no review. Now they live in a flat k8s/ folder at the repo root next to the code, applied as a directory with kubectl apply -f ./k8s/, and every label change goes through a PR where a mismatch like mine is a one-line diff a teammate spots. apply (declarative) beats create (imperative) for anything in version control — create fails if the resource exists and can't update, useless for repeatable manifests — and the declarative-config guide covers the reasoning end to end. I'd have started there instead of pasting from three tutorials.

Bottom line: when a Service drops all traffic with no error, run kubectl describe svc and read the Endpoints line — <none> is a label/selector mismatch every time, not a networking failure.

Sources

Run Postgres in Kubernetes for Local Development: Killing the docker-compose Parity Gap

Metronom — Thu, 06 Aug 2026 09:29:20 +0000

For a year the app ran in Kubernetes and its Postgres, Redis and RabbitMQ ran in a docker compose up on the side. It worked until it didn't. If you're going to run Postgres in Kubernetes for local development, do it in the same cluster as the app — two infra descriptions drift, and the drift is where the rollout bugs come from. Here's the migration, including the Helm licensing ambush that nearly derailed it.

Symptom

myapp (FastAPI on 8080) ran in a k3d dev cluster, restarted by Tilt on every save. Postgres/Redis/RabbitMQ lived in a compose.yaml next to it. Two worlds.

The app reached its DB at localhost:5432 locally but needed a Service DNS name in the cluster. Every deploy, the same class of break:

sqlalchemy.exc.OperationalError: could not translate host name "localhost"
  to address: Name or service not known
# prod wanted: postgresql://myapp:...@postgres:5432/myapp

Everything we'd debugged in Compose — service addresses, env vars, startup order — had to be rewritten from scratch for Kubernetes, and that rewrite was exactly where the bugs lived.

Root cause

Two parallel infra descriptions. In production only the Kubernetes one exists. Anything validated in Compose proved nothing about the manifests that actually ship. This is the Twelve-Factor dev/prod parity principle: keep dev and prod the same, especially the tools. There's a solid breakdown of bringing dependencies into the cluster instead of Compose on the side at this dependencies writeup.

Bring the dependencies up in the same cluster with the same manifests and you get unified DNS (the app finds the DB by service name postgres, not localhost:5432), shared Secrets and ConfigMaps, and you catch manifest problems — RBAC, resource limits, probes, network policies — before the rollout, not on a Friday. Heuristic: if the service is going to Kubernetes, run its dependencies in the cluster.

The ambush: the 2025 Bitnami brownout

The default for stateful dependencies has always been a Helm chart, historically Bitnami (bitnami/postgresql, etc.). In 2025 that broke. Per Bitnami's own catalog-change notice, public image brownouts started 2025-08-28, and from 2025-09-29 most public Bitnami OCI charts and images moved behind a Broadcom commercial subscription. What's left public is limited and only on -latest; the rest sits in a bitnamilegacy repo, unsupported, no security patches.

Blindly running helm install bitnami/* in 2026 is no longer safe. Options: Chainguard's drop-in replacement charts (forked for compatibility), official vendor charts, or a deliberately pinned legacy tag (acceptable-ish for local, not prod).

The fix: a raw Postgres manifest

The most reliable dev Postgres turned out not to be Helm at all — a raw three-object manifest, nothing to look up in a chart repo, no subscription dependency:

# postgres.yaml -- Deployment + Service + PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: postgres-data, namespace: myapp }
spec:
  accessModes: ["ReadWriteOnce"]
  resources: { requests: { storage: 1Gi } }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: postgres, namespace: myapp }
spec:
  replicas: 1
  selector: { matchLabels: { app: postgres } }
  template:
    metadata: { labels: { app: postgres } }
    spec:
      containers:
        - name: postgres
          image: postgres:17
          env:
            - { name: POSTGRES_DB, value: myapp }
            - { name: POSTGRES_USER, value: myapp }
            - { name: POSTGRES_PASSWORD, value: devpass }
            - { name: PGDATA, value: /var/lib/postgresql/data/pgdata }
          volumeMounts:
            - { name: data, mountPath: /var/lib/postgresql/data }
          readinessProbe:
            exec: { command: ["pg_isready", "-U", "myapp", "-d", "myapp"] }
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: data
          persistentVolumeClaim: { claimName: postgres-data }
---
apiVersion: v1
kind: Service
metadata: { name: postgres, namespace: myapp }
spec:
  selector: { app: postgres }
  ports: [{ port: 5432, targetPort: 5432 }]

Now myapp reaches the DB at postgresql://myapp:devpass@postgres:5432/myapp by service name — the same shape as prod.

Wiring Tilt and getting startup order right

Tilt brings dependencies up and tears them down with the project. Ordering is the point:

k8s_yaml('postgres.yaml')
k8s_resource('postgres', port_forwards=['5432:5432'])
k8s_resource('myapp', resource_deps=['postgres'])

resource_deps makes Tilt wait until postgres passes its pg_isready readiness probe before starting myapp. Verify the ordering held with a quick check — the app should never log a connection error on a cold start:

tilt up
kubectl logs deploy/myapp -n myapp | grep -i "could not translate\|connection refused" || echo "clean start"

A trap Tilt's own Helm docs confirm: Tilt's built-in helm() is really helm template and skips Helm hooks, so for third-party charts with initialization you want the helm_resource extension (a real helm install). That distinction costs hours if you miss it.

The surprise that ate an afternoon: ephemeral data

My whole database vanished after k3d cluster delete. In k3d the default local-path provisioner k3s ships writes to /var/lib/rancher/k3s/storage inside the k3d node's container filesystem — ephemeral by default. Often desirable (clean seed every run). To persist across cluster recreation, map a host directory at creation:

k3d cluster create dev \
  --volume $HOME/k3d-storage:/var/lib/rancher/k3s/storage@all

The PVC detail that confuses everyone once: k3d's local-path StorageClass uses WaitForFirstConsumer binding and a reclaimPolicy of Delete, and its volumes are node-local — as the local-path-provisioner project notes, a local-path PVC is pinned to a node via kubernetes.io/hostname, so the pod always reschedules to the same node. None of it matters until you delete the cluster; then survival comes down entirely to whether you mapped a host dir.

Migrations: a separate Job, not the app's initContainer

An empty Postgres isn't enough — myapp expects a schema. Tempting to put alembic upgrade head in myapp's initContainer, but with multiple replicas each pod races to migrate, and a long migration risks a probe killing it mid-run. Use a standalone migrations Job, ordered in Tilt:

k8s_resource('myapp-migrate', resource_deps=['postgres'])
k8s_resource('myapp-seed',    resource_deps=['myapp-migrate'])
k8s_resource('myapp',         resource_deps=['myapp-seed'])

Seeding is another Job after migrations — trivial to disable in prod by just not wiring it up. The Job also gives you a clean audit trail: kubectl logs job/myapp-migrate shows exactly which migrations ran, and a failed migration blocks the app start instead of half-migrating under live traffic.

Before / after

	Before	After
Dependencies	docker-compose on the side	in-cluster, like prod
DB address	`localhost:5432` locally	`postgres` service name everywhere
Rollout surprises	env/order rewritten for k8s	caught locally
Chart risk	unknowingly on Bitnami	raw manifest / Chainguard
Migrations	ad hoc	separate Job, ordered by Tilt

Guardrail

Make data survival an explicit per-project decision: ephemeral for a clean seed, host-mapped for continuity across k3d cluster delete. Never a surprise.
OSS dependencies run as-is in the cluster. Managed services (S3, SQS, DynamoDB) get LocalStack for the fast loop, but verify against a real dev account for anything with subtle semantics — an emulator is never 1:1 with the cloud.
Audit every helm install for a Bitnami source before you build on it. That chart may rot underneath you.
Migrations and seed are separate Jobs, never baked into the app's startup.

What I'd do differently

Move off Compose-on-the-side the day the app went to Kubernetes, not a year later. Every week of two-worlds cost us a rollout surprise that a single cluster would have caught for free.

Bottom line: run your Postgres/Redis/RabbitMQ in the same cluster as the app so the DB address, startup order, and manifests are validated locally — and audit any Bitnami chart before it rots under you.

DEV Community: Metronom

Every MCP revision, and what each one took back

Hacker News + dev.to (a concrete '2-5 minutes to a few seconds' DX win is a proven front-page hook)

We got the Kubernetes inner loop from 2–5 minutes back to seconds with Tilt

Key takeaways

Local Kubernetes Dev — Part 8: Tilt — a fast development loop

What GitHub's MCP Server Costs You Before You Ask It Anything

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

The golden set

ImagePullBackOff — and the k3d containerd trap

Symptom

Root cause

The fix

CrashLoopBackOff — the status that hides its own cause

Symptom

The fix

Pending — the scheduler couldn't place it

Symptom

Root cause

The fix

OOMKilled — the exit-137 signature

Symptom

Root cause

The Service that's green but silent

Symptom

The fix

Before / after

Guardrail

What I'd do differently

Sources

docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

Symptom

Root cause

The fix that didn't work: Kompose

The fix that worked: a production-like k3d cluster

Proving the three bugs are dead, locally

Before / after

Guardrail

What I'd do differently

Sources

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

The Kubernetes :latest Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

Symptom

Root cause

The fix

1. Immutable tags, generated in CI

2. One base, kill the copy-pasted k8s-prod/

3. A pipeline that physically cannot ship :latest

Reproducing the outage on k3d first

Before / after

Guardrail

What I'd do differently

Sources

k3d Ingress Localhost Setup: Fixing "curl: could not resolve host" and the Dead Ingress

k3d Ingress Localhost Setup: Fixing "curl: could not resolve host" and the Dead Ingress

What broke

Symptom 1: the Ingress existed and did nothing

Root cause

Symptom 2: no route from the host

The fix

Symptom 3: Chrome worked, curl and Safari didn't

Root cause

The fix

The DNS trap that bit a week later

Runbook: prod-like local routing from scratch

When I reach for what

Guardrail

What I'd do differently

Sources

Zero-Downtime Kubernetes Deployment: Killing Deploy 5xx With k3d Before Prod

Zero-Downtime Kubernetes Deployment: Killing Deploy 5xx With k3d Before Prod

Symptom

Root cause

The fix

1. Readiness is the gate

2. SIGTERM, PID 1, and the shell that ate the signal

3. The endpoints race, and the preStop sleep

4. maxUnavailable: 0

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

2. One base, kill the copy-pasted `k8s-prod/`

3. A pipeline that physically cannot ship `:latest`