DEV Community: Metronom

You can catch production failures on your laptop: probes, OOMKilled, and zero-downtime in k3d

Metronom — Tue, 08 Sep 2026 10:27:02 +0000

"Works on my laptop" and "survives production" are different things — but almost everything separating them is testable locally in k3d. Probe behavior, liveness restarts, rolling updates, SIGTERM shutdown: k3d reproduces all of it.

Takeaways

CPU throttles, memory kills. CPU is compressible — hit the limit and the container just slows down. Memory is not — exceed limits.memory and the kernel kills the process: OOMKilled, exit code 137. So a low CPU limit = slowness; a low memory limit = instant Pod death. Give memory headroom (request ~P99 + 20%); CPU can ride closer to P95.
Three probes, three jobs:
- liveness fails -> container is restarted. Keep it simple (GET /healthz), never hit the DB — a dependency blip would restart a healthy container.
- readiness fails -> Pod is pulled from the Service endpoints (no traffic), not restarted. This is the right place to check PostgreSQL.
- startup gates the other two — until it passes, liveness/readiness are disabled. Use it for slow starters instead of a big initialDelaySeconds.
QoS: Guaranteed (request=limit everywhere) evicted last; BestEffort (no requests/limits) evicted first. Always set at least requests.
Graceful shutdown: on delete, the Pod goes Terminating, preStop runs, kubelet sends SIGTERM to PID 1, waits terminationGracePeriodSeconds (default 30s), then SIGKILL. Your app must catch SIGTERM — and uvicorn must be PID 1 (exec-form CMD, no shell wrapper) or the signal never arrives.
The endpoints race: removing the Pod from endpoints and sending SIGTERM happen in parallel, so traffic can still hit a shutting-down Pod (5xx during deploy). Fix: a preStop: sleep 15 so routing updates first. Keep preStop well under the grace period — it counts against the same budget.
Zero-downtime rollout: maxUnavailable: 0, maxSurge: 1, plus minReadySeconds. Without a readiness probe, zero-downtime is impossible — the rollout waits on readiness before retiring old Pods.

Verify locally: kubectl get endpointslices -n myapp -w while running a while true; do curl ... loop during a rollout.

Full article: https://dorokhovich.com/blog/local-k8s-making-it-production-like?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-making-it-production-like

Local Kubernetes Dev — Part 13: Making your local setup truly production-like

Metronom — Sun, 06 Sep 2026 08:23:06 +0000

"Works on my laptop" ≠ "survives in prod." And you can verify almost all of that difference locally in k3d.

Probe behavior, restarts from liveness, how a rolling update proceeds, how the app shuts down on a signal — k3d reproduces all of it like a real cluster. Which means the traps get caught at your desk, not in prod on a Friday evening.

In the new article (part 13) we add, one at a time, the things that make a service resilient to the Deployment:
• requests/limits and the key distinction: CPU is throttled when exceeded (slows down), but memory is NOT compressible — the process gets killed, that's OOMKilled with code 137;
• the three probes and who does what: liveness restarts a hung container, readiness pulls the Pod out of endpoints (no restart, DB checks go here too), startup gives a slow start room to come up;
• QoS classes and who gets evicted first (BestEffort);
• graceful shutdown: why the app must catch SIGTERM, the race with endpoints, and the cure — a preStop sleep;
• zero-downtime rollout: maxUnavailable: 0 + maxSurge: 1, and why zero downtime is fundamentally impossible without readiness.

At the end — a prod-readiness checklist. Read it: https://dorokhovich.com/blog/local-k8s-making-it-production-like?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-making-it-production-like

events -> describe -> logs: the Kubernetes debugging chain (and the --previous flag you're not using)

Metronom — Sat, 05 Sep 2026 12:30:23 +0000

Almost every Kubernetes debugging session follows the same route. Memorize it: events -> describe -> logs. First see what happened at the cluster level, then the details of a specific Pod, then what the app itself said.

Takeaways

kubectl logs --previous (-p) is the key to CrashLoopBackOff. The current container just restarted and has written nothing (or crashed again), so plain kubectl logs shows nothing useful. -p pulls the logs of the previous, already-dead instance — that's where the real stack trace lives. Note: -f and -p don't combine (you can't stream something already dead).
kubectl describe pod → read the Events block bottom-up. Look for Warning with reasons like Failed, BackOff, FailedScheduling. describe tells you what happened at the k8s level ("container crashing, so BackOff"); logs -p tells you why.
CrashLoopBackOff and ImagePullBackOff are NOT Pod phases. The five real phases are Pending, Running, Succeeded, Failed, Unknown. Those *BackOff strings are display fields kubectl synthesizes from container state — the underlying phase often stays Pending. Confusing the two makes the docs harder to read.
kubectl get events isn't sorted by time by default — add --sort-by='.lastTimestamp'. Events also have a ~1h TTL, so yesterday's incident is gone.
Check from inside with port-forward (tunnel via the API server) and exec. No shell (distroless)? Use kubectl debug to attach an ephemeral container.
k9s is a terminal UI over the same kubectl actions — fast, but understand the commands first.
Don't over-build. For local dev, logs + describe + events + k9s are enough. A full Prometheus/Grafana/OTel stack is a direction to grow, not a local requirement. Mnemonic: metrics = what, logs = details, traces = why.

Full article: https://dorokhovich.com/blog/local-k8s-debugging-and-observability?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-debugging-and-observability

Local Kubernetes Dev — Part 12: Debugging and observability

Metronom — Wed, 02 Sep 2026 11:09:18 +0000

"Pod's in CrashLoopBackOff, but kubectl logs is empty" — sound familiar?

The logs are empty because you're looking at the freshly restarted container, which hasn't written anything yet. The real cause of the crash is in the logs of the previous, already-dead instance. And you pull them with one flag many people don't know about:

kubectl logs -p -n myapp

In the new article (part 12) I've collected the essential debugging toolkit — and one idea worth remembering forever: almost any problem in Kubernetes unwinds along the chain events → describe → logs.

I cover:
• kubectl logs with -f, --previous, --since, --tail and the multi-container trap;
• kubectl describe pod and reading the Events block from the bottom up;
• why CrashLoopBackOff and ImagePullBackOff are NOT pod phases (the phase stays Pending) but status fields assembled by kubectl;
• port-forward and exec to check the app from the inside (and kubectl debug for distroless with no shell);
• k9s — the whole cluster in one window;
• the three pillars of observability and an honest warning: don't drag a full Prometheus stack into local.

Read it and stop fearing red statuses: https://dorokhovich.com/blog/local-k8s-debugging-and-observability?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-debugging-and-observability

The third level of failure — when the tool says ok and lies

Metronom — Thu, 27 Aug 2026 20:42:20 +0000

MCP defines two ways for a tool call to fail. There is a third, no protocol can catch it, and it cost me twelve articles.

On 25 July 2026 I audited every entry in my publishing calendar that carried a published URL — twenty-five of them. Twelve live posts on Medium were truncated: between 41% and 72% of the original text had survived, and some had lost their code blocks entirely. A thirteenth URL answered with HTTP 410. Every one had been reported as a success, with a valid URL, weeks earlier.

Here is the mechanism. My publishing server puts drafts on Medium by driving a signed-in browser, because Medium's write API has been left to rot. The adapter waited for the address bar to change to the new draft's edit URL, treated that as done, and closed the tab. Medium, meanwhile, saves the editor's contents incrementally, over background requests that had not finished. Everything that had not reached the server by the time the tab closed was lost, silently, and the adapter returned ok. There was no failed request to catch: the page that would have made it no longer existed.

Now the part I got wrong. For weeks I read this as "browser automation is unreliable and APIs are reliable". That is not what the data says. dev.to came through completely intact — and so did two other platforms that I drive through exactly the same browser, over exactly the same debugging protocol, in the same run. The dividing line was never browser against API. It was one submission against many: those platforms hand over the whole text in a single POST with nothing left to arrive afterwards, while Medium accepted it in instalments and my adapter stopped watching after the first one. The same fact explains the survivors: short announcements of around 1,500 characters finished saving before the tab closed. It was a race, and short texts won it.

And the reason nobody noticed for weeks is the most ordinary reason there is. The check that existed asked whether the URL was alive. It was. A truncated article is a perfectly healthy web page: 200, renders, title, hero image, three paragraphs, and then it simply stops.

The second bug, which is better than the first

The adapter also had to notice when Medium's editor failed to save, so it searched the page text for the phrases Medium shows on failure — one of which is "something is wrong". It searched document.body.innerText, which on that page includes the article being published.

One of the articles contained the sentence "if something is wrong you flip back".

A successful run was declared a failure, the retry ran, and a duplicate draft appeared. The rule generalises well past browsers: never read status out of prose. If the only way to learn an outcome is to hunt for words in running text, you have guessed it rather than learned it, and prose will eventually contain your keyword by accident.

Why the protocol cannot help

MCP's two levels are a good pair. A protocol error means nothing ran — no such tool, arguments that failed the schema. An execution error means the tool ran and the news is bad, and it comes back as a successful response carrying a flag, in text, because the recipient is a model in the middle of a task and can act on it.

Neither level sees this. From the protocol's point of view the call went perfectly: request sent, well-formed response, no error flag, a URL inside. The flag that reports failure is set by the same code that was wrong about what happened. A protocol can carry a verdict; it cannot audit one.

There is a bill attached, too. On that request the model paid for twenty tool descriptions, paid again for the call, received ok, and moved on. The spend was real. The work was not done.

What replaced the check

The check that the work happened has to be separate from the tool that did it, and it has to ask a different question: not "did the call return" but "is what is now on the server the same as what I sent".

Mine reads every published article back and diffs it against the source, rendering each URL in the signed-in browser that owns it, because Medium blocks anonymous fetches and drafts are private to their account. It compares two signals: the number of code blocks, and whether the text of the last link is present — in these articles the last link sits in the closing "further reading" block, which makes it the first casualty of truncation. It has a second mode that opens everything in a profile signed in to nothing, because an author always sees his own article in full, so a paywall or a login wall is invisible from the account that published it.

The full audit table, the script, and the fix that now refuses to return a URL if less than 98% of the text landed on the server are on my own site: https://dorokhovich.com/blog/mcp-false-success — and the third article of the series, on what an MCP server is made of, is at https://dorokhovich.com/blog/how-mcp-works

Source: my own publishing server and its runbook, audited 25 July 2026. One server, one platform, one bug, a sample of twenty-five. It is not evidence about AI agents in general and I have no money figure to give you, because I never calculated one.

One real question, and I want the disagreement rather than the agreement: does your post-deploy check ask whether the thing is reachable, or whether it is correct? If you think the distinction does not apply to what you run, tell me what your check compares against, because mine compared against nothing and looked green for weeks.

Every MCP revision, and what each one took back

Metronom — Thu, 20 Aug 2026 08:55:34 +0000

MCP is twenty months old and has had five specification revisions. I put them all in one table because the shape of the list says more than any single entry does.

Read the removals, not the additions. Batching arrived in March 2025 and was gone by June, before it was three months old, which tells you it was never load-bearing: nothing had been built on it, so the specification was free to admit the mistake. Sessions and the opening handshake went on 28 July 2026, and that is a different kind of news entirely. A capability deleted in month seven is a correction. A rework of the foundations in month twenty means the foundations were wrong.

Revision	Added	Taken back
2024-11-05	JSON-RPC; stdio and HTTP+SSE; tools, resources, prompts	—
2025-03-26	OAuth 2.1; Streamable HTTP; batching	HTTP+SSE
2025-06-18	Structured output	Batching, under three months old
2025-11-25	Icons; schema defaults; JSON Schema 2020-12	Nothing the revision list records
2026-07-28	Capability-listing call; "input required"; response cache hints; routing headers	Sessions; the handshake; stream resumption. Deprecated with a year of grace: server-to-model calls, folder access, protocol-level logging, dynamic client registration

The 2026-07-28 row is the one worth arguing about. Removing sessions made MCP stateless, which is Fielding's second REST constraint, adopted twenty months late by a protocol that had started from the opposite premise. Everything else in that revision follows from it: response cache hints, and headers carrying enough routing information that a proxy can direct a request without parsing the body. These are the features you add when your protocol has to survive contact with ordinary infrastructure.

My reading, and you are welcome to disagree with it: the reversal is late and right. Sessions in a protocol living between two processes owned by different people were a mistake from the beginning. The two ends are deployed, restarted, scaled and upgraded on separate schedules, and a shared memory between them is a liability one side always ends up holding.

Nothing broke on 28 July. The revision came out as a Release Candidate, the SDKs are in beta, and everything deprecated has a year of grace. The practical consequence is not "wait", it is "budget for rewriting".

Source: modelcontextprotocol.io, revisions 2024-11-05, 2025-03-26, 2025-06-18, 2025-11-25 and 2026-07-28, plus the official changelog for the last of them. I keep this table current as revisions land, with per-revision anchors so you can link to a row rather than to a page: https://dorokhovich.com/blog/mcp-revision-history

One real question: is there a row here you would have filled in differently? The 2025-11-25 line in particular says "nothing the revision list records", and that is a claim about a changelog rather than a line-by-line diff. If you have diffed it, tell me what I missed.

Hacker News + dev.to (a concrete '2-5 minutes to a few seconds' DX win is a proven front-page hook)

Metronom — Wed, 19 Aug 2026 08:59:47 +0000

We got the Kubernetes inner loop from 2–5 minutes back to seconds with Tilt

Chapter 8 of a local-Kubernetes series — the payoff that closes the loop chapter 1 opened. Tilt watches files, rebuilds, deploys, and with Live Update syncs code into a running container in seconds.

Key takeaways

The pain, quantified: by hand, every edit is docker build → push → kubectl rollout restart → wait → check logs — a minute or two each. Ten edits = half an hour gone.
A Tiltfile is a program, not YAML — it's Starlark (a Python dialect). tilt up executes it top to bottom; edit the Tiltfile and Tilt re-executes it, nothing to restart.
The three pillars: docker_build('tag','ctx') (how to build), k8s_yaml('file') (what to deploy), k8s_resource('name', port_forwards=...) (fine-tuning). Tilt matches built images to manifest images by tag and swaps in the fresh build.
Live Update is the star: instead of rebuild+redeploy, Tilt copies changed files straight into the running container. Steps run in a strict order — fall_back_on → sync → run → restart_container — and the order can't be broken.
Two newbie traps: sync paths must live inside the same docker_build context ("if Tilt is watching it, you can sync it"); run() can't come before sync(). Also: the first deploy is always full — Live Update needs a running container.
Hot reload, two scenarios: (A) framework reloads itself — uvicorn --reload --reload-dir app, sync is enough; (B) no reload — use the restart_process extension (docker_build_with_restart), the modern replacement for the deprecated restart_container() on k8s.
Dashboard at localhost:10350: per-resource update + runtime status, logs with filtering, endpoints as links, manual Trigger Update, optional TRIGGER_MODE_MANUAL.
vs Skaffold/DevSpace: Tilt = UI-first + Starlark + Live Update; Skaffold/DevSpace = CLI-first + YAML. Fair caveat: pick by team taste.

Full article: https://dorokhovich.com/blog/local-k8s-tilt-fast-dev-loop?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-tilt-fast-dev-loop

Local Kubernetes Dev — Part 8: Tilt — a fast development loop

Metronom — Mon, 17 Aug 2026 09:41:15 +0000

Remember the loop from chapter one that stretched from seconds to 2–5 minutes on every change? This is the chapter where we bring it back down to seconds.

Part eight of the series is the emotional climax. Tilt orchestrates your inner loop: it watches files, builds the image for you, deploys to the cluster for you, and — crucially — can update code right inside the running container in seconds. Plus a web dashboard where all your services, statuses, and logs live on one screen, no juggling ten terminals.

The three pillars of any Tiltfile: docker_build (how to build), k8s_yaml (what to deploy), k8s_resource (fine-tuning — port forwarding). The magic is that Tilt matches images from docker_build to images in the manifests by tag and substitutes the fresh build. The star of the show is Live Update: instead of "rebuild image → redeploy pod," Tilt copies the changed files straight into the live container. With FastAPI it pairs beautifully with uvicorn --reload.

I break down the strict order of live_update steps (fall_back_on → sync → run → restart), two beginner traps (sync only within the build context; run can't come before sync), the no-hot-reload path via restart_process, and an honest comparison of Tilt vs Skaffold vs DevSpace. https://dorokhovich.com/blog/local-k8s-tilt-fast-dev-loop?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-tilt-fast-dev-loop

What GitHub's MCP Server Costs You Before You Ask It Anything

Metronom — Thu, 13 Aug 2026 14:02:21 +0000

TL;DR
GitHub's official MCP server spends roughly 17,600 tokens on tool descriptions before you ask it anything, with every toolset switched on.
The same operation through MCP costs 4 to 32 times what the equivalent command-line call costs.
If you can write the list of calls in advance, write it, and keep REST underneath.

Attach GitHub's official MCP server to an assistant, every toolset switched on, and the conversation opens with a bill of roughly 17,600 tokens. Nobody has asked anything yet. That is the cost of the tool descriptions alone — the names, the sentence explaining what each tool does, the shape of the arguments it expects — and it is paid again on every request for as long as the server stays attached. The figure is a count of the server's own tool schemas, published by people who counted them; it is not a vendor's number.

Written 5 August 2026. Every figure below was measured between late 2025 and July 2026, before the revision of 28 July changed some of the rules they were measured under. The prices move; the arithmetic does not. Measure on your own workload before you plan around them.

I want to be careful with that number, because it belongs to a configuration rather than to a product. The server groups its tools into toolsets you can switch off, and switching the unused ones off cuts the figure several times over. Which is exactly the point: nobody can honestly quote it without saying which toolsets were on. In the measurement above, all of them were.

The "MCP versus REST" argument is almost always conducted as though it were a matter of taste, settled by whichever side sounds more modern. It is not a matter of taste. There is a price list, the numbers on it are fairly harsh, and once you have seen them the choice stops being about architecture and starts being about whether the thing you are building actually needs a model to decide anything.

Where the numbers come from, before anyone asks in the comments: public analyses of the GitHub MCP Server and survey write-ups of context overhead from late 2025 and early 2026, plus the schema-overhead discussion in issue #2808 of the specification repository. Specification revisions used: 2025-06-18 and 2026-07-28. The truncation incident later in this piece is from my own publishing server, audited on 25 July 2026 — that part is not a citation, it is my own damage.

Start with the bill

The bill is denominated in two things, and only one of them is money. A token is the unit a language model is billed in and bounded by. The model here is any LLM you like; the brand changes nothing about the unit. It is a fragment of text — roughly three quarters of a word in English, rather less in Russian, where the same sentence costs more tokens to say. The bound is the context window: the fixed amount of text a model can hold in view at one time, covering the conversation, whatever you pasted into it, and everything the tools have said about themselves. Every number below is a share of that window, spent before the model does anything you would recognise as work.

A model does not "connect" to a tool the way a program connects to a database. It is handed a written description of every available tool, in full, at the start of every request, and it reads that description again each time
GitHub's official MCP server: roughly 17,600 tokens of tool descriptions with the full toolset enabled, before a question is asked. That number belongs to a configuration rather than to the product — the server groups its tools into toolsets that can be switched off, and switching the unused ones off cuts the figure several times over. Which is the point: the number cannot honestly be quoted without saying which toolsets were on
Attach several servers at once — an issue tracker, a browser, a database — and 30,000 tokens and up is ordinary
Estimates from early 2026 put a typical desktop assistant at 40–50% of its context window consumed before the first question. Percentages are the wrong unit here, and I quote them only because that is how the estimates were published. They were published without the configurations they were measured on, which is exactly the thing I have just said makes a number unquotable. The tokens are the fact: 17,600 for one server with everything enabled, past 30,000 once several are attached. A larger context window makes the same descriptions look cheaper without making them cheaper — the share falls, the count does not move, and you pay it on every request
The comparison that sobers people up: the same operation performed through MCP costs 4 to 32 times more tokens than the equivalent command-line call

Four to 32 times is not overhead. Overhead is a percentage. This is a different order of magnitude, and it deserves to be said plainly: for any task you can solve with a script, MCP is not worth paying for. A nightly job that reads a file and posts it to an endpoint needs no tool descriptions, no model reading them, and no bill. The price starts buying you something at exactly the point where you can no longer write the list of calls in advance.

The cost that appears on no invoice

The costs so far can be counted before you spend them. This one cannot, and it appears on no invoice at all. I paid it personally. I run an MCP server with twenty tools that manages my publication calendar and puts articles on Medium, dev.to, Stack Exchange, Hacker News and Telegram. Its Medium adapter drives a signed-in Chrome over the debugging protocol, because Medium's write API has been left to rot. The adapter considered its work finished the moment the browser's address changed to the editing URL of a new draft, and then closed the tab. Medium, meanwhile, saves the editor's contents incrementally, over background requests that had not yet finished. Everything that had not reached the server by the time the tab closed was lost, silently. The adapter returned a valid URL and a success.

The audit on 25 July 2026 covered all 25 calendar entries that carried a published URL. Twelve live posts on Medium were truncated, retaining between 41% and 72% of their text, and some had lost code blocks. A thirteenth URL answered with HTTP 410. Short announcements of around 1,500 characters had survived, because they finished saving before the tab closed. Nothing else was affected: dev.to, Stack Overflow and Hacker News all came through intact. That is the detail worth pausing on, because Stack Overflow and Hacker News are driven through a browser exactly as Medium is. The dividing line was never browser against API. It was one submission against many: those three hand over the whole text in a single POST, while Medium accepted it in instalments and my adapter stopped watching after the first one.

The reason it went unnoticed for weeks is the ordinary reason. The check that existed asked whether the URL was alive, not whether the article was all there. The post loads. The link works. Only the second half is missing.

Two things follow, and neither of them is about Medium. The first is that a tool returning success has told you nothing about whether the work happened. MCP distinguishes two levels of failure — the protocol broke, or the tool ran and reported bad news — and this is a third kind that neither level can see, because from the protocol's point of view the call went perfectly. The second is about tokens. On that request the model paid for twenty tool descriptions, paid again for the call, received ok, and moved on to the next item in its plan. The spend was real. The work was not done. Anonymous statistics about tool calls that never complete get quoted a lot and sourced almost never. I cannot tell you what is behind them. I can tell you what was behind mine, and that nothing will tell you when it is happening to you.

A second bug sat beside the first, and it cost a duplicate draft rather than a truncated article: the adapter decided whether Medium had failed to save by looking for failure phrases in the page text, and one of the articles being published contained one of the phrases. The rule that comes out of that is in the third article.

Why "just add more tools" makes things worse

A task that ran reliably with ten tools starts coming apart at fifty. The instinct is to blame volume — more text, a fuller window, worse answers — and volume is the smaller half of the problem. The real problem is selection. The model is choosing one item from a list, and every item it did not pick is a distraction it had to read and rule out. Two tools whose descriptions differ by a single clause are not two options; they are a coin toss the model performs on your behalf, once per call, silently. Long lists also encourage the model to plan in tool calls rather than in steps, which is how a job that needed three calls turns into eleven.

Tool responses cost more than tool descriptions

The second cost gets discussed less and is frequently larger than the first: the responses. Descriptions are a fixed charge, paid per request. Responses are metered. A call that lists pull requests returns author objects, timestamps, avatar URLs and review metadata that nobody asked for, and all of it lands in the same window as the descriptions. It is entirely normal for a session to spend more context on what the tools said back than on the catalogue of what they could do.

Where REST is still the better answer

Everything in that publishing server except the browser adapters is a plain HTTP call to a REST API, and that half has never lost anything. REST did not get worse when models arrived. It stayed exactly as good as it was, and the cases it was good at are still the majority of cases.

Keep it wherever the calls are known in advance and made by a program rather than chosen by a model. If you can enumerate the calls while writing the code, enumerate them: you get a compiler, a code review, a stack trace and a test suite, none of which a model's choice of tool will ever give you. High volumes of similar requests are the other obvious case — a quarter of a century of HTTP caching, ETag, CDNs, load balancers and per-key limits are sitting there, built and debugged by other people, and none of it knows what an MCP server is. A public interface has its own reason to stay a REST API: outside developers want stability, versioning and an OpenAPI description they read once at integration time, not flexibility to be rediscovered on every call. And there is cost. A REST call has a number you can put in a spreadsheet; an MCP call costs whatever the user happened to have attached that day.

The simple rule holds up better than any diagram: if you can write the list of calls in advance, write it. An MCP server sitting inside a scheduled job, with no human in the loop and nothing for a model to decide, is a very expensive HTTP client.

Where MCP earns its keep

The price buys one thing, and it is the thing that makes an AI agent an agent rather than a script: the ability not to know, at the time you write the program, what the program will be asked to do. Where that ignorance is genuine, the bill is fair. Where it is a habit of mind, you are paying to avoid making a decision.

The fork, in practice

Two questions decide it, and they have to be asked in order.

The first is who makes the call: a programmer, or a model? If a programmer, the argument is over before it starts. You have documentation, types, a compiler and a review process, and none of the machinery MCP exists to provide is any use to you. Make the HTTP call.

The second question only applies if the answer to the first was "a model": is the list of actions known when you ship? If it is known, small and fixed — three actions, always the same three — then function calling against your own API will get you most of the way there. You describe the three functions to the model directly, in the request, and skip the protocol between the parts entirely. Model Context Protocol starts paying for itself when the set is genuinely open: when the user attaches what they need, when a server you have never seen announces its tools at runtime, when the next step depends on what the last one returned in a way you cannot enumerate ahead of time.

In practice the answer is almost always "both", and not as a diplomatic compromise. It is what the systems look like. The MCP server I run is a wrapper — five hundred lines of tool definitions, and about as much again in the adapters and verifiers behind them — sitting in front of the same admin API the web interface uses. The endpoints are identical; only the authentication differs — a session cookie for the browser, an API key for the server. Nothing in the calendar was redesigned to be MCP-shaped, and nothing needed to be. What the wrapper adds is exactly the set of things a model needs and a program does not: descriptions written to be read rather than parsed, a deliberately narrower set of actions than the API exposes, and a point where a human can look at what is about to be sent and say no.

So the honest summary is MCP on the outside, REST underneath. It is worth stating the inverse too, because it is the mistake I see most often: if your design has MCP underneath as well — services calling each other over it, batch jobs speaking it, a scheduler holding a tool catalogue — you have bought the price without buying the reason.

How to cut the MCP token bill

Four mechanisms exist, and one habit beats most of them.

Tool search: moving the catalogue out of the request

Tool search shipped in late 2025 as part of Anthropic's advanced tool-use release, and it moves the catalogue out of the request: the client keeps the full list, and the model is handed only the tools that match the task in front of it. The reduction in startup cost reported in that announcement is 95%. I do not believe that as a general rule. The figure is true for setups that had far too many tools attached to begin with, which means it measures somebody's earlier mistake rather than the effect of the mechanism. Run it against a considered set of fifteen tools and you will not see anything like 95%, because there was never 95% of waste there to remove.

Code execution instead of chains of tool calls

Executing code instead of chaining calls is the interesting one, and the only one of the four that attacks the larger cost. Instead of asking the model to make eleven calls and read eleven responses, you let it write a short program that makes the calls itself. The intermediate data — the four-thousand-line list it filtered down to three rows — never passes through the context window at all. Only the three rows do. Since responses are usually where the money goes, this is where the money comes back.

Sub-agents do not save tokens

Sub-agents are worth stating carefully, because the claim is usually made backwards. Sub-agents do not reduce total token spend. They generally increase it: each one re-reads its own instructions and its own slice of the tool catalogue, and the coordinator pays again for the summaries coming back. What they protect is the coordinator's context, which stays clear of forty tool descriptions and thousand-line outputs it will never need. That is a real benefit and often worth the money. It is not a saving, and budgeting for it as one is how a bill doubles.

Cacheable tool lists (2026-07-28)

Cacheable tool lists came with the 2026-07-28 revision: a hint from the server saying, in effect, that the client need not ask for this again yet — a direct consequence of the protocol becoming stateless. Mechanical, modest, and most useful in exactly the setups where descriptions dominate the bill.

The cheapest fix: fewer tools, shorter descriptions

Then there is the discipline that requires no technology whatsoever, works today, and outperforms everything above: fewer tools, shorter descriptions, filtered responses. Fifteen tools with one-line descriptions, returning only the fields somebody actually reads, will beat every mechanism in this section, and you can do it this afternoon without waiting for a specification revision or an SDK. My own version of it is not a mechanism at all: a deliberately narrower set of tools than the API exposes, and three verifiers that exist only because the caller is a model.

Two numbers are worth carrying out of this. 17,600 tokens on every request, for one server with everything switched on; and 4 to 32 times the cost of the equivalent command-line call. Neither is an argument against MCP. Both are an argument for knowing, before you attach anything, which decision you are paying for the right to postpone. If you already know the answer, you have your list of calls, and you should write it. (You will not find "MCP versus REST" speed benchmarks here. They measure implementations rather than protocols, and the number that comes out says more about the server somebody tested than about the choice in front of you.)

This is the second of three articles. The first is about where MCP and REST came from and why the July 2026 revision turned MCP back toward REST; the third is about what an MCP server is actually made of.

The three articles this comes from are on my own site: where MCP and REST came from, this one on what MCP costs in tokens, and what an MCP server is actually made of.

One real question, and I want the disagreement rather than the agreement: have you measured what your own tool catalogue costs per request? Not the vendor's number for their server — yours, counted on your own descriptions. I have not published mine, and that is a fair thing to hold against this article: I run a server with twenty tools and I have quoted somebody else's figure rather than my own. If you have counted yours, I would like the number and the method, because the method is the part nobody publishes.

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

Metronom — Wed, 12 Aug 2026 12:32:08 +0000

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

Most Kubernetes ImagePullBackOff / CrashLoopBackOff troubleshooting is not deep. The statuses read like spaceship error codes, but nearly all of them fall out of three commands and a short list of causes. This is the playbook I run on autopilot, plus the one k3d trap that cost me a morning.

The golden set

Whatever the status, start here. Same describe / logs / exec loop the Kubernetes 'Debug Running Pods' guide recommends.

# 1. State, Reason, and the Events block at the bottom — ALWAYS first
kubectl describe pod <pod> -n myapp

# 2. App logs, and the CRASHED container's logs via --previous
kubectl logs <pod> -n myapp
kubectl logs <pod> -n myapp --previous

# 3. Namespace timeline
kubectl get events -n myapp --sort-by=.lastTimestamp

describe hands you three fields: State, Reason, and Events. That's the loop. The rest is pattern-matching. There's a solid field guide to these same statuses at this troubleshooting writeup.

ImagePullBackOff — and the k3d containerd trap

Symptom

ErrImagePull first, then ImagePullBackOff after backoff. Pod stuck Waiting. Events line usually says it outright: not found / unauthorized / no such host.

Root cause

The boring causes are enumerated in the Kubernetes images docs: typo in name or tag, private registry with no imagePullSecrets, Docker Hub rate limit (toomanyrequests), bad registry address.

The trap that isn't boring: k3d nodes run on containerd, isolated from your Docker daemon. The image you just built with docker build — visible right there in docker images — the cluster cannot see. Docker has it; containerd doesn't. ImagePullBackOff, even though "the image is obviously built."

The fix

Two correct ways to get the image into the cluster, both in the k3d image-import docs:

# Option 1: import into the nodes directly
docker build -t myapp:dev .
k3d image import myapp:dev -c dev

# Option 2 (recommended): a local registry created with the cluster
k3d cluster create dev --registry-create k3d-registry.localhost:5000

# then the FULL name matters: address + port + tag
image: k3d-registry.localhost:5000/myapp:dev

The classic follow-up mistake is an incomplete image name — myapp:dev instead of k3d-registry.localhost:5000/myapp:dev. The cluster looks in the wrong place and you're back in backoff. If you use Tilt, docker_build handles delivery, but the Tiltfile image name must match the manifest exactly or you get the same error.

CrashLoopBackOff — the status that hides its own cause

Symptom

Container starts, crashes, kubelet restarts it, it crashes again — with exponential backoff, roughly 10s → 20s → 40s, capped at 5 minutes. The status itself tells you nothing.

The fix

The command I wish I'd known on day one:

kubectl logs <pod> -n myapp --previous

Without --previous you get the freshly-started container, which usually hasn't written anything. --previous pulls the stack trace from the instance that actually died. Also read Last State: Terminated and its Exit Code in describe.

Causes I check in order:

Startup bug — exception, non-zero exit. For a FastAPI service, missing DB_HOST / DB_PASSWORD crashes it on connect.
Missing env/config — a ConfigMap/Secret isn't mounted.
Dependency not up — Postgres hasn't started and the app doesn't wait.
OOM — out of memory (below).
Liveness too strict — kills the app before it warms up. Fix with initialDelaySeconds or a startup probe.
Exit code 0 — for a long-running service, "exited successfully" is still a crash loop. Usually a wrong entrypoint that runs and exits instead of starting uvicorn.

When the container dies instantly and logs are empty, keep it alive and climb inside:

# temporary, in the Deployment
command: ["sleep", "infinity"]

kubectl exec -it <pod> -n myapp -- sh
env | grep DB_
uvicorn app.main:app --host 0.0.0.0 --port 8080   # run by hand, read the real error

Pending — the scheduler couldn't place it

Symptom

Pod never leaves Pending. describe → Events → FailedScheduling, usually naming the cause: Insufficient cpu / Insufficient memory.

Root cause

On a small local cluster this is the most common one. I once set requests: 4 CPUs on a 2-CPU k3d VM and the pod simply never scheduled. Other causes: nodeSelector/affinity mismatch, taints without tolerations, an unbound PVC, a taken hostPort.

The fix

Lower requests, fix the selector/PVC, or grow the cluster:

k3d node create extra --cluster dev --role agent

For a local FastAPI service, 100m CPU / 128Mi memory is fine. Don't paste production numbers into a laptop cluster.

OOMKilled — the exit-137 signature

Symptom

Process blew past its memory limit and the kernel SIGKILL'd it. Recognize it by exit code 137 (128 + 9, SIGKILL) and Reason: OOMKilled in describe. That Reason field distinguishes a memory kill from other SIGKILLs, and OOM is often the hidden cause behind a CrashLoopBackOff.

Root cause

Two flavors:

Container-level — exceeded resources.limits.memory. Raise the limit or fix a leak.
Node-level (sneaky on k3d) — the whole Docker VM ran out of memory, so pods get OOM-killed even while each app is within its own limit. The fix isn't Kubernetes limits; it's giving the Docker VM more memory, or not overcommitting — the sum of all pods' limits shouldn't exceed the VM's memory.

The Service that's green but silent

Symptom

Pods Running, everything looks fine, requests never reach the app. Almost always one of three breaks in Service → endpoints → Pod.

The fix

Check endpoints first:

kubectl get endpointslices -n myapp -l kubernetes.io/service-name=myapp

Empty means nothing is behind the Service. Three causes:

Selector mismatch — labels are case-sensitive, app: Web ≠ app: web.
Port mismatch — targetPort must equal the real containerPort.
Readiness not passing — pod shows 0/1 Ready, Kubernetes pulls it from endpoints so no traffic flows, but the container doesn't restart and still reads Running.

Localize by hitting the pod directly, bypassing the Service:

kubectl port-forward <pod> -n myapp 8080:8080
curl http://localhost:8080/healthz   # works? then it's selector/endpoints/readiness, not the app

Before / after

Status	Old reflex	The move
ImagePullBackOff	re-`docker build`, panic	check k3d containerd/registry gotcha; full image name
CrashLoopBackOff	stare at empty logs	`logs --previous` + exit code
Pending	assume cluster broken	read `FailedScheduling`; shrink requests
OOMKilled	confused restarts	spot exit 137 + `Reason: OOMKilled`
Service silent	blame the app	check endpoints/selector/readiness

Guardrail

The whole thing is one algorithm: get pods shows what → describe explains why in Reason/Events → logs --previous gives the crash detail. Ninety percent of cases are on the list above.
Watch disk creep. Kubelet image GC is lazy — it won't clean until the node's disk crosses its high-threshold, 85% by default — so images pile up on purpose. docker system df to see it, prune carefully; docker system prune -a can delete images the cluster still needs.
Keep the golden set in muscle memory. It's faster than any search bar.

What I'd do differently

Stop treating a red status as a mystery. Every one of them resolves to a Reason field and a short cause list. The dread was never warranted — the statuses just have unfriendly names.

Bottom line: memorize describe + logs --previous + get events, learn the five common statuses, and remember that in k3d "the image is built" and "the cluster can see it" are two different facts.

Sources

docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

Metronom — Tue, 11 Aug 2026 19:26:02 +0000

docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

The service passed every local test in docker-compose. The first Kubernetes rollout fell over inside an hour on things Compose had never modeled: readiness, Secrets, startup order, service discovery. This is the docker-compose vs kubernetes dev/prod parity gap, and it cost us an outage before I admitted it existed.

Symptom

First prod rollout. Pod Running, but the Service sent it zero traffic.

$ kubectl get pod -l app=api
NAME                   READY   STATUS    RESTARTS   AGE
api-6c9f8b7d4-2xk9p    0/1     Running   3          2m

$ kubectl describe pod api-6c9f8b7d4-2xk9p | tail
  Warning  Unhealthy  readiness probe failed: connection refused
  ...
$ kubectl logs api-6c9f8b7d4-2xk9p --previous
  sqlalchemy.exc.OperationalError: could not connect to server: Connection refused
  Is the server running on host "postgres" (10.43.x.x) and accepting connections?

Three separate failures, none of which any green Compose test had touched:

No readiness probe existed, so I'd never configured one — Compose has no such concept.
A plaintext env var in Compose was supposed to be a Secret in k8s.
The app crashed reaching the DB before the DB was up. Compose's depends_on had papered over that for years.

Root cause

I'd assumed a Compose file and a set of k8s manifests were the same idea at different fidelity. They aren't. The Twelve-Factor App's tenth factor, dev/prod parity, names the trap: keep the gaps between dev and prod small — especially the tools gap, where "tiny incompatibilities crop up, causing code that worked and passed tests in development to fail in production."

Two different problems live under that one heading, and I'd conflated them:

Backing-service parity. The canonical sin is SQLite locally, Postgres in prod. Sooner or later a behavioral difference bites at the worst moment. Same type, same version, both sides. No substitutions.
Manifest-level bugs. A whole class of failure lives only at the Kubernetes object layer: misconfigured probes, thin RBAC, service discovery by in-cluster DNS, missing resource limits. Compose has none of these concepts, so it can surface none of these bugs.

That second bucket is the one that got me. A single Compose service is not one Kubernetes object — it's several, each carrying its own slice of production behavior:

a Deployment — how the pods run
a Service — how they're reached
a ConfigMap + Secret — config and secrets
an Ingress — external access

Every one of those is a place to misconfigure something, and a place Compose never made me think about. There's a solid breakdown of what "production-like local" actually means at this writeup on production-like local environments.

The fix that didn't work: Kompose

First instinct was to cheat and auto-convert. Kompose is a real project under the Kubernetes org, so I ran it:

# a starting point, NOT a prod manifest
kompose convert -f compose.yaml

Kompose's own authors are honest: "our conversions are not always 1-1... but will get you 99% of the way there." Production lives in that last 1%. What the conversion silently drops or mangles:

depends_on is ignored. Kubernetes has no "start B after A." You use init containers or, better, app-level retries — reconnect to Postgres until it answers.
build: doesn't build. Kubernetes can't build from source. The image must already be pushed to a registry.
network_mode: host and custom networks map poorly or not at all.
Bind mounts vanish. You're expected to use a ConfigMap/Secret.

The kicker: default Kompose output ships without requests/limits, without probes, with env vars in plaintext instead of Secrets. The automation strips out exactly the production-like properties you were trying to gain. A draft, not a deploy.

The fix that worked: a production-like k3d cluster

I switched to running the service in a real local cluster with k3d (k3s in Docker). The discipline that mattered was deciding, on purpose, what to reproduce and what to simplify.

Reproduce, no exceptions:

# pin the SAME minor as prod. Docker tags can't use '+',
# so it's 'v1.31.5-k3s1', not 'v1.30.2+k3s1'
k3d cluster create dev --image rancher/k3s:v1.31.5-k3s1

Same backing services by type and version — Postgres, never SQLite.
Real requests/limits, so scheduler and eviction behavior is realistic.
The exact probe pair that would've saved the first rollout — liveness and readiness: /healthz liveness (process is up), /ready readiness (checks the Postgres connection; no traffic while the DB is unreachable, but no restart either).
ConfigMap + Secret instead of hardcoding; Ingress + in-cluster DNS so routing matches prod.

Same real manifests, deployed the same declarative way as prod:

kubectl apply -k k8s/overlays/dev
kubectl rollout status deploy/api

Deliberately simplify, and write down why: prod scale and topology, managed cloud services (Postgres in a container instead of RDS), and load testing (doesn't belong on the box running the app it tests).

Proving the three bugs are dead, locally

The point of parity is that the failures now reproduce on the laptop. I ran each one on purpose before trusting the rollout:

# 1. startup order: kill the DB, restart the app, watch it retry not crashloop
kubectl delete pod -l app=postgres
kubectl logs -f deploy/api        # "waiting for postgres... retry 3" then ready

# 2. readiness gates traffic: app is up but DB down -> 0/1, no endpoint
kubectl get endpointslices -l kubernetes.io/service-name=api

# 3. secret is wired, not baked into the image
kubectl exec deploy/api -- printenv DATABASE_URL   # from Secret, not Dockerfile

All three passed at my desk. The next prod rollout was uneventful — the first one in that project that ever had been.

Before / after

	docker-compose only	Production-like k3d
Backing service	often SQLite/stand-in	same Postgres + version as prod
K8s version	not modeled	pinned to match prod
Probes	none	liveness + readiness, real
Secrets	plaintext env	ConfigMap + Secret
Service discovery	Compose networking	real CoreDNS / Ingress
`depends_on`	hides startup-order bugs	forces real retry logic
First prod rollout	collapsed on unmodeled objects	boring, matched local

Guardrail

"Works in Compose" validates the app, not the manifests. Manifest bugs only appear after a real rollout — so test in k8s.
Production-like is not "everything like prod." It's an environment where you know exactly what matches, what doesn't, and why. The list of documented simplifications is the most underrated line item on the whole checklist.
You don't need the whole prod zoo. Bring up only the namespace and services you're touching this session. Parity isn't recreating everything at once.
Don't try to copy prod exactly. Copying prod onto a laptop is impossible and pointless. The goal is narrower: shrink the gap between how the service behaves at your desk and how it behaves in prod, so packaging and deployment bugs get caught before the rollout.

What I'd do differently

Keep Compose for the earliest, fastest hacking loop — it still earns its place there. Just never again mistake a green Compose run for validation of a Kubernetes deploy. The moment a service is bound for k8s, it gets tested in a local cluster on pinned versions and the same manifests as prod. My rollouts have since gotten boring, which after that first outage is the highest praise I have.

Bottom line: a Compose file is structurally not equivalent to k8s manifests — pin the same versions and run the same manifests in a local cluster, or the manifest-layer bugs wait for prod to find them.

Sources

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

Metronom — Mon, 10 Aug 2026 08:15:50 +0000

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

Nobody deployed. Nobody ran kubectl apply. At 02:00 the orders API started throwing errors we'd fixed three weeks earlier, and it stayed broken for 40 minutes. The kubernetes latest tag problem is not theoretical. This is the incident that made me delete :latest from every manifest we own.

Symptom

On-call paged. Errors that were fixed weeks ago, live again.

2025-... POST /orders 500  InvalidStateError: order already settled
2025-... POST /orders 500
2025-... POST /orders 500

Deploy history: empty. No release. No apply. No human near the cluster. Running code had changed anyway. First reflex was to blame a rogue actor with cluster creds. kubectl get events and the audit log both said no — nobody had written to the API server in hours. The image had moved without a single Kubernetes write.

Root cause

The Deployment referenced a mutable tag.

containers:
  - name: orders
    image: orders:latest

Timeline, reconstructed from the kubelet events and registry push log:

Someone pushed a mid-refactor build and tagged it latest. Never promoted it.
Hours later a pod OOM'd and restarted.
On restart the kubelet pulled latest again — now pointing at the broken build.
Pod came up on code nobody chose to deploy. Prod changed its own version while we slept.

Three separate properties of latest conspired here, and the Kubernetes image docs warn about all of them:

latest is mutable. It points at a different digest tomorrow. You cannot answer "what is running in prod?"
Re-running apply is a no-op. Kubernetes rolls out only when the pod template changes. The literal string image: orders:latest never changes, so the cluster sees nothing to do even when the registry digest is completely different.
A restart re-pulls. imagePullPolicy defaults to Always for latest (and IfNotPresent for a pinned tag). So latest maximizes the odds of a surprise pull on every restart. The inverse trap is just as ugly: a pinned tag with IfNotPresent can stick on a stale local layer and never pull the fix you just pushed. Both are the same disease — the running image and the name you typed have drifted apart.

There's a solid breakdown of the full local-to-CI path that this class of bug forced on us at this writeup on preparing a local setup for deployment and CI. I'll stick to the fixes.

The fix

1. Immutable tags, generated in CI

Rule: every image is tagged by something that resolves to exactly one build. Short SHA, semver, or a digest pin for maximum determinism.

image: ghcr.io/acme/orders@sha256:45b23dee...

Don't hand-write tags. docker/metadata-action emits them from the git context and writes OCI labels like org.opencontainers.image.revision. "What's in prod?" collapses from an investigation to one label read:

docker inspect ghcr.io/acme/orders:sha-abc1234 \
  --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}'

2. One base, kill the copy-pasted `k8s-prod/`

The other sin: a k8s/ folder copied to k8s-prod/ months earlier. Two copies drift. Ours had, and prod's memory limit no longer matched what we'd tested. Configuration drift, cured by a single base with only the diffs layered on top.

Kustomize — built into kubectl, plain YAML, no templating engine to reason about:

# overlays/prod/kustomization.yaml
resources:
  - ../../base
namePrefix: prod-
images:
  - name: orders
    newName: ghcr.io/acme/orders
    newTag: sha-abc1234    # swap the image without touching the Deployment
patches:
  - path: replicas-patch.yaml

# render and eyeball it before it touches the cluster
kustomize build overlays/prod | kubectl apply -f -

The images field is the quiet win: CI rewrites the tag without editing the Deployment YAML. Helm with a per-env values-prod.yaml lands the same place — just mind -f ordering, last file wins, and swapping the order silently changes your prod replica count.

3. A pipeline that physically cannot ship `:latest`

Push-based and deliberately dumb: commit → build → push → update manifest to the immutable tag → apply. GitHub Actions skeleton, using Docker's build-push-action:

name: build-and-deploy
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/acme/orders
          tags: |
            type=sha
            type=semver,pattern={{version}}
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
      - name: Deploy
        run: |
          echo "${{ secrets.KUBECONFIG }}" > kubeconfig
          export KUBECONFIG=kubeconfig
          kubectl set image deployment/orders \
            orders=ghcr.io/acme/orders:sha-${GITHUB_SHA::7}

The deploy step sets an immutable tag, never latest. Now the pod template genuinely differs, the rollout is honest, and a restart can't re-pull a mystery image.

Reproducing the outage on k3d first

I didn't ship any of this blind. I reproduced the exact failure on a laptop k3d cluster, then confirmed the fix killed it. The whole loop is a few minutes and costs nothing.

k3d cluster create latest-repro
# push two builds to the same mutable tag
docker build -t k3d-registry:5000/orders:latest ./good && docker push k3d-registry:5000/orders:latest
kubectl apply -k overlays/local           # Deployment references :latest
kubectl rollout status deploy/orders

# now poison latest and force a restart, no apply involved
docker build -t k3d-registry:5000/orders:latest ./broken && docker push k3d-registry:5000/orders:latest
kubectl delete pod -l app=orders          # simulate the 02:00 crash
kubectl get pod -l app=orders -o jsonpath='{.items[0].spec.containers[0].image}'

The pod came back on the broken image with an empty deploy history — the outage, on demand. Re-run the same script against the SHA-pinned overlay and the delete/restart brings the same image back every time. That's the whole proof. Once it held, I trusted it in prod.

Before / after

	Before	After
Image tags	`:latest`	immutable `sha-…` / digest
"What's in prod?"	an investigation	one label read
Environments	copy-pasted `k8s-prod/`	one base + overlays
Drift	silent, unbounded	the diff is the only diff
Restart	may pull broken `latest`	pinned, deterministic
Rollout on redeploy	sometimes a no-op	always honest

Guardrail

Grep CI and manifests for :latest in a pre-merge check. No merge if it hits.
Most local Kubernetes work carried straight to prod: same Dockerfile, same Kustomize base, same probes. Prod parity is the payoff. Prod only adds — registry, Secrets, TLS, HPA, PodDisruptionBudget.
Some things must never cross the line: --reload, file-sync dev accelerators, exposed debug ports. The prod overlay is the mechanism that keeps them out.
Pin a digest, not just a tag, for anything you truly can't afford to move — @sha256:... is the only reference in Kubernetes that a registry cannot repoint under you.

What I'd do differently

Move off the push pipeline sooner. It has three built-in weaknesses I already know will bite:

CI holds the kubeconfig. CI access equals prod access.
Drift is invisible. Someone runs kubectl edit by hand and CI never notices.
No automatic rollback. Ship broken, revert by hand, at 02:00.

Those three are exactly what GitOps removes. Per the CNCF OpenGitOps principles, Git is the source of truth and an in-cluster agent (Argo CD or Flux) pulls and reconciles drift. CI's job ends at pushing the image and committing the tag. That's the next rung: CI builds an immutable image → a bot bumps the tag in Git → the agent applies it.

Bottom line: the outage wasn't a Kubernetes flaw, it was a mutable tag plus a copy-pasted folder — pin an immutable SHA and collapse onto one base before your own 02:00 page.

DEV Community: Metronom

You can catch production failures on your laptop: probes, OOMKilled, and zero-downtime in k3d

Takeaways

Local Kubernetes Dev — Part 13: Making your local setup truly production-like

events -> describe -> logs: the Kubernetes debugging chain (and the --previous flag you're not using)

Takeaways

Local Kubernetes Dev — Part 12: Debugging and observability

The third level of failure — when the tool says ok and lies

The second bug, which is better than the first

Why the protocol cannot help

What replaced the check

Every MCP revision, and what each one took back

Hacker News + dev.to (a concrete '2-5 minutes to a few seconds' DX win is a proven front-page hook)

We got the Kubernetes inner loop from 2–5 minutes back to seconds with Tilt

Key takeaways

Local Kubernetes Dev — Part 8: Tilt — a fast development loop

What GitHub's MCP Server Costs You Before You Ask It Anything

Start with the bill

The cost that appears on no invoice

Why "just add more tools" makes things worse

Tool responses cost more than tool descriptions

Where REST is still the better answer

Where MCP earns its keep

The fork, in practice

How to cut the MCP token bill

Tool search: moving the catalogue out of the request

Code execution instead of chains of tool calls

Sub-agents do not save tokens

Cacheable tool lists (2026-07-28)

The cheapest fix: fewer tools, shorter descriptions

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

The golden set

ImagePullBackOff — and the k3d containerd trap

Symptom

Root cause

The fix

CrashLoopBackOff — the status that hides its own cause

Symptom

The fix

Pending — the scheduler couldn't place it

Symptom

Root cause

The fix

OOMKilled — the exit-137 signature

Symptom

Root cause

The Service that's green but silent

Symptom

The fix

Before / after

Guardrail

What I'd do differently

Sources

docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

Symptom

Root cause

The fix that didn't work: Kompose

The fix that worked: a production-like k3d cluster

Proving the three bugs are dead, locally

Before / after

Guardrail

What I'd do differently

Sources

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

The Kubernetes :latest Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

Symptom

Root cause

The fix

1. Immutable tags, generated in CI

2. One base, kill the copy-pasted k8s-prod/

3. A pipeline that physically cannot ship :latest

Reproducing the outage on k3d first

Before / after

Guardrail

What I'd do differently

Sources

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

2. One base, kill the copy-pasted `k8s-prod/`

3. A pipeline that physically cannot ship `:latest`