DEV Community: Billy Walker

Nobody warns you about year two of platform engineering

Billy Walker — Thu, 30 Jul 2026 08:43:31 +0000

Year one of a platform gets a budget, a roadmap, and a launch deck. Year two gets the support queue, the upgrade backlog, and the first awkward realisation that the golden path you shipped six months ago is already drifting away from the estate it was meant to simplify. That is the part nobody budgets for properly: a platform is a product, so shipping it once is the same mistake as shipping any product once.

This is also why so many internal platforms feel promising in the demo and frustrating a year later. The visible work in year one is building: templates, guardrails, self-service workflows, maybe a portal, maybe a paved road people can actually use. The invisible work in year two is keeping the thing alive while the world underneath it keeps moving: Kubernetes releases, Helm chart churn, identity changes, cloud policy updates, framework upgrades, and the steady trickle of support requests from teams whose use case never quite fit the happy path.

Platform engineering does not usually fail because the idea was wrong. It stalls because teams fund the build and underfund the maintenance.

The launch was the easy part

There is a familiar shape to the first year of a platform effort.

A team gets a mandate to improve developer experience, standardise delivery, or reduce ticket-driven toil. It has executive attention. It has a date to hit. It has a story people can understand: we are building the internal platform.

That phase is real work, but it has one advantage: the scope looks finite. You can imagine a first version. You can plan the launch. You can decide what the thinnest viable platform looks like and cut everything else.

Year two is harder because the work stops looking like a project and starts looking like ownership.

Now you are carrying things like:

patching and upgrading the platform's own dependencies
repaving templates and starter repos as upstream defaults change
deciding which one-off exceptions should become product features
pruning workflows nobody uses before they become permanent support burdens
proving that adoption is still growing, rather than assuming it is

None of that produces the same tidy before-and-after slide as the original launch. It is still the work that determines whether the platform becomes part of how the organisation ships software or just another ambitious internal system people route around.

That is why Team Topologies keeps insisting on platform as a product rather than platform as a shared tools pile. The point is not that platforms need prettier language. The point is that products have ongoing discovery, maintenance, and evolution. A platform that is not continuously evolved is just a frozen opinion about how engineering used to work.

Most of the field is only just getting to year two

One reason there is so little useful day-two writing is that the discipline itself is still young.

The 2024 State of Platform Engineering report from Humanitec and Gitpod, based on a vendor-sponsored survey of platform teams, found that 56% of respondent teams were less than two years old, and only 13% had been doing platform engineering for more than five years. That is enough to explain why there is still so little hard-won writing about the maintenance phase: a lot of the field is only just arriving there.

In other words, there is not a deep institutional playbook here. There are patterns. There are good principles. There is some solid research. But there is also a lot of collective improvisation.

That matters because year-two problems do not look like year-one problems.

The first year asks whether you can build a paved road. The second asks whether you can keep repaving it while traffic is already on it.

The first year asks whether you can make self-service possible. The second asks whether the self-service path still fits the organisation as it changes.

The first year asks whether the platform team can ship. The second asks whether it can avoid becoming the bottleneck it was created to remove.

Golden paths do not stay golden by themselves

A platform's main promise is usually straightforward: give teams a safer, faster default path than stitching everything together from scratch.

That is a good promise. It is also one with a permanent maintenance bill attached.

The paved road starts decaying the moment upstream moves. A base chart changes shape. A cluster policy tightens. A framework introduces a new recommended default. A cloud provider deprecates some corner of the identity model you buried in a template months ago. A service team needs one capability the platform path did not anticipate, so it forks the template "just for now" and never really comes back.

This is where platform credibility leaks away quietly. Not in one dramatic outage, but in a long series of tiny moments where engineers learn that the official path is slightly stale, slightly rigid, or slightly behind the reality of the stack. Once that happens often enough, people stop asking whether they should use the platform and start assuming they will need to work around it.

That is also why the advice in Stop copying big-tech platform architecture: right-size to your scale matters even more in year two than in year one. Over-built platform machinery does not just cost more to launch. It costs more to keep current. Every extra control plane, abstraction layer, and bespoke workflow becomes another thing that needs tending after the excitement has worn off.

A golden path without a gardener is just a future migration project.

The upgrade treadmill is not a side quest

If you want one brutally practical example of year-two reality, use Kubernetes itself.

Kubernetes now ships three minor releases a year, and the project maintains patch support for only the most recent three minor versions. Under the yearly support policy that works out to roughly a 14-month window per release: twelve months of standard patches, then two months of maintenance mode before end of life. If your platform sits on Kubernetes, the upgrade tax is not occasional. It is structural. You are budgeting for two to three cluster upgrades a year, forever, before you even count the other components attached to the platform.

And of course it is never just Kubernetes.

There is ingress or Gateway API, cert management, policy tooling, observability agents, CI runners, secret syncing, identity plumbing, image build tooling, and whatever home-grown wrappers your platform created around those things to make them usable. One upstream change can ripple through templates, documentation, tests, and support expectations all at once.

Teams often talk about this as if it were operational overhead adjacent to the roadmap. It is not. For a platform team, the upgrade treadmill is part of the roadmap. If you do not plan for it explicitly, it eats the roadmap anyway.

This is the part that breaks year-two planning. A team thinks it has funded one staff engineer to "keep the lights on" while everyone else moves on to the next strategic thing. Then the calendar turns, the supported version window narrows, and suddenly the strategic thing is an upgrade programme whether anyone likes it or not.

Adoption stalls when nobody is listening

There is a second year-two trap that looks softer but is just as destructive: the team assumes that shipping the platform was the same thing as winning adoption.

It is not.

DORA's 2024 Accelerate State of DevOps report is useful here because it is more honest than most of the hype. It found that internal platforms were associated with better individual productivity and team performance, but also with weaker throughput and change stability where the implementation added complexity or reduced developer independence. DORA describes this as a J-curve: you can see early gains, then hit a dip, and only recover if the platform matures in a user-centred way.

That finding lands because it matches what many teams feel in practice. A platform can absolutely remove toil. It can also create a new layer of process, abstraction, and waiting if the product discipline is weak.

The warning signs are usually familiar:

teams need tickets for routine changes the platform was meant to self-serve
exceptions pile up because the happy path is too narrow
the platform roadmap is driven by internal architecture debates rather than user pain
the team measures launches and features, but not adoption or abandonment

This is why Team Topologies talks about sensing user needs rather than just building services for them. The thinnest viable platform is only a starting point. If you stop there, the platform freezes while the organisation keeps changing around it.

Puppet's 2024 State of DevOps report makes the same point from a different angle: 52% of respondents said a product manager was crucial to a platform team's success, with a further 21% calling one nice to have. That is not a plea to add ceremony. It is a recognition that somebody needs to own adoption, prioritisation, and the discipline of learning what the users of the platform actually need next.

Without that function, year two becomes a familiar kind of drift. The platform team keeps shipping things. The users keep needing different things. Everyone is technically busy, and the gap still widens.

This is how the platform team becomes the bottleneck again

Platform engineering is often sold as the way out of ticket Ops. Build the paved road, standardise the sharp edges, push routine work into self-service, and let product teams move without waiting for a central platform group.

That is the right goal. But an under-staffed year-two platform team can relapse into the exact behaviour it was supposed to replace.

You see it when every unusual request becomes a handoff. You see it when upgrades consume so much attention that user-facing platform work slips for a quarter. You see it when the team starts protecting itself with queues and process because it no longer has the slack to evolve the underlying system.

At that point the platform still exists, but the interaction model has changed. The platform is no longer a capability. It is a department.

And yes, that sounds harsh. It is still a useful test. If the main experience of the platform for most engineers is waiting for another team, something important has gone wrong even if the underlying architecture is perfectly respectable.

What year-two funding actually looks like

If the first-year mistake is treating the platform like a one-off delivery project, the year-two fix is not mysterious. It is just less glamorous than the original launch.

A serious year-two platform budget needs to fund four things explicitly:

maintenance as first-class work: upgrades, repaving, dependency churn, template hygiene, and pruning stale paths
product management: somebody accountable for adoption, prioritisation, and the sensing loop with platform users
operational slack: enough capacity that support and upgrades do not consume every sprint
evidence: metrics for adoption, developer independence, and the support burden the platform is actually removing

That last point matters because year-two platform conversations often become strangely qualitative. People can tell that the platform feels helpful or painful, but the organisation stops short of measuring the signs that matter.

A better year-two review asks questions like:

how many teams are still on the paved path versus a forked one?
which support requests repeat often enough to deserve product work?
how old are the core templates and platform dependencies?
where does a platform user still need a human instead of self-service?
did the last upgrade consume planned capacity, or did it blow up the quarter?

Those are product questions as much as technical ones. Which is the point.

The real work starts after the applause

Platform engineering year two is where the discipline becomes real. The launch matters, but the launch is only the opening move. After that comes the unglamorous part: keeping the path current, paying the upgrade tax, listening hard enough to notice when adoption is flattening, and funding the platform team as owners of a product rather than custodians of a project.

If you get that part right, the J-curve recovers and the platform becomes genuinely compounding infrastructure for the organisation. If you get it wrong, the platform slowly turns back into the queue it was meant to remove.

Nobody warns you about year two because year one is easier to tell as a story. Year two is mostly maintenance, judgement, and sustained ownership. It is also the part that decides whether the platform was ever really a platform at all.

Observability in the age of LLMs: tracing what your AI actually did

Billy Walker — Tue, 28 Jul 2026 08:24:40 +0000

A feature built on an LLM can hit every familiar SLO and still be failing in the way that matters. The API is up. The median latency looks fine. Error rate is flat. Meanwhile the model is taking the scenic route through three tool calls, burning tokens, and returning answers you would not want a customer to trust. The thing you need to see is not just whether the service is healthy. It is what the system actually did.

That is why LLM observability is best thought of as an extension of ordinary tracing, not a fresh religion with its own priesthood. An LLM call is a span. An agent run is a span tree. The useful questions are still familiar: where did the time go, which step failed, what changed, and how much did this request cost? What changes is the shape of the work. Now you also care about token usage, model choices, tool execution, and whether the answer was any good.

If you already run OpenTelemetry, that is good news. You do not need to throw away your tracing model to observe AI features properly. You need to extend it.

Why classic observability stops short

Traditional service telemetry assumes that the interesting failures are mostly about infrastructure or request handling: timeouts, retries, saturation, bad downstreams, deployment regressions. Those problems still exist in LLM-backed systems, but they are no longer the whole story.

A generated answer can be wrong while every system metric stays green. A retrieval step can bring back unhelpful context without producing an obvious exception. An agent can choose a slow tool path, or call the same tool twice, or hand the model a bloated prompt that quietly doubles token spend. Keep the old graphs; just stop expecting them to catch this class of failure.

OpenTelemetry made the same point in its 2026 GenAI observability post: every application call to an LLM hides "a chain of model calls, tool invocations, and token exchanges" behind the scenes. That is the part standard web-service monitoring does not naturally reveal. When a request takes 45 seconds, the question is no longer just "which service was slow?" but "which decision inside the trace made it slow?"

This is the same reason fast feedback loops matter more when LLMs write your code. When generation gets cheaper, validation becomes the scarce resource. In production, observability is one of those validation loops.

The debugging primitive is still the trace

There is a mild industry temptation to treat LLM systems as too exotic for the tooling we already have. That is usually the wrong instinct.

A useful mental model looks more like this:

agent.run
├── llm.chat openai/gpt-5.6
├── tool.retrieval postgres/pgvector
├── llm.chat openai/gpt-5.6
├── tool.http billing-api
└── llm.chat openai/gpt-5.6

That tree is already enough to answer a lot of awkward production questions:

which step dominated latency
whether the model asked for tools more than once
whether retrieval happened before the answer changed course
whether one provider or model variant behaved differently
where token usage and therefore spend started to climb

None of that needs a brand-new theory of observability. It just means treating model calls and tool calls as first-class trace events instead of leaving them hidden inside application logs or framework internals.

That framing matters because it keeps you portable. If the trace is the source of truth, you can ship it to a general-purpose backend, to an AI-focused product, or to both. If the observability story starts and ends inside one vendor's SDK, you have made your debugging model harder to move.

What the OpenTelemetry GenAI conventions give you

The concrete standard here is the OpenTelemetry semantic conventions for Generative AI. As of July 2026 the gen_ai work has moved into its own dedicated repository, semantic-conventions-genai, and every document in it is still marked Development rather than stable. The new repository does not even have a tagged release yet, so there is no version number to pin against; the honest options are pinning a commit, or recording the last main-repo release that carried the gen_ai conventions before the move. That does not make them useless. It means you should adopt them with your eyes open and expect some movement.

You do not need to memorise every field. The useful thing is the shape of the telemetry the conventions are trying to standardise.

For spans and attributes, the useful core includes fields such as:

gen_ai.operation.name for the kind of model operation
gen_ai.provider.name for the provider
gen_ai.request.model and gen_ai.response.model
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
gen_ai.response.finish_reasons
gen_ai.tool.name for tool execution
gen_ai.agent.name and gen_ai.conversation.id when you need to group larger workflows

For metrics, the current GenAI docs define client-side measurements such as gen_ai.client.token.usage and gen_ai.client.operation.duration.

That is enough to capture the parts of an LLM request you usually end up caring about first: how many tokens went in, how many came out, how long the operation took, which model handled it, and where tool execution sat in the middle.

It is also worth being precise about one thing the conventions do not settle for you: cost. Cost is still best treated as a derived metric rather than a magical standard field. In practice that means joining token counts to your model pricing table and calculating spend from there. If someone promises a universal LLM cost number with no context, raise an eyebrow.

Instrument these first

You can spend a lot of time arguing about evals, guardrails, and which agent framework is fashionable this month. Instrumentation is less glamorous, but it is the bit that keeps the rest honest.

If you are tracing LLM-backed features today, start with these:

One span per model call. Capture provider, model, operation, latency, finish reason, and token counts.
One span per tool call. Retrieval, HTTP calls, database lookups, and internal helpers should sit in the same trace, not in a separate logging universe.
Prompt and response references, not reckless raw dumps. Keep enough data to debug, but be deliberate about privacy, redaction, and storage costs.
A derived cost metric. Token counts are useful; token counts joined to pricing are useful to finance and engineering at the same time.
Quality signals beside the trace. User rating, evaluator score, moderation outcome, or task success should sit close enough to the trace that you can compare bad answers with the path that produced them.

Getting the first span is cheaper than most teams expect. If your stack is Python calling OpenAI, the contrib instrumentation does it with zero code changes:

pip install opentelemetry-distro opentelemetry-instrumentation-openai-v2
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true opentelemetry-instrument python app.py

Note the environment variable. Prompt and completion content is off by default and opt-in, which is exactly the right default: you get spans, models, token counts, and finish reasons without capturing user text, and you turn on content capture deliberately, with redaction and retention already thought through.

That is the practical core. You can add fancier workflow views later.

Experimental is fine. Invisible is not.

Some teams hesitate because the GenAI conventions are still moving. That is a real caveat, but it is not a good reason to keep model behaviour opaque.

The honest trade-off is this:

if you wait for the conventions to feel perfectly settled, you stay blind longer
if you adopt them now, you may need a migration later

I would take the migration.

A moving standard is still much better than bespoke telemetry that only one SDK understands. The conventions already give you a common vocabulary for spans, token usage, and model metadata. Even if a field changes later, the shape of the problem is now visible in your traces instead of trapped inside ad hoc logs.

The practical discipline is boring, which is usually how you know it is right:

record the commit or snapshot of the conventions you instrumented against, since there is no tagged release to pin yet
note that it is a Development-status schema
keep your instrumentation behind a thin abstraction where you can
re-check the docs before each major upgrade

That is manageable. Debugging a production agent system with no trace of its internal choices is much less manageable.

The AI observability tools are mostly OpenTelemetry underneath

Once you start looking around, the AI observability market can seem louder than the underlying technical differences justify.

There are good tools here. Langfuse's own docs position its OpenTelemetry integration around ingesting OTLP traces for LLM observability. MLflow describes its tracing product as OpenTelemetry-compatible and aimed at LLM and agent observability. OpenInference, from the Arize ecosystem, describes itself more plainly still: OpenTelemetry instrumentation for AI observability.

That is the important pattern.

The credible tools are increasingly building on OpenTelemetry rather than asking you to abandon it. What they compete on is everything around the wire format:

workflow visualisation
evaluator and feedback loops
prompt inspection
dataset and experiment tooling
agent-specific views
hosted versus self-managed trade-offs

Those are meaningful differences. They just are not a reason to skip portable traces.

If you already have an observability stack you trust, start there. Send the traces somewhere you can query them. Add a specialised tool when you have a concrete gap: better prompt inspection, easier annotation workflows, richer experiment tracking, or an agent timeline that your current backend does not give you.

What I would not do is buy an AI observability platform before I had first proved that we were actually capturing the right spans.

Agent observability is where this gets interesting

Plain LLM tracing is useful. Agent tracing is where the debugging payoff gets larger.

The difficult production questions are rarely about one completion in isolation. They are about execution paths:

why did this agent choose that tool
why did it call the same tool twice
why did it stop early
why did one branch of the workflow take ten times longer than the others
which step turned a reasonable user request into an expensive one

That is why the recent push toward agent timelines is worth paying attention to. Honeycomb launched its Agent Observability features on 12 May 2026, with an Agent Timeline view that is basically a more opinionated way to walk the same trace tree. That kind of view earns its keep because agent systems are not just slow or fast. They are branching, recursive, and occasionally a bit absurd.

The frontier here is less about inventing new telemetry categories than about making traces easier for humans to read. Multi-agent systems create deeper trees, more spans, and more "why on earth did it do that?" moments. The backend that wins will be the one that makes that path legible under pressure.

What to do this week

If you are shipping LLM-backed features and do not yet have a plan, the sensible order of work is refreshingly unromantic:

instrument model calls as spans
instrument tool calls in the same trace
capture token counts and duration
derive cost from usage and pricing
attach one quality signal you trust
only then decide whether you need a specialised platform

That sequence gets you out of the worst failure mode, which is running a system that can think, call tools, and spend money without leaving behind an explanation.

LLM observability is not separate from good observability practice. It is what happens when the service starts making choices on your behalf. Start with traces. Make token usage and cost visible. Keep the data portable. Then, if you need a shinier lens on top, add one later.

Stop copying big-tech platform architecture: right-size to your scale

Billy Walker — Sat, 11 Jul 2026 07:57:12 +0000

A lot of platform over-engineering starts with a diagram that looks serious. A small team has a few services, a modest growth curve, and perfectly ordinary operational needs, but the design doc still reaches for the full hyperscaler silhouette: Kubernetes everywhere, a service mesh, Kafka, multi-cluster, and a bespoke internal platform before anyone has proved they need one. The problem is not that those tools are bad. It is that they are answers to constraints most teams do not have. Right-sizing platform architecture means copying the principles behind big-tech systems, not the stack itself.

That becomes expensive quickly because complexity is not free. Every extra control plane becomes another upgrade path, another source of pages, another thing a new engineer has to understand before they can safely ship a change. Big companies take on that cost because, at their scale, it often pays for itself. Small and mid-sized teams usually take it on up front and hope the benefits show up later.

They often do not.

Big-tech architecture is the output of big-tech constraints

It helps to say the quiet part out loud. Google, Amazon, and the rest did not end up with sprawling platform stacks because they enjoy making life difficult. They built those systems under pressures most teams will never see all at once:

traffic and data volumes that make single-node or single-database assumptions collapse
hundreds of teams that need to change software independently
regulatory or blast-radius boundaries that force hard isolation
dedicated platform and reliability teams to operate the machinery

If your team has three product engineers and one person who gets the on-call phone, you are operating under a different set of forces. Copying the architecture without copying the forces is how you end up with enterprise-grade overhead and startup-grade staffing.

That is also the practical reading of Conway’s Law. System design follows communication structure. A microservice estate with separate deployment pipelines, cross-cutting policy layers, and multiple control planes assumes an organisation shaped to own those boundaries. If you do not have that organisation, the elegant diagram turns into one team carrying all of the complexity by hand.

Copy the principles, not the implementation

The useful parts of big-tech engineering are usually the ideas underneath the stack choice.

You can borrow those almost anywhere:

reduce cognitive load so engineers can ship without learning every platform detail
improve feedback loops so changes are cheap to validate and rollback
create clear boundaries between services, teams, and operational ownership
standardise the paved road so the safe path is also the easy path
treat cost as a design constraint, not something to apologise for later

What does not travel as cleanly is the implementation detail. Service mesh, multi-cluster, event streaming everywhere, and a bespoke internal developer platform are all legitimate responses to particular thresholds. They are poor defaults.

This is where Dan McKinley’s Choose Boring Technology still lands so cleanly. His line about companies having roughly three innovation tokens has aged well because the underlying point is still right: novelty is expensive, and you should spend it where it creates advantage rather than where it merely creates architecture. “Boring” in McKinley’s sense is not old or unfashionable. It is technology whose failure modes are already well understood.

For platform teams, that is a strong default. If the thing you are building is not your differentiator, buying more operational complexity than you need is rarely a flex. It is a loan.

“You are not Google” is a useful design rule, not a slogan

Oz Nova’s You Are Not Google remains one of the clearest antidotes to architecture cargo culting because it forces a more honest sequence of questions. What problem are you actually solving? Which candidate solutions exist? What historical pressure produced the famous answer everyone keeps citing? Does that answer fit your team, budget, and operational maturity now?

Those questions are annoyingly effective because they make a lot of platform theatre disappear.

A Kubernetes platform is a good example. Plenty of teams do need Kubernetes. Plenty also adopt it because it feels like the proper destination for anything serious. The real question is not “is Kubernetes modern?” It is “what operational problem does this solve for us today that simpler deployment machinery does not?” If the answer is mostly social proof, that is a warning light.

The same applies higher up the stack. A bespoke internal platform can be the right move when many teams need a common workflow and the cost of hand-held support keeps rising. But Team Topologies’ thinnest viable platform is a better starting point than an immediate platform product. Sometimes the right platform is a template, a few scripts, and good docs. It does not have to begin life as an ecosystem.

The Prime Video story is a correction, not a manifesto

The most abused recent example in this space is Prime Video’s write-up on reducing the cost of one monitoring service by more than 90%. The point of that post was not “Amazon abandoned microservices” and it certainly was not “monoliths always win”. It was narrower, and more useful.

One team had built a distributed serverless design around Step Functions, Lambda, and S3. At the workload they were handling, the orchestration overhead had become the cost problem. They moved to a single process running on ECS and passed data in memory instead of via a chain of external services. The result was a cost reduction above 90% for that tool.

That matters because it shows what right-sizing actually looks like in practice. The original design was valid in the abstract. It was just mis-sized for the workload and economics in front of the team. Adrian Cockcroft’s corrective take on the reaction was useful here too: this was a refactoring decision inside one service, not a grand ideological retreat.

The lesson is not “always build a monolith”. It is “distributed systems premiums only make sense when you are actually collecting the benefit”.

Martin Fowler has been making the same point for years. Monolith First is still one of the sanest defaults in software architecture because a modular monolith lets you prove boundaries before turning them into network calls. You can extract a service later. You cannot cheaply un-distribute a badly fragmented system that nobody really needed.

A right-sizing table for platform teams

If you want one decision aid from this post, make it this: treat big-tech patterns as threshold tools, not identity markers.

Pattern people cargo-cult	Right-sized default for most teams	Real signal the bigger pattern may now pay off
Service mesh	Use ingress, `NetworkPolicy`, and straightforward app-level telemetry first. Add mTLS or traffic shaping only when you can name the requirement.	You need per-service identity, consistent mTLS, or progressive traffic control across many teams and many services.
Multi-cluster Kubernetes	Start with one well-run cluster and strong namespace, RBAC, quota, and environment boundaries.	You have hard isolation needs: data residency, materially different blast radii, or control-plane scale limits.
Kafka for everything	Start with a managed queue, an outbox pattern, or even Postgres when throughput and replay needs are modest.	You genuinely need a durable, replayable event log with multiple independent consumers and sustained high throughput.
Bespoke internal developer platform	Start with a thinnest viable platform: templates, automation, docs, and one paved path that removes toil.	Many teams are repeating the same workflow, support demand is compounding, and a platform product will clearly save more time than it costs to run.
Microservices from day one	Start with a modular monolith and explicit internal boundaries.	Teams need independent deployability, boundaries are already clear, and you have the operational discipline to support distributed systems well.

This is also where the DORA research is often flattened into the wrong slogan. High-performing teams benefit from loose coupling and independent deployability. That does not automatically mean “split everything into microservices”. You can get plenty of that benefit from a well-structured monolith, a clean delivery pipeline, and sensible ownership boundaries.

The label matters less than the change cost.

Cost, team shape, and on-call load should be first-class inputs

Werner Vogels’ Frugal Architect material is useful beyond AWS because it frames architecture as a series of trade-offs rather than a hunt for the most sophisticated-looking answer. That sounds obvious, but teams forget it surprisingly easily when a design starts accumulating fashionable parts.

A few questions cut through the theatre quickly:

What does this add to the pager surface area?
Who upgrades it, patches it, and debugs it at 02:00?
What skill set does it require from every new engineer?
What failure mode becomes easier because we adopted it?
What failure mode becomes harder or more likely?
Could a simpler design buy us another year before this complexity is justified?

If those questions make the design feel awkward, that is usually a useful signal rather than conservatism.

This is especially true in platform engineering because platforms quietly multiply costs across the organisation. A hard-to-understand abstraction does not stay local to the team that built it. Every product engineer pays rent on it.

When the bigger architecture is the right call

None of this is an argument for permanent smallness or for avoiding ambitious systems. Some teams absolutely do cross the threshold where the heavier pattern is worth it.

You are probably getting there when several of these are true at the same time:

multiple teams are blocked on each other because deployments cannot move independently
reliability problems are coming from shared blast radius rather than application bugs
platform support has turned into a queue because manual help does not scale
data residency, compliance, or tenancy requirements demand hard technical boundaries
workload volume has made the simpler option materially more expensive or less reliable

At that point, more machinery can be the cheaper choice overall. But it is still better to adopt it deliberately than to inherit it by imitation.

That is the part people miss when they turn “choose boring technology” into a bumper sticker. McKinley’s innovation tokens exist so you can spend them where they matter. The platform question is whether you are spending one on a real constraint or on architecture envy.

The sophisticated choice is often the smaller one

The cheapest architecture that meets your actual constraints is not the amateur option. Usually it is the disciplined one.

Small teams do not fail because they lacked enough control planes. They usually fail because they made delivery too slow, operations too fragile, and ownership too muddy for their actual size. Big-tech architecture can solve those problems at big-tech scale. At smaller scale, it often creates them.

So copy the habits that travel: clear interfaces, good defaults, fast feedback, cost awareness, and platforms that remove toil rather than advertise sophistication. Leave the rest until the numbers, the team shape, and the pager all agree that you have earned it.

Progressive delivery on Kubernetes with Argo Rollouts

Billy Walker — Thu, 09 Jul 2026 19:05:06 +0000

Most teams meet the limits of a plain Kubernetes Deployment at the worst possible moment: mid-rollout, watching a new version march out to every pod while a graph slowly turns the wrong colour. A Deployment rolls pods out carefully enough — it respects readiness, it surges and drains in order — but it has one stubborn blind spot. Once the new ReplicaSet is Ready, it keeps going unless something stops it. There is no built-in way to say “send 10% of traffic here, wait, check an SLO, then continue”, or “only promote this preview stack if the metrics hold up”.

That missing control loop is exactly what Argo Rollouts adds. It keeps the Deployment mental model you already know and layers canary, blue-green, analysis, and automated rollback on top. For platform teams, that quietly turns rollout safety from a runbook someone has to remember into a controller that does it the same way every time. As of Argo Rollouts v1.9.0, that controller can drive percentage-based canaries, blue-green cutovers, and metric-gated promotion without asking you to rip out the rest of your Kubernetes delivery stack.

The gap in a normal Deployment

A Deployment is genuinely good at one thing: swapping one ReplicaSet for another while keeping enough healthy pods around to serve traffic. Readiness probes, maxUnavailable, and maxSurge all pull their weight here. What none of them give you is a first-class sense of whether the new version is actually any good.

That distinction matters most when “pods are Ready” and “this version is safe” quietly stop meaning the same thing. A service can start up cleanly and still come apart under real load. Error rate can sit flat through the first few requests and then climb once traffic arrives in earnest. Latency can drift at the 95th percentile while the average stays there looking reassuring. And a bad release often hurts just one path through the app — which is cold comfort when a straight rollout has already reached every user before anyone has the evidence to hit stop.

Argo Rollouts lives precisely in that gap. It is not a replacement for the Kubernetes scheduler, and it is not a GitOps controller — it does not want either job. It is a rollout controller that takes over the update strategy for a single workload and adds the parts a Deployment never had: promotion steps, pause points, analysis runs, and traffic-shaping integrations.

The mental model: Deployment behaviour, but with a promotion engine

The easiest way to hold Argo Rollouts in your head is “a Deployment with a state machine bolted on”. The Rollout resource still owns ReplicaSets and still reacts to pod template changes, but on top of that it keeps track of where a release currently sits in its promotion flow.

At a high level, the loop runs like this:

A new pod template lands in Git and is applied to the cluster.
The Rollout controller creates a new ReplicaSet.
The strategy decides what happens next: shift some traffic, create a preview stack, pause, or run analysis.
If the checks pass, the controller promotes the release.
If the checks fail, the controller aborts and keeps the stable version serving traffic.

That separation is really the whole idea. Kubernetes carries on running the pods, exactly as it did before. Argo Rollouts is the part that decides whether a new version has earned more traffic yet.

The pieces that matter

As of v1.9.0, the CRDs worth genuinely understanding are Rollout, AnalysisTemplate, ClusterAnalysisTemplate, AnalysisRun, and Experiment. That list looks longer than it needs to be, though — in day-to-day work most teams spend nearly all their time in the first three.

`Rollout`

The Rollout (apiVersion: argoproj.io/v1alpha1, kind: Rollout) sits at the centre of everything. It is deliberately built to be a drop-in replacement for a Deployment, which is why migrating an existing workload is mostly a matter of changing the apiVersion, the kind, and the strategy — the pod template underneath can stay much as it was.

A Rollout can either carry its own pod template or point at an existing Deployment through workloadRef. That second option is the gentle on-ramp: it lets you bring a live workload under Rollouts gradually, rather than rewriting the whole object in one nervous commit.

`AnalysisTemplate` and `AnalysisRun`

AnalysisTemplate describes the check; AnalysisRun is that check actually running during a rollout. The template might ask Prometheus for a success rate, a latency figure, or some service-specific business metric that only your team would think to measure. The run takes the answer and decides whether the rollout carries on, pauses, or aborts.

This is the part that makes Argo Rollouts something more than “rolling updates, but slower”. The controller is not simply counting down a timer between steps — it is waiting for evidence before it commits.

Canary vs blue-green

Argo Rollouts supports both, and it helps to remember that they are answers to different questions.

Canary is about gradual exposure. Raise the traffic a step at a time, watch what happens, then decide whether to keep going.
Blue-green is about a clean cutover. Stand the new version up beside the old one, satisfy yourself it works, then flip the active service across.

Neither is universally the right answer. Canary tends to be the better default for customer-facing services, where the whole point is for a bad version to touch as little traffic as possible before you catch it. Blue-green comes into its own when you want an obvious stable-versus-preview split and the reassurance of a fast switch back.

A concrete canary with analysis

The heart of a real Argo Rollouts setup is rarely the sheer volume of YAML — it is the promotion logic hiding inside it. The example below is deliberately small: a canary that shifts traffic in stages and refuses to promote unless a Prometheus-backed success-rate check agrees.

First, install the controller:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Then define the analysis template:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 5m
      successCondition: result[0] >= 0.99
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(
              rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[5m])
            ) /
            sum(
              rate(http_requests_total{service="{{args.service-name}}"}[5m])
            )

Then wire it into the rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 8
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/example/checkout:v2
          ports:
            - containerPort: 8080
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause:
            duration: 5m
        - analysis:
            templates:
              - templateName: checkout-success-rate
            args:
              - name: service-name
                value: checkout
        - setWeight: 30
        - pause:
            duration: 10m
        - setWeight: 100

The interesting part here is not the syntax — it is the sequence.

At 10% traffic, the rollout deliberately pauses long enough to gather some real request data. The AnalysisRun then measures the service against a threshold you wrote down in advance, and if that metric fails three times it aborts, rather than quietly waving the release on to 30% and then to everyone.

There is one subtlety worth internalising early. Without a traffic-routing integration, setWeight is an approximation drawn from ReplicaSet sizes rather than exact request-level shaping. At low replica counts that is often good enough, but it is not the same thing as a true 10% split. When you genuinely need that precision, wire Rollouts up to a traffic provider and give it stableService, canaryService, and trafficRouting. That is the line between “best effort, by pod count” and “deliberate, by request path”.

Where blue-green is simpler

Blue-green is the model most teams find easiest to explain out loud, mostly because it has fewer moving parts. There is an active stack, a preview stack, and a single moment of promotion between them.

In Rollouts, that shape becomes an activeService carrying live traffic and, optionally, a previewService pointed at the new ReplicaSet before it goes live. The part that makes it genuinely interesting is what you can hang either side of the cutover with prePromotionAnalysis and postPromotionAnalysis:

prePromotionAnalysis has to be satisfied that the preview version is good enough before the switch happens at all.
postPromotionAnalysis keeps watching once traffic has moved across, and can send it back to the old version if the new one starts to degrade.

The field I would commit to memory here is scaleDownDelaySeconds. It defaults to 30 seconds, giving the service selectors and traffic infrastructure time to converge before the old ReplicaSet is scaled away. It sounds like a trivial setting, but it is exactly the sort of thing that separates a rollout which “worked” as far as the controller was concerned from one that quietly dropped traffic down in the dataplane.

How this fits with GitOps

Argo Rollouts and Argo CD get along well, but it is worth being clear that they are doing two different jobs.

Argo CD reconciles the desired state in Git into the cluster. Argo Rollouts decides how a workload that has just changed should be promoted, once that change has landed. The boundary between them is clean, and that cleanness is the point.

In practice it means you do not have to smuggle rollout logic into your CI pipelines or bend your GitOps repo layout around it. Git still holds the desired pod template, Argo CD still applies it, and Rollouts takes over only for the runtime question of whether the new version deserves to keep advancing.

It also means rollback here is operational rather than declarative — a distinction that trips people up more often than you would expect. When a rollout aborts, Argo Rollouts does not go and rewrite Git on your behalf. It scales the release back to the stable version in the cluster, but the repository still holds the bad image tag until someone, or another controller, reverts it. That gap has teeth: if Argo CD is running with auto-sync and self-heal, it can read the aborted rollout as simply out of sync and try to re-apply the very version that just failed. Teams new to the model tend to walk straight past that boundary, and only notice it the first time the two controllers start pulling against each other.

Production reality: the sharp edges

Argo Rollouts demos beautifully and is surprisingly easy to misuse once real traffic is involved. The failure modes that actually bite are rarely exotic — they are the boring, avoidable ones.

Canary percentages are only exact if your traffic layer is exact

A setWeight: 10 step reads as though it means precisely ten percent, but without traffic routing it is really just a pod-count approximation. If your service runs seven replicas, there is no way for the controller to carve traffic into a clean tenth. When exact exposure genuinely matters, reach for a supported traffic manager rather than trusting the arithmetic.

Traffic-routed canaries can overprovision

The moment you add trafficRouting, the replica arithmetic shifts under you. maxSurge and maxUnavailable are no longer the whole story, particularly once setCanaryScale is in the mix, and it is easy to find yourself running more stable and canary pods at once than you had budgeted for during a promotion. That is usually the right trade for safety — just make sure it is a trade you planned for, rather than one you discover on the cluster.

Blue-green is simple, but not cheap

Running blue and green side by side doubles parts of your footprint by design — that is the whole mechanism, not a bug. If you lean on preview environments heavily, keep them short-lived. Rollouts is built for promotion windows measured in minutes, not for quietly maintaining a second long-lived environment that drifts further from the first with every passing day.

HPA needs to target the `Rollout`

If the workload is autoscaled, point the HPA at the Rollout itself — not the Deployment you migrated from, and certainly not an individual ReplicaSet. Get this wrong and you end up refereeing two controllers that hold different opinions about who owns the replica count, usually at the least convenient moment.

When Argo Rollouts is the right call

Reach for Argo Rollouts when the cost of a bad release is high enough that “the pods became Ready” simply is not a strong enough gate to stand behind. In practice that means customer-facing APIs, services with SLOs you actually measure, and teams who already have decent metrics but are still making the promotion call by hand, one deploy at a time.

What it is not is a decoration. Adopting it to make a deployment diagram look more sophisticated is a poor reason on its own. If a service is low-risk, its traffic layer is simple, and rollback is already fast, a plain Deployment may well remain the honest choice.

The real win was never that Rollouts hands Kubernetes yet another CRD. It is that it gives your release process a proper control loop: expose a little, measure what happens, then decide. That single step — pausing to look before committing — is the one a plain Deployment has never been able to take on its own.

EKS node autoscaling with Karpenter: just-in-time capacity without node groups

Billy Walker — Thu, 09 Jul 2026 19:01:51 +0000

Most people arrive at Karpenter after a slow build-up of small frustrations rather than one dramatic outage. Managed node groups start out simple, and then one day you notice you are spending real time deciding which group should be a little bigger, which instance family to add next, and how much headroom to leave "just in case". It starts to feel less like running a cluster and more like tending a spreadsheet.

Karpenter offers a gentler mental model. Instead of pre-sizing groups and hoping one of them fits, it watches the pods that cannot be scheduled, works out the capacity they actually need, and launches EC2 nodes to match. Two things shift as a result. Scaling up follows the workload rather than a template you wrote weeks ago, and scaling down becomes a policy you set once rather than a clean-up job you keep meaning to get to. On a busy cluster, that is often the difference between paying for a permanent buffer and only asking for capacity when there is real work waiting.

How Karpenter changes EKS node autoscaling

The simplest way to describe the change is this: Cluster Autoscaler works at the level of node groups, while Karpenter works at the level of individual pending pods. On paper that reads like a minor implementation detail. In day-to-day operation it turns out to be most of the story.

With Cluster Autoscaler, you make the important decisions up front — which instance families exist, how much spare room each group keeps, and which group grows when pods start to queue. Karpenter takes those same decisions and turns them into guardrails, then chooses the actual node shape at launch time, once it can see what is really being asked for.

That tends to suit clusters where the workloads do not all look alike:

web services that spike during deployments
queue workers whose demand follows traffic rather than CPU
short-lived CI or batch pods
teams mixing Spot and on-demand capacity in the same cluster

None of this means node groups are a mistake. They are simply static by design, and that is perfectly fine right up until your workload mix changes often enough that static sizing starts to feel expensive or brittle. That is usually the moment Karpenter starts to earn its place.

NodePool and EC2NodeClass split policy from AWS plumbing

On EKS, two Karpenter resources do most of the work: NodePool and EC2NodeClass.

NodePool describes scheduling and disruption policy — what kinds of nodes are allowed, how aggressively Karpenter can consolidate, and which workloads are welcome to land there.
EC2NodeClass describes the AWS side — subnets, security groups, AMI family, the IAM role or instance profile, storage, and the other EC2-specific details.

Keeping these separate is more useful than it first appears, because it holds your capacity intent apart from the cloud plumbing underneath it. A single EC2NodeClass can back several NodePools, and each NodePool can express a different set of rules about the workloads it serves.

A minimal setup looks like this:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ['c', 'm', 'r']
        - key: karpenter.sh/capacity-type
          operator: In
          values: ['spot', 'on-demand']
        - key: kubernetes.io/arch
          operator: In
          values: ['amd64']
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 10m

Two things are worth remembering here, and the docs are clear about both. Karpenter will not do anything at all until at least one NodePool exists — there is no implicit default waiting in the background. And a pod's own scheduling constraints are intersected with the NodePool requirements, so if a workload asks for something the pool does not allow — an instance family, a zone, an architecture — Karpenter simply will not launch a node for it. It sounds obvious written down, but it is one of the more common ways for a pending pod to look like a capacity shortage when it is really a policy mismatch.

Why just-in-time provisioning beats a larger buffer

The traditional habit on EKS is to keep enough spare capacity around that the next deployment or burst will probably fit. The trouble with "probably" is that it quietly shows up on the bill every month, whether or not you ever use it.

Karpenter lets you run a little closer to the line, because it provisions from real pending demand rather than a guess. If a rollout suddenly needs six more replicas, it can pick capacity that fits those pods instead of inflating a general-purpose group and hoping the shape is close. When the burst passes, the same controller can quietly remove or replace whatever it over-provisioned.

This helps most when your requests are uneven. A memory-heavy worker, an ARM-only build job, and a stateless frontend rollout do not really want to share a node. Node groups force you to approximate and accept some waste, whereas Karpenter can stay much closer to what each workload actually asked for.

The catch is that your constraints have to be explicit. Instance family, architecture, capacity type, disruption tolerance — all of it needs to be written down properly. Karpenter takes away the provisioning ceremony, but it does not take away the thinking.

Consolidation, drift, and interruption handling are the real day-two features

Fast provisioning is the headline, but the part you live with day to day is what happens after a node has landed.

Karpenter's disruption controller looks after two cases that quietly save a lot of effort:

Consolidation removes empty nodes, or swaps an underutilised one for a cheaper shape when the same pods can comfortably run elsewhere.
Drift replaces nodes that have fallen out of step with the current NodePool or EC2NodeClass — a new AMI selection, say, or changed launch requirements.

In a node-group-heavy setup, those two jobs — right-sizing and AMI rollovers — usually turn into separate pieces of scheduled maintenance. Karpenter keeps comparing what you asked for with what is actually running, so the fleet does not slowly drift out of shape between maintenance windows.

There is an honest caveat here, and the Karpenter docs do not hide from it: consolidation is only ever as good as the disruption rules you allow. Preferred anti-affinity and topology spread can quietly stop nodes from consolidating even when the scheduler could place the pods elsewhere, and PodDisruptionBudgets can do the same. So when someone says Karpenter is not saving them any money, the controller is rarely the first place to look. More often it is the workload policy wrapped around the nodes it is trying to drain.

Interruption handling matters too, especially if you lean on Spot. Karpenter can react to a Spot interruption warning, bring up replacement capacity ahead of time, and drain the old node in parallel — but all of that still depends on the workload tolerating disruption. Two minutes is not long when graceful shutdown, PDBs, and placement rules are all pulling in different directions.

The cost story here is the unglamorous one: a cluster that already had plenty of capacity in aggregate, just trapped in the wrong shape. Consolidation and drift are what turn that from a tidy architecture diagram into an actual lower bill.

Spot works well when the fallback path is real

Karpenter and Spot are often mentioned in the same breath, and for good reason — it can choose from a far wider range of instance types than a hand-maintained node-group layout, which is exactly what makes Spot workable. The part worth dwelling on is that Spot only pays off when the fallback is genuinely there.

A sensible EKS pattern usually looks something like:

prefer Spot for workloads that can tolerate interruption
allow on-demand in the same NodePool for when the service cannot wait for spare Spot capacity to appear
keep topology spread and PDBs sensible, so replacing one node does not set off a second incident
test your interruption handling before you actually need it

That last point carries more weight than it might seem. "We allow on-demand fallback" is a line of configuration; proving that a constrained workload can still schedule when a Spot pool dries up is a different thing entirely, and a good deal more reassuring when it happens at an awkward hour.

Where Karpenter is better than Cluster Autoscaler, and where it is not

Karpenter tends to win out over Cluster Autoscaler when an EKS cluster carries a genuine mix of workloads and their shapes change often. It is good at turning pending work into right-sized nodes without asking you to pre-bake every possible capacity combination into managed groups ahead of time.

It is a less obvious choice when the fleet is already settled — the node groups are well understood, the workload mix barely moves, and nothing about the current setup is causing pain. If what you have today is simple and cheap, Karpenter can end up being a sophisticated answer to a question nobody was really asking.

So the useful question is not whether Karpenter is newer or cleverer, because it is comfortably both. It is whether managing node groups has quietly become a drag on your time. If it has, the added complexity is usually a fair trade. If it has not, a well-kept Cluster Autoscaler setup may still be all you need.

Takeaway

Karpenter moves EKS node autoscaling away from managing node groups and towards provisioning that follows the workload. You describe the policy once through NodePool and EC2NodeClass, and from then on capacity is launched to match the pods actually in front of the scheduler, rather than the guess someone made last quarter.

The real payoff is not only quicker scale-up. It is the combination of right-sized launches, consolidation, drift replacement, and Spot fallback that holds together under pressure. Between them, consolidation and Spot can take a meaningful bite out of node spend without making scale-up feel fragile, though the exact number will always depend on your particular workload mix.

If you are already spending your week wrestling oversized node groups, slow scale-up, or a fleet that drifts out of shape between maintenance windows, Karpenter is well worth a serious look on EKS.