The deploy job turns green at 2:14 p.m. Someone drops the link in the team channel, and the on-call engineer opens the dashboards. Error rates and latency are holding flat. Ten minutes later, the team calls the change good. But when those dashboards opened, a third of the pods were still on the previous version. One region didn't finish syncing until 2:31. The flat graphs mostly showed the old code behaving the way it always had.
The pipeline reported accurately on everything it controls, and nothing in it failed. The mistake was in how the team read the green check. The check confirmed that the pipeline had finished its steps, but the new code hadn't reached most of the traffic yet, so the team judged a change that had barely run. Every signal the team looked at described the deploy, and none came from the change itself.
A change passes through four events
The green check marks just one of four events a change passes through on its way to users. Many teams already keep two of them apart, deploying code to production behind a feature flag and releasing the new behavior to users later. Even so, they tend to treat each event as proof of the next.
CI finishes first. Then the code deploys. Next, the code is running everywhere, in every environment and region that users actually reach. Finally, the feature releases, and users can experience the new behavior.
Those events don't always line up. The pipeline can report success while the new code is still rolling out to some pods or regions. That gap between deploying the code and running it everywhere usually closes within minutes, so most checks skip it. There are three scenarios where this lag is most common:
- A Kubernetes rollout or GitOps sync can report success while pods are still restarting or failing readiness checks.
- A CDN can keep serving stale assets from edge caches until their TTL runs out or someone purges them.
- A multi-region deploy can finish in one region well before another.
In each case, the deploy finishes before the change has handled real traffic, so verification has to confirm that the change ran before its metrics can mean anything.
Existing signals describe the deploy
The signals most often used to confirm a change is live describe the deploy as a whole.
| Signal | Proves | Doesn't prove |
|---|---|---|
| Rollout status or sync result | The pipeline finished its steps | Every instance is serving new code |
| Health and readiness checks | The process is up and accepting traffic | The new code path is running |
/version endpoint |
Which build an instance has | That users reach that instance or that code path |
| Deploy markers in observability | When the deploy happened | When the change actually took effect |
Each of these signals answers a valid question about the deploy. A status field reports what the pipeline believes happened. A fixed bake time proves only that time passed. A quiet dashboard is an inference drawn from silence. Canary analysis gets closer, but it still compares aggregate metrics across versions, and aggregates can't show that any one change in a build ever ran. None of these signals shows that a specific change ran in production for an actual request.
The change leaves its own evidence
The most obvious path is often the most overlooked, and feature flags are a good example. A flag inside the new code can be the signal that proves it's running. Say a flag key exists only in the new build. An evaluation of that flag can happen only if the new build is running and a request has reached that code path. The evaluation is therefore a signal the code produces about itself. For instance, in a product catalog, this LaunchDarkly flag check sits in the new code path that ranks search results.
import ldclient
from ldclient import Context
client = ldclient.get()
def search_products(user, query):
context = Context.builder(user.id).kind("user").build()
# This flag only exists in the new build. If it's being evaluated,
# at least one instance is running the new code.
if client.variation("new-catalog-ranker", context, False):
return new_ranking(query)
return legacy_ranking(query)
A team already wrapping risky changes in a release flag can use that same flag to carry this evidence without extra instrumentation. The flag can stay off the whole time, and users keep getting the legacy ranking. Each call still counts as an evaluation as long as the SDK sends events. That team can therefore see that the new code is running before it releases anything to users.
LaunchDarkly shows those evaluations per environment, over time, and by variation. After a deploy, an engineer can check whether the new flag has been evaluated in production at all. An empty chart means either no instance is running the new code or no request has reached it yet. SDKs send events in batches, so a minute or two of silence right after a deploy is normal.
Rollout status still tracks whether every pod and region has the new build. Evaluations answer the question rollout status can't, which is whether the change has handled requests yet. Once the change has handled requests, its metrics can start to say something about it.
Any signal emitted from inside the new code path proves the same thing, whether it's a metric, a log line, or a trace. A rarely hit code path may take a while to show any evidence at all. If no request has reached the new code, nothing in production has tested it yet. That check is easy to run by hand on one change and one code path, and harder when a single deploy carries dozens of agent-written changes.
Agent volume widens the gap
Coding agents raise the number of changes in each deploy, so evidence per change matters more. When a deploy carries a few human-written changes, the gap stays small and easy to catch. Someone usually notices that the dashboards look too calm and checks again.
At agent speed, "the deploy is live" says little about any one change. It can't show which changes are running or which code paths have received requests. It also can't show which changes are still waiting on a cache or a region.
The cost of checking too early repeats with every change. A change judged against old code was never really tested, yet the team still decides to keep it, ramp it up, or move on. At a handful of changes a week, that cost is a rounding error. At agent volume, the same mistake can hit many of the changes in a single deploy.
So the unit of verification moves from the deploy to the change. Each change needs its own evidence that it's running before anyone decides what happens to it next.
Know what you're waiting for
Before a release starts, the team should already know which evidence it's waiting for, such as a first evaluation in production, a log line from the new path, or a version-tagged trace. A finished pipeline gives the team permission to start checking. Evidence from production decides the verdict.
Collecting that evidence is the first step. The decisions that follow depend on it, such as whether to ramp up the rollout, pause, or roll back. As agents add more changes to every deploy, teams will make those decisions more often. They'll also have less time to spend on each change.
Top comments (0)