Why post-deploy verification deserves its own category

Lazypl82 — Tue, 12 May 2026 04:37:32 +0000

After a deploy, the hard part is often not the deploy.

CI has already passed. The rollout has started. Dashboards mostly look normal. Then one small runtime signal moves, and someone has to decide whether it came from this deploy or whether it is just noise.

That window is short. Usually the first 15 minutes. But it shows up every time a team ships, and the more I worked around it, the less it felt like a normal monitoring problem.

The decision is not "is the system healthy"

When I first started, I assumed this was a subset of observability. The signals are runtime signals. The dashboards are the same dashboards. The Sentry events are the same Sentry events.

But the decision is different.

Monitoring is mostly about ongoing system state. Is the service up. Are the latencies in range. Are the error rates trending. The decisions monitoring drives are continuous: alert on threshold breach, page on saturation, dashboards for capacity planning.

The post-deploy decision is much narrower. This deploy went out a few minutes ago. A signal just shifted. Is that shift attributable to the deploy? Is it strong enough to act on? What should happen next?

Notice that the inputs overlap with monitoring. The decision does not.

Why existing observability tools do not close it

The existing stack does inspect the right signals. Datadog can show error rate per service. Sentry can show new exception fingerprints. Grafana can correlate latency to deploy markers.

What none of them do, by default, is close the decision.

They give you the inputs. They show you the picture. The interpretation sits in the operator's head: is this fine, is this WATCH, is this RISK. That interpretation is the actual bottleneck.

In small teams, that bottleneck is invisible because one person knows the deploy, knows the service, sees the dashboards, and makes the call in a few seconds. The mental model is in their head.

In larger teams, especially in MSA setups where multiple teams ship in parallel, the bottleneck shows up. Service A and Service B both deployed in the last 30 minutes. Error rate is up on a checkout API. Whose deploy is responsible? Should anyone roll back? Whose call is that, and on what evidence?

The existing stack supplies the data. It does not supply the verdict.

What "verdict" means here

I have been calling the output a verdict because that is what it has to be: a closed decision, not another open dashboard.

The shape that has worked is three states.

STABLE means the post-deploy window looked normal. Move on.
WATCH means something shifted, but not enough to act. Stay close.
RISK means the pattern is strong enough that doing nothing is the wrong call.

Two states would have been simpler, and that is what I tried first. But binary collapses the middle, and the post-deploy middle is real. Most days, the verdict is WATCH. The deploy did not break anything outright, but something is moving. The team needs to know that without escalating.

Three states. STABLE, WATCH, RISK. Each with the affected API, a recommended action, and the next move. Not a chart. Not an alert. A short structured verdict that closes the decision.

Why the input has to stay narrow

Here is the part I want to be honest about.

The temptation here is to ingest more. More signals, more derived metrics, more partner-side measurement. The product gets bigger. It looks more capable.

That is also exactly when this category turns into a worse APM.

The boundary that keeps this category narrow is on the input side, not the output side. Teams send raw events they already have: error logs, stack traces, deploy markers, environment metadata. They do not measure latency, throughput, p95, ratios, counters, CPU, or memory just to send them somewhere else. The tool derives what it needs from the raw events.

If a partner has to install something that measures throughput just to use the verification layer, the layer is no longer narrow. It is just another collector with a verdict on top.

Raw in, verdict out. That is the shape that makes the separate tool make sense. Without that boundary, the category collapses back into monitoring.

Why the output has to be readable by agents

There is one more piece that I think matters more than it looks.

The verdict has to be readable by something other than a human dashboard.

In modern deploy pipelines, humans are not always the only consumer of deploy outcomes. An internal workflow might need to read the result. An agent might need a decision input. A release process might need a shared object that says what happened after the deploy, without scraping a dashboard or reading a Slack thread.

A dashboard does not serve those consumers well. A structured verdict does. STABLE, WATCH, RISK as discrete values, plus affected API and decision tier. That shape can be read by an agent and handed to a policy the team owns.

That second consumer is one of the reasons I stopped trying to make this look like a monitoring tool. Monitoring tools are designed for human reading. Verdict tools have to be readable by both humans and software.

Where this leaves the category

I do not think every team needs a separate post-deploy verification tool. Small teams with one engineer per deploy mostly do not. The bottleneck is not visible at that scale.

The teams where it shows up are the ones where multiple services ship in parallel, where multiple humans need a shared verdict, and where workflows downstream of the deploy need to read structured output.

For those teams, the post-deploy 15 minutes is not just "a feature your monitoring tool should add". It is its own category, with its own narrow contract: raw events in, structured verdict out. It does not fit cleanly inside an existing observability surface.

That is the bet I have been working on.

I am building Relivio, a small verdict layer for the first 15 minutes after a production deploy. If you are working on something similar, or running into the same problem, happy to talk. relivio.dev