Our Configuration Manager asked me a question I could not answer.
"For release 20260829.1, show me the list of changes, who approved each one, and confirm nothing else went out."
I had everything and nothing. Seven applications, three pipelines each (front end, back end, database), a Saturday production window, complete build history going back years. Every deployment logged, every commit traceable, every pipeline run timestamped. And I could not answer the question.
Here is why, and what I built to fix it.
The shape of the gap
Azure DevOps is pipeline-centric. Its unit of truth is a run: this YAML, this commit, this agent, this artifact, this environment, at this time. That model is excellent for the question "what is running in production right now" and useless for the question "what was release 20260829.1."
A release, in the sense an auditor means it, is a different kind of object:
- a named, agreed scope of changes
- decided before deployment
- approved by someone who did not write the code
- verified after deployment to have shipped exactly that scope and nothing more
ADO has no such object. It has runs, and a run is evidence of deployment, not evidence of intent. You can reconstruct intent by hand: pull the run, get the commits, walk the commits back to pull requests, walk the PRs back to work items, then go read each work item's comment thread looking for something that resembles an approval. I did that once, for one release, for one application. It took most of a day and produced an answer I would not defend in an audit, because the reconstruction had no fixed point. If a change shipped and nobody had approved it, the reconstruction would happily tell me it shipped. It had no way to notice that nobody had said yes.
That is the actual failure. Not missing data. Missing intent recorded ahead of time, against which the data can be checked.
The fix: write down the intent first
The whole system rests on one idea. Before a deployment, create a durable object that says "this release is these changes, approved by these people, on this date." After the deployment, go and find what actually shipped. Compare the two lists.
That comparison is the product. Everything else is plumbing.
The object itself is thin: a release tag, a target application, a window, a scope of work items, an approval record per item, a state. Three states turned out to be enough: Planned, Deploying, Closed. Planned means intent is recorded and approved. Deploying means the window has opened and discovery is running. Closed means reconciliation was reviewed and someone signed off in writing.
I built it backwards the first time
My first version was pipeline-first. Pick the pipeline runs that made up the release, then walk back to find the stories. It demoed beautifully and was wrong.
Pipeline-first can only describe what happened. You cannot plan a release out of pipeline runs, because at planning time the runs do not exist yet. Worse, it structurally cannot detect the case the auditor actually cares about, which is a change that shipped without approval. If your source of truth is the set of runs, everything in it is by definition part of the release.
The second version is story-first. Planning starts from the backlog: select the work items you intend to ship, check each one against readiness criteria, classify the change, route it for approval. The approval is recorded against the work item, by a named person, with a timestamp, and the approver cannot be the person who wrote the change. That last rule is one line of validation and the single most valuable thing in the entire application.
If you build something like this, start with the approval record. It is the fixed point everything else is measured against.
Finding what actually shipped
Post-deployment discovery is the part I expected to be hard and turned out to be mechanical, once I stopped trying to use the obvious link.
The obvious link is commits. Take the pipeline run, get its commit list, map commits to work items. This is fragile: squash merges collapse history, cherry-picks duplicate it, reverts pollute it, and the commit-to-work-item association depends on developers remembering to type #71549 in a message.
Pull requests are far better. A completed PR in ADO has a stable, queryable relationship to its work items and appears in the build's associated changes. So discovery runs in the other direction:
for each work item in the approved scope:
find its completed pull requests
for each PR, find the pipeline runs that included it
tag those runs with the release tag
for each discovered run:
list every PR it carried
for each PR, resolve its work items
any work item not in the approved scope is a passenger
Two things fall out of this for free. Runs get tagged in ADO with the release tag, so the evidence lives where auditors already look rather than only inside my tool. And the passenger detection is automatic. I did not write a rule to find unapproved changes. It is just the set difference.
Three buckets
Reconciliation produces exactly three lists, and the entire compliance conversation happens inside them.
Delivered as planned. Approved, shipped. The boring bucket. It should be nearly everything.
Planned but not delivered. Approved and did not ship. Usually harmless, occasionally a live problem: a story that was approved, marked done, communicated to a client, and is not actually in production. Nobody notices this without an explicit check, because nothing failed. The pipeline went green.
Delivered but never approved. Shipped and nobody said yes. This is the bucket the whole system exists for. A change rides along on a shared branch, catches the release train, and reaches production without review. The pipeline is green, the deployment is clean, the audit trail is a hole. My tool raises a hard stop on this and will not let the release close until it is either explained or recorded as an exception.
The first time I ran it against real history, it found one. That finding paid for the build.
Closing a release requires a written sign-off. Free text, mandatory, stored. Not a checkbox, because a checkbox tells a future reader nothing. A sentence like "confirmed login, encryption and notifications on production at 09:40" tells them what was actually verified and by whom.
What it does not do
It does not gate deployments. The pipelines still run whether my console approves anything or not, deliberately, because a compliance tool that can block a Saturday production window is a compliance tool that gets bypassed in the first emergency. It observes and it reports, and the pressure it creates is social, not technical. That is a real trade-off and reasonable people would make it differently.
It also does not eliminate the possibility of someone lying to it. Segregation of duties is enforced on the approval record, not on human behaviour. Two colleagues can still rubber-stamp each other. What the tool guarantees is that the rubber-stamping is legible afterwards, with names and timestamps attached. That is a lower bar than prevention and it is the bar most audit frameworks actually set.
What I'd tell you if you have the same problem
Your CI system's data model is not your governance model, and no amount of dashboards on top of pipeline runs will bridge that. The missing piece is almost always a record of intent, written before the event, owned by someone who is not the implementer.
Once you have that record, the useful output is not a green tick. It is a diff.
Top comments (0)