It's 17:40 on a Thursday. Sixty-one commits have landed since the last tag. CI is green. Someone in the release channel types: "are we good to ship?"
And then a strange thing happens. Nobody actually knows.
The person who asked doesn't know — they can't read sixty-one commits. The tech lead skims the compare view, sees nothing on fire, and says "should be fine." The QA engineer asks what changed and gets a link to a diff. Somebody ships it.
Most of the time it works. That's the problem. It works often enough that we never notice we replaced a decision with a mood.
The bottleneck moved and nobody moved with it
For fifteen years the constraint on shipping software was writing it. Everything we built — CI, code review, trunk-based development, feature flags — was designed around that constraint.
That constraint is gone. In 2025 and 2026, a mid-sized team generates more code in a week than it used to generate in a month.
The 2025 DORA report caught the consequence precisely: teams that increased AI adoption reported better code quality and better documentation at the individual level — and worse delivery stability at the team level. Change failure rate went up. Rework went up. Individual output improved while the system got less reliable.
That is not a paradox. It's a queue. Code generation got 5x faster. Review, judgement and release verification did not get faster at all. The volume now outruns the only part of the pipeline that was never automated: someone deciding whether this particular set of changes is safe to put in front of users.
And the benchmarks say we were not good at it before the volume went up. Only about 8.5% of teams achieve the ideal 0–2% change failure rate. Elite performers sit around 5%. Low performers run at 45–60% — meaning nearly one in two of their deploys causes a degradation that needs a fix.
Half. Of deploys.
What CI actually tells you
Green CI means: the code you wrote does what you told it to do, on the tests you remembered to write, in an environment that is not production.
That's valuable. It is also a much smaller claim than the one we use it for.
CI does not know that this release contains a database migration and code that still reads the old column. It does not know that auth and payments changed in the same window, which is the pairing behind a disproportionate share of the worst outages. It does not know that one of these PRs was merged with an unresolved "changes requested" review. It does not know that 140 production files changed and not one test file was touched.
Every one of those is visible in the repository. None of them is visible in a green checkmark. We just never built the thing that looks.
The four questions
Strip away the tooling and every deploy decision is an attempt to answer four questions:
- What actually changed? Not the commit messages — a real answer, inferred from which code paths moved.
- Is any of it risky? Which specific changes carry a blast radius, and how big.
- What should we test before we ship? Tied to this release, not a generic regression suite.
- Can we deploy, yes or no? And if not, what has to be true first.
Notice that release notes answer none of these. Release notes are a communication artifact — a list of what happened, written for people who weren't there. They are not a risk artifact. Generating them with AI makes you faster at producing the wrong document.
The rubric, if you want to do this by hand
You don't need a tool to start. Here is the signal list I'd use, and — more importantly — why each one earns its place. Steal it.
Schema migrations. The highest-severity routine change in most codebases, because it's the one you cannot roll back by redeploying. Weight it heaviest.
Payment and billing paths. Failures here are the only bugs that are simultaneously a customer incident, a finance incident, and sometimes a compliance incident.
Auth, session, token, RBAC. Same reasoning. A subtle auth regression is worse than an outage: an outage announces itself.
Declared breaking changes. A ! in a conventional commit, a BREAKING CHANGE: footer, a breaking label. Someone already told you. Read it.
PRs merged over an unresolved "changes requested." This is the highest-value signal on the list and almost nobody checks it. A reviewer looked at the code, said no, and it shipped anyway. That's not a code smell, it's a process breakdown.
Release size. Not because big releases are inherently bad, but because they destroy your ability to bisect. When something breaks in a 200-commit release, your mean time to recovery is a function of how long it takes to find which commit did it.
Production files changed, zero test files touched. Not a moral failing — sometimes correct. But it should be a choice someone made, not a fact nobody noticed.
No pull requests traceable at all. If a range of commits has no PRs behind it, nothing in this list applies, because nothing was reviewed. That's the finding.
Infra, CI config, .env, lockfiles. Lower individually. Dangerous in combination with anything above, because they change the ground the release stands on.
Run a release through that list by hand once. It takes about ten minutes and I promise you find something.
The part I got wrong twice
I built a scoring engine on top of that rubric — every release starts at 100, rules subtract. Two design decisions took me three attempts each, and I think they generalise to any risk score anyone reading this ever builds.
Penalties have to compound, not add. My first version subtracted flat points: migration −18, auth −14, no tests −10, and so on. Any real release of decent size hit zero. Which means a messy-but-shippable Tuesday release and an actual disaster both scored 0 — the score lost all resolution exactly where you need it most. Now each penalty removes a share of the confidence still remaining. A bad release approaches zero asymptotically and never quite gets there, so "bad" and "catastrophic" stay distinguishable.
Positive signals must be worth exactly zero points. This one is counterintuitive and I resisted it. Green CI, approvals, tests shipped — surely those should add? No. If good signals add points, a clean release banks headroom above 100, and then adding a database migration to it leaves the score at 100. The score would silently absorb the risk. A user seeing that reads it as a bug, and they're right. The absence of a problem is not evidence of safety. It's just the absence of a problem.
That second one is really a statement about risk scoring in general: you cannot earn credit against future harm. Every dashboard that lets a green metric offset a red one is lying to somebody.
The engine described above is what we're building here — point it at two refs of a repo, it reads the PRs, commits, changed files and CI checks between them, scores the release, and gives you a QA checklist for what it actually touched. It isn't open to sign-ups yet; right now it's being calibrated against real releases of large open-source projects, which is its own article.
If your CI is green and you still hesitate before hitting deploy, that hesitation is information. It's worth building the thing that explains it.
What's on your deploy checklist that isn't on mine? I'm genuinely collecting these — the rubric above is version four and it got better every time someone told me what I was missing.
Top comments (0)