Release notes answer the wrong question

Deployment Confidence — Sat, 01 Aug 2026 17:34:29 +0000

Before every release, somebody generates the notes. It used to take twenty minutes of scrolling the commit log. Now a model does it in four seconds, and there are a dozen products that will do it for you.

We got dramatically faster at producing a document nobody uses to decide anything.

Two different artifacts

Release notes are a communication artifact. Past tense, written outward, for people who weren't in the room — users, support, the changelog page. "Added X. Fixed Y." Their job is to tell someone what already happened.

The deploy decision needs a risk artifact. Future tense, written inward, for the person about to press the button. Its job is to say what could go wrong and what to check first.

Different tense, different reader, different content. We keep producing the first one and then making the second decision on vibes.

Put them side by side

Here is a realistic release note:

Added dark mode

Fixed login redirect

Updated dependencies

Nothing in there is false. Nothing in there is what you need either.

What's worth knowing is that the login redirect fix touched session handling — the same code path as authentication. That the dependency update was a lockfile bump underneath that same auth code. That the two arrived together, in a PR one person approved at 18:00 on a Thursday.

The release note summarises what. The decision needs where, together with what, and reviewed how carefully.

Why AI made this worse rather than better

Summarising a diff is a solved problem. It's among the easiest things a language model does: the input is structured, the output is a list, and the failure mode is boring rather than dangerous. That's exactly why the category is crowded.

Deciding whether a set of changes is safe to ship is not that. It requires knowing which paths are load-bearing, what changed together, what the review actually said, and what the CI state means. Almost none of that lives in the diff text. It lives in the pull requests, the review states, the check runs, and the file paths.

So we automated the easy half and called the job done. The document got faster. The decision didn't move at all.

The tell

Ask anyone on your team what they read before approving a deploy.

Nobody says "the release notes." They say they skim the compare view — which is another way of saying that the artifact we generate is not the artifact we use.

A document produced for every release and consulted for none of them isn't a documentation problem. It's a signal that the real work is happening somewhere with no tooling at all.

Keep writing them anyway

None of this makes release notes useless. They're good at their actual job: users need to know what changed, support needs to know what to expect, and the changelog is a real product surface.

Just stop expecting them to answer a question they were never shaped to answer. Ship the notes to your users. Build something else for the person holding the deploy button.

What do you actually read before approving a deploy? I'm collecting answers, and so far "the compare view, quickly" is winning by a lot.

Green CI is not a deploy decision

Deployment Confidence — Sat, 01 Aug 2026 11:44:47 +0000

It's 17:40 on a Thursday. Sixty-one commits have landed since the last tag. CI is green. Someone in the release channel types: "are we good to ship?"

And then a strange thing happens. Nobody actually knows.

The person who asked doesn't know — they can't read sixty-one commits. The tech lead skims the compare view, sees nothing on fire, and says "should be fine." The QA engineer asks what changed and gets a link to a diff. Somebody ships it.

Most of the time it works. That's the problem. It works often enough that we never notice we replaced a decision with a mood.

The bottleneck moved and nobody moved with it

For fifteen years the constraint on shipping software was writing it. Everything we built — CI, code review, trunk-based development, feature flags — was designed around that constraint.

That constraint is gone. In 2025 and 2026, a mid-sized team generates more code in a week than it used to generate in a month.

The 2025 DORA report caught the consequence precisely: teams that increased AI adoption reported better code quality and better documentation at the individual level — and worse delivery stability at the team level. Change failure rate went up. Rework went up. Individual output improved while the system got less reliable.

That is not a paradox. It's a queue. Code generation got 5x faster. Review, judgement and release verification did not get faster at all. The volume now outruns the only part of the pipeline that was never automated: someone deciding whether this particular set of changes is safe to put in front of users.

And the benchmarks say we were not good at it before the volume went up. Only about 8.5% of teams achieve the ideal 0–2% change failure rate. Elite performers sit around 5%. Low performers run at 45–60% — meaning nearly one in two of their deploys causes a degradation that needs a fix.

Half. Of deploys.

What CI actually tells you

Green CI means: the code you wrote does what you told it to do, on the tests you remembered to write, in an environment that is not production.

That's valuable. It is also a much smaller claim than the one we use it for.

CI does not know that this release contains a database migration and code that still reads the old column. It does not know that auth and payments changed in the same window, which is the pairing behind a disproportionate share of the worst outages. It does not know that one of these PRs was merged with an unresolved "changes requested" review. It does not know that 140 production files changed and not one test file was touched.

Every one of those is visible in the repository. None of them is visible in a green checkmark. We just never built the thing that looks.

The four questions

Strip away the tooling and every deploy decision is an attempt to answer four questions:

What actually changed? Not the commit messages — a real answer, inferred from which code paths moved.
Is any of it risky? Which specific changes carry a blast radius, and how big.
What should we test before we ship? Tied to this release, not a generic regression suite.
Can we deploy, yes or no? And if not, what has to be true first.

Notice that release notes answer none of these. Release notes are a communication artifact — a list of what happened, written for people who weren't there. They are not a risk artifact. Generating them with AI makes you faster at producing the wrong document.

The rubric, if you want to do this by hand

You don't need a tool to start. Here is the signal list I'd use, and — more importantly — why each one earns its place. Steal it.

Schema migrations. The highest-severity routine change in most codebases, because it's the one you cannot roll back by redeploying. Weight it heaviest.

Payment and billing paths. Failures here are the only bugs that are simultaneously a customer incident, a finance incident, and sometimes a compliance incident.

Auth, session, token, RBAC. Same reasoning. A subtle auth regression is worse than an outage: an outage announces itself.

Declared breaking changes. A ! in a conventional commit, a BREAKING CHANGE: footer, a breaking label. Someone already told you. Read it.

PRs merged over an unresolved "changes requested." This is the highest-value signal on the list and almost nobody checks it. A reviewer looked at the code, said no, and it shipped anyway. That's not a code smell, it's a process breakdown.

Release size. Not because big releases are inherently bad, but because they destroy your ability to bisect. When something breaks in a 200-commit release, your mean time to recovery is a function of how long it takes to find which commit did it.

Production files changed, zero test files touched. Not a moral failing — sometimes correct. But it should be a choice someone made, not a fact nobody noticed.

No pull requests traceable at all. If a range of commits has no PRs behind it, nothing in this list applies, because nothing was reviewed. That's the finding.

Infra, CI config, .env, lockfiles. Lower individually. Dangerous in combination with anything above, because they change the ground the release stands on.

Run a release through that list by hand once. It takes about ten minutes and I promise you find something.

The part I got wrong twice

I built a scoring engine on top of that rubric — every release starts at 100, rules subtract. Two design decisions took me three attempts each, and I think they generalise to any risk score anyone reading this ever builds.

Penalties have to compound, not add. My first version subtracted flat points: migration −18, auth −14, no tests −10, and so on. Any real release of decent size hit zero. Which means a messy-but-shippable Tuesday release and an actual disaster both scored 0 — the score lost all resolution exactly where you need it most. Now each penalty removes a share of the confidence still remaining. A bad release approaches zero asymptotically and never quite gets there, so "bad" and "catastrophic" stay distinguishable.

Positive signals must be worth exactly zero points. This one is counterintuitive and I resisted it. Green CI, approvals, tests shipped — surely those should add? No. If good signals add points, a clean release banks headroom above 100, and then adding a database migration to it leaves the score at 100. The score would silently absorb the risk. A user seeing that reads it as a bug, and they're right. The absence of a problem is not evidence of safety. It's just the absence of a problem.

That second one is really a statement about risk scoring in general: you cannot earn credit against future harm. Every dashboard that lets a green metric offset a red one is lying to somebody.

The engine described above is what we're building here — point it at two refs of a repo, it reads the PRs, commits, changed files and CI checks between them, scores the release, and gives you a QA checklist for what it actually touched. It isn't open to sign-ups yet; right now it's being calibrated against real releases of large open-source projects, which is its own article.

If your CI is green and you still hesitate before hitting deploy, that hesitation is information. It's worth building the thing that explains it.

What's on your deploy checklist that isn't on mine? I'm genuinely collecting these — the rubric above is version four and it got better every time someone told me what I was missing.

DEV Community: Deployment Confidence