DEV Community

Katarina Supe
Katarina Supe

Posted on Edited on AI-assisted

The release shipped. Did the docs?

In my last post, I flipped who writes the first docs draft. I ended it with a question I left open: how do you know which PRs in a milestone need docs, and how do you get them all done before release day?

This post is my answer, and the last one in this series.

Late docs used to be an annoyance

Docs have always lagged a little behind releases. A feature ships on Tuesday, its page shows up a week later, and in between, developers piece it together from the release notes or ask someone. Annoying, but survivable.

Agents don't wait for the page. They read the docs as they are today and take them as the truth. If the docs don't mention a new feature, for the agent it doesn't exist. If they don't mention a bug fix, the bug is still there. You can ship the best feature of the year, and an agent will tell your user it can't be done.

So "the docs will catch up" doesn't work anymore. Whatever the docs say on release day is what your users hear.

Every PR has to answer one question

The first step is the simplest: a PR can't merge until it says whether it needs docs. A check blocks the merge until the PR has one of three labels:

  • Docs needed: something a user sees changed, and a page has to say so.
  • Changelog only: worth a line in the release notes, but no page changes.
  • Docs unnecessary: nothing a user would notice.

That gives me a list. It doesn't give me the truth. The engineer sets the label while thinking about the code, and the code isn't always where the user impact is.

My favorite example from the release I'm working on now: a fix to a shortest-path query was labeled "docs unnecessary". Fair enough, from the engineer's side. Most of the PR was a refactor, and no flag, command or syntax changed. But inside it was a security fix: the query could return a path ending at a node the user wasn't allowed to read. Anyone using permissions would want to know that.

No page had to change. It needed one line in the release notes. A small changelog entry can go a long way, so I read the "unnecessary" labels too, not just the "needed" ones.

Check as you go, not on release day

If you check the docs on release day, you find the gaps on release day. By then the engineer is three PRs further, everyone is busy with the release, and whatever doesn't make it waits for the next one. In practice, the docs stay a release behind for good.

So I check while the release is still in progress, as PRs merge. The check compares the milestone with the docs: which merged PRs need docs, which already have a docs PR, which release notes are still missing, and which labels look wrong. It takes minutes, and the gaps show up while the engineer still remembers the PR.

Release day turns into a final check instead of a scramble.

Write the release note after someone tried it

The engineer writes the first release note in the PR, while building the change. It describes what they built. That isn't always what matters to the person who uses it.

I write the final note from the docs PR, once the change has been run and the examples work. By then someone has looked at it from the user's side, and that often moves the focus.

One fix was about hitting the memory limit inside an internal store. The engineer's note talked about a possible crash and a fairly unusual way to trigger it. What a user would actually see was different: after one failed write, every later write to that store hung until a restart. That's what someone would remember, so that's what the note says now.

This matters most for breaking changes. Another change added a long-requested option to a path query, and its note read like a new feature, with the breaking part at the end of the sentence. For anyone already using that query, the end of the sentence was the whole story: their query would stop working after the upgrade. So it became two entries, a new feature and a breaking change. Every breaking change gets the same three parts:

  • Before: how it worked.
  • Now: how it works.
  • What to do: the change you need to make.

A developer who upgrades doesn't want to know what we built. They want to know what breaks and how to fix it.

What's left for humans

Agents read everything, fast, and believe all of it. That's what started this whole series. But look at where it brought us.

Running every example, checking every page that mentions a feature, drafting the docs for every PR, comparing a milestone with the docs while it fills up: none of that was possible by hand at today's pace. With AI, it is. And the developer who opens the page reads the same docs the agent does. We did this for agents, and humans got docs with better coverage and accuracy than ever.

What AI hasn't solved is whether the docs are any good. A page can be true, up to date and shipped on time, and still not help anyone: the wrong example, the wrong level of detail, the point buried in the third paragraph. Coverage and accuracy can be checked. Quality still needs someone who knows the reader. In my opinion, that's where the human in the loop belongs now.

How does your team know what needs docs before a release? I'd love to hear it.

Top comments (0)