DEV Community

Cover image for QualityLayer: agree what done means before the agent writes code
Max Ritter
Max Ritter

Posted on Originally published at qualitylayer.dev

QualityLayer: agree what done means before the agent writes code

#claudecode #testing #productivity #ai

Coding agents write most of my code now, and the code itself is rarely the problem.

Even the best models don't build software the way an experienced engineer would. They don't know the users, and they don't know what the team considers good design.

An engineer settles those questions before writing anything. An agent guesses and keeps going.

A small task that showed me the problem

I saw this most clearly on a small task. I asked an agent to add CSV export to a reports page, and it wrote the whole thing without asking me a single question.

Then I read the diff and found out what it had decided for me:

  • whether the file contains only what's filtered on the page or everything
  • what happens when someone exports 50,000 rows
  • which columns end up in the file

Each of those depends on who uses the report. The agent just chose something and moved on.

Moving the decisions to the start

Working decisions back out of a diff is the slowest part of working with agents for me. So I moved them to the start.

Before the agent writes any code, we agree on a numbered list of results. The change counts as done when every one of them holds. I call the list "Done means".

It works with any agent, with or without a tool.

Let the agent ask the questions

I let the agent do most of the work. In plan mode in Claude Code or Codex I give it something like this:

Read the code that this change touches. Then ask me up to six questions
that decide how the change behaves. For each question, give your
recommended answer. After my answers, write a numbered "Done means" list.
Each item must be a result a person can check on the running program.
Enter fullscreen mode Exit fullscreen mode

The recommended answers are what makes this fast. I accept most of them with one keystroke.

The two or three I change are the decisions that needed an engineer. I make them now, before any code exists.

What the list looked like for the export

For the export, the list looked like this:

  1. The reports page shows an "Export CSV" button next to the filters.
  2. The file contains the rows that match the current filters, in the current sort order.
  3. The columns match the visible table columns, with the same headers.
  4. An export of 50,000 rows finishes while the page stays responsive.
  5. The existing PDF export keeps working.

The Done means list for the CSV export as a numbered checklist with five items. Item 5, the existing PDF export keeps working, is highlighted as the one people skip.

I can check each line myself in a minute. "Export works" would be useless as an item, because it doesn't say which behavior to look at.

Item 5 is the one people skip. I care about it most, because agents break nearby things quietly. Every list I write names something that has to keep working.

For a bug I use four parts:

  1. the steps that trigger it
  2. what happens
  3. what should happen
  4. what the fix must not change

Checking it with a session that didn't write the code

The list pays off a second time after the build. I open a new session that hasn't seen any of the work and give it the list. Then I ask it to check one point at a time on the running program:

For each item in Done means, run the program or the tests and show me
the evidence: command output, a screenshot or a file. If you cannot check
an item, say so and tell me what I need to do to check it myself.
Enter fullscreen mode Exit fullscreen mode

It has to be a new session. The session that wrote the code knows what it meant to do, and it tends to report that instead of what the code actually does.

A fresh session only has the list and the program in front of it.

What comes back is a short report with evidence for each point. I read that first. When I open the diff afterwards, I already know where to look.

Example report from the fresh session for the CSV export. Each point has its evidence next to it. One point is open, with a note on how I can check it myself.

What QualityLayer does with it

I built QualityLayer around this list. Its Discuss step runs this conversation for every change, and the list goes into the plan I approve before any code exists.

After the build, in the Verify step, an agent that didn't write the code checks every point and shows me the evidence. Anything it can't check stays open, and I look at it myself at Review.

Flow from Discuss to the Done means list, to the build session and then to a fresh session that checks each point. Review comes last. The session that wrote the code never checks it.

The list is a Markdown file in the repo. It works with Claude Code, Codex, OpenCode, Pi and others.

Staying the engineer

I think everybody who writes software is turning into a manager of agents. This list is how I stay the engineer in that setup. I make the calls and check the results, and the agent does the typing.

You don't need my tool for any of it, though. Write the list before your next change and check the result against it.

I'd be curious whether it changes how you review.

https://github.com/maxritter/qualitylayer

Top comments (0)