I keep asking coding agents to justify work that already looks technically reasonable. They can explain the design, implement it, and write the tests. I still have to ask why the user needs it.
My jev-reranker CLI is one example. Its design included --model, --batch-size, and --timeout-ms, each with a default and validation. All three made it into the implementation. For the original MVP, I wanted to send retrieved documents to Jev, rank them, preserve their fields, and report failures safely. I didn't need to expose those internal settings as caller choices.
A review against the design could confirm that all three options worked correctly. The design had already made them part of the job.
That's the gap I wanted to check: whether the work itself was justified by what the user wanted to achieve.
How a suggestion becomes a requirement
I've kept revising codex-workflows and claude-code-workflows to control this. I changed the review process so an agent has to evaluate a finding before turning it into work. I also made technical choices open to reconsideration, even after they'd passed an earlier design review. The corresponding change in the Claude workflow preserves the user's requirements while allowing the agent to choose simpler ways to meet them.
The loop I'm trying to interrupt is familiar. A reviewer raises an edge case. The implementation agent accepts the proposed fix, adds a mechanism, and writes tests for it. The next reviewer checks that mechanism. After a few rounds, everyone is discussing how to make the addition correct, and nobody is asking whether it needed to exist.
With the coding models I use now, the extra work can spread through the whole development process. A suggestion becomes a design decision, then a task, then code whose tests make it look settled. I've had to keep returning to the user's purpose at each of those steps.
In When Better Models Make Old Agent Workflows Worse, I wrote about the cost of adding workflow rules after every failure. I didn't want another growing list of things the model should avoid. I wanted a separate check that could ask the same question about a design, a response to a reviewer, or an implementation change.
Judge the work against the outcome
Technical review gives us useful questions about complexity, duplication, abstractions, and correctness. An outcome check adds another reference point: what would the user lose if we left this proposed work out?
The timeout option makes the distinction concrete. I can remove --timeout-ms while keeping an internal deadline and safe error reporting. Removing the timeout mechanism itself could leave requests hanging. Those changes might both be described as simplifying timeout support, but they preserve different behavior.
The same issue appears before any code is written. While adding Antigravity support to sub-agents-skills, a review raised the possibility of running agy --version before every invocation. The minimum version was already documented, and normal CLI errors reached the caller. I couldn't identify a current requirement that needed another subprocess, a version parser, and a separate rejection path.
In that work, preserving interrupted output through the runner's existing partial status was necessary. Cutting that handling would make the implementation smaller and break the behavior callers expected.
A small diff can add an unnecessary obligation. A larger change can be the least work that actually meets the requirement. I wanted the check to compare the proposed behavior with a concrete alternative and identify which requirement each one preserves or loses.
Turning that question into a skill
A general-purpose LLM can make this judgment too. Asking it to review its work doesn't ensure that it will revisit the premises it just used to produce that work. I wanted the outcome question to be an explicit step, with a separate classification I could inspect alongside the agent's reasoning.
Working with Jev on retrieval suggested a way to try this cheaply. Its Choice API takes material and defined categories, then returns a choice and probabilities. I could give it a bounded judgment without starting another full review agent.
I built that into Outcome Doctor, a skill in agent-clinic. It's configured for the agent to use when choosing an implementation approach, including a design or a response to code review, without waiting for me to request an assessment. The coding agent reads the source and prepares the user's required outcome, relevant facts, proposed behavior, and an alternative. For implementation changes, that includes the affected behavior and actual callers. A Python script sends decisions from the same case together, with one classification question per decision.
Jev classifies the resulting approach as sufficient, excessive, insufficient, mixed, or unknown. Mixed covers a proposal that adds unnecessary work while leaving a requirement unmet. Unknown means a missing fact could change the judgment. The check needs to be able to say both “keep this” and “there isn't enough evidence yet.”
The coding agent then checks the result against the sources and uses that assessment to keep or revise its approach within the work I've authorized. It explains the decision from the evidence; Jev returns the classification and probabilities.
The first agent still decides what reaches Jev. That dependency became important when I tested the skill in fresh sessions.
Does the judgment follow the requirement?
The useful test was whether the same capability could earn a different judgment when the user needed it.
Against the original CLI requirement, the public model, timeout, and batch-size settings were classified as excessive. I then supplied explicit needs for those capabilities in authored comparison cases:
| Capability | Classified as excessive | Classified as sufficient |
|---|---|---|
| Model selection | A fixed model is enough | Compare caller-selected models in one installation |
| Timeout setting | One internal deadline is enough | Existing jobs require different deadlines |
| Batch-size setting | One internal batch size is enough | Service limits and throughput require different sizes |
These are synthetic requirement changes, not additional historical incidents. They test a property I need from the tool: it must be able to keep the very mechanism it would otherwise recommend removing.
The OSS review-response cases gave another useful separation. Jev classified the extra version preflight and unused output-field additions as excessive, while retaining required caller detection and partial-result handling. These inputs reconstructed the review decisions and their context. They tested whether the proposed responses served the existing runner's obligations.
That gave me evidence for the classification step. Running the skill exposed a different problem.
The agent left out the decisions I wanted checked
In fresh claude -p sessions on the CLI design, the agent sometimes omitted --model and --timeout-ms from the records it sent to Jev. The API call succeeded, and the skill produced an assessment. Two of the choices I already considered unnecessary had never reached the classifier.
A correct classifier cannot flag an item it never receives.
Other runs supplied possible future benefits as facts. Model selection might be useful for comparing versions someday, so the prepared input started giving Jev a reason to preserve it. Sharing a mistaken premise with another model can produce agreement very cheaply.
I changed how the skill prepares the input: carry the source's listed choices into the assessment, including ones that look necessary, and distinguish observed needs from hypothetical benefits. I also adjusted the Jev instructions to evaluate the behavior remaining after a change. A proposal to remove an unnecessary setting needs to be judged by the resulting implementation, just as an addition does.
In the final two fresh sessions on the original design, the agent included all three settings and Jev classified them as excessive. A session using the historical src/options.rs diff also identified all three. The proposed reductions kept the required internal behavior.
This is what made the idea usable as a skill: the agent could gather the evidence, submit the decisions, and return an assessment tied to the sources. Prepared-input results alone hadn't established that.
I ran these checks with jev-1.13.0 on a small development set that I reused during tuning. Expected judgments came from my explicit decisions and interpretations made during evaluation. Some design judgments still varied, particularly around proof obligations and support artifacts. I keep the source evidence available so a questionable result can be checked.
In these runs, the Jev calls were cheap and finished in about a second or less. The full Claude sessions took minutes to read the material, prepare the input, and explain the result. Most of the time went into preparing and explaining the judgment.
Before the next task gets added
I want this check where a review response is about to become an implementation task. It can also run while a design is still easy to change, or against an implementation when the relevant callers and requirements are available.
The outcome should remain visible in the review. If the agent proposes another mechanism, it should be able to explain which requirement the simpler approach would fail. If Jev disagrees, that gives the agent a specific judgment to revisit against the source. Treating Jev as an authority to obey would recreate the review loop I was trying to fix.
I haven't measured whether this beats an equally careful review by the coding model alone, or how much unnecessary work it prevents across ongoing development. The current evidence supports an experimental review aid: it caught known excesses, retained required behavior in the comparison cases, and worked through the skill on real design and code inputs.
The next useful evidence will come from review decisions it changes. I want to see which additions we decline, which simpler approaches we accept, and whether the required behavior survives those choices, before another proposed mechanism acquires its own tests, documentation, and maintenance.
Top comments (0)