"Fix checkout" is enough to start a conversation. It is a terrible unit of work for a coding agent.
The agent can inspect the repository, find something suspicious, write a plausible patch, and report success. The reviewer still has no shared answer to two basic questions: what exactly was broken, and what observable behavior should now be different?
That gap matters more as implementation gets cheaper. A coding task should begin with a known before-state and a declared after-state. The patch is only an attempt to move the system between them.
A task is a state transition, not a sentence
A vague prompt usually contains intent:
Fix the checkout timeout when a session expires.
It does not contain a task contract. Which revision has the bug? Which fixture exposes it? Does the request hang, return the wrong status, or discard the cart? What behavior must remain unchanged? Which command evaluates the result?
Without those details, the agent has to invent part of the acceptance test while implementing the fix. That is a bad delegation boundary. The same system is choosing what "done" means and announcing that it got there.
A reviewable task has two observable states:
- The starting state can be recreated before any edit.
- The target state can be evaluated after the edit without trusting the agent's summary.
The agent can choose the implementation between those states, while the task author owns the surrounding contract.
This framing also keeps the human out of low-value supervision. You do not need to narrate every code change if you have already defined the behavior, constraints, and evidence that will make the change acceptable.
Start from failure on purpose
CLI-Gym is a useful example of failure-first task construction. Its project documentation describes a pipeline that begins with healthy repository environments, introduces controlled failures, and turns those states into reproducible command-line tasks. The README reports 1,655 tasks from 29 repositories.
Those numbers are project-authored, and they do not prove benchmark quality or production reliability. The task design is more useful here than the headline count.
A controlled failure gives the agent and reviewer the same starting point. It can be reset. It can be observed before the patch. It can be checked again afterward. If the failure disappears for reasons unrelated to the patch, that becomes visible too.
Compare that with dropping an agent into a moving branch and saying, "The build is weird, please fix it." The agent may find a real problem. It may also repair an incidental symptom, update a flaky snapshot, or change enough code that nobody can tell which behavior mattered.
Failure-first does not mean every assignment needs an existing failing test. Feature work often starts without one. In that case, define an observable before-state and a concrete acceptance example. A missing endpoint, an unsupported interaction, or a fixture that currently produces no result can still anchor the transition.
Put the contract in an artifact
Chat is convenient for exploration. It is weak as the only record of a coding task.
A durable setup keeps intent, scope, checks, and evidence in artifacts that survive the session. A task file can be reviewed before execution. A plan can be compared with the allowed scope. Test output and the final diff can stay attached to the same work unit.
This changes the review from "Does the completion message sound convincing?" to a comparison:
- Did the run begin from the declared base state?
- Does the original reproduction now reach the target behavior?
- Did the diff stay inside the stated boundaries?
- Which named checks ran, and which did not?
- Did the agent stop when the task became ambiguous?
Treat a confident completion paragraph as context, then verify it against the artifacts.
The contract should also remain implementation-neutral where possible. "Add a retry around refreshSession()" may be a reasonable plan, but it is already prescribing a patch. "An expired session returns the typed authentication error within the configured timeout and preserves the cart" gives the agent room to inspect the code before choosing a change.
A six-field task card
You do not need a large specification system to get this benefit. A compact task card is enough for many repository changes.
The following example is hypothetical. The commands and fixture names stand in for whatever your repository actually uses.
Base
- Revision:
<commit SHA> - Fixture:
expired-session.json - Environment assumption: the local test services start with the documented setup command
- Initial state: the cart contains one item and the session token is expired
Reproduce
- Run
pnpm test checkout -- --grep "expired session" - Current result: the checkout request waits until the test timeout
- Save the pre-change failure output with the task
Target
- Checkout returns the repository's existing typed authentication error before the configured timeout
- The cart remains intact so the user can authenticate and try again
Boundaries
- Do not change the session lifetime
- Do not add a dependency
- Preserve checkout behavior for valid sessions
- Keep unrelated formatting and refactors out of the diff
Checks
- Re-run the exact reproduction command
- Run the existing checkout test group
- Run the repository's type check
- Inspect the final diff against the boundaries above
Stop
Hand the task back unresolved if:
- the failure does not reproduce from the declared base
- the fixture or local setup is broken
- the expected result conflicts with the current API contract
- a required service or credential is unavailable
- the fix needs a wider product or architecture decision
Omitting the Stop section leaves the agent with no defined way to return an unresolved task. An agent that has only success criteria will keep searching for a patch. Explicit stop conditions let it return a useful failure report before it widens the change or guesses at product intent.
Passing checks is evidence with a boundary
Named checks make the transition reviewable. They do not certify the entire system.
A green targeted test says something about the behavior encoded in that test. It does not prove that the architecture is healthy, that the change is secure, or that every product expectation has been captured. A reproducible failure narrows uncertainty around the named behavior while leaving architectural and product questions for review.
The final diff still needs review. Compare the code with the declared target and non-goals, then decide whether broader tests or specialist review are needed. A two-line fix in an authentication path may deserve more scrutiny than a larger change in an isolated internal tool.
The task card helps make that judgment legible. It shows what the checks were supposed to prove and where their authority ends.
Let an unresolved run count as a valid result
The Hacker News and Reddit threads in the source notes are anecdotal, but some participants favor small, auditable loops: inspect, change, run a real check, review the diff, and preserve enough state for the next step. Tool preferences vary, while every unattended run still needs a clear handoff.
A run that reports "cannot reproduce from the named revision" has produced useful information. So has one that finds contradictory acceptance checks or a missing service. Those results protect the repository from a guessed patch and give the task author a specific problem to resolve.
Treating every run as patch-or-failure creates pressure to hide uncertainty. Treating a justified stop as part of the contract makes the workflow more honest. The agent can state what it inspected, where the transition broke down, and what decision or dependency is missing.
That is a better result than green output from a check nobody agreed would define success.
Define the transition before asking for the patch
Return to "fix checkout." With a base revision, fixture, reproduction command, behavioral target, boundaries, checks, and stop conditions, the assignment becomes reviewable before the agent writes a line of code.
The agent may discover that the expected fix is tiny. It may find that the task is underspecified. Both outcomes are easier to handle when the before-state and after-state are explicit.
A stronger model can propose a better implementation. It cannot recover a success condition the team never wrote down.
Top comments (0)