Most agent workflows I see are not one request. They are a run of them: add the model, wire the endpoint, add validation, write the migration, build the report. Each step lands on code the agent wrote a few minutes earlier.
Each step also ends the same way. The agent runs its tests, they pass, and it tells you it's done.
A preprint from two Amazon researchers measures what that "done" is worth when you stack twenty of them. Short version: not enough to build on without checking.
What the benchmark did
BOBA-Bench (Cuadron and Gupta, arXiv 2611.02207, 1 November 2026) turns the commit histories of 140 open-source Python projects into 280 chains of 20 requests each. Each request goes to the agent in a fresh conversation, on whatever the previous request left. Hidden tests grade every step. The agent is never told the result.
Ten agents, from open-weight to frontier models. Headline numbers:
- 51% of chains completed with agents left alone. 39% got through with no missed request and no broken earlier test.
- 66 to 84% of the individual requests inside those chains were completed.
- A request issued on broken code failed about twice as often as one on a clean base.
- Only 14 to 33% of failing tests were ever repaired later.
- Asked whether an implementation was correct, agents were right 53% of the time.
- One external "that's wrong" plus one repair round lifted chain completion from 51% to 59%.
Two details matter for engineers. First, once checked, agent-written code was nearly as good a base as the original developers' code (within 4 points for every model). The code wasn't the problem. Unnoticed failures were. Second, in 23 to 43% of attempts a later request broke a test that had passed earlier, usually one of the agent's own.
A second preprint, Multi-step SWE (Klavis AI, arXiv 2611.07166), found the same shape on 5 to 15 step tasks: the best of nine models passed 56.8% of steps but solved 18.5% of whole tasks.
Mapping it onto engineering work
This part is my inference, not the paper's.
- The agent's own tests are not your gate. If it misreads the request, it tests its misreading. Your gate is a check it didn't write and can't edit.
- Some steps are foundations. A schema, an interface, a shared helper. Later steps inherit their mistakes. Those need a real check before you issue the next request.
- Regression runs every step. Re-run the earlier steps' checks, not only the new one.
- Agent review is a hint, not a gate. The reviewing models either waved through most wrong code (77 to 95% accepted) or rejected about half the correct code (41 to 58%).
- Feed back facts. The paper found that naming the failing test once did about what three rounds of bare "that's wrong" did. Paste the test name and the error.
- Multi-agent setups: the authors recommend that the verifier's result reach the sub-agent that did the work, not stop at the orchestrator.
The artefact: a chain plan
Paste this into the PR description, the ticket, or a CHAIN.md before you start a multi-step agent session. Fill the left columns first. Fill "Result fed back" as you go.
## Chain plan
| # | Request | Builds on | Foundation? | Check the agent can't edit | Earlier checks re-run | Result fed back |
|---|---|---|---|---|---|---|
| 1 | Example: add `discount_code` to Order model | - | Yes | Migration applies on a copy of prod schema; one real order loads | - | Pass |
| 2 | Example: expose field on `GET /orders/:id` | 1 | Yes | Contract test from API spec (held outside agent workspace) | 1 | Fail: `test_order_contract_discount_null` expected null, got "" |
| 2r | Example: repair after named failure | 1 | - | Same contract test | 1 | Pass |
| 3 | Example: validate code format on POST | 2 | No | Hand test: one valid, one invalid code via curl | 1, 2 | Pass |
| 4 | Example: monthly discount report | 2, 3 | No | Report totals match a SQL query written by a person | 1, 2, 3 | Not run yet: do not start step 5 |
Rules I'd attach to it (again, mine):
- A step marked Foundation = Yes does not get a successor until its check column says Pass.
- "Check the agent can't edit" names a file, test or person. "Agent's tests pass" is not allowed in this column.
- Failures go back as the exact test name or observed behaviour, then one repair round, then re-check.
- If nobody can run a step's check today, the chain stops there today.
Limits
Both papers are preprints written by the benchmark builders. BOBA-Bench is Python only and removes the human on purpose, so real chains with someone glancing between steps may do better. The authors call their own advice "patches, not a principled approach", and expect it to become unnecessary once agents can certify their own work. Even perfect feedback recovered only part of the loss, and the repair round raised regression rates by 3 to 9 points.
Sources
- Cuadron and Gupta, "Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone", arXiv 2611.02207: https://arxiv.org/abs/2611.02207
- Zeng, Liu and Lin, "Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?", arXiv 2611.07166: https://arxiv.org/abs/2611.07166
How many requests do you let an agent stack before you run the thing yourself, and has a silent miss ever surprised you several steps later?
Top comments (0)