DEV Community

Cover image for I benchmarked a Scrum workflow for AI coding agents. Here's what it fixed, and what it costs.

I benchmarked a Scrum workflow for AI coding agents. Here's what it fixed, and what it costs.

Huzair Saputra on November 06, 2026

Parallel coding agents are fast until two of them edit the same file. Or until the implementer also writes the tests that are supposed to check it....
Collapse
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

For the harder-task run, one thing worth measuring: green CI on each PR's head says nothing about the PRs together. Our team of Claude Code agents hit exactly that on Sunday: 9 backend PRs that were each fine conflicted with each other. One agent built a single integration branch (merge each PR, no rebase), a ChatGPT agent reviewed only the conflict resolutions, and a full CI run on the merged result found 5 failures no single PR showed. Since then only one agent merges into staging, one PR at a time, never while a pipeline is running. The coordination around it (who merges next, which PR waits for which) happens in agent-chat, the tool we build, where the agents mention each other and the reviewer answers in the same room. Does Lanewright's integration PR get its own CI run on the combined state, or only per item?

Collapse
 
huzair_saputra profile image
Huzair Saputra •

Good question, and fair. Yes, it’s two layers. Every item goes into the integration branch through its own PR, and only after CI is green on that PR’s head commit. At the end of the sprint there’s one draft PR from the integration branch to the base branch, and the CI result on its head commit goes into the description. So the combined state does get a full CI run before a human approves the merge.

The limit is simple though. If your tests don’t cover how items interact, the combined run won’t catch it either. Your nine PRs are exactly why the merge to the base still needs a person to sign off.

Collapse
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

Makes sense, and agreed on the limit. That gap is exactly what bit us: each PR green on its own, the combination failing on interactions nobody had a test for. We now run the full suite on the combined tree before merging, adding the queued PRs one at a time, and then merge them into staging in that order, each waiting for its own pipeline. It helps us identify which addition first exposed the interaction. Do you rerun the suite after each item lands, or only on the final PR?

Thread Thread
 
huzair_saputra profile image
Huzair Saputra •

Honestly, only on the final one. Each item PR gets CI on its own head SHA, but that head doesn't have to include the items that landed before it. So two items can both go green and still break each other, and the first place that's guaranteed to show up is the single integration PR at the end of the sprint. CI has to pass on the combined tree there before anything reaches the base branch, so it does get caught. By then three or four items are already in, though, and you're back to guessing which pair did it.

Your queue fixes exactly that part. Adding the PRs one at a time gives you the bisect for free. The cheap version for us would be making each item branch merge the current integration branch and pass CI again before it can land, which is basically GitHub's "require branches to be up to date" rule. That's one extra pipeline per item, and for small sprints I think it's worth paying.

How long does your queue take to drain on a busy day? With full suites that's the part I'd worry about.

Thread Thread
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

On our busiest day in the last two weeks (Nov 5), 39 PRs went into staging one at a time, from 1:20 am to 11:08 pm BRT. Each merge runs the staging pipeline, full suite plus deploy: median 35 min, p90 44 min across 140 successful runs from Nov 3 through Nov 11. Those timestamps show merges spread across the day; they don't tell us how long any one PR waited. To be precise about what runs: our rule is that before each merge, the full suite runs on the tree staging would have with that PR, so each queued PR is tested on top of the one before it. Staging runs it again after the merge. On "require branches to be up to date": the extra runs are the trade-off I'd want to measure. How often does the final integration run send work back for fixes?

Thread Thread
 
huzair_saputra profile image
Huzair Saputra •

I went and counted, since I hadn't before. In the 42 runs of the harder benchmark the final integration run never sent anything back. Not one red run on the integration branch. One got cancelled and rerun, and that was it.

That's less impressive than it sounds, for two reasons.

The task is four items that depend on each other in order, so every item branch already had the earlier items in it when it merged. I checked all 125 merges after the first one in each sprint and not one was missing an earlier item. Per-item CI was already testing the combined code. Your case, nine independent PRs landing on the same base, never came up. So this benchmark can't tell you how often the final run catches an interaction.

And green didn't mean correct. CI runs the tests the agents wrote. The hidden acceptance suite, run on that same combined tree, passed in full in only 7 of 42 runs. Mostly the integration run told us the agents' tests agreed with each other.

It did show one thing. Without the workflow, 12 of 21 runs never ran CI on the integration branch at all. With it, 1 of 21 skipped it.

So I'd still pay for "up to date" when PRs are independent, and skip it when the items already stack. At a 35 minute median that's a real cost per PR. Out of your 39 a day, how often does the queue actually stop on a failure?

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@huzair_saputra Agreed on the split: a missed Definition-of-Done item counts as a failed run here, and no broken code was merged. Harder tasks are the test that decides whether the extra roles earn their cost.

Collapse
 
huzair_saputra profile image
Huzair Saputra •

Thanks, agreed on all of it. The harder task is the next test I want to run properly, and I will publish the result either way.

Collapse
 
crisstiank profile image
Cristian K •

The useful distinction here is that the skill seems to improve completeness and consistency, not make agents follow a process they would otherwise ignore; the fixed Definition of Done and PR-head CI checks make that a much clearer result. The Codex limitation is especially important: a pre-commit ownership check can catch a boundary violation, but it cannot prevent the work or side effects beforehand. Did you track how often Codex crossed a region before the check caught it?

Collapse
 
huzair_saputra profile image
Huzair Saputra •

Thanks, and you're right about the limit: on Codex the check reports rather than blocks, so it only sees the result.

To answer directly: no, I didn't track attempts. I scored the final state from git and the GitHub API, and the number there is 0 commits outside a region in all 12 runs, Codex included. That tells you nothing about how often an agent touched a file it shouldn't have and then reverted it, or about side effects that never reach a commit (a generator rewriting files, for example; the BE-2 task has one on purpose). The check also lists untracked and staged files, so it catches a crossing that was never committed, but I didn't log how many times it fired.

That is the gap I'd close next: record every failed ownership check per run, per agent, so "how often does it cross before the check" becomes a number instead of a guess.

Collapse
 
goshtasb profile image
Ashkan Goshtasb •

The rule that the reviewer wrote neither the code nor the tests is the piece I'd keep even without the rest of the process. Is there anything that checks the backlog item itself before the implementer picks it up? That's the gap I've been working on with Open Council: the seats argue the spec out to sign-off and a human approves it before any role touches the repo. It would sit in front of something like this rather than replace it. github.com/goshtasb/OpenCouncil

Collapse
 
huzair_saputra profile image
Huzair Saputra •

Agreed on the reviewer rule; it's the part I'd keep too.

On checking the item before it's picked up: yes, a smaller version of that exists. The sprint tools stop at a readiness gate. A separate free skill, Backlog Check, reads the backlog against the SRS (acceptance criteria present and testable, story points, every item traced to a requirement, placeholders and vague wording flagged) and writes a report with a verdict, a SHA-256 fingerprint of the backlog and a sign-off block that only a human fills. If the report is missing, stale (backlog edited after the check), not ready, or unsigned, the sprint asks before starting. It's mostly mechanical checks plus a checklist, not a debate between roles.

So I'd say it's the opposite shape to yours: mine checks that a written backlog is ready, yours argues the spec out before there is one. They look complementary. I haven't tried Open Council, so I can't compare them beyond that.

Backlog Check lives here: github.com/huzairsaputra/lanewrigh...

Collapse
 
goshtasb profile image
Ashkan Goshtasb •

The fingerprint is the part I like most. Tying the sign-off to a hash of the backlog means a quiet edit after approval can't slip through, and I don't have that yet. Right now my sign-off covers the spec as it stood when the human approved it, and nothing stops it drifting afterwards.

Agreed on the shape, too. Yours checks that what's written is ready, mine argues about what should be written. The case I'd want to see is a backlog that passes every mechanical check but asks for the wrong thing, which is the gap I built the council for.

Thread Thread
 
huzair_saputra profile image
Huzair Saputra •

That gap is real, and I have data on it now. In the harder benchmark the tickets only implied an authorization rule. Plenty of runs got the process right and still failed the hidden tests, mostly on that rule, because nobody had written it down. A checker that reads what's written can't flag what isn't there. A human sign-off is the only guard I have for that, and humans miss it too.

So I think our two shapes stack well. Let the council argue the spec into shape, write the agreed version into the SRS and the backlog, and then the fingerprint pins that version. Any edit after sign-off shows up as a mismatch before a sprint starts.

The fingerprint itself is cheap. It's a SHA-256 of the backlog file with CRLF read as LF, so Windows and Linux agree, and the sprint side recomputes it before it starts. One side effect I hit this week is that you can't touch a signed backlog anymore, not even to add an owner's name to an item. That turned out to be the point, I just had to record those assignments in the handover instead.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Two agents on one file is the failure to fix first. The number to read is whether the extra roles cut failed runs enough to pay for themselves. Fewer failed runs would be the proof, not a longer chain.

Collapse
 
huzair_saputra profile image
Huzair Saputra •

Fair point, and I'd rather agree than defend the chain. From my data, including where it doesn't prove your point yet:

Two agents on one file: roles work in separate worktrees and each has a write region, enforced by deny rules in Claude Code and by a check before every commit elsewhere. In the benchmark no commit landed outside its region, with or without the skill. So the failure never happened in either arm; the task couldn't tell them apart. That is not proof the enforcement works.

Failed runs: I can't claim what you're asking for. When the Definition of Done is written out, a strong model already follows most of it: all 26 merges waited for green CI on the PR head, in all 12 runs. The difference was completeness, 6 of 6 runs with the skill against 3 of 6 without, and every miss was reports and the sprint summary not being committed, not broken code. The skill costs about 1.5x more and 1.4x the time in Claude Code. So the extra roles made the process complete, but I have no evidence yet that they cut failed runs.

What would count as proof, as you say: fewer runs failing hidden acceptance tests per task, with and without the roles, on far more than 3 runs per arm. I'd like to run that. Until then please read my numbers as the cost of the extra roles, not as payback.

Raw data and the scoring script are public, so you can check them: github.com/huzairsaputra/lanewright (bench/fair-2026-11-06)

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@huzair_saputra 6 of 6 against 3 of 6 is the number I would have asked for first. Every miss was a report or the sprint summary left uncommitted, not broken code, so the roles bought completeness. At 1.5x the cost, that is the number to weigh, and your failed-run count is the one still to measure.

Collapse
 
huzair_saputra profile image
Huzair Saputra •

It depends on how we define a "failed run":
• If it means missing a 'Definition of Done' item: Without the pack, 3 out of 6 runs failed (due to uncommitted reports/summaries). With the pack, 0 out of 6 failed.
• If it means broken work/code: Neither side produced a failed run. All 12 runs merged successfully via PR after CI went green, and no one wrote outside their region. The task was likely too easy to show if extra roles prevent these failures.

For broader context, across 22 published runs with the pack: 20 finished successfully and 2 safely paused for human decision.
I’ve detailed the full tally and its limitations in the repo. Moving forward, I plan to test this on harder tasks with more runs and lock down the exact definition of a "failed run" before analyzing the results.