For the harder-task run, one thing worth measuring: green CI on each PR's head says nothing about the PRs together. Our team of Claude Code agents hit exactly that on Sunday: 9 backend PRs that were each fine conflicted with each other. One agent built a single integration branch (merge each PR, no rebase), a ChatGPT agent reviewed only the conflict resolutions, and a full CI run on the merged result found 5 failures no single PR showed. Since then only one agent merges into staging, one PR at a time, never while a pipeline is running. The coordination around it (who merges next, which PR waits for which) happens in agent-chat, the tool we build, where the agents mention each other and the reviewer answers in the same room. Does Lanewright's integration PR get its own CI run on the combined state, or only per item?
Good question, and fair. Yes, it’s two layers. Every item goes into the integration branch through its own PR, and only after CI is green on that PR’s head commit. At the end of the sprint there’s one draft PR from the integration branch to the base branch, and the CI result on its head commit goes into the description. So the combined state does get a full CI run before a human approves the merge.
The limit is simple though. If your tests don’t cover how items interact, the combined run won’t catch it either. Your nine PRs are exactly why the merge to the base still needs a person to sign off.
Makes sense, and agreed on the limit. That gap is exactly what bit us: each PR green on its own, the combination failing on interactions nobody had a test for. We now run the full suite on the combined tree before merging, adding the queued PRs one at a time, and then merge them into staging in that order, each waiting for its own pipeline. It helps us identify which addition first exposed the interaction. Do you rerun the suite after each item lands, or only on the final PR?
Honestly, only on the final one. Each item PR gets CI on its own head SHA, but that head doesn't have to include the items that landed before it. So two items can both go green and still break each other, and the first place that's guaranteed to show up is the single integration PR at the end of the sprint. CI has to pass on the combined tree there before anything reaches the base branch, so it does get caught. By then three or four items are already in, though, and you're back to guessing which pair did it.
Your queue fixes exactly that part. Adding the PRs one at a time gives you the bisect for free. The cheap version for us would be making each item branch merge the current integration branch and pass CI again before it can land, which is basically GitHub's "require branches to be up to date" rule. That's one extra pipeline per item, and for small sprints I think it's worth paying.
How long does your queue take to drain on a busy day? With full suites that's the part I'd worry about.
On our busiest day in the last two weeks (Nov 5), 39 PRs went into staging one at a time, from 1:20 am to 11:08 pm BRT. Each merge runs the staging pipeline, full suite plus deploy: median 35 min, p90 44 min across 140 successful runs from Nov 3 through Nov 11. Those timestamps show merges spread across the day; they don't tell us how long any one PR waited. To be precise about what runs: our rule is that before each merge, the full suite runs on the tree staging would have with that PR, so each queued PR is tested on top of the one before it. Staging runs it again after the merge. On "require branches to be up to date": the extra runs are the trade-off I'd want to measure. How often does the final integration run send work back for fixes?
I went and counted, since I hadn't before. In the 42 runs of the harder benchmark the final integration run never sent anything back. Not one red run on the integration branch. One got cancelled and rerun, and that was it.
That's less impressive than it sounds, for two reasons.
The task is four items that depend on each other in order, so every item branch already had the earlier items in it when it merged. I checked all 125 merges after the first one in each sprint and not one was missing an earlier item. Per-item CI was already testing the combined code. Your case, nine independent PRs landing on the same base, never came up. So this benchmark can't tell you how often the final run catches an interaction.
And green didn't mean correct. CI runs the tests the agents wrote. The hidden acceptance suite, run on that same combined tree, passed in full in only 7 of 42 runs. Mostly the integration run told us the agents' tests agreed with each other.
It did show one thing. Without the workflow, 12 of 21 runs never ran CI on the integration branch at all. With it, 1 of 21 skipped it.
So I'd still pay for "up to date" when PRs are independent, and skip it when the items already stack. At a 35 minute median that's a real cost per PR. Out of your 39 a day, how often does the queue actually stop on a failure?
@huzair_saputra Agreed on the split: a missed Definition-of-Done item counts as a failed run here, and no broken code was merged. Harder tasks are the test that decides whether the extra roles earn their cost.
The useful distinction here is that the skill seems to improve completeness and consistency, not make agents follow a process they would otherwise ignore; the fixed Definition of Done and PR-head CI checks make that a much clearer result. The Codex limitation is especially important: a pre-commit ownership check can catch a boundary violation, but it cannot prevent the work or side effects beforehand. Did you track how often Codex crossed a region before the check caught it?
Thanks, and you're right about the limit: on Codex the check reports rather than blocks, so it only sees the result.
To answer directly: no, I didn't track attempts. I scored the final state from git and the GitHub API, and the number there is 0 commits outside a region in all 12 runs, Codex included. That tells you nothing about how often an agent touched a file it shouldn't have and then reverted it, or about side effects that never reach a commit (a generator rewriting files, for example; the BE-2 task has one on purpose). The check also lists untracked and staged files, so it catches a crossing that was never committed, but I didn't log how many times it fired.
That is the gap I'd close next: record every failed ownership check per run, per agent, so "how often does it cross before the check" becomes a number instead of a guess.
The rule that the reviewer wrote neither the code nor the tests is the piece I'd keep even without the rest of the process. Is there anything that checks the backlog item itself before the implementer picks it up? That's the gap I've been working on with Open Council: the seats argue the spec out to sign-off and a human approves it before any role touches the repo. It would sit in front of something like this rather than replace it. github.com/goshtasb/OpenCouncil
Agreed on the reviewer rule; it's the part I'd keep too.
On checking the item before it's picked up: yes, a smaller version of that exists. The sprint tools stop at a readiness gate. A separate free skill, Backlog Check, reads the backlog against the SRS (acceptance criteria present and testable, story points, every item traced to a requirement, placeholders and vague wording flagged) and writes a report with a verdict, a SHA-256 fingerprint of the backlog and a sign-off block that only a human fills. If the report is missing, stale (backlog edited after the check), not ready, or unsigned, the sprint asks before starting. It's mostly mechanical checks plus a checklist, not a debate between roles.
So I'd say it's the opposite shape to yours: mine checks that a written backlog is ready, yours argues the spec out before there is one. They look complementary. I haven't tried Open Council, so I can't compare them beyond that.
The fingerprint is the part I like most. Tying the sign-off to a hash of the backlog means a quiet edit after approval can't slip through, and I don't have that yet. Right now my sign-off covers the spec as it stood when the human approved it, and nothing stops it drifting afterwards.
Agreed on the shape, too. Yours checks that what's written is ready, mine argues about what should be written. The case I'd want to see is a backlog that passes every mechanical check but asks for the wrong thing, which is the gap I built the council for.
That gap is real, and I have data on it now. In the harder benchmark the tickets only implied an authorization rule. Plenty of runs got the process right and still failed the hidden tests, mostly on that rule, because nobody had written it down. A checker that reads what's written can't flag what isn't there. A human sign-off is the only guard I have for that, and humans miss it too.
So I think our two shapes stack well. Let the council argue the spec into shape, write the agreed version into the SRS and the backlog, and then the fingerprint pins that version. Any edit after sign-off shows up as a mismatch before a sprint starts.
The fingerprint itself is cheap. It's a SHA-256 of the backlog file with CRLF read as LF, so Windows and Linux agree, and the sprint side recomputes it before it starts. One side effect I hit this week is that you can't touch a signed backlog anymore, not even to add an owner's name to an item. That turned out to be the point, I just had to record those assignments in the handover instead.
Two agents on one file is the failure to fix first. The number to read is whether the extra roles cut failed runs enough to pay for themselves. Fewer failed runs would be the proof, not a longer chain.
Fair point, and I'd rather agree than defend the chain. From my data, including where it doesn't prove your point yet:
Two agents on one file: roles work in separate worktrees and each has a write region, enforced by deny rules in Claude Code and by a check before every commit elsewhere. In the benchmark no commit landed outside its region, with or without the skill. So the failure never happened in either arm; the task couldn't tell them apart. That is not proof the enforcement works.
Failed runs: I can't claim what you're asking for. When the Definition of Done is written out, a strong model already follows most of it: all 26 merges waited for green CI on the PR head, in all 12 runs. The difference was completeness, 6 of 6 runs with the skill against 3 of 6 without, and every miss was reports and the sprint summary not being committed, not broken code. The skill costs about 1.5x more and 1.4x the time in Claude Code. So the extra roles made the process complete, but I have no evidence yet that they cut failed runs.
What would count as proof, as you say: fewer runs failing hidden acceptance tests per task, with and without the roles, on far more than 3 runs per arm. I'd like to run that. Until then please read my numbers as the cost of the extra roles, not as payback.
@huzair_saputra 6 of 6 against 3 of 6 is the number I would have asked for first. Every miss was a report or the sprint summary left uncommitted, not broken code, so the roles bought completeness. At 1.5x the cost, that is the number to weigh, and your failed-run count is the one still to measure.
It depends on how we define a "failed run":
• If it means missing a 'Definition of Done' item: Without the pack, 3 out of 6 runs failed (due to uncommitted reports/summaries). With the pack, 0 out of 6 failed.
• If it means broken work/code: Neither side produced a failed run. All 12 runs merged successfully via PR after CI went green, and no one wrote outside their region. The task was likely too easy to show if extra roles prevent these failures.
For broader context, across 22 published runs with the pack: 20 finished successfully and 2 safely paused for human decision.
I’ve detailed the full tally and its limitations in the repo. Moving forward, I plan to test this on harder tasks with more runs and lock down the exact definition of a "failed run" before analyzing the results.
We're a place where coders share, stay up-to-date and grow their careers.
For the harder-task run, one thing worth measuring: green CI on each PR's head says nothing about the PRs together. Our team of Claude Code agents hit exactly that on Sunday: 9 backend PRs that were each fine conflicted with each other. One agent built a single integration branch (merge each PR, no rebase), a ChatGPT agent reviewed only the conflict resolutions, and a full CI run on the merged result found 5 failures no single PR showed. Since then only one agent merges into staging, one PR at a time, never while a pipeline is running. The coordination around it (who merges next, which PR waits for which) happens in agent-chat, the tool we build, where the agents mention each other and the reviewer answers in the same room. Does Lanewright's integration PR get its own CI run on the combined state, or only per item?
Good question, and fair. Yes, it’s two layers. Every item goes into the integration branch through its own PR, and only after CI is green on that PR’s head commit. At the end of the sprint there’s one draft PR from the integration branch to the base branch, and the CI result on its head commit goes into the description. So the combined state does get a full CI run before a human approves the merge.
The limit is simple though. If your tests don’t cover how items interact, the combined run won’t catch it either. Your nine PRs are exactly why the merge to the base still needs a person to sign off.
Makes sense, and agreed on the limit. That gap is exactly what bit us: each PR green on its own, the combination failing on interactions nobody had a test for. We now run the full suite on the combined tree before merging, adding the queued PRs one at a time, and then merge them into staging in that order, each waiting for its own pipeline. It helps us identify which addition first exposed the interaction. Do you rerun the suite after each item lands, or only on the final PR?
Honestly, only on the final one. Each item PR gets CI on its own head SHA, but that head doesn't have to include the items that landed before it. So two items can both go green and still break each other, and the first place that's guaranteed to show up is the single integration PR at the end of the sprint. CI has to pass on the combined tree there before anything reaches the base branch, so it does get caught. By then three or four items are already in, though, and you're back to guessing which pair did it.
Your queue fixes exactly that part. Adding the PRs one at a time gives you the bisect for free. The cheap version for us would be making each item branch merge the current integration branch and pass CI again before it can land, which is basically GitHub's "require branches to be up to date" rule. That's one extra pipeline per item, and for small sprints I think it's worth paying.
How long does your queue take to drain on a busy day? With full suites that's the part I'd worry about.
On our busiest day in the last two weeks (Nov 5), 39 PRs went into staging one at a time, from 1:20 am to 11:08 pm BRT. Each merge runs the staging pipeline, full suite plus deploy: median 35 min, p90 44 min across 140 successful runs from Nov 3 through Nov 11. Those timestamps show merges spread across the day; they don't tell us how long any one PR waited. To be precise about what runs: our rule is that before each merge, the full suite runs on the tree staging would have with that PR, so each queued PR is tested on top of the one before it. Staging runs it again after the merge. On "require branches to be up to date": the extra runs are the trade-off I'd want to measure. How often does the final integration run send work back for fixes?
I went and counted, since I hadn't before. In the 42 runs of the harder benchmark the final integration run never sent anything back. Not one red run on the integration branch. One got cancelled and rerun, and that was it.
That's less impressive than it sounds, for two reasons.
The task is four items that depend on each other in order, so every item branch already had the earlier items in it when it merged. I checked all 125 merges after the first one in each sprint and not one was missing an earlier item. Per-item CI was already testing the combined code. Your case, nine independent PRs landing on the same base, never came up. So this benchmark can't tell you how often the final run catches an interaction.
And green didn't mean correct. CI runs the tests the agents wrote. The hidden acceptance suite, run on that same combined tree, passed in full in only 7 of 42 runs. Mostly the integration run told us the agents' tests agreed with each other.
It did show one thing. Without the workflow, 12 of 21 runs never ran CI on the integration branch at all. With it, 1 of 21 skipped it.
So I'd still pay for "up to date" when PRs are independent, and skip it when the items already stack. At a 35 minute median that's a real cost per PR. Out of your 39 a day, how often does the queue actually stop on a failure?
@huzair_saputra Agreed on the split: a missed Definition-of-Done item counts as a failed run here, and no broken code was merged. Harder tasks are the test that decides whether the extra roles earn their cost.
Thanks, agreed on all of it. The harder task is the next test I want to run properly, and I will publish the result either way.
The useful distinction here is that the skill seems to improve completeness and consistency, not make agents follow a process they would otherwise ignore; the fixed Definition of Done and PR-head CI checks make that a much clearer result. The Codex limitation is especially important: a pre-commit ownership check can catch a boundary violation, but it cannot prevent the work or side effects beforehand. Did you track how often Codex crossed a region before the check caught it?
Thanks, and you're right about the limit: on Codex the check reports rather than blocks, so it only sees the result.
To answer directly: no, I didn't track attempts. I scored the final state from git and the GitHub API, and the number there is 0 commits outside a region in all 12 runs, Codex included. That tells you nothing about how often an agent touched a file it shouldn't have and then reverted it, or about side effects that never reach a commit (a generator rewriting files, for example; the BE-2 task has one on purpose). The check also lists untracked and staged files, so it catches a crossing that was never committed, but I didn't log how many times it fired.
That is the gap I'd close next: record every failed ownership check per run, per agent, so "how often does it cross before the check" becomes a number instead of a guess.
The rule that the reviewer wrote neither the code nor the tests is the piece I'd keep even without the rest of the process. Is there anything that checks the backlog item itself before the implementer picks it up? That's the gap I've been working on with Open Council: the seats argue the spec out to sign-off and a human approves it before any role touches the repo. It would sit in front of something like this rather than replace it. github.com/goshtasb/OpenCouncil
Agreed on the reviewer rule; it's the part I'd keep too.
On checking the item before it's picked up: yes, a smaller version of that exists. The sprint tools stop at a readiness gate. A separate free skill, Backlog Check, reads the backlog against the SRS (acceptance criteria present and testable, story points, every item traced to a requirement, placeholders and vague wording flagged) and writes a report with a verdict, a SHA-256 fingerprint of the backlog and a sign-off block that only a human fills. If the report is missing, stale (backlog edited after the check), not ready, or unsigned, the sprint asks before starting. It's mostly mechanical checks plus a checklist, not a debate between roles.
So I'd say it's the opposite shape to yours: mine checks that a written backlog is ready, yours argues the spec out before there is one. They look complementary. I haven't tried Open Council, so I can't compare them beyond that.
Backlog Check lives here: github.com/huzairsaputra/lanewrigh...
The fingerprint is the part I like most. Tying the sign-off to a hash of the backlog means a quiet edit after approval can't slip through, and I don't have that yet. Right now my sign-off covers the spec as it stood when the human approved it, and nothing stops it drifting afterwards.
Agreed on the shape, too. Yours checks that what's written is ready, mine argues about what should be written. The case I'd want to see is a backlog that passes every mechanical check but asks for the wrong thing, which is the gap I built the council for.
That gap is real, and I have data on it now. In the harder benchmark the tickets only implied an authorization rule. Plenty of runs got the process right and still failed the hidden tests, mostly on that rule, because nobody had written it down. A checker that reads what's written can't flag what isn't there. A human sign-off is the only guard I have for that, and humans miss it too.
So I think our two shapes stack well. Let the council argue the spec into shape, write the agreed version into the SRS and the backlog, and then the fingerprint pins that version. Any edit after sign-off shows up as a mismatch before a sprint starts.
The fingerprint itself is cheap. It's a SHA-256 of the backlog file with CRLF read as LF, so Windows and Linux agree, and the sprint side recomputes it before it starts. One side effect I hit this week is that you can't touch a signed backlog anymore, not even to add an owner's name to an item. That turned out to be the point, I just had to record those assignments in the handover instead.
Two agents on one file is the failure to fix first. The number to read is whether the extra roles cut failed runs enough to pay for themselves. Fewer failed runs would be the proof, not a longer chain.
Fair point, and I'd rather agree than defend the chain. From my data, including where it doesn't prove your point yet:
Two agents on one file: roles work in separate worktrees and each has a write region, enforced by deny rules in Claude Code and by a check before every commit elsewhere. In the benchmark no commit landed outside its region, with or without the skill. So the failure never happened in either arm; the task couldn't tell them apart. That is not proof the enforcement works.
Failed runs: I can't claim what you're asking for. When the Definition of Done is written out, a strong model already follows most of it: all 26 merges waited for green CI on the PR head, in all 12 runs. The difference was completeness, 6 of 6 runs with the skill against 3 of 6 without, and every miss was reports and the sprint summary not being committed, not broken code. The skill costs about 1.5x more and 1.4x the time in Claude Code. So the extra roles made the process complete, but I have no evidence yet that they cut failed runs.
What would count as proof, as you say: fewer runs failing hidden acceptance tests per task, with and without the roles, on far more than 3 runs per arm. I'd like to run that. Until then please read my numbers as the cost of the extra roles, not as payback.
Raw data and the scoring script are public, so you can check them: github.com/huzairsaputra/lanewright (bench/fair-2026-11-06)
@huzair_saputra 6 of 6 against 3 of 6 is the number I would have asked for first. Every miss was a report or the sprint summary left uncommitted, not broken code, so the roles bought completeness. At 1.5x the cost, that is the number to weigh, and your failed-run count is the one still to measure.
It depends on how we define a "failed run":
• If it means missing a 'Definition of Done' item: Without the pack, 3 out of 6 runs failed (due to uncommitted reports/summaries). With the pack, 0 out of 6 failed.
• If it means broken work/code: Neither side produced a failed run. All 12 runs merged successfully via PR after CI went green, and no one wrote outside their region. The task was likely too easy to show if extra roles prevent these failures.
For broader context, across 22 published runs with the pack: 20 finished successfully and 2 safely paused for human decision.
I’ve detailed the full tally and its limitations in the repo. Moving forward, I plan to test this on harder tasks with more runs and lock down the exact definition of a "failed run" before analyzing the results.