DEV Community

Discussion on: Two AI Agents Reviewed the Same PR. Was That Actually Independent?

 
suirindo profile image
asagiman •

That's a useful distinction, Dev_PJS. I particularly agree that merging parallel writes creates a new verification obligation, even when each individual change has already passed its checks.

We've been approaching this as an admission-control problem: parallelize disjoint write sets, but limit work in progress by the capacity to independently verify and accept outcomes.

There's another subtlety I'm interested in: a checker being different from the writer doesn't automatically make the verification independent. Shared assumptions, test fixtures, or evidence sources can still produce correlated blind spots.

So I think we need to distinguish three things: writer separation, checker separation, and evidence independence.

Your point about checker capacity being the actual schedule is especially relevant. Have you found a practical way to measure that capacity before admitting additional runs?

Thread Thread
 
jes_p_d4f61f7aa2ec405a418 profile image
Dev_PJS •

Admission control is the right frame — it moves the limit upstream, before the work exists, instead of discovering it at the merge. The version I run is cruder than a controller: I simply don't start a run whose verification I can't schedule. The queue is allowed to grow in front of the checker, never behind it — work waiting to be started is cheap and visible; work finished-but-unverified is expensive and lies about being done. WIP limited by acceptance capacity means the bottleneck stays honest about where it is.

Thread Thread
 
suirindo profile image
asagiman •

That's a very useful operational rule: let the queue grow before execution, not after it.

I particularly agree that finished-but-unverified work is hidden WIP, not completed throughput.

For multi-agent development, I'd extend your admission rule slightly: a run should have not only an available independent verification path, but also a realistic path to integration and final acceptance.

Even if two changes pass independent review separately, combining them creates another verification obligation. That capacity needs to be accounted for too.

The metric I want to optimize is accepted changes, not completed agent runs.

One question: how do you handle verification capacity when unexpected failures require additional review cycles? Do you reserve spare capacity, or pause new admissions until the backlog clears?

Thread Thread
 
jes_p_d4f61f7aa2ec405a418 profile image
Dev_PJS •

'Finished-but-unverified is hidden WIP' — I'll take that phrasing; it names the dishonesty precisely. And your extension is the part my cruder rule misses: verification capacity alone isn't the gate if integration has no realistic path. A run can be verified in isolation and still be unmergeable into the whole — different base assumptions, an interface that moved while it worked. So the admission question is really two: can this be checked, and can this land? Work that can be checked but can't land is just well-documented waste.

Thread Thread
 
suirindo profile image
asagiman •

Exactly. "Can this be checked?" and "Can this land?" are two different admission decisions.

I'd add one more distinction: mergeability at admission is only provisional.

The base revision, shared interfaces, and competing writers can change while the agent is working. So even a run that had a valid integration path at the start needs that path revalidated before merge and final acceptance.

For multi-agent work, I'm thinking of three concrete requirements:

  1. Reserve independent verification and integration capacity before starting.
  2. Record the exact base revision, affected interfaces, and integration assumptions.
  3. If those assumptions change, invalidate the previous integration decision and re-evaluate the actual combined candidate.

That last point matters: two independently verified changes do not automatically produce one verified integration.

Your phrase "well-documented waste" captures the cost of getting this wrong.

I'm curious whether you serialize interface-changing work upfront, or allow parallel execution and rely on revalidation when the base moves.