DEV Community

YuhaoLin2005
YuhaoLin2005

Posted on

I Built a Dual-Pool Adversarial Review System for AI Agents — And It Actually Works

AI code review has a problem: abstract roles produce generic feedback. "Saboteur" says "add error handling." "New Hire" says "this is confusing." Useful? Sometimes. Specific? Rarely.

I built something different: a review system that uses real engineers with searchable philosophies instead of abstract roles. Linus Torvalds doesn't say "consider error handling" — he says "eliminate the special case entirely." That's not a wording difference. That's a completely different action.

The Core Idea: Two Pools, Cross-Orchestrated

Fixed Pool (Convergence)          Random Pool (Divergence)
Digital-twin matched              Web-searched fresh each time
Stability & depth                 Surprise & blind-spot coverage
    │                                    │
    └────────── Cross-orchestrated ──────┘
              explore ←→ exploit
Enter fullscreen mode Exit fullscreen mode

Fixed Pool

9 workers + 2 managers, curated to match the user's expertise, personality, and goals. Patty McCord (Netflix's former Chief Talent Officer) and Ed Catmull (Pixar's Braintrust creator) serve as managers who recruit teams per task instead of using a fixed template.

Random Pool

Fresh personas via web search each session. No preset list — the manager defines search keywords based on what the task needs. This is where the surprises come from.

How One Round Works

  1. Manager drawn from the pool
  2. Manager analyzes task → decides depth + required roles
  3. Manager recruits specific workers (2 engineers + 1 product/designer)
  4. Team reviews — each person MUST find at least 1 issue
  5. Output: [Manager] picked [A,B,C]. Found N issues. Verdict: BLOCK/CONCERNS/CLEAN

Next round: new manager, keep at most 2 previous members.

Real Validation Data

I tested this on my own PR to alirezarezvani/claude-skills (18.7K stars):

  • Round 1 (Fixed/McCord): 10 findings — structure, format, adoption gaps
  • Round 2 (Fixed/Catmull): 8 findings — clarity, edge cases, UX
  • Round 3 (Random/Spolsky+DuVander): 3 findings — positioning, first impression, output destination

The random pool found things both fixed-pool rounds completely missed. Fixed pool reviewers — who know me — were blind to how an outsider would perceive the skill.

Key Innovations vs Existing Systems

alirezarezvani adversarial-reviewer gaurav-yadav adversarial-ai-review This System
Reviewers Abstract roles Domain agents Real people + searchable philosophy
Team formation Fixed 3-template 22 agent pairs Manager-curated per task
Cross-round Rotate roles Same agent set Swap pool + manager + workers
Personalization None None Digital twin matching
Evolution Static Static Promote/demote/audit cycle

Open Source (MIT)

What I Learned

  1. Real philosophers > abstract roles. "What would Torvalds say?" produces different code than "be more defensive."
  2. Managers matter more than workers. McCord replacing one designer with another was the highest-leverage decision.
  3. Random pools catch what fixed pools can't. If you only use people who know you, you have blind spots.
  4. A system that reviews itself is essential. The review system found 14 issues with itself across 3 rounds.

Top comments (1)

Collapse
 
nazar_boyko profile image
Nazar Boyko

The "each person must find at least one issue" rule is the bit I'd worry about. On genuinely clean code that quota forces every reviewer to invent something, and invented findings are exactly the generic noise ("add error handling") you set out to escape. How do you tell a finding that's actually real from one that only exists because the persona was required to produce it? The two-pool idea is clever, I just think forcing everyone to find something works against the specificity that's the whole point.