Last quarter, one of my client's team hit an inflection point that almost broke our engineering velocity.
We had fully integrated autonomous coding agents into our daily workflow. Code was being written faster than ever. But our pull request queue quickly turned into an absolute nightmare. We went from reviewing 15 human-crafted PRs a week to facing down 60+ agent-generated PRs each sitting at +800 lines of code.
Human code review became the ultimate bottleneck. Senior engineers were spending four hours a day clicking through massive diffs. Worse, "PR fatigue" kicked in: engineers started glossing over complex logic, leaving superficial "LGTM!" comments just to clear their queues. Predictably, subtle memory leaks, race conditions, and breaking API changes started sneaking into staging.
We realized very quickly: You cannot review agent-generated diffs with human-only review bandwidth.
Instead of dumping AI code directly onto human reviewers, we built a Multi-Agent PR Tribunal an automated CI pipeline where four specialized agent personas attack, audit, and verify every diff before a human ever sees it.
Here is how we built it, how it runs in our pipeline, where it broke, and why this pattern saved our team.
The Architecture: The Multi-Agent PR Tribunal
A single LLM prompt asking "Does this PR look good?" is useless. It suffers from confirmation bias and gives generic praise.
To get real rigor, you have to force agents into adversarial, highly specialized roles. Our tribunal consists of four distinct agent personas that run in parallel on every pull request:
┌────────────────────────┐
│ Pull Request Opened │
└───────────┬────────────┘
│
┌───────────────────────┼───────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Security Agent │ │ Performance Agent│ │ Architecture/API │
└─────────┬────────┘ └─────────┬────────┘ └─────────┬────────┘
│ │ │
└───────────────────────┼───────────────────────┘
│
▼
┌───────────────────────────┐
│ Dynamic Verification Agent│
│ (Runs Executable AST) │
└─────────────┬─────────────┘
│
▼
┌───────────────────────────┐
│ Consensus Engine │
└─────────────┬─────────────┘
│
┌──────────────────┴──────────────────┐
▼ ▼
[Unanimous Pass / Minor] [Blocking Disagreement]
│ │
▼ ▼
Surfaced to Human Review Blocked & Routed Back
(With Audit Summary) To Author Agent
The Security Audit Agent: Cold, paranoid. Scans strictly for OWASP top 10, unsanitized inputs, auth bypasses, and secret leaks.
The Performance & Memory Leak Agent: Focuses on algorithmic complexity ((O(n^2)) database queries), missing indexes, unclosed stream handles, and memory leaks.
The API & Architectural Compatibility Agent: Verifies breaking changes in public contracts, schema migrations, and module boundaries.
The Dynamic Verification Agent: Actually executes the proposed diff inside an isolated container, mutating edge-case inputs to verify runtime behavior.
1. Concrete Tribunal Configuration
Here is how we structure our reviewer agent prompts and schemas inside .github/tribunal/. We enforce JSON schemas so the agents cannot return hand-wavy conversational prose.
.github/tribunal/schemas/review-output.schema.json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"agentPersona": { "type": "string" },
"verdict": { "type": "string", "enum": ["APPROVE", "REQUEST_CHANGES", "NEEDS_HUMAN_ELEVATION"] },
"confidenceScore": { "type": "number", "minimum": 0.0, "maximum": 1.0 },
"findings": {
"type": "array",
"items": {
"type": "object",
"properties": {
"filePath": { "type": "string" },
"lineRange": { "type": "string" },
"severity": { "type": "string", "enum": ["CRITICAL", "MAJOR", "MINOR"] },
"ruleId": { "type": "string" },
"description": { "type": "string" },
"suggestedFix": { "type": "string" }
},
"required": ["filePath", "lineRange", "severity", "ruleId", "description"]
}
}
},
"required": ["agentPersona", "verdict", "confidenceScore", "findings"]
}
.github/tribunal/prompts/performance-agent.md
You are an uncompromising Performance & Systems Engineer auditing a Pull Request diff.
Your ONLY goal is to reject code that introduces runtime bottlenecks, memory leaks, or unoptimized I/O.
RULES:
1. Do NOT compliment the author or comment on code style.
2. Search for N+1 ORM queries, unindexed queries, blocking event loops, and missing connection pools.
3. If a loop contains an `await` call to an external network service, you MUST flag it as CRITICAL unless wrapped in batch concurrency primitives.
4. Output MUST strictly match `.github/tribunal/schemas/review-output.schema.json`.
2. The Consensus Engine: Arbitration Script
Once all agents complete their independent reviews, our consensus engine aggregates the results. If any agent flags a CRITICAL issue, or if confidence drops below 0.85, the PR is automatically blocked before a human engineer ever gets pinged.
Here is a simplified version of our production consensus script (scripts/tribunal-consensus.ts):
// scripts/tribunal-consensus.ts
import * as fs from "fs";
import * as path from "path";
interface Finding {
filePath: string;
lineRange: string;
severity: "CRITICAL" | "MAJOR" | "MINOR";
ruleId: string;
description: string;
suggestedFix?: string;
}
interface ReviewOutput {
agentPersona: string;
verdict: "APPROVE" | "REQUEST_CHANGES" | "NEEDS_HUMAN_ELEVATION";
confidenceScore: number;
findings: Finding[];
}
function evaluateConsensus(resultsDir: string) {
const files = fs.readdirSync(resultsDir).filter(f => f.endsWith(".json"));
const reviews: ReviewOutput[] = files.map(f =>
JSON.parse(fs.readFileSync(path.join(resultsDir, f), "utf-8"))
);
let hasCritical = false;
let blockReasons: string[] = [];
let totalFindings = 0;
for (const review of reviews) {
console.log(`[Tribunal] Processing ${review.agentPersona}... Verdict: ${review.verdict} (Confidence: ${review.confidenceScore})`);
for (const finding of review.findings) {
totalFindings++;
if (finding.severity === "CRITICAL") {
hasCritical = true;
blockReasons.push(`❌ [${review.agentPersona}] ${finding.filePath}:${finding.lineRange} - ${finding.description}`);
}
}
}
if (hasCritical) {
console.error("\n🚨 PR BLOCKED BY TRIBUNAL CONSENSUS ENGINE");
blockReasons.forEach(reason => console.error(reason));
process.exit(1);
}
console.log(`\n✅ PR Approved by Tribunal (${reviews.length} agents, ${totalFindings} non-critical findings). Routing to Human Reviewer.`);
}
evaluateConsensus(process.argv[2] || "./tribunal-results");
3. Real-World Failure Modes: Where the Tribunal Broke
Setting this up wasn't smooth sailing. In our first three weeks, we hit three major edge-case failures that forced us to redesign the pipeline:
Failure Mode 1:
The Hallucinated Rule Vulnerability What Happened: Our Security Agent flagged an $oid string conversion in a Mongo query as a high-severity Remote Code Execution (RCE) vulnerability, completely inventing a non-existent CVE. It blocked 12 PRs in one morning.
How We Fixed It: We grounded the Security Agent with an explicit rule-verification step. Before flagging a CRITICAL vulnerability, the agent must execute a static analysis linter (like Semgrep) via tool-use to confirm the AST matches the vulnerability pattern.
Failure Mode 2:
"Praise Drift" Between Agents What Happened: When an Author Agent submitted a fix for a PR, the Reviewer Agents read the commit message ("Fixed issue reported by review"), assumed the problem was solved, and dropped their confidence threshold to automatically pass the PR without re-evaluating the diff.
How We Fixed It: Reviewer Agents are completely stateless and isolated. They receive only the git diff and the base contracts. They are never given commit messages, PR descriptions, or chat history from previous review loops.
4. Non-Trivial Terminal Execution
Here is what it looks like when a pull request runs through the tribunal in CI:
# 1. Trigger the Parallel Agent Reviewers on PR #412
$ npx tribunal-cli audit --pr 412 --out-dir ./tribunal-results
[Agent: Security] Analyzing 6 modified files...
[Agent: Performance] Analyzing 6 modified files...
[Agent: API-Compatibility] Analyzing 6 modified files...
[Agent: Security] Complete. Verdict: REQUEST_CHANGES (1 Critical, 0 Minor)
[Agent: Performance] Complete. Verdict: APPROVE (0 Critical, 2 Minor)
[Agent: API-Compatibility] Complete. Verdict: APPROVE (0 Critical, 0 Minor)
# 2. Run the Consensus Engine against the outputs
$ npx ts-node scripts/tribunal-consensus.ts ./tribunal-results
[Tribunal] Processing Security Audit Agent... Verdict: REQUEST_CHANGES (Confidence: 0.94)
[Tribunal] Processing Performance Agent... Verdict: APPROVE (Confidence: 0.89)
[Tribunal] Processing API Compatibility Agent... Verdict: APPROVE (Confidence: 0.98)
🚨 PR BLOCKED BY TRIBUNAL CONSENSUS ENGINE
❌ [Security Audit Agent] src/modules/user/service.ts:L42-L48 - Unsanitized input passed directly to raw SQL query execution. Potential SQL Injection.
# 3. Post summary comment to GitHub PR and reject merge gate
$ gh pr comment 412 --body-file ./tribunal-results/summary.md
$ gh pr edit 412 --add-label "tribunal/changes-requested"
The Verdict
Metric
|
Human-Only Code Review
|
Single AI Prompt Review
|
Multi-Agent PR Tribunal
|
|
Review Bottleneck
|
Extremely High
|
Low
|
Minimal
|
|
Hallucination Rate
|
N/A
|
High (Praise Drift)
|
Very Low (Isolated AST Audit)
|
|
Catastrophic Bug Catch Rate
|
Medium (PR Fatigue)
|
Low
|
High (Dedicated Security/Perf Personas)
|
|
Cost per PR
|
High (Human Hours)
|
~$0.05
|
~$0.40
|
My Takeaway: AI is writing code faster than human brains can parse raw text. If you're building software in 2026, relying purely on human eyes to catch bugs in agent-generated diffs is a recipe for outage alerts at 3 AM. Give your CI pipeline specialized reviewer agents with strict JSON schemas, enforce deterministic consensus gates, and let your human engineers focus on high-level system design.
### 💡 Need High-Impact Technical Content for Your Engineering Team?
I partner with developer-tooling startups, SaaS platforms, and engineering teams to translate complex infrastructure, agentic systems, and backend architecture into publication-grade technical writing.
Whether you need deep-dive architecture essays, hands-on developer tutorials, or technical counter-narratives:
📩 Email: abhishekninja2018@gmail.com
💼 LinkedIn: linkedin.com/in/abhishekninja
🐦 X (Twitter): @AvishekBanzzov
✍️ Medium: medium.com/@abhishekninja2018
💻 Dev.to: dev.to/abhishekninja_writer
🛠️ Capabilities: Long-form Technical Essays | Hands-On Tutorials | Developer Tooling Deep-Dives | Technical Counter-Narratives
Top comments (0)