Two days before my hackathon deadline, I re-read the judging criteria for
the track I was submitting to and found a sentence I'd apparently skipped
the first time through:
"...a measurable efficiency gain over single-agent baselines."
My project, RedCouncil, is an adversarial multi-agent system — five
specialized AI agents (Growth, Risk, Legal, TechDebt, Customer) debate a
business decision, cross-examine each other, and a Synthesizer produces a
severity-scored report. 68 tests passing, auth, storage, deployment, all
done. I had never once measured whether it actually beat a single model
call.
My first instinct was to bail. My second, better instinct was to ask: how
fast can I actually get real evidence for this, instead of assuming it's
either fine or hopeless?
Building the eval, fast
The fastest honest version of this isn't a new subsystem — it's reusing
everything you already have. Same scenarios, same model, one new prompt
that asks a single Qwen call to do what all five agents do combined, and a
purely deterministic scoring function — no LLM judge, just counting and set
math. Two numbers mattered most:
- Cross-agent conflicts surfaced. A single model call has no mechanism to represent disagreement between specialist viewpoints. This number is 0 for the baseline by construction, which makes it a free, structurally guaranteed data point.
- High-severity findings caught, compared head-to-head on the same 1–10 scale, same model, so the comparison isolates the architecture, not model size.
I ran it. Numbers came back. They looked good. I started building the
slide deck.
Then I actually read the results file instead of just the summary, and
found the first problem.
Bug 1: the metric that was empty by design
One of the fields I'd built, domains_missed_by_baseline, was empty in
every single scenario — which I initially read as "RedCouncil never misses
anything the baseline catches." It took a second look to realize the field
was always going to be empty, because of how I'd defined it: it only
checked domains present in RedCouncil's output but absent from the
baseline's. Since the baseline prompt forced one finding per domain every
time, that set difference was structurally guaranteed to be empty
regardless of how RedCouncil actually performed. Not a finding — an
artifact of how I'd defined the metric.
Worse, when I checked the reverse direction, the one that actually
mattered, I found RedCouncil was missing an entire domain — Growth — in 3
of 5 scenarios. My own multi-agent debate system was covering less ground
than the single dumb baseline it was supposed to beat.
Bug 2: the retry that gave up too early
Root-causing this with real trace data (not speculation) showed a
two-part failure: the coverage checker only flagged a domain as "missing"
if its findings scored above a severity threshold — so a legitimately
lower-severity Growth finding could slip through undetected. And when a
retry did get triggered, the acceptance logic only kept the retry if it
strictly improved on the previous attempt. If the model's retry attempt
failed to recover the missing domain, the system gave up and kept the
broken result — because "no better than before" and "still broken" looked
identical to the acceptance check.
Fix: lower the threshold so every domain gets checked regardless of
severity, and give the retry loop several bounded attempts instead of one,
tracking convergence properly instead of a single strict comparison.
I re-ran the comparison. Growth showed up in all 5 scenarios. I updated the
deck. I thought I was done.
Bug 3: zero, again
Then I actually opened the live product UI to record demo footage, and the
"conflicts surfaced" counter read 0 — on a scenario that, reading the
transcript, clearly had agents contradicting each other's core claims in
plain English.
Round one of debugging this found a genuine plumbing bug: one part of the
code wrote a result to a field called conflicts, another part read from a
field called conflict_count. Nobody had ever written to that second key.
Classic. Fixed it, redeployed, tested again.
Still zero.
Round two went deeper, and this is the part I actually learned something
from: the conflict detector was gating candidate disagreements through a
lexical-overlap check before it would even evaluate whether they opposed
each other. My Growth agent argues in terms of savings, cost, ROI. My
Customer agent argues in terms of churn, friction, retention. Zero
shared vocabulary — so the gate filtered the pair out before the actual
stance-comparison logic ever ran, even though the underlying disagreement
was completely real.
This is the kind of bug that's obvious once you see it and invisible until
you do: specialists with genuinely different mandates use genuinely
different words by design. A similarity gate built for a system where
everyone talks about the same thing is exactly the wrong tool for a system
whose entire premise is that the agents don't think alike.
The fix: stop inferring disagreement from vocabulary overlap. Trust the
signal the agents themselves already produce when they explicitly rebut
each other during cross-examination.
Bug 4: the one hiding inside the fix
Trusting each agent's own rebuttal signal introduced a new failure mode
almost immediately: rebuttals are often mutual. If Growth rebuts Risk and
Risk rebuts Growth in the same exchange, and each rebuttal becomes its own
conflict entry, you double-count every disagreement that goes both ways.
Caught this one before it shipped, with a simple fix: dedupe on the
unordered agent pair before counting, not the directional edge.
What actually changed by the end
Conflict detection went from a broken 0, to a plumbing fix that still
read 0, to a real but partial fix, to a detector that could finally see
disagreements that don't share vocabulary — at which point the true
conflict count roughly doubled from what the vocabulary-limited version had
found. Every one of those jumps looked like "the fix worked" right up until
the next layer surfaced.
What I'd actually tell someone building an eval harness
- A metric that's always the same value isn't a metric — check what it's structurally capable of showing before you trust it. Mine was empty-by-definition for an entire category of finding.
- If your system's whole premise is that components think differently, don't build detection logic that assumes they'll describe things similarly. That's not an edge case, it's the design working as intended running into a detector that assumes the opposite.
- Fixing your eval can be as much engineering work as fixing your product — arguably more, because a broken product fails loudly and a broken eval fails silently, as a confident, wrong number.
- Non-determinism compounds across every layer you don't pin down. Between run-to-run model variance and the layers of retry logic in the system itself, "the number" was never really one number — it was a distribution I was sampling from once and calling final. Worth deciding on purpose whether that's good enough for your use case, rather than discovering it by accident three reruns in.
RedCouncil runs on Qwen Cloud (Qwen-Max) end to end — all five debate
agents plus the Synthesizer. If you're building anything where multiple
model calls are supposed to disagree with each other on purpose, budget
real time for your evaluation code, not just your product code. Mine
needed just as much debugging.
Top comments (5)
I was particularly intrigued by the
domains_missed_by_baselinemetric and how it was initially misinterpreted due to its definition. The fact that it was structurally guaranteed to be empty highlights the importance of thoroughly reviewing our metrics and their implications. The subsequent discovery of RedCouncil missing an entire domain in 3 out of 5 scenarios serves as a great reminder to consider multiple perspectives and potential blind spots in our systems. The retry mechanism's early give-up also underscores the need for more robust error handling and convergence tracking. What strategies do you think are most effective in identifying and mitigating such blind spots in complex systems like RedCouncil?Hey, did the final deck ship with numbers from a single run in the end, or did you re-run a few times once you realized the number was really a distribution you'd sampled once?
This is a strong example of why an eval needs tests of its own. I would turn each failure you found into a small metric-contract suite before the next benchmark run: construct fixtures where a metric must be zero, non-zero, symmetric, and invariant to agent ordering; then add metamorphic tests such as paraphrasing the same disagreement with disjoint vocabulary, swapping agent labels, and duplicating reciprocal rebuttals. Also report distributions across seeded runs rather than one aggregate score, with paired scenario-level deltas against the baseline. That separates a real architecture gain from retry luck. The most valuable artifact may be the catalog of ways each metric can lie, because future changes can be checked against those known blind spots.
This really highlights something teams often underestimate: the evaluation layer becomes part of the product. It's easy to optimize prompts or agent orchestration while assuming the metrics are telling the truth, but an "always green" eval or a structurally flawed metric can send you in the wrong direction for weeks. We've seen similar challenges while building multi-agent workflows at IT Path Solutions the hardest bugs often aren't inside the agents, they're in the harness measuring them. Treating evals, retry logic, and observability as first-class engineering components usually pays off much more than another prompt iteration. Great write-up and a very relatable debugging journey.
All three bugs are the same bug in three costumes, and naming it is worth more than any single fix. Each is a check whose blind spot is correlated with the exact thing it was built to measure. The domains-missed metric was constructed from the assumption the system was supposed to violate, baseline covers every domain, so it could not see the gap it existed to find. The conflict detector assumed disagreeing agents share vocabulary, but you designed the agents to disagree across vocabularies, so it was blind in precisely the dimension the system varies. The lexical gate dropping "savings/ROI" against "churn/retention" is not an edge case, it is the design working as intended, read by an instrument that did not know the design.
That is why the bugs clustered where they did. The eval was sharp where the system was ordinary and blind where the system was novel, because it inherited your mental model, and the system's interesting behavior lives exactly where that model does not hold yet. An eval built by the same head that built the system is blind in the system's new dimensions by default, not by accident. Which is the real reason fixing an eval can cost as much as fixing the product: you are not debugging code, you are removing the shared assumptions between the checker and the thing checked, one at a time. Every one of your fixes is that same move, stop letting the eval assume what the system assumes.
One thing your last insight hands you against your first one for free: variance across seeds is a structural-pin detector. A metric that is empty by construction shows zero variance, and so does a metric that is genuinely stable. The first reads flat because it is disconnected, the second because it is precise, and they look identical until you ask why it is not moving. If you are already reporting distributions from seeded runs, a flat distribution where you expected spread is the same smell as domains_missed always returning empty, caught by the instrument you already built. Non-determinism is a free perturbation test on your own metrics, as long as you read the flat ones as suspects rather than as successes.