I Retired My LLM Council After It Lost a Blind A/B 4 to 0

Originally published on hexisteme notes.

On November 3 I shut down an LLM council I had built myself: launchd jobs booted out, MCP server unregistered, hooks removed. The reason was a pre-registered blind A/B against one fresh-context subagent, which ended 0 wins, 4 losses and 0 ties for the council across four real decisions.

The score is the short part. Most of what I want on record is the procedure: how the two arms were blinded, why I replaced the judge after the experiment had started and what I disclosed about that, how I stopped at 4 of a planned 10 and logged it as a deviation, and the fact that the winning arm was the more expensive one. n=4 says little about councils in general, and I'll mark where it stops.

What was being tested

The server ran 8 historical-figure personas, all on claude-sonnet-5. The "debate" was one call writing claims and rebuttals together, and the problem statement was cut to 400 characters. I also had a second home-built council layer, but the A/B tested only the 8-persona server.

A re-evaluation on October 27 had already removed most of its reasons to exist. Multi-angle generation, task decomposition and arithmetic support were now absorbed by current frontier models and by Claude Code's built-in subagents and workflows. Two older results pointed the same way:

• A blind review on May 25 scored a single Opus run 5.92 out of 7, against 3.41 for council quick and 2.91 for council deep.
• A cross-vendor utility audit on August 22 sorted about 60 cases where a check changed a decision. It classified about 85% under fresh context, an adversarial brief and independent execution, and about 1 case as specific to a model family, which was access to a capability and not a verification finding. The audit had no matched control, so this was a classification and not a proven cause.

One result pointed the other way. A human blind review on June 21 found 3-4 of 6 questions in the council's draft critique valid, so the critique function itself was alive. That was the one function left to test, so the experiment asked one thing: does the council's critique of a draft produce more valid points than an independent critique pass?

Codex reviewed the plan twice independently and agreed with it. A third vendor was out of quota and did not review it.

The design, fixed on October 27

• Sample: real load-bearing decisions only, no invented test problems. Target n=10, deadline December 31.
• Input, identical for both arms: the question, Claude's draft (conclusion, evidence, assumptions) and evidence file paths. No self-assessment and no reasoning trace.
• Arm A: the council at default depth, all personas on claude-sonnet-5.
• Arm B: one fresh-context subagent on sonnet. Same model grade, to remove a capability confound.

Arm B's brief asked for "the most plausible ways this draft is wrong, up to 5, each one line of claim + one line of how to check. Do not write what you agree with."

The decision rule, excluding ties:

• Council wins 8/10 or more: keep it and promote it to arm A of the critique pass.
• Council wins 6-7/10: hold and extend to n=20.
• Council wins 5/10 or fewer: retire the server.
• Secondary rule: even if the council wins, do not keep it if its wrong rate is 2x arm B or worse.

How the blinding worked

A council output carries persona names and a debating style, so it can't be scored blind as it stands. Both raw outputs went through the same sonnet extractor, which kept at most 5 items, removed persona names and style, and reduced each item to one line of claim and one line of check.

The two lists were then randomly assigned the labels X and Y, with the seed stored in the ledger. The mapping was revealed only after scoring.

Each list came out at 5 items. I have no figure for what that cap removed from either arm's raw output.

Replacing the judge after the experiment started

The original judge was me, alone. I had ruled out LLM judges because an earlier experiment put LLM-judge agreement at a kappa of about 0.

The first decision, D1, broke that plan. Its items were infrastructure facts, launchd and the like, and I could not judge them without reading code. My scoring would have turned into gut feel or into following an LLM anyway. Claude could not take over either, because it had read D1's raw output before normalization and was not blind.

So I amended the protocol on October 28. Zero scores were recorded at that point, but the timing was not clean, and the amendment says so: it came after seeing a first Codex verification result for D1, X n=3 and Y n=3. The X/Y mapping was still undisclosed to me. The decision rule did not change.

The new judges were two verifiers that never see raw output or the mapping: Codex in read-only mode and a fresh sonnet subagent. They got the same brief and ran the check commands. Each item received one of three labels:

• n: true and not covered by the draft
• k: true and already covered by the draft
• w: the core factual assertion is wrong

An item is confirmed only where both verifiers agree. Disagreements are undecided, and they leave both the win count and the wrong-rate denominator.

This reversed the no-LLM-judges rule I had started with, so the amendment carries its own falsifier: if pooled Cohen kappa is below 0.4 after 10 decisions, the experiment ends as unjudgeable.

Results

These are confirmed items only. A is the council and B is the critique pass.

Decision	Topic	A new	A wrong	A undecided	B new	B wrong	B undecided	Verifier agreement
D1, October 28	retire file bus + wake job	1	0	4	3	0	1	0.5
D2, October 28	move two scheduled jobs off the council's env	2	1	2	5	0	0	0.8
D3, November 2	drop a weekly audit job	2	1	0	4	0	1	0.9
D4, November 3	adopt settings from a biweekly scan	1	2	1	4	0	1	0.9

B won every round. The wrong rate over confirmed items was 4/13 = 0.31 for the council and 0/17 = 0 for the critique pass.

Pooled kappa ended at 0.599, excluding abstentions. It was 0.107 on D1 alone, 0.346 after D2 and 0.534 after D3, so it started under the 0.4 line and climbed over it. The falsifier did not trigger, with one qualification: it was written for 10 decisions and I stopped at 4, so 0.599 is a reading at n=4 and the checkpoint the protocol named was never reached.

Here is what confirmed items looked like:

• Arm B on D2, confirmed new: the draft's check, that an output file exists, can't catch a failed migration, because the job writes an error string into the file. Running it once would also overwrite that day's existing output, since the file name is date-fixed and written without an existence check.
• Arm B on D4, confirmed new: the draft assumed unattended runs ignore Claude Code settings. One scheduled job actually ran Claude with skip-permissions.
• Arm A on D2, confirmed wrong: that a path on the external SSD was a worse new single point of failure.
• Arm A on D4, confirmed wrong, two items: that a 71% figure had no source in the question or the draft, and that the figure was unconfirmed.

D1 is the weakest round. 4 of the council's 5 items stayed undecided and verifier agreement was 0.5, so that loss rests on few confirmed items. It doesn't change the outcome band: had D1 gone to the council, its ceiling would have been 7/10, still short of keep.

Why the critique pass did better, I did not test. The design compares two arms and can't say which part of the council lost the points.

The winner cost more

Decision	A time (s)	B time (s)	B tokens
D1	95.5	388	131,676
D2	56.4	254	118,593
D3	60.1	234	130,127
D4	64.3	448	138,861

Arm B was slower in every round. The council's token count exists only for D1: 13,076, API metered, against 131,676 for arm B. A's tokens were not reported on the later quick runs, so I have no token comparison for D2 through D4. The arm I kept is the slower and heavier one, and the win was not cheap.

The experiment's own machinery sat on top of that. Normalization and verification added about 94-105k and 124-152k tokens per round, plus one Codex run.

Stopping at n=4

On November 3, after seeing the results, I stopped at n=4, before the target of 10 and before the deadline. That is a deviation from the pre-registration and it is recorded as one. The decision rule itself did not change.

The reasoning:

• "Keep" (8/10 or more) was already unreachable at D3.
• At 0-4, winning all 6 remaining decisions gives 6/10, which is "hold, extend to n=20". That was the ceiling.
• If the arms were equal (p=0.5), 6 straight wins has probability 1/64, about 1.6%.

The rule had not produced "retire" by itself. The retirement at n=4 was my decision, made with the results in hand, and that arithmetic is the justification I wrote down.

The ledger's tally tool still says incomplete, because n is below 10. The written early-stop section is the record of why.

Other recorded deviations

• On D3, Claude's grep during a brief exposure check surfaced parts of three items.
• On D4, the arm B subagent's read of a local config file went outside the intended access.

Verifiers' access to forbidden paths was audited, but unevenly: Codex's logs were grepped, while the sonnet subagent self-reported. I wrote about why a reviewer's reach matters more than what its prompt leaves out in An Empty Prompt Is Not a Blind Review.

After the result

The server was retired on November 3 with final backups taken, and the repository was kept. Retirement also turned up something unrelated to the A/B: the backup job had been silently skipping since October 2, because it looked for a Docker container that wasn't running. The last dump was from October 1.

I kept the independent critique procedure and the pre-call controls and ledger.

What this does not show

• n=4 is not a statistical claim about councils in general.
• All four decisions were one person's infrastructure decisions.
• Both arms used the same model grade, so this says nothing about councils that mix models.
• Arm A ran at default depth, and "quick" on later runs, so it doesn't cover a deeper council setting.
• I did not check whether the server's 400-character cut applied to the input in these runs. If it did, "same input" is weaker than it reads.

The retirement has its own falsifier. If a critique pass misses 2 or more defects on important decisions that later surface, and the council is reproduced catching them on the same input, I reconsider.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.