I ran the same job past three different AI providers and had them vote on the answer, and I did it for a hobby project about a Sunday night D&D game. That sounds a tad extravagant, I know, right up until you look at what the job actually is.
What the job actually is
A tabletop campaign runs for years. Mine's been going a while and meets most weeks, and in that time the party has met hundreds of people, made dozens of promises, and learned things they've long since forgotten they know. Very little of it gets written down properly (I speak as the DM who was supposed to be writing it down).
The Dungeon Master invents a barkeep on the spot, gives him a name, and six weeks later the party walks back through that door expecting him to still exist. I've written about that particular horror before. Scout listens to the session and turns it into structured campaign facts. Who was in the room, who said what, what got promised, what the party now knows that it didn't at teatime.
What I was slow to appreciate is that every proper noun in that transcript is invented. Not obscure or niche, invented that week by one of us sat at the table, and meaningless everywhere else in the universe.
A model pulling facts out of a news article can lean on knowing what Paris is. A model pulling facts out of my Sunday night has no idea whether candlefoot is a place, a person, an illness or a poor life choice, because until three weeks ago it wasn't anything at all. There's no world knowledge to fall back on, only what was said and how carefully you listened.
Extraction is a write
That sounds like transcription, but it isn't quite. Transcription is a read, and getting it wrong produces one duff sentence that a reader notices and ignores. Extraction is a write: the facts go into a canonical store, and every answer Scout gives afterwards is drawn from that store.
So a wrong fact doesn't produce a bad answer once. It sits there being wrong, poisoning everything downstream of it, and a confidently recorded fact looks no different from a correct one. Six months later a player asks who the party bought the horses from, Scout tells them, and it's wrong, and there's no way to tell because the answer arrives as confidently as all the right ones. That asymmetry is most of the argument for me: where a bad write is permanent and a duplicate read costs pennies, redundancy is the cheap side of the trade.
Why three models rather than one better prompt
The obvious objection is that this is a prompting problem. Write a better prompt, pick the best model, spend the money once. I don't think that holds, and the reason is what the three of them actually did. They didn't fail less or more than each other, they failed in different directions. Gemini in particular kept reaching for the source material instead of the transcript in front of it, which is behaviour I'd already met in NotebookLM and had to prompt my way around there too.
That's what a panel is for. Three copies of one model at temperature zero don't give you a panel, they give you the same answer three times with a rosette on it! The redundancy only buys anything if the members are wrong about different things, and providers trained differently on different data are about the best independent-ish failures you'll get without building your own.
The voting was the easy part
Here's where I lost a weekend. Majority voting over a set of facts assumes you can tell when two facts are the same fact, and that is a much harder question than it sounds. I ended up refereeing cases by hand just to find out how hard, and here's a real one: two models extracted the same thirty seconds of play, and both produced a statement about it.
The invented names won't mean anything to you, and that's rather the point. Read them as three opaque tokens that happen to be a place, a substance and an event.
Both statements contained the same trio: kettlesteam, candlefoot and the stolen voice. Same three things, same relationship between them, unambiguously the same moment. A hard match, no argument... except each model had also bundled other facts into the same statement, and those didn't correspond at all. One had folded in who was present. The other had folded in what it cost.
So the two agree to a high degree of probability, are not identical, and are really three separate facts that happen to share a sentence. Then a second case, where one model named a thing outright and the other only implied it. Jingle Jangle is inferred rather than named, which is still true so long as the inference is correct, and there's nothing in the transcript that lets a vote-counter decide whether it was.
So do those agree? I don't think there's a clean answer, because the question is wrong. "Do these two agree" is really a question about how big a fact is, and until you've settled that, a majority vote has nothing well-defined to count. Three models can produce three statements describing the same events and still cast three different votes, purely because they chopped those events up differently. The voting mechanism is about forty lines. The fact-granularity problem is the actual project (I did say I lost a weekend).
The bill, which is not a footnote
Three inferences where a normal pipeline has one, on every session, forever. For a product whose viability rests on unit cost, that's a design decision with an invoice stapled to it. Early passes went through the local Claude backend specifically to spend a subscription I'd already paid for instead of API credit, which is every bit as cheap as it sounds, and I'd recommend it to anyone prototyping (a subscription you've already paid for is the best price there is).
Testing a panel to understand what it buys is fine. Shipping one before I know what it costs per session is not, and those are different decisions that are very easy to blur when the thing works nicely in a spike.
The roster is not the design
The three names are not the durable part of this. Anthropic, Gemini and OpenAI are the panel today, and I'd be surprised if that survives the next eighteen months: providers change, prices change, models get deprecated with a fortnight's notice, and something will turn up that's better at this specific job than any of the three currently sat on it. I suspect none of that touches the design.
What I'm testing isn't "are these the right three". It's whether a panel with a reconciliation step produces a more trustworthy write than a single call does, and whether the disagreements are informative enough to be worth paying for. If that holds, the roster is an implementation detail: swap a member, swap two, run five when the stakes are high and one when they aren't. The structure is the thing that has to be right, and the structure is what the experiment is actually interrogating. It's the same instinct as not building on one vendor's shape, designing for the role and not the incumbent.
It talks through my own modules
All of that is a good deal easier because the panel reaches all three providers through the estate's own chat modules instead of raw HTTP calls: go/chat with chat-anthropic, chat-gemini and chat-openai underneath it, the shape the extraction programme built them into.
Three providers, one interface. Changing who's on the panel is a config change and a dependency, not a rewrite, and that's why the roster gets to be disposable. It also promptly exposed a rate-limiting gap in those modules, which got filed upstream against them (by me, against me) rather than patched around locally. That's the deal you make with dogfooding: the tool you built tells you about itself, in public, at the least convenient moment.
Where it's got to
Scout is still private and still very much unfinished, and the panel is an investigation rather than a shipped feature. What the investigation produced wasn't a voting mechanism so much as the discovery that I'd been asking the wrong question for nearly two weeks. I thought the problem was picking the right model... it was working out what a fact is.
Originally published at phpboyscout.uk on 14 October 2026.
Top comments (2)
The fact-granularity problem you hit is real, and I think it also changes what "agree" should be measured on. If each model returns free-text statements that bundle several facts, the vote is over sentences, and sentences are the wrong unit. Forcing each fact down to one subject, one relation, one object (plus a flag for named versus inferred, like your Jingle Jangle case) lets you vote per triple, and the bundled "who was present / what it cost" parts stop dragging the matched part down.
Two cheap checks on whether the panel earns its three-times bill: hand-label one session's facts once, then compare (a) each single model, (b) majority of three, and (c) "at least one model says it" against that labelled set. And look at the pairwise agreement between providers on the facts that turned out wrong. If two of them make the same mistake together, a 2-of-3 vote will confidently write it into the store, which is the failure you said is permanent.
On the inferred facts: storing them with their evidence span from the transcript, and having the vote require a span from at least two members, would give you something a counter can actually check.
Did the three disagree more on which facts exist, or on how they were worded?
A combination of both! The results were wildly unpredictable and it was difficult to even get consistent results. I realised that I was asking for far too much structure put of what was natural speech. So I pivoted and went back to some good old fashioned ML using NLP to do an extraction pass over the text first to give consistent semantic meaning to the text before it went to an agent. This meant that they all started evaluating the results in a much more consistent manner because they were able to treat each word identically. It's still not perfect, but it's massively improved to a point where a full panel shouldn't be necessary for anything other than calibration (hopefully) and the more fun generative stuff