Five Prompts Beat Five Samples: 21/24 vs 15/24 on the Identical Five-Call Bill

Two methods, same model, same 24 questions, same budget of five calls per question.

Self-consistency — one prompt, five samples at temperature 0.3, majority vote — scores 15/24.

An ensemble of five different prompts, plurality vote, scores 21/24. The best single prompt on its own scores 17/24.

👉 Live, with the correlation dial and every aggregation rule on sliders: https://dev48.in/prompt/day83-prompt-ensembling.html

These are not the same technique

Self-consistency varies the sampler's dice. Every one of the k samples read identical words, so the vote cancels sampling noise and nothing else.

Prompt ensembling varies the prompt: a different phrasing, a different output format, the options listed in a different order. The members can disagree for a reason, and a vote only buys you something when the voters can disagree for a reason.

The ceiling makes the difference concrete. Push self-consistency to k = 301 and it lands on 17/24 — and not approximately. At k = 301 it reproduces that one prompt's own greedy pick on 24 of 24 questions. The plurality over samples converges to the prompt's most likely answer, so the prompt is the ceiling. k buys stability, never a second point of view.

The engine simulates nothing about language

24 arithmetic questions are DATA. Each has four options: the true value, an off-by-one, a digit swap, and a rounded figure. The in-page self-check re-evaluates 17 × 24 and the other 23 rather than trusting the stored answer.

A "prompt variant" is a scoring profile over those four options — how strongly it is drawn to the truth, to each of the three traps, and to whatever is listed first. Each question is tempting for one particular trap. A member goes wrong where its weakness meets a matching question, which is what makes two variants fail on different questions rather than the same ones.

Noise is a hash of (seed, member, question, option), not a stream. Reordering the member array therefore cannot change what any member answers, and the verifier proves it across all 120 permutations of five members on all 24 questions.

prompt variant	accuracy
plain ask	17/24
options as a table	17/24
work it out step by step	16/24
answer with the value only	11/24
same ask, options reversed	15/24
plurality over all five	21/24
confidence-weighted	22/24
Borda (rank aggregation)	23/24

Mean pairwise error correlation across those five: −0.144.

The finding: an ensemble only pays when the errors are uncorrelated

Same machinery, same five members, same 24 questions, same seed, same 5× cost. One dial: blend each member's profile toward the first member's, so the set slides from five genuinely different prompts to five rewordings of one.

blend	error correlation	best single	ensemble	gain	agreement when wrong
0.0 — five different prompts	−0.144	17/24	21/24	+4	53.3%
0.2	−0.079	18/24	22/24	+4	60.0%
0.4	0.000	18/24	19/24	+1	64.0%
0.6	0.172	17/24	18/24	+1	73.3%
0.8	0.346	17/24	17/24	0	80.0%
1.0 — five rewordings of one	0.508	17/24	16/24	−1	87.5%

The gain does not merely shrink. It crosses zero and goes negative. Cost does not move.

The question to measure before shipping an ensemble is not "are my prompts different?" It is "do my prompts fail on different questions?"

A majority of biased members is confidently wrong

That last column is the part that should worry you.

member set	error correlation	best single	ensemble	agreement when wrong
5 different prompts	−0.144	17/24	21/24	53.3%
5 rewordings of one prompt	0.644	17/24	16/24	92.5%
5 identical calls	1.000	17/24	17/24	100.0%

A diverse ensemble that is wrong is wrong about 3-to-2 and looks shaky. A correlated ensemble that is wrong is wrong 5-to-0 and looks certain. The correlated set is less accurate and more convincing about it — so a gate that routes high-agreement answers past review fails in exactly the same direction the ensemble does.

And the bottom row is the limit case. Five identical calls at temperature zero return the single prompt's own sheet under 12 of 12 rule × tie-break combinations, for 5× the bill. If your "variants" are politeness edits, that row is what you bought.

The tie-break nobody writes down is order-dependent

Five members over four options tie more often than people expect. Somebody has to decide, and in most code nobody decided: a loop that keeps a running maximum returns whichever tied option the first member voted for.

Over all 120 permutations of the five members, that default returns different answers on 2 of 24 questions. Listing the same five members in reverse takes the sheet from 21/24 to 20/24. Nothing about the model, the prompts or the seed changed — only the order of an array.

Breaking ties on a property of the options instead (lowest listed position, summed confidence, Borda score) is invariant across all 120 orderings. It costs nothing.

k is not a slider you push right

k	cost	plurality	questions decided by tie-break
1	1×	17/24	0
2	2×	18/24	12
3	3×	20/24	2
4	4×	18/24	9
5	5×	21/24	2

Accuracy goes 17 → 18 → 20 → 18 → 21. Adding a member is not a safe move; a weak member's votes land somewhere. And an even k sends half your sheet to the tie-break: k = 2 decides 12 of 24 questions by policy rather than by vote.

If you ensemble, use an odd k.

And no aggregation rule wins everywhere

Same engine, seven seeds:

seed	best single	plurality	confidence-weighted	Borda
20260083	17/24	21/24	22/24	23/24
7	19/24	20/24	19/24	21/24
101	18/24	19/24	19/24	22/24
4242	20/24	22/24	20/24	22/24
90210	18/24	21/24	20/24	22/24
31337	17/24	20/24	21/24	22/24
555	17/24	19/24	21/24	21/24

Plurality beats confidence-weighted on three seeds and loses on three. One run does not establish an ordering between them.

Borda is at least as good as both on 7 of 7, which is the strongest claim this page will make for any rule — and it is the only rule with a cost that is not in the call count. Borda needs every member to rank all the options, which is more output tokens per call, and a member that will not rank reliably cannot be in a Borda ensemble at all. It is also the only rule that can elect an option nobody put first: five members whose first choices are 0, 0, 1, 1, 3 elect option 2, the universal second choice, with zero first-place votes. A plurality vote cannot see it.

What to actually do

1. Write variants that fail differently, not variants that read differently. Change the format, the option order, the reasoning demanded. Politeness edits produce the 5-identical-calls row.
2. Fan out in parallel, or the latency is k times too.
3. Pick an aggregation rule on purpose, and write the tie-break down.
4. Measure, on a held-out set, before shipping: every single variant's accuracy, the pairwise error correlation, the agreement on the wrong answers, and accuracy at k = 1, 3, 5.

If the ensemble is not beating your best single variant, or the correlation is high, ship k = 1 and keep the money. k prompts cost k times as much for a gain you have to measure rather than assume.

186 verifier asserts over 120 member permutations, 42 in-page checks, 0 failures. Vanilla JS, one file, no build step, no network.

👉 https://dev48.in/prompt/day83-prompt-ensembling.html