Sixty-six entries in this series changed the prompt. CoT-decoding (Wang & Zhou, 2024) changes nothing about it. Branch the top-k first tokens instead of taking the argmax one, greedy-decode each, and several of those branches turn out to contain a chain of thought no instruction asked for. Then choose between them by the confidence gap over the answer span, Δ = p(top-1) − p(top-2).
Greedy scores 39.6%. Max-Δ scores 75.9%. And the free control — take the longest branch, no logprobs, one line — scores 76.2%.
Every number computed live: https://dev48.infy.uk/prompt/day67-cot-decoding.html
The null result comes first, because it is the finding
The control exists because a technique with dependencies has to beat the version without them, not just beat doing nothing. Reasoning paths are long and direct answers are short, so:
longest: b => argmaxBy(b, x => x.tokens).answer, // free, no logprobs
deltaAnswer: b => argmaxBy(b, x => x.deltaAnswer).answer, // CoT-decoding
Dead level. Logprob access, answer-span localisation and per-token arithmetic bought nothing over argmax(len). That is pinned:
assert(acc(longest) >= acc(deltaAnswer) - 0.02) at the defaults; // the null result
Writing a test around your own null result is the only way it survives the next edit.
Where it stops being true: the slope, not the level
| k | max Δ | longest |
|---|---|---|
| 8 | 72.8% | 75.2% |
| 30 | 82.2% | 76.3% |
"Longest" saturates at the accuracy of a reasoning branch — once it reliably finds one, more branches give it nothing new. Δ keeps climbing, because it discriminates between reasoning branches. The value of the confidence signal is in the slope.
The other crossover is ρ, how much of the correctness margin the gap actually carries. Below about 0.5 max-Δ is worse than picking the longest branch; above it, better. Every published Δ result is implicitly a claim about ρ for that model on that task.
The selector that wins nearly everywhere is neither: Δ-weighted voting at 82.2%, which reads both signals. Majority voting fails when the errors agree (78.3% → 52.9% across the sweep) while max-Δ barely moves, because it never looks at the other branches. Δ-ranking fails when the confidence is uninformative: at ρ = 0 it is 56.5%, exactly picking at random, while majority sits flat at 74.0%.
Averaging Δ over the whole path is worse than the free control
deltaPath: (dRea * reaTokens + dAns * ansTokens) / (reaTokens + ansTokens)
75.9% over the answer span, 67.5% over the path. The dilution is not a coefficient anyone tuned — it falls out of three answer tokens against sixty reasoning ones, and it gets worse the better the reasoning is.
The assertion I got wrong
I asserted that a constant added to every Δ is invisible to every selector, since it cannot reorder an argmax. One of nine cases failed, and the engine was right. A weighted sum is not shift-invariant: adding c adds c × (number of votes) to every answer's total, so overconfidence is really a knob that turns Δ-weighted voting into plain majority. Unclipped, +0.25 leaves max-Δ at exactly 75.9%, bit for bit, and moves the weighted vote 82.2% → 80.5%, converging at 77.6%.
The ordering claim is not safe either once the scale has an edge to pile up against. Clip Δ at 1, as a probability gap must be, and the same +0.25 costs 6.3 points with 75.2% of questions ending in a tie at the top. The tie breaks to the lowest rank. The lowest rank is the greedy branch you branched to escape.
Part of a from-scratch series — one prompting technique a day, measured rather than described: https://dev48.infy.uk/promptfromzero.php
Top comments (0)