"Ask the model to check its own answer" is one of those techniques everybody repeats and nobody measures. So I measured it, on 4,000 generated problems where the ground truth is known by construction.
The first version of my own page got the measurement wrong, and the way it was wrong is the whole article.
Live, all 4,000 runs in-browser: https://dev48.infy.uk/prompt/day64-self-verification.html
The setup
Word problems with an exact answer: rate * boxes + extra. A solver with a dial-able error rate. A verifier that does not re-solve — it reconstructs a masked given from the candidate answer and checks it comes back:
// given rate and boxes, and the model's answer, recover 'extra'
const extra = answer - rate * boxes;
const ok = Number.isInteger(extra) && extra >= 0 && extra === trueExtra;
That asymmetry is why self-verification can work at all. Solving is search. Checking is arithmetic. When the two are genuinely different operations, the second one can catch the first.
And it does: 100% of the "forgot the extra term" slips are caught, and 100% of off-by-one answers reconstruct to a fraction, which is an instant reject.
Where the measurement went wrong
The obvious question is "when does verification stop helping?" — and the obvious way to answer it is to plot accuracy with the verifier against accuracy without.
Except the moment you filter, "accuracy" is ambiguous. I plotted precision: of the answers the pipeline delivered, how many were right?
Precision looks fantastic. Raw solver 66.0%, filtered 97.4%. Crank the solver to 97% accurate with a poor 25% verifier and precision still reads 99.3% against a raw 96.9%.
Verification wins at every setting. Which should have been the tell: precision beats raw accuracy essentially always, because throwing away answers you are unsure about cannot lower it. I had plotted a quantity that structurally cannot cross.
The metric that can cross
You do not ship precision. You ship answers. So count what actually reaches the user:
delivered = precision × coverage
Same setting — solver 97%, verifier 25% — with coverage included:
| raw | verified | |
|---|---|---|
| precision | 96.9% | 99.3% |
| coverage | 100% | 73.7% |
| delivered correct | 96.9% | 73.2% |
The pipeline is now worse, by 24 points. A fallible verifier does not only reject wrong answers. It rejects correct ones — 25.7% of them here, exactly as configured.
Sweep the solver and there is a real crossing point, somewhere around 60% solver error. Below it, verification costs you. Above it, verification pays.
The confusion matrix, because two rates are not one
TP 2533 · FP 118 · FN 268 · TN 1081 — sums to 4000, and it should always be printed. A verifier has a catch rate and a false-reject rate, and any single number you quote hides one of them.
Sampling changes the trade, and it is not free
Reject-and-resample instead of reject-and-give-up, and coverage comes back:
| k | coverage | model calls |
|---|---|---|
| 1 | 60.3% | 2.00 |
| 5 | 98.7% | ~3.1 |
| 8 | 99.4% | 3.30 |
The call counter is attached to the real loop, not to a formula. Verification is never one extra call — it is one per attempt, plus the attempts you would not have made.
What I would keep
- Ask what the metric can do before you plot it. Precision under filtering cannot fall. A quantity that structurally cannot cross will always show your technique winning.
- Report coverage next to every filtered accuracy. Always. Without it, "we improved accuracy by filtering" is not a claim, it is a definition.
- Verification is a filter, and filters have two error rates. Budget for both.
Part of a from-scratch series — one prompting technique a day, measured rather than described: https://dev48.infy.uk/promptfromzero.php
Top comments (0)