Sixty-four techniques into a daily prompting series, few-shot prompting had always been described the way everyone describes it: pick good examples, put them in. Picking which examples got its own day. Picking what order had never come up, and it moves the number more than most rewording does.
Lu et al. (2021) found accuracy swinging from near-random to near state-of-the-art across permutations of the same examples. So I built the measurement: all 720 orderings of a 6-shot prompt, enumerated rather than sampled.
Live: https://dev48.infy.uk/prompt/day65-exemplar-ordering.html
32 points of spread on identical content. Exhaustive matters here because the interesting part is the tail — a mean over sampled orderings hides that one specific arrangement is much worse, and in production you did not sample, you picked one and shipped it.
Three biases, dialled separately
Zhao et al. (2021) isolated three, and implementing them as separate coefficients is what makes them three things rather than one vague thing:
- Recency — the last example weighs most. Measured slot influence: 0.05 at position 1 rising to 0.38 at position 6.
- Majority label — a 5-positive/1-negative prompt predicts positive 67% of the time on a balanced test set; a 1-positive/5-negative prompt, 7.5%.
- Common token — one label string is simply likelier and wins the close calls.
Set recency to zero and the ordering spread goes to exactly 0. That is the argument for separability, and it is one assertion.
Then the part that rewrote the page
I set out to demonstrate the standard caveat: contextual calibration helps but does not eliminate the ordering effect. Feed a content-free input — literally "N/A" — take whatever the model predicts as pure bias, divide it out.
It eliminated 100% of the spread. To floating point.
That is not a bug. It is the structural result, and it is sharper than what I set out to show:
A bias that is an additive constant in logit space is exactly what a content-free probe measures. Dividing it out is not an approximation — it is exact.
My three biases were all additive constants. Of course the probe caught all of them.
The second attempt failed more sharply
Fine: make the bias interact with the input. I scaled it by how ambiguous the input was, expecting a residual.
Worse. "N/A" is the most ambiguous input there is, so the probe saw that term at full strength and removed it too.
What actually survives calibration
A term that satisfies three conditions:
- It needs content to exist — it vanishes at the probe, so calibration cannot see it.
- It is order-dependent — here, it reads the last exemplar.
- It saturates rather than scaling with the input's own strength.
That third one took a fourth attempt to notice. A term proportional to x.polarity is just a rescaling of the signal: it can never outvote the signal, so it can never change a decision, so it produces no residual in accuracy at all. It has to be additive, with a sign the input does not control, saturating at a low threshold — so a weak input gets nearly the full effect while its own evidence is still tiny.
With that in place: calibration removes 97% of the spread and leaves 1.5 points. Turn the term off and it removes 100%, exactly.
So the advice is sharper than "calibrate and hope"
Not "calibration helps a bit". It is:
Calibration removes constant bias completely and interaction bias not at all.
Balance your labels — free, and the largest single effect here. Calibrate. Then still evaluate more than one ordering, because the part calibration cannot reach is the part that depends on your actual inputs, and you only get to ship one order.
The thing I would keep
I spent three model designs chasing a result I had already written down as a caveat. Each failure was informative and none of them would have surfaced if the page had just asserted the caveat and moved on. All three attempts are on the page, in the order they failed.
Part of a from-scratch series — one prompting technique a day, measured rather than described: https://dev48.infy.uk/promptfromzero.php
Top comments (0)