Few-shot prompting under a token budget is a selection problem, not a template. To price the selection you need a model of what a demonstration mechanically does, and the only thing it can do is eliminate hypotheses it contradicts. So the engine is a version space: 81 possible inputs, a hidden rule drawn from an enumerated space of 256 conjunctions, bitsets intersected rather than sampled. A query is pinned when every surviving rule labels it the same way — an upper bound on what any reader could get right, because when the survivors disagree the answer is not in the prompt at all.
function vsAdd(vs, sp, xi, lab){
var b = xi * sp.Wd;
for (var w = 0; w < sp.Wd; w++)
vs[w] = lab ? (vs[w] & sp.pos[b + w]) : (vs[w] & ~sp.pos[b + w]);
}
Every number below is exact arithmetic over enumerated sets: https://dev48.infy.uk/prompt/day72-few-shot-budget.html
The probe has no parameters in it
Take one record. Write it as a demonstration. Change one word. Ending it -> YES deletes 240 of the 256 rules; ending it -> NO deletes 16. The token cost is identical on 81 of 81 records, and across all 81 the deleted count takes exactly two values at a ratio of exactly 15.00. That is where "lead with the positives" comes from, and it is a fact about the rule space rather than about writing.
Give the reader blacklists as well — the same 256 patterns plus their negations — and the identical pair of demonstrations delete 256 and 256, a ratio of 1.00. Nothing in the prompt says which space the reader is in.
The recommendation that follows from it dies
| strategy, 150 trials, 160-token budget | queries pinned |
|---|---|
| greedy, rules killed per token — nine lines | 65.69% |
| exact 0/1 knapsack, standalone values | 61.20% |
| lead with the positives | 54.43% |
| alternate the two labels | 54.43% |
| cheapest first | 45.95% |
| no examples at all | 0.00% |
"Lead with the positives" ties the folk rule it was built to beat, and on a curated pool it is the second-worst strategy on the board, beaten by taking the cheapest examples — 59.36% against 82.79%.
The knapsack losing is the sharper one. It is exactly right and still loses, because it prices each demonstration standalone and set value is submodular: two demonstrations that kill the same rules get paid twice, so the optimiser buys ten of them where a greedy that re-scores after every pick buys six better ones.
What the measurement contradicted
I expected the accuracy column to at least rank these. It cannot see any of it. The empty prompt scores 92.59% accuracy, because the labels are skewed and a reader with no information says NO to everything. A 156-token prompt of NO demonstrations scores 92.61%. That is the whole return: two hundredths of a point for 156 tokens, while the same prompt pins under one percent of the inputs it did not already contain.
Across eleven strategies the accuracy column spans four points and the pinned column spans sixty-six. If you are tuning against an accuracy number on a skewed task, it will move when you edit the prompt and it will not tell you whether the prompt taught the reader anything.
24,674 assertions in the page, 6,480 in the extracted-engine verifier. Self-contained: one file, inline CSS, no external asset.
Part of a from-scratch series — one prompting technique a day, measured rather than described: https://dev48.infy.uk/promptfromzero.php
Top comments (0)