Batch prompting (Cheng, Kasai & Yu, 2023) is one observation: put k independent items in a single call and the shared instruction and exemplars are paid once instead of k times. Per-item input cost goes from P + q to P/k + q. That is the whole technique.
Every number computed live: https://dev48.infy.uk/prompt/day68-batch-prompting.html
The saving is exact, and capped twice
Cap one is k, and it is not an approximation: batching k items captures precisely 1 − 1/k of the available saving, measured residual 1.1e-16, depending on nothing else — not the prompt, not the task, not the model. k=4 already holds three quarters of it, and the journey from 8 to 64 is worth eleven points of a saving you mostly had.
Cap two is content. The most batching can ever save is P/(P+q+a), the share of a single call that is shared overhead. With a 180-token instruction plus four exemplars that is 84%. With a one-line instruction it is 47%, and no batch size improves on it. "Batching cut our bill 5×" is a statement about how bloated the prompt was.
The other side of the ledger has no cap
k answers have to be re-attached to the k questions they belong to, and the obvious way is by position. Now let the response omit one — not garble it, omit it.
const parsePositional = (lines, k) => range(k).map(i => lines[i] || null);
Everything after that point shifts up by one, so one missing line corrupts up to k−1 others. At k=16 that is 22.7% of items carrying another item's answer across 43.3% of batches, and not one of them looks like a wrong answer. They look like answers.
The free control I expected to win
Count the lines; if you did not get k of them, run the batch again. One line, no format change, no parser change. It works: 61.5% back to 74.0%, the perfect-channel number to within a tenth. I built the page around it.
| strategy at k=16 | accuracy | tok/item | misaligned | silent batches |
|---|---|---|---|---|
| never batch | 79.8% | 214.6 | 0.0% | 0.0% |
| positional, unchecked | 61.5% | 45.2 | 22.7% | 43.3% |
| tagged, unchecked | 71.2% | 48.2 | 0.3% | 4.0% |
| positional + count check | 74.0% | 92.2 | 1.3% | 4.0% |
| tagged + re-ask the gaps | 74.0% | 55.4 | 0.3% | 4.0% |
Identical to two decimal places, at 40% less. The mechanism is observability, not reliability: a count check knows something is missing and cannot know what, so its only available repair is batch-sized — 9.3% of items dropped into single calls and 60% of batches re-run, against 3.4% of items re-asked. You pay for the repair, not the detection.
By k=64 the safety net has stopped batching. 66.7% of items have fallen back to single calls, after paying for the failed batch first, at 342.8 tokens per correct answer against 269.1 for never batching at all. Its accuracy rises over that range, which is the tell. A strategy whose accuracy improves with batch size is not batching.
Two claims I had to take back
I wrote "index your answers, it is never wrong to do". False in 3 of 45 grid configurations, all at small k. A printed index introduces a failure positional parsing does not have — the index itself can be wrong — and that rate does not depend on k, while the cascade it prevents grows with k. At k=4 with a low slip rate, positional misaligns 0.08% of items and tagged misaligns 0.25%. By k=16 the tag removes 98.7% of the misalignment and the argument is over.
And a count check is not sufficient, for combinatorial rather than unlucky reasons. A dropped answer removes a line; an answer arriving over two lines adds one. Every assignment of {clean, dropped, split} to k items was enumerated for k ≤ 7 — 3,279 of them — and 608 come back with exactly k lines and a scrambled interior. The smallest case is k=2.
Part of a from-scratch series — one prompting technique a day, measured rather than described: https://dev48.infy.uk/promptfromzero.php
Top comments (0)