Modern LLM APIs cache the key-value state of a prompt prefix. Send the same opening tokens twice and the second call is billed at roughly a tenth of the input price and returns noticeably faster. This one fact reorganized how we write every high-volume prompt.
The rule
Everything stable goes first, byte-identical on every call: system rules, output schema, few-shot examples, reference tables. Everything volatile goes last: the task, the retrieved documents, the user text. The cache breaks at the first differing byte, so a single timestamp in the header poisons the whole prefix.
[ system rules ] stable, versioned ─┐
[ output schema ] stable, versioned ├─ cached prefix, ~91% of tokens
[ 12 few-shot examples ] stable, versioned ─┘
[ retrieved context ] volatile ──────────────── paid in full
[ the actual task ] volatile
Measured on one of our extraction pipelines
| prompt layout | prefix reuse | cost per 1k calls | p50 latency |
|---|---|---|---|
| task first, rules inline, timestamp in header | 0% | $31.40 | 4.1 s |
| stable prefix, task last | 91% | $6.90 | 1.8 s |
Same model, same accuracy, 78% cheaper and twice as fast, for a diff that only moved paragraphs.
Operational consequences
- The prefix is versioned like an API. Editing it is a deliberate release, because every edit is a fleet-wide cache flush and a visible cost spike. We tag prompts with a version constant and roll them like schema migrations.
- Batch queues are sorted by prefix. Requests sharing a prefix run adjacently so the cache stays warm instead of being evicted between hits.
- Hit rate is a logged metric. The API reports cached token counts per call; we graph them. A silent drop in cache hit rate has caught two accidental prompt edits before the invoice did.
- Randomness is banned from the prefix. No timestamps, no request IDs, no dict-ordering roulette. Serialization is canonical, keys sorted, for the same reason your build system wants deterministic inputs.
None of this touches quality. It is pure systems hygiene applied to a resource most teams do not know they are wasting.
Top comments (2)
"A single timestamp in the header poisons the whole prefix" is the line that bites everyone eventually. The failure we saw most was subtler than a literal timestamp, though — it was serialization nondeterminism. Two runs building the same tool schema or the same retrieved-doc blob from a dict with unstable key ordering produce byte-different prefixes that look identical when you print them, and the cache silently misses. Your "canonical serialization, keys sorted" rule is the actual fix, and it deserves the emphasis you gave it.
The operational move I'd underline for anyone skimming: making cache hit rate a graphed, alerting metric. It turns a prompt edit from an invisible correctness change into a visible cost regression the moment it ships — you caught two edits before the invoice, which is the whole game. One thing worth adding is that the cache also has a TTL, so low-QPS prompts can miss even with a perfect prefix simply because the entry expired between calls; your "sort the batch queue by prefix so hits run adjacently" trick is what saves those, and it's a genuinely non-obvious optimization. Did you find a QPS floor below which prefix-sorting stopped being worth the batching latency?
No, we did not find one, and I would rather say that than invent a number. The paper leaves TTL and eviction out of the cost model on purpose.
But your question changed my mind about our own sentence, so it is worth working out loud. Take one prefix with arrival rate λ and cache TTL T. With no batching at all, consecutive calls hit whenever the gap is under T, so the natural hit rate is 1 − e^(−λT). Prefix-sorting inside a window W can only add hits when a second same-prefix request actually lands inside that window: 1 − e^(−λW).
Put numbers in and the band collapses. With T = 300s, λ only has to clear ~0.003 req/s, one call per five minutes, for the entry to stay warm on its own. With W = 1s, λ has to clear ~1 req/s before the window groups anything at all. That is a 300x gap pointing the wrong way: any traffic dense enough for a one-second batch window to be worth its latency is already two orders of magnitude denser than the TTL ever needed.
So for time-based expiry there is no floor to find, because there is no band. Sorting pays in the other case, the one the post actually named and I did not explain: evicted, not expired. Under capacity pressure many distinct prefixes compete and yours is pushed out by other traffic rather than by the clock, and there it is your neighbours in the queue, not the clock, that decide whether you survive. It also pays when W is genuinely large, offline batch runs where the window is minutes, so W approaches T.
The honest answer to "what is the QPS floor" is therefore a shape, not a number. Below λ ≈ 1/W the sorting buys nothing and costs latency, above λ ≈ 1/T you never needed it, and if W ≪ T the two bounds cross.
Good catch on serialization nondeterminism, too. Dict ordering is the one that prints identically and misses silently, which is why it survives review.