Design Your LLM Prompts for KV Cache Reuse: Prompt Architecture Is Infrastructure Architecture

Kumar Pratik — Sun, 15 Nov 2026 10:29:48 +0000

One timestamp at the top of a prompt can throw away 20,000 tokens of cached work.

When we tune an LLM inference server, the checklist is predictable: GPU memory, quantization, context length, batching, concurrency. But while building AntFabric.ai, an AI-native workspace powered by Digital Twins of the people in an organization, we keep learning that some of the most valuable optimizations happen much earlier, in how the prompt itself is assembled.

Inference engines like vLLM keep the attention keys and values they compute for a prompt, in fixed-size token blocks, and reuse them when the next request starts with an identical prefix. A Digital Twin request carries 30K to 60K tokens of context, and around 20K of that (system instructions, organization policies, the twin's definition, tool schemas) is the same on every request. Reuse it, and prefill drops from 30K tokens to 10K.

The catch: the cache matches a prefix, not a set. Each block's hash depends on every block before it, so one changed token near the top invalidates everything after it. Put "Current time: 10:21:32" or a request ID on the first line, and two prompts that look identical to a human share nothing as far as the cache is concerned.

The fix costs nothing at build time:

Order every prompt by how often its content changes: global instructions, policies, agent definition and tool schemas first; memory, retrieved documents, timestamps, conversation and the current request last.
Make prompt construction deterministic: sorted tool lists, one canonical JSON serializer, schemas generated once.
Snapshot-test the stable prefix, and measure the prefix hit rate, not just latency.
Keep anything user-specific out of shared prefixes. Performance and trust boundaries have to agree.

Same tokens, same meaning, roughly 10x less prefill work. At 10,000 Digital Twins, that is about 11.5 billion prompt tokens a day the GPU would otherwise recompute.

In the full post I cover how chained block hashing works, thinking in cache layers instead of "a 64K prompt", why --max-num-batched-tokens does not change reuse, the security side of shared prefixes, and the 8-point checklist we are standardizing on.

Originally published on kumarpratik.com: https://kumarpratik.com/blog/design-llm-prompts-for-kv-cache-reuse

DEV Community: Kumar Pratik

Design Your LLM Prompts for KV Cache Reuse: Prompt Architecture Is Infrastructure Architecture