A Timestamp in Your System Prompt Is Multiplying Your LLM Bill

The problem

Open the usage dashboard of any team running LLM agents and you'll see the same shape: input tokens dwarfing output tokens, often 30-50x. That ratio isn't the model being verbose. It's your architecture re-buying the same tokens over and over.

LLM APIs are stateless. Every call starts cold and processes your entire prompt from token zero. An agent loop resends the whole conversation every turn — system prompt, tool schemas, every prior tool call and result — plus one new message.

Run the numbers on a modest agent: an 8,000-token prefix (system prompt + tools) and ~1,000 new tokens per turn (a tool call plus its result). By turn 50, each call sends 57,000 tokens, and the run has consumed 1.62M cumulative input tokens. Only 58,000 of those were new. Everything else was tokens the model had already processed — some of them 49 times.

At $3 per million input tokens, that's $4.88 per run. A thousand runs a day is ~$4,900/day, nearly all of it redundant compute. You pay for it in latency too: processing input ("prefill") is most of what you wait for before the first output token on long prompts.

Prompt caching exists precisely for this — and most teams either don't use it or silently break it.

Why it happens

Transformers process a request in two phases: prefill, where the model computes internal state (attention keys and values) for every input token, and decode, where it generates output. Prefill cost scales with prompt length. That's where your input bill and your time-to-first-token live.

Here's the exploitable property: the same token prefix through the same model produces the same internal state. So providers cache that state, keyed by your exact token prefix. On a hit, the model skips recomputing everything up to the point where your prompt diverges from the cached one.

Read that again: exact prefix. The cache is positional, not semantic. The first byte that differs invalidates everything after it — the cache has no idea your timestamp is "just metadata."

Which is how teams turn the discount off without noticing:

• You are a helpful assistant. Current time: 2026-07-08T09:14:23Z — a per-request timestamp in the first line. Hit rate: 0%.
• A request ID or session ID interpolated near the top.
• Tool schemas serialized from an unordered map, so tools appear in a different order per process.
• JSON serialization with unstable key order.
• Rewriting earlier conversation history each turn (re-summarizing, re-ranking) instead of appending.

What to do about it

1. Order your prompt by volatility. Static system prompt first, then tool definitions, then few-shot examples, then conversation history, then the current input. Stable content up top, variable content at the end. An append-only conversation is naturally cache-friendly: each turn extends yesterday's prefix instead of replacing it.

2. Evict dynamic bytes from the prefix. If the model needs the current date, inject it once per session or put it in the latest message — not interpolated into the top of the system prompt on every call. Same for user names, request IDs, feature flags.

3. Serialize deterministically. Fixed tool order, sorted JSON keys, no map-iteration-order roulette. Two byte-identical prompts are the whole contract.

4. Know your provider's mechanics. On Anthropic's API, caching is explicit: you place cache_control breakpoints; cache reads cost 0.1x the base input price, writes 1.25x (5-minute TTL) or 2x (1-hour) — it pays for itself after a single hit. On OpenAI's API it's automatic for prompts over 1,024 tokens, matching in 128-token increments, with cached input around 50% off (up to 90% on some models) and up to ~80% faster time-to-first-token on long prompts.

5. Measure it. The API tells you: cache_read_input_tokens (Anthropic) or usage.prompt_tokens_details.cached_tokens (OpenAI). Log it, chart the hit rate, and alert on regressions — because an innocent prompt tweak can silently zero your cache. If you've never looked, your hit rate is probably 0%.

With a warm cache, the 50-turn run above costs the equivalent of ~228K full-price tokens instead of 1.62M — $0.68 instead of $4.88, about 7x cheaper — and the model stops re-reading 56K tokens before every response.

Key takeaways

• Agent loops re-send everything every turn: input cost grows quadratically with turns. In a 50-turn run, ~98% of input tokens are ones the model already processed.
• Prompt caching skips that recomputation, but it matches your exact token prefix — it's positional, not semantic.
• One moving byte near the top (timestamp, request ID, unstable serialization) invalidates the cache for everything after it.
• Order content by volatility, append instead of rewriting, serialize deterministically.
• Track your cache hit rate like a real metric. It's one of the few places in engineering where a config-level change is worth 7-10x.