DEV Community

Cover image for cached_tokens is 0 because your system prompt isn't stable
Chad Priest
Chad Priest

Posted on Originally published at blog.vodou.ai

cached_tokens is 0 because your system prompt isn't stable

#llm #caching #typescript #debugging

I added a [cache] log line next to our usage tracker and read usage.prompt_tokens_details.cached_tokens off a warm conversation. It said 0. Then 1. Out of roughly 16,000 prompt tokens, every single turn. We were paying full price for the same prefix over and over, and the 273K-token board balloon I'd been blaming on the agent loop was just real money.

Session affinity was the wrong theory for the first day

My first theory was session affinity. Fireworks' docs say serverless needs a routing hint or turn 2 lands on a replica that never saw turn 1, and we were sending neither user nor x-session-affinity. Obvious culprit. Cheap fix.

So I wrote a direct probe against api.fireworks.ai with a byte-identical 4,034-token prefix:

call1 (cold,  user=A): prompt=4034 cached=0      0%
call2 (warm,  user=A): prompt=4034 cached=0      0%   ← still cold
call3 (warm,  user=A): prompt=4034 cached=4033 100%
call4 (no user field): prompt=4034 cached=0      0%
call5 (no user field): prompt=4034 cached=4033 100%
Enter fullscreen mode Exit fullscreen mode

Affinity wasn't decisive: calls 4 and 5 hit 100% with no hint at all. Caching was automatic and total. It just needed the prefix to actually be identical, and ours wasn't. Two things were rotting it: the ~24KB workspace bootstrap rode turn 1 and vanished on turn 2, and the ~15K query-dependent memory recall was concatenated straight onto the front of the system message.

chatWithOpenAICompat freezes the system prompt behind a flag

In MCP-servers/Vodou-Console/src/llm.ts, chatWithOpenAICompat now has a mode where the system prompt is deliberately frozen:

if (STABLE_PREFIX) {
  systemPrompt = staticParts;                // frozen → cacheable prefix
  lateContextBlock = memoryForSystem || '';  // volatile → relocated
} else {
  systemPrompt = staticParts + '\n\n---\n\n' + memoryForSystem;
}
Enter fullscreen mode Exit fullscreen mode

The relocated memory gets spliced in as a late system turn, immediately before the current user message:

const insertAt = Math.max(1, m.length - 1);
m.splice(insertAt, 0, { role: 'system', content: '### Relevant context for this turn\n\n' + lateContextBlock });
Enter fullscreen mode Exit fullscreen mode

Flag on: turn 1 and 2 read 0% (the write lag the probe already warned me about), then 93% → 96% → 97%. Flag off, same conversation: 0 / 81 / 0 / 90 / 95 / 58. The signature of a prefix that busts whenever retrieval returns something new.

Re-sending the bootstrap every turn cost 2.1x more, not less

I also "fixed" the bootstrap by re-sending it every turn so the provider could cache it. Six turns measured 49K raw / ~33K billed, against 19.6K / ~16K for the old drop-after-turn-1 behavior. About 2.1× worse. Fireworks discounts cached input ~50%, not the ~90% Anthropic-style number I had in my head, so re-sending 6K to cache it loses to simply not sending it. I reverted that half.

The rule that transfers: in a frozen-prefix design, the system prompt is a data structure with an invariant, not a string you can append to. Anything query-dependent goes in the message array. And before you build for a cache, measure its discount: a 50%-off cache and a 90%-off cache reward opposite architectures.

Source: cached_tokens is 0 because your system prompt isn't stable by Chad Priest, from Building Vodou in Public.

Top comments (0)