Decide which prompt prefix to cache vs re-run to reduce LLM latency

Implementation summary

This post gives a practical decision flow for splitting an LLM prompt into a reusable prefix and a volatile suffix so you can use provider-side prompt caching safely and effectively. The included helper finds the longest contiguous leading stable prefix, produces a conservative token estimate for that prefix, and compares it to a configurable minimum cacheable token threshold. Use the server-inserted example to plug this helper into tests and CI.

Expected behavior

• The helper treats only a contiguous leading block of parts marked stable as a single cacheable prefix. Stable parts appearing after the first volatile message do not extend that leading prefix for reuse.
• If the estimated prefix token count is below your configured minimum, the helper marks the prefix as not cacheable so you avoid treating tiny stable blocks as worth caching.
• For modern models (GPT-5.6+), the helper recommends explicit-only mode when later stable parts exist or when the prefix is borderline; it suggests implicit mode only for robust leading stable blocks with no later stable sections.
• Use cache keys to isolate reuse by customer/session/workspace and pick TTLs aligned with expected reuse: shorter TTLs for frequently changing prefixes and longer TTLs for durable reference material.

Limitations

• The token estimator is intentionally small and conservative; it is a planning tool, not the provider tokenizer. For production measurement, use a tokenizer that matches the actual model.
• The helper does not guarantee provider-side cache hits or quantify latency reduction. You must instrument the real system to measure cache hit rate, warm vs cold latency, and whether the prefix choice improves user-visible performance.
• Prompt caching is sensitive to prompt shape: append-only updates or compaction can invalidate reuse even when much of the text looks similar.

Use this post as a checklist: identify ownership (what changes each request), measure prefix size before committing to cache, pick keys and TTLs to separate scope, prefer explicit-only when later stable blocks exist, and verify with real instrumentation. The server will append the canonical URL and insert the verified implementation example.

Working example

export function simpleTokenEstimate(text) {
  if (!text) return 0;
  // Very small, deterministic estimator: split on whitespace and punctuation groups.
  // This is intentionally conservative and inexpensive; real apps should use a tokenizer
  // matching the actual model.
  return String(text)
    .trim()
    .split(/\s+|(?=[.,;:!?()\[\]{}"'])|(?<=[.,;:!?()\[\]{}"'])/)
    .filter(Boolean).length;
}

export function analyzePromptParts(parts, { model = 'gpt-5.6', minCacheableTokens = 64 } = {}) {
  // parts: array of { role: 'system'|'developer'|'user'|'assistant', text: string, stable?: boolean }
  // Determine the longest leading contiguous stable prefix (from start) that can be cached.
  if (!Array.isArray(parts)) throw new TypeError('parts must be an array');

  let prefixParts = [];
  let prefixTokenCount = 0;
  for (const p of parts) {
    if (!p || typeof p.text !== 'string') throw new TypeError('each part must have a text string');
    const isStable = Boolean(p.stable);
    if (!isStable) break; // only contiguous leading stable parts are cacheable as one prefix
    prefixParts.push(p);
    prefixTokenCount += simpleTokenEstimate(p.text);
  }

  const prefixText = prefixParts.map(p => p.text).join('\n');
  const isCacheable = prefixTokenCount >= minCacheableTokens;

  // Model-specific guidance: on gpt-5.6+, explicit-only is recommended when suffixes are volatile.
  const major = String(model).toLowerCase();
  const prefersExplicit = major.includes('gpt-5.6') || major.includes('gpt-6') || major.includes('gpt-5.');

  // Suggest cache mode: if there are trailing stable parts later (non-leading), explicit is safer.
  const hasLaterStable = parts.slice(prefixParts.length).some(p => p.stable);
  const suggestedMode = (!isCacheable || hasLaterStable) ? 'explicit' : (prefersExplicit ? 'implicit' : 'explicit');

  const contiguousLeadingStableCount = prefixParts.length;

  return {
    prefixText,
    prefixTokenCount,
    isCacheable,
    minCacheableTokens,
    suggestedMode,
    // top-level convenience for tests and callers
    contiguousLeadingStableCount,
    reason: {
      contiguousLeadingStableCount,
      hasLaterStable,
      model,
    }
  };
}

export function chooseCacheKey(baseKey, { scope = 'global', id = null } = {}) {
  // Simple helper to build a cache-accounting key. Do not include secrets here.
  if (typeof baseKey !== 'string') throw new TypeError('baseKey must be a string');
  const safeScope = String(scope).replace(/[^a-z0-9_:-]/gi, '_');
  if (id == null) return `${safeScope}:${baseKey}`;
  return `${safeScope}:${id}:${baseKey}`;
}

export default { simpleTokenEstimate, analyzePromptParts, chooseCacheKey };

Read the full article: https://chriseugenerodriguez.com/blog/prompt-prefix-caching-decision