Prompt Caching Strategies to Cut LLM Costs by 70%

LLM API costs add up fast when you're running a production application. If your system prompt is 2,000 tokens and you're processing 10,000 requests a day, you're paying for those same tokens 10,000 times. Prompt caching lets you pay once and reuse — and the savings can be dramatic.

What Prompt Caching Actually Is

Most LLM providers charge for every token you send in a request — input tokens plus output tokens. Prompt caching allows the provider to store the computed state (the KV cache) of a prompt prefix on their servers, so subsequent requests that share that prefix only pay a fraction of the original input token cost.

The key insight: prompt caching only works on static prefix content. The cached portion must be at the beginning of your prompt and must be identical across requests. If you change anything in that prefix — even a single character — the cache misses and you pay full price.

Typical savings with caching enabled:

• Input tokens in the cached prefix: ~90% cheaper
• Output tokens: unchanged
• Net reduction on token-heavy workloads: 50–75%

Structuring Prompts for Maximum Cache Hit Rate

The single biggest mistake developers make is putting dynamic content early in the prompt. User-specific data, timestamps, or session IDs near the top will break the cache for every single request.

The correct structure is:

[SYSTEM PROMPT — static, long, cached]
[FEW-SHOT EXAMPLES — static, cached]
[RETRIEVED CONTEXT — cacheable if content-addressed]
[USER QUERY — dynamic, never cached]

Here's a quick analyzer to check your prompt layout before deploying:

def analyze_prompt_structure(messages: list[dict]) -> dict:
    """
    Estimate how much of a conversation benefits from caching.
    """
    total_tokens = 0
    static_tokens = 0

    for i, msg in enumerate(messages):
        content = msg.get("content", "")
        estimated = len(content.split()) * 1.3  # rough token estimate
        total_tokens += estimated

        if msg["role"] == "system":
            static_tokens += estimated
        elif i < len(messages) - 2:  # all but the last user turn
            static_tokens += estimated

    cache_ratio = static_tokens / total_tokens if total_tokens > 0 else 0

    return {
        "total_tokens_estimate": int(total_tokens),
        "cacheable_tokens_estimate": int(static_tokens),
        "cache_ratio": round(cache_ratio, 2),
        "recommendation": (
            "Good caching potential"
            if cache_ratio > 0.6
            else "Restructure — dynamic content is too early"
        ),
    }


messages = [
    {"role": "system", "content": "You are an expert security analyst. " * 100},
    {"role": "user", "content": "Analyze this log file..."},
    {"role": "assistant", "content": "I found the following issues..."},
    {"role": "user", "content": "What about this IP: 192.168.1.1?"},
]

print(analyze_prompt_structure(messages))
# {'total_tokens_estimate': 481, 'cacheable_tokens_estimate': 390,
#  'cache_ratio': 0.81, 'recommendation': 'Good caching potential'}

Marking Cache Breakpoints Explicitly

Some providers let you annotate cache boundaries directly in your request payload. This gives fine-grained control over what gets stored versus what flows through uncached.

import httpx

SYSTEM_PROMPT = "You are a security analyst assistant..." * 50  # ~2 000 tokens


def build_cached_request(user_query: str, history: list, api_key: str) -> dict:
    return {
        "model": "your-model-id",
        "max_tokens": 1024,
        "system": [
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},  # provider-specific marker
            }
        ],
        "messages": history + [{"role": "user", "content": user_query}],
    }


def call_llm(query: str, history: list, api_key: str) -> dict:
    payload = build_cached_request(query, history, api_key)

    resp = httpx.post(
        "https://api.your-llm-provider.com/v1/messages",
        headers={"x-api-key": api_key, "content-type": "application/json"},
        json=payload,
        timeout=30,
    )
    resp.raise_for_status()
    data = resp.json()

    usage = data.get("usage", {})
    cache_read = usage.get("cache_read_input_tokens", 0)
    cache_write = usage.get("cache_creation_input_tokens", 0)
    print(f"Cache read: {cache_read} | Cache write: {cache_write}")

    return data

The cache_read_input_tokens field in the response is your ground truth. On the first request, you'll see cache_creation_input_tokens — you paid to populate the cache. On every subsequent request with the same prefix, cache_read_input_tokens climbs and creation tokens drop to zero.

Content-Addressed Caching for RAG Pipelines

In Retrieval-Augmented Generation (RAG) systems, retrieved documents change per query — which looks like it defeats caching. But if you sort and deduplicate chunks by content hash before injecting them, identical queries retrieve identical chunks in the same order. Same order means the same prompt prefix, which means a cache hit.

import hashlib


def stable_chunk_id(chunk: str) -> str:
    return hashlib.sha256(chunk.encode()).hexdigest()[:16]


def build_rag_prompt(query: str, chunks: list[str]) -> tuple[str, str]:
    """
    Returns (context_section, query_section).
    Deterministic ordering ensures cache hits across equivalent retrieval sets.
    """
    deduped = {stable_chunk_id(c): c for c in chunks}
    sorted_chunks = sorted(deduped.values(), key=stable_chunk_id)
    context = "\n\n---\n\n".join(sorted_chunks)
    return context, query


# Two different queries that retrieve the same three chunks
# will produce byte-for-byte identical context → cache hit
context, query = build_rag_prompt(
    "What is the NIST framework?",
    [
        "NIST CSF overview: identify, protect, detect, respond, recover.",
        "NIST SP 800-53 defines control families for federal systems.",
        "Risk management tiers range from 1 (partial) to 4 (adaptive).",
    ],
)

This technique is especially effective for FAQ-style applications where 20% of unique queries account for 80% of traffic. You can also pre-warm the cache by sending your top 100 most common context sets at startup — the first 100 requests pay to write the cache, and everything after that reads it for a fraction of the cost.

Monitoring Cache Efficiency in Production

Shipping caching without instrumentation is flying blind. Track these three numbers from day one:

• Cache hit rate = cache_read_tokens / (cache_read_tokens + cache_creation_tokens)
• Cost per request = derived from input/output/cache_read/cache_write token counts and provider pricing
• Cache miss patterns — which requests miss and why (dynamic content creeping into the prefix)

from dataclasses import dataclass


@dataclass
class CacheMetrics:
    hits: int = 0
    misses: int = 0
    tokens_saved: int = 0

    @property
    def hit_rate(self) -> float:
        total = self.hits + self.misses
        return self.hits / total if total > 0 else 0.0


metrics = CacheMetrics()


def track_usage(usage: dict) -> None:
    cache_read = usage.get("cache_read_input_tokens", 0)
    cache_write = usage.get("cache_creation_input_tokens", 0)

    if cache_read > 0:
        metrics.hits += 1
        metrics.tokens_saved += cache_read
    elif cache_write > 0:
        metrics.misses += 1

    if (metrics.hits + metrics.misses) % 100 == 0:
        print(
            f"Hit rate: {metrics.hit_rate:.1%} | "
            f"Tokens saved: {metrics.tokens_saved:,}"
        )

In production, pipe these numbers into Prometheus or your existing observability stack. If your hit rate drops below 70% on a workload that should be caching well, something dynamic is leaking into the prefix. The LLM security and configuration checklists at AYI NEDJIMI Consultants treat cost controls and access controls in the same review — both are operational risks.

The Takeaway

Prompt caching is one of the few LLM optimizations that requires no model changes, no architecture redesign, and no quality tradeoff. The entire gain comes from reorganizing your prompts and measuring the outcome:

1. Move all static content to the top — system instructions, examples, knowledge bases
2. Keep dynamic content (user queries, timestamps, session data) at the very end
3. Use content-addressing in RAG pipelines to maximize hit rate across semantically similar queries
4. Track cache_read_input_tokens per request from day one

On a real workload with a 3,000-token system prompt and 100 tokens of user input, caching reduces input costs by ~97% for the static portion. At scale, that is not a rounding error.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.