DEV Community

aiunplugged
aiunplugged

Posted on Originally published at aiunplugged.in

Prompt caching — the LLM discount most teams aren't using

Most LLM applications pay full input price on the same prefix tokens, request after request. A 20,000-token system prompt sent 100 times a day is 2 million tokens billed at the standard input rate — most of them repeat. Prompt caching is the API-level feature that stops the billing clock on those repeats. On the current pricing of every major provider, cached-prefix tokens cost between 5% and 50% of standard input tokens, which translates to 50-95% off the input bill for workloads with long stable prefixes.

By late 2026, every major hosted LLM supports prompt caching in some form. Teams that haven't turned it on are paying 2-20x more per request than teams that have. This article covers what caching is, how each provider implements it, when it helps and when it doesn't, with the current (late 2026) pricing shape.

What prompt caching actually is

An LLM processes a prompt token-by-token, building up an internal representation (the KV cache) as it goes. For a stable prefix — a long system prompt, a document pasted in before a question, a tool-schema preamble — that internal state is identical every time the prefix is sent. Recomputing it from scratch on every request wastes GPU time at the provider's side and bills the user for tokens the provider didn't really have to process.

Prompt caching exposes that reuse. The provider stores the KV cache for recent long prefixes, and when the next request arrives with the same prefix, it skips the recompute. The savings get passed to the user as a steep discount on cached-input tokens, and the request also runs faster because the first N tokens of inference are already done.

Two things determine whether caching helps:

  1. Prefix stability — the first K tokens of every request must be byte-identical. One changed character at position 50 invalidates everything cached after it.
  2. Reuse frequency — the prefix must be sent again before the cache expires. All providers have TTLs; some are 5 minutes, some are 24 hours.

Everything else is implementation detail.

The pricing shape — late 2026

Rough shape across the four major providers. Verify current numbers on each provider's live pricing page before budgeting; this landscape moves monthly.

Provider Cached-input discount Cache-write surcharge Minimum cacheable prefix Default TTL Model/mode
Anthropic 90% off (0.1× base) on most models; 95% off (0.05×) on Opus 5.5 and Sonnet 5.5 +25% on first write (1.25× base) for 5-min TTL; +100% (2× base) for 1-hour TTL 1,024 tokens (varies by model) 5 minutes (default), 1 hour (opt-in) Explicit cache_control field
OpenAI ~90% off cached input (0.10× base) across current GPT-5.6 / GPT-6 families; 50% off on older GPT-4-era models None on current models (cache writes billed at uncached-input rate) 1,024 tokens ~30 minutes typical; up to 24 hours on extended-retention models Automatic + optional explicit breakpoints on newest models
Google Gemini 90% off cached input (Gemini 2.5+); 75% off (Gemini 2.0) Cache creation billed at standard input rate; plus a per-token-hour storage charge on explicit caches 2,048 tokens (Gemini 2.5), 4,096 tokens (Gemini 3.x), varies Implicit: automatic; explicit: user-chosen TTL with storage cost Implicit (automatic) + explicit API
DeepSeek ~90% off cached input None (automatic) Varies by model Hours (disk-based) Automatic

Three practical takeaways from the table:

  1. OpenAI and DeepSeek are "turn it on by default" — caching happens automatically, no code change required beyond keeping the prefix stable.
  2. Anthropic requires an explicit cache_control marker — more work, but gives finer control over which parts of the prompt to cache and for how long.
  3. Gemini's explicit caching is unique in charging storage — the cache is cheap to read, but you pay by the token-hour to keep it warm. Only worth it when reuse is high enough that storage pays for itself.

When caching helps most

Three workload shapes see the biggest gains:

Long system prompts reused across users. A chatbot with a 10,000-token system prompt covering tone, policies, allowed tools, and example interactions. Every user turn sends that 10K prefix again. With Anthropic 5-minute caching, after the first request each subsequent request within 5 minutes pays 10% of standard for those 10K tokens — a 90% discount on the dominant input line.

Document Q&A with the same document. User uploads a 50-page PDF, then asks 20 follow-up questions over the next 10 minutes. Without caching: 20 × 50-page prefix = full price. With caching: 1 full-price request + 19 cached reads = ~95% off the document portion.

Code assistants over the same codebase. Cursor, Claude Code, and similar tools re-send large chunks of the codebase on every turn as context. The codebase changes slowly within a session, so cache hit rates can be high. Both Anthropic's and OpenAI's APIs are built with this workload specifically in mind.

Three workload shapes see little or no gain:

  • One-shot single-turn requests where no two requests share a prefix. There is nothing to cache.
  • Highly variable prompts where the first 1000 tokens change on every request. Caching requires byte-identical prefixes.
  • Short prompts below the provider's minimum cacheable length. The 1,024-token floor cuts out most quick-chat use cases.

Anthropic — explicit cache_control markers

Anthropic's SDK requires an explicit marker on each content block the caller wants cached. Everything up to and including that marker becomes the cache boundary.

from anthropic import Anthropic

client = Anthropic()

LONG_SYSTEM_PROMPT = """\
You are an assistant for a legal document review platform.
Answer questions only using the attached contract.
Cite clause numbers.
...
(10,000 more tokens of policy, examples, allowed-tools, formatting rules)
"""


def ask(question: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": LONG_SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},   # default 5-minute TTL
            }
        ],
        messages=[{"role": "user", "content": question}],
    )
    return response.content[0].text
Enter fullscreen mode Exit fullscreen mode

The first call writes the cache at 1.25× standard input price for the system prompt tokens. Every subsequent call within 5 minutes reads from the cache at 0.1× (or 0.05× on Sonnet 5.5 and Opus 5.5). For a 10,000-token prompt called 20 times in 5 minutes, that's roughly:

  • Without caching: 20 × 10,000 = 200,000 input tokens at base price
  • With caching: 1 × 10,000 × 1.25 + 19 × 10,000 × 0.1 = 12,500 + 19,000 = 31,500 tokens-equivalent — about 85% off.

For a 1-hour TTL (useful when reuse is sparser), add "ttl": "1h":

cache_control = {"type": "ephemeral", "ttl": "1h"}
Enter fullscreen mode Exit fullscreen mode

The 1-hour cache costs 2× on write but still 0.1× on read, so it pays off after two reads instead of one. The 1-hour TTL became generally available in August 2025 and no longer requires a beta header.

OpenAI — automatic caching, optional breakpoints

OpenAI's caching requires zero code change on current models. Any prompt prefix at least 1,024 tokens long that gets re-sent within the cache window (typically 30 minutes on GPT-5.6 and later; up to 24 hours on extended-retention models) gets charged at the cached-input rate automatically.

from openai import OpenAI

client = OpenAI()

LONG_SYSTEM_PROMPT = "..." * 2000  # 10,000+ tokens


def ask(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-5.6-terra",
        messages=[
            {"role": "system", "content": LONG_SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ],
    )
    # Cache hit telemetry
    cached = response.usage.input_tokens_details.cached_tokens
    print(f"Cached tokens hit: {cached}/{response.usage.prompt_tokens}")
    return response.choices[0].message.content
Enter fullscreen mode Exit fullscreen mode

Two things to notice:

  1. Nothing in the request body signals caching. OpenAI handles routing automatically. The only observable effect is the cached_tokens field on usage.input_tokens_details.
  2. Observability is on the response, not the request. Always log cached_tokens / prompt_tokens to catch cache misses early — a small change in the system prompt (a date rotated in, a dynamic value inserted at position 50) will silently drop the hit rate without any error.

On GPT-6 family models, developers can also set explicit cache breakpoints to force the cache boundary where they want it, in addition to the automatic behaviour. Useful for prompts where the stable prefix ends mid-paragraph and automatic detection might cache less than the full stable portion.

Google Gemini — implicit (free) and explicit (storage-billed)

Gemini supports two caching modes:

Implicit caching — automatic, no code change, 90% discount on hits (on Gemini 2.5 and 3.x models), no storage fee. The system detects repeated prefixes and routes to the cached version transparently.

Explicit caching — the developer creates a named cache object, which stores the prefix on Google's side for a chosen TTL. Reads are 90% off standard input, but there is a per-token-hour storage charge while the cache is alive.

from google import genai
from google.genai import types

client = genai.Client()

# Create an explicit cache from a 50-page PDF uploaded earlier
cache = client.caches.create(
    model="models/gemini-3.1-pro-preview",
    config=types.CreateCachedContentConfig(
        display_name="legal-contract-123",
        system_instruction="You are a legal reviewer. Answer only from the attached contract.",
        contents=[my_uploaded_pdf],
        ttl="3600s",  # 1 hour; billed for each hour of storage
    ),
)

# Use the cache in later requests
response = client.models.generate_content(
    model="models/gemini-3.1-pro-preview",
    contents="What is the termination clause?",
    config=types.GenerateContentConfig(cached_content=cache.name),
)
Enter fullscreen mode Exit fullscreen mode

The storage cost matters. As of October 2026, Flash-tier models bill around $1.00 per million tokens per hour (with Gemini 3.8 Flash discounted to $0.50 through December 31, 2026, then rising to $1.00 on January 1, 2027), and Pro-tier (both Gemini 2.5 Pro and 3.1 Pro Preview) is $4.50 per million tokens per hour. For a 100,000-token cached PDF held for a 1-hour conversation on Flash, storage adds $0.05-$0.10 — negligible relative to the ~90% savings on reads if there are more than one or two queries. For a 1-million-token cached corpus held for 24 hours on Gemini Pro, storage adds ~$108 — only worth it with substantial reuse. Verify current rates on Google's pricing page before planning.

Implicit caching sidesteps the storage charge entirely. The trade-off: no guarantee the cache will be there when needed, since the system decides what to retain.

Common gotchas

Six patterns that silently kill cache hit rates:

  1. A timestamp or current date rendered into the system prompt. "Today is 2026-11-24" means every request has a unique prefix. Move the dynamic data into the user message, or round the date to the day/hour and accept slight staleness.

  2. Streaming responses that join concatenated chat history differently each turn. The join logic has to be byte-identical across calls. Different whitespace in message separators invalidates the cache.

  3. Tool/function definitions changing order between calls. Even if the tools themselves are the same, re-ordering their position in the request changes the prefix bytes. Sort tool lists deterministically.

  4. A/B tests or feature flags that rotate parts of the prompt. Each flag variant has its own cache, and total hit rate drops. Keep the flag variants above the cache boundary, not inside it.

  5. A userId or sessionId interpolated into the system prompt. Every user has their own cache, hit rates stay poor across users. Move user-specific data to the user message.

  6. Model upgrades without re-planning cache budgets. Each model has its own cache namespace. The day a service switches from Sonnet 5 to Sonnet 5.5, every cache resets.

When caching doesn't help, and what to do instead

If the workload has none of the three "helps most" shapes above, caching may not be the biggest cost lever. Three alternatives:

  • Shrink the prompt. Most 10,000-token system prompts have 3,000 tokens of dead text that could be removed with no quality loss. Shrinking input beats caching 90% of input for one-shot workloads.
  • Downgrade the model. For retrieval-heavy or classification-style queries, Haiku-tier models at 1/15th the price of Opus-tier produce acceptable quality. Caching doesn't close that gap.
  • Switch to batch API. For non-latency-sensitive workloads, OpenAI's and Anthropic's batch endpoints halve the input price for processing within 24 hours. Caching + batch stacks for even bigger savings.

Caching is one tool in the cost toolbox. Teams that treat it as the only lever often leave bigger savings elsewhere.

Bottom line

Prompt caching is a near-free optimisation for any LLM workload with a stable prefix over 1,024 tokens that gets reused within minutes. On the current pricing of every major provider, it cuts input cost 50-95% for the cached portion. The engineering cost is roughly one hour: identify the stable prefix, add the right marker (or just let auto-caching work), log the hit rate, and move on.

For teams that haven't turned it on, this is almost certainly the highest-ROI cost cut sitting in their LLM bill today. For teams that have turned it on but see poor hit rates, the six gotchas above cover 90% of the real-world misses.

Verify current pricing on each provider's live docs — the specific percentages above are accurate as of late 2026 but the LLM pricing landscape moves monthly.

Related reading:

Top comments (0)