DEV Community: synthorai

Prompt Caching for Open-Weight LLMs: Provider Roulette

synthorai — Mon, 15 Jun 2026 11:52:43 +0000

With a closed model, prompt caching is one documented contract. Claude has cache_control breakpoints; OpenAI and Gemini cache automatically above a token floor; the discounts are published and stable. You read one page and you're done.

Open weights break that assumption. The same Qwen or Llama checkpoint is served by a dozen hosts, and caching is not a property of the model — it's a property of where the model runs. To show how far that goes, here's one measured request — an identical ~4.7K-token prompt sent to the same Qwen model through a multi-provider router six times, no upstream pinned:

Call	Upstream the router picked	Cost	Cached tokens
1	Upstream A	$0.0141	0
2	Upstream B	$0.000709	0 (cold)
3–6	Upstream B	$0.000286	4,224 (warm)

Same model, same router, same prompt: the bill ranged from $0.0141 to $0.000286 — a 49× spread — purely on which upstream the router chose and whether that upstream had the prefix warm.

Summary

Prompt caching for open-weight models is a routing outcome, not a model feature. It's implemented — free and automatic — in the inference engine, then preserved or destroyed by every layer above it.
Five layers: one provides caching, three can break it. The model (sets cacheability, serves no cache) → the inference engine (caching, free) → the compute host (productizes it, unevenly) → the gateway (multi-cluster routing) → the router (scatters across vendors with disjoint caches).
Measured. An identical request, scattered by a router, cost 49× more on one pick than another; on one model, one host delivered 59.6% off and another 0%; published cache discounts span 0% to ~98% across models.
What to do. Pin your route so repeated prefixes hit the same warm cache; audit by the cost delta, not the cached_tokens field (which often reads 0 on a real hit); weigh latency separately — warm prefills run 2–10× faster even at ~0% cost discount.

Live figures were measured on 2026-06-14 against a multi-provider router and our own gateway, with a fixed ~4.7K-token English prompt, small max_tokens, sequential runs. Documented pricing was checked against primary provider docs the same day and cross-verified adversarially. Ratios (percent discount, latency change) are the portable part; absolute dollars depend on the venue, your prompt, and load. Reproduce before quoting.

The cache types you'll actually meet

Before the stack, the vocabulary. Across open-weight hosts there are four distinct cache shapes, and they bill differently.

1. Automatic prefix caching (no markers). The dominant pattern. The server hashes your prompt prefix, reuses the KV state if it matches an earlier request, and applies the discount on its own — no cache_control, no code change, often impossible to disable. DeepSeek, Zhipu GLM, and most open-weight hosts do this. Writes are free; the cache lives anywhere from VRAM (minutes) to disk (DeepSeek keeps prefixes "a few hours to a few days").

2. Explicit breakpoint caching (cache_control). The Anthropic shape, which a few open-weight hosts also offer. Alibaba's Model Studio takes "cache_control": {"type": "ephemeral"} on a Qwen message block; some serving platforms expose an equivalent marker. You mark the boundary, pay a write surcharge, and get a deeper read discount in return.

3. Rented cache objects (with a storage fee). The one to watch. Moonshot's legacy moonshot-v1 family makes you POST /v1/caching to create a cache, then bills a write fee, a per-token-per-minute storage fee, and a per-call hit fee. Google's explicit Gemini caching is the same idea — input cost plus storage at roughly $1.00–$4.50 per 1M-tokens per hour. The cache is a resource you rent and must garbage-collect.

4. Self-host KV reuse (free). Run the weights yourself and the inference engine caches for free and automatically. No write fee, no read rate, no storage rental — a hit just skips prefill.

Cache type	Markers?	Write fee	Storage fee	Where you meet it
Automatic prefix	No	Free	No	Most open-weight hosts; DeepSeek, GLM
Explicit breakpoint	`cache_control`	Surcharge	No	Qwen (explicit mode); some platforms
Rented cache object	Create/TTL/delete	Yes	Yes	Moonshot `moonshot-v1`, Gemini explicit
Self-host KV reuse	No	Free	No	vLLM, SGLang, TensorRT-LLM

Qwen on Model Studio offers both automatic and explicit modes, with a real tradeoff: implicit bills a hit at 20% of input with free writes; explicit bills a hit at 10% of input but charges 125% on the write and bounds the entry to a 5-minute TTL. Deeper discount, but you pay to populate and pay again each time it expires.

Where caching lives in the stack

Here is the key idea. Prompt caching for open weights is solved at exactly one layer and endangered at every layer above it. Walk the stack from the weights up, and at each layer ask: does this layer provide caching, or merely forward it — and can it break what the layer below already did?

  request
     |
     v
  +--------------------------------------------------+
  | L5  router             scatters across vendors   |  can break it
  | L4  gateway            multi-cluster routing     |  can break it
  | L3  compute host       uneven delivery           |  can break it
  |==================================================|
  | L2  inference engine   CACHING LIVES HERE, free  |  <-- the cache is born here
  |==================================================|
  | L1  model              cacheability: MLA / GQA   |  sets the ceiling
  +--------------------------------------------------+

  A cache hit is born at L2 and must survive L3-L5 routing to reach you;
  every layer above L2 is a chance to land where your prefix isn't.

Layer 1 — The model: cacheability, not a cache

This is the layer most people think caching lives in — "DeepSeek has caching" — so it's the first one to get precise about. A checkpoint is a bag of weights; it runs the same attention whether or not a KV cache exists. It ships no cache, no discount, no TTL, no cache_control marker — those are serving-layer features. In that strict sense the weights provide no caching product.

But the weights are not neutral, and DeepSeek is the perfect example of why. The model's attention architecture decides how big the KV cache is, and therefore how cheap caching can ever be:

DeepSeek's Multi-head Latent Attention (MLA) compresses the KV cache into a low-rank latent — to roughly 4–14% of a standard multi-head cache. That compression is exactly what lets DeepSeek's API persist prefixes to disk and price a cache read at ~2% of input. The architecture is the enabler; the disk cache is a product built on top of it.
Grouped-Query Attention (GQA) — used by Llama, Qwen, Mistral, and DeepSeek — shares KV heads to shrink the cache by the group factor (≈8× on Llama-3).

So Layer 1's contribution is cacheability, not a cache: the architecture sets the ceiling on how cheap every layer above can make caching, but the weights never serve a cached token themselves. And "DeepSeek has caching" quietly merges two different things wearing the same name — the weights (this layer, which give you MLA) and DeepSeek's API and serving stack (Layers 2–3, which give you the disk cache, the discount, and the usage fields). Download the open weights and run them yourself and you keep MLA's small KV cache, but the disk-cache product stays on DeepSeek's servers — you inherit whatever Layer 2 you deploy in its place. So the operational move still holds: stop asking whether a model caches and start asking where it's served — just don't mistake that for the architecture not mattering. It sets the ceiling; the path sets what you actually get.

Layer 2 — The inference engine: where caching is built, and free

One layer up, caching is not just present — it's solved, and free. Modern inference engines cache prefixes automatically:

vLLM — Automatic Prefix Caching: hashes each KV block, reuses any block whose prefix hash it has seen, LRU-evicts. On by default in V1.
SGLang — RadixAttention: stores the KV cache in a radix tree so any shared prefix is reused, with cache-aware scheduling.
TensorRT-LLM — block reuse (enable_block_reuse, default on), with optional offload of KV blocks to host memory.

Projects like LMCache extend this further — offloading KV to CPU/disk and sharing it across instances, which is the seed of solving the routing problem we're about to hit. The point: if you self-host, you are done. Caching is automatic, costs nothing beyond the GPUs you already run, evicts by LRU, and you own it — a hit simply skips prefill, lowering TTFT and raising throughput. There is no cached_tokens billing field because nothing is billed; the payoff shows up in your own latency metrics. For a closed model you rent caching; for an open one you can own it outright. The catch is the inverse of the hosted world: the cache is ephemeral (VRAM, LRU), so it survives only while the prefix stays hot — which is precisely what the layers above must preserve.

Layer 3 — The compute host: productizing it, unevenly

Commercial inference hosts wrap Layer 2 and run fleets of replicas. They inherit free automatic caching — the question is whether they implement it well, and the answer is mixed on two axes.

First, exposure and price vary wildly. Among the major open-weight hosts: one applies a flat 50% to cached input and lets cached tokens skip rate limits; another defaults to 50% off on serverless; a third prices cached input per model (e.g. a Qwen tier at ~80% off) and exposes a cache-key hint to improve affinity; a fourth makes caching always-on and undiscloseable on dedicated endpoints. Same underlying engine, four pricing philosophies.

Second — and this is the first place caching breaks — the multi-replica problem. Your warm prefix lives in the VRAM of the one replica that served the cold request. The host's own load balancer may send your next request to a different replica with a cold cache. We saw exactly this: pinning the same Qwen model to one upstream at a time and running cold→warm:

Pinned upstream	Cold	Warm	Discount	`cached_tokens`
Provider A	$0.000709	$0.000286	59.6%	4,224 ✓
Provider B	$0.000662	$0.000662	0%	0

Provider A cached cleanly and reported it. Provider B — which advertises a cache-read price for this model — returned no discount across a cold call and two warm calls in our test. Whether that's eligibility, replica fan-out, or a longer warm-up than two requests, the measured result on this path was zero. The capability is solved at Layer 2; whether you actually receive it is a Layer-3 execution detail, and it differs by host.

Layer 4 — The gateway: the multi-cluster problem

A gateway sits in front of one or more upstreams and multiplies the replica problem into a cluster problem. If it round-robins requests across clusters or providers without cache affinity, the warm cache becomes structurally unreachable — every request lands somewhere the prefix isn't. A cache-aware gateway must route by prefix hash so identical prefixes stick to the same upstream, the same way Layer 2 sticks them to the same KV blocks.

We ran a cold→warm battery across open-weight models on a third-party gateway, reading the per-request cost directly:

Model	Cold	Warm	Discount	Latency
`deepseek-v4-pro`	$0.00189	$0.0000155	99.2%	6.0s → 1.1s
`deepseek-v4-flash`	$0.000564	$0.0000116	97.9%	4.9s → 1.2s
`qwen3.5-flash`	$0.000561	$0.0000853	84.8%	10.2s → 1.0s
`kimi-k2.5`	$0.00242	$0.000469	80.6%	3.2s → 1.2s
`qwen3-max`	$0.00350	$0.00336	3.8%	2.2s → 1.1s
`qwen3.5-plus`	$0.00114	$0.00114	0.0%	1.8s → 1.0s

DeepSeek-V4 hit 97–99% (affinity working end to end); qwen3.5-plus and qwen3-max returned ~0% on the warm call despite carrying a cache-read price in the catalog. Two more gateway lessons hide in this table:

The usage field lies; the cost doesn't. cached_tokens read 0 on every call here, including the 99% cost drops. Many OpenAI-compatible gateways don't populate the cached-token field for upstreams that cache automatically. Audit by the cost delta between a cold and warm call, not by the token field — the same lesson as auditing a gateway's cache claims.
Latency improves even when cost doesn't. Every warm call was 2–10× faster — qwen3.5-flash went 10.2s→1.0s — including the ~0%-discount ones. A hit skips prefill regardless of how the host prices it, so caching can pay off in TTFT on a gateway that gives you nothing on the bill.

A gateway that doesn't preserve affinity hands you a cache you can't reach; one that doesn't surface cache cost hands you one you can't verify.

Layer 5 — The router: random distribution across providers

At the top, a multi-provider router load-balances one model ID across different companies' clusters — each with a separate cache. Now even perfect affinity within a provider can't save you: if call 1 goes to one vendor and call 2 to another, there is no shared cache to hit. This is the scatter from the top of this post, and it compounds Layer 4 — not just multiple clusters, but multiple vendors with disjoint cache state and disjoint prices (the priciest pick billed 20× the cheapest upstream's base rate). The cache only engaged once routing happened to stick to one provider.

The fix is to collapse the randomness — make routing deterministic so repeated prefixes land on the same warm cache:

# Pin the upstream; otherwise load-balancing scatters you across disjoint caches.
# (field names follow a common multi-provider router's API)
import requests

requests.post(f"{ROUTER_BASE}/chat/completions",
  headers={"Authorization": f"Bearer {API_KEY}"},
  json={
    "model": "qwen/qwen3.5-35b-a3b",
    "messages": messages,
    "usage": {"include": True},              # return cost + cached_tokens
    "provider": {                            # the part that makes caching work
        "order": ["<your-chosen-upstream>"],
        "allow_fallbacks": False,
    },
  })

To its credit, the router did report cached_tokens (4,224 on the hit) and a per-request cost, so you can verify both — better than the Layer-4 gateway that read 0. But the routing is yours to constrain. Caching is a routing problem dressed up as a pricing feature: the cache is free at Layer 2, and Layers 3, 4, and 5 are three escalating ways to route yourself away from it.

How deep is the discount? It's all over the map

When the routing does line up, how much do you save? For closed models the cache-read discount clusters near 90%. For open weights the published cache-read price ranges from a token gesture to near-total, even within one vendor's lineup. First-party published rates:

Model (first-party / mode)	Input $/M	Cache read $/M	Discount	Layer-2 type
DeepSeek-v4-flash	0.14	0.0028	~98%	auto disk
DeepSeek-v4-pro	1.74	0.145	~92%	auto disk
Qwen (explicit mode)	base	0.10× base	90%	explicit
Kimi K2.6	0.95	0.16	~83%	auto
GLM-5	1.0	0.20	80%	auto implicit
Qwen (implicit mode)	base	0.20× base	80%	auto

DeepSeek's automatic disk cache is the deepest in the field — deepseek-v4-flash reads cached input at $0.0028/M against a $0.14/M miss, a 1:50 ratio, which our Layer-4 test reproduced at 97.9%. Third-party hosts of these same open weights price cached input independently — some apply a flat ~50%, others vary per model from ~50% to ~90% — so the discount you get is a function of which host you land on, not just the model. Same feature name, a 48-point spread.

Because the discount is a venue property, one model carries different cache economics everywhere it's served. deepseek-v4-pro, four ways:

Where (layer)	Cache-read discount	Source
First-party API (L3)	~92% ($1.74 → $0.145)	documented
Third-party host A (L3)	~89% ($1.74 → $0.20)	documented
Third-party host B (L3)	~92% ($1.6 → $0.135)	documented
Third-party gateway (L4)	99.2%	measured (cold→warm)

"DeepSeek-V4-Pro supports caching" is true and nearly useless; the operational question is "supports caching where, at what rate, reported how."

A decision checklist

✅ The model sets the ceiling, not the cache (Layer 1). Its attention architecture (MLA, GQA) decides how cheap caching can be, but it never serves a cached token — so still ask where it's served and what that host's stack does.
✅ Self-hosting? You already have it free (Layer 2). Confirm automatic prefix caching is on (it is by default in vLLM/SGLang) and watch your prefix hit rate.
✅ On a compute host, verify delivery, not the price column (Layer 3). A cache-read price is a claim; measure a cold→warm cost delta. Use a cache-key affinity hint where the host offers one.
✅ Through a gateway, demand cache-affinity routing and cost reporting (Layer 4). If identical prefixes don't stick to one upstream, or cost doesn't drop on a warm call, the cache is unreachable or unverifiable.
✅ On a router, pin the upstream (Layer 5). Constrain routing (e.g. a provider-order field with fallbacks off), or you forfeit hits to load-balancing across disjoint caches — and risk a 20–50× pricier upstream.
✅ Weigh latency separately from cost. Warm prefills are 2–10× faster even when the dollar discount is ~0.
✅ Watch for storage-fee cache types. Rented caches (Moonshot moonshot-v1, Gemini explicit) bill per-token-time for an idle cache; automatic prefix caches don't.

Bottom line

For closed models, "does it cache?" has one answer. For open weights the capability was solved years ago at the inference-engine layer — vLLM and SGLang cache every prefix, for free, automatically. Everything above that layer is plumbing that either preserves the hit or scatters you away from it: a compute host's replica balancer, a gateway's cluster routing, a router's random spread across vendors. The model's architecture sets the ceiling on how cheap caching can be — MLA and GQA are real, model-level wins — but the path your request takes decides what you actually get. Treat cache behavior as a routing property — measure it in cost terms on the exact path you'll run, pin the route so the cache you warmed is the one you hit, and remember that the deepest discount in the world is worth nothing if request two lands somewhere request one never touched.

For the mechanics of why a KV cache exists and how TTLs work, start with How KV Cache & TTL Work; to audit a gateway's cache claims, see Does Your LLM Gateway Lie About Cache?.

FAQ

Do open-weight models support prompt caching?
The weights set how cheap caching can be — attention architectures like MLA and GQA shrink the KV cache — but the cache itself, the discount, and the API come from the serving stack. Caching is implemented in the inference engine (vLLM, SGLang, TensorRT-LLM), inherited by compute hosts, and forwarded (or scattered) by gateways and routers. Ship the same checkpoint to three hosts and you can get free automatic caching, none, or explicit-only.

Why did the same model cost 49× more on one call than another?
On a multi-provider router, an un-pinned request is load-balanced across different vendors' clusters with different base prices and disjoint cache state. One call hit a pricey provider cold; another hit a cheap one warm. Pin the upstream (constrain provider order, fallbacks off) to control both.

If I self-host, do I need to pay for caching?
No. Automatic prefix caching in vLLM, SGLang, and TensorRT-LLM is on by default and free — a hit just skips prefill. You pay only for the GPUs you already run, and the cache is yours, evicted by LRU when VRAM is needed.

The API says cached_tokens: 0 but my bill dropped — did caching work?
Probably yes. Many gateways don't populate cached_tokens for upstreams that cache automatically. Trust the cost field: a large drop between a cold and an identical warm call means the cache hit.

Which open-weight model has the deepest cache discount?
DeepSeek's automatic disk cache: deepseek-v4-flash reads cached input at ~$0.0028/M against $0.14/M uncached (~98% off), reproduced at 97.9–99.2% across the V4 line in our cold→warm tests. Many third-party hosts apply a flat ~50% instead.

Is there a catch with caches that charge a storage fee?
Yes — Moonshot's moonshot-v1 explicit cache and Gemini's explicit cache bill per-token-time to keep the cache alive (Gemini ~$1–4.50 / 1M-tokens / hour). An idle cache you forgot to delete keeps charging. Automatic prefix caches have no storage fee.

Verification: live cost/latency figures measured 2026-06-14 against a multi-provider router and our own gateway, using a fixed ~4.7K-token prompt, small max_tokens, sequential cold→warm runs; discounts computed from the returned per-request cost. Documented pricing and cache mechanics checked against primary provider docs the same day and cross-verified adversarially; a few vendor figures (notably Moonshot's explicit-cache fees) move frequently — confirm current values before quoting. Your numbers will vary with provider, prompt, region, and load.

Sources

All checked 2026-06-14. Not financial advice; verify current pricing before relying on it.

Claude Fable 5's 30-Day Retention: ZDR, HIPAA, COPPA

synthorai — Fri, 12 Jun 2026 06:54:46 +0000

If your organization runs Claude under a zero-data-retention (ZDR) agreement, your first request to claude-fable-5 didn't return a completion. It returned 400 invalid_request_error. That's not an outage — it's policy. Fable 5 is the first generally available Claude model that cannot be used without 30-day data retention, and the requirement follows the model onto every platform: the Claude API, AWS Bedrock, Google Vertex AI, and Microsoft Foundry each gate it behind an explicit retention opt-in.

For teams that treated "we don't retain your data" as a settled property of their LLM stack, this is an architectural event. This post covers what the policy says, why the window exists, how each cloud implements it, and what it changes for consumer products and sensitive-data industries.

Policy details were checked against Anthropic's, AWS's, Google's, and Microsoft's published documentation on 2026-06-12. Policies change; verify against the linked primary sources and your own contracts. This is an engineering overview, not legal advice.

What the policy actually says

Anthropic designates Claude Fable 5 and Claude Mythos 5 as Covered Models. Per the API data retention docs and the Mythos-class retention practices article (effective 2026-06-09):

Prompts and completions are retained for 30 days, then automatically deleted — unless flagged for an active safety investigation or required by law.
There is no opt-out. Retention is a condition of using the model. A request from an organization whose retention configuration doesn't meet the requirement returns 400 invalid_request_error.
Access is narrow by design. Automated safety systems screen the data; only a small group of approved personnel can review flagged conversations, they cannot export, copy, or download it, and every access lands in tamper-proof logs.
Existing ZDR agreements do not carry over to Covered Model traffic — including through cloud platforms.

Consumer plans (Claude Free/Pro/Max) are unaffected — they already operate under their own retention terms. This policy targets the commercial API surface, exactly where "we never retain" promises tend to live.

Why a 30-day window exists

The rationale in the Covered Models article is specific: these models have substantially advanced capabilities in software engineering, agentic workflows, and cybersecurity, and "some forms of misuse only become detectable across many requests." The cited examples — best-of-N jailbreaking, state-sponsored espionage — are attack patterns where each prompt looks benign and only the sequence is diagnostic. You can't detect a sequence you've deleted.

Two things the window is not:

Not training data. Anthropic states retained data is never used for training without express permission. The purpose is abuse detection, full stop.
Not new in kind — new in enforceability. A ~30-day abuse-monitoring window has been the industry default for years: OpenAI keeps API abuse logs up to 30 days (ZDR by approval); Azure OpenAI stores prompts up to 30 days unless approved for modified abuse monitoring. What changed is that the window became non-negotiable for one model class — previously every provider offered a zero-retention escape hatch.

One pre-existing caveat that surprises people: even under ZDR, Anthropic retains safety classifier results, and content flagged for Usage Policy violations can be kept up to 2 years. Zero data retention has never meant zero data — it means zero retention of unflagged content in the normal path.

Same requirement, three clouds, three mechanisms

The retention applies wherever the model runs, but each platform wires the opt-in differently — and the differences decide who processes your data and where your controls live.

Platform	Opt-in mechanism	Scope	Without opt-in
Claude API	30-day retention in Privacy controls	Organization or workspace	`400 invalid_request_error`
AWS Bedrock	`data_retention_mode: provider_data_share`	Account or project	Model listed `unavailable`; requests blocked
Google Vertex AI	Anthropic data sharing + Model Garden terms	Project	Requests blocked until enabled
Microsoft Foundry	Anthropic's terms accepted at deployment	Subscription/deployment	Not covered by Azure's ZDR program at all

AWS Bedrock is the most explicit. Data retention is a configurable mode (default / provider_data_share / none), resolved project → account → model default. Fable 5 declares allowed_modes: ["provider_data_share"]: prompts and completions are shared with Anthropic and retained up to 30 days. Under any other mode:

{
  "id": "anthropic.claude-fable-5",
  "status": "unavailable",
  "status_reason": "This model is not available under data retention mode 'default'.",
  "data_retention": {
    "mode": "default",
    "source": "account",
    "allowed_modes": ["provider_data_share"]
  }
}

Nothing changed for pre-Fable-5 models, and an SCP on the bedrock:DataRetentionMode condition key can enforce your posture org-wide — nobody quietly flips the account to try the new model. Note: with cross-region inference, the retained copy lives in the destination region, which matters if you carry residency commitments.

Google Vertex AI gates the model behind a project-level Anthropic data-sharing setting (setPublisherModelConfig with dataSharingEnabledProvider: "anthropic") plus terms acceptance in Model Garden, per Google's Fable 5 documentation. General data handling follows Vertex AI's data-governance policy; for residency-sensitive workloads, Vertex's regional and multi-region endpoints control where inference runs — which now includes where the retained copy lives.

Microsoft Foundry is structurally different. Microsoft's data and privacy documentation is explicit that Claude models are third-party marketplace services: you accept Anthropic's terms at deployment, and Anthropic — not Microsoft — is the data processor. Azure OpenAI's ZDR and modified-abuse-monitoring programs don't extend to Claude deployments. Organizations with ZDR postures elsewhere typically isolate Covered Model use in a dedicated subscription, making the retention boundary structural rather than procedural.

The pattern across all three: retention class became a first-class, machine-readable model attribute — a mode, a flag, a terms gate — rather than a paragraph in a contract. Your infrastructure can now enforce your data posture, and it should.

What it means for enterprise deployments

With no ZDR agreement, nothing changes mechanically — you were already in a 30-day-style posture, possibly without realizing it. The work is making it explicit in your vendor documentation.

With a ZDR agreement, you have a three-way choice:

Skip Covered Models. ZDR stays uniform; you give up the model. Viable if your workloads don't need it — see our measured Fable 5 evaluation for what it costs and where it differs.
Split by workspace or project. Every platform supports a scoped opt-in: a designated Claude API workspace (Console → Settings → Workspaces → Privacy controls), a Bedrock project with provider_data_share, a separate Vertex project or Azure subscription. Route only retention-tolerant workloads there.
Accept retention org-wide. Simplest to operate, but it silently downgrades the guarantee for every workload — including the ones whose sensitivity justified ZDR. That's a decision for your data-protection owner, not a config change.

Regardless of provider: your own logging is a second retention surface. If your gateway or observability stack logs full prompts, you're running a longer window than your provider, under your own roof. Provider guarantees are only as meaningful as the layer in front of them — the same audit logic we applied to cache claims applies here.

What it means for consumer-facing products

If you serve consumers and route their content through a Covered Model, the change propagates into your own legal surface — ZDR agreement or not. Three concrete consequences:

1. Your privacy notice probably needs an update. Most regimes require disclosing retention, not just collection: GDPR Article 13(2)(a) requires the storage period (or criteria) at collection time; California's CPRA requires the notice at collection to state retention per category of personal information. If your notice says — or implies — that conversation data isn't retained anywhere, a processor holding a 30-day copy makes it wrong. Update the notice, the records of processing, and the DPA inventory.

2. You cannot offer users an opt-out you don't have. The retention has no exception mechanism, so there is no toggle you can build that exempts a user's prompts while still using that model. The lever you actually hold is routing: a consent-aware gateway sends users who decline data sharing to ZDR-eligible models and everyone else to the Covered Model — a legal constraint turned into an ordinary routing rule. Far better than a preference checkbox that does nothing.

3. Deletion requests need accurate plumbing. Erasure obligations (GDPR Art. 17, CPRA deletion, and their equivalents) extend to processors. A bounded window that auto-deletes within 30 days is generally a defensible processor posture — but your DSAR playbook should say that, not promise immediate downstream deletion you can't execute.

The global dimension compounds this: the same disclosure-and-processor logic appears in the UK GDPR, Brazil's LGPD, and the spreading family of US state privacy laws. For users in China, PIPL adds two sharper edges — providing personal information to another processor generally requires separate consent, and routing Chinese users' content to an overseas LLM endpoint is a cross-border transfer needing a recognized mechanism (security assessment, standard contract, or certification). A model upgrade that changes who retains what, where, for how long is exactly the change these frameworks expect you to re-paper.

Sensitive-data industries: where 30 days bites hardest

For most products the provider's window is a documentation problem. For industries whose data is itself regulated, it's an architecture problem: the retained copy is regulated data at rest at a vendor, and sector rules govern exactly that.

Healthcare (HIPAA)

HIPAA doesn't require zero retention — it requires that any vendor holding protected health information does so under a Business Associate Agreement (BAA) with appropriate safeguards. The 30-day copy of your prompts is PHI at rest at a business associate; the question is whether your BAA covers it. The two major API vendors structure this differently, and the difference now matters: Anthropic's HIPAA-ready API access explicitly doesn't require ZDR — it's built on retention-with-safeguards (encryption, access controls, audit logging, enforced feature restrictions). OpenAI's API BAA covers endpoints eligible for zero data retention — and a BAA scoped to ZDR endpoints structurally cannot cover a model that mandates retention.

A model's retention class is now a BAA-eligibility question. Confirm in writing that your BAA covers the specific model before routing PHI to it — and remember the chain shifts on clouds: on Bedrock the platform is your business associate; on Foundry, Anthropic processes the data directly. One sharp edge: PHI must never appear in JSON schema definitions for structured outputs — cached schemas don't get the same protections as message content.

Children's products (COPPA)

The timing is awkward: the FTC's amended COPPA Rule took effect June 23, 2025, with compliance on most provisions due April 22, 2026 — the first model with mandatory provider-side retention arrived just as operators finished implementing the new retention obligations. Two of those interact directly with the 30-day window: a written, public data retention policy is now mandatory (§312.10) — what children's data is collected, why, and when it's deleted — and indefinite retention is prohibited, with retention limited to what's reasonably necessary for the collected purpose.

A bounded 30-day window with automatic deletion is the compatible shape — but the provider retains for its trust-and-safety purpose, not the purpose you collected the child's data for, and your notice must describe the processor relationship accurately. For child-directed products that adopted ZDR specifically to minimize the data trail, the routing answer applies with higher stakes: children's traffic stays on ZDR-eligible models, or the Covered Model window goes into your §312.10 policy first.

The same pattern, other sectors

Once you see the structure — regulated data, retained copy at a vendor, sector rule governing retention — it recurs:

Biometrics (Illinois BIPA): operators need a written, publicly available retention schedule and destruction guidelines for biometric data. A provider's 30-day copy of prompts containing biometric identifiers belongs in that schedule.
Payments (PCI DSS / GLBA): PCI DSS prohibits storing sensitive authentication data after authorization — anywhere. Card data pasted into a prompt becomes card data retained at a provider for 30 days. The clean answer is upstream redaction, not downstream paperwork.
Education (FERPA): vendors handling student records under the school-official exception must remain under the school's direct control. A safety-retention copy the school cannot access or delete early sits uneasily with that standard — a question for counsel before EdTech traffic hits a Covered Model.
Financial services — the inversion (SEC/FINRA): broker-dealers must retain business communications under books-and-records rules. For them the provider's window isn't the problem; capturing their own compliant copy is. Same retention question, opposite sign.

The common thread: sector rules regulate retention in both directions, and a provider-side window you don't control must be mapped into whichever direction your sector points.

A decision checklist

✅ Inventory which models your traffic actually touches. Retention class is now a per-model attribute, not a per-provider one.
✅ If you have ZDR: decide deliberately — skip Covered Models, split by workspace/project/subscription, or accept retention org-wide. Don't let it happen implicitly.
✅ Enforce the posture in infrastructure — Bedrock SCPs, workspace privacy controls, separate cloud projects — not in a wiki page.
✅ B2C: update privacy notices and DSAR playbooks; route non-consenting users to ZDR-eligible models instead of building opt-outs that can't work.
✅ Regulated data: confirm coverage per model, in writing — BAA for PHI, §312.10 policy for children's data, retention schedules for biometrics — before routing that data to a retention-required model.
✅ Audit your own logging. A provider's 30-day window is irrelevant if your gateway logs prompts indefinitely.

Bottom line

The 30-day window attached to Fable 5 is not a data grab — it's bounded, purpose-limited abuse monitoring, consistent with what most of the industry already does by default, made mandatory for one model class because cross-request misuse detection doesn't work on deleted data. For most teams the engineering impact is zero and the governance impact is a paragraph in a vendor review.

But for organizations whose compliance position assumed zero retention — ZDR-scoped BAAs, privacy notices that say nothing persists, children's products built on data minimization — Fable 5 is the moment that assumption stopped being uniform across models. The fix isn't avoiding the model. It's making retention class an explicit, per-model input to routing decisions, the same way you already treat price and context window.

FAQ

Can I use Claude Fable 5 under a zero-data-retention agreement?
No. Fable 5 and Mythos 5 are Covered Models requiring 30-day retention; ZDR organizations get a 400 invalid_request_error unless they enable 30-day retention for a workspace and route Fable 5 traffic through it.

Does going through AWS Bedrock, Vertex AI, or Microsoft Foundry avoid the requirement?
No. Each platform gates the model behind its own retention opt-in: provider_data_share on Bedrock, Anthropic data sharing plus Model Garden terms on Vertex, Anthropic's terms at deployment on Foundry (where Anthropic, not Microsoft, is the data processor). Existing ZDR arrangements don't carry over on any of them.

Can my end users opt out of the retention?
No — there is no opt-out mechanism. The lever you hold is routing: send users who decline data sharing to ZDR-eligible models. Don't ship a preference toggle that doesn't change anything.

Is the retained data used to train models?
Anthropic states retained data is never used for training without express permission. The purpose is trust-and-safety review: automated screening, with flagged conversations reviewable only by approved personnel who cannot export the data, under tamper-proof access logs.

Does the 30-day retention change how prompt caching works?
No. Cache entries follow their own short TTLs (5 minutes or 1 hour) and the caching contract on Fable 5 is unchanged — see our measured evaluation. The 30-day window is a separate, parallel retention for safety review.

Sources

All checked 2026-06-12. Policies change — verify against current documents and your own contracts. Not legal advice.

Claude Fable 5: Caching, Tokenizer & Cost vs Opus 4.6

synthorai — Thu, 11 Jun 2026 15:49:46 +0000

claude-fable-5 is now available on the Synthorai gateway. If you cache against the Claude line, the good news is that the caching and TTL contract is a carry-over: same cache_control markers, same 5-minute and 1-hour TTLs, same write premiums, same deep read discount. Your caching code moves over by changing one string.

The thing to budget for isn't the cache mechanics — it's the bill. Fable 5 lists at 2x the Opus token price, and it tokenizes the same English text into ~45% more tokens than Opus 4.6 (it's on the post-4.6 tokenizer, identical to Opus 4.8). Those two multipliers stack. This post measures all of it so you don't have to.

All numbers below were measured against https://synthorai.io/ (Anthropic-native /v1/messages) on 2026-06-10 with a stable ~6.6–9.6K-token English system prompt, max_tokens small, single sequential run. Cost figures are read from the gateway usage.cost field; ratios (token counts, write premium, read discount, cross-model cost) are the portable part — absolute dollars scale with your prompt. Reproduce against your own prompt before quoting them.

Availability

import os
from anthropic import Anthropic

anth = Anthropic(
    api_key=os.environ["SYNTHORAI_KEY"],
    base_url="https://synthorai.io/",   # SDK appends /v1/messages
)

msg = anth.messages.create(
    model="claude-fable-5",             # the only line that changes
    max_tokens=512,
    system=[
        {"type": "text", "text": SYSTEM_PROMPT,
         "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": question}],
)
print(msg.usage)   # input_tokens, cache_creation_input_tokens, cache_read_input_tokens, cost

Swap claude-opus-4-6 → claude-fable-5 and nothing in your caching path needs to move. Fable 5 is an Anthropic-native model with a 1M-token context window. One behavioral note: it is a reasoning model and emits thinking tokens by default — even a trivial "reply OK" returned output_tokens_details.thinking_tokens > 0 in our runs, where Opus 4.6/4.8 returned zero. Budget output tokens accordingly. The mechanics behind cache_control are covered in the caching tutorial; the architecture of why the cache exists is in Part 1 of the series.

The headline: Fable 5 is on the new tokenizer

The token count for the Opus line jumped at the 4.7 generation: the same English text that counted as ~6.6K tokens on 4.6 counts as ~9.6K on 4.8. Fable 5 lands on the new side — identical text reports the exact same token count as Opus 4.8.

Model	Input tokens (identical text)	Tokenizer generation
`claude-opus-4-6`	6,614	pre-4.7
`claude-opus-4-8`	9,619	post-4.7
`claude-fable-5`	9,619	post-4.7 (identical to 4.8)

The same system prompt is ~45% more tokens on Fable 5 than on Opus 4.6 (9,619 / 6,614 = 1.45). This is the single most important number to internalize before you migrate, because every downstream figure — cost, the 1,024-token cache-eligibility floor, your per-call budget — is computed in tokens.

We're describing a measured observation — identical text, identical token count on Fable 5 and Opus 4.8, ~45% above Opus 4.6 — most consistent with the tokenizer/vocabulary update that shipped at the 4.7 generation. If you're coming from 4.6 or earlier, re-measure; if you're coming from 4.7/4.8, expect parity.

Caching behavior: the contract is unchanged

We ran the same no-cache / cold-write / warm-read sequence on each model. The discount structure is identical end to end — Fable 5 honors cache_control and reports the same usage fields (cache_creation_input_tokens, cache_read_input_tokens, and the ephemeral_5m / ephemeral_1h buckets).

Model	5m cache write	1h cache write	Warm read
`claude-opus-4-6`	1.25x	2.00x	~9% of no-cache
`claude-opus-4-8`	1.25x	2.00x	~6% of no-cache
`claude-fable-5`	1.24x	1.99x	~6% of no-cache

Two invariants hold across all three:

Write premium ≈ 1.25x (5m), ≈ 2x (1h). The first (cold) call costs ~1.25x the no-cache price to populate a 5-minute entry, or ~2x for a 1-hour entry. Break-even is one hit.
Read discount ≈ 90%+. A warm cache read on Fable 5 cost ~6% of the no-cache call — a ~94% discount, in line with (slightly better than) Anthropic's documented ~90% cached-read economics. Reads stay deeply discounted regardless of TTL.

The percentages are flat across the line. As with the Opus 4.7 → 4.8 step, the higher absolute bill on Fable 5 is a price-and-token story, not a cache-economics story — covered next.

TTL behavior: both windows honored

Fable 5 supports the same two TTLs as the rest of the line: a 5-minute sliding default and an opt-in 1-hour window. We isolated each TTL with a unique prefix per call (so no stale entry could contaminate the result) and confirmed the usage object reports the correct bucket — cache_creation.ephemeral_5m_input_tokens or ephemeral_1h_input_tokens.

# 1-hour TTL — same marker syntax on Fable 5 as on the Opus line
"cache_control": {"type": "ephemeral", "ttl": "1h"}

The 1-hour write costs ~2x no-cache (vs ~1.25x for the 5-minute write), and reads stay at the deep discount regardless of TTL — identical to Opus 4.6/4.8. If you picked 5m for live chat and 1h for agents with human-in-the-loop pauses on Opus, keep those choices on Fable 5.

The cost story: 2x price x 1.45x tokens

Here is where Fable 5 actually differs. Two things push the bill up, and they multiply.

1. List price is 2x the Opus tier.

Model	Input ($/M)	Output ($/M)	Cache read ($/M)
`claude-opus-4-6` / `4-8`	5	25	0.5
`claude-fable-5`	10	50	1

2. The same text is ~45% more tokens than on 4.6 (the tokenizer shift above).

Multiply them and the same English prompt costs materially more. Measured against the identical system prompt on each model (gateway usage.cost, same single run):

Comparison	Token ratio	Price ratio	Same-prompt cost ratio (measured)
Fable 5 vs Opus 4.8	1.00x	2.0x	2.0x
Fable 5 vs Opus 4.6	1.45x	2.0x	2.9x

So against Opus 4.8 (same tokenizer), Fable 5 is a clean 2x — pure price premium. Against Opus 4.6, the tokenizer change compounds the price change into roughly 2.9x the cost for the same prompt. Your cache discount is unchanged, but the absolute base it applies to is ~2.9x larger than it was on 4.6. If you sized a per-call budget against 4.6, re-do it.

A practical consequence: re-check the 1,024-token cache-eligibility floor. Anthropic only caches prefixes at or above a minimum size. A prompt that sat just under the floor on 4.6 (in old-tokenizer tokens) may clear it on Fable 5 (~45% more tokens) — and vice versa for size estimates built on the old count. Always read cache_creation_input_tokens / cache_read_input_tokens from the live response rather than estimating from a local tokenizer that may not match.

Migration checklist (Opus → Fable 5)

✅ Caching code carries over verbatim. cache_control markers, breakpoint count (up to 4), ttl: "1h", usage-field names — all identical.
✅ TTL choices carry over. 5m for live/session workloads, 1h for bursty/agent-with-pauses.
✅ Discount economics carry over. ~90%+ read, ~1.25x write (5m), ~2x write (1h).
⚠️ Re-budget on absolute cost. Fable 5 is ~2x Opus per token, and ~2.9x the same-prompt cost vs Opus 4.6. The discount percentage is unchanged; the base it applies to is not.
⚠️ Re-measure token counts if coming from 4.6 or earlier (expect ~45% more for the same text). From 4.7/4.8, expect parity.
⚠️ Account for default thinking tokens. Fable 5 emits reasoning tokens by default — they bill at the output rate ($50/M). Cap or disable thinking if you don't need it.

Bottom line

For a team already caching against Claude, claude-fable-5 is an easy integration: the entire caching and TTL surface is stable, so there's nothing to relearn and no code to rewrite. It is not an easy budget swap from Opus 4.6 — between the 2x token price and the ~45% tokenizer inflation, the same prompt runs ~2.9x the cost. Confirm your numbers against the live usage object, decide whether you need the default thinking tokens, and size the cache breakpoints against the new token counts.

For the full caching playbook — prompt structure, hit-rate debugging, TTL-aware patterns — see the four-part series starting with How KV Cache & TTL Work and the working Python tutorial.

FAQ

Do I need to change my cache_control code to use Fable 5?
No. The marker syntax, breakpoint limit, and TTL options are identical to the Opus line. Change the model field and nothing else in the caching path.

Did the cache read discount change on Fable 5?
No. A warm read is a small single-digit fraction of the no-cache input price (~90%+ off) — we measured ~94% on Fable 5, consistent with Anthropic's documented cached-read economics.

Does Fable 5 support the 1-hour TTL?
Yes. {"type": "ephemeral", "ttl": "1h"} works exactly as on Opus. The 1-hour write costs ~2x no-cache; the 5-minute write ~1.25x. Reads stay deeply discounted on both.

Why is the same prompt so much more expensive on Fable 5 than on Opus 4.6?
Two stacked multipliers: Fable 5 lists at 2x the per-token price, and the same English text counts as ~45% more tokens (it uses the post-4.6 tokenizer). Together that's ~2.9x the cost for an identical prompt. The cache discount is unchanged.

Is Fable 5 a drop-in replacement for Opus 4.8?
On the caching/TTL surface and token counts, yes — token counts are identical, so the only delta is the 2x price and Fable 5's default thinking tokens. We don't publish capability benchmarks we haven't run; for quality and reasoning claims, see Anthropic's model card.

Verification: all token-count, cost, write-premium, and read-discount figures measured against https://synthorai.io/ on 2026-06-10 using the official anthropic SDK, single tenant, single sequential run. Cost is read from the gateway usage.cost field; cross-model and premium/discount ratios are computed from those measured costs and are independent of any account-level promotion. Discount/premium ratios cross-checked against Anthropic Prompt Caching docs. Warm-read latency (TTFT) was dominated by network jitter in our run and is omitted as unreliable. Your numbers will vary with prompt, region, and load.

Provider Drift: How Default Routing Inflates LLM Cost 3.9 — A Measurement

synthorai — Fri, 05 Jun 2026 09:34:34 +0000

You turned on prompt caching, the hit counter ticks now and then, but your bill barely moved. Before blaming your prompt structure, look at something the dashboard hides: which upstream actually served each request.

Multi-provider gateways spread a single model across several upstream providers and pick one per request. Prompt caches are per-provider (often per-node inside a provider). So when your second identical request lands on a different upstream than the first, it is a cache miss, even though your prompt did not change one byte. This is provider drift, and on a pay-per-token model it quietly multiplies your cost.

The two conditions that trigger it

This is not a misconfiguration you opted into. It is what you get out of the box:

Default auto routing. The request is sent to the model without pinning an upstream, so the gateway chooses one per call.
Default provider sort = "default (balanced)". The gateway load-balances across eligible upstreams rather than sticking to one.

Both are the factory defaults. You do not have to touch anything to get drift; you have to touch settings to avoid it.

What 20 identical requests look like

We sent the same ~8K-token prefix 20 times in a row to one popular multi-provider gateway, on the defaults above, asking for the upstream's own reported provider and cache fields each time. For a disk-cached model in the DeepSeek family:

9 distinct upstreams served the 20 calls: N***a, S***w, M***h, D***a, A***L, P***l, S***e, V***e, A***d.
Cache hit rate: 4/20 (20%). You only hit on the calls that happened to land on an upstream that had already cached your prefix.

Run the same 20 calls against a single-backend gateway (one model, one upstream, no balancing) and the hit rate is 19/20 (95%) on the identical workload. Same model, same prompt, same number of calls. The only variable is whether routing drifts.

For contrast, on the very same multi-provider gateway a GPT-class model was routed to one upstream (A***e) for all 20 calls and hit 19/20. Drift is not uniform; it bites whichever model the gateway happens to spread, and on this run that was the DeepSeek-family model.

Conclusion A: the cost you expected vs the cost you paid

Per-call cost on the drifting model split cleanly by cache outcome:

call type	median cost / call
cache hit	~$0.00015
cache miss	~$0.00062

A miss costs about 4x a hit on this model (on raw input tokens the published gap is wider still, roughly 50x). Now total it across the 20 calls:

scenario	hit rate	cost for 20 identical calls
expected (cache reachable)	95%	$0.0026
actual (default drift)	20%	$0.0102

Same model, same prompt, same 20 requests. Provider drift made the run cost ~3.9x more. The caching was "on" the whole time; the routing layer simply billed most of your tokens at the miss rate. Scale that to a production endpoint replaying a large stable prefix all day and the gap is the bulk of your input spend.

Conclusion B: no cache also means no latency win

Caching is not only a cost lever. A warm prefill returns the first token sooner. When drift denies you the cache, you forfeit that speedup too. We measured time-to-first-token (TTFT) on repeated identical calls:

GPT-class model (routed to one consistent upstream, cache reachable):

call	TTFT
1st (cold, miss)	~1760 ms
subsequent (warm, hit)	~1130 ms

Caching buys roughly a 36% faster first token, and it is steady: every warm call lands in a tight band.

DeepSeek-family model (default drift, cache rarely reachable):

Cache hits across a 10-call repeat: 0.
TTFT swung from ~1000 ms to ~4500 ms call to call, with occasional empty responses.

Because almost every request is a fresh upstream, you stay at cold-prefill latency and inherit the variance of whichever provider answered. The GPT model got a 36% TTFT improvement from a reachable cache; the drifting model got none, plus a 4.5x spread between its fastest and slowest call.

Audit your own setup in five minutes

Do not trust these numbers, or anyone's. Send the same long prefix several times and watch two fields. No domains hardcoded; point it at your own gateway with env vars.

import os, uuid
from openai import OpenAI

client = OpenAI(api_key=os.environ["GW_KEY"], base_url=os.environ["GW_BASE"])
SYS = f"[probe {uuid.uuid4().hex}]\n\n" + ("You are a support assistant. " * 300)

seen, hits = {}, 0
for i in range(20):
    r = client.chat.completions.create(
        model=os.environ["GW_MODEL"], max_tokens=16,
        messages=[{"role": "system", "content": SYS},
                  {"role": "user", "content": f"q{i}"}],
        extra_body={"usage": {"include": True}})
    d = r.model_dump()
    det = r.usage.prompt_tokens_details
    cached = (getattr(det, "cached_tokens", 0) or 0) if det else 0
    seen[d.get("provider")] = seen.get(d.get("provider"), 0) + 1   # populated when exposed
    hits += 1 if cached else 0

print(f"hit rate {hits}/20; upstreams seen: {len(seen)}")

More than one upstream for the same model means drift. A hit rate well below your prompt stability means it is taxing you. The fuller method is in Does Your LLM Gateway Lie About Cache?.

What to look for

The cure for drift is structural: route a given model to a consistent backend so a warm cache is actually reachable on the next request, instead of load-balancing each call onto a fresh upstream that has never seen your prefix. When you evaluate a gateway, send the same prefix 20 times and count the upstreams. One is what you want. Nine is a tax.

A fair caveat: prompt caching is best-effort everywhere, and on disk-cached models the hit rate still softens over long idle gaps even with a single backend. Eliminating drift does not hand you an infinite cache. It removes the largest and most wasteful source of misses, the one you never agreed to and cannot see.

Closing

"Supports prompt caching" and "your cache is reachable" are different claims. A gateway that scatters one model across a rotating cast of upstreams can report cache support truthfully while delivering a 20% hit rate, a ~4x bill, and first-token latency that swings 4.5x. The number to watch is not whether caching is advertised. It is your measured hit rate and how many upstreams your identical requests touch. Run the probe and let the data settle it.

For the broader audit method see Does Your LLM Gateway Lie About Cache?; for why caches exist at all, see How KV Cache & TTL Work.

FAQ

Is this a misconfiguration on my side?
No. It happens on the factory defaults: auto routing with the provider sort left at "default (balanced)." Avoiding drift requires actively pinning an upstream, not the other way around.

Does pinning one upstream fix it?
It removes cross-provider drift, but a single upstream often runs multiple replicas without prefix affinity, so hits can still flip-flop. Measure after pinning rather than assuming.

Why did the GPT-class model not drift?
On this run the gateway happened to route it to a single upstream. Drift is per-model and depends on how many eligible upstreams the gateway balances across; it is not uniform.

Is the cost gap really ~4x?
On the per-call totals we measured, a miss was ~4x a hit; on raw input-token pricing for this model class the published hit-vs-miss gap is closer to 50x. Either way, turning expected hits into misses is the expensive part.

What single metric should I monitor?
Cache hit rate per model over time, alongside the count of distinct upstreams per model. If hit rate falls or upstream count rises, your effective token cost just went up.

Does Your LLM Gateway Lie About Cache? A 5-Min Audit

synthorai — Tue, 02 Jun 2026 15:30:00 +0000

A gateway sits between your code and the model provider. You read cached_tokens back from the response, you see a smaller number, and you trust the dollars saved are real. But you never see the upstream call. The gateway could report a cache hit and still bill the full input rate. It could fail to cache at all behind a perfectly clean response. It could strip usage metadata on streaming, the path most of your production traffic runs on, so you can't tell either way.

This isn't hypothetical. A Hacker News PSA reported that routing DeepSeek V4 through a popular gateway returned 2–3× fewer cached tokens than calling DeepSeek directly; one commenter posted bills showing the caching stats weren't reported through the gateway at all. The gateway's team replied that they couldn't reproduce it and were investigating. That disagreement is the whole point. When two parties can't agree on whether your cache is working, the only tiebreaker is a measurement you ran yourself.

Usually this isn't malice. It's a translation gap or an unfinished code path. The effect on your invoice is the same either way. This post is one runnable script that audits both styles of prompt caching, automatic (DeepSeek) and marker-based (Claude), against any gateway, including this one. It prints a side-by-side scorecard in under five minutes.

Four ways a gateway can lie about cache

Failure mode	What you see	What's actually happening
Silent no-cache	A clean response, no error	Nothing was cached; you pay full price every call
Cache theater	`cached_tokens` > 0 in the response	…but the billed cost is the full input rate
Markup creep	A plausible cost number	The gateway's markup quietly eats the discount
Metadata blackout	Clean text output	Usage fields stripped (esp. on streaming), so you can't audit it

The dangerous ones are the first two: the response looks like caching is working. You find out at the end of the month.

Two cache mechanisms, one audit

Providers expose caching in two shapes, and a real gateway has to pass both through faithfully:

Automatic (DeepSeek, GPT, Gemini, Qwen): the provider caches any sufficiently long prefix on its own. No markers. Hits appear in usage.prompt_tokens_details.cached_tokens.
Marker-based (Anthropic Claude): you tag cacheable spans with cache_control. Hits appear as cache_read_input_tokens.

The script hides that difference behind a thin Lane adapter, then runs all five checks against both. Here is the whole thing: two lanes and one audit() that performs every check.

import os, time, uuid
from openai import OpenAI
from anthropic import Anthropic

KEY  = os.environ["GATEWAY_KEY"]
oai  = OpenAI(api_key=KEY,    base_url="https://synthorai.io/v1")   # auto lane
anth = Anthropic(api_key=KEY, base_url="https://synthorai.io/")     # marker lane

class AutoLane:      # DeepSeek / GPT / Gemini / Qwen: provider caches automatically
    mode = "auto"
    def __init__(self, model): self.model = model
    def call(self, sys, q, stream=False):
        if stream:
            cached = cost = None
            s = oai.chat.completions.create(model=self.model, max_tokens=48, stream=True,
                stream_options={"include_usage": True},
                messages=[{"role":"system","content":sys},{"role":"user","content":q}])
            for ev in s:
                if ev.usage:
                    d = ev.usage.prompt_tokens_details
                    cached, cost = (d.cached_tokens if d else None), getattr(ev.usage,"cost",None)
            return {"cached": cached or 0, "cost": cost, "prompt_total": None}
        u = oai.chat.completions.create(model=self.model, max_tokens=48,
            messages=[{"role":"system","content":sys},{"role":"user","content":q}]).usage
        cached = u.prompt_tokens_details.cached_tokens if u.prompt_tokens_details else 0
        return {"cached": cached or 0, "cost": u.cost, "prompt_total": u.prompt_tokens}

class MarkerLane:    # Anthropic Claude: explicit cache_control markers
    mode = "marker"
    def __init__(self, model): self.model = model
    def call(self, sys, q, stream=False):
        block = {"type":"text","text":sys,"cache_control":{"type":"ephemeral"}}
        if stream:
            with anth.messages.stream(model=self.model, max_tokens=48, system=[block],
                    messages=[{"role":"user","content":q}]) as s:
                for _ in s.text_stream: pass
                u = s.get_final_message().usage.model_dump()
            return {"cached": u.get("cache_read_input_tokens") or 0,
                    "cost": u.get("cost"), "prompt_total": None}
        u = anth.messages.create(model=self.model, max_tokens=48, system=[block],
            messages=[{"role":"user","content":q}]).usage.model_dump()
        read, created = u.get("cache_read_input_tokens",0), u.get("cache_creation_input_tokens",0)
        return {"cached": read, "cost": u.get("cost"),
                "prompt_total": u.get("input_tokens",0) + read + created}

def audit(lane, long_prompt):
    SYS = f"[audit {uuid.uuid4().hex}]\n\n" + long_prompt    # unique => guaranteed cold start
    r = {"lane": lane.model, "mode": lane.mode}

    # CHECK 1: cache engages. Cold misses; a repeat should hit. A cache can
    # take a moment to become readable, so poll the warm read (sleep 1s between
    # attempts) before concluding "no cache".
    cold = lane.call(SYS, "Q1")
    warm = cold
    for i in range(4):
        warm = lane.call(SYS, f"warm {i}")
        if warm["cached"] > 0: break
        time.sleep(1.0)
    r["cold"], r["warm"] = cold, warm
    r["check1"] = cold["cached"] == 0 and warm["cached"] > 0

    # CHECK 2: cost reflects the discount (catches "cache theater").
    disc = (1 - warm["cost"]/cold["cost"])*100 if cold["cost"] and warm["cost"] else None
    r["discount"], r["check2"] = disc, (disc is not None and disc > 30)

    # CHECK 3: token accounting. cached fits inside the prompt total.
    r["check3"] = warm["prompt_total"] is None or warm["cached"] <= warm["prompt_total"]

    # CHECK 4: streaming preserves usage metadata (cache count AND cost).
    st = lane.call(SYS, "stream", stream=True)
    r["stream_cached"], r["stream_cost"] = st["cached"] > 0, st["cost"] is not None
    r["check4"] = r["stream_cached"] and r["stream_cost"]

    # CHECK 5: negative control. a unique prefix must always miss.
    n1 = lane.call(f"[uniq {uuid.uuid4().hex}]\n\n"+long_prompt, "x")
    n2 = lane.call(f"[uniq {uuid.uuid4().hex}]\n\n"+long_prompt, "y")
    r["check5"] = n1["cached"] == 0 and n2["cached"] == 0
    return r

# Any long, STABLE text works as the cacheable prefix: a system prompt, tool
# schemas, or a retrieved document. It only needs to clear the provider's
# minimum cacheable size (see Check 1). Load yours however you like.
LONG_SYSTEM_PROMPT = open("system_prompt.txt").read()   # ~8K+ tokens

for lane in [AutoLane("deepseek-v4-flash"), MarkerLane("claude-opus-4-8")]:
    print(audit(lane, LONG_SYSTEM_PROMPT))

The rest of the post walks each check: the lines that implement it, what both lanes returned, and how to read the result.

Check 1: does the cache engage?

cold = lane.call(SYS, "Q1")
warm = cold
for i in range(4):                       # poll: a cache may take a beat to be readable
    warm = lane.call(SYS, f"warm {i}")
    if warm["cached"] > 0: break
    time.sleep(1.0)
r["check1"] = cold["cached"] == 0 and warm["cached"] > 0

	cold cached	warm cached	result
`deepseek-v4-flash`	0	7,552 / 7,870 (96%)	PASS
`claude-opus-4-8`	0	12,446 / 12,454 (99.9%)	PASS

A cold call on a unique prefix must cache nothing; a repeat must hit. The single most common false alarm is declaring "no cache" after one warm call, because caches don't always become readable instantly. The loop polls a few times with a 1-second pause, which removes the flakiness. If you still get 0 after several warm calls on a prompt above the size floor (~1,024 tokens for most providers; DeepSeek matches at a finer 64), the cache genuinely isn't engaging.

Check 2: does the cost reflect the discount?

disc = (1 - warm["cost"]/cold["cost"])*100 if cold["cost"] and warm["cost"] else None
r["check2"] = disc is not None and disc > 30

	cold cost	warm cost	discount	result
`deepseek-v4-flash`	$0.00107	$0.00030	72.3%	PASS
`claude-opus-4-8`	$0.07112	$0.00672	90.6%	PASS

This is the check that catches cache theater. The warm call's cost must actually drop. DeepSeek's per-call total fell ~72% (the cached input is discounted more steeply; output and the uncached remainder dilute the headline). Claude's cached read is ~90% off. The failure signal is unmistakable: cached_tokens > 0 with identical cold and warm cost means the gateway is reporting a hit it isn't pricing. You're paying full freight for a cache that "works" on paper.

Check 3: do the token counts add up?

r["check3"] = warm["prompt_total"] is None or warm["cached"] <= warm["prompt_total"]

	cached	prompt total	result
`deepseek-v4-flash`	7,552	7,870	PASS
`claude-opus-4-8`	12,446	12,454	PASS

cached has to sit inside the prompt total, with the remainder billed as uncached input. Both reconcile. If cached_tokens exceeds prompt_tokens, or the uncached remainder is implausibly large for a stable prefix, the gateway is mis-accounting: re-tokenizing or double-counting somewhere in the translation.

Check 4: does streaming preserve the metadata?

st = lane.call(SYS, "stream", stream=True)
r["stream_cached"], r["stream_cost"] = st["cached"] > 0, st["cost"] is not None
r["check4"] = r["stream_cached"] and r["stream_cost"]

	stream cached	stream cost	result
`deepseek-v4-flash`	preserved	preserved	PASS
`claude-opus-4-8`	preserved	preserved	PASS

Most production chat streams, so this is the path that matters most. On both lanes the cache hit signal and the cost both survive the stream. cached_tokens and cost come through in the final usage chunk, so your highest-volume path stays auditable. The failure mode to watch for is a gateway that drops usage on streaming: a clean token output with no cached_tokens or cost means you're flying blind on the path you run most. (Pass stream_options={"include_usage": True} so the usage chunk is emitted at all.)

Check 5: the negative control

n1 = lane.call(f"[uniq {uuid.uuid4().hex}]\n\n"+long_prompt, "x")
n2 = lane.call(f"[uniq {uuid.uuid4().hex}]\n\n"+long_prompt, "y")
r["check5"] = n1["cached"] == 0 and n2["cached"] == 0

	unique-prefix A	unique-prefix B	result
`deepseek-v4-flash`	cached 0	cached 0	PASS
`claude-opus-4-8`	cached 0	cached 0	PASS

Send a unique prefix every call; it must never hit. Both lanes correctly reported cached=0 at full cost for distinct prefixes. A "hit" here would make the cache reporting a false positive you could never trust. The clean negative control is what makes the positive results in Checks 1–2 meaningful in the first place.

Reading your scorecard

Check	Healthy result	Red flag
1. cache engages	`0` cold, `>0` warm (after polling)	`0` after several warm calls, above the size floor
2. cost reflects discount	warm cost ≪ cold cost	`cached > 0` but costs equal
3. token accounting	`cached ≤ prompt_total`, reconciles	counts don't add up
4. streaming metadata	cache + cost survive the stream	usage missing on streamed calls
5. negative control	unique prefix always misses	a distinct prefix "hits"

The two that cost money silently are 2 (full price for a reported hit) and 1 (no caching behind a clean response). Run both on every model you bill against.

Closing

Caching is the highest-leverage cost lever in an LLM app, which is exactly why "the cache is working" deserves a test, not an assumption. Wire Check 1 + Check 2 into CI against each model you bill against, alert if the discount drifts below your expected band, and you'll catch a silent regression the day a gateway or upstream provider changes behavior, instead of at the end of the billing cycle. And whatever your audit does, poll the warm read before you call a cache broken.

For the mechanics behind these numbers (prefill, KV cache, TTLs) start with How KV Cache & TTL Work. For working caching patterns per provider, see the tutorial.

FAQ

My Check 1 shows 0 on the warm call. Is my gateway lying?
Check three things first. (1) Does your prompt clear the provider's minimum cacheable size (~1,024 tokens for most; DeepSeek matches at finer 64-token granularity)? (2) Did you poll the warm read a few times? Caches don't always become readable on the very next call. (3) Is the prefix byte-identical between calls, with no timestamps or per-request IDs at the front? Only after all three should you suspect the gateway.

What does "cache theater" cost me in practice?
You pay the full input rate on every call while believing you pay a fraction. On a high-volume endpoint with a large stable prefix, that's your bill being several times what you modeled. Check 2 is the one to alert on.

Why is DeepSeek's discount lower than Claude's here?
Different things are being measured. Claude's ~90% is the read discount on cached input. DeepSeek's ~72% is the per-call total reduction, where output and the uncached remainder are billed at full rate and dilute the headline. Compare like with like for your own prompt shape.

Does this work for GPT, Gemini, Qwen too?
Yes. They're all automatic, so they use the AutoLane unchanged with a different model. Only Claude needs the MarkerLane. Same five checks either way.

Should this live in CI?
Yes. Run Check 1 + Check 2 against every model you bill against, on a schedule, and alert when the observed discount drifts outside your expected band. A standing audit turns a silent regression into a notification.

Claude Opus 4.8 on Synthorai: Caching & TTL vs 4.7/4.6

synthorai — Fri, 29 May 2026 14:47:14 +0000

claude-opus-4-8 is now available on the Synthorai gateway. If you already run prompt caching against the Opus line, the headline is reassuring and slightly boring: nothing about the caching or TTL contract changed from 4.7 or 4.6. Same cache_control markers, same 5-minute and 1-hour TTLs, same read discount, same write premiums. Your caching code is a drop-in carry-over.

There is exactly one thing that did change — and it changed back at 4.7, not at 4.8 — that affects your token budget. This post measures it so you don't have to.

All numbers below were measured against https://synthorai.io/ (Anthropic-native /v1/messages) on 2026-05-29 with a ~8K-character English system prompt, max_tokens small, single sequential run. Reproduce against your own prompt before quoting them.

Availability

import os
from anthropic import Anthropic

anth = Anthropic(
    api_key=os.environ["SYNTHORAI_KEY"],
    base_url="https://synthorai.io/",   # SDK appends /v1/messages
)

msg = anth.messages.create(
    model="claude-opus-4-8",            # the only line that changes
    max_tokens=512,
    system=[
        {"type": "text", "text": SYSTEM_PROMPT,
         "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": question}],
)
print(msg.usage)   # cache_creation_input_tokens, cache_read_input_tokens, cost

Swap claude-opus-4-7 → claude-opus-4-8 and nothing else in your caching path needs to move. The mechanics behind cache_control are covered in the caching tutorial; the architecture of why the cache exists is in Part 1 of the series.

Caching behavior: unchanged from 4.7/4.6

We ran the same cache write / cache read / no-cache sequence across the recent Opus line. The discount structure is identical end to end.

Model	No-cache cost	5m cache write	Cache read	Read discount
`claude-opus-4-5`	$0.0364	$0.0452	$0.0041	88.8%
`claude-opus-4-6`	$0.0364	$0.0452	$0.0041	88.7%
`claude-opus-4-7`	$0.0522	$0.0654	$0.0059	88.7%
`claude-opus-4-8`	$0.0520	$0.0654	$0.0059	88.6%

Two invariants hold across all four versions:

Read discount ≈ 89%. A warm cache read costs ~11% of the no-cache input price. This is Anthropic's documented 10% cached-read rate, unchanged.
Write premium ≈ 25%. The first (cold) call costs ~1.25× the no-cache price to populate the cache. Break-even is one hit.

The absolute dollar figures for 4.7 and 4.8 are higher than 4.5/4.6, but as we'll see in a moment that's a token-count story, not a cache-economics story — the percentages are flat.

TTL behavior: unchanged from 4.7/4.6

Opus 4.8 honors the same two TTLs as the rest of the line: a 5-minute sliding default and an opt-in 1-hour window. We isolated the TTL path with a unique prefix per call (so no stale cache entry could contaminate the result) and measured the write premium for each TTL:

Model	TTL	Cache write	Write premium vs no-cache
`claude-opus-4-7`	5m	$0.0650	~1.25×
`claude-opus-4-7`	1h	$0.1036	~2×
`claude-opus-4-8`	5m	$0.0650	~1.25×
`claude-opus-4-8`	1h	$0.1036	~2×

# 1-hour TTL — same marker syntax on 4.8 as on 4.7/4.6
"cache_control": {"type": "ephemeral", "ttl": "1h"}

The usage object reports the TTL bucket exactly as before — cache_creation.ephemeral_5m_input_tokens or ephemeral_1h_input_tokens. The 1-hour write costs ~2× no-cache (vs ~1.25× for the 5-minute write), and reads stay at ~11% regardless of TTL. Identical to 4.7. If you picked 5m for live chat and 1h for agents with human-in-the-loop pauses on 4.7, keep those choices on 4.8.

Time-to-first-token: flat across the line

We measured warm-read TTFT with a streaming call (5 samples per model after a gateway warm-up, median reported). On this ~8–11K-token prompt, TTFT sits in a ~2.2–2.8 s band with no material per-version trend — the sample ranges overlap, so the differences are jitter, not a version effect.

Model	Warm-read TTFT (median)	Range (n=5)
`claude-opus-4-5`	2.72 s	2.58 – 2.78 s
`claude-opus-4-6`	2.76 s	2.65 – 3.01 s
`claude-opus-4-7`	2.21 s	1.98 – 2.97 s
`claude-opus-4-8`	2.47 s	2.23 – 4.38 s

Two caveats worth stating plainly:

Don't read a ranking into this. The ranges overlap heavily (4.8's high sample was an outlier at 4.38 s); on this prompt size TTFT is dominated by network and queueing jitter, not the model version. Treat ~2.2–2.8 s as the warm band for all four.
The cache TTFT win scales with prompt length. At ~8–11K tokens the prefill saved by a cache hit is small, so cold and warm TTFT are close (both ~2–3 s on a warmed gateway). The gap widens substantially at 100K+ tokens, where prefill dominates — that's where a warm cache turns a multi-second wait into a fast first token. The mechanics are in Part 1: How KV Cache & TTL Work.

The one real change: tokenization (since 4.7)

Here is the thing to re-check before you migrate. The same system text reports ~43% more input tokens on 4.7/4.8 than on 4.5/4.6.

Model	Input tokens (identical text)	No-cache cost
`claude-opus-4-5`	~7,976	$0.0364
`claude-opus-4-6`	~7,977	$0.0364
`claude-opus-4-7`	~11,393	$0.0522
`claude-opus-4-8`	~11,394	$0.0520

The token count jumps at the 4.7 generation and carries into 4.8. The cost tracks the token count almost exactly: the cost ratio (4.8 / 4.5) is 1.43, and the token ratio is 1.429. In other words, the per-token price is the same across the whole line — the higher bill on 4.7/4.8 comes entirely from the same text counting as more tokens.

Two practical consequences:

Re-budget on absolute cost, not on discount. Your cache discount is unchanged (~89% read), but the same English prompt is ~43% more expensive in absolute terms on 4.7/4.8 than it was on 4.6. If you sized a per-call budget against 4.6 token counts, it will be off.
Re-check the 1,024-token cache-eligibility floor. Anthropic only caches prefixes at or above a minimum size. A prompt that sat just under the floor on 4.6 may clear it on 4.7/4.8 (more tokens), and a prompt sized in tokens for the old tokenizer needs re-measuring. Always read cache_creation_input_tokens / cache_read_input_tokens from the live response rather than estimating from a local tokenizer that may not match.

We're describing a measured observation — identical text, ~43% more reported input tokens on 4.7/4.8 — most consistent with a tokenizer/vocabulary update at the 4.7 generation. The takeaway doesn't depend on the root cause: re-measure token counts when you migrate, because the cache math is token-based.

Migration checklist (4.6/4.7 → 4.8)

✅ Caching code carries over verbatim. cache_control markers, breakpoint count (up to 4), ttl: "1h", usage-field names — all identical.
✅ TTL choices carry over. 5m for live/session workloads, 1h for bursty/agent-with-pauses.
✅ Discount economics carry over. ~89% read, ~1.25× write (5m), ~2× write (1h).
⚠️ Re-measure token counts. If you're coming from 4.5/4.6, expect ~40%+ more input tokens for the same text (this happened at 4.7). Coming from 4.7, expect parity.
⚠️ Re-validate cost dashboards. Trust usage.cost and the *_input_tokens fields from the live response, not a cached estimate from the old generation.

Bottom line

For an engineering team already caching against Opus, claude-opus-4-8 is the easy kind of upgrade: the entire caching and TTL surface is stable, so there's nothing to relearn and no code to rewrite. Budget for the tokenizer shift if you're jumping from 4.6 or earlier, confirm your numbers against the live usage object, and ship.

For the full caching playbook — prompt structure, hit-rate debugging, TTL-aware patterns — see the four-part series starting with How KV Cache & TTL Work and the working Python tutorial.

FAQ

Do I need to change my cache_control code to use Opus 4.8?
No. The marker syntax, breakpoint limit, and TTL options are identical to 4.7/4.6. Change the model field and nothing else.

Did the cache read discount change on 4.8?
No. A warm read is ~11% of the no-cache input price (~89% off) on 4.5 through 4.8, matching Anthropic's documented rate.

Did the 1-hour TTL premium change?
No. The 1-hour write costs ~2× the no-cache input price; the 5-minute write costs ~1.25×. Reads are ~11% regardless of TTL. Same as 4.7.

Why is the same prompt more expensive on 4.8 than on 4.6?
The per-token price is the same — the prompt simply counts as more tokens. Identical text reported ~8.0K tokens on 4.5/4.6 and ~11.4K on 4.7/4.8 in our measurements (a ~43% increase), most consistent with a tokenizer change at the 4.7 generation. The cache discount is unchanged.

Is 4.8 a drop-in replacement for 4.7?
On the caching/TTL surface, yes — token counts and economics were already at the 4.7 level, so migration from 4.7 is parity. We don't publish capability benchmarks we haven't run; for quality and reasoning claims, see Anthropic's model card.

Verification: all caching, TTL, token-count, cost, and TTFT figures measured against https://synthorai.io/ on 2026-05-29 using the official anthropic SDK, single tenant. Cost/token figures are a single sequential run; TTFT is a 5-sample median per model after gateway warm-up. Discount/premium ratios cross-checked against Anthropic Prompt Caching docs. Your numbers will vary with prompt, region, and load.

LLM Prompt Caching: The Complete 2026 Guide

synthorai — Wed, 27 May 2026 15:30:00 +0000

If you ship a chatbot, a RAG app, or an AI agent against a large language model, prompt caching is the single optimization that gives you back 50–90% of input cost and 3–10× of time-to-first-token at no quality cost. It isn't a bolt-on trick — it falls directly out of how Transformer attention is defined. Once you understand that, the rest of the stack (TTLs, provider differences, prompt structure) lines up cleanly.

This page is the index to a four-part series that takes you from the theory to a production decision matrix. Pick where to enter based on what you already know.

Where to enter

If you want to...	Start at
Understand why caching exists and what KV cache actually is	Part 1 — How KV Cache & TTL Work
Pick a provider and know what's different about each	Part 2 — Compare Claude, GPT, Gemini, DeepSeek
Copy-paste working Python and measure your own numbers	Part 3 — Working Python Tutorial
Match a chatbot / RAG / agent workload to the right model	Part 4 — Best Model for Chat, RAG & Agents

Each part stands alone but they're written so reading them in order builds the picture without redundancy.

Part 1 — How LLM Prompt Caching Works

LLM Prompt Caching #1: How KV Cache & TTL Work →

The architectural article. Walks through self-attention as a single equation, explains why the K and V vectors of a stable prefix are mathematically reusable, and shows how the memory-vs-compute tradeoff produces the TTL behavior every developer has to design around.

Key takeaways:

Prompt caching isn't an optimization layered on top — it's a direct consequence of causal-masked attention. K/V at position i is a deterministic function of tokens 1…i, so identical prefixes give bit-identical K/V.
Prefill (compute-bound, O(N²)) is what caching saves; decode (memory-bandwidth-bound, O(N) per token) is what every inference engine already optimizes.
TTLs exist because KV cache is enormous (~10 GB for a 32K context on a 70B model). 5 minutes is the GPU memory-pressure horizon; hours-to-days are only possible with disk-backed caches (DeepSeek's MLA architecture).
Caching wins both cost (50–90% off input on cache hits) and latency (TTFT drops 3–10× for prompts in the 5–10K-token range and much more for 100K+).

Part 2 — Compare LLM Prompt Caching Across Providers

LLM Prompt Caching #2: Compare Claude, GPT, Gemini, DeepSeek →

The buyer's guide. Five providers expose prompt caching in five very different shapes — explicit markers (Claude), fully automatic (GPT-5, DeepSeek-v4), hybrid implicit+explicit (Gemini, Qwen), or architectural disk-backing (DeepSeek's MLA). The article gives a feature-by-feature comparison plus a 5-dimension evaluation framework to score them for your specific workload.

Key takeaways:

Don't compare base prices — compare effective cost weighted by your hit rate (formula in §4.1).
Claude has the deepest single-call discount (~90%) but requires explicit cache_control markers.
DeepSeek-v4 is the only provider with disk-backed caches at scale; partial-prefix matches earn discounts because the granularity is 64 tokens instead of 1,024.
Gemini's explicit cache costs hourly storage fees — break-even depends on call frequency.
API ergonomics, hit-rate predictability, TTL fit, latency under miss, and migration cost are the five dimensions that actually distinguish providers once you control for hit rate.

Part 3 — Working Python Tutorial

LLM Prompt Caching #3: Working Python Tutorial →

The hands-on article. One OpenAI SDK + one Anthropic SDK against a single gateway, with measured numbers from 2026-05-25 across the full Claude family (haiku-4-5 through opus-4-7), GPT-5.x, Gemini 2.5, DeepSeek-v4, and Qwen3.

Key takeaways:

Claude with cache_control markers: measured 88–89% cost reduction uniformly across haiku/sonnet/opus 4-x. Use the Anthropic SDK with base_url="https://synthorai.io/".
GPT-5.4-mini auto-cache: 5× TTFT improvement (3.6 s → 0.73 s on a 7K-token prompt), 93% cache hit rate on the system tokens.
Gemini 2.5-flash implicit: 88% cost reduction on cache hits when streaming usage is captured.
DeepSeek-v4-flash: 74% off, disk-backed (cache survives hour-scale idle).
TTL-aware patterns: keep-alive heartbeat for cron, prefix stability rules, what to log per call.

Part 4 — Best Model by Use Case

LLM Prompt Caching #4: Best Model for Chat, RAG & Agents →

The decision article. Different workloads pull the cost/latency levers differently — chat is naturally cache-friendly, RAG fights the prefix-stability problem, agents depend on cumulative prefix discipline. The article gives a model recommendation by workload shape with cost estimates.

Key takeaways:

Chatbots: any model with auto-cache works; sessions hit naturally. Pick on cost/quality. gpt-5.4-nano cheapest, gpt-5.4-mini fastest cached TTFT, claude-haiku-4-5 best instruction-following at modest premium.
RAG: retrieved-doc reordering kills mid-prompt cache hits. Three fixes — push references to the end, deterministic chunk ordering, or Claude's multi-cache_control breakpoints.
Agents: tool calls and results must be append-only and byte-identical step-to-step. claude-sonnet-4-5 with 4 cache_control markers gives the strongest cumulative-prefix discount; gpt-5.4-mini works without code changes at 50% savings.
TTL match: 5 min for chat, 1 hour for agents with human-in-the-loop steps, disk-backed for sporadic batch.

How to read this

Engineer new to the topic: read in order. The architecture in Part 1 makes Parts 2–4 click instantly.
PM or architect doing vendor selection: jump to Part 2 + Part 4. Reference Part 1 if a teammate asks "but why TTL exists".
Engineer with a specific workload to ship today: Part 4 first (find your row in the matrix), then Part 3 for the exact code.
Anyone optimizing an existing app: Part 3 §6 cross-provider benchmark — reproduce it against your own prompt; that's a one-day exercise, not a multi-week migration.

Numbers in this series

All measured numbers were captured on 2026-05-25 against the Synthorai gateway (https://synthorai.io/v1 for OpenAI-compat, https://synthorai.io/ for Anthropic-native), single-tenant, single sequential run, no concurrent load. Your numbers will move with region, time-of-day, and competing tenant load — treat them as a starting point and reproduce against your own traffic before quoting them.

Pricing tables and TTL behavior reflect vendor public documentation as of 2026-05. Providers update these every few months; the architectural reasoning (Part 1) is stable, the comparative numbers (Part 2 & 3) drift.