synthorai

Posted on Jul 4 • Originally published at synthorai.io

LLM Prompt Caching #5: LangChain Setups That Actually Hit

#caching #langchain #tutorial

Here is a LangChain system prompt that looks perfectly reasonable and caches nothing:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", BIG_STABLE_SYSTEM_PROMPT),   # the syntax every tutorial uses
    ("human", "{question}"),
])

We ran this against claude-sonnet-5 twice with an identical 1,800-token system prompt and read the usage fields back. Both calls: cache writes 0, cache reads 0. Not a partial hit, not a fragmented cache. Nothing. The reason: Anthropic only caches what you mark with cache_control, and a plain string in a ("system", ...) tuple has nowhere to put the marker. The most convenient syntax in LangChain is also the one that leaves the entire discount on the table, and no error tells you.

This is part 5 of the caching series. Part 1 covers how prefix caching works, part 3 does the raw-SDK tutorial. This part is about what changes when LangChain assembles your prompts for you. Everything below was measured on 2026-07-04 through the Synthorai gateway with langchain-core 1.4.8, langchain-anthropic 1.4.8, and langchain-openai 1.3.3.

First, which "caching" are you looking for?

Two unrelated features share the word, and the LangChain docs page you land on when you search is usually the wrong one.

	Response caching (LangChain's `InMemoryCache`)	Prompt caching (this series)
What it stores	The whole completion, in your app	The prompt prefix's KV state, on the provider
When it saves money	The exact same request repeats	Different requests share a prefix
Where	`set_llm_cache(InMemoryCache())`, SQLite, Redis	`cache_control` markers or automatic prefix matching
Agent loops, RAG, chat	Almost useless (every request differs)	The main lever, since system + tools repeat every turn

And "exact same request" means exact: the built-ins key on the pair (serialized prompt, model-config string). Measured: an identical repeat returned in 0 ms with no API call; adding one space to the prompt missed; the same prompt with max_tokens changed by one also missed. (The cached replay also returns the original call's usage numbers, so naive token accounting double-counts.) Semantic caches exist as third-party integrations; the built-ins are exact-match only.

So set_llm_cache is fine for deduplicating identical calls in tests; the 2,000-token system prompt you re-send on every agent turn is prompt caching's job, and it needs the prompt assembled the right way.

The fix: content blocks, not strings

cache_control travels inside a content block, so the system message has to be a SystemMessage with block content rather than a bare string:

from langchain_anthropic import ChatAnthropic
from langchain_core.messages import SystemMessage
from langchain_core.prompts import ChatPromptTemplate

llm = ChatAnthropic(
    model="claude-sonnet-5",
    base_url="https://synthorai.io",   # any Anthropic-compatible endpoint
)

prompt = ChatPromptTemplate.from_messages([
    SystemMessage(content=[{
        "type": "text",
        "text": BIG_STABLE_SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"},   # a bare string has nowhere to put this
    }]),
    ("human", "{question}"),
])
chain = prompt | llm

Same 1,800-token system prompt, measured through the same gateway:

Call	String-tuple syntax	Content-block syntax
1st (cold)	write 0 / read 0	write 1,875 / read 0
2nd, different question	write 0 / read 0	write 0 / read 1,875

A warm read bills at roughly 10% of the input price, so on Claude this one structural change is the difference between paying full price forever and a 90% discount on the stable half of every call. The economics are in part 1; the marker mechanics match the raw SDK usage in the LangChain Anthropic integration docs and Anthropic's prompt caching guide.

Where your template variables go decides your hit rate

LangChain templates make it effortless to interpolate variables anywhere, which is exactly the hazard. The cache key is the byte-exact prefix. We put a date inside the cached block and measured:

SystemMessage(content=[{
    "type": "text",
    "text": f"Today is {today}. " + BIG_STABLE_SYSTEM_PROMPT,   # variable INSIDE the block
    "cache_control": {"type": "ephemeral"},
}])

Call	Result
day A, question 1	write 1,865 (cold for this value)
day A, question 2	read 1,865 (same value, hit)
day B, question 1	write 1,865 (new value, cold again)

The cache did not break. It got keyed on the variable. A value that repeats, like a date, costs one cache write per value and hits after that. A value that is unique per call, like a timestamp or a request ID, makes every call a cold write and the hit rate exactly zero.

The expensive real-world version of this mistake is RAG. The template many chains reach for puts the retrieved context at the top of the system prompt, before the static instructions. We measured both orders with an 800-token retrieved context that changes per query and a marked system block:

Order inside the prompt	Call 1	Call 2 (new query, new context)
Context first, then rules	write 3,133	write 3,133 again, read 0
Rules first (marked), context in the human turn	write 1,852	read 1,852

The wrong row is not merely "no discount": every call pays the cache-write premium, about 1.25× the normal input price, on the full 3,133 tokens, and nothing is ever read back. A mis-ordered RAG prompt with caching enabled costs more than not caching at all. The fixed content sits after the variable content, so it might as well not exist.

The rule that falls out of the measurements:

Static text first, inside the marked block. System rules, tool definitions, few-shot examples.
Anything that varies goes after the marker, ideally in the human turn: retrieved context, dates, user questions.
A variable inside the block is acceptable only if it repeats often enough to amortize its own cache write.

Tool definitions get cached too

An agent re-sends its tool schemas on every call, and in Anthropic's request layout the tools sit before the system prompt. Since a marker means "cache everything from the start of the request up to here," that raises two practical questions. Does a marker on the system block also cover the tools in front of it? And does LangChain's bind_tools turn your tools into the exact same bytes on every call, because if the serialization wobbles, the prefix changes and every call misses.

Measured answers to both. With the same marked system prompt, the warm cache read was 1,861 tokens without tools and 2,389 tokens with two tools bound: the extra 528 tokens are the tool schemas coming back out of the cache. And that 2,389 repeated exactly across three consecutive calls, which means bind_tools serializes the same way every time; the framework does not leak noise into the prefix. So to be explicit: as long as the system block carries the marker, the tools themselves need no cache_control; that one marker behind them does all the work.

The opposite arrangement exists for one specific shape: the tools are the biggest stable thing you send and the system prompt is thin or absent. Then the request still needs a marker somewhere, and it can live on a tool. This only works with the raw Anthropic-format dict, because a @tool-decorated function has no field to carry it; bind_tools passes the dict through untouched:

# variant: NO marked system block anywhere; the tool carries the request's only marker
llm.bind_tools([{
    "name": "get_weather",
    "description": LONG_TOOL_DESCRIPTION,
    "input_schema": {...},
    "cache_control": {"type": "ephemeral"},   # passes through bind_tools verbatim
}])

Measured: write 3,002 cold, read 3,002 warm, with no marked system message in the request.

Multi-turn: move the marker to the last message

A conversation looks like another ordering problem, but it is the opposite case: the order is already perfect, because history only ever appends, so the whole transcript is stable prefix. The problem here is coverage. A marker on the system block caches the system block and nothing after it: as the history grows, the warm read stays flat at the system size while every accumulated turn bills as ordinary input.

The pattern that fixes it is the same one the raw SDK uses: put the marker on the latest message, so the breakpoint slides forward and the whole conversation-so-far becomes the cached prefix:

def marked(text):
    return HumanMessage(content=[{
        "type": "text", "text": text,
        "cache_control": {"type": "ephemeral"},
    }])

# each turn: history stays plain, only the newest human message carries the marker
llm.invoke([system, *history, marked(new_question)])

Measured across two turns: turn 1 wrote 1,864; turn 2 read 1,864 and wrote only the 15-token delta (the previous answer plus the new question), the prior prefix billing at the ≈10% read rate. That is the shape an agent loop wants, and LangChain expresses it with an ordinary message list. Anthropic allows up to four markers per request, so the sliding marker composes with a fixed marker on the system block or on the tools.

Read the meters, and know their names

LangChain standardizes usage into usage_metadata, and here is the gotcha we hit: with langchain-anthropic 1.4.8, on every response in our runs, the standard input_token_details.cache_creation field stayed 0 even when a cache write happened. The real write count lands in a nonstandard key:

r = chain.invoke({"question": "..."})
det = r.usage_metadata["input_token_details"]
det["cache_read"]                  # correct on hits (1875 above)
det["cache_creation"]              # 0 even on a cold write; do not alert on this
det["ephemeral_5m_input_tokens"]   # the actual write count (1875)

The provider reported the write correctly (cache_creation_input_tokens: 1875 in the raw response, visible via r.response_metadata["usage"]); the standardized mapping just files it under the TTL-bucket key. A cost dashboard watching cache_creation will tell you caching is free while the write premium quietly accrues. Trust the raw usage object or know the bucket keys. This is the same class of problem as gateways mis-reporting cache fields, which we audit in Does Your LLM Gateway Lie About Cache?.

Implicit caches: mis-ordering fails silently, so watch it hardest here

Claude's cache is explicit. GPT and most open-weight providers cache automatically on prefix match, no markers, and through LangChain the same chain works after one constructor change:

llm = ChatOpenAI(model="glm-5.2", base_url="https://synthorai.io/v1")

Plain string system prompt, no markers: GLM 5.2's second call read 1,088 tokens of the roughly 1,850-token prefix. (Not all of it: automatic caches match in coarse block increments rather than byte-for-byte to the end; OpenAI, for instance, documents 128-token granularity.) So far, free money. But the mis-ordering hazard from the RAG table above applies with full force here, and with a nastier failure mode. We reran the same order experiment on the automatic path, new retrieved context on every call:

Order (no markers, automatic cache)	Call 1	Call 2 (new query, new context)
Context first, then rules	read 0	read 0
Rules first, context in the human turn	read 0	read 1,088

The wrong order is an unconditional zero: the changing context sits at the front, no two calls share a prefix, and the discount never arrives. On the explicit path the same mistake at least shows up in the bill as a cache-write premium on every call; on the implicit path there is no premium, no error, and no signal. The prompt just quietly never qualifies, while you assume "automatic" means "working." And since there is no marker to place, prompt order is the only knob the implicit path gives you.

So verify from the meters, in production rather than once in a test: input_token_details.cache_read (LangChain) or prompt_tokens_details.cached_tokens (raw). OpenAI's automatic caching additionally documents a 1,024-token minimum prefix, and per-provider TTL and eligibility differ, which is part 2's territory.

Checklist

On Claude, a ("system", "...") string tuple has nowhere to carry cache_control: nothing gets cached, nothing warns you. Cacheable system prompts go in a SystemMessage with content blocks and the marker.
The cache key is the byte-exact prefix: static content first, variables after the marker or in the human turn. RAG context before the rules does not just miss; it pays the write premium every call.
A variable inside the cached block creates one cache entry per value: repeating values amortize, per-call-unique values (timestamps, request IDs) never hit.
Tools sit before the system prompt in the prefix, so the system marker caches bound tools too (bind_tools serializes deterministically). If tools are your biggest stable block, the marker can go on an Anthropic-format tool dict instead.
In conversations, a marker fixed on the system block leaves the growing history at full price; put it on the latest message so each turn reads the prior prefix and writes only the delta.
Do not monitor input_token_details.cache_creation: it stays 0 even on writes, so a dashboard concludes caching is free while write premiums accrue. The real count is in ephemeral_5m_input_tokens, or read the raw response_metadata["usage"].
On automatic-cache models (GPT, GLM, DeepSeek), prompt order is the only knob and mis-ordering fails silently: no premium, no error, just a discount that never arrives. Verify hits from the usage fields.
set_llm_cache stores whole responses keyed on the exact prompt and model config; it only pays off when identical requests repeat, never on an agent loop.

The habits are small: a content block instead of a string, static before variable, a marker that slides with the conversation, one usage field read correctly. The measured difference was a 90% discount on every stable token versus nothing, and in the mis-ordered RAG case, versus paying extra. LangChain does not get in the way of prompt caching; it just makes the wrong prompt shape as easy to write as the right one.

Disclaimer

Measured on 2026-07-04 against https://synthorai.io/ with langchain-core 1.4.8, langchain-anthropic 1.4.8, langchain-openai 1.3.3, models claude-sonnet-5 and glm-5.2, a roughly 1,800-token English system prefix, small samples, and a 1–2 second gap between consecutive calls so cache writes have time to land. Each experiment used a fresh randomized prefix to guarantee a cold cache, which is why the baseline token counts differ slightly between tables (1,852 to 1,875). Library field mappings and provider cache behavior change between versions; re-measure against your own stack before depending on the numbers.

DEV Community