DEV Community: Julia Denysova

Spring AI Retries and Embeddings: Failed Answers and Full Re-indexes — LLM Cost Control 4/4

Julia Denysova — Tue, 25 Aug 2026 07:10:00 +0000

Across the first three parts of this series, the goal has stayed the same: send fewer tokens without losing what the model actually needs to answer well. Part 1 started with measurement and model choice, so you know what you are paying for before you optimise anything. Part 2 looked at the tokens you already send: how long the answer runs, how much history you carry on every turn, and how caching can make repeated content cheaper. Part 3 tuned retrieval and tool calling, so a request now carries only the context that question actually needs.

All of that assumes each request is sent only once. In practice, a failed answer usually means a retry, and a retry sends everything again — so one usable answer can cost you several requests.

This part is about controls for failed responses and for the indexing side of RAG. It also closes the series: all ten drivers are listed on one map at the end.

Driver #8 — Malformed output: paying full price for a failed answer
Driver #9 — Embeddings at scale: small numbers, big multipliers
The one-glance map: all ten drivers

Prices: list prices where a ratio matters, an example rate of $1 per million input tokens elsewhere. Full note in Part 1.

Driver #8 — Malformed output: paying full price for a failed answer

When the model returns JSON that your code cannot parse, the usual fix is to retry the call. Each retry resends the request context — system prompt, conversation history, RAG context, and tool schemas — so the input tokens are billed again and the failed answer is billed as output in full, cut off or not. For example, if 5% of requests require one retry, input token usage increases by roughly 5% without producing additional business value. A retry policy allowing three retries can make worst-case requests cost up to four times more.

Spring AI's .entity() convenience method on ChatClient's CallResponseSpec traditionally used a prompt-based structured output approach: the StructuredOutputConverter generated format instructions (including the JSON schema) and added them to the prompt. The model was then expected to return JSON matching that schema, which Spring AI parsed into the target Java type afterwards.

Spring AI 2.0 adds two per-call controls through EntityParamSpec: .validateSchema() and .useProviderStructuredOutput(). The first enables schema validation; the second switches from prompt-based instructions to the provider's native structured output mechanism, where supported.


Ticket ticket = chatClient.prompt()
    .user(text)
    .call()
    .entity(Ticket.class, spec -> spec
        .useProviderStructuredOutput()  // schema enforced by the provider
        .validateSchema());             // fallback: validate and retry

.useProviderStructuredOutput() sends the schema to the provider as an API-level rule instead of as prompt text. The provider runtime enforces schema conformance for supported models, so invalid structured responses are prevented during generation. The schema instructions also disappear from the prompt, reducing input token usage on every call. Spring AI 2.0 supports this for providers with native structured output capabilities, including OpenAI, Anthropic, Google GenAI, Mistral AI, and model-dependent Ollama support depending on the model. Check the limitations for your provider before relying on it: OpenAI, for example, rejects top-level array schemas, so requesting a List<T> fails, and Ollama models with reasoning mode may still return plain text.

.validateSchema() adds StructuredOutputValidationAdvisor: the response is checked against the schema, and if validation fails, the validation error is appended to the user message and the model is called again, up to 3 retry attempts by default. Each retry is another model call and consumes tokens, so use validation as a recovery mechanism rather than the primary strategy. Combine it with provider-native structured output when available to prevent failures before they happen.

Driver #9 — Embeddings at scale: small numbers, big multipliers

Embedding calls look cheap on a per-request basis, but a RAG pipeline runs them across your entire document collection, and may run them again whenever documents are added, updated, or a full re-index is required.

Three things drive that cost: the embedding model's price, the dimensionality of the vectors (which affects storage and retrieval costs), and how much text you actually need to embed during each indexing run. A fourth source of embedding calls has nothing to do with documents at all — that one comes at the end.

The embedding model

Model price comes first. According to OpenAI's pricing (August 2026), text-embedding-3-small costs $0.02 per million input tokens, compared to $0.13 per million for text-embedding-3-large — making the small model about 6.5× cheaper. Embedding a 50-million-token document collection therefore costs about $1 with the small model versus $6.50 with the large one each time you re-embed the entire corpus, such as during a full re-index. For many retrieval workloads, the smaller model delivers sufficient retrieval quality. Rather than assuming the larger model is necessary, evaluate both using the same fixed test set of questions introduced in Driver #6, Part 3.

Vector dimensionality

The second lever is vector dimensionality. EmbeddingOptions exposes a dimensions setting for providers that support it, such as OpenAI's text-embedding-3 models (including deployments on Azure). Reducing the number of dimensions does not lower the embedding API cost, which is based on input tokens, but it significantly reduces vector storage requirements. Assuming float32 vectors (4 bytes per dimension), one million vectors at 3,072 dimensions occupy about 12 GB of raw storage, while 512-dimensional vectors require about 2 GB. Smaller vectors also reduce index size and memory usage and often reduce similarity search latency because less data must be stored and processed.


spring.ai.openai.embedding.model=text-embedding-3-small
spring.ai.openai.embedding.dimensions=512

Set the number of dimensions before you start loading data. A vector index normally requires all vectors to have the same dimensionality, so changing this setting later usually requires creating a new index and re-embedding the existing documents.

Batching and incremental indexing

Third: batch requests, and skip what has not changed. When you add documents through a VectorStore, Spring AI uses the default TokenCountBatchingStrategy, which groups documents into batches based on token count. It keeps batches below the embedding model's token limit (OpenAI's 8,191-token limit with a default 10% safety margin) instead of sending each chunk as a separate embedding request. You can replace this with your own BatchingStrategy bean if needed.

What the framework cannot know automatically is which documents actually changed. Store a content hash in each Document's metadata and re-embed only chunks whose hash differs. If only 2% of chunks require updating each week, embedding work can be roughly 50× smaller than a full re-index.

Chat memory is an embedding cost too

Everything above assumes you are embedding documents. There is a second source, and it is easy to miss because it does not look like indexing at all.

Part 2, Driver #4 offered VectorStoreChatMemoryAdvisor as an option for very long conversations. Instead of resending the whole chat on every turn, it saves the messages in a vector store and adds back only the ones that match the current question. Each request then stays the same size, however long the session runs. That is a real saving on the chat side.

The cost does not disappear, though. It moves. Every message the advisor saves has to be turned into a vector first, and the search itself embeds the question too — so each turn brings its own embedding cost. A document collection is embedded once and updated from time to time; a busy support chat produces new messages all day, so here the number of embedding calls follows your traffic, not the size of your collection. The controls are the same — a smaller model, fewer dimensions — but count the calls before you choose this advisor over a message window.

Wrap-up: the one-glance map

Every driver above follows the same basic pattern: tokens repeat or grow somewhere, and Spring AI gives you a control to cut them. Here is the full map:

#	Cost driver	Symptom on the bill	Spring AI control
0	No measurement	Unexpected costs at the end of the month	Micrometer token metrics; operational alerts; provider spending limits
1	Model choice	Flagship prices for routine tasks	configurable model selection; one `ChatClient` per cost tier
2	One shared client	Requests include defaults they do not need	Per-task clients from the prototype `ChatClient.Builder`
3	Output & reasoning tokens	Unnecessarily long responses or expensive reasoning	`maxTokens` ceiling; thinking/reasoning-effort options (where supported)
4	Conversation history	Cost per request grows a lot as conversations get longer	`MessageWindowChatMemory` with a fixed window size
5	Repeated static content	The same prompt content is sent again and again	Static-first prompt structure; provider prompt caching (where supported); `promptCacheKey` on OpenAI
6	RAG stuffing	Too much retrieved content increases input tokens	`topK`, similarity threshold, rerank/compress post-processors
7	Tool schemas & loops	Tool definitions are sent unnecessarily; loops run too long	`ToolSearchToolCallingAdvisor`; a custom ToolCallingAdvisor subclass for loop limits; McpToolFilter
8	Malformed output	Failed responses require retries	Native structured output (`ChatClient.EntityParamSpec.useProviderStructuredOutput()`) with schema enforcement
9	Embeddings at scale	Indexing and vector storage costs keep growing	Smaller embedding model; dimensions; batching; application-side incremental indexing

Where to start

Ten drivers is a list, not a plan. Working through them in order is the slowest way to use this map, because the drivers are not equal — in most applications two or three of them carry almost the whole bill, and which ones depends on what you built.

So start at Driver #0. Set up your metrics, watch prompt and completion tokens, and let the graph choose your next move.

One last thing worth saying: every control in this series trades something away. A smaller window forgets more. A tighter topK misses more facts. A lower reasoning effort reasons less. None of these is free, and a cheap application that gives poor answers is not cheap — users simply ask again. The main point here is knowing what each token buys you, and paying only for the ones that buy something.

That is the series. If you have hit a cost driver that is not on the map, or your numbers came out differently from the estimates here, the comments are the right place for it — several ideas from readers of Part 1 ended up shaping the later parts.

Spring AI RAG and Tool Calling: Paying for Context You Don't Use — LLM Cost Control 3/4

Julia Denysova — Wed, 12 Aug 2026 07:00:00 +0000

Suppose responses now have a token limit, the memory window is set, and the prompt begins with content a provider can cache — that was Part 2. Every control there worked on content that was already in the request: how long the answer may be, how much history travels with it, how the fixed part is arranged. You set a limit or an order, and that was it.

The next thing to look at is the context your application adds by itself at runtime. Document chunks arrive from a vector store, tool definitions arrive from every server you connect to. Both are billed as input tokens, both are sent whether the model uses them or not, and both grow with how much you have indexed or connected — not with what the current question actually needs.

This part is about controls for retrieved context and tool schemas.

Driver #6 — RAG context stuffing: paying for noise
Driver #7 — Tool schemas and agent loops: paying for a toolbox you rarely open

Prices: list prices where a ratio matters, an example rate of $1 per million input tokens elsewhere. Full note in Part 1.

Driver #6 — RAG context stuffing: paying for noise

Every retrieved document chunk becomes input tokens on every request, whether the model actually needs it or not. RAG bills you for relevance you never checked: retrieve 10 chunks of about 800 tokens each, and every question now carries 8,000 tokens of context. At 50,000 requests a month, that is 400 million tokens — $400 a month for retrieval alone at the example rate. And if only 3 of those 10 chunks are actually relevant, roughly $280 of that is spent on noise. Noise costs you twice: you pay for the tokens, and irrelevant context also lowers answer quality, which tends to lead to retries and follow-up questions.

The first control is retrieving less. Spring AI's VectorStoreDocumentRetriever offers two settings: topK sets a hard limit on how many chunks are retrieved, and similarityThreshold (0.0–1.0) filters out results whose similarity score falls below the specified threshold. By default, the retriever requests the top 4 results with a similarity threshold of 0.0, which disables similarity filtering and accepts all returned matches. As a result, the retriever accepts the vector store's nearest-neighbor results up to topK, which can include weakly related chunks when no truly relevant documents exist.


Advisor ragAdvisor = RetrievalAugmentationAdvisor.builder()
    .documentRetriever(VectorStoreDocumentRetriever.builder()
        .vectorStore(vectorStore)
        .topK(3)                    // retrieve at most 3 documents
        .similarityThreshold(0.65)  // exclude documents below the similarity threshold
        .build())
    .build();

String answer = chatClient.prompt()
    .advisors(ragAdvisor)
    .user(question)
    .call()
    .content();

The second control is cleaning up what was retrieved. RetrievalAugmentationAdvisor is Spring AI's modular RAG pipeline, and its DocumentPostProcessor stage runs between document retrieval and prompt generation. This extension point lets you transform or filter the retrieved documents before they are added to the prompt. For example, you can implement custom post-processors to re-rank documents and keep only the most relevant ones, remove near-duplicate documents (common with overlapping chunking strategies), or compress each document to only the passages needed to answer the question. Each of these steps reduces the amount of context that reaches the model, lowering token usage while often improving answer quality.

One thing to watch for: some RAG pipeline steps spend tokens in order to save tokens later. LLM-based compression or query rewriting adds additional model calls and token usage for each request. That trade can still be worth it — for example, using a cheaper model to compress context before sending it to a more expensive flagship model — but the optimization is not free, so measure both sides (Driver #0).

A simple rule for tuning: if answers are missing relevant facts, try increasing topK or lowering the similarity threshold; if prompts are too large or expensive, try reducing topK or increasing the threshold. Evaluate changes against a fixed test set of questions rather than adjusting by intuition.

Driver #7 — Tool schemas and agent loops: paying for a toolbox you rarely open

Every tool made available to the model is included in the chat request as a tool definition containing its name, description, and JSON schema. Spring AI warns that large agent setups connected to services such as Slack, GitHub, Jira, and MCP servers can easily expose 50+ tools, consuming 55,000+ tokens before the first user message. It also notes that tool selection accuracy degrades when models face dozens of similarly named tools. At an example input price of $1 per million tokens, 55,000 schema tokens would cost about $0.055 per request, or $5,500 per 100,000 requests, before considering caching or tool-discovery approaches.

In Spring AI 2.0, ToolCallingAdvisor is the standard tool execution mechanism used by ChatClient; the tool-call loop was moved out of individual ChatModel implementations and into the advisor chain. (ToolCallAdvisor was renamed during the 2.0 development cycle and remains available as a deprecated compatibility class.) By default, ToolCallingAdvisor still sends all available tool definitions to the model. For large tool libraries, Spring AI 2.0 provides ToolSearchToolCallingAdvisor, which uses dynamic tool discovery: the model initially receives a search tool, then relevant tool definitions are added only when discovered. Spring's benchmark reports 34–64% token reduction across OpenAI, Anthropic, and Gemini models, although actual savings depend on the number of tools, schema size, model pricing, and workload.


spring.ai.chat.client.tool-search-advisor.enabled=true
spring.ai.chat.client.tool-search-advisor.tool-index-type=lucene

This replaces the default ToolCallingAdvisor behind the scenes. Three ToolIndex implementations are available: regex (the default — lightweight, no extra dependencies), lucene keyword search (included with the starter), and vector semantic search (needs a VectorStore bean).

The loop itself is another place where tokens can accumulate. ToolCallingAdvisor continues calling the model while responses contain tool calls, and every iteration is another model request carrying the conversation state accumulated so far. A model repeatedly failing the same tool call can therefore consume additional tokens. If you need an iteration limit as a hard safety boundary, as of today the only way is to build it yourself on the advisor's extension points — ToolCallingAdvisor is designed for subclassing, with protected hook methods (such as doBeforeCall) called on every iteration, and the accumulated history in each request already tells you how many tool rounds have run.


public class BoundedToolCallingAdvisor extends ToolCallingAdvisor {

    private final int maxIterations;

    // constructor via the self-referential Builder pattern omitted for brevity

    @Override
    protected ChatClientRequest doBeforeCall(ChatClientRequest request,
                                             CallAdvisorChain chain) {
        // method logic goes here    
        return request;
    }
}

A custom ToolExecutionEligibilityChecker can control whether a returned tool call should be executed, but it only sees the latest response, so it cannot count rounds — it is not a replacement for a loop-iteration limit.

If the default tool-calling loop needs additional guardrails, there are several levels of control. You can disable automatic registration globally through configuration or for a single ChatClient call when you want to manage tool execution yourself. For reusable policies such as custom stopping rules, tool-specific restrictions, or additional observability, implement a custom ToolCallingAdvisor. If the application requires a completely different agent workflow, take control of the loop directly at the ChatModel level and manage tool execution explicitly.

MCP can make the schema problem larger: each connected server may contribute its available tool definitions, and a single session can end up exposing dozens of tools across multiple servers. Two controls help here: an McpToolFilter bean (covered in the Tool Calling reference) lets you control which MCP tools are exposed, for example by server or tool metadata, while the tool-search advisor can index MCP-provided tool callbacks and retrieve only relevant tool definitions when they are needed. Setting spring.ai.mcp.client.toolcallback.enabled=false opts out of exposing MCP tools entirely.

What's next

Drivers #6 and #7 solve the same problem, just in different parts of the prompt — they keep the model from receiving context this question does not need. The first does it by retrieving and keeping fewer document chunks, the second by sending tool definitions only when the model asks for them.

That holds as long as the request succeeds. When the model returns JSON your code cannot parse, the retry sends the whole request again — system prompt, history, retrieved chunks, tool schemas — and you pay for the same context a second time.

Driver #6 also left something out. Before a vector store can return anything, every document in your collection has to be turned into a vector, and that is a paid API call for each chunk. The retrieval side of RAG is what this part explained; the indexing side has its own bill, and it arrives again whenever you re-index.

Part 4 covers those two, and closes the series with all ten drivers on a single map.

Spring AI Prompt Caching and Chat Memory: Where the Tokens Go — LLM Cost Control 2/4

Julia Denysova — Tue, 04 Aug 2026 07:04:09 +0000

Suppose the metrics are in place, each task has the model it actually needs, and every feature has its own client — that was Part 1. The next thing to look at is what those clients send and receive.

This part is about controls for output token generation, chat memory and static input tokens.

Driver #3 — Output and reasoning tokens: the expensive direction
Driver #4 — Conversation history: you pay for the whole chat, every turn
Driver #5 — Repeated static content: paying full price for the same tokens

Prices: list prices where a ratio matters, an example rate of $1 per million
input tokens elsewhere. Full note in Part 1.

Driver #3 — Output and reasoning tokens: the expensive direction

Every token the model generates costs several times more than a token you send it. Reasoning models also generate hidden "thinking" tokens, and these are billed at the same, higher output rate.

The gap is on every price sheet. On OpenAI's list (August 2026), gpt-5 input costs $1.25 per million tokens and output costs $10 — an 8× difference. On Anthropic's price sheet, Sonnet 5 is $2 in and $10 out — a 5× difference. (That is introductory pricing to 31 August 2026; the standard $3/$15 keeps the same 5× ratio.)

The exact numbers change often, but the pattern does not: output tokens have cost several times more than input tokens for years. Reasoning makes this worse. A model may use 2,000 thinking tokens to produce a 200-token answer, so you pay for 2,200 output tokens in total — eleven times the text the user actually sees. A verbose model on a busy endpoint can end up costing more than all your input traffic combined.

Spring AI gives you two controls here. The first works everywhere: ChatOptions.builder().maxTokens() provides a portable completion-length limit, when supported by the provider. Think of it as a safety net against runaway responses, not a tool for improving quality — a cut-off answer is still billed in full, so pair the limit with prompt instructions that ask for a short answer. The second control is provider-specific and kept isolated in that provider's options object: OpenAI has a reasoning-effort setting, Anthropic has a thinking-token budget.


// Provider-independent token limit 
ChatOptions.Builder capped = ChatOptions.builder()
    .maxTokens(400);

// Provider-specific reasoning control, isolated in one options object
OpenAiChatOptions.Builder lowEffort = OpenAiChatOptions.builder()
    .model("gpt-5-mini")
    .reasoningEffort("low")
    .maxCompletionTokens(400);

// Ollama: disable reasoning for thinking-capable models
OllamaOptions.Builder noThink = OllamaOptions.builder()
    .model("qwen3")
    .think(false);

One 2.0 upgrade note belongs here: the Anthropic module's maxTokens default rose from 500 to 4096. If you relied — even without noticing it — on the old 500-token cap to limit the cost of responses, they can now run up to eight times longer after the upgrade. Set the limit explicitly if you want to keep the old behaviour.

One trap deserves its own warning. Several popular Ollama models — including qwen3 and deepseek-r1 — use reasoning by default. During local development there is no per-token API bill, so it is easy to miss that a prompt pattern has become dependent on lengthy reasoning. Move the same prompts to a provider that bills reasoning tokens, and that hidden reasoning can become part of your completion cost. For simple workloads such as extraction, classification, or reformatting, disable or limit thinking during local development as well. This keeps token usage, latency, and production costs easier to predict.

Driver #4 — Conversation history: you pay for the whole chat, every turn

LLMs have no built-in memory, so "memory" in practice means sending the full conversation history with every request. Every past message is billed again, as if it were new input.

The cost grows faster than you might expect. Assume a 500-token system prompt, roughly 50-token user messages, and roughly 200-token replies. Each finished turn adds about 250 tokens of history, and every later turn has to carry that history too. A message window bounds how much of that history each request includes. Two details must be noted here: the window counts the stored messages, so ten messages means the last five complete user–assistant exchanges, and the current user message always travels outside the window. From turn 6 onwards, every request is therefore the same size: 500 + 1,250 + 50 = 1,800 input tokens.

Turn	Input tokens, unbounded history	Input tokens, window = 10 messages
1	550	550
5	1,550	1,550
6	1,800	1,800
20	5,300	1,800
50	12,800	1,800
Whole 50-turn session	≈ 333,750	≈ 86,250

Without a limit, the cost of each turn grows in a straight line, but the cost of the whole session grows much faster than that: a 20-turn session sends about 58,500 input tokens in total, and a 50-turn support chat about 333,750 — for a single user. The same 50-turn chat with a 10-message window sends about 86,250, roughly a quarter of the unbounded total.

One Spring AI 2.0 detail is worth knowing here: eviction now removes whole turns. A turn starts at a user message, so the kept window always begins with one, and maxMessages is an upper bound rather than a guarantee. For plain chat, choose an even window size so it maps cleanly onto complete exchanges.

Spring AI's control here is MessageWindowChatMemory, a sliding window of at most N messages, added through a memory advisor:


@Bean
ChatClient chatClient(ChatClient.Builder builder,
                      ChatMemoryRepository repository) {

    ChatMemory memory = MessageWindowChatMemory.builder()
        .chatMemoryRepository(repository)
        .maxMessages(10)   // overrides the default 20-message window
        .build();
    return builder
        .defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())
        .build();
}

The service that injects this client selects the conversation on each call:


chatClient.prompt()
    .user(message)
    // mandatory in Spring AI 2.0 — there is no default conversation id
    .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, sessionId))
    .call()
    .content();

ChatMemory and ChatMemoryRepository are documented together in Spring AI's Chat Memory reference. Spring AI sets this up automatically (an in-memory repository and a 20-message window). Treat maxMessages as a cost-control decision, not just a convenience setting: a 20-message window from short Q&A may be cheap, while the same window from long conversations can add thousands of tokens to every request. MessageWindowChatMemory limits the number of messages, not tokens, so choose the window size based on the typical token size of your conversations and the context you actually need.

Two things to keep in mind. First, a smaller window trades cost for what the model remembers. The model genuinely forgets removed turns, so choose the smallest window your use case can accept, not simply the smallest window possible. Second, the window affects Driver #5 too: a stable start to the prompt (system prompt, then the oldest history) is what makes provider-side caching work well. Removing messages aggressively, in a way that changes the start of the prompt on every turn, can cost you the cache discount.

For very long sessions, VectorStoreChatMemoryAdvisor is an alternative. It stores history in a vector store and adds back only the messages relevant to the current question, so the input size per turn stays flat no matter how long the session runs. The trade-off: every message has to be turned into an embedding and stored (Part 4), and the history added back changes with each question, which works against caching. Caching is next.

Driver #5 — Repeated static content: paying full price for the same tokens

The system prompt, tool definitions (including schemas), and few-shot examples are often the same across requests, and they are part of the model input context. Without caching, providers generally process and charge these repeated input tokens on every request. Prompt caching reduces this cost: supported providers store already-processed prompt prefixes and apply a lower input-token rate when the same content is reused. Every provider implements caching differently, with its own rules for eligibility, expiration, and configuration.

Anthropic and AWS Bedrock: you choose what to cache

Spring AI exposes named caching strategies — SYSTEM_ONLY, TOOLS_ONLY, SYSTEM_AND_TOOLS, and CONVERSATION_HISTORY — through provider-specific enums for Anthropic and AWS Bedrock. These strategies define where Spring AI places cache breakpoints while respecting provider limitations, but cache lifetime and expiration remain managed by the underlying provider:


AnthropicChatOptions.builder()
    .cacheOptions(AnthropicCacheOptions.builder()
        .strategy(AnthropicCacheStrategy.SYSTEM_AND_TOOLS)
        .build())

Anthropic prompt caching charges a higher price for cache writes and a lower price for cache reads, which makes the feature valuable for workloads with repeated, stable prompts. For example, at $1 per million input tokens, a 2,000-token system prompt sent 300,000 times per month costs $600 without caching; with a high cache hit rate and 0.1× cached reads, the cost can drop to roughly $60. The main things to watch are cache lifetime and traffic patterns: caches expire, and low-traffic endpoints may pay cache-write costs without enough cache hits to recover the overhead.

For agent-style workloads, there is one more Anthropic-specific setting worth knowing: cacheToolResults. With CONVERSATION_HISTORY caching, each tool-calling round adds tool results after the default cache point, so those tool outputs are billed as new, uncached input on later rounds. Enabling cacheToolResults moves the cache point to the last tool result, allowing the next round to read the previous round's (often large) tool output from the cache instead of processing it again. This pairs directly with tool schemas and agent loops (Part 3).

OpenAI: automatic, but no longer free

Prompts of at least 1,024 tokens are cached automatically, with no code changes. Two things changed with the GPT-5.6 family, and both affect the bill.

First, cache writes are billed at 1.25× the uncached input rate. On earlier models they were free.

Second, the service caches exact prefixes at breakpoints. By default it places one implicit breakpoint at the latest user or tool message, and it no longer falls back to the longest matching prefix before that point. A request can therefore share thousands of identical tokens with the previous one, report zero cached tokens, and pay to write the changing prefix again.

The lever Spring AI gives you is the cache key. Requests that share a prefix should carry the same one:


OpenAiChatOptions.Builder shared = OpenAiChatOptions.builder()
    .model("gpt-5.6-terra")
    .promptCacheKey("support-assistant-v1");

or spring.ai.openai.chat.prompt-cache-key=support-assistant-v1. On GPT-5.6 and later, this key is required for the more reliable matching. Keep traffic per key to roughly 15 requests per minute and partition busier workloads across several keys.

Spring AI 2.0 does not expose explicit cache breakpoints, so on OpenAI the cacheable prefix is whatever your prompt structure gives you — which makes the static-first rule below a cost control, not a style preference. It also reports cache reads but not cache writes for OpenAI, so the 1.25× write charge is invisible in your token metrics. Watch it on the provider's dashboard instead.

Local models (Ollama): free, but still worth planning

Local models also use a cache, but it saves time, not money. Before a model can answer, it must first process every token of the prompt — an expensive "reading" step on the GPU. The engine keeps the result of this step in memory (the KV cache). If the next request begins with exactly the same text, the engine skips that part and processes only what is new. The result: a faster first token and more requests per GPU. But nothing on a bill gets smaller, because locally there are no per-token charges. The cache also lives only while the model is loaded — Ollama unloads idle models after five minutes, though keep_alive can hold them in memory for longer.

Note that the cache only matches from the start of the prompt: at the first token that differs, everything after it is processed again. So the static-first rule applies here too.

The general rule: static content first, changing content last

Provider prompt caching mechanisms depend on matching the beginning of the prompt. A cache hit requires an identical prefix, so fixed content — instructions, examples, and tool definitions — should be placed before changing content. A single timestamp placed at the top of the system prompt can invalidate the cached prefix that follows it. Spring AI 2.0 supports separating static and dynamic SystemMessage blocks for providers that support multi-block system caching, such as Anthropic and AWS Bedrock Converse. With .multiBlockSystemCaching(true), Spring AI can preserve cacheable system blocks while allowing later changing system content to remain outside the cached prefix.

Structure your prompts this way even before you turn caching on — it is what makes every provider's cache, automatic or explicit, actually work for you.

What's next

There is a limit to how far prompt structure can take you, though. Everything in this part assumed the content was yours: your system prompt, your conversation, your examples. The next part deals with the context your application adds automatically — document chunks from a vector store and tool definitions from every server you connect to. Both arrive as input tokens, both are sent whether the model uses them or not, and both scale with how much you have indexed rather than with how much you need.

Spring AI Token Usage: Measure Cost Before You Pick a Model — LLM Cost Control 1/4

Julia Denysova — Thu, 30 Jul 2026 15:26:31 +0000

Cutting LLM costs in Spring AI starts with two choices: which model answers a request, and what defaults your ChatClient adds to every one it sends.

Neither is worth changing until you can see where the tokens go. That is why this article starts with measurement.

This is Part 1 of four, and it covers the first three of ten cost drivers. Driver #0 tells you where the money actually goes; #1 and #2 are the two decisions that shape every request your application sends. The remaining seven attach to what you build here.

A note on the numbers: where a price ratio matters for the argument (input vs. output, cache read vs. write), this series quotes real August 2026 list prices with a link. All other examples use a flat rate of $1 per million input tokens, so you can redo the calculation with your own provider's price sheet. You should do that, because these prices change every few months.

Driver #0 — Spring AI observability measurement: you cannot cut what you cannot see

Provider invoices and usage dashboards usually show your spending by model and by token type — input, output, and cached. That is useful, but it is not enough. The numbers cannot tell you which feature, client, or advisor inside your application was responsible for that usage.

Spring AI integrates with Spring Boot's Micrometer-based observability to fill this gap. Its core AI components automatically emit that data. ChatModel, EmbeddingModel, and ImageModel implementations (support varies by provider) publish model-level observations, including token usage where available. ChatClient (including advisors) and VectorStore report execution observations and traces rather than token usage metrics.

Each metric includes built-in tags, such as the model name and token type. These tags separate models and providers, but not callers: every request to the same model carries the same tag values, so they cannot tell two features apart on their own. Spring AI marks tags as low- or high-cardinality: low-cardinality tags go on both metrics and traces, high-cardinality tags go on traces only. To attribute usage to a specific feature or client, add your own low-cardinality tags. Keep those values few and stable, not one per user or request. For more on how tags work, and how to keep their number under control, see Micrometer's documentation on tags and naming.

With this data in place, you can answer the questions the later drivers raise. Is history growth (Part 2) really increasing costs? Watch prompt token usage over time. Did the tool-search advisor (Part 3) reduce the amount of context sent to the model? Compare prompt token usage before and after enabling it. Is a reasoning model becoming more expensive than expected (Part 2)? Output token trends can reveal unexpectedly long responses and rising generation costs.

Two more steps make this setup safer, not just more informative. First, create an alert for sudden increases in token usage or request volume. A bug that puts an LLM call inside a loop can burn through your budget fast, and an alert catches it within minutes instead of at the end of the month. Second, configure provider-side spending controls where available, such as budgets, quotas, or usage limits. This is a final safeguard that no application framework can enforce.

Treat observability as step zero. Set it up before optimising anything else, so every later change has a measurable before-and-after comparison.

Driver #1 — Model choice: stop paying flagship prices for mini-model work

The fastest way to overspend on an LLM application is to send every request to the most capable model you can find. Flagship and budget models on the same provider's price sheet can differ tenfold in price per token, or more. But a large share of typical workloads — classification, extraction, routing, short summaries — passes the quality bar on the smaller model already. Sending those requests to a flagship model increases cost without improving the outcome.

Price is only one factor, though. Models also differ in what they can do: some accept images or audio, some are built for tool calling or strict structured output, some reason step by step while others just answer directly. The rule is the same either way: pick the smallest model that covers what the task actually needs. Do not pay for image support on a text-only pipeline, and do not pay for a reasoning model just to reformat JSON. Requirements change over time, so treat your model choice as temporary, not final. Providers release cheaper and better models every few months, and a task that needed the flagship model yesterday often works fine on the mini model next quarter.

This only pays off if switching models is easy. Spring AI's answer is that the model is configuration, not architecture. ChatClient does not depend on any one provider or model: your service code does not need to know which model answers, portable options like maxTokens and temperature carry over between models, and provider-specific settings stay isolated in one options object. Spring AI 2.0 makes this explicit: default options set on a ChatClient act as a partial "delta" over the model's start-up configuration. The framework signals this same idea in its own defaults: the OpenAI integration in 2.0 defaults to gpt-5-mini, not the flagship model. A good pattern is one client per cost tier:

@Configuration
public class ChatClients {

    @Bean
    @Primary
    ChatClient cheapClient(ChatClient.Builder builder) {
        return builder
              .defaultOptions(ChatOptions.builder()
                  .model("gpt-5-mini"))
              .build();
    }

    @Bean
    ChatClient flagshipClient(ChatClient.Builder builder) {
        return builder
              .defaultOptions(ChatOptions.builder()
                  .model("gpt-5"))
              .build();
    }
}

A ticket classifier feature could inject @Qualifier("cheapClient") ChatClient, while a contract-analysis feature injects the flagship one. Two details matter here. First, build clients from the auto-configured builder, not from a raw ChatClient.builder(chatModel). The auto-configured builder already has observability wired in, and the token metrics from Driver #0 depend on that. Second, this example covers two tiers on one provider. If you mix vendors in the same application, create a separate ChatClient from each auto-configured ChatModel instead. Either way, the size of the change stays small: swapping models is a property change, and swapping vendors just adds a starter dependency plus properties. No code changes, no prompt rewrites, no new SDK to learn.

Driver #2 — One shared client: every request carries every default

A ChatClient default is not free. Everything attached to the client — system prompt, tools, memory advisor — is included in every request that client sends. One shared "do-everything" client means a simple 10-token call (e.g. a classifier feature) carries the same 2,000-token system prompt, tool schemas, and conversation history as your most complex feature. If the classifier runs 50,000 times a month, that is 100 million input tokens of extra content it never actually needs — $100 a month at the example rate, for nothing.

Spring AI's unit of cost control is the client, not the whole application. ChatClient.Builder is auto-configured as a prototype bean, so each task can build its own client with only the defaults that task needs: .defaultSystem() for the prompt, .defaultOptions() for model and token settings, .defaultAdvisors() and .defaultTools() for the rest. Every later driver in this series — memory windows, RAG advisors, tool search — is attached at exactly this level. This is why building per-task clients should come before the drivers that follow.

@Configuration
class TaskClients {

    @Bean
    ChatClient classifierClient(ChatClient.Builder builder) {
        return builder
                .defaultSystem("Classify the ticket. Reply with one word: BILLING, TECH, or OTHER.")
                .defaultOptions(ChatOptions.builder()
                        .model("gpt-5-mini")
                        .maxTokens(5)
                        .temperature(0.0))
                .build();
    }

    @Bean
    ChatClient supportChatClient(ChatClient.Builder builder, ChatMemory memory) {
        return builder
                .defaultSystem(new ClassPathResource("prompts/support.st"))
                .defaultOptions(ChatOptions.builder()
                        .model("gpt-5")
                        .maxTokens(800))
                .defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())
                .build();
    }
}

Note the Spring AI 2.0 change: .defaultOptions() and the per-call .options() now take a ChatOptions.Builder, not a fully built instance. This builder is merged with the model's defaults before the first advisor runs, and only the fields you explicitly set override them. That merge lets you override only the options you need without replacing the model's configured defaults:

classifierClient.prompt(text)
    .options(ChatOptions.builder().maxTokens(1)) // override for a yes/no case
    .call()
    .content();

The result: each request pays only for the defaults its own task actually declared. That is a requirement for every control that follows.

What's next

Drivers #1 and #2 decide which model answers a request and which client sends it; Driver #0 shows you what those decisions cost. You now have one client per task, each carrying only the defaults that task needs, and the token metrics to prove it.

None of that limits the size of a single request. A reasoning model can bill more hidden thinking than visible answer. A support conversation resends its whole history on every turn. The same system prompt goes out thousands of times a day at full price, unless you structure it so the provider can cache it.

Part 2 covers those three. Each one is a limit or an advisor you attach to the clients you have just built.

How to Reduce LLM Costs in Spring AI 2.0: 10 Practical Controls

Julia Denysova — Thu, 30 Jul 2026 15:26:02 +0000

Spring AI's defaults are built for a fast start; they do not guarantee a low monthly cost. Shipping an LLM feature is easy — making it cost-efficient is not. This series shows the spots where money leaks, along with the control that closes each one.

Spring AI 2.0 reached GA on 12 June 2026. It needs Spring Boot 4, moves the tool-calling loop out of the ChatModel, adds tool search, and extends structured outputs. Tool search and the extended structured-output controls point in the same direction: they determine how many tokens your application sends and receives.

The bill grows quietly. A chatbot with a 2,000-token system prompt, run 100,000 times a month, sends 200 million tokens of the same text. At an example rate of $1 per million input tokens, that is $200 a month — before a single user message. Then add conversation history, which is sent in full on every turn. Add retrieved RAG documents and the JSON schema of every registered tool. The input side can grow 10× with no change in traffic at all. Output tokens cost several times more per token than input tokens, and reasoning models bill their hidden "thinking" as output too.

The provider sets the prices. The framework gives you controls that can reduce the number of tokens you pay for.

This series works through ten cost drivers, numbered #0 to #9. Each one is a place where tokens repeat or grow without anyone deciding they should, and each comes with the Spring AI control that cuts it. They are spread across four parts.

Part 1 — Token Usage: Measure Cost Before You Pick a Model (Drivers #0–#2)

Provider dashboards show what you spent, but not which feature spent it. Spring AI's observability closes that gap, and from there you can match each model to its task and stop features from carrying defaults they never needed.

Part 2 — Prompt Caching and Chat Memory: Where the Tokens Go (Drivers #3–#5)

This part covers limiting response length, bounding how much conversation history you resend, and structuring prompts so that provider caching actually works.

Part 3 — RAG and Tool Calling: Paying for Context You Don't Use (Drivers #6–#7)

Retrieved documents and tool definitions are added to every request, whether the model needs them or not. This part covers retrieving less, cleaning up what was retrieved, and sending tool schemas only when they are relevant.

Part 4 — Retries and Embeddings: Failed Answers and Full Re-indexes (Drivers #8–#9)

A failed response is billed in full, and retrying it resends the entire context. This part covers preventing malformed output at the provider level, then the indexing side: embedding model choice, vector size, batching, and re-embedding only what has changed.

Start with Part 1. Every later driver asks you to trade something — context, memory, response length — and per-feature metrics are what tell you which trade is worth making. Part 1 also covers the cheapest change available: sending routine work to a smaller model, which usually needs no more than a property change.