Suppose responses now have a token limit, the memory window is set, and the prompt begins with content a provider can cache — that was Part 2. Every control there worked on content that was already in the request: how long the answer may be, how much history travels with it, how the fixed part is arranged. You set a limit or an order, and that was it.
The next thing to look at is the context your application adds by itself at runtime. Document chunks arrive from a vector store, tool definitions arrive from every server you connect to. Both are billed as input tokens, both are sent whether the model uses them or not, and both grow with how much you have indexed or connected — not with what the current question actually needs.
This part is about controls for retrieved context and tool schemas.
- Driver #6 — RAG context stuffing: paying for noise
- Driver #7 — Tool schemas and agent loops: paying for a toolbox you rarely open
Prices: list prices where a ratio matters, an example rate of $1 per million input tokens elsewhere. Full note in Part 1.
Driver #6 — RAG context stuffing: paying for noise
Every retrieved document chunk becomes input tokens on every request, whether the model actually needs it or not. RAG bills you for relevance you never checked: retrieve 10 chunks of about 800 tokens each, and every question now carries 8,000 tokens of context. At 50,000 requests a month, that is 400 million tokens — $400 a month for retrieval alone at the example rate. And if only 3 of those 10 chunks are actually relevant, roughly $280 of that is spent on noise. Noise costs you twice: you pay for the tokens, and irrelevant context also lowers answer quality, which tends to lead to retries and follow-up questions.
The first control is retrieving less. Spring AI's VectorStoreDocumentRetriever offers two settings: topK sets a hard limit on how many chunks are retrieved, and similarityThreshold (0.0–1.0) filters out results whose similarity score falls below the specified threshold. By default, the retriever requests the top 4 results with a similarity threshold of 0.0, which disables similarity filtering and accepts all returned matches. As a result, the retriever accepts the vector store's nearest-neighbor results up to topK, which can include weakly related chunks when no truly relevant documents exist.
Advisor ragAdvisor = RetrievalAugmentationAdvisor.builder()
.documentRetriever(VectorStoreDocumentRetriever.builder()
.vectorStore(vectorStore)
.topK(3) // retrieve at most 3 documents
.similarityThreshold(0.65) // exclude documents below the similarity threshold
.build())
.build();
String answer = chatClient.prompt()
.advisors(ragAdvisor)
.user(question)
.call()
.content();
The second control is cleaning up what was retrieved. RetrievalAugmentationAdvisor is Spring AI's modular RAG pipeline, and its DocumentPostProcessor stage runs between document retrieval and prompt generation. This extension point lets you transform or filter the retrieved documents before they are added to the prompt. For example, you can implement custom post-processors to re-rank documents and keep only the most relevant ones, remove near-duplicate documents (common with overlapping chunking strategies), or compress each document to only the passages needed to answer the question. Each of these steps reduces the amount of context that reaches the model, lowering token usage while often improving answer quality.
One thing to watch for: some RAG pipeline steps spend tokens in order to save tokens later. LLM-based compression or query rewriting adds additional model calls and token usage for each request. That trade can still be worth it — for example, using a cheaper model to compress context before sending it to a more expensive flagship model — but the optimization is not free, so measure both sides (Driver #0).
A simple rule for tuning: if answers are missing relevant facts, try increasing topK or lowering the similarity threshold; if prompts are too large or expensive, try reducing topK or increasing the threshold. Evaluate changes against a fixed test set of questions rather than adjusting by intuition.
Driver #7 — Tool schemas and agent loops: paying for a toolbox you rarely open
Every tool made available to the model is included in the chat request as a tool definition containing its name, description, and JSON schema. Spring AI warns that large agent setups connected to services such as Slack, GitHub, Jira, and MCP servers can easily expose 50+ tools, consuming 55,000+ tokens before the first user message. It also notes that tool selection accuracy degrades when models face dozens of similarly named tools. At an example input price of $1 per million tokens, 55,000 schema tokens would cost about $0.055 per request, or $5,500 per 100,000 requests, before considering caching or tool-discovery approaches.
In Spring AI 2.0, ToolCallingAdvisor is the standard tool execution mechanism used by ChatClient; the tool-call loop was moved out of individual ChatModel implementations and into the advisor chain. (ToolCallAdvisor was renamed during the 2.0 development cycle and remains available as a deprecated compatibility class.) By default, ToolCallingAdvisor still sends all available tool definitions to the model. For large tool libraries, Spring AI 2.0 provides ToolSearchToolCallingAdvisor, which uses dynamic tool discovery: the model initially receives a search tool, then relevant tool definitions are added only when discovered. Spring's benchmark reports 34–64% token reduction across OpenAI, Anthropic, and Gemini models, although actual savings depend on the number of tools, schema size, model pricing, and workload.
spring.ai.chat.client.tool-search-advisor.enabled=true
spring.ai.chat.client.tool-search-advisor.tool-index-type=lucene
This replaces the default ToolCallingAdvisor behind the scenes. Three ToolIndex implementations are available: regex (the default — lightweight, no extra dependencies), lucene keyword search (included with the starter), and vector semantic search (needs a VectorStore bean).
The loop itself is another place where tokens can accumulate. ToolCallingAdvisor continues calling the model while responses contain tool calls, and every iteration is another model request carrying the conversation state accumulated so far. A model repeatedly failing the same tool call can therefore consume additional tokens. If you need an iteration limit as a hard safety boundary, as of today the only way is to build it yourself on the advisor's extension points — ToolCallingAdvisor is designed for subclassing, with protected hook methods (such as doBeforeCall) called on every iteration, and the accumulated history in each request already tells you how many tool rounds have run.
public class BoundedToolCallingAdvisor extends ToolCallingAdvisor {
private final int maxIterations;
// constructor via the self-referential Builder pattern omitted for brevity
@Override
protected ChatClientRequest doBeforeCall(ChatClientRequest request,
CallAdvisorChain chain) {
// method logic goes here
return request;
}
}
A custom ToolExecutionEligibilityChecker can control whether a returned tool call should be executed, but it only sees the latest response, so it cannot count rounds — it is not a replacement for a loop-iteration limit.
If the default tool-calling loop needs additional guardrails, there are several levels of control. You can disable automatic registration globally through configuration or for a single ChatClient call when you want to manage tool execution yourself. For reusable policies such as custom stopping rules, tool-specific restrictions, or additional observability, implement a custom ToolCallingAdvisor. If the application requires a completely different agent workflow, take control of the loop directly at the ChatModel level and manage tool execution explicitly.
MCP can make the schema problem larger: each connected server may contribute its available tool definitions, and a single session can end up exposing dozens of tools across multiple servers. Two controls help here: an McpToolFilter bean (covered in the Tool Calling reference) lets you control which MCP tools are exposed, for example by server or tool metadata, while the tool-search advisor can index MCP-provided tool callbacks and retrieve only relevant tool definitions when they are needed. Setting spring.ai.mcp.client.toolcallback.enabled=false opts out of exposing MCP tools entirely.
What's next
Drivers #6 and #7 solve the same problem, just in different parts of the prompt — they keep the model from receiving context this question does not need. The first does it by retrieving and keeping fewer document chunks, the second by sending tool definitions only when the model asks for them.
That holds as long as the request succeeds. When the model returns JSON your code cannot parse, the retry sends the whole request again — system prompt, history, retrieved chunks, tool schemas — and you pay for the same context a second time.
Driver #6 also left something out. Before a vector store can return anything, every document in your collection has to be turned into a vector, and that is a paid API call for each chunk. The retrieval side of RAG is what this part explained; the indexing side has its own bill, and it arrives again whenever you re-index.
Part 4 is coming. It covers those two, and closes the series with all ten drivers on a single map.
Top comments (0)