Model API costs are invisible until they're not. During development you make a few hundred calls and barely notice the bill. In production with real users, the same agent can cost ten times what you expected — not because something is wrong, but because small inefficiencies multiply at scale.
The good news: most of the cost in a typical AI agent comes from a handful of sources, and most of those sources have straightforward fixes that don't affect output quality at all.
This article covers where token costs actually come from and how to reduce them without making your agent worse.
Where the money goes
Before optimising anything, measure where tokens are actually being spent. Log usage on every request:
ChatResponse chatResponse = chatClient.prompt()
.user(request.message())
.call()
.chatResponse();
Usage usage = chatResponse.getMetadata().getUsage();
log.info("[cost] conversationId={} inputTokens={} outputTokens={} totalTokens={}",
request.conversationId(),
usage.getPromptTokens(),
usage.getGenerationTokens(),
usage.getTotalTokens());
After a few days of production traffic, query the averages:
SELECT
AVG(input_tokens) AS avg_input,
AVG(output_tokens) AS avg_output,
AVG(total_tokens) AS avg_total,
MAX(total_tokens) AS max_total
FROM request_usage
WHERE created_at > NOW() - INTERVAL '7 days';
This tells you where to focus. In most agents the breakdown looks like:
| Source | Typical share of input tokens |
|---|---|
| System prompt | 15–30% |
| Tool definitions | 10–25% |
| Conversation history | 30–50% |
| User message | 5–15% |
Output tokens are usually 10–20% of total. Input tokens are where most of the cost hides.
Trim the system prompt
Every word in your system prompt fires on every single request. A 500-token system prompt on 10,000 daily requests is 5 million tokens just for instructions.
Common bloat to cut:
// Before — verbose, repetitive
"""
You are a helpful AI assistant specialising in travel planning for India.
Your job is to help users plan their trips by providing information about
weather, attractions, and transportation options. You should always be
polite and professional. When users ask about travel, use the available
tools to get real-time information. Always provide accurate information
based on the tool results and never make up information that isn't
provided by the tools.
"""
// After — same behaviour, ~40% fewer tokens
"""
You are a travel planning assistant for India.
Use tools for weather, attractions, and transport.
Only state facts returned by tools.
"""
Test that the shortened prompt produces equivalent output quality using your eval dataset before shipping it. Prompt compression is one of the highest-leverage cost reductions — it compounds across every request.
Bound the conversation history window
Conversation history is the biggest token cost driver for multi-turn agents. Each request includes all prior messages up to the window size.
A conversation that's been going for 20 turns might look like:
Turn 1: user (15 tokens) + assistant (80 tokens)
Turn 2: user (20 tokens) + assistant (120 tokens) + 2 tool calls (60 tokens)
...
Turn 20: all of the above replayed = ~2,000 tokens of history
+ new user message (25 tokens)
+ system prompt (80 tokens)
+ tool definitions (200 tokens)
= ~2,300 tokens input just for context
The window size directly controls this cost. Keep it as small as correctness allows:
new MessageChatMemoryAdvisor(
redisChatMemory,
AbstractChatMemoryAdvisor.DEFAULT_CHAT_MEMORY_CONVERSATION_ID,
10 // last 10 messages instead of 20
)
Test different window sizes against your eval dataset. For most travel-planning agents, a window of 8–12 messages is enough — users rarely reference something from more than 5 turns ago.
Cache tool results
When multiple users ask about weather in Mysore on the same day, your agent calls getWeather("Mysore") hundreds of times and pays for the same tool result on every call. Cache it.
You already added @Cacheable for scaling (article 21). The same caching reduces costs:
@Cacheable(value = "weather", key = "#city")
@Tool(description = "Get the current weather for a city in India.")
public String getWeather(@ToolParam(description = "City name") String city) {
return weatherApi.fetch(city);
}
The token cost reduction here isn't direct — the tool result is a small part of the input. The real saving is that cached tool results mean shorter tool response messages, which means slightly fewer tokens per request across all users asking about the same city.
More importantly, caching prevents tool API costs, which for some providers (maps, weather, ticketing) can exceed model costs.
Use smaller models for simpler tasks
The most impactful cost lever: not every task needs your most capable model.
A request like "What's the weather in Mysore?" needs basic reasoning and one tool call. A request like "Plan a 7-day budget trip across Karnataka with weather, attractions, and transport for each city" needs complex multi-step reasoning.
Routing by complexity:
@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
String model = selectModel(request.message());
String response = callAgentWithModel(request, model);
return ResponseEntity.ok(response);
}
private String selectModel(String message) {
// simple heuristics — refine based on your traffic patterns
boolean isComplex = message.length() > 200
|| message.contains("plan")
|| message.contains("itinerary")
|| message.contains("compare")
|| message.toLowerCase().contains("week");
return isComplex ? "gemini-1.5-pro" : "gemini-1.5-flash";
}
Flash-class models typically cost 10–20× less than pro-class models. If 70% of your requests are simple factual questions and only 30% need complex reasoning, you've reduced model costs by 60–70% with no change to quality on either class.
Validate this with your eval dataset — run the simple cases through the cheaper model and confirm pass rate stays the same.
Shorten tool return values
Tool results become part of the input on every subsequent turn in the conversation. A tool that returns verbose JSON pays that token cost on every future message in the conversation.
// Before — verbose, gets replayed in history
public String getAttractions(String city) {
return """
{
"city": "Mysore",
"attractions": [
{"name": "Mysore Palace", "rating": 4.8, "category": "Heritage",
"openingHours": "10:00-17:30", "entryFee": "₹50"},
{"name": "Chamundi Hills", "rating": 4.5, "category": "Religious",
"openingHours": "06:00-20:00", "entryFee": "Free"}
],
"lastUpdated": "2026-09-27"
}
""";
}
// After — readable string, same information, ~60% fewer tokens
public String getAttractions(String city) {
return "Top attractions in " + city + ": " +
"Mysore Palace (₹50, Heritage) · " +
"Chamundi Hills (Free, Religious)";
}
The model doesn't need JSON structure to understand tool results — it reads natural language just as well. Plain readable strings are significantly cheaper than structured JSON for the same semantic content.
Avoid redundant tool calls
The model sometimes calls the same tool twice in one conversation — once to answer the current question and again when the user asks a follow-up. If the tool result is already in the conversation history, the model already has the answer.
Enforce this in the system prompt:
"""
You are a travel planning assistant for India.
Use tools for weather, attractions, and transport.
Only state facts returned by tools.
Do not call a tool if its result is already in the conversation history.
"""
This single instruction can eliminate 20–30% of tool calls in multi-turn conversations. Check your logs — if you see the same tool called twice with the same arguments within a conversation, this prompt addition will catch it.
Batch requests where possible
If your application processes multiple independent requests — a daily summary, a batch of user queries, a scheduled report — the Batch API lets you process them at roughly half the cost of individual requests:
// Instead of 100 individual calls at full price:
for (String query : queries) {
results.add(chatClient.prompt().user(query).call().content());
}
// Use batch processing (check your provider's batch API):
// 50% cost reduction, results within 24 hours
Batch processing isn't suitable for real-time user interactions, but it's a significant win for any offline processing your system does — ingesting documents, generating summaries, running eval datasets overnight.
Cost optimisation checklist
| Action | Typical saving | Effort |
|---|---|---|
| Trim system prompt | 10–30% of input tokens | Low |
| Reduce history window size | 20–40% of input tokens | Low |
| Route simple queries to cheaper model | 50–70% of model cost | Medium |
| Shorten tool return values | 5–15% of input tokens | Low |
| Add "don't repeat tool calls" to system prompt | 10–20% of tool calls | Low |
| Cache stable tool results | Variable | Already done |
| Use batch API for offline processing | ~50% of batch costs | Medium |
Start with the top three — they cover most of the saving with the least risk of affecting output quality. Always verify with your eval dataset before shipping any prompt change.
The quality trade-off
Every cost reduction carries some risk of quality degradation. The discipline is:
- Measure baseline pass rate on your eval dataset
- Apply one change at a time
- Re-run eval — confirm pass rate hasn't dropped
- Monitor online eval scores for a week after deploying
If a change drops pass rate even slightly, either revert it or investigate why. Cost optimisation that makes your agent worse isn't worth it.
What's next
This is the final article in the series. We've gone from a basic Spring Boot setup through RAG, agents, tools, memory, patterns, deployment, scaling, MCP, evaluation, and cost optimisation. The last article wraps up what this journey looked like end to end — and what I'd do differently if I started again.
Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.
Top comments (0)