DEV Community: Sham Prakash K

Free Cloud PC That Works in Any Browser — Oracle Cloud Setup Guide (2026)

Sham Prakash K — Sun, 08 Nov 2026 15:34:43 +0000

Ever wanted a computer in the cloud you can open from your phone, laptop, or any device — just by typing a URL in a browser?

No app to install. No VPN. Just open a link and you're staring at a full desktop.

I set this up for free using Oracle Cloud's Always Free tier. Here's exactly how I did it, including the mistakes I made along the way.

Why Oracle Cloud?

Before we start, here's why Oracle Cloud is the best free option:

Provider	Free CPU	Free RAM	Free Forever?
Oracle Cloud	2 ARM cores	12 GB	✅ Yes
Google Cloud	1 shared vCPU	0.6 GB	✅ Yes (but too slow for a browser)
AWS	1 vCPU	1 GB	❌ 12 months only
Azure	1 vCPU	1 GB	❌ 12 months only

Note: Oracle reduced their Always Free ARM tier from 4 OCPUs/24 GB to 2 OCPUs/12 GB in August 2026. It's still the most generous free tier available — Google's free VM has 0.6 GB RAM which can't even run a browser properly.

What You'll Have at the End

A free Ubuntu 22.04 VM running 24/7
Full XFCE desktop environment
Firefox browser running in that VM
Access from any browser via http://YOUR_IP:6080/vnc.html
Works on mobile, laptop, tablet — anything with a browser
A platform to host web servers, bots, scripts, and side projects — all for free

Step 1: Create an Oracle Cloud Account

Go to cloud.oracle.com → Start for free

You'll need:

An email address
A Visa or Mastercard for identity verification (a ~$1 temporary hold is placed and auto-refunded in 3-5 days — RuPay is not accepted)
Choose your Home Region carefully — you can't change it later

Indian users: Oracle charges through their Singapore entity for APAC. The temporary hold appears in SGD (~₹85), not INR. It disappears on its own.

Step 2: Create the Free VM

Go to Compute → Instances → Create Instance
Click Change Image → Select Ubuntu 22.04
Click Change Shape → Ampere (ARM) → VM.Standard.A1.Flex
Set OCPUs: 2 and Memory: 12 GB
Under Networking → Select Create new virtual cloud network
Under Add SSH Keys → Download the private key and save it safely
Click Create and wait 2-3 minutes

Step 3: Assign a Public IP

New VMs sometimes don't get a public IP automatically. Here's how to fix that:

Click your instance → Networking tab
Click the VNIC name → IP Administration tab
Click ⋯ on the row → Edit
Select Ephemeral public IP → Update

Note your public IP — you'll need it throughout this guide.

Step 4: SSH Into the VM

Windows (PowerShell):

ssh -i C:\path\to\your-key.key ubuntu@YOUR_PUBLIC_IP

Mac/Linux:

ssh -i ~/path/to/your-key.key ubuntu@YOUR_PUBLIC_IP

Type yes when asked about the fingerprint. You should see the Ubuntu welcome screen.

Step 5: Install Everything

Run these commands one by one:

# Update the system
sudo apt update && sudo apt upgrade -y

When prompted about restarting services, just press Enter to accept defaults.

# Install XFCE desktop + VNC + noVNC
sudo apt install -y xfce4 xfce4-goodies tightvncserver novnc websockify

# Install Firefox (non-snap version — important!)
sudo add-apt-repository ppa:mozillateam/ppa -y
sudo apt install -y firefox-esr

Why non-snap Firefox? Ubuntu's default Firefox is installed as a snap package, which doesn't work inside a VNC session. Always use firefox-esr from the Mozilla PPA instead.

Step 6: Configure VNC

Set your VNC password:

vncserver :1

Enter a password (max 8 characters). Type n when asked about a view-only password.

Configure XFCE as the desktop:

vncserver -kill :1

cat > ~/.vnc/xstartup << 'EOF'
#!/bin/bash
xrdb $HOME/.Xresources
startxfce4 &
EOF

chmod +x ~/.vnc/xstartup

Start VNC with your screen resolution:

vncserver :1 -geometry 1920x1080 -depth 24

Step 7: Start noVNC

noVNC is what turns the VNC desktop into something you can open in any browser:

websockify --web=/usr/share/novnc/ 6080 localhost:5901 &

Step 8: Open the Firewall

You need to open port 6080 in two places:

1. Inside the VM:

sudo apt install -y iptables-persistent
sudo iptables -I INPUT -p tcp --dport 6080 -j ACCEPT
sudo netfilter-persistent save

The iptables-persistent part is important — without it, the firewall rule disappears every time the VM reboots.

2. In Oracle Cloud Console:

Go to Networking → Virtual Cloud Networks → Your VCN
Click Security tab → Default Security List
Click Add Ingress Rules:
- Source CIDR: 0.0.0.0/0
- Protocol: TCP
- Destination Port Range: 6080
Click Add Ingress Rules

Step 9: Access from Any Device

Open any browser and go to:

http://YOUR_PUBLIC_IP:6080/vnc.html

Enter your VNC password → Click Connect.

You'll see the full Ubuntu desktop. Open Firefox from Applications → Web Browser.

That's it. You can now open this URL from your phone, your office laptop, a friend's computer — anywhere.

Make It Survive Reboots

Set up systemd services so everything starts automatically:

sudo bash -c 'cat > /etc/systemd/system/vncserver.service << EOF
[Unit]
Description=VNC Server
After=network.target

[Service]
User=ubuntu
Type=forking
ExecStartPre=-/usr/bin/vncserver -kill :1
ExecStartPre=-/bin/rm -f /tmp/.X1-lock /tmp/.X11-unix/X1
ExecStart=/usr/bin/vncserver :1 -geometry 1920x1080 -depth 24
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF'

sudo bash -c 'cat > /etc/systemd/system/novnc.service << EOF
[Unit]
Description=noVNC
After=vncserver.service
Requires=vncserver.service

[Service]
User=ubuntu
ExecStart=/usr/bin/websockify --web=/usr/share/novnc/ 6080 localhost:5901
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF'

sudo systemctl daemon-reload
sudo systemctl enable vncserver novnc
sudo systemctl start vncserver novnc

Now VNC + noVNC start automatically on every reboot. No SSH needed to get back in.

Transferring Files to Your VM

Copy a file from your local machine to the VM:

scp -i /path/to/key.key /local/file.pdf ubuntu@YOUR_IP:/home/ubuntu/

What Else Can You Do With This VM?

A free cloud VM with 2 cores and 12 GB RAM is surprisingly capable. Here's what you can run on it beyond just browsing:

🛠️ Development & Coding

Host a web server — deploy a Node.js, Python Flask, or any backend API for free
VS Code in the browser — install code-server and get a full IDE accessible from any device
Database server — run PostgreSQL, MongoDB, or Redis
Docker containers — spin up multiple services on one VM

📚 Learning & Practice

Linux practice — a safe sandbox to learn terminal commands without breaking your own machine
Learn DevOps — practice Docker, CI/CD pipelines, nginx configs
Self-host your side projects — deploy and share your projects for free
Run Python/ML scripts — small AI experiments and data processing

⚡ Always-On Tasks

Background downloads — leave large downloads running 24/7 without keeping your laptop on
Scheduled cron jobs — run scripts at specific times automatically
Telegram/Discord bots — keep them always online
Web scrapers — run data collection scripts continuously

🔒 Privacy

Browse with a different IP — websites see the VM's IP, not yours
Isolated browsing — keep work and personal browsing completely separate

Gotchas I Hit (So You Don't Have To)

1. Public IP not assigned by default
New Oracle VMs sometimes have no public IP. Go to Networking → VNIC → IP Administration to assign one.

2. Snap browsers don't work in VNC
Both chromium-browser and firefox on Ubuntu 22.04 install as snap packages which crash inside VNC with a cgroup error. Use firefox-esr from the Mozilla PPA instead.

3. iptables rules reset on reboot
After resizing the VM (changing CPU/RAM), Oracle reboots it and iptables resets. The fix is iptables-persistent which saves rules permanently.

4. Mumbai region capacity
If you're in India, the Mumbai region is often at capacity for free ARM VMs. You may get fewer OCPUs/RAM than the limit allows. Try resizing after creation, or add the Singapore region to your account.

Security

Don't share your URL + VNC password with anyone you don't trust
Enable MFA on your Oracle account: Profile → Security → MFA
The connection is HTTP (not HTTPS) — don't do internet banking or enter real passwords inside the VM
Change your VNC password anytime with vncpasswd

Final Thoughts

For ₹0 a month, you get a cloud computer with 2 cores and 12 GB RAM running 24/7, accessible from any device on the planet.

Is it blazing fast? No. Is it free forever and genuinely useful? Absolutely.

I use it for browsing, running scripts, and keeping things running in the background — without touching my local machine's resources.

If this helped you, drop a ❤️! And if you hit any issues, ask in the comments — I've probably already made that mistake. 😄

The AI Was Never the Hard Part

Sham Prakash K — Sun, 01 Nov 2026 14:30:00 +0000

Twenty-four articles. One agent. A lot of things I got wrong before I got them right.

This one isn't about code. It's about what I'd tell myself at article 1 if I could go back.

I thought the model was the hard part

It isn't. Spring AI handles the model. You call chatClient.prompt().user(message).call().content() and it works. Five lines.

The hard part is everything around that call. How do you pass context without spending a fortune on tokens? What happens when a tool fails mid-conversation? How do you know if the response is actually correct? How do you ship without putting your API key in git?

Those questions have nothing to do with AI. They're the same questions you'd ask about any external service you're integrating with. I wasted the first few articles thinking I needed to understand the model better. I should have been thinking about the backend.

I should have built the eval dataset first

This is the biggest regret.

I built the eval dataset in article 23. Twenty-two articles after I made my first API call. Before that, every time I changed a prompt or adjusted a tool, I was testing by hand — send a message, read the response, guess whether it was better or worse.

That's not testing. That's hoping.

The eval dataset forces you to define what "correct" means before you build anything. Once you have it, changing the system prompt goes from "seems okay?" to "pass rate held at 87%." The difference is enormous. I'd build five test cases in article 1 if I started again — even dumb ones. Anything is better than nothing.

The tool design mistake I kept making

Early tools returned verbose JSON. Something like:

{
  "city": "Mysore",
  "weather": "partly cloudy",
  "temperature": "24°C",
  "humidity": "68%",
  "lastUpdated": "2026-09-27T10:30:00"
}

That structure replays in every subsequent message in the conversation. The model doesn't need it. It can read "Mysore: 24°C, partly cloudy, 68% humidity" just as well — and that's 60% fewer tokens in every follow-up turn.

I changed this late. By the time I got to the cost optimisation article, I'd been paying for bloated tool returns for weeks. Small thing, big compounding effect.

The other tool mistake: throwing exceptions. A tool that throws NoRouteException crashes the agent and surfaces a 500 to the user. A tool that returns "No direct route found between Mumbai and Mysore" lets the model acknowledge the gap and keep going. I had to learn this the hard way — an exception in getTicketPrice that took down the whole conversation before I caught it.

Log tokens from day one

I added token logging in article 20. I should have added it in article 2.

Every request logs inputTokens and outputTokens. Without that data, you're guessing where the cost is going. When I finally queried the averages after a few days of traffic, the breakdown was immediately obvious: history was 45% of every request, the system prompt was another 25%. Together they were 70% of input tokens — and both were fixable with no change to output quality.

But I didn't know that until I measured it. If I'd been logging from the start, I would have caught it much earlier and built with more cost awareness from the beginning.

Memory architecture matters more than model choice

Swapping models changes the quality of responses. Getting memory wrong breaks the agent.

A conversation window that's too big makes every request expensive. A window that's too small makes the agent forget context three turns in. Finding the right size — 10 messages in this case — required thinking carefully about actual user behaviour, not just picking a number.

I spent more time tuning the model and its prompts than I spent thinking about memory. In hindsight, that's backwards. RedisChatMemory with a sensible window size was the most impactful architectural decision in the whole project. The model barely matters if the context management is wrong.

What I'd do differently

Build the eval dataset first. Log tokens from the first API call. Use TravelToolsPort as an interface from article 3, not article 23 — testability from the start, not retrofitted later. Think about MCP earlier — not because local @Tool methods are wrong, but because understanding the coupling between the agent and its tools changes how you design tool boundaries.

Everything else I'd keep. The bottom-up approach — one concept per article, each building on the last — was right. Spring AI's abstractions are at the right level. Starting small was right.

The short version

AI backend engineering is backend engineering. The model is the new external service — you call it, handle its failures, measure its cost, and test against it. Everything else is the same: interfaces, observability, evaluation, operational readiness.

Start small. Measure everything. Build testability in from the start. The AI-specific parts follow the same principles as the rest.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Token Economics: Optimising Cost in a Spring AI Agent

Sham Prakash K — Thu, 29 Oct 2026 14:30:00 +0000

Model API costs are invisible until they're not. During development you make a few hundred calls and barely notice the bill. In production with real users, the same agent can cost ten times what you expected — not because something is wrong, but because small inefficiencies multiply at scale.

The good news: most of the cost in a typical AI agent comes from a handful of sources, and most of those sources have straightforward fixes that don't affect output quality at all.

This article covers where token costs actually come from and how to reduce them without making your agent worse.

Where the money goes

Before optimising anything, measure where tokens are actually being spent. Log usage on every request:

ChatResponse chatResponse = chatClient.prompt()
    .user(request.message())
    .call()
    .chatResponse();

Usage usage = chatResponse.getMetadata().getUsage();
log.info("[cost] conversationId={} inputTokens={} outputTokens={} totalTokens={}",
    request.conversationId(),
    usage.getPromptTokens(),
    usage.getGenerationTokens(),
    usage.getTotalTokens());

After a few days of production traffic, query the averages:

SELECT
  AVG(input_tokens)  AS avg_input,
  AVG(output_tokens) AS avg_output,
  AVG(total_tokens)  AS avg_total,
  MAX(total_tokens)  AS max_total
FROM request_usage
WHERE created_at > NOW() - INTERVAL '7 days';

This tells you where to focus. In most agents the breakdown looks like:

Source	Typical share of input tokens
System prompt	15–30%
Tool definitions	10–25%
Conversation history	30–50%
User message	5–15%

Output tokens are usually 10–20% of total. Input tokens are where most of the cost hides.

Trim the system prompt

Every word in your system prompt fires on every single request. A 500-token system prompt on 10,000 daily requests is 5 million tokens just for instructions.

Common bloat to cut:

// Before — verbose, repetitive
"""
You are a helpful AI assistant specialising in travel planning for India.
Your job is to help users plan their trips by providing information about
weather, attractions, and transportation options. You should always be
polite and professional. When users ask about travel, use the available
tools to get real-time information. Always provide accurate information
based on the tool results and never make up information that isn't
provided by the tools.
"""

// After — same behaviour, ~40% fewer tokens
"""
You are a travel planning assistant for India.
Use tools for weather, attractions, and transport.
Only state facts returned by tools.
"""

Test that the shortened prompt produces equivalent output quality using your eval dataset before shipping it. Prompt compression is one of the highest-leverage cost reductions — it compounds across every request.

Bound the conversation history window

Conversation history is the biggest token cost driver for multi-turn agents. Each request includes all prior messages up to the window size.

A conversation that's been going for 20 turns might look like:

Turn 1:  user (15 tokens) + assistant (80 tokens)
Turn 2:  user (20 tokens) + assistant (120 tokens) + 2 tool calls (60 tokens)
...
Turn 20: all of the above replayed = ~2,000 tokens of history
         + new user message (25 tokens)
         + system prompt (80 tokens)
         + tool definitions (200 tokens)
         = ~2,300 tokens input just for context

The window size directly controls this cost. Keep it as small as correctness allows:

new MessageChatMemoryAdvisor(
    redisChatMemory,
    AbstractChatMemoryAdvisor.DEFAULT_CHAT_MEMORY_CONVERSATION_ID,
    10  // last 10 messages instead of 20
)

Test different window sizes against your eval dataset. For most travel-planning agents, a window of 8–12 messages is enough — users rarely reference something from more than 5 turns ago.

Cache tool results

When multiple users ask about weather in Mysore on the same day, your agent calls getWeather("Mysore") hundreds of times and pays for the same tool result on every call. Cache it.

You already added @Cacheable for scaling (article 21). The same caching reduces costs:

@Cacheable(value = "weather", key = "#city")
@Tool(description = "Get the current weather for a city in India.")
public String getWeather(@ToolParam(description = "City name") String city) {
    return weatherApi.fetch(city);
}

The token cost reduction here isn't direct — the tool result is a small part of the input. The real saving is that cached tool results mean shorter tool response messages, which means slightly fewer tokens per request across all users asking about the same city.

More importantly, caching prevents tool API costs, which for some providers (maps, weather, ticketing) can exceed model costs.

Use smaller models for simpler tasks

The most impactful cost lever: not every task needs your most capable model.

A request like "What's the weather in Mysore?" needs basic reasoning and one tool call. A request like "Plan a 7-day budget trip across Karnataka with weather, attractions, and transport for each city" needs complex multi-step reasoning.

Routing by complexity:

@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    String model = selectModel(request.message());
    String response = callAgentWithModel(request, model);
    return ResponseEntity.ok(response);
}

private String selectModel(String message) {
    // simple heuristics — refine based on your traffic patterns
    boolean isComplex = message.length() > 200
        || message.contains("plan")
        || message.contains("itinerary")
        || message.contains("compare")
        || message.toLowerCase().contains("week");

    return isComplex ? "gemini-1.5-pro" : "gemini-1.5-flash";
}

Flash-class models typically cost 10–20× less than pro-class models. If 70% of your requests are simple factual questions and only 30% need complex reasoning, you've reduced model costs by 60–70% with no change to quality on either class.

Validate this with your eval dataset — run the simple cases through the cheaper model and confirm pass rate stays the same.

Shorten tool return values

Tool results become part of the input on every subsequent turn in the conversation. A tool that returns verbose JSON pays that token cost on every future message in the conversation.

// Before — verbose, gets replayed in history
public String getAttractions(String city) {
    return """
        {
          "city": "Mysore",
          "attractions": [
            {"name": "Mysore Palace", "rating": 4.8, "category": "Heritage",
             "openingHours": "10:00-17:30", "entryFee": "₹50"},
            {"name": "Chamundi Hills", "rating": 4.5, "category": "Religious",
             "openingHours": "06:00-20:00", "entryFee": "Free"}
          ],
          "lastUpdated": "2026-09-27"
        }
        """;
}

// After — readable string, same information, ~60% fewer tokens
public String getAttractions(String city) {
    return "Top attractions in " + city + ": " +
           "Mysore Palace (₹50, Heritage) · " +
           "Chamundi Hills (Free, Religious)";
}

The model doesn't need JSON structure to understand tool results — it reads natural language just as well. Plain readable strings are significantly cheaper than structured JSON for the same semantic content.

Avoid redundant tool calls

The model sometimes calls the same tool twice in one conversation — once to answer the current question and again when the user asks a follow-up. If the tool result is already in the conversation history, the model already has the answer.

Enforce this in the system prompt:

"""
You are a travel planning assistant for India.
Use tools for weather, attractions, and transport.
Only state facts returned by tools.
Do not call a tool if its result is already in the conversation history.
"""

This single instruction can eliminate 20–30% of tool calls in multi-turn conversations. Check your logs — if you see the same tool called twice with the same arguments within a conversation, this prompt addition will catch it.

Batch requests where possible

If your application processes multiple independent requests — a daily summary, a batch of user queries, a scheduled report — the Batch API lets you process them at roughly half the cost of individual requests:

// Instead of 100 individual calls at full price:
for (String query : queries) {
    results.add(chatClient.prompt().user(query).call().content());
}

// Use batch processing (check your provider's batch API):
// 50% cost reduction, results within 24 hours

Batch processing isn't suitable for real-time user interactions, but it's a significant win for any offline processing your system does — ingesting documents, generating summaries, running eval datasets overnight.

Cost optimisation checklist

Action	Typical saving	Effort
Trim system prompt	10–30% of input tokens	Low
Reduce history window size	20–40% of input tokens	Low
Route simple queries to cheaper model	50–70% of model cost	Medium
Shorten tool return values	5–15% of input tokens	Low
Add "don't repeat tool calls" to system prompt	10–20% of tool calls	Low
Cache stable tool results	Variable	Already done
Use batch API for offline processing	~50% of batch costs	Medium

Start with the top three — they cover most of the saving with the least risk of affecting output quality. Always verify with your eval dataset before shipping any prompt change.

The quality trade-off

Every cost reduction carries some risk of quality degradation. The discipline is:

Measure baseline pass rate on your eval dataset
Apply one change at a time
Re-run eval — confirm pass rate hasn't dropped
Monitor online eval scores for a week after deploying

If a change drops pass rate even slightly, either revert it or investigate why. Cost optimisation that makes your agent worse isn't worth it.

What's next

This is the final article in the series. We've gone from a basic Spring Boot setup through RAG, agents, tools, memory, patterns, deployment, scaling, MCP, evaluation, and cost optimisation. The last article wraps up what this journey looked like end to end — and what I'd do differently if I started again.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

How Do You Know Your AI Agent Is Wrong? Evaluation for Java Backends

Sham Prakash K — Tue, 27 Oct 2026 14:30:00 +0000

A user reports that your agent gave them wrong information. You check the logs — the tool calls look correct, the model returned a response, no exceptions were thrown. Everything ran, but the answer was bad.

This is the hardest problem in AI backend development. The agent doesn't crash when it's wrong. It confidently returns a plausible-sounding answer that happens to be incorrect, and your standard monitoring missed it entirely.

Evaluating AI output means building systems that can catch wrong answers before they reach users — or at least before they become a pattern. This article covers how to approach it practically.

Why traditional testing doesn't work

Unit tests verify deterministic behaviour. Call a function with input X, expect output Y. AI models are not deterministic — the same prompt can produce slightly different answers on different runs, and many correct answers exist for the same question.

You can't write:

assertEquals("24°C", agent.ask("What's the weather in Mysore?"));

The agent might say "It's 24 degrees and sunny" or "Currently 24°C with clear skies" — both correct, neither matching the expected string.

What you need instead: evaluation criteria, not expected strings. The question isn't "did the agent return exactly this?" but "did the agent return something that satisfies these conditions?"

The three types of checks

1. Structural checks — does the response have the right shape?

These are the easiest to automate. They don't require understanding the content.

@Test
void agentResponseShouldNotBeEmpty() {
    String response = agent.chat("Plan a trip to Mysore");
    assertThat(response).isNotBlank();
    assertThat(response.length()).isGreaterThan(50);
}

@Test
void agentShouldMentionTemperature() {
    String response = agent.chat("What's the weather in Mysore?");
    assertThat(response).containsAnyOf("°C", "degrees", "temperature", "warm", "hot", "cool");
}

@Test
void agentShouldNotHallucinate() {
    String response = agent.chat("What is the weather in Mysore?");
    assertThat(response).doesNotContain("I don't know");
    assertThat(response).doesNotContain("I cannot access");
}

These catch obvious failures: blank responses, refusals when the tool should have answered, missing key information.

2. Semantic checks — is the content correct?

These require either a known ground truth or a scoring mechanism.

For known ground truth (the answer is a fact you can verify):

@Test
void agentWeatherShouldMatchActualData() {
    // inject a mock tool that always returns "Mysore: 24°C, clear"
    String response = agentWithMockTools.chat("What's the weather in Mysore?");
    assertThat(response).contains("24");
}

Using mock tools with fixed return values turns semantic checks into structural ones — you control the data, so you know what the answer should contain.

3. Consistency checks — does the agent behave the same way across runs?

Run the same prompt multiple times and compare outputs:

@Test
void agentShouldBeConsistentOnFactualQuestions() {
    Set<String> responses = new HashSet<>();
    for (int i = 0; i < 5; i++) {
        responses.add(agentWithMockTools.chat("What's the weather in Mysore?"));
    }
    // All responses should mention 24°C since the mock always returns it
    responses.forEach(r ->
        assertThat(r).contains("24")
    );
}

High variance on factual questions (where tool data is fixed) indicates a prompt problem — the model isn't reliably grounding its answer in the tool result.

Building a test dataset

Ad-hoc tests catch known problems. A test dataset catches regressions — problems you didn't know to look for.

Structure each test case with: input, expected properties, and what would count as failure.

public record EvalCase(
    String input,
    List<String> mustContain,
    List<String> mustNotContain,
    String description
) {}

A starter dataset for a travel agent:

List<EvalCase> evalDataset = List.of(
    new EvalCase(
        "What's the weather in Mysore?",
        List.of("24", "°C"),
        List.of("I cannot", "I don't know"),
        "weather query should return temperature"
    ),
    new EvalCase(
        "How do I get from Bangalore to Mysore?",
        List.of("bus", "train", "₹"),
        List.of("I cannot"),
        "travel query should return route and cost"
    ),
    new EvalCase(
        "Plan a 2-day trip to Mysore within ₹3,000",
        List.of("₹", "Mysore", "day"),
        List.of("I cannot", "unable"),
        "trip plan should be structured and within budget context"
    ),
    new EvalCase(
        "What is the capital of France?",
        List.of(),
        List.of(),
        "out-of-scope question — agent should handle gracefully"
    )
);

Run the dataset:

@Test
void evalDatasetShouldPass() {
    List<String> failures = new ArrayList<>();

    for (EvalCase evalCase : evalDataset) {
        String response = agentWithMockTools.chat(evalCase.input());

        for (String must : evalCase.mustContain()) {
            if (!response.toLowerCase().contains(must.toLowerCase())) {
                failures.add(String.format(
                    "[%s] missing '%s' in: %s",
                    evalCase.description(), must, response.substring(0, Math.min(100, response.length()))
                ));
            }
        }

        for (String mustNot : evalCase.mustNotContain()) {
            if (response.toLowerCase().contains(mustNot.toLowerCase())) {
                failures.add(String.format(
                    "[%s] found forbidden '%s' in: %s",
                    evalCase.description(), mustNot, response.substring(0, Math.min(100, response.length()))
                ));
            }
        }
    }

    assertThat(failures)
        .withFailMessage("Eval failures:\n" + String.join("\n", failures))
        .isEmpty();
}

This runs the full dataset as one test. Any failure prints the case description and the offending response. Add cases as you discover new failure modes in production.

Mock tools for evaluation

The hardest part of evaluating an agent in a test environment is that its tools call external APIs. Those APIs are slow, cost money, and return different values over time.

The solution: inject mock tools during evaluation.

Define an interface for your tools:

public interface TravelToolsPort {
    String getWeather(String city);
    String getAttractions(String city);
    String getTicketPrice(String from, String to);
}

Create a fixed mock implementation:

public class MockTravelTools implements TravelToolsPort {

    @Override
    @Tool(description = "Get the current weather for a city in India.")
    public String getWeather(@ToolParam(description = "City name") String city) {
        return switch (city.toLowerCase()) {
            case "mysore"    -> "Mysore: 24°C, clear skies";
            case "bangalore" -> "Bangalore: 22°C, partly cloudy";
            default          -> "Weather data not available for '" + city + "'.";
        };
    }

    @Override
    @Tool(description = "Get the top tourist attractions in a city.")
    public String getAttractions(@ToolParam(description = "City name") String city) {
        return switch (city.toLowerCase()) {
            case "mysore" -> "Top attractions: Mysore Palace, Chamundi Hills, Brindavan Gardens";
            default       -> "No attraction data for '" + city + "'.";
        };
    }

    @Override
    @Tool(description = "Get the cheapest ticket price between two cities.")
    public String getTicketPrice(
            @ToolParam(description = "Departure city") String from,
            @ToolParam(description = "Destination city") String to) {
        if (from.equalsIgnoreCase("bangalore") && to.equalsIgnoreCase("mysore")) {
            return "Cheapest: KSRTC bus ₹180, journey 3 hours";
        }
        return "No direct route found between " + from + " and " + to + ".";
    }
}

Wire the mock into a test-only ChatClient bean:

@TestConfiguration
public class EvalConfig {

    @Bean
    @Primary
    public ChatClient evalChatClient(ChatModel chatModel) {
        MockTravelTools mockTools = new MockTravelTools();
        return ChatClient.builder(chatModel)
            .defaultSystem("You are a travel planning assistant for India.")
            .defaultTools(mockTools)
            .build();
    }
}

Now eval tests run without hitting any external API. They're fast, free, and deterministic.

Tracking pass rate over time

A single test run tells you if something is broken now. A history of runs tells you if quality is drifting.

Log each eval run to a table:

@PostMapping("/eval/run")
public EvalResult runEval() {
    int passed = 0;
    int failed = 0;
    List<String> failures = new ArrayList<>();

    for (EvalCase evalCase : evalDataset) {
        String response = evalChatClient.prompt()
            .user(evalCase.input())
            .call()
            .content();

        boolean pass = evalCase.mustContain().stream()
            .allMatch(s -> response.toLowerCase().contains(s.toLowerCase()))
            && evalCase.mustNotContain().stream()
            .noneMatch(s -> response.toLowerCase().contains(s.toLowerCase()));

        if (pass) passed++;
        else {
            failed++;
            failures.add(evalCase.description());
        }
    }

    EvalResult result = new EvalResult(
        Instant.now(), passed, failed,
        (double) passed / (passed + failed), failures
    );

    evalResultRepository.save(result);
    return result;
}

Run this endpoint after every deployment. If pass rate drops, a recent change broke something. If it drops gradually over days, your prompt or tool data is drifting against user expectations.

What to do when an agent gives a wrong answer in production

When a user reports a bad answer, the investigation follows the same path every time:

Find the conversation ID in the logs
Replay the tool calls from the log — did they return correct data?
If tool data was correct, the model misused it — update the system prompt or tool description
If tool data was wrong, fix the tool
Add a new eval case covering this exact failure so it can't silently regress

Step 5 is the most important one. Every production failure is an opportunity to make your eval dataset stronger.

LLM-as-a-judge

String-based checks (mustContain, mustNotContain) are fast and cheap but limited. They can't tell whether an answer is helpful, accurate in tone, or logically sound. They only check whether certain words appear.

LLM-as-a-judge fixes this: you send the question and the agent's answer to a second model and ask it to score the response. The judge model reads the full answer and reasons about its quality — the way a human reviewer would.

@Service
public class LlmJudge {

    private final ChatClient judgeClient;

    public LlmJudge(ChatModel chatModel) {
        this.judgeClient = ChatClient.builder(chatModel)
            .defaultSystem("""
                You are an evaluator for an AI travel assistant.
                Given a user question and the assistant's response,
                score the response on a scale of 1 to 5:

                5 — complete, accurate, directly answers the question
                4 — mostly correct, minor gaps
                3 — partially answers but missing key information
                2 — vague or unhelpful despite having the right data
                1 — wrong, hallucinated, or refused without reason

                Reply with ONLY a JSON object:
                {"score": <1-5>, "reason": "<one sentence>"}
                """)
            .build();
    }

    public JudgeResult score(String question, String answer) {
        String prompt = "Question: " + question + "\n\nAnswer: " + answer;

        String raw = judgeClient.prompt()
            .user(prompt)
            .call()
            .content();

        try {
            BeanOutputConverter<JudgeResult> converter =
                new BeanOutputConverter<>(JudgeResult.class);
            return converter.convert(raw);
        } catch (Exception e) {
            return new JudgeResult(0, "failed to parse judge response");
        }
    }
}

public record JudgeResult(int score, String reason) {}

Use it in your eval dataset runner alongside string checks:

for (EvalCase evalCase : evalDataset) {
    String response = agentWithMockTools.chat(evalCase.input());

    // existing string checks
    boolean structuralPass = evalCase.mustContain().stream()
        .allMatch(s -> response.toLowerCase().contains(s.toLowerCase()));

    // LLM judge score
    JudgeResult judgeResult = llmJudge.score(evalCase.input(), response);

    log.info("[eval] case='{}' structural={} judgeScore={} reason='{}'",
        evalCase.description(), structuralPass,
        judgeResult.score(), judgeResult.reason());

    if (judgeResult.score() < 3) {
        failures.add(String.format("[%s] judge score %d: %s",
            evalCase.description(), judgeResult.score(), judgeResult.reason()));
    }
}

What LLM-as-a-judge catches that string checks miss:

An answer that contains "24°C" but buries it in a confusing paragraph — string check passes, judge scores it 3/5
An answer that invents a detail not in the tool data — structurally fine, judge flags the hallucination
An answer that refuses when it should answer — structural check might not have a mustNotContain for this specific refusal phrasing, but the judge scores it 1/5

The cost trade-off: every judge call is a full model API call. For a dataset of 20 cases, that's 20 extra calls per eval run. Use string checks as the first pass — only run the judge on cases that pass string checks, or on a random sample of production traffic.

Which model to use as judge? The judge should be at least as capable as the agent, ideally more capable. If your agent uses a smaller model, judge with a larger one. The judge's output is only as reliable as its own reasoning ability.

Online evaluation — scoring live production traffic

Everything so far runs offline: in a test suite before deployment. Online evaluation runs in production, scoring real responses as they happen.

Why bother? Your eval dataset covers cases you anticipated. Real users ask things you never thought of. Online evaluation catches quality problems that your test cases missed.

The pattern: after the agent produces a response, score it before (or after) returning it to the user.

@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    String requestId = UUID.randomUUID().toString();
    long start = System.currentTimeMillis();

    String response = callAgent(request.message(), request.conversationId());

    // score asynchronously — don't block the user
    CompletableFuture.runAsync(() ->
        scoreOnline(requestId, request.message(), response)
    );

    return ResponseEntity.ok(response);
}

private void scoreOnline(String requestId, String question, String answer) {
    try {
        // fast structural check first
        boolean hasContent = answer.length() > 30
            && !answer.toLowerCase().contains("i cannot")
            && !answer.toLowerCase().contains("i don't know");

        if (!hasContent) {
            log.warn("[eval:online] requestId={} structural=FAIL answer='{}'",
                requestId, answer.substring(0, Math.min(80, answer.length())));
            onlineEvalRepository.save(new OnlineEvalResult(
                requestId, question, answer, false, 0, "structural fail"));
            return;
        }

        // LLM judge for a sample (10% of traffic)
        if (Math.random() < 0.10) {
            JudgeResult judgeResult = llmJudge.score(question, answer);
            log.info("[eval:online] requestId={} judgeScore={} reason='{}'",
                requestId, judgeResult.score(), judgeResult.reason());
            onlineEvalRepository.save(new OnlineEvalResult(
                requestId, question, answer, true,
                judgeResult.score(), judgeResult.reason()));
        }
    } catch (Exception e) {
        log.warn("[eval:online] scoring failed for requestId={}: {}", requestId, e.getMessage());
    }
}

Key design decisions in this pattern:

Score asynchronously. Never make the user wait for the evaluation. Fire it off with CompletableFuture.runAsync() and return the response immediately.

Sample for LLM judging. Running the judge on 100% of traffic doubles your model costs. Start with 10% — enough to detect quality trends without doubling the bill. Increase the sample rate temporarily after a deployment to validate the new version.

Always do structural checks. Structural checks are free (just string operations). Run them on 100% of traffic. Only invoke the judge on a sample.

Store results. Every scored response goes to a table. After a week of production traffic, you can query: "what's the average judge score this week vs last week?" and "which question types score lowest?"

@Query("SELECT AVG(judgeScore) FROM OnlineEvalResult " +
       "WHERE judgeScore > 0 AND createdAt > :since")
double averageScoreSince(@Param("since") Instant since);

Combining offline and online:

	Offline eval	Online eval
When	Before every deploy	On every live request
What it catches	Regressions vs known cases	Unknown failure patterns
Cost	Fixed (N cases × model call)	Proportional to traffic
Action	Block deploy if pass rate drops	Alert if average score drops

The two complement each other. Offline eval is your gate — it prevents known regressions from reaching production. Online eval is your monitor — it catches new failure modes that your test dataset didn't anticipate.

Limits of automated evaluation

Automated evals catch structural and factual problems. They don't catch:

Tone and helpfulness — the answer is technically correct but unhelpful or rude. Requires human review.

Subtle hallucinations — the agent makes up a plausible detail not in the tool data. Hard to catch without comprehensive ground truth.

Edge case combinations — real users ask things your eval dataset never anticipated. Eval coverage grows over time, it's never complete.

Prompt injection in user input — a user sends "ignore previous instructions and...". Structural checks won't catch this; requires specific adversarial test cases.

The right mental model: automated evaluation is a regression net, not a quality guarantee. It tells you the agent isn't obviously broken. Human review for a sample of real conversations tells you whether it's actually good.

What's next

The agent is evaluated and monitored. The last pieces of the production puzzle are cost and efficiency — how to reduce what you spend on the model without reducing what users get. The next article covers cost optimisation: prompt caching, smaller models for simpler tasks, and reducing token waste.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Beyond @Tool: Connecting Your Spring AI Agent to External Tool Servers

Sham Prakash K — Sun, 25 Oct 2026 14:30:00 +0000

Every tool we've built so far lives inside the Spring Boot application. The @Tool method is in the same JAR as the agent. That works fine for small projects — but it doesn't scale well as the number of tools grows, and it means every tool change requires rebuilding and redeploying the agent.

The Model Context Protocol (MCP) changes this. It's a standard that lets tools live in a separate process — their own server — while the agent connects to them over a network. The agent doesn't care where the tool runs. It just calls it.

This article covers what MCP is, how to build an MCP server in Java, and how to connect it to a Spring AI agent.

What MCP actually is

MCP is a protocol — a defined format for how an AI agent discovers and calls tools that live outside it.

Without MCP, tools are tightly coupled to the agent:

agent → calls → @Tool method → inside the same JVM

With MCP, tools live in a separate server:

agent → HTTP/SSE → MCP server → runs the tool → returns result

The agent asks the MCP server: "what tools do you have?" The server responds with a list of tool names, descriptions, and parameter schemas — exactly like the @Tool and @ToolParam annotations define locally. The agent then calls those tools over the wire just as if they were local.

Why does this matter?

Tools written in different languages can serve the same agent
Tools can be updated and redeployed without touching the agent
Multiple agents can share the same MCP server
External teams can publish tools without access to your agent codebase

Building an MCP server with Spring AI

Spring AI includes built-in support for exposing tools as an MCP server. You add a dependency, annotate your tool class, and the framework handles the protocol.

Add the dependency:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-mcp-server-spring-boot-starter</artifactId>
</dependency>

Configure it in application.properties:

spring.ai.mcp.server.name=travel-tools
spring.ai.mcp.server.version=1.0.0
spring.ai.mcp.server.type=SYNC

Write the tool class — same @Tool and @ToolParam annotations you already know:

@Service
public class TravelTools {

    @Tool(description = "Get the current weather for a city in India. " +
                        "Use when the user asks about weather or travel conditions.")
    public String getWeather(
            @ToolParam(description = "City name, e.g. Mysore") String city) {
        try {
            return weatherApi.fetch(city);
        } catch (Exception e) {
            return "Weather data unavailable for '" + city + "'. Try a nearby city.";
        }
    }

    @Tool(description = "Get the top tourist attractions in a city. " +
                        "Use when planning activities or asking what to visit.")
    public String getAttractions(
            @ToolParam(description = "City name") String city) {
        return attractionApi.fetch(city);
    }

    @Tool(description = "Get the cheapest ticket price between two cities in India. " +
                        "Use when asking about travel cost or how to get somewhere.")
    public String getTicketPrice(
            @ToolParam(description = "Departure city") String from,
            @ToolParam(description = "Destination city") String to) {
        try {
            return ticketApi.cheapest(from, to);
        } catch (NoRouteException e) {
            return "No direct route found between " + from + " and " + to + ".";
        }
    }
}

@Configuration
public class McpServerConfig {

    @Bean
    public List<McpServerFeatures.SyncToolSpecification> tools(TravelTools travelTools) {
        ToolCallbackProvider provider =
            MethodToolCallbackProvider.builder()
                .toolObjects(travelTools)
                .build();

        return Arrays.stream(provider.getToolCallbacks())
            .map(callback -> new McpServerFeatures.SyncToolSpecification(
                callback.getToolDefinition(),
                (exchange, args) -> new McpSchema.CallToolResult(
                    List.of(new McpSchema.TextContent(
                        callback.call(String.valueOf(args))
                    )), false)
            ))
            .toList();
    }
}

Start the server. It exposes an SSE endpoint at /sse by default. Run it on port 8081.

server.port=8081

The MCP server is now running — it lists your tools and handles calls to them.

Connecting the agent to the MCP server

On the agent side (your Spring Boot application running on port 8080), add the MCP client dependency:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-mcp-client-spring-boot-starter</artifactId>
</dependency>

Configure the connection:

spring.ai.mcp.client.sse.connections.travel-server.url=http://localhost:8081

Spring AI automatically discovers the tools from the MCP server at startup. You can see this in the log:

Discovered 3 tools from MCP server 'travel-server':
  - getWeather
  - getAttractions
  - getTicketPrice

Use the discovered tools in the agent — they arrive as ToolCallback beans:

@Component
public class TravelAgent {

    private final ChatClient chatClient;

    public TravelAgent(ChatClient.Builder builder,
                       List<McpSyncClient> mcpClients) {
        List<ToolCallback> mcpTools = new McpToolCallbackProvider(mcpClients)
            .getToolCallbacks();

        this.chatClient = builder
            .defaultSystem("You are a travel planning assistant for India.")
            .defaultTools(mcpTools)
            .build();
    }

    public String chat(String message, String conversationId) {
        return chatClient.prompt()
            .user(message)
            .advisors(a -> a.param(
                AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY,
                conversationId))
            .call()
            .content();
    }
}

The agent calls the MCP tools exactly as if they were local @Tool methods. The model sees the same tool names, descriptions, and parameter schemas. The network hop is invisible to the model.

The tool discovery lifecycle

When the agent starts, it connects to each configured MCP server and requests a list of available tools. This is the discovery step. The tool definitions are loaded once at startup and cached in the agent.

1. Agent starts
2. Agent connects to MCP server at http://localhost:8081/sse
3. Agent calls: tools/list
4. MCP server returns: [{name, description, inputSchema}, ...]
5. Agent registers these as available tools
6. User sends a message
7. Model selects a tool → agent calls MCP server: tools/call
8. MCP server runs the tool → returns result
9. Agent returns result to model → model generates response

If you add a new tool to the MCP server and restart it, the agent picks it up on its next restart (or reconnection). Tools can evolve independently of the agent.

Running multiple MCP servers

One agent can connect to multiple MCP servers — each responsible for a different domain:

spring.ai.mcp.client.sse.connections.travel-server.url=http://localhost:8081
spring.ai.mcp.client.sse.connections.payments-server.url=http://localhost:8082
spring.ai.mcp.client.sse.connections.docs-server.url=http://localhost:8083

The agent merges all discovered tools into one pool. When the model picks a tool, the agent routes the call to the correct server automatically — you don't write any routing logic.

This is powerful for larger systems: a payments team owns the payments MCP server, a content team owns the docs MCP server, and the agent team owns the agent. Each deploys and scales independently.

Error handling across the network

Local tool errors were simple — the tool returned a string. MCP tools can also fail at the network level: the server is down, the connection times out, the SSE stream drops.

Spring AI surfaces these as McpException. Catch them in your agent layer:

public String chat(String message, String conversationId) {
    try {
        return chatClient.prompt()
            .user(message)
            .call()
            .content();
    } catch (McpException e) {
        log.error("[agent] MCP error: {}", e.getMessage());
        return "One of the services I rely on is temporarily unavailable. " +
               "Please try again in a moment.";
    }
}

For resilience, add health checks on the MCP server connections — if a server is unreachable at startup, the agent should log a warning and continue without those tools rather than failing to start entirely.

MCP vs local @tool — when to use each

	Local `@Tool`	MCP server
Setup	Annotate a method	Separate Spring Boot app
Deployment	Redeploy agent to change tools	Deploy MCP server independently
Language	Java only	Any language with an MCP library
Latency	Zero (same JVM)	Network hop (usually <5ms local)
Sharing	One agent	Multiple agents
Use for	Simple, agent-specific tools	Shared tools, independent teams, non-Java tools

Start with local @Tool. Move to MCP when you have tools shared across agents, tools owned by other teams, or tools written in a different language.

What's next

The agent can now call tools that live anywhere on the network. The next question is quality — how do you know the agent is actually giving correct answers? The next article covers evaluating AI agent output: building test cases, scoring responses, and catching regressions before they reach users.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Scaling Your Spring AI Agent: From One User to Many

Sham Prakash K — Fri, 23 Oct 2026 14:30:00 +0000

One user chatting with your agent is easy. Ten concurrent users is fine. A hundred concurrent users exposes every assumption you made when building for one.

The agent code itself rarely needs to change. What breaks is everything around it — conversation memory tied to a single instance, no limits on how much work one request can do, API connections opened and closed on every call, and identical questions hitting the model repeatedly when a cached answer would do.

This article covers how to design your Spring AI agent to scale horizontally: stateless services, persistent memory, connection pooling, response caching, and async request handling.

The stateless rule

The single most important principle for scaling: your application instances must be stateless.

Stateless means: any instance can handle any request. No local data that only lives in one instance's memory.

The problem with InMemoryChatMemory at scale:

User A's first message  → hits Instance 1 → history stored in Instance 1
User A's second message → hits Instance 2 → no history found → agent has no context

With a load balancer distributing requests across instances, a user's conversation can land on a different instance every time. InMemoryChatMemory breaks immediately.

The fix: move conversation history out of the JVM and into a shared store.

Persistent conversation memory with Redis

Redis is the standard choice for session data at scale — fast, simple, and supported natively by Spring.

Add the dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

Configure the connection:

spring.data.redis.host=${REDIS_HOST:localhost}
spring.data.redis.port=6379
spring.data.redis.password=${REDIS_PASSWORD:}

Implement ChatMemory backed by Redis:

@Component
public class RedisChatMemory implements ChatMemory {

    private static final String KEY_PREFIX = "chat:";
    private static final Duration TTL = Duration.ofHours(24);

    private final RedisTemplate<String, String> redis;
    private final ObjectMapper objectMapper;

    public RedisChatMemory(RedisTemplate<String, String> redis, ObjectMapper objectMapper) {
        this.redis = redis;
        this.objectMapper = objectMapper;
    }

    @Override
    public void add(String conversationId, List<Message> messages) {
        String key = KEY_PREFIX + conversationId;
        messages.forEach(msg -> {
            try {
                String json = objectMapper.writeValueAsString(
                    Map.of("role", msg.getMessageType().name(),
                           "content", msg.getContent()));
                redis.opsForList().rightPush(key, json);
            } catch (JsonProcessingException e) {
                throw new RuntimeException(e);
            }
        });
        redis.expire(key, TTL);
    }

    @Override
    public List<Message> get(String conversationId, int lastN) {
        String key = KEY_PREFIX + conversationId;
        long size = Optional.ofNullable(redis.opsForList().size(key)).orElse(0L);
        long start = Math.max(0, size - lastN);

        List<String> raw = redis.opsForList().range(key, start, -1);
        if (raw == null) return List.of();

        return raw.stream()
            .map(this::deserialize)
            .filter(Objects::nonNull)
            .toList();
    }

    @Override
    public void clear(String conversationId) {
        redis.delete(KEY_PREFIX + conversationId);
    }

    private Message deserialize(String json) {
        try {
            Map<?, ?> map = objectMapper.readValue(json, Map.class);
            String role = (String) map.get("role");
            String content = (String) map.get("content");
            return switch (role) {
                case "USER"      -> new UserMessage(content);
                case "ASSISTANT" -> new AssistantMessage(content);
                default          -> null;
            };
        } catch (JsonProcessingException e) {
            return null;
        }
    }
}

Wire it into the ChatClient:

@Bean
public ChatClient chatClient(ChatModel chatModel, RedisChatMemory redisChatMemory) {
    return ChatClient.builder(chatModel)
        .defaultAdvisors(
            new MessageChatMemoryAdvisor(redisChatMemory)
        )
        .build();
}

Now any instance can handle any conversation — history lives in Redis, not in the JVM.

Connection pooling for the model API

Every call to the model goes over HTTP. Without connection pooling, each request opens a new TCP connection, completes TLS handshake, sends the request, and closes the connection. Under load, this overhead adds up.

Spring AI uses an HTTP client under the hood. Configure it to keep connections alive and reuse them:

@Configuration
public class HttpClientConfig {

    @Bean
    public RestClient.Builder restClientBuilder() {
        HttpClient httpClient = HttpClient.newBuilder()
            .connectTimeout(Duration.ofSeconds(10))
            .executor(Executors.newVirtualThreadPerTaskExecutor())
            .build();

        ClientHttpRequestFactory factory =
            new JdkClientHttpRequestFactory(httpClient);

        return RestClient.builder()
            .requestFactory(factory);
    }
}

For higher throughput, use Apache HttpClient with explicit pool configuration:

<dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5</artifactId>
</dependency>

@Bean
public PoolingHttpClientConnectionManager connectionManager() {
    PoolingHttpClientConnectionManager manager =
        new PoolingHttpClientConnectionManager();
    manager.setMaxTotal(200);           // total connections across all routes
    manager.setDefaultMaxPerRoute(50);  // connections to the model API endpoint
    return manager;
}

Connection pooling is one of the cheapest performance wins — a few lines of config, meaningful latency reduction under load.

Caching repeated queries

Some questions get asked over and over:

"What are the top tourist attractions in Mysore?"
"What's the cheapest way to travel from Bangalore to Mysore?"

These don't change hour to hour. Calling the model every time wastes money and adds latency. Cache the response.

Spring's caching abstraction with Redis makes this straightforward:

Enable caching in your main class:

@EnableCaching
@SpringBootApplication
public class Application { ... }

Configure the cache manager:

@Bean
public RedisCacheManager cacheManager(RedisConnectionFactory factory) {
    RedisCacheConfiguration config = RedisCacheConfiguration.defaultCacheConfig()
        .entryTtl(Duration.ofMinutes(30))
        .serializeValuesWith(
            RedisSerializationContext.SerializationPair
                .fromSerializer(new GenericJackson2JsonRedisSerializer()));

    return RedisCacheManager.builder(factory)
        .cacheDefaults(config)
        .build();
}

Add @Cacheable to tool methods that return stable data:

@Cacheable(value = "attractions", key = "#city")
@Tool(description = "Get the top tourist attractions in a city.")
public String getAttractions(@ToolParam(description = "City name") String city) {
    return attractionApi.fetch(city);
}

@Cacheable(value = "weather", key = "#city")
@Tool(description = "Get the current weather for a city.")
public String getWeather(@ToolParam(description = "City name") String city) {
    return weatherApi.fetch(city);
}

The first call for "Mysore" hits the API and stores the result. The next 100 calls for "Mysore" return from Redis instantly — no model call, no tool API call.

What to cache and what not to:

Cache: attraction lists, route options, general information that changes slowly
Do not cache: user-specific data, prices that change frequently, anything time-sensitive

Async request handling

By default, Spring MVC handles each request on a thread from a thread pool. A 5-second model call holds that thread for 5 seconds. With 100 concurrent users and a 20-thread pool, requests queue up fast.

Switch agent calls to async processing to free threads while waiting for the model:

@PostMapping("/chat")
public CompletableFuture<ResponseEntity<String>> chat(@RequestBody ChatRequest request) {
    return CompletableFuture
        .supplyAsync(() -> callAgent(request), agentExecutor)
        .orTimeout(30, TimeUnit.SECONDS)
        .thenApply(ResponseEntity::ok)
        .exceptionally(e -> {
            if (e.getCause() instanceof TimeoutException) {
                return ResponseEntity.status(504)
                    .body("Request timed out.");
            }
            return ResponseEntity.status(500)
                .body("Something went wrong.");
        });
}

Configure a dedicated thread pool for agent calls:

@Bean("agentExecutor")
public Executor agentExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(20);
    executor.setMaxPoolSize(50);
    executor.setQueueCapacity(100);
    executor.setThreadNamePrefix("agent-");
    executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
    executor.initialize();
    return executor;
}

With Java 21+, virtual threads handle this even more cleanly:

spring.threads.virtual.enabled=true

One line in properties — Spring Boot automatically uses virtual threads for all servlet request handling. Each request gets its own lightweight thread, and blocking on the model call costs almost nothing.

Rate limiting at the application layer

Under load, a single slow user can consume a disproportionate share of your model API quota. Enforce per-user limits at the application layer before requests reach the model.

Using Bucket4j for token bucket rate limiting:

<dependency>
    <groupId>com.github.vladimir-bukhtoyarov</groupId>
    <artifactId>bucket4j-core</artifactId>
    <version>8.10.1</version>
</dependency>

@Component
public class UserRateLimiter {

    private final Map<String, Bucket> buckets = new ConcurrentHashMap<>();

    private Bucket bucketFor(String userId) {
        return buckets.computeIfAbsent(userId, k ->
            Bucket.builder()
                .addLimit(Bandwidth.classic(10, Refill.greedy(10, Duration.ofMinutes(1))))
                .build()
        );
    }

    public boolean tryConsume(String userId) {
        return bucketFor(userId).tryConsume(1);
    }
}

Check in the controller:

if (!userRateLimiter.tryConsume(request.userId())) {
    return ResponseEntity.status(429)
        .body("You've sent too many messages. Please wait a moment.");
}

Ten requests per minute per user is a reasonable starting limit for an agent that makes multiple tool calls per request.

What to monitor under load

Once traffic scales up, these metrics tell you where the system is struggling:

Latency percentiles — not averages

Average latency hides tail problems. Watch p95 and p99 (the slowest 5% and 1% of requests). If p99 is 30 seconds and p50 is 2 seconds, a small number of users are timing out while most are fine.

@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
    return new TimedAspect(registry);
}

@Timed(value = "agent.call", percentiles = {0.5, 0.95, 0.99})
public String callAgent(String message, String conversationId) {
    // ...
}

Thread pool utilisation

If your agent thread pool is at capacity, requests queue. Watch agent.pool.active vs agent.pool.size — if active consistently approaches max, increase pool size or reduce per-request latency.

Redis hit rate for cached tools

If your cache hit rate is low, either the cache TTL is too short, keys are not being reused across users, or the queries are naturally unique. A hit rate below 20% for stable data suggests a caching problem.

Token usage per request over time

If average token count per request is growing, conversation histories are getting longer or system prompts changed. Unbounded growth here becomes a cost problem.

A scaling checklist

Before putting your agent behind a load balancer:

[ ] Conversation history is stored in Redis or a database, not in-memory
[ ] Application instances have no local state that affects responses
[ ] HTTP connection pooling is configured for the model API client
[ ] Stable tool results are cached with an appropriate TTL
[ ] Async request handling is in place (virtual threads or dedicated pool)
[ ] Per-user rate limiting is enforced before reaching the model
[ ] p95 and p99 latency are being tracked
[ ] Thread pool utilisation is monitored

What's next

The agent scales horizontally. The next step is expanding what it can do — connecting it to external systems through the Model Context Protocol (MCP), so tools don't have to live inside your Spring Boot application.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

From Localhost to Production: Deploying Your Spring AI Agent

Sham Prakash K — Wed, 21 Oct 2026 14:30:00 +0000

Running an AI agent on your laptop is one thing. Putting it in front of real users is another. The agent itself doesn't change — but everything around it does.

API keys need to stay out of your code. Token costs add up fast if you're not watching. A slow model call that was fine during testing becomes a timeout in production. And when something goes wrong, you need logs that actually tell you what happened.

This article covers the practical side of shipping a Spring AI agent: secrets management, cost awareness, rate limiting, timeouts, and what to monitor once you're live.

The API key problem

During development, it's tempting to put your API key directly in application.properties:

spring.ai.openai.api-key=sk-proj-abc123...

Don't do this. That file gets committed to version control. The key leaks. You get a bill.

The right pattern: read the key from an environment variable.

spring.ai.openai.api-key=${OPENAI_API_KEY}

Now start the application with the key set in the environment:

export OPENAI_API_KEY=sk-proj-abc123...
java -jar your-app.jar

On a cloud platform (Railway, Render, Fly.io, AWS), set this as a secret environment variable in the deployment settings — never in a config file that gets pushed.

What about local development? Use a .env file that is listed in .gitignore, and load it with a tool like direnv or just set the variable in your shell profile. Never commit .env.

Separating configuration by environment

Your app needs different settings in development vs production:

Dev: verbose logging, low timeouts, a small token limit so you don't burn through a quota while testing
Prod: tighter timeouts, higher token limits, structured logging

Spring Boot's profile system handles this cleanly.

src/main/resources/application.properties — shared defaults:

spring.ai.openai.api-key=${OPENAI_API_KEY}
app.agent.timeout-seconds=30
app.agent.max-tokens=2000

src/main/resources/application-dev.properties — development overrides:

logging.level.com.example=DEBUG
app.agent.timeout-seconds=60
app.agent.max-tokens=500

src/main/resources/application-prod.properties — production overrides:

logging.level.com.example=INFO
app.agent.timeout-seconds=20
app.agent.max-tokens=4000

Activate a profile at startup:

java -jar your-app.jar --spring.profiles.active=prod

Your agent code reads these values without knowing which profile is active:

@Value("${app.agent.timeout-seconds}")
private int timeoutSeconds;

@Value("${app.agent.max-tokens}")
private int maxTokens;

Timeouts — set them everywhere

In development, a slow model call is annoying. In production, it blocks a thread, holds a connection, and can cascade into a full outage.

Set a timeout at every layer.

On the ChatClient call:

String response = CompletableFuture
    .supplyAsync(() -> chatClient.prompt()
        .user(request.message())
        .tools(travelTools)
        .call()
        .content())
    .get(timeoutSeconds, TimeUnit.SECONDS);

On the HTTP client Spring AI uses internally, you can configure it via properties:

spring.ai.openai.chat.options.timeout=20000

On your REST endpoint with Spring MVC:

@PostMapping("/chat")
@ResponseTimeout(20000)
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    // ...
}

Pick one consistent timeout value and apply it everywhere. Thirty seconds is a reasonable ceiling for most agent calls. If your agent needs longer, it's probably doing too much in a single request — consider breaking it into smaller steps.

Token costs — what you're actually paying for

Every call to the model costs money based on tokens — roughly three-quarters of a word each. The bill comes from two places:

Input tokens: your system prompt + conversation history + the user's message + tool definitions
Output tokens: the model's response

Output tokens cost more than input tokens. A verbose system prompt that runs 500 tokens fires on every single request. Multiply by 10,000 users and it becomes significant.

Where token costs hide:

Source	What it contains	What to watch
System prompt	Your instructions	Keep it concise — every word fires on every call
Tool definitions	@tool descriptions	Long descriptions add up across many tools
Conversation history	All prior messages	Window size directly controls cost
Tool results	What tools return	Verbose JSON returns cost more than plain strings

How to track usage in Spring AI:

Every ChatResponse contains usage metadata:

ChatResponse chatResponse = chatClient.prompt()
    .user(request.message())
    .call()
    .chatResponse();

Usage usage = chatResponse.getMetadata().getUsage();
log.info("tokens — input: {}, output: {}, total: {}",
    usage.getPromptTokens(),
    usage.getGenerationTokens(),
    usage.getTotalTokens());

Log this on every production request. After a few days you'll see exactly where cost is coming from and whether it's growing as expected.

Rate limiting — protect yourself and the API

The model provider's API has rate limits — requests per minute and tokens per minute. Hit them and your app gets 429 errors.

Two problems to solve:

1. Protect against a single user hammering the API:

Use a simple in-memory rate limiter per user (or per IP if you don't have user identity):

@Component
public class RateLimiter {

    private final Map<String, AtomicInteger> callCount = new ConcurrentHashMap<>();
    private final Map<String, Long> windowStart = new ConcurrentHashMap<>();
    private static final int MAX_CALLS_PER_MINUTE = 10;

    public boolean allow(String userId) {
        long now = System.currentTimeMillis();
        long start = windowStart.computeIfAbsent(userId, k -> now);

        if (now - start > 60_000) {
            callCount.put(userId, new AtomicInteger(0));
            windowStart.put(userId, now);
        }

        return callCount.computeIfAbsent(userId, k -> new AtomicInteger(0))
                        .incrementAndGet() <= MAX_CALLS_PER_MINUTE;
    }
}

Check it in your controller:

if (!rateLimiter.allow(request.userId())) {
    return ResponseEntity.status(429)
        .body("Too many requests. Please wait a moment.");
}

2. Handle 429s from the provider gracefully:

try {
    return chatClient.prompt().user(message).call().content();
} catch (OpenAiHttpException e) {
    if (e.statusCode() == 429) {
        return "The service is busy right now. Please try again in a moment.";
    }
    throw e;
}

Never let a raw 429 surface to the user as a 500 error.

Retry with backoff — for transient failures

Model APIs have transient failures — brief network blips, occasional 500s. A single retry with a short wait handles most of them.

Spring Retry makes this straightforward:

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>

Enable it in your main class:

@EnableRetry
@SpringBootApplication
public class Application { ... }

Add retry to your agent call:

@Retryable(
    retryFor = { OpenAiHttpException.class },
    maxAttempts = 3,
    backoff = @Backoff(delay = 1000, multiplier = 2)
)
public String callAgent(String message) {
    return chatClient.prompt().user(message).call().content();
}

This retries up to 3 times, waiting 1s then 2s between attempts. It only retries OpenAiHttpException — not your own application errors.

Do not retry on 429 errors with short delays — you'll just accelerate into the rate limit. Retry on 500s and network errors only.

What to log in production

Good production logs answer: "What did the agent do, for whom, and how did it go?"

Minimum log fields on every agent request:

@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    String requestId = UUID.randomUUID().toString();
    long start = System.currentTimeMillis();

    log.info("[req:{}] agent call — userId={} conversationId={}",
        requestId, request.userId(), request.conversationId());

    try {
        String response = callAgent(request.message(), request.conversationId());
        long durationMs = System.currentTimeMillis() - start;

        log.info("[req:{}] agent complete — durationMs={} responseLength={}",
            requestId, durationMs, response.length());

        return ResponseEntity.ok(response);
    } catch (TimeoutException e) {
        log.warn("[req:{}] agent timeout after {}ms", requestId, timeoutSeconds * 1000);
        return ResponseEntity.status(504).body("Request timed out.");
    } catch (Exception e) {
        log.error("[req:{}] agent error — {}", requestId, e.getMessage(), e);
        return ResponseEntity.status(500).body("Something went wrong.");
    }
}

The requestId ties every log line for a single request together. The durationMs tells you if calls are getting slower over time. The error path always logs the full stack trace.

Health checks and readiness

Before a request reaches your agent, you want to know the application is healthy. Spring Boot Actuator gives you this for free:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

management.endpoints.web.exposure.include=health,info
management.endpoint.health.show-details=when-authorized

GET /actuator/health returns {"status":"UP"} when the app is running. Your deployment platform (Render, Railway, Kubernetes) pings this endpoint — if it returns anything other than UP, traffic stops going to that instance.

Add a custom health check that verifies your model API is reachable:

@Component
public class ModelApiHealthIndicator implements HealthIndicator {

    private final ChatModel chatModel;

    public ModelApiHealthIndicator(ChatModel chatModel) {
        this.chatModel = chatModel;
    }

    @Override
    public Health health() {
        try {
            chatModel.call(new Prompt("ping"));
            return Health.up().build();
        } catch (Exception e) {
            return Health.down()
                .withDetail("reason", e.getMessage())
                .build();
        }
    }
}

If the model API is down, your health check returns DOWN and the platform can route around the broken instance or alert you.

A deployment checklist

Before you ship:

Secrets

[ ] API key is in an environment variable, not in any committed file
[ ] .env is in .gitignore
[ ] No secrets appear in log output

Reliability

[ ] Every agent call has a timeout
[ ] 429 errors are caught and returned as readable messages
[ ] Transient errors have retry with backoff

Cost

[ ] Token usage is logged on every request
[ ] System prompt is as concise as possible
[ ] Conversation history window is bounded (chatHistoryWindowSize)

Observability

[ ] Every request gets a unique request ID
[ ] Duration is logged on every request
[ ] Errors log the full stack trace

Operations

[ ] Health check endpoint is configured
[ ] Logs go somewhere you can actually read them (not just stdout on a server you can't SSH into)

What's next

The agent is deployed and handling real traffic. The next question is scale — what happens when many users hit the agent at the same time, when conversation histories grow large, and when you need to update the model or tools without downtime.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Building Agents That Don't Break — Error Handling and Patterns in Spring AI

Sham Prakash K — Mon, 19 Oct 2026 14:30:00 +0000

A basic agent works well when tools succeed, inputs are valid, and the task is straightforward. Real usage is messier. Tools time out. The model calls a tool with an argument it can't handle. A query returns no results. The agent loops without making progress.

This article covers how to design agents that handle these situations cleanly — patterns for planning, error recovery, structured output, observability, and preventing runaway behaviour.

What happens when a tool fails

When a tool throws an exception in Spring AI, the agent loop breaks. The model never gets a chance to handle the failure — it just sees an error response or nothing at all.

The better pattern: catch failures inside the tool and return them as readable strings.

@Tool(description = "Get the current weather for a city in India.")
public String getWeather(@ToolParam(description = "City name") String city) {
    try {
        return weatherApi.fetch(city);
    } catch (CityNotFoundException e) {
        return "Weather data not available for '" + city + "'. " +
               "This city may not be supported. Try a nearby major city.";
    } catch (ApiTimeoutException e) {
        return "Weather service is temporarily unavailable. " +
               "Please try again or skip weather for this trip.";
    }
}

When the model receives a failure message like this, it can respond intelligently: suggest an alternative city, skip that part of the answer, or tell the user the service is down. If the tool throws an uncaught exception, the model gets no information and the agent fails silently.

Rule: tools should never throw. They should return.

Returning partial results

When a task requires multiple tool calls and one fails, the agent doesn't have to fail entirely. Return what you have.

@Tool(description = "Get the cheapest ticket price between two cities.")
public String getTicketPrice(
        @ToolParam(description = "Departure city") String from,
        @ToolParam(description = "Destination city") String to) {
    try {
        String price = ticketApi.cheapest(from, to);
        return "Cheapest ticket from " + from + " to " + to + ": " + price;
    } catch (NoRouteException e) {
        return "No direct bus or train route found between " + from + " and " + to + ". " +
               "Consider checking flights or breaking the journey.";
    }
}

The model receives "no direct route found" and continues building the rest of the trip plan — weather, attractions, hotels — without the ticket price. A partial, honest answer is far better than a broken agent.

Validating arguments inside tools

The model constructs tool arguments from natural language. It usually gets them right, but edge cases slip through — wrong formats, unsupported values, empty strings.

Validate at the start of every tool:

@Tool(description = "Get flight prices for a date. Date must be in YYYY-MM-DD format.")
public String getFlightPrice(
        @ToolParam(description = "Departure city") String from,
        @ToolParam(description = "Destination city") String to,
        @ToolParam(description = "Travel date in YYYY-MM-DD format, e.g. 2026-11-15") String date) {

    if (from == null || from.isBlank()) {
        return "Error: departure city is required.";
    }
    if (!date.matches("\\d{4}-\\d{2}-\\d{2}")) {
        return "Error: date must be in YYYY-MM-DD format (e.g. 2026-11-15). Received: " + date;
    }

    return flightApi.search(from, to, date);
}

Returning the invalid input in the error message is deliberate — the model sees what it passed, recognises the format mismatch, and either corrects it on a retry or tells the user what's missing.

Planning before acting

For complex tasks — multi-city trips, research tasks, anything with many steps — prompting the model to plan before calling tools improves output quality significantly.

ChatClient.builder(chatModel)
    .defaultSystemPrompt("""
        You are a travel planning assistant.

        When given a complex request, first outline the steps you will take,
        then call the tools in that order.

        Do not call multiple tools simultaneously for a multi-step plan —
        work through the steps in sequence so each result can inform the next.

        If a tool returns an error or no data, note it in your answer
        and continue with the remaining steps.
        """)
    .build();

"First outline the steps, then call tools in that order" encourages the model to reason about the full task before starting. Without this, the model sometimes starts calling tools immediately, misses dependencies between steps, and produces a less coherent result.

Guarding against infinite loops

An agent that keeps calling tools without producing a final answer is a runaway agent. This can happen when:

Tools keep returning errors and the model keeps retrying
The model interprets each tool result as requiring another tool call
A bug in tool logic returns something that always looks like "incomplete"

Spring AI's ChatClient doesn't impose a hard limit on tool call iterations by default. You can wrap the agent call with a timeout:

@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    try {
        String response = CompletableFuture
            .supplyAsync(() -> runAgent(request))
            .get(30, TimeUnit.SECONDS);
        return ResponseEntity.ok(response);
    } catch (TimeoutException e) {
        return ResponseEntity.status(504)
            .body("The agent took too long to respond. Please try a simpler question.");
    }
}

You can also add a counter inside a custom advisor or tool execution wrapper that aborts after N tool calls. Thirty seconds and ten tool calls per request are reasonable starting limits.

Sequential vs parallel tool execution

The default behaviour: the model calls tools in whatever order and parallelism it determines. For most tasks, this is correct. But some tasks have dependencies — the result of one tool call should inform the next.

Parallel is correct when tools are independent:

"What's the weather in Mysore, and what are the top attractions?"

Weather and attractions don't depend on each other. The model will correctly issue both calls simultaneously.

Sequential is correct when steps have dependencies:

"Find the cheapest travel option from Bangalore to my destination, then find hotels near the bus station."

The hotel search depends on knowing which bus station — which depends on the travel result. The model needs the first result before making the second call.

For dependency-heavy tasks, make this explicit in the system prompt: "Work through steps one at a time — each step may depend on the previous result."

Structured output

By default, the agent returns its final answer as a plain string. That works fine for chat — but when the response needs to feed into another system, be stored in a database, or be rendered as a structured UI component, plain text breaks down fast.

Spring AI's BeanOutputConverter lets you tell the model to return JSON matching a specific Java record. The converter injects formatting instructions into the system prompt automatically and parses the response back into a typed object.

Define the output shape:

public record TripPlan(
    String destination,
    String weather,
    List<String> attractions,
    String cheapestTransport,
    int estimatedCostInr
) {}

Use the converter in your agent call:

@PostMapping("/plan")
public TripPlan plan(@RequestBody ChatRequest request) {
    BeanOutputConverter<TripPlan> converter =
        new BeanOutputConverter<>(TripPlan.class);

    String response = chatClient
        .prompt()
        .user(request.message() + "\n\n" + converter.getFormat())
        .tools(travelTools)
        .call()
        .content();

    return converter.convert(response);
}

converter.getFormat() appends a JSON schema instruction to the user message — something like "Respond with a JSON object matching this schema: {...}". The model fills in the values; converter.convert() parses the JSON into your TripPlan record.

The output is now typed Java — you can store it, return it as an API response, or pass it downstream.

One thing to watch: if the model returns slightly malformed JSON (which happens, especially with complex schemas), convert() will throw. Wrap it in a try-catch and fall back to the raw string response when parsing fails. Keep output schemas flat and simple — deeply nested structures increase the chance of malformed output.

Observability — knowing what the agent actually did

In production, a user reports a wrong answer. Your first question is: which tools fired, with what arguments, and what did they return?

Without logging at the tool level, that question has no answer. You're left guessing.

The simplest approach: log at the start and end of every tool method.

@Tool(description = "Get the current weather for a city in India.")
public String getWeather(@ToolParam(description = "City name") String city) {
    log.info("[tool:getWeather] called with city='{}'", city);

    String result = weatherApi.fetch(city);

    log.info("[tool:getWeather] returned: {}", result);
    return result;
}

This gives you a complete trace of every agent execution in your logs — which tools ran, what the model passed, and what they returned. When an answer is wrong, you trace the conversation ID to the logs and see exactly what the agent did.

For a cleaner approach, Spring AI's Advisor interface lets you intercept every tool call centrally without touching each tool method:

@Component
public class ToolLoggingAdvisor implements RequestResponseAdvisor {

    private static final Logger log = LoggerFactory.getLogger(ToolLoggingAdvisor.class);

    @Override
    public AdvisedRequest adviseRequest(AdvisedRequest request, Map<String, Object> context) {
        log.info("[agent] incoming message: {}", request.userText());
        return request;
    }

    @Override
    public ChatResponse adviseResponse(ChatResponse response, Map<String, Object> context) {
        log.info("[agent] final response: {}",
            response.getResult().getOutput().getContent());
        return response;
    }
}

ChatClient.builder(chatModel)
    .defaultAdvisors(
        new ToolLoggingAdvisor(),
        new MessageChatMemoryAdvisor(new InMemoryChatMemory()),
        new QuestionAnswerAdvisor(vectorStore, SearchRequest.defaults())
    )
    .build();

The advisor logs every request before it reaches the model and every response before it's returned to the caller. Combined with per-tool logging, you have full visibility into what the agent did on every request — without modifying controller or service code.

What to log at minimum:

Conversation ID on every request
Each tool name and the arguments the model passed
Each tool's return value (or error)
Total time taken for the agent call

Streaming agent responses

For long-running agent calls — multiple tool calls, slow APIs — streaming lets users see progress rather than waiting for a blank screen.

Spring AI supports streaming with tool use:

@PostMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> streamChat(@RequestBody ChatRequest request) {
    return chatClient
        .prompt()
        .user(request.message())
        .tools(travelTools)
        .advisors(a -> a.param(
            AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY,
            request.conversationId()
        ))
        .stream()
        .content();
}

.stream().content() returns a Flux<String> that emits tokens as the model generates them. Tool calls still happen synchronously inside the loop — the model pauses streaming while a tool executes, then resumes streaming the result.

The user experience: they see text appearing token by token, with brief pauses when tools are called. This is significantly better than staring at a loading indicator for five seconds.

A checklist before shipping an agent

Before putting an agent in front of real users, run through these:

Tools all return strings, never throw. Exceptions break the loop invisibly.

Every tool has a "use this when..." clause. Without it, the model skips tools or uses the wrong one.

You've tested the "nothing found" case. Ask something the tools can't answer. The agent should say so honestly, not hallucinate.

You've set a timeout. An unbounded agent call is a denial-of-service risk.

You've tested with bad inputs. What happens when the user sends "???" or a very long message? The agent should degrade gracefully.

The system prompt includes a failure instruction. "If a tool returns an error, note it and continue" prevents the model from getting stuck retrying a broken tool.

Structured output has a fallback. If BeanOutputConverter.convert() throws on malformed JSON, catch it and return the raw string rather than crashing.

Tool calls are logged. Conversation ID, tool name, arguments, and return value — without this, debugging a wrong answer in production is guesswork.

What's next

The articles so far have covered building agents locally. The next phase is getting them in front of users — deploying to production, managing API keys, handling costs, and monitoring what the agent actually does in the wild.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Giving Your Spring AI Agent a Memory — Conversations That Actually Make Sense

Sham Prakash K — Sat, 17 Oct 2026 14:30:00 +0000

Send two messages to a basic agent:

"My budget for the trip is ₹5,000."
"What can I do within that budget?"

The second message makes no sense without the first. If the agent has no memory, it treats every message as a fresh conversation with no history. It doesn't know what trip. It doesn't know what budget. It has to ask again — or worse, it guesses.

Memory is what turns a stateless request-response system into an agent that can hold a real conversation.

What memory means in this context

Memory in an AI agent means one thing: the conversation history that gets sent to the model on each request.

The model itself has no persistent state. Between requests, it remembers nothing. What creates the illusion of memory is appending previous messages to each new request, so the model always sees the full conversation up to that point.

Request 1:  [user: "My budget is ₹5,000"]
            ↓ model responds
            [assistant: "Got it, ₹5,000 budget."]

Request 2:  [user: "My budget is ₹5,000"]        ← history replayed
            [assistant: "Got it, ₹5,000 budget."] ← history replayed
            [user: "What can I do within that budget?"]  ← new message
            ↓ model responds with full context

Your application stores the conversation history. On every new request, it prepends the stored history before sending to the model. That's the entire mechanism.

MessageChatMemoryAdvisor in Spring AI

Spring AI handles conversation history automatically through an advisor called MessageChatMemoryAdvisor. You saw it briefly in the RAG article — here's what it actually does.

@Bean
public ChatClient chatClient(ChatModel chatModel) {
    return ChatClient.builder(chatModel)
        .defaultAdvisors(
            new MessageChatMemoryAdvisor(new InMemoryChatMemory())
        )
        .build();
}

InMemoryChatMemory is a simple in-process store — a Map from conversation ID to message list. Before every request, the advisor loads the history for the current conversation ID and prepends it to the messages. After every request, it appends the new user message and the model's response to the history.

You don't write any of this yourself. The advisor handles it.

Conversation IDs — one per user session

Memory is per-conversation, not global. Each user gets their own conversation history. You identify conversations with a string ID — typically a UUID generated when the session starts.

@PostMapping("/chat")
public String chat(@RequestBody ChatRequest request) {
    return chatClient
        .prompt()
        .user(request.message())
        .advisors(a -> a.param(
            AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY,
            request.conversationId()
        ))
        .call()
        .content();
}

public record ChatRequest(String message, String conversationId) {}

The client generates a conversationId once (e.g. UUID.randomUUID().toString()) and sends it with every message in a session. The advisor uses it to look up and update the right history.

Two users with different conversation IDs get completely separate histories. Same agent, isolated memory.

What gets stored in history

The history isn't just user messages and assistant responses. In an agent with tools, the conversation history also includes tool calls and their results.

A typical history after one tool-calling turn looks like this:

[user]      "What's the weather in Chennai?"
[assistant] → tool_call: getWeather("Chennai")
[tool]      "Chennai: 34°C, humid, partly cloudy."
[assistant] "It's 34°C and humid in Chennai right now with some cloud cover."

All four messages are stored and replayed on the next turn. This is what allows the agent to remember not just what it said, but what it did — which tools it called, what they returned, and what conclusion it drew.

If a user follows up: "Is that hotter than usual for this time of year?" — the model has the tool result in context and can reason about it without calling getWeather again.

The context window limit

Conversation history grows with every turn. The model has a context window — a maximum number of tokens it can process in a single request. Pack enough history into a conversation and you hit that limit.

When history overflows the context window, one of a few things happens depending on how your application handles it:

The request fails with a token limit error
The oldest messages get silently truncated (you lose early context)
The model gets confused because the history appears to start mid-conversation

Spring AI's MessageChatMemoryAdvisor accepts a chatHistoryWindowSize parameter that limits how many previous messages to include:

new MessageChatMemoryAdvisor(
    new InMemoryChatMemory(),
    AbstractChatMemoryAdvisor.DEFAULT_CHAT_MEMORY_CONVERSATION_ID,
    20  // keep last 20 messages
)

With a window of 20, the advisor includes only the 20 most recent messages in each request. Older messages are still stored — the window just controls what gets sent to the model.

What window size to use? Start with 20–30 messages. A message is typically 50–200 tokens; 20 messages is roughly 1,000–4,000 tokens of history, leaving plenty of room for the system prompt, tool definitions, and the current response.

In-memory vs persistent memory

InMemoryChatMemory stores history in a Map inside your running application. It's fast and requires no infrastructure. The trade-off: it disappears when the application restarts.

For a development server or a demo, in-memory is fine. For production, you need persistent storage.

Spring AI provides a CassandraChatMemory and a Neo4jChatMemory, and the ChatMemory interface is straightforward to implement against any database:

public class PostgresChatMemory implements ChatMemory {

    private final JdbcTemplate jdbc;

    @Override
    public void add(String conversationId, List<Message> messages) {
        messages.forEach(msg ->
            jdbc.update(
                "INSERT INTO chat_history (conversation_id, role, content, created_at) VALUES (?,?,?,?)",
                conversationId, msg.getMessageType().name(),
                msg.getContent(), Instant.now()
            )
        );
    }

    @Override
    public List<Message> get(String conversationId, int lastN) {
        return jdbc.query(
            "SELECT role, content FROM chat_history " +
            "WHERE conversation_id = ? ORDER BY created_at DESC LIMIT ?",
            (rs, i) -> toMessage(rs.getString("role"), rs.getString("content")),
            conversationId, lastN
        );
    }

    @Override
    public void clear(String conversationId) {
        jdbc.update("DELETE FROM chat_history WHERE conversation_id = ?", conversationId);
    }
}

Swap new InMemoryChatMemory() for new PostgresChatMemory(jdbc) and history survives restarts, scales across multiple instances, and can be queried or audited.

Clearing history

When a user starts a new topic or wants a clean slate, clear the history:

chatMemory.clear(conversationId);

Without explicit clearing, history accumulates indefinitely (or until the window drops the oldest messages). For session-based use cases, clear on logout or on session expiry.

Memory and RAG together

When both MessageChatMemoryAdvisor and QuestionAnswerAdvisor are registered, the order matters:

ChatClient.builder(chatModel)
    .defaultAdvisors(
        new MessageChatMemoryAdvisor(new InMemoryChatMemory()),  // runs first
        new QuestionAnswerAdvisor(vectorStore,
            SearchRequest.defaults().withTopK(5))               // runs second
    )
    .build();

Memory runs first so that follow-up questions like "tell me more about that" contain the previous turn's context before RAG tries to retrieve relevant chunks. With the order reversed, RAG searches based on an incomplete message that has no conversational context.

What memory doesn't solve

Memory gives the model access to recent conversation history. It doesn't give the model access to:

Facts from much earlier in a long conversation. Once messages scroll past the history window, they're gone from the model's view. If a user mentioned their budget 40 messages ago and the window is 20, the model won't see it.

Knowledge from other conversations. Each conversation ID has its own isolated history. An insight the model developed in one session isn't available in another.

Structured data. History is a list of text messages. If you need to remember structured facts — the user's name, preferences, profile — store them in your database and inject them via the system prompt rather than relying on them being in conversation history.

What's next

Memory keeps context across turns. But there's another dimension to agent design: what happens when a task is too complex to complete in one shot — when the agent needs to break a goal into sub-tasks, work through them over multiple steps, and adapt its plan when something doesn't work?

The next article covers structured agent patterns — how to design agents that plan, delegate, and recover from failures.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Designing Tools Your AI Agent Will Actually Use Correctly

Sham Prakash K — Thu, 15 Oct 2026 14:30:00 +0000

The previous article showed how to wire tools into a Spring AI agent. The mechanics are straightforward — annotate a method, pass it to the ChatClient, done. What's less obvious is that the quality of your tools determines whether the agent works correctly.

A poorly described tool gets called at the wrong time, with the wrong arguments, or not at all. A well-designed tool gets called exactly when it should, with exactly the right input.

The description is an instruction to the model

When you write @Tool(description = "..."), that string isn't documentation for your team. It's a runtime instruction that the model reads before every single request.

The model's decision process looks like this:

What did the user ask?
What tools do I have available? (reads each description)
Do I need a tool to answer this? Which one?
What arguments should I pass? (reads each @ToolParam description)

If your description is vague, the model makes a guess. Sometimes the guess is right. Often it isn't — and because it's a language model, it won't tell you it was guessing. It will just proceed with what seemed most reasonable.

Vague description:

@Tool(description = "Get weather")
public String getWeather(String city) { ... }

The model doesn't know: Is this current weather or a forecast? What does it return — temperature? conditions? Both? What format is the city name expected in?

Clear description:

@Tool(description = "Get the current weather for a city in India. " +
    "Returns temperature in Celsius and sky conditions (sunny, cloudy, rainy). " +
    "Use this when the user asks about current weather or whether to carry an umbrella.")
public String getWeather(
        @ToolParam(description = "City name in English, e.g. Mumbai, Chennai, Bengaluru") 
        String city) { ... }

Now the model knows exactly when to call it, what it returns, and how to pass the city name. Same method, entirely different instruction.

What to include in a tool description

A useful tool description answers four questions:

What does it do? State the action clearly. "Fetches", "returns", "calculates", "queries" — be specific about the operation.

When should the model use it? This is the most commonly missed part. Add a "use this when..." clause. Without it, the model may skip the tool in cases where it should call it, because it isn't sure the tool applies.

What does it return? Describe the shape of the output. If it returns a price, say what currency. If it returns a date, say what format. If it returns a list, say roughly how many items.

What are the constraints? If the tool only works for certain inputs — cities in India, dates in the future, IDs that exist in the database — say that. The model will avoid calling it in unsupported cases, which prevents unnecessary tool failures.

Example applying all four:

@Tool(description = "Get the cheapest available bus or train ticket price between two cities in India. " +
    "Returns the price in INR and the transport type (bus or train). " +
    "Use this when the user asks about travel cost, ticket price, or how to get between cities. " +
    "Only works for cities with bus or rail connections — does not cover flights.")
public String getTicketPrice(
        @ToolParam(description = "Departure city name in English, e.g. Bengaluru") String from,
        @ToolParam(description = "Destination city name in English, e.g. Mysore") String to) {
    // ...
}

Writing good parameter descriptions

Parameter descriptions are just as important as the tool description. The model constructs arguments from them.

Be specific about format:

// Bad — model doesn't know what format to use
@ToolParam(description = "Date") String date

// Good — model knows exactly what to pass
@ToolParam(description = "Travel date in YYYY-MM-DD format, e.g. 2026-10-15") String date

Specify units:

// Bad
@ToolParam(description = "Budget") int amount

// Good
@ToolParam(description = "Maximum budget in INR, e.g. 5000") int amountInr

Clarify expected values for enums or limited options:

@ToolParam(description = "Seat class: SLEEPER, AC_3_TIER, or AC_2_TIER") String seatClass

When the model knows the exact expected values, it passes valid ones instead of guessing.

Returning results the model can reason about

Your tool's return value is what the model reads next. It uses that text to decide what to do — call another tool, ask a clarifying question, or produce the final answer.

Don't return raw JSON:

// Bad — the model has to parse this in its head
return "{\"temp\":28,\"condition\":\"PC\",\"humidity\":85}";

The model can handle JSON, but it has to mentally parse it first. That's cognitive overhead that introduces errors. Write the return value as natural language.

Return readable sentences:

// Good — the model reads this like a human would
return "Current weather in Mumbai: 28°C, partly cloudy, humidity 85%.";

Include context in the result:

If a tool fetches a price, include what the price is for. The model may call multiple tools and combine results — it needs context in each result to do that correctly.

// Without context — confusing when combined with other results
return "₹3,240";

// With context — clear when combined
return "Flight from Bengaluru to Goa on 2026-11-15: ₹3,240 (economy, non-stop, IndiGo 6E-502).";

Common failure modes

The model answers without calling the tool

This happens when the description doesn't say when to use the tool, and the model happens to know something relevant from training data. It answers directly, skipping the tool entirely.

Fix: add "Use this when..." to the description. Add to the system prompt: "Always use available tools to get current information — do not answer from memory."

The model calls the tool with the wrong argument format

A date tool receiving "15th October 2026" instead of "2026-10-15". A city tool receiving "Blr" instead of "Bengaluru".

Fix: specify the exact format in @ToolParam. Add examples: "e.g. 2026-10-15", "e.g. Bengaluru".

The model calls the wrong tool

Two tools with similar descriptions cause the model to mix them up. A getWeather and a getForecast tool with overlapping descriptions leads to unpredictable choices.

Fix: make descriptions explicitly distinct. State what each tool does NOT do: "Returns current conditions — for forecasts, use getForecast."

The tool returns an error and the model hallucinates a result

The tool throws an exception, returns an empty string, or returns an error message the model didn't expect. The model sees this as an unhelpful result and fabricates what the result probably would have been.

Fix: return meaningful error strings, not exceptions. "Unable to fetch weather for this city — city not found." is something the model can relay honestly to the user. An uncaught exception that surfaces as an empty result is something the model will try to paper over.

How many tools is too many?

There's no hard rule, but there's a practical limit. A model given 20 tools spends more time deciding which one to use. The chance of picking the wrong tool increases. The context window fills with tool definitions.

Keep tools focused. One tool should do one thing. A tool that fetches weather AND attractions AND ticket prices is three tools masquerading as one — split it.

Name tools clearly. The tool name is part of what the model reads. getWeather, getAttractions, getTicketPrice are clear. getData, fetch, process are not.

Remove tools the agent doesn't need. If a tool exists but the current use case will never need it, don't register it. Fewer tools mean faster, more accurate decisions.

Testing your tools

Tool quality is easy to verify. Ask questions that should and shouldn't trigger each tool, then inspect whether the right tools were called with the right arguments.

Three test cases every tool should pass:

Positive case — ask something that clearly requires this tool. Verify it was called and returned a sensible result.

Negative case — ask something unrelated. Verify the tool was NOT called.

Edge case — ask something ambiguous or with unusual input. Verify the arguments the model constructed are valid.

Add logging to each tool method during development:

@Tool(description = "...")
public String getWeather(@ToolParam(description = "City name") String city) {
    log.info("getWeather called with city={}", city);
    // ...
}

The log shows you exactly what the model passed. Wrong argument format? Fix the @ToolParam description. Tool not called at all? Fix the tool description or the system prompt.

What's next

Tools are methods. The agent loop runs them in sequence or parallel. But there's a limit to what single tool calls can do — some problems require a model to plan across many steps, remember intermediate results, and course-correct when something goes wrong. That's where more structured agent patterns come in.

The next article covers memory in agents — how to give your agent context from previous turns so it can handle multi-turn conversations, remember what it learned from earlier tool calls, and build on its own previous answers.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

AI Agents Explained — Tools, the ReAct Loop, and Spring Boot

Sham Prakash K — Tue, 13 Oct 2026 14:30:00 +0000

RAG gives the model access to your documents. It retrieves, it answers — but it never does anything. If answering a question requires checking a live API, querying a database, or chaining several steps together, RAG isn't enough.

That's where agents come in.

What an agent actually is

A regular chat call is one-shot: you send a message, the model responds, done. The model produces text — and only text. It has no way to interact with the outside world.

An agent breaks that limitation.

The idea is simple: you give the model a list of tools — capabilities your application exposes — and the model can invoke them during its response. It can call a tool, read the result, decide what to do next, call another tool, and eventually produce a final answer.

The model doesn't execute the tools itself. Your code does. The model just decides when to call them and what arguments to pass.

Think of it like a senior engineer delegating work. The manager (the model) decides what needs to be done. The engineers (your tool functions) actually do the work and report back. The manager reviews the results and figures out the next step.

What is a "tool"?

In the context of AI agents, a tool is just a function in your application that the model is allowed to call.

That function can do anything a normal Java method can do: call an external API, run a database query, read a file, do a calculation, send a notification. From the model's perspective, a tool is a named capability with a description and a set of inputs. From your code's perspective, it's a plain method.

Before the model responds to a user message, it sees:

The user's question
A list of available tools (name, description, what parameters each expects)

The model decides whether it needs a tool to answer. If yes, it responds not with text but with a tool call request: "call this tool with these arguments." Your application executes the function, sends the result back, and the model continues.

The ReAct pattern — how agents reason

The ReAct pattern (Reason + Act) describes how a model behaves inside this loop:

Reason — the model thinks about what it knows, what it still needs, and which tool will help next.

Act — it calls a tool.

Observe — it receives the tool's result.

Reason again — given the new information, what should happen next? Call another tool? Combine results and answer?

Act or answer — this repeats until the model has enough to respond.

Here's a concrete example. A user asks: "What's the weather like in Mumbai, and should I carry an umbrella?"

Without tools, the model can only guess based on training data. With a weather tool wired up:

Reason: I need live weather data for Mumbai. I have a getWeather tool.
Act: Call getWeather("Mumbai")
Observe: Result: "32°C, humidity 88%, chance of rain 70%"
Reason: I now have the data. I can answer.
Answer: "Mumbai is hot and humid today with a 70% chance of rain. Yes, carry an umbrella."

The model drove that entire sequence. Your code just executed the getWeather function when asked.

The agent loop — step by step

Before looking at code, it helps to understand exactly what happens behind the scenes on every request:

1. User sends a message
2. Your app sends: message + list of available tools → model
3. Model responds with a tool call request (not final text yet)
4. Your app executes the tool, gets the result
5. Your app sends: original conversation + tool result → model
6. Model either calls another tool (back to step 3)
   or produces the final answer (done)

Step 6 is the key: the loop continues until the model decides it has enough information. One question might trigger three tool calls. Another might need none. The model decides.

In Spring AI, you don't write this loop manually. The framework handles it. You write the tools; Spring AI runs the loop and returns the final answer.

Defining tools in Spring AI

Spring AI uses a @Tool annotation to mark methods as callable by the model. Any public method in a Spring bean can become a tool.

Here's the simplest possible tool:

@Service
public class WeatherTools {

    @Tool(description = "Get the current weather for a city. Returns temperature in Celsius and a short description.")
    public String getWeather(
            @ToolParam(description = "The city name, e.g. Mumbai or Chennai") String city) {

        // In a real app, this calls a weather API
        // For now, returning dummy data
        return "32°C, partly cloudy with high humidity";
    }
}

A few things to understand here:

@Tool marks the method as a tool. The description is what the model reads to decide whether this tool is relevant to the current task. Write it as if you're explaining the tool to a new developer: what does it do, what does it return?

@ToolParam describes each parameter. The model reads this description when deciding what argument to pass. If the user says "weather in Bombay", the model needs to know whether to pass "Bombay" or "Mumbai" or a city code — your description guides that decision.

The return value is what the model sees. Whatever your method returns gets sent back to the model as the result of that tool call. Return a plain, readable string — not raw JSON that the model has to parse and interpret. Write it like you'd write a sentence explaining the result.

Wiring tools into the ChatClient

Once the tools are defined, you pass them to the ChatClient when making a request:

@RestController
@RequestMapping("/api/agent")
public class AgentController {

    private final ChatClient chatClient;
    private final WeatherTools weatherTools;

    public AgentController(ChatClient.Builder builder, WeatherTools weatherTools) {
        this.chatClient = builder.build();
        this.weatherTools = weatherTools;
    }

    @PostMapping("/chat")
    public String chat(@RequestBody ChatRequest request) {
        return chatClient
            .prompt()
            .user(request.message())
            .tools(weatherTools)       // register tools for this request
            .call()
            .content();
    }
}

.tools(weatherTools) tells Spring AI: "for this request, the model is allowed to call any @Tool method in this bean."

Spring AI automatically converts your annotated methods into the tool definition format the model expects — name, description, parameter names, types. You don't write any JSON schema by hand.

.call().content() waits for the entire agent loop to finish and returns the model's final text response. The tool calls, the loop, the result passing — all of that happens inside this one call. From your controller's perspective, it looks like a regular chat request.

Multiple tools and multi-step reasoning

An agent can have many tools. The model decides which ones to call, in what order, and how many times.

@Service
public class TravelTools {

    @Tool(description = "Get current weather for a city. Returns temperature in Celsius and sky conditions.")
    public String getWeather(@ToolParam(description = "City name") String city) {
        return "24°C, clear skies";
    }

    @Tool(description = "Get top tourist attractions for a city. Returns a list of places to visit.")
    public String getAttractions(@ToolParam(description = "City name") String city) {
        return "Mysore Palace, Chamundi Hills, Brindavan Gardens, Devaraja Market";
    }

    @Tool(description = "Get current bus or train ticket price between two cities in India. Returns price in INR.")
    public String getTicketPrice(
            @ToolParam(description = "Departure city") String from,
            @ToolParam(description = "Destination city") String to) {
        return "Bus: ₹180, Train: ₹240 (2nd class)";
    }
}

Now ask: "Plan a weekend trip from Bangalore to Mysore — weather, what to see, and how to get there."

The model has three tools and an open-ended request. It will:

Call getWeather("Mysore") → "24°C, clear skies"
Call getAttractions("Mysore") → list of places
Call getTicketPrice("Bangalore", "Mysore") → fare info
Combine all three into a readable trip plan

Three tool calls, one user message, one final answer. The model orchestrated the sequence — your code just ran the functions.

Parallel tool calling

When multiple tool calls are independent of each other — when neither result depends on the other — the model can request them all at once.

For the trip planning example: weather, attractions, and ticket price can all be fetched simultaneously. None of them depend on each other's results. A capable model will issue all three calls in a single step, and Spring AI will execute them in parallel.

You write no parallel execution code. Spring AI detects that the model returned multiple tool calls in one response, runs them concurrently, and sends all results back together. The model then continues with the full set of results.

This matters for performance. Three sequential tool calls in a slow API might take three seconds. In parallel, they take the time of the slowest one.

System prompt for agents

Just like RAG needs a grounding instruction in the system prompt, agents benefit from a clear directive about when to use tools.

ChatClient.builder(chatModel)
    .defaultSystemPrompt("""
        You are a travel assistant for India.
        Use the available tools to answer questions — always call tools for current information.
        Do not answer from memory when a tool can provide the answer.
        If the user asks about something outside travel, politely say you can only help with travel questions.
        """)
    .build();

"Always call tools for current information — do not answer from memory" prevents the model from skipping tool calls when it happens to know something from training data. Without this, the model might answer a weather question confidently from stale knowledge instead of calling your live weather tool.

When to use an agent — and when not to

Agents are powerful but have a real cost: every tool call is a round trip to your application. A three-tool agent call makes four calls to the model (initial request + one per tool result). That's slower and more expensive than a plain chat call.

The decision is straightforward:

Plain chat call — the model reads input and produces text. Summarisation, classification, translation, writing. No external data needed.

RAG — the model needs knowledge from your documents. No actions, no live data — just retrieval and response.

Agent — answering requires doing something: checking a live API, running a database query, performing a multi-step process where each step's result shapes the next.

Only reach for agents when the task genuinely requires action. An agent that just wraps a static lookup is overkill — that's a plain function call.

What's next

Tools are only as good as their descriptions. The next article covers tool design in depth — how to write descriptions the model understands, how to return results the model can reason about, and the failure modes that come from poorly designed tools.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Building a Full RAG Pipeline in Spring Boot — From Ingest to Grounded Answer

Sham Prakash K — Sun, 11 Oct 2026 14:30:00 +0000

Previous articles covered the individual pieces of RAG. You now know what embeddings are, how to chunk documents, and how Pinecone stores and searches vectors. This article connects those pieces into a working pipeline in Spring Boot — and shows exactly what happens at each step when a real question comes in.

Two flows, one shared store

Every RAG system has two separate flows:

Ingest — runs once per document. Takes raw text, splits it into chunks, converts each chunk into a vector, stores it in Pinecone.

Query — runs on every user message. Takes the question, converts it to a vector, finds the most relevant chunks, injects them into the prompt, sends the enriched prompt to the model.

These two flows are completely independent. They share exactly one thing: the vector store. The ingest flow writes to it. The query flow reads from it.

The ingest flow

Splitting and storing

@Service
public class IngestService {

    private final VectorStore vectorStore;
    private final TokenTextSplitter splitter;

    public IngestService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
        this.splitter = new TokenTextSplitter(350, 50, 10, 10000, true);
    }

    public void ingest(String sourceId, String title, String content) {
        vectorStore.delete(List.of(sourceId));

        Document doc = new Document(content, Map.of(
            "source", sourceId,
            "title",  title
        ));

        List<Document> chunks = splitter.split(doc);
        vectorStore.add(chunks);
    }
}

vectorStore.delete() runs first — it removes any previously stored chunks from this document before adding new ones. Skip this step and re-uploading a document creates two sets of chunks in Pinecone. Retrieval will silently mix answers from both versions.

vectorStore.add(chunks) does three things in one call: embeds each chunk using text-embedding-004, then stores the text, the vector, and the metadata together in Pinecone. You don't call the embedding API separately — Spring AI handles it.

Ingest endpoint

@RestController
@RequestMapping("/api/documents")
public class IngestController {

    private final IngestService ingestService;

    @PostMapping
    public ResponseEntity<String> ingest(@RequestBody IngestRequest request) {
        ingestService.ingest(request.sourceId(), request.title(), request.content());
        return ResponseEntity.ok("Ingested: " + request.sourceId());
    }
}

Try it:

curl -X POST http://localhost:8080/api/documents \
  -H "Content-Type: application/json" \
  -d '{
    "sourceId": "return-policy-v2",
    "title": "Electronics Return Policy",
    "content": "Electronics purchased at full price may be returned within 30 days with original packaging and receipt..."
  }'

The response is just "Ingested: return-policy-v2". Nothing tells you how many chunks were created or what they contain — the work happened inside vectorStore.add().

The query flow

Wiring retrieval into the chat client

QuestionAnswerAdvisor is the piece that connects the vector store to every chat request. Configure it once on the ChatClient bean and it runs automatically on every message — you never call Pinecone directly from your controller.

@Configuration
public class ChatConfig {

    @Bean
    public ChatClient chatClient(ChatModel chatModel, VectorStore vectorStore) {
        return ChatClient.builder(chatModel)
            .defaultAdvisors(
                new MessageChatMemoryAdvisor(new InMemoryChatMemory()),
                new QuestionAnswerAdvisor(
                    vectorStore,
                    SearchRequest.defaults().withTopK(5)
                )
            )
            .defaultSystemPrompt("""
                You are a helpful assistant.
                Answer ONLY based on the provided context.
                If the context does not contain enough information, say so clearly.
                Do not use knowledge from outside the provided context.
                """)
            .build();
    }
}

Two things here worth understanding deeply:

withTopK(5) tells the advisor to retrieve the 5 most similar chunks for every query. That number directly affects what the model receives — too few and the answer might be in the 6th chunk, too many and the model gets overwhelmed with context. Five is a reasonable starting point.

The system prompt instruction is the enforcement mechanism for grounding. Answer ONLY based on the provided context tells the model not to use its training knowledge to fill gaps. Without this instruction, the model will — confidently, and invisibly — supplement retrieved context with things it learned during training. That's the exact problem RAG is supposed to solve.

Chat endpoint

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatClient chatClient;

    @PostMapping
    public String chat(@RequestBody ChatRequest request) {
        return chatClient
            .prompt()
            .user(request.message())
            .advisors(a -> a.param(
                AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY,
                request.conversationId()
            ))
            .call()
            .content();
    }
}

What the model actually receives

This is the detail that makes the whole system make sense.

When a user asks "What is the return policy for electronics?", the model doesn't receive that question alone. By the time QuestionAnswerAdvisor runs, the prompt looks like this:

System:
You are a helpful assistant.
Answer ONLY based on the provided context.
If the context does not contain enough information, say so clearly.

Context information is below.
---------------------
[Chunk 1: "Electronics purchased at full price may be returned within 30 days
with original packaging and receipt. Items must be in original condition."]

[Chunk 2: "Exceptions: laptops and tablets cannot be returned after the seal
is broken unless the item is defective."]

[Chunk 3: "To initiate a return, contact customer support with your order
number. Refunds are processed within 5–7 business days."]
---------------------
Given the context information and not prior knowledge, answer the question.

User: What is the return policy for electronics?

The model reads those chunks and answers from them. It did not search your documents — you retrieved the relevant sections and placed them directly in front of it. The model's job is to read and summarise. Your job is to retrieve well.

When there is no answer in the documents

Ask the system something that isn't in any of your uploaded documents. Pinecone still returns 5 chunks — the 5 least-dissimilar ones regardless of actual relevance. The model receives those chunks, finds no useful information in them, and because of the system prompt responds: "I don't have enough information to answer that based on the available documents."

That is the correct behaviour. If the system gives a detailed, confident answer to a question that isn't in your documents, check the system prompt — the grounding instruction is either missing or being ignored.

You can also add a similarity threshold to reject low-quality retrievals outright:

SearchRequest.defaults()
    .withTopK(5)
    .withSimilarityThreshold(0.70)

Chunks with similarity below 0.70 are discarded before they reach the prompt. If nothing passes the threshold, the model gets an empty context section and says so.

Tuning retrieval quality

Once the pipeline is running, these are the variables to adjust — in order of how much impact they have:

Chunk size — the biggest lever. Vague answers that feel like summaries usually mean chunks are too large. Drop from 350 to 200–250 tokens and re-ingest. Fragmented answers that miss context usually mean chunks are too small — try going larger.

topK — if the correct answer is in your documents but the model keeps missing it, the relevant chunk might be ranked 6th or 7th. Try increasing topK to 7 or 8.

Similarity threshold — add it only after you observe irrelevant chunks appearing in the context and causing wrong answers. Starting without a threshold is fine.

Test changes by writing 10–15 representative questions, running them before and after, and comparing the answers. Intuition alone won't tell you which direction to go.

A common failure mode

There's one failure that appears after the pipeline is running and the initial tests look fine: a question comes in that the documents don't answer, and the system gives a confident, detailed response anyway.

This happens when the system prompt grounding instruction is missing or soft. During initial testing, questions tend to be things the documents do answer — so the failure is invisible. It surfaces when real users ask things outside the scope of what was uploaded.

Always test the "not in the documents" case before considering the pipeline done. Ask something the system could not possibly know. If the answer sounds specific and confident, the system prompt is not enforcing grounding.

What's next

RAG gives the model knowledge from your documents. But knowledge isn't enough for every task. Some things require the model to act — query a database, call an API, run a multi-step process.

That's what AI agents do. The next phase of this series covers how agents work, what the ReAct loop is, and how to build one in Spring Boot.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.