DEV Community: Sham Prakash K

Scaling Your Spring AI Agent: From One User to Many

Sham Prakash K — Fri, 23 Oct 2026 14:30:00 +0000

One user chatting with your agent is easy. Ten concurrent users is fine. A hundred concurrent users exposes every assumption you made when building for one.

The agent code itself rarely needs to change. What breaks is everything around it — conversation memory tied to a single instance, no limits on how much work one request can do, API connections opened and closed on every call, and identical questions hitting the model repeatedly when a cached answer would do.

This article covers how to design your Spring AI agent to scale horizontally: stateless services, persistent memory, connection pooling, response caching, and async request handling.

The stateless rule

The single most important principle for scaling: your application instances must be stateless.

Stateless means: any instance can handle any request. No local data that only lives in one instance's memory.

The problem with InMemoryChatMemory at scale:

User A's first message  → hits Instance 1 → history stored in Instance 1
User A's second message → hits Instance 2 → no history found → agent has no context

With a load balancer distributing requests across instances, a user's conversation can land on a different instance every time. InMemoryChatMemory breaks immediately.

The fix: move conversation history out of the JVM and into a shared store.

Persistent conversation memory with Redis

Redis is the standard choice for session data at scale — fast, simple, and supported natively by Spring.

Add the dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

Configure the connection:

spring.data.redis.host=${REDIS_HOST:localhost}
spring.data.redis.port=6379
spring.data.redis.password=${REDIS_PASSWORD:}

Implement ChatMemory backed by Redis:

@Component
public class RedisChatMemory implements ChatMemory {

    private static final String KEY_PREFIX = "chat:";
    private static final Duration TTL = Duration.ofHours(24);

    private final RedisTemplate<String, String> redis;
    private final ObjectMapper objectMapper;

    public RedisChatMemory(RedisTemplate<String, String> redis, ObjectMapper objectMapper) {
        this.redis = redis;
        this.objectMapper = objectMapper;
    }

    @Override
    public void add(String conversationId, List<Message> messages) {
        String key = KEY_PREFIX + conversationId;
        messages.forEach(msg -> {
            try {
                String json = objectMapper.writeValueAsString(
                    Map.of("role", msg.getMessageType().name(),
                           "content", msg.getContent()));
                redis.opsForList().rightPush(key, json);
            } catch (JsonProcessingException e) {
                throw new RuntimeException(e);
            }
        });
        redis.expire(key, TTL);
    }

    @Override
    public List<Message> get(String conversationId, int lastN) {
        String key = KEY_PREFIX + conversationId;
        long size = Optional.ofNullable(redis.opsForList().size(key)).orElse(0L);
        long start = Math.max(0, size - lastN);

        List<String> raw = redis.opsForList().range(key, start, -1);
        if (raw == null) return List.of();

        return raw.stream()
            .map(this::deserialize)
            .filter(Objects::nonNull)
            .toList();
    }

    @Override
    public void clear(String conversationId) {
        redis.delete(KEY_PREFIX + conversationId);
    }

    private Message deserialize(String json) {
        try {
            Map<?, ?> map = objectMapper.readValue(json, Map.class);
            String role = (String) map.get("role");
            String content = (String) map.get("content");
            return switch (role) {
                case "USER"      -> new UserMessage(content);
                case "ASSISTANT" -> new AssistantMessage(content);
                default          -> null;
            };
        } catch (JsonProcessingException e) {
            return null;
        }
    }
}

Wire it into the ChatClient:

@Bean
public ChatClient chatClient(ChatModel chatModel, RedisChatMemory redisChatMemory) {
    return ChatClient.builder(chatModel)
        .defaultAdvisors(
            new MessageChatMemoryAdvisor(redisChatMemory)
        )
        .build();
}

Now any instance can handle any conversation — history lives in Redis, not in the JVM.

Connection pooling for the model API

Every call to the model goes over HTTP. Without connection pooling, each request opens a new TCP connection, completes TLS handshake, sends the request, and closes the connection. Under load, this overhead adds up.

Spring AI uses an HTTP client under the hood. Configure it to keep connections alive and reuse them:

@Configuration
public class HttpClientConfig {

    @Bean
    public RestClient.Builder restClientBuilder() {
        HttpClient httpClient = HttpClient.newBuilder()
            .connectTimeout(Duration.ofSeconds(10))
            .executor(Executors.newVirtualThreadPerTaskExecutor())
            .build();

        ClientHttpRequestFactory factory =
            new JdkClientHttpRequestFactory(httpClient);

        return RestClient.builder()
            .requestFactory(factory);
    }
}

For higher throughput, use Apache HttpClient with explicit pool configuration:

<dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5</artifactId>
</dependency>

@Bean
public PoolingHttpClientConnectionManager connectionManager() {
    PoolingHttpClientConnectionManager manager =
        new PoolingHttpClientConnectionManager();
    manager.setMaxTotal(200);           // total connections across all routes
    manager.setDefaultMaxPerRoute(50);  // connections to the model API endpoint
    return manager;
}

Connection pooling is one of the cheapest performance wins — a few lines of config, meaningful latency reduction under load.

Caching repeated queries

Some questions get asked over and over:

"What are the top tourist attractions in Mysore?"
"What's the cheapest way to travel from Bangalore to Mysore?"

These don't change hour to hour. Calling the model every time wastes money and adds latency. Cache the response.

Spring's caching abstraction with Redis makes this straightforward:

Enable caching in your main class:

@EnableCaching
@SpringBootApplication
public class Application { ... }

Configure the cache manager:

@Bean
public RedisCacheManager cacheManager(RedisConnectionFactory factory) {
    RedisCacheConfiguration config = RedisCacheConfiguration.defaultCacheConfig()
        .entryTtl(Duration.ofMinutes(30))
        .serializeValuesWith(
            RedisSerializationContext.SerializationPair
                .fromSerializer(new GenericJackson2JsonRedisSerializer()));

    return RedisCacheManager.builder(factory)
        .cacheDefaults(config)
        .build();
}

Add @Cacheable to tool methods that return stable data:

@Cacheable(value = "attractions", key = "#city")
@Tool(description = "Get the top tourist attractions in a city.")
public String getAttractions(@ToolParam(description = "City name") String city) {
    return attractionApi.fetch(city);
}

@Cacheable(value = "weather", key = "#city")
@Tool(description = "Get the current weather for a city.")
public String getWeather(@ToolParam(description = "City name") String city) {
    return weatherApi.fetch(city);
}

The first call for "Mysore" hits the API and stores the result. The next 100 calls for "Mysore" return from Redis instantly — no model call, no tool API call.

What to cache and what not to:

Cache: attraction lists, route options, general information that changes slowly
Do not cache: user-specific data, prices that change frequently, anything time-sensitive

Async request handling

By default, Spring MVC handles each request on a thread from a thread pool. A 5-second model call holds that thread for 5 seconds. With 100 concurrent users and a 20-thread pool, requests queue up fast.

Switch agent calls to async processing to free threads while waiting for the model:

@PostMapping("/chat")
public CompletableFuture<ResponseEntity<String>> chat(@RequestBody ChatRequest request) {
    return CompletableFuture
        .supplyAsync(() -> callAgent(request), agentExecutor)
        .orTimeout(30, TimeUnit.SECONDS)
        .thenApply(ResponseEntity::ok)
        .exceptionally(e -> {
            if (e.getCause() instanceof TimeoutException) {
                return ResponseEntity.status(504)
                    .body("Request timed out.");
            }
            return ResponseEntity.status(500)
                .body("Something went wrong.");
        });
}

Configure a dedicated thread pool for agent calls:

@Bean("agentExecutor")
public Executor agentExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(20);
    executor.setMaxPoolSize(50);
    executor.setQueueCapacity(100);
    executor.setThreadNamePrefix("agent-");
    executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
    executor.initialize();
    return executor;
}

With Java 21+, virtual threads handle this even more cleanly:

spring.threads.virtual.enabled=true

One line in properties — Spring Boot automatically uses virtual threads for all servlet request handling. Each request gets its own lightweight thread, and blocking on the model call costs almost nothing.

Rate limiting at the application layer

Under load, a single slow user can consume a disproportionate share of your model API quota. Enforce per-user limits at the application layer before requests reach the model.

Using Bucket4j for token bucket rate limiting:

<dependency>
    <groupId>com.github.vladimir-bukhtoyarov</groupId>
    <artifactId>bucket4j-core</artifactId>
    <version>8.10.1</version>
</dependency>

@Component
public class UserRateLimiter {

    private final Map<String, Bucket> buckets = new ConcurrentHashMap<>();

    private Bucket bucketFor(String userId) {
        return buckets.computeIfAbsent(userId, k ->
            Bucket.builder()
                .addLimit(Bandwidth.classic(10, Refill.greedy(10, Duration.ofMinutes(1))))
                .build()
        );
    }

    public boolean tryConsume(String userId) {
        return bucketFor(userId).tryConsume(1);
    }
}

Check in the controller:

if (!userRateLimiter.tryConsume(request.userId())) {
    return ResponseEntity.status(429)
        .body("You've sent too many messages. Please wait a moment.");
}

Ten requests per minute per user is a reasonable starting limit for an agent that makes multiple tool calls per request.

What to monitor under load

Once traffic scales up, these metrics tell you where the system is struggling:

Latency percentiles — not averages

Average latency hides tail problems. Watch p95 and p99 (the slowest 5% and 1% of requests). If p99 is 30 seconds and p50 is 2 seconds, a small number of users are timing out while most are fine.

@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
    return new TimedAspect(registry);
}

@Timed(value = "agent.call", percentiles = {0.5, 0.95, 0.99})
public String callAgent(String message, String conversationId) {
    // ...
}

Thread pool utilisation

If your agent thread pool is at capacity, requests queue. Watch agent.pool.active vs agent.pool.size — if active consistently approaches max, increase pool size or reduce per-request latency.

Redis hit rate for cached tools

If your cache hit rate is low, either the cache TTL is too short, keys are not being reused across users, or the queries are naturally unique. A hit rate below 20% for stable data suggests a caching problem.

Token usage per request over time

If average token count per request is growing, conversation histories are getting longer or system prompts changed. Unbounded growth here becomes a cost problem.

A scaling checklist

Before putting your agent behind a load balancer:

[ ] Conversation history is stored in Redis or a database, not in-memory
[ ] Application instances have no local state that affects responses
[ ] HTTP connection pooling is configured for the model API client
[ ] Stable tool results are cached with an appropriate TTL
[ ] Async request handling is in place (virtual threads or dedicated pool)
[ ] Per-user rate limiting is enforced before reaching the model
[ ] p95 and p99 latency are being tracked
[ ] Thread pool utilisation is monitored

What's next

The agent scales horizontally. The next step is expanding what it can do — connecting it to external systems through the Model Context Protocol (MCP), so tools don't have to live inside your Spring Boot application.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

From Localhost to Production: Deploying Your Spring AI Agent

Sham Prakash K — Wed, 21 Oct 2026 14:30:00 +0000

Running an AI agent on your laptop is one thing. Putting it in front of real users is another. The agent itself doesn't change — but everything around it does.

API keys need to stay out of your code. Token costs add up fast if you're not watching. A slow model call that was fine during testing becomes a timeout in production. And when something goes wrong, you need logs that actually tell you what happened.

This article covers the practical side of shipping a Spring AI agent: secrets management, cost awareness, rate limiting, timeouts, and what to monitor once you're live.

The API key problem

During development, it's tempting to put your API key directly in application.properties:

spring.ai.openai.api-key=sk-proj-abc123...

Don't do this. That file gets committed to version control. The key leaks. You get a bill.

The right pattern: read the key from an environment variable.

spring.ai.openai.api-key=${OPENAI_API_KEY}

Now start the application with the key set in the environment:

export OPENAI_API_KEY=sk-proj-abc123...
java -jar your-app.jar

On a cloud platform (Railway, Render, Fly.io, AWS), set this as a secret environment variable in the deployment settings — never in a config file that gets pushed.

What about local development? Use a .env file that is listed in .gitignore, and load it with a tool like direnv or just set the variable in your shell profile. Never commit .env.

Separating configuration by environment

Your app needs different settings in development vs production:

Dev: verbose logging, low timeouts, a small token limit so you don't burn through a quota while testing
Prod: tighter timeouts, higher token limits, structured logging

Spring Boot's profile system handles this cleanly.

src/main/resources/application.properties — shared defaults:

spring.ai.openai.api-key=${OPENAI_API_KEY}
app.agent.timeout-seconds=30
app.agent.max-tokens=2000

src/main/resources/application-dev.properties — development overrides:

logging.level.com.example=DEBUG
app.agent.timeout-seconds=60
app.agent.max-tokens=500

src/main/resources/application-prod.properties — production overrides:

logging.level.com.example=INFO
app.agent.timeout-seconds=20
app.agent.max-tokens=4000

Activate a profile at startup:

java -jar your-app.jar --spring.profiles.active=prod

Your agent code reads these values without knowing which profile is active:

@Value("${app.agent.timeout-seconds}")
private int timeoutSeconds;

@Value("${app.agent.max-tokens}")
private int maxTokens;

Timeouts — set them everywhere

In development, a slow model call is annoying. In production, it blocks a thread, holds a connection, and can cascade into a full outage.

Set a timeout at every layer.

On the ChatClient call:

String response = CompletableFuture
    .supplyAsync(() -> chatClient.prompt()
        .user(request.message())
        .tools(travelTools)
        .call()
        .content())
    .get(timeoutSeconds, TimeUnit.SECONDS);

On the HTTP client Spring AI uses internally, you can configure it via properties:

spring.ai.openai.chat.options.timeout=20000

On your REST endpoint with Spring MVC:

@PostMapping("/chat")
@ResponseTimeout(20000)
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    // ...
}

Pick one consistent timeout value and apply it everywhere. Thirty seconds is a reasonable ceiling for most agent calls. If your agent needs longer, it's probably doing too much in a single request — consider breaking it into smaller steps.

Token costs — what you're actually paying for

Every call to the model costs money based on tokens — roughly three-quarters of a word each. The bill comes from two places:

Input tokens: your system prompt + conversation history + the user's message + tool definitions
Output tokens: the model's response

Output tokens cost more than input tokens. A verbose system prompt that runs 500 tokens fires on every single request. Multiply by 10,000 users and it becomes significant.

Where token costs hide:

Source	What it contains	What to watch
System prompt	Your instructions	Keep it concise — every word fires on every call
Tool definitions	@tool descriptions	Long descriptions add up across many tools
Conversation history	All prior messages	Window size directly controls cost
Tool results	What tools return	Verbose JSON returns cost more than plain strings

How to track usage in Spring AI:

Every ChatResponse contains usage metadata:

ChatResponse chatResponse = chatClient.prompt()
    .user(request.message())
    .call()
    .chatResponse();

Usage usage = chatResponse.getMetadata().getUsage();
log.info("tokens — input: {}, output: {}, total: {}",
    usage.getPromptTokens(),
    usage.getGenerationTokens(),
    usage.getTotalTokens());

Log this on every production request. After a few days you'll see exactly where cost is coming from and whether it's growing as expected.

Rate limiting — protect yourself and the API

The model provider's API has rate limits — requests per minute and tokens per minute. Hit them and your app gets 429 errors.

Two problems to solve:

1. Protect against a single user hammering the API:

Use a simple in-memory rate limiter per user (or per IP if you don't have user identity):

@Component
public class RateLimiter {

    private final Map<String, AtomicInteger> callCount = new ConcurrentHashMap<>();
    private final Map<String, Long> windowStart = new ConcurrentHashMap<>();
    private static final int MAX_CALLS_PER_MINUTE = 10;

    public boolean allow(String userId) {
        long now = System.currentTimeMillis();
        long start = windowStart.computeIfAbsent(userId, k -> now);

        if (now - start > 60_000) {
            callCount.put(userId, new AtomicInteger(0));
            windowStart.put(userId, now);
        }

        return callCount.computeIfAbsent(userId, k -> new AtomicInteger(0))
                        .incrementAndGet() <= MAX_CALLS_PER_MINUTE;
    }
}

Check it in your controller:

if (!rateLimiter.allow(request.userId())) {
    return ResponseEntity.status(429)
        .body("Too many requests. Please wait a moment.");
}

2. Handle 429s from the provider gracefully:

try {
    return chatClient.prompt().user(message).call().content();
} catch (OpenAiHttpException e) {
    if (e.statusCode() == 429) {
        return "The service is busy right now. Please try again in a moment.";
    }
    throw e;
}

Never let a raw 429 surface to the user as a 500 error.

Retry with backoff — for transient failures

Model APIs have transient failures — brief network blips, occasional 500s. A single retry with a short wait handles most of them.

Spring Retry makes this straightforward:

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>

Enable it in your main class:

@EnableRetry
@SpringBootApplication
public class Application { ... }

Add retry to your agent call:

@Retryable(
    retryFor = { OpenAiHttpException.class },
    maxAttempts = 3,
    backoff = @Backoff(delay = 1000, multiplier = 2)
)
public String callAgent(String message) {
    return chatClient.prompt().user(message).call().content();
}

This retries up to 3 times, waiting 1s then 2s between attempts. It only retries OpenAiHttpException — not your own application errors.

Do not retry on 429 errors with short delays — you'll just accelerate into the rate limit. Retry on 500s and network errors only.

What to log in production

Good production logs answer: "What did the agent do, for whom, and how did it go?"

Minimum log fields on every agent request:

@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    String requestId = UUID.randomUUID().toString();
    long start = System.currentTimeMillis();

    log.info("[req:{}] agent call — userId={} conversationId={}",
        requestId, request.userId(), request.conversationId());

    try {
        String response = callAgent(request.message(), request.conversationId());
        long durationMs = System.currentTimeMillis() - start;

        log.info("[req:{}] agent complete — durationMs={} responseLength={}",
            requestId, durationMs, response.length());

        return ResponseEntity.ok(response);
    } catch (TimeoutException e) {
        log.warn("[req:{}] agent timeout after {}ms", requestId, timeoutSeconds * 1000);
        return ResponseEntity.status(504).body("Request timed out.");
    } catch (Exception e) {
        log.error("[req:{}] agent error — {}", requestId, e.getMessage(), e);
        return ResponseEntity.status(500).body("Something went wrong.");
    }
}

The requestId ties every log line for a single request together. The durationMs tells you if calls are getting slower over time. The error path always logs the full stack trace.

Health checks and readiness

Before a request reaches your agent, you want to know the application is healthy. Spring Boot Actuator gives you this for free:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

management.endpoints.web.exposure.include=health,info
management.endpoint.health.show-details=when-authorized

GET /actuator/health returns {"status":"UP"} when the app is running. Your deployment platform (Render, Railway, Kubernetes) pings this endpoint — if it returns anything other than UP, traffic stops going to that instance.

Add a custom health check that verifies your model API is reachable:

@Component
public class ModelApiHealthIndicator implements HealthIndicator {

    private final ChatModel chatModel;

    public ModelApiHealthIndicator(ChatModel chatModel) {
        this.chatModel = chatModel;
    }

    @Override
    public Health health() {
        try {
            chatModel.call(new Prompt("ping"));
            return Health.up().build();
        } catch (Exception e) {
            return Health.down()
                .withDetail("reason", e.getMessage())
                .build();
        }
    }
}

If the model API is down, your health check returns DOWN and the platform can route around the broken instance or alert you.

A deployment checklist

Before you ship:

Secrets

[ ] API key is in an environment variable, not in any committed file
[ ] .env is in .gitignore
[ ] No secrets appear in log output

Reliability

[ ] Every agent call has a timeout
[ ] 429 errors are caught and returned as readable messages
[ ] Transient errors have retry with backoff

Cost

[ ] Token usage is logged on every request
[ ] System prompt is as concise as possible
[ ] Conversation history window is bounded (chatHistoryWindowSize)

Observability

[ ] Every request gets a unique request ID
[ ] Duration is logged on every request
[ ] Errors log the full stack trace

Operations

[ ] Health check endpoint is configured
[ ] Logs go somewhere you can actually read them (not just stdout on a server you can't SSH into)

What's next

The agent is deployed and handling real traffic. The next question is scale — what happens when many users hit the agent at the same time, when conversation histories grow large, and when you need to update the model or tools without downtime.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Building Agents That Don't Break — Error Handling and Patterns in Spring AI

Sham Prakash K — Mon, 19 Oct 2026 14:30:00 +0000

A basic agent works well when tools succeed, inputs are valid, and the task is straightforward. Real usage is messier. Tools time out. The model calls a tool with an argument it can't handle. A query returns no results. The agent loops without making progress.

This article covers how to design agents that handle these situations cleanly — patterns for planning, error recovery, structured output, observability, and preventing runaway behaviour.

What happens when a tool fails

When a tool throws an exception in Spring AI, the agent loop breaks. The model never gets a chance to handle the failure — it just sees an error response or nothing at all.

The better pattern: catch failures inside the tool and return them as readable strings.

@Tool(description = "Get the current weather for a city in India.")
public String getWeather(@ToolParam(description = "City name") String city) {
    try {
        return weatherApi.fetch(city);
    } catch (CityNotFoundException e) {
        return "Weather data not available for '" + city + "'. " +
               "This city may not be supported. Try a nearby major city.";
    } catch (ApiTimeoutException e) {
        return "Weather service is temporarily unavailable. " +
               "Please try again or skip weather for this trip.";
    }
}

When the model receives a failure message like this, it can respond intelligently: suggest an alternative city, skip that part of the answer, or tell the user the service is down. If the tool throws an uncaught exception, the model gets no information and the agent fails silently.

Rule: tools should never throw. They should return.

Returning partial results

When a task requires multiple tool calls and one fails, the agent doesn't have to fail entirely. Return what you have.

@Tool(description = "Get the cheapest ticket price between two cities.")
public String getTicketPrice(
        @ToolParam(description = "Departure city") String from,
        @ToolParam(description = "Destination city") String to) {
    try {
        String price = ticketApi.cheapest(from, to);
        return "Cheapest ticket from " + from + " to " + to + ": " + price;
    } catch (NoRouteException e) {
        return "No direct bus or train route found between " + from + " and " + to + ". " +
               "Consider checking flights or breaking the journey.";
    }
}

The model receives "no direct route found" and continues building the rest of the trip plan — weather, attractions, hotels — without the ticket price. A partial, honest answer is far better than a broken agent.

Validating arguments inside tools

The model constructs tool arguments from natural language. It usually gets them right, but edge cases slip through — wrong formats, unsupported values, empty strings.

Validate at the start of every tool:

@Tool(description = "Get flight prices for a date. Date must be in YYYY-MM-DD format.")
public String getFlightPrice(
        @ToolParam(description = "Departure city") String from,
        @ToolParam(description = "Destination city") String to,
        @ToolParam(description = "Travel date in YYYY-MM-DD format, e.g. 2026-11-15") String date) {

    if (from == null || from.isBlank()) {
        return "Error: departure city is required.";
    }
    if (!date.matches("\\d{4}-\\d{2}-\\d{2}")) {
        return "Error: date must be in YYYY-MM-DD format (e.g. 2026-11-15). Received: " + date;
    }

    return flightApi.search(from, to, date);
}

Returning the invalid input in the error message is deliberate — the model sees what it passed, recognises the format mismatch, and either corrects it on a retry or tells the user what's missing.

Planning before acting

For complex tasks — multi-city trips, research tasks, anything with many steps — prompting the model to plan before calling tools improves output quality significantly.

ChatClient.builder(chatModel)
    .defaultSystemPrompt("""
        You are a travel planning assistant.

        When given a complex request, first outline the steps you will take,
        then call the tools in that order.

        Do not call multiple tools simultaneously for a multi-step plan —
        work through the steps in sequence so each result can inform the next.

        If a tool returns an error or no data, note it in your answer
        and continue with the remaining steps.
        """)
    .build();

"First outline the steps, then call tools in that order" encourages the model to reason about the full task before starting. Without this, the model sometimes starts calling tools immediately, misses dependencies between steps, and produces a less coherent result.

Guarding against infinite loops

An agent that keeps calling tools without producing a final answer is a runaway agent. This can happen when:

Tools keep returning errors and the model keeps retrying
The model interprets each tool result as requiring another tool call
A bug in tool logic returns something that always looks like "incomplete"

Spring AI's ChatClient doesn't impose a hard limit on tool call iterations by default. You can wrap the agent call with a timeout:

@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
    try {
        String response = CompletableFuture
            .supplyAsync(() -> runAgent(request))
            .get(30, TimeUnit.SECONDS);
        return ResponseEntity.ok(response);
    } catch (TimeoutException e) {
        return ResponseEntity.status(504)
            .body("The agent took too long to respond. Please try a simpler question.");
    }
}

You can also add a counter inside a custom advisor or tool execution wrapper that aborts after N tool calls. Thirty seconds and ten tool calls per request are reasonable starting limits.

Sequential vs parallel tool execution

The default behaviour: the model calls tools in whatever order and parallelism it determines. For most tasks, this is correct. But some tasks have dependencies — the result of one tool call should inform the next.

Parallel is correct when tools are independent:

"What's the weather in Mysore, and what are the top attractions?"

Weather and attractions don't depend on each other. The model will correctly issue both calls simultaneously.

Sequential is correct when steps have dependencies:

"Find the cheapest travel option from Bangalore to my destination, then find hotels near the bus station."

The hotel search depends on knowing which bus station — which depends on the travel result. The model needs the first result before making the second call.

For dependency-heavy tasks, make this explicit in the system prompt: "Work through steps one at a time — each step may depend on the previous result."

Structured output

By default, the agent returns its final answer as a plain string. That works fine for chat — but when the response needs to feed into another system, be stored in a database, or be rendered as a structured UI component, plain text breaks down fast.

Spring AI's BeanOutputConverter lets you tell the model to return JSON matching a specific Java record. The converter injects formatting instructions into the system prompt automatically and parses the response back into a typed object.

Define the output shape:

public record TripPlan(
    String destination,
    String weather,
    List<String> attractions,
    String cheapestTransport,
    int estimatedCostInr
) {}

Use the converter in your agent call:

@PostMapping("/plan")
public TripPlan plan(@RequestBody ChatRequest request) {
    BeanOutputConverter<TripPlan> converter =
        new BeanOutputConverter<>(TripPlan.class);

    String response = chatClient
        .prompt()
        .user(request.message() + "\n\n" + converter.getFormat())
        .tools(travelTools)
        .call()
        .content();

    return converter.convert(response);
}

converter.getFormat() appends a JSON schema instruction to the user message — something like "Respond with a JSON object matching this schema: {...}". The model fills in the values; converter.convert() parses the JSON into your TripPlan record.

The output is now typed Java — you can store it, return it as an API response, or pass it downstream.

One thing to watch: if the model returns slightly malformed JSON (which happens, especially with complex schemas), convert() will throw. Wrap it in a try-catch and fall back to the raw string response when parsing fails. Keep output schemas flat and simple — deeply nested structures increase the chance of malformed output.

Observability — knowing what the agent actually did

In production, a user reports a wrong answer. Your first question is: which tools fired, with what arguments, and what did they return?

Without logging at the tool level, that question has no answer. You're left guessing.

The simplest approach: log at the start and end of every tool method.

@Tool(description = "Get the current weather for a city in India.")
public String getWeather(@ToolParam(description = "City name") String city) {
    log.info("[tool:getWeather] called with city='{}'", city);

    String result = weatherApi.fetch(city);

    log.info("[tool:getWeather] returned: {}", result);
    return result;
}

This gives you a complete trace of every agent execution in your logs — which tools ran, what the model passed, and what they returned. When an answer is wrong, you trace the conversation ID to the logs and see exactly what the agent did.

For a cleaner approach, Spring AI's Advisor interface lets you intercept every tool call centrally without touching each tool method:

@Component
public class ToolLoggingAdvisor implements RequestResponseAdvisor {

    private static final Logger log = LoggerFactory.getLogger(ToolLoggingAdvisor.class);

    @Override
    public AdvisedRequest adviseRequest(AdvisedRequest request, Map<String, Object> context) {
        log.info("[agent] incoming message: {}", request.userText());
        return request;
    }

    @Override
    public ChatResponse adviseResponse(ChatResponse response, Map<String, Object> context) {
        log.info("[agent] final response: {}",
            response.getResult().getOutput().getContent());
        return response;
    }
}

ChatClient.builder(chatModel)
    .defaultAdvisors(
        new ToolLoggingAdvisor(),
        new MessageChatMemoryAdvisor(new InMemoryChatMemory()),
        new QuestionAnswerAdvisor(vectorStore, SearchRequest.defaults())
    )
    .build();

The advisor logs every request before it reaches the model and every response before it's returned to the caller. Combined with per-tool logging, you have full visibility into what the agent did on every request — without modifying controller or service code.

What to log at minimum:

Conversation ID on every request
Each tool name and the arguments the model passed
Each tool's return value (or error)
Total time taken for the agent call

Streaming agent responses

For long-running agent calls — multiple tool calls, slow APIs — streaming lets users see progress rather than waiting for a blank screen.

Spring AI supports streaming with tool use:

@PostMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> streamChat(@RequestBody ChatRequest request) {
    return chatClient
        .prompt()
        .user(request.message())
        .tools(travelTools)
        .advisors(a -> a.param(
            AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY,
            request.conversationId()
        ))
        .stream()
        .content();
}

.stream().content() returns a Flux<String> that emits tokens as the model generates them. Tool calls still happen synchronously inside the loop — the model pauses streaming while a tool executes, then resumes streaming the result.

The user experience: they see text appearing token by token, with brief pauses when tools are called. This is significantly better than staring at a loading indicator for five seconds.

A checklist before shipping an agent

Before putting an agent in front of real users, run through these:

Tools all return strings, never throw. Exceptions break the loop invisibly.

Every tool has a "use this when..." clause. Without it, the model skips tools or uses the wrong one.

You've tested the "nothing found" case. Ask something the tools can't answer. The agent should say so honestly, not hallucinate.

You've set a timeout. An unbounded agent call is a denial-of-service risk.

You've tested with bad inputs. What happens when the user sends "???" or a very long message? The agent should degrade gracefully.

The system prompt includes a failure instruction. "If a tool returns an error, note it and continue" prevents the model from getting stuck retrying a broken tool.

Structured output has a fallback. If BeanOutputConverter.convert() throws on malformed JSON, catch it and return the raw string rather than crashing.

Tool calls are logged. Conversation ID, tool name, arguments, and return value — without this, debugging a wrong answer in production is guesswork.

What's next

The articles so far have covered building agents locally. The next phase is getting them in front of users — deploying to production, managing API keys, handling costs, and monitoring what the agent actually does in the wild.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Giving Your Spring AI Agent a Memory — Conversations That Actually Make Sense

Sham Prakash K — Sat, 17 Oct 2026 14:30:00 +0000

Send two messages to a basic agent:

"My budget for the trip is ₹5,000."
"What can I do within that budget?"

The second message makes no sense without the first. If the agent has no memory, it treats every message as a fresh conversation with no history. It doesn't know what trip. It doesn't know what budget. It has to ask again — or worse, it guesses.

Memory is what turns a stateless request-response system into an agent that can hold a real conversation.

What memory means in this context

Memory in an AI agent means one thing: the conversation history that gets sent to the model on each request.

The model itself has no persistent state. Between requests, it remembers nothing. What creates the illusion of memory is appending previous messages to each new request, so the model always sees the full conversation up to that point.

Request 1:  [user: "My budget is ₹5,000"]
            ↓ model responds
            [assistant: "Got it, ₹5,000 budget."]

Request 2:  [user: "My budget is ₹5,000"]        ← history replayed
            [assistant: "Got it, ₹5,000 budget."] ← history replayed
            [user: "What can I do within that budget?"]  ← new message
            ↓ model responds with full context

Your application stores the conversation history. On every new request, it prepends the stored history before sending to the model. That's the entire mechanism.

MessageChatMemoryAdvisor in Spring AI

Spring AI handles conversation history automatically through an advisor called MessageChatMemoryAdvisor. You saw it briefly in the RAG article — here's what it actually does.

@Bean
public ChatClient chatClient(ChatModel chatModel) {
    return ChatClient.builder(chatModel)
        .defaultAdvisors(
            new MessageChatMemoryAdvisor(new InMemoryChatMemory())
        )
        .build();
}

InMemoryChatMemory is a simple in-process store — a Map from conversation ID to message list. Before every request, the advisor loads the history for the current conversation ID and prepends it to the messages. After every request, it appends the new user message and the model's response to the history.

You don't write any of this yourself. The advisor handles it.

Conversation IDs — one per user session

Memory is per-conversation, not global. Each user gets their own conversation history. You identify conversations with a string ID — typically a UUID generated when the session starts.

@PostMapping("/chat")
public String chat(@RequestBody ChatRequest request) {
    return chatClient
        .prompt()
        .user(request.message())
        .advisors(a -> a.param(
            AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY,
            request.conversationId()
        ))
        .call()
        .content();
}

public record ChatRequest(String message, String conversationId) {}

The client generates a conversationId once (e.g. UUID.randomUUID().toString()) and sends it with every message in a session. The advisor uses it to look up and update the right history.

Two users with different conversation IDs get completely separate histories. Same agent, isolated memory.

What gets stored in history

The history isn't just user messages and assistant responses. In an agent with tools, the conversation history also includes tool calls and their results.

A typical history after one tool-calling turn looks like this:

[user]      "What's the weather in Chennai?"
[assistant] → tool_call: getWeather("Chennai")
[tool]      "Chennai: 34°C, humid, partly cloudy."
[assistant] "It's 34°C and humid in Chennai right now with some cloud cover."

All four messages are stored and replayed on the next turn. This is what allows the agent to remember not just what it said, but what it did — which tools it called, what they returned, and what conclusion it drew.

If a user follows up: "Is that hotter than usual for this time of year?" — the model has the tool result in context and can reason about it without calling getWeather again.

The context window limit

Conversation history grows with every turn. The model has a context window — a maximum number of tokens it can process in a single request. Pack enough history into a conversation and you hit that limit.

When history overflows the context window, one of a few things happens depending on how your application handles it:

The request fails with a token limit error
The oldest messages get silently truncated (you lose early context)
The model gets confused because the history appears to start mid-conversation

Spring AI's MessageChatMemoryAdvisor accepts a chatHistoryWindowSize parameter that limits how many previous messages to include:

new MessageChatMemoryAdvisor(
    new InMemoryChatMemory(),
    AbstractChatMemoryAdvisor.DEFAULT_CHAT_MEMORY_CONVERSATION_ID,
    20  // keep last 20 messages
)

With a window of 20, the advisor includes only the 20 most recent messages in each request. Older messages are still stored — the window just controls what gets sent to the model.

What window size to use? Start with 20–30 messages. A message is typically 50–200 tokens; 20 messages is roughly 1,000–4,000 tokens of history, leaving plenty of room for the system prompt, tool definitions, and the current response.

In-memory vs persistent memory

InMemoryChatMemory stores history in a Map inside your running application. It's fast and requires no infrastructure. The trade-off: it disappears when the application restarts.

For a development server or a demo, in-memory is fine. For production, you need persistent storage.

Spring AI provides a CassandraChatMemory and a Neo4jChatMemory, and the ChatMemory interface is straightforward to implement against any database:

public class PostgresChatMemory implements ChatMemory {

    private final JdbcTemplate jdbc;

    @Override
    public void add(String conversationId, List<Message> messages) {
        messages.forEach(msg ->
            jdbc.update(
                "INSERT INTO chat_history (conversation_id, role, content, created_at) VALUES (?,?,?,?)",
                conversationId, msg.getMessageType().name(),
                msg.getContent(), Instant.now()
            )
        );
    }

    @Override
    public List<Message> get(String conversationId, int lastN) {
        return jdbc.query(
            "SELECT role, content FROM chat_history " +
            "WHERE conversation_id = ? ORDER BY created_at DESC LIMIT ?",
            (rs, i) -> toMessage(rs.getString("role"), rs.getString("content")),
            conversationId, lastN
        );
    }

    @Override
    public void clear(String conversationId) {
        jdbc.update("DELETE FROM chat_history WHERE conversation_id = ?", conversationId);
    }
}

Swap new InMemoryChatMemory() for new PostgresChatMemory(jdbc) and history survives restarts, scales across multiple instances, and can be queried or audited.

Clearing history

When a user starts a new topic or wants a clean slate, clear the history:

chatMemory.clear(conversationId);

Without explicit clearing, history accumulates indefinitely (or until the window drops the oldest messages). For session-based use cases, clear on logout or on session expiry.

Memory and RAG together

When both MessageChatMemoryAdvisor and QuestionAnswerAdvisor are registered, the order matters:

ChatClient.builder(chatModel)
    .defaultAdvisors(
        new MessageChatMemoryAdvisor(new InMemoryChatMemory()),  // runs first
        new QuestionAnswerAdvisor(vectorStore,
            SearchRequest.defaults().withTopK(5))               // runs second
    )
    .build();

Memory runs first so that follow-up questions like "tell me more about that" contain the previous turn's context before RAG tries to retrieve relevant chunks. With the order reversed, RAG searches based on an incomplete message that has no conversational context.

What memory doesn't solve

Memory gives the model access to recent conversation history. It doesn't give the model access to:

Facts from much earlier in a long conversation. Once messages scroll past the history window, they're gone from the model's view. If a user mentioned their budget 40 messages ago and the window is 20, the model won't see it.

Knowledge from other conversations. Each conversation ID has its own isolated history. An insight the model developed in one session isn't available in another.

Structured data. History is a list of text messages. If you need to remember structured facts — the user's name, preferences, profile — store them in your database and inject them via the system prompt rather than relying on them being in conversation history.

What's next

Memory keeps context across turns. But there's another dimension to agent design: what happens when a task is too complex to complete in one shot — when the agent needs to break a goal into sub-tasks, work through them over multiple steps, and adapt its plan when something doesn't work?

The next article covers structured agent patterns — how to design agents that plan, delegate, and recover from failures.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Designing Tools Your AI Agent Will Actually Use Correctly

Sham Prakash K — Thu, 15 Oct 2026 14:30:00 +0000

The previous article showed how to wire tools into a Spring AI agent. The mechanics are straightforward — annotate a method, pass it to the ChatClient, done. What's less obvious is that the quality of your tools determines whether the agent works correctly.

A poorly described tool gets called at the wrong time, with the wrong arguments, or not at all. A well-designed tool gets called exactly when it should, with exactly the right input.

The description is an instruction to the model

When you write @Tool(description = "..."), that string isn't documentation for your team. It's a runtime instruction that the model reads before every single request.

The model's decision process looks like this:

What did the user ask?
What tools do I have available? (reads each description)
Do I need a tool to answer this? Which one?
What arguments should I pass? (reads each @ToolParam description)

If your description is vague, the model makes a guess. Sometimes the guess is right. Often it isn't — and because it's a language model, it won't tell you it was guessing. It will just proceed with what seemed most reasonable.

Vague description:

@Tool(description = "Get weather")
public String getWeather(String city) { ... }

The model doesn't know: Is this current weather or a forecast? What does it return — temperature? conditions? Both? What format is the city name expected in?

Clear description:

@Tool(description = "Get the current weather for a city in India. " +
    "Returns temperature in Celsius and sky conditions (sunny, cloudy, rainy). " +
    "Use this when the user asks about current weather or whether to carry an umbrella.")
public String getWeather(
        @ToolParam(description = "City name in English, e.g. Mumbai, Chennai, Bengaluru") 
        String city) { ... }

Now the model knows exactly when to call it, what it returns, and how to pass the city name. Same method, entirely different instruction.

What to include in a tool description

A useful tool description answers four questions:

What does it do? State the action clearly. "Fetches", "returns", "calculates", "queries" — be specific about the operation.

When should the model use it? This is the most commonly missed part. Add a "use this when..." clause. Without it, the model may skip the tool in cases where it should call it, because it isn't sure the tool applies.

What does it return? Describe the shape of the output. If it returns a price, say what currency. If it returns a date, say what format. If it returns a list, say roughly how many items.

What are the constraints? If the tool only works for certain inputs — cities in India, dates in the future, IDs that exist in the database — say that. The model will avoid calling it in unsupported cases, which prevents unnecessary tool failures.

Example applying all four:

@Tool(description = "Get the cheapest available bus or train ticket price between two cities in India. " +
    "Returns the price in INR and the transport type (bus or train). " +
    "Use this when the user asks about travel cost, ticket price, or how to get between cities. " +
    "Only works for cities with bus or rail connections — does not cover flights.")
public String getTicketPrice(
        @ToolParam(description = "Departure city name in English, e.g. Bengaluru") String from,
        @ToolParam(description = "Destination city name in English, e.g. Mysore") String to) {
    // ...
}

Writing good parameter descriptions

Parameter descriptions are just as important as the tool description. The model constructs arguments from them.

Be specific about format:

// Bad — model doesn't know what format to use
@ToolParam(description = "Date") String date

// Good — model knows exactly what to pass
@ToolParam(description = "Travel date in YYYY-MM-DD format, e.g. 2026-10-15") String date

Specify units:

// Bad
@ToolParam(description = "Budget") int amount

// Good
@ToolParam(description = "Maximum budget in INR, e.g. 5000") int amountInr

Clarify expected values for enums or limited options:

@ToolParam(description = "Seat class: SLEEPER, AC_3_TIER, or AC_2_TIER") String seatClass

When the model knows the exact expected values, it passes valid ones instead of guessing.

Returning results the model can reason about

Your tool's return value is what the model reads next. It uses that text to decide what to do — call another tool, ask a clarifying question, or produce the final answer.

Don't return raw JSON:

// Bad — the model has to parse this in its head
return "{\"temp\":28,\"condition\":\"PC\",\"humidity\":85}";

The model can handle JSON, but it has to mentally parse it first. That's cognitive overhead that introduces errors. Write the return value as natural language.

Return readable sentences:

// Good — the model reads this like a human would
return "Current weather in Mumbai: 28°C, partly cloudy, humidity 85%.";

Include context in the result:

If a tool fetches a price, include what the price is for. The model may call multiple tools and combine results — it needs context in each result to do that correctly.

// Without context — confusing when combined with other results
return "₹3,240";

// With context — clear when combined
return "Flight from Bengaluru to Goa on 2026-11-15: ₹3,240 (economy, non-stop, IndiGo 6E-502).";

Common failure modes

The model answers without calling the tool

This happens when the description doesn't say when to use the tool, and the model happens to know something relevant from training data. It answers directly, skipping the tool entirely.

Fix: add "Use this when..." to the description. Add to the system prompt: "Always use available tools to get current information — do not answer from memory."

The model calls the tool with the wrong argument format

A date tool receiving "15th October 2026" instead of "2026-10-15". A city tool receiving "Blr" instead of "Bengaluru".

Fix: specify the exact format in @ToolParam. Add examples: "e.g. 2026-10-15", "e.g. Bengaluru".

The model calls the wrong tool

Two tools with similar descriptions cause the model to mix them up. A getWeather and a getForecast tool with overlapping descriptions leads to unpredictable choices.

Fix: make descriptions explicitly distinct. State what each tool does NOT do: "Returns current conditions — for forecasts, use getForecast."

The tool returns an error and the model hallucinates a result

The tool throws an exception, returns an empty string, or returns an error message the model didn't expect. The model sees this as an unhelpful result and fabricates what the result probably would have been.

Fix: return meaningful error strings, not exceptions. "Unable to fetch weather for this city — city not found." is something the model can relay honestly to the user. An uncaught exception that surfaces as an empty result is something the model will try to paper over.

How many tools is too many?

There's no hard rule, but there's a practical limit. A model given 20 tools spends more time deciding which one to use. The chance of picking the wrong tool increases. The context window fills with tool definitions.

Keep tools focused. One tool should do one thing. A tool that fetches weather AND attractions AND ticket prices is three tools masquerading as one — split it.

Name tools clearly. The tool name is part of what the model reads. getWeather, getAttractions, getTicketPrice are clear. getData, fetch, process are not.

Remove tools the agent doesn't need. If a tool exists but the current use case will never need it, don't register it. Fewer tools mean faster, more accurate decisions.

Testing your tools

Tool quality is easy to verify. Ask questions that should and shouldn't trigger each tool, then inspect whether the right tools were called with the right arguments.

Three test cases every tool should pass:

Positive case — ask something that clearly requires this tool. Verify it was called and returned a sensible result.

Negative case — ask something unrelated. Verify the tool was NOT called.

Edge case — ask something ambiguous or with unusual input. Verify the arguments the model constructed are valid.

Add logging to each tool method during development:

@Tool(description = "...")
public String getWeather(@ToolParam(description = "City name") String city) {
    log.info("getWeather called with city={}", city);
    // ...
}

The log shows you exactly what the model passed. Wrong argument format? Fix the @ToolParam description. Tool not called at all? Fix the tool description or the system prompt.

What's next

Tools are methods. The agent loop runs them in sequence or parallel. But there's a limit to what single tool calls can do — some problems require a model to plan across many steps, remember intermediate results, and course-correct when something goes wrong. That's where more structured agent patterns come in.

The next article covers memory in agents — how to give your agent context from previous turns so it can handle multi-turn conversations, remember what it learned from earlier tool calls, and build on its own previous answers.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

AI Agents Explained — Tools, the ReAct Loop, and Spring Boot

Sham Prakash K — Tue, 13 Oct 2026 14:30:00 +0000

RAG gives the model access to your documents. It retrieves, it answers — but it never does anything. If answering a question requires checking a live API, querying a database, or chaining several steps together, RAG isn't enough.

That's where agents come in.

What an agent actually is

A regular chat call is one-shot: you send a message, the model responds, done. The model produces text — and only text. It has no way to interact with the outside world.

An agent breaks that limitation.

The idea is simple: you give the model a list of tools — capabilities your application exposes — and the model can invoke them during its response. It can call a tool, read the result, decide what to do next, call another tool, and eventually produce a final answer.

The model doesn't execute the tools itself. Your code does. The model just decides when to call them and what arguments to pass.

Think of it like a senior engineer delegating work. The manager (the model) decides what needs to be done. The engineers (your tool functions) actually do the work and report back. The manager reviews the results and figures out the next step.

What is a "tool"?

In the context of AI agents, a tool is just a function in your application that the model is allowed to call.

That function can do anything a normal Java method can do: call an external API, run a database query, read a file, do a calculation, send a notification. From the model's perspective, a tool is a named capability with a description and a set of inputs. From your code's perspective, it's a plain method.

Before the model responds to a user message, it sees:

The user's question
A list of available tools (name, description, what parameters each expects)

The model decides whether it needs a tool to answer. If yes, it responds not with text but with a tool call request: "call this tool with these arguments." Your application executes the function, sends the result back, and the model continues.

The ReAct pattern — how agents reason

The ReAct pattern (Reason + Act) describes how a model behaves inside this loop:

Reason — the model thinks about what it knows, what it still needs, and which tool will help next.

Act — it calls a tool.

Observe — it receives the tool's result.

Reason again — given the new information, what should happen next? Call another tool? Combine results and answer?

Act or answer — this repeats until the model has enough to respond.

Here's a concrete example. A user asks: "What's the weather like in Mumbai, and should I carry an umbrella?"

Without tools, the model can only guess based on training data. With a weather tool wired up:

Reason: I need live weather data for Mumbai. I have a getWeather tool.
Act: Call getWeather("Mumbai")
Observe: Result: "32°C, humidity 88%, chance of rain 70%"
Reason: I now have the data. I can answer.
Answer: "Mumbai is hot and humid today with a 70% chance of rain. Yes, carry an umbrella."

The model drove that entire sequence. Your code just executed the getWeather function when asked.

The agent loop — step by step

Before looking at code, it helps to understand exactly what happens behind the scenes on every request:

1. User sends a message
2. Your app sends: message + list of available tools → model
3. Model responds with a tool call request (not final text yet)
4. Your app executes the tool, gets the result
5. Your app sends: original conversation + tool result → model
6. Model either calls another tool (back to step 3)
   or produces the final answer (done)

Step 6 is the key: the loop continues until the model decides it has enough information. One question might trigger three tool calls. Another might need none. The model decides.

In Spring AI, you don't write this loop manually. The framework handles it. You write the tools; Spring AI runs the loop and returns the final answer.

Defining tools in Spring AI

Spring AI uses a @Tool annotation to mark methods as callable by the model. Any public method in a Spring bean can become a tool.

Here's the simplest possible tool:

@Service
public class WeatherTools {

    @Tool(description = "Get the current weather for a city. Returns temperature in Celsius and a short description.")
    public String getWeather(
            @ToolParam(description = "The city name, e.g. Mumbai or Chennai") String city) {

        // In a real app, this calls a weather API
        // For now, returning dummy data
        return "32°C, partly cloudy with high humidity";
    }
}

A few things to understand here:

@Tool marks the method as a tool. The description is what the model reads to decide whether this tool is relevant to the current task. Write it as if you're explaining the tool to a new developer: what does it do, what does it return?

@ToolParam describes each parameter. The model reads this description when deciding what argument to pass. If the user says "weather in Bombay", the model needs to know whether to pass "Bombay" or "Mumbai" or a city code — your description guides that decision.

The return value is what the model sees. Whatever your method returns gets sent back to the model as the result of that tool call. Return a plain, readable string — not raw JSON that the model has to parse and interpret. Write it like you'd write a sentence explaining the result.

Wiring tools into the ChatClient

Once the tools are defined, you pass them to the ChatClient when making a request:

@RestController
@RequestMapping("/api/agent")
public class AgentController {

    private final ChatClient chatClient;
    private final WeatherTools weatherTools;

    public AgentController(ChatClient.Builder builder, WeatherTools weatherTools) {
        this.chatClient = builder.build();
        this.weatherTools = weatherTools;
    }

    @PostMapping("/chat")
    public String chat(@RequestBody ChatRequest request) {
        return chatClient
            .prompt()
            .user(request.message())
            .tools(weatherTools)       // register tools for this request
            .call()
            .content();
    }
}

.tools(weatherTools) tells Spring AI: "for this request, the model is allowed to call any @Tool method in this bean."

Spring AI automatically converts your annotated methods into the tool definition format the model expects — name, description, parameter names, types. You don't write any JSON schema by hand.

.call().content() waits for the entire agent loop to finish and returns the model's final text response. The tool calls, the loop, the result passing — all of that happens inside this one call. From your controller's perspective, it looks like a regular chat request.

Multiple tools and multi-step reasoning

An agent can have many tools. The model decides which ones to call, in what order, and how many times.

@Service
public class TravelTools {

    @Tool(description = "Get current weather for a city. Returns temperature in Celsius and sky conditions.")
    public String getWeather(@ToolParam(description = "City name") String city) {
        return "24°C, clear skies";
    }

    @Tool(description = "Get top tourist attractions for a city. Returns a list of places to visit.")
    public String getAttractions(@ToolParam(description = "City name") String city) {
        return "Mysore Palace, Chamundi Hills, Brindavan Gardens, Devaraja Market";
    }

    @Tool(description = "Get current bus or train ticket price between two cities in India. Returns price in INR.")
    public String getTicketPrice(
            @ToolParam(description = "Departure city") String from,
            @ToolParam(description = "Destination city") String to) {
        return "Bus: ₹180, Train: ₹240 (2nd class)";
    }
}

Now ask: "Plan a weekend trip from Bangalore to Mysore — weather, what to see, and how to get there."

The model has three tools and an open-ended request. It will:

Call getWeather("Mysore") → "24°C, clear skies"
Call getAttractions("Mysore") → list of places
Call getTicketPrice("Bangalore", "Mysore") → fare info
Combine all three into a readable trip plan

Three tool calls, one user message, one final answer. The model orchestrated the sequence — your code just ran the functions.

Parallel tool calling

When multiple tool calls are independent of each other — when neither result depends on the other — the model can request them all at once.

For the trip planning example: weather, attractions, and ticket price can all be fetched simultaneously. None of them depend on each other's results. A capable model will issue all three calls in a single step, and Spring AI will execute them in parallel.

You write no parallel execution code. Spring AI detects that the model returned multiple tool calls in one response, runs them concurrently, and sends all results back together. The model then continues with the full set of results.

This matters for performance. Three sequential tool calls in a slow API might take three seconds. In parallel, they take the time of the slowest one.

System prompt for agents

Just like RAG needs a grounding instruction in the system prompt, agents benefit from a clear directive about when to use tools.

ChatClient.builder(chatModel)
    .defaultSystemPrompt("""
        You are a travel assistant for India.
        Use the available tools to answer questions — always call tools for current information.
        Do not answer from memory when a tool can provide the answer.
        If the user asks about something outside travel, politely say you can only help with travel questions.
        """)
    .build();

"Always call tools for current information — do not answer from memory" prevents the model from skipping tool calls when it happens to know something from training data. Without this, the model might answer a weather question confidently from stale knowledge instead of calling your live weather tool.

When to use an agent — and when not to

Agents are powerful but have a real cost: every tool call is a round trip to your application. A three-tool agent call makes four calls to the model (initial request + one per tool result). That's slower and more expensive than a plain chat call.

The decision is straightforward:

Plain chat call — the model reads input and produces text. Summarisation, classification, translation, writing. No external data needed.

RAG — the model needs knowledge from your documents. No actions, no live data — just retrieval and response.

Agent — answering requires doing something: checking a live API, running a database query, performing a multi-step process where each step's result shapes the next.

Only reach for agents when the task genuinely requires action. An agent that just wraps a static lookup is overkill — that's a plain function call.

What's next

Tools are only as good as their descriptions. The next article covers tool design in depth — how to write descriptions the model understands, how to return results the model can reason about, and the failure modes that come from poorly designed tools.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Building a Full RAG Pipeline in Spring Boot — From Ingest to Grounded Answer

Sham Prakash K — Sun, 11 Oct 2026 14:30:00 +0000

Previous articles covered the individual pieces of RAG. You now know what embeddings are, how to chunk documents, and how Pinecone stores and searches vectors. This article connects those pieces into a working pipeline in Spring Boot — and shows exactly what happens at each step when a real question comes in.

Two flows, one shared store

Every RAG system has two separate flows:

Ingest — runs once per document. Takes raw text, splits it into chunks, converts each chunk into a vector, stores it in Pinecone.

Query — runs on every user message. Takes the question, converts it to a vector, finds the most relevant chunks, injects them into the prompt, sends the enriched prompt to the model.

These two flows are completely independent. They share exactly one thing: the vector store. The ingest flow writes to it. The query flow reads from it.

The ingest flow

Splitting and storing

@Service
public class IngestService {

    private final VectorStore vectorStore;
    private final TokenTextSplitter splitter;

    public IngestService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
        this.splitter = new TokenTextSplitter(350, 50, 10, 10000, true);
    }

    public void ingest(String sourceId, String title, String content) {
        vectorStore.delete(List.of(sourceId));

        Document doc = new Document(content, Map.of(
            "source", sourceId,
            "title",  title
        ));

        List<Document> chunks = splitter.split(doc);
        vectorStore.add(chunks);
    }
}

vectorStore.delete() runs first — it removes any previously stored chunks from this document before adding new ones. Skip this step and re-uploading a document creates two sets of chunks in Pinecone. Retrieval will silently mix answers from both versions.

vectorStore.add(chunks) does three things in one call: embeds each chunk using text-embedding-004, then stores the text, the vector, and the metadata together in Pinecone. You don't call the embedding API separately — Spring AI handles it.

Ingest endpoint

@RestController
@RequestMapping("/api/documents")
public class IngestController {

    private final IngestService ingestService;

    @PostMapping
    public ResponseEntity<String> ingest(@RequestBody IngestRequest request) {
        ingestService.ingest(request.sourceId(), request.title(), request.content());
        return ResponseEntity.ok("Ingested: " + request.sourceId());
    }
}

Try it:

curl -X POST http://localhost:8080/api/documents \
  -H "Content-Type: application/json" \
  -d '{
    "sourceId": "return-policy-v2",
    "title": "Electronics Return Policy",
    "content": "Electronics purchased at full price may be returned within 30 days with original packaging and receipt..."
  }'

The response is just "Ingested: return-policy-v2". Nothing tells you how many chunks were created or what they contain — the work happened inside vectorStore.add().

The query flow

Wiring retrieval into the chat client

QuestionAnswerAdvisor is the piece that connects the vector store to every chat request. Configure it once on the ChatClient bean and it runs automatically on every message — you never call Pinecone directly from your controller.

@Configuration
public class ChatConfig {

    @Bean
    public ChatClient chatClient(ChatModel chatModel, VectorStore vectorStore) {
        return ChatClient.builder(chatModel)
            .defaultAdvisors(
                new MessageChatMemoryAdvisor(new InMemoryChatMemory()),
                new QuestionAnswerAdvisor(
                    vectorStore,
                    SearchRequest.defaults().withTopK(5)
                )
            )
            .defaultSystemPrompt("""
                You are a helpful assistant.
                Answer ONLY based on the provided context.
                If the context does not contain enough information, say so clearly.
                Do not use knowledge from outside the provided context.
                """)
            .build();
    }
}

Two things here worth understanding deeply:

withTopK(5) tells the advisor to retrieve the 5 most similar chunks for every query. That number directly affects what the model receives — too few and the answer might be in the 6th chunk, too many and the model gets overwhelmed with context. Five is a reasonable starting point.

The system prompt instruction is the enforcement mechanism for grounding. Answer ONLY based on the provided context tells the model not to use its training knowledge to fill gaps. Without this instruction, the model will — confidently, and invisibly — supplement retrieved context with things it learned during training. That's the exact problem RAG is supposed to solve.

Chat endpoint

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatClient chatClient;

    @PostMapping
    public String chat(@RequestBody ChatRequest request) {
        return chatClient
            .prompt()
            .user(request.message())
            .advisors(a -> a.param(
                AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY,
                request.conversationId()
            ))
            .call()
            .content();
    }
}

What the model actually receives

This is the detail that makes the whole system make sense.

When a user asks "What is the return policy for electronics?", the model doesn't receive that question alone. By the time QuestionAnswerAdvisor runs, the prompt looks like this:

System:
You are a helpful assistant.
Answer ONLY based on the provided context.
If the context does not contain enough information, say so clearly.

Context information is below.
---------------------
[Chunk 1: "Electronics purchased at full price may be returned within 30 days
with original packaging and receipt. Items must be in original condition."]

[Chunk 2: "Exceptions: laptops and tablets cannot be returned after the seal
is broken unless the item is defective."]

[Chunk 3: "To initiate a return, contact customer support with your order
number. Refunds are processed within 5–7 business days."]
---------------------
Given the context information and not prior knowledge, answer the question.

User: What is the return policy for electronics?

The model reads those chunks and answers from them. It did not search your documents — you retrieved the relevant sections and placed them directly in front of it. The model's job is to read and summarise. Your job is to retrieve well.

When there is no answer in the documents

Ask the system something that isn't in any of your uploaded documents. Pinecone still returns 5 chunks — the 5 least-dissimilar ones regardless of actual relevance. The model receives those chunks, finds no useful information in them, and because of the system prompt responds: "I don't have enough information to answer that based on the available documents."

That is the correct behaviour. If the system gives a detailed, confident answer to a question that isn't in your documents, check the system prompt — the grounding instruction is either missing or being ignored.

You can also add a similarity threshold to reject low-quality retrievals outright:

SearchRequest.defaults()
    .withTopK(5)
    .withSimilarityThreshold(0.70)

Chunks with similarity below 0.70 are discarded before they reach the prompt. If nothing passes the threshold, the model gets an empty context section and says so.

Tuning retrieval quality

Once the pipeline is running, these are the variables to adjust — in order of how much impact they have:

Chunk size — the biggest lever. Vague answers that feel like summaries usually mean chunks are too large. Drop from 350 to 200–250 tokens and re-ingest. Fragmented answers that miss context usually mean chunks are too small — try going larger.

topK — if the correct answer is in your documents but the model keeps missing it, the relevant chunk might be ranked 6th or 7th. Try increasing topK to 7 or 8.

Similarity threshold — add it only after you observe irrelevant chunks appearing in the context and causing wrong answers. Starting without a threshold is fine.

Test changes by writing 10–15 representative questions, running them before and after, and comparing the answers. Intuition alone won't tell you which direction to go.

A common failure mode

There's one failure that appears after the pipeline is running and the initial tests look fine: a question comes in that the documents don't answer, and the system gives a confident, detailed response anyway.

This happens when the system prompt grounding instruction is missing or soft. During initial testing, questions tend to be things the documents do answer — so the failure is invisible. It surfaces when real users ask things outside the scope of what was uploaded.

Always test the "not in the documents" case before considering the pipeline done. Ask something the system could not possibly know. If the answer sounds specific and confident, the system prompt is not enforcing grounding.

What's next

RAG gives the model knowledge from your documents. But knowledge isn't enough for every task. Some things require the model to act — query a database, call an API, run a multi-step process.

That's what AI agents do. The next phase of this series covers how agents work, what the ReAct loop is, and how to build one in Spring Boot.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Vector Databases — What They Are and How to Use Pinecone with Spring AI

Sham Prakash K — Fri, 09 Oct 2026 14:30:00 +0000

After chunking and embedding, you have a collection of vectors — one for each chunk of your documents. Now you need somewhere to store them, and more importantly, somewhere that can answer the question: "which of these stored vectors is closest in meaning to this new query vector?"

That's not a question a regular database is designed to answer.

What a dimension actually is

Before talking about vector databases, it's worth making "dimensions" concrete — because this is where most explanations lose people.

A dimension is just one number in the vector. A 3-dimensional vector has 3 numbers. A 5-dimensional vector has 5 numbers.

Imagine you described every piece of text along just two dimensions: how technical it is and how positive its tone is. You could plot every chunk on a flat 2D plane:

positive tone
      ↑
      │  "Great product, fast shipping!"        ● 
      │                          ● "Easy setup guide"
      │
──────┼──────────────────────────────────────────→ technical
      │           "JWT token validation fails" ●
      │  "Database connection pool exhausted" ●
      ↓
negative tone

Chunks that are similar in both dimensions end up close together. That's the entire idea.

Real embedding models don't use 2 dimensions — they use hundreds or thousands, because no two dimensions can capture all the nuance of language. But the intuition is the same: each dimension captures some aspect of meaning, and vectors with similar meanings end up close together in that space — regardless of how many dimensions there are.

The number of dimensions is fixed per embedding model. Google's text-embedding-004 uses 768. OpenAI's text-embedding-3-small uses 1536. You don't choose it — the model decides it. What matters is that you use the same model (and therefore the same number of dimensions) consistently.

Why a regular database can't search by similarity

A standard database index — a B-tree — works by sorting values so it can binary-search through them. Looking for rows where price < 50? The database jumps to the right part of the sorted index and reads from there.

Vector similarity search asks a different question: "which stored vectors point in the most similar direction to this query vector?" You can't sort vectors in a way that puts similar ones next to each other in a linear index. Similarity in high-dimensional space doesn't work like sorting numbers on a line.

The naive solution — compare the query vector against every single stored vector and rank them — works technically, but it scans the entire table on every query. With 10,000 chunks it's slow. With a million it's unusable.

A vector database solves this with specialised index structures that group similar vectors together, so a similarity search only has to examine a fraction of your data rather than all of it.

Pinecone — a managed vector database

Pinecone is a purpose-built vector database, fully managed. You don't install anything. You don't maintain a server. You create an index, store vectors, and query them through an API. There's a free tier that covers everything you need for learning and small projects.

This is why it's a good first choice for RAG: it removes all the infrastructure overhead so you can focus on understanding how vector storage and retrieval actually work.

The core concepts in Pinecone:

Index — a named collection of vectors. You create one per use case. Each index is configured with a dimension count that must match your embedding model.

Namespace — a partition inside an index. Useful for separating vectors by user, by document type, or by environment (dev vs production).

Record — one stored vector, with an ID, the vector values, and optional metadata.

Storing and searching with Pinecone

After creating an account and a free index at pinecone.io, you get an API key and an index host URL.

In Spring AI, add the Pinecone dependency:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-pinecone-store-spring-boot-autoconfigure</artifactId>
</dependency>

Configure in application.properties:

spring.ai.vectorstore.pinecone.api-key=${PINECONE_API_KEY}
spring.ai.vectorstore.pinecone.index-name=your-index-name
spring.ai.vectorstore.pinecone.namespace=documents

That's the entire setup. Spring AI handles the rest — embedding, storing, and searching all go through the same VectorStore interface you already know:

@Autowired
private VectorStore vectorStore;

// Ingest — stores chunks + their vectors in Pinecone
vectorStore.add(chunks);

// Search — embeds the query and finds the closest chunks
List<Document> results = vectorStore.similaritySearch(
    SearchRequest.query("What is the return policy?").withTopK(5)
);

similaritySearch embeds your query using the same embedding model configured for the app, sends the query vector to Pinecone, and returns the top K chunks ranked by similarity. The chunks come back as Document objects with content and metadata — ready to inject into a prompt.

Spring AI Advisors — automatic retrieval in every chat call

Manually calling similaritySearch and building the prompt yourself works, but Spring AI provides a higher-level abstraction that does it automatically: Advisors.

QuestionAnswerAdvisor intercepts every user message before it reaches the model, retrieves relevant chunks from your vector store, and injects them into the prompt as context:

@Bean
public ChatClient chatClient(ChatModel chatModel, VectorStore vectorStore) {
    return ChatClient.builder(chatModel)
        .defaultAdvisors(
            new MessageChatMemoryAdvisor(new InMemoryChatMemory()),
            new QuestionAnswerAdvisor(
                vectorStore,
                SearchRequest.defaults().withTopK(5)
            )
        )
        .build();
}

With this wired up, every call to the chat client automatically:

Takes the user's message and embeds it
Searches the vector store for the top 5 similar chunks
Injects those chunks into the system prompt before sending to the model

The injected context looks like this in the actual prompt:

Context information is below.
---------------------
[Chunk 1: "Electronics purchased at full price may be returned within 30 days..."]
[Chunk 2: "Exceptions: laptops and tablets cannot be returned after the seal is broken..."]
[Chunk 3: "To initiate a return, contact customer support with your order number..."]
---------------------
Given the context information and not prior knowledge, answer the question.

The model reads the injected chunks and answers from them — not from its training data. That's RAG running end to end through a single advisor.

The dimension mismatch bug

When you create a Pinecone index, you set a dimension count. That count must match the number of dimensions your embedding model produces.

If you create an index with dimension 768 and embed with Google's text-embedding-004 (which outputs 768 dimensions), everything works. If you later switch to a different embedding model that outputs a different number of dimensions, Pinecone rejects the vectors with a dimension mismatch error — and that error is visible, so you catch it.

The silent version is harder: switching to a different model that happens to output the same number of dimensions. Pinecone accepts the vectors. No error. But vectors from two different models are not comparable — they live in completely different spaces, like two maps drawn at different scales and orientations. Similarity scores between them are meaningless. Retrieval returns confidently wrong results with no warning.

The rule is the same as the one from the embeddings article: same model at ingest, same model at query, always. If you change the embedding model, delete your index and re-embed all documents from scratch.

A practical habit: store the embedding model name in your chunk metadata so you can always tell which model produced which vectors.

Document doc = new Document(chunkText, Map.of(
    "source",          "return-policy-v2",
    "embeddingModel",  "text-embedding-004"
));

What's next

You now have all the pieces: documents chunked, embedded, stored in Pinecone, and searchable via Spring AI. The next article puts it all together — a full working RAG pipeline in Spring Boot from the ingest endpoint to the retrieval-augmented chat response, with the complete architecture wired up and running.

Setting up Pinecone for the first time? Drop any issues in the comments — dimension mismatches and API key config are the two that catch people most often.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Chunking — Why How You Split Documents Decides Retrieval Quality

Sham Prakash K — Wed, 07 Oct 2026 14:30:00 +0000

The previous article explained how embeddings convert text into vectors so similar meanings land close together in vector space. Before you can embed a document, though, you have to decide how to cut it up. That decision — chunk size, chunk strategy, chunk overlap — has more impact on retrieval quality than almost anything else in a RAG system. And it fails silently. You won't get an error. You'll just get bad answers.

Why chunking exists

An embedding model converts a piece of text into a single vector. That vector represents the overall meaning of the entire input.

If you embed a 2,000-word document as one piece, you get one vector that represents the average meaning of all 2,000 words. When a user asks a specific question, the retrieved vector is the right document — but the 2,000 words contain dozens of topics, and the specific answer is buried somewhere in the middle. The model receives 2,000 words of context to search through and often glosses over the detail it needs.

If you split that same document into 300-word pieces and embed each piece separately, each vector represents a focused slice of meaning. When a user asks a specific question, the retrieved chunk contains mostly relevant content — and the model can extract the answer cleanly.

Chunking is how you control what unit of meaning gets embedded and retrieved.

Fixed-size chunking

The simplest strategy: split text every N characters, regardless of where sentences or paragraphs end.

// Spring AI: TokenTextSplitter with fixed token count
TokenTextSplitter splitter = new TokenTextSplitter(
    400,   // target chunk size in tokens
    50,    // minimum chunk size
    10,    // number of sentences to use for estimation
    10000, // max number of characters per chunk
    true   // keep separator
);

List<Document> chunks = splitter.split(document);

What it does well: Simple. Fast. Predictable chunk sizes. Good for structured content like product listings or short FAQ entries where every item is roughly self-contained.

What it does poorly: It cuts mid-sentence. A sentence that starts near the end of one chunk continues in the next. The embedding for the first chunk includes an incomplete thought. The embedding for the second chunk starts mid-sentence with no context. Both vectors are slightly off.

For prose documents — policies, articles, manuals — fixed-size splitting consistently degrades retrieval quality at the boundaries.

Recursive character splitting

A smarter approach: try to split at natural boundaries first, fall back to smaller boundaries only when needed.

The algorithm works like this:

Try to split at double newlines (\n\n) — paragraph breaks
If a resulting piece is still too large, split at single newlines (\n)
If still too large, split at sentence endings (., !, ?)
If still too large, split at spaces
If still too large, split at characters

The result is chunks that respect the natural structure of the text as much as possible. Paragraphs stay together until they're too large. Sentences stay together until they're too large.

// In Spring AI, configure the splitter to respect sentence boundaries
TokenTextSplitter splitter = new TokenTextSplitter();
splitter.setDefaultChunkSize(350);
splitter.setMinChunkSizeChars(100);

For most documents — policies, documentation, articles — recursive splitting produces noticeably better retrieval than fixed-size splitting. The chunks are semantically coherent because they follow the author's own structure.

Semantic chunking

The most advanced strategy: instead of splitting by size, split where the meaning changes.

The algorithm embeds every sentence, then compares the embedding of each sentence to its neighbours. When the cosine similarity drops sharply — when two adjacent sentences are less related than the average — that's a topic boundary. Split there.

Sentence 1: "The return policy for electronics covers 30 days."     → embed
Sentence 2: "Items must be in original condition with receipt."      → embed
Sentence 3: "For software products, all sales are final."           → embed
Sentence 4: "Our shipping partners include FedEx and UPS."          → embed ← similarity drops here
Sentence 5: "Standard delivery takes 3–5 business days."            → embed

Sentences 1–3 are about return policy. Sentence 4 shifts to shipping. The similarity drop triggers a split. The result is chunks that are semantically complete — each one covers one topic from start to finish.

The tradeoff: semantic chunking requires embedding every sentence during ingest, which is slower and more expensive. For large document libraries, the cost adds up. It produces the best retrieval quality, but recursive splitting gets you 80% of the way there at a fraction of the compute.

Chunk size: the single most important parameter

Whatever strategy you use, chunk size is the most impactful variable. Here is what happens at each end of the range:

Too large (1,000–2,000+ tokens):

One vector covers too many topics. Retrieval finds the right section but returns far more context than needed.
The model receives a wall of text with the answer buried inside. Response quality degrades — the model summarises instead of answering specifically.
Fewer chunks are stored, so similarity scores are spread thin across large sections.

Too small (30–50 tokens):

A sentence fragment gets its own vector. The answer to a multi-sentence question is split across five different chunks. Retrieval might return three fragments that together form an answer — but out of order, with gaps.
Short chunks lack context. "It must be returned within 30 days" is meaningless without the sentence before it that says what "it" refers to.

The practical starting range: 200–400 tokens.

This is where most production RAG systems land after tuning. It's small enough that each chunk covers one focused idea, and large enough that each chunk has enough context to be useful on its own. Adjust from there based on your document structure and what your retrieval results actually look like.

Chunk overlap

Even with good splitting, a fact sometimes lands exactly at a boundary. The end of one chunk contains the first half of a sentence; the beginning of the next chunk contains the second half. Both chunks miss the complete thought.

Overlap solves this: each chunk shares N tokens with the previous chunk.

Chunk 1: [...sentence A... sentence B... sentence C...]
Chunk 2:                  [sentence B... sentence C... sentence D... sentence E...]
Chunk 3:                                              [sentence D... sentence E... sentence F...]

Sentences B and C appear in both chunk 1 and chunk 2. If a question's answer spans the boundary between chunk 1 and chunk 2, either chunk can surface it.

Typical overlap: 10–20% of chunk size. For a 300-token chunk, 30–50 tokens of overlap is a reasonable starting point. More overlap means more redundant storage and more tokens injected into prompts; less overlap risks missing boundary-spanning facts.

Tables and code blocks

Recursive splitting and fixed-size splitting both have a blind spot: structured content.

Tables split badly. A markdown or HTML table has a header row and data rows. If the splitter cuts mid-table, you get two chunks: one with the header and a few rows, one with the remaining rows and no header. Both embed with degraded meaning — the second chunk has no column names, so the model has no idea what the numbers represent when it reads that chunk.

Code blocks have the same problem. A function split across two chunks produces one chunk ending mid-function and one starting in the middle of a method body. Neither chunk embeds or reads correctly.

The fix: treat tables and code blocks as atomic units. Never split inside them. Before running your splitting logic, identify blocks of this type and mark them as unsplittable — pass them through as a single chunk regardless of size. Most document parsers can identify these boundaries before splitting begins.

If a table or code block is genuinely too large to pass as one chunk, extract it separately with its own pre- and post-context (the heading above it, the sentence that introduces it) so it embeds with enough surrounding meaning to be retrievable.

Chunk metadata — tracking where each chunk came from

Every chunk should carry metadata: which document it came from, and ideally which section or page.

public void ingest(String documentText, String sourceId, String title) {
    Document doc = new Document(
        documentText,
        Map.of(
            "source", sourceId,   // document ID or filename
            "title",  title       // human-readable name
        )
    );
    List<Document> chunks = splitter.split(doc);
    vectorStore.add(chunks);
}

Spring AI propagates metadata from the parent Document to every chunk automatically. When a chunk is retrieved, its metadata comes back with it.

This matters for two reasons.

Citations. When the model answers from retrieved chunks, you can tell the user which document the answer came from: "Based on the Electronics Return Policy (v2.3)..." Without metadata, retrieved chunks are anonymous — you can't tell the user where the answer came from and you can't verify it yourself.

Debugging. When retrieval returns the wrong chunks, metadata is how you diagnose it. You can log which chunks were retrieved for each query and trace them back to the source document. Without metadata, a bad retrieval result is opaque — you see the text but have no path back to the original.

A minimum useful metadata set: source (document ID or filename), title (human-readable name). Add page or section when your documents have them.

Document updates and re-ingestion

Documents change. Policies get updated. Products get new specs. When a document changes, you need to update the chunks in your vector store.

The naive approach — just re-run ingest — creates a problem: you now have two sets of chunks from the same document. The old chunks and the new chunks both live in the vector store. Retrieval will return chunks from both versions, potentially mixing outdated content with current content. The model won't know which version is correct.

The right approach: delete before re-ingesting.

public void update(String sourceId, String newText, String title) {
    // Delete all chunks from the old version
    vectorStore.delete(List.of(sourceId));

    // Re-ingest the new version
    ingest(newText, sourceId, title);
}

This requires that sourceId is consistent across versions — the same document always uses the same ID. With pgvector, the delete is a SQL DELETE WHERE metadata->>'source' = ?. Spring AI's VectorStore interface exposes a delete(List<String> ids) method that handles this.

If you skip the delete step, stale chunks accumulate silently. Retrieval starts mixing answers from old and new versions with no indication that anything is wrong.

Ingestion pipeline design

Every document goes through the same pipeline before it can be retrieved:

Document (PDF, text, markdown)
     ↓
1. Parse — extract plain text from the file format
     ↓
2. Clean — remove headers, footers, boilerplate, formatting artefacts
     ↓
3. Split — cut into chunks (respecting tables and code blocks as atomic units)
     ↓
4. Embed — convert each chunk to a vector using the embedding model
     ↓
5. Store — save each chunk + its vector + metadata in the vector database

Steps 1–3 happen entirely in your application. Steps 4–5 involve the embedding API and the vector database. The whole pipeline runs once per document. Add a new document today, it's searchable immediately — no retraining, no redeployment.

In Spring AI, the pipeline collapses to a few lines:

@Autowired
private VectorStore vectorStore;

@Autowired
private TokenTextSplitter splitter;

public void ingest(String documentText, String sourceId, String title) {
    Document doc = new Document(
        documentText,
        Map.of("source", sourceId, "title", title)
    );
    List<Document> chunks = splitter.split(doc);
    vectorStore.add(chunks);  // embeds and stores in one call
}

VectorStore.add() handles the embedding and storage together. Spring AI calls the embedding API for each chunk and writes the vectors to the configured store — pgvector, Pinecone, or any other supported backend.

For batch ingestion of multiple documents:

public void ingestAll(List<Map<String, String>> documents) {
    List<Document> allChunks = documents.stream()
        .map(d -> new Document(d.get("text"), Map.of(
            "source", d.get("id"),
            "title",  d.get("title")
        )))
        .flatMap(doc -> splitter.split(doc).stream())
        .toList();

    vectorStore.add(allChunks);
}

Batch is significantly faster than ingesting one document at a time — the embedding API processes chunks in batches, reducing the number of HTTP round trips.

A mistake worth knowing

When I first built the ingest pipeline, I used 2,000-token chunks — roughly half a page of text. My thinking was: bigger chunks mean more context per retrieval, so the model has more to work with.

Retrieval looked fine on the surface. The top chunks coming back were semantically relevant — they were from the right sections. But the actual responses were vague. Questions that should have had specific, crisp answers were getting general summaries instead.

The problem took a while to diagnose because the retrieval metrics looked correct. The issue was in what the model was receiving: a 2,000-token chunk contains enough content to embed as semantically close to the query, but also contains 1,800 tokens of surrounding text that dilutes the specific answer. The model was reading the right chapter but looking for a needle in a haystack every time.

Dropping to 300-token chunks with 50-token overlap fixed it. Specific questions started getting specific answers. The retrieval precision went up because each chunk now represented one coherent idea rather than a sprawling section.

The rule I use now: start at 300 tokens, 50-token overlap. Run real queries against real data. If answers are too vague, chunk smaller. If answers are too fragmented or lack context, chunk larger.

What's next

Your documents are now split into focused chunks and stored as vectors. The piece you need next is the database that stores and searches those vectors efficiently. A regular database can't do this — you need a vector database, and for a Spring Boot application the most practical option is pgvector: the vector extension for PostgreSQL.

That's what the next article covers — what a vector database does, why a regular index doesn't work for similarity search, and how to set up pgvector with Spring AI.

How are you splitting your documents? Fixed chunks or something smarter? Drop your setup in the comments — I'm curious what chunk sizes people are landing on.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

How AI Understands Meaning — Embeddings Explained for Backend Engineers

Sham Prakash K — Mon, 05 Oct 2026 14:30:00 +0000

The previous article explained that RAG retrieves "semantically similar" chunks before generating an answer. But that phrase hides an entire question: how does a computer measure semantic similarity?

A computer doesn't understand meaning. It processes numbers. You can't tell a computer that "return policy" and "refund rules" are about the same thing — at least not directly. What you can do is convert both into numbers in such a way that similar meanings produce similar numbers. That conversion is what an embedding is.

What an embedding is

An embedding is a fixed-length list of numbers that represents a piece of text.

Take the sentence "What is the return policy for electronics?" — an embedding model converts that into something like:

[0.023, -0.417, 0.882, 0.001, -0.334, ... ]  ← 768 numbers total

Every piece of text — every word, sentence, paragraph — maps to a list of numbers like this. The length of the list depends on the model (768, 1024, 1536 are common). This list is called a vector.

The numbers themselves mean nothing individually. What matters is their relationship to other vectors. Similar texts produce vectors that are numerically close to each other.

Why numbers can capture meaning

This sounds like it shouldn't work. How can a list of numbers "know" that "return policy" and "refund rules" are related?

The answer is in how embedding models are trained.

An embedding model is trained on a massive amount of text — books, articles, documentation, code. During training, it learns to predict which words and phrases appear near each other. The model learns that "dog" and "cat" appear in similar contexts: they're both followed by words like "food", "vet", "sleep", "play". They're both preceded by words like "my", "the", "a cute". Over billions of examples, the model builds an internal map where things that appear in similar contexts end up in similar positions.

That internal map is the vector space. When you ask the model to embed a word, it returns the coordinates of that word on the map.

"Dog" and "cat" end up close together on the map — not because anyone programmed that, but because the training data made them end up near each other. "Cricket" and "sports" end up close for the same reason. "Dog" and "democracy" end up far apart — almost no overlap in the contexts where they appear.

Vector space intuition

Imagine your embedding model produces 2-dimensional vectors. Every piece of text maps to a point on a flat plane.

                    Y
                    |
   "cricket" •  •  "football"
                    |
"sports"       •   |
               "basketball"
─────────────────────────────── X
                    |
"democracy"  •      |
   "parliament" •   |
                    |

Words and phrases with similar meanings cluster together. Words with very different meanings are far apart.

Real embeddings have hundreds or thousands of dimensions instead of 2. You can't visualize them, but the math works the same way: similar meanings land in nearby positions in that high-dimensional space.

How similarity is measured — cosine similarity

Once you have two vectors, you measure how close they are using cosine similarity.

Cosine similarity measures the angle between two vectors. If two vectors point in almost the same direction, the angle between them is small and the similarity score is close to 1. If they point in opposite directions, the score is close to -1. Perpendicular vectors — totally unrelated — score around 0.

High similarity:   vector A and vector B point in the same direction → score ≈ 1.0
Low similarity:    vector A and vector C point in different directions → score ≈ 0.1

Why angle instead of distance? Because distance is sensitive to the magnitude of the vector — a longer document would produce a vector further from the origin even if it's about the same topic. Cosine similarity ignores magnitude and only looks at direction, which makes it much more reliable for comparing meaning regardless of text length.

In practice, your vector database handles all of this for you. You ask "give me the 5 vectors closest to this query vector" and it returns them, ranked by cosine similarity. You never implement the math yourself.

Cosine similarity vs dot product

Vector databases give you a choice of similarity metric. The two most common are cosine similarity and dot product. They look similar — and under one condition they're identical — but they behave differently and the choice matters.

Dot product multiplies the two vectors element by element and sums the results:

dot(A, B) = A[0]×B[0] + A[1]×B[1] + ... + A[n]×B[n]

The result depends on both the direction and the magnitude of the vectors. A longer vector (one with larger numbers overall) will produce a higher dot product score even if it isn't pointing in a more similar direction.

Cosine similarity normalises both vectors to unit length first, then computes the dot product:

cosine(A, B) = dot(A, B) / (magnitude(A) × magnitude(B))

The result depends only on direction — magnitude is cancelled out.

When they're identical: If your embedding model produces unit-normalised vectors (vectors with magnitude = 1), dot product and cosine similarity give exactly the same score. Many modern embedding models, including Google's text-embedding-004, output unit-normalised vectors specifically so you can use the faster dot product operation without needing to normalise first.

When the difference matters:

Metric	Sensitive to magnitude?	Use when
Dot product	Yes	Vectors are unit-normalised (most modern models)
Cosine similarity	No	Vectors have varying magnitude; you want direction only

For RAG with a modern embedding API, dot product is the practical choice — it's faster and the model is already normalising for you. Cosine similarity is the safer default if you're unsure whether the model normalises its output.

In pgvector (the PostgreSQL vector extension we'll use in the next article), the operators are:

-- Cosine distance (1 - cosine similarity; lower = more similar)
ORDER BY embedding <=> query_vector

-- Dot product (negative; lower = more similar, because pgvector uses distance)
ORDER BY embedding <#> query_vector

-- Euclidean distance (less common for semantic search)
ORDER BY embedding <-> query_vector

Spring AI's VectorStore handles the metric selection for you when you configure the store, so you rarely write this SQL directly — but knowing what's underneath matters when retrieval results look unexpected.

How this connects to RAG

When you embed your documents during the ingest phase, each chunk gets converted into a vector and stored in the vector database:

"Electronics purchased at full price may be returned within 30 days..."
     ↓ embedding model
[0.023, -0.417, 0.882, ...] → stored in vector DB

When a user asks a question, the same process runs on the question:

"What is the return policy for electronics?"
     ↓ same embedding model
[0.021, -0.408, 0.879, ...]  ← numerically close to the chunk above

The vector database compares the question vector against all stored chunk vectors and returns the ones with the highest cosine similarity. Those chunks are the semantically relevant ones — and they get injected into the prompt.

The reason semantic retrieval works is that the question vector and the relevant chunk vector end up close in vector space, even if they share no words. You never searched for "return" or "electronics" as keywords. You searched by meaning.

The same-model constraint

In previous article mentioned this briefly: the embedding model used during ingest must be the same one used during query.

Here's why that matters.

Every embedding model has its own internal map — its own way of laying out text in vector space. Google's text-embedding-004 and OpenAI's text-embedding-3-small both produce vectors, but those vectors live in completely different spaces. The numbers are incomparable across models.

If you embed your documents with model A and embed your query with model B, the vectors don't exist in the same space. There's no meaningful angle between them. Cosine similarity returns a number, but that number is noise — it has no relationship to actual semantic similarity.

The failure is silent. The vector database still runs the search, still returns 5 results, still looks like it's working. But the results will be wrong or random. This is one of the hardest bugs to catch because there's no error — just quietly bad retrieval.

Same model, both ways, always.

Embedding dimensions

When you call an embedding API, you get back a vector of a specific length — this is called the number of dimensions.

Google's text-embedding-004 produces 768-dimensional vectors by default (it supports up to 3072). OpenAI's text-embedding-3-large produces 3072-dimensional vectors. More dimensions generally means more nuance — the model can encode more information about meaning. But it also means larger storage and slower similarity search.

The dimensions are fixed per model and model version. If you switch from one model to another during a project, you have to re-embed all your documents from scratch because the vector length changes. Your old vectors become unusable — a 768-dimensional vector and a 1024-dimensional vector can't be compared.

This is another reason to choose your embedding model carefully at the start and not change it.

Calling the embedding API in Java

Using Spring AI, embedding a text string looks like this:

@Autowired
private EmbeddingModel embeddingModel;

public float[] embed(String text) {
    EmbeddingResponse response = embeddingModel.embedForResponse(List.of(text));
    return response.getResults().get(0).getOutput();
}

Spring AI's EmbeddingModel interface abstracts the provider — whether you're using Google's text-embedding-004, OpenAI's embedding API, or another provider, the call looks the same. The dependency injection handles the HTTP call and API key.

To embed multiple chunks at once (more efficient during ingest):

public List<float[]> embedBatch(List<String> texts) {
    EmbeddingResponse response = embeddingModel.embedForResponse(texts);
    return response.getResults().stream()
        .map(EmbeddingResultData::getOutput)
        .toList();
}

Batching is significantly faster and cheaper than embedding one chunk at a time — the API processes the batch in a single call instead of making one HTTP request per chunk.

A mistake worth knowing

Early in the project, I was testing the ingest pipeline with Google's text-embedding-004 and the retrieval looked fine in isolation. Then I swapped to OpenAI's API to test something unrelated, and in the process I accidentally left the embedding configuration pointing to OpenAI's embedding model while the stored vectors had been produced by Google's model.

Retrieval completely broke. Every question returned the same two chunks regardless of what the question was. But nothing threw an error. The search ran fine, just silently against incompatible vectors.

It took two hours to realise the issue was in the embedding configuration, not in the retrieval logic or the vector database setup. The fix was one line — pointing the query embedding back to text-embedding-004 — and then re-testing with a fresh batch of stored chunks to confirm the vectors were now in the same space.

The lesson: whenever retrieval looks wrong, check your embedding model configuration first, both ingest-side and query-side. It's the most common silent failure in RAG systems.

What's next

You now know what embeddings are and how semantic search works at the level that matters for building. The next piece is: how do you split your documents into chunks before embedding them?

The chunk size you choose has a bigger impact on retrieval quality than almost any other variable. Too large and you get vague results. Too small and you get fragments that miss context. There's a specific pattern that works, and a specific mistake I made that cost me two days of debugging — that's what the next article covers.

Built an embedding pipeline and hit unexpected results? Drop what you observed in the comments — it's almost always one of the issues above.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

What Is RAG — And Why Every AI App That Touches Real Data Needs It

Sham Prakash K — Sat, 03 Oct 2026 14:30:00 +0000

The model doesn't know your data. That's the sentence most AI tutorials skip.

You ask it about your product catalog — it hallucinates one. You ask it about your internal policy — it describes something it saw during training. You ask it what happened last week — it has no idea. Not because the model is bad. Because your data was never in the training set.

This is the problem RAG solves. And when the answer genuinely isn't in your data, RAG makes the model say so — instead of making something up. That's the other half of what it fixes.

Why the model doesn't know your data

An LLM learns by reading an enormous amount of public text — web pages, books, code, articles. After training, those patterns are baked into the model's weights. That's how it "knows" things.

But training has two hard limits.

A cutoff date. Anything published after the training cutoff doesn't exist for the model. It doesn't know what happened last month. It doesn't know what your team shipped last week.

Only what was in the training data. Your company's internal docs were never there. Your product catalog, your customer records, your policies — none of it was public text the model could learn from.

So when you ask about your specific domain, the model doesn't retrieve from a database. It generates text that fits the pattern based on what it learned during training. For general questions, that's fine. For questions about your data, it guesses — and guesses confidently.

The naive fix and why it fails

The first thought most people have: just include everything in the prompt.

Before the model replies, paste your entire document library into the context. All the product docs, all the policies, all the records. Give it everything — it'll find what it needs.

This fails for three reasons.

Token limits. A 50-page document is roughly 35,000 tokens. Even with a million-token context window, you can't dump an entire document library into every request. Real systems have thousands of documents.

The "lost in the middle" problem. We covered this in article 03. When you send a very long prompt, the model pays less attention to content buried in the middle. Relevant information sitting in paragraph 47 gets deprioritised. More context is not always better — it can actively hurt response quality.

Cost. If your context is 50,000 tokens and you're paying per million tokens, that's real money per API call — multiplied across every request from every user.

Dumping everything in the prompt doesn't scale. You need something smarter.

"Why not just fine-tune the model on my data?"

This is the first question most engineers ask. It sounds cleaner — train the model once on your data, and it knows everything.

Fine-tuning doesn't solve this problem. Here's why.

Fine-tuning teaches a model a style or a behaviour, not facts. You can fine-tune a model to always respond in JSON, or to follow a specific tone, or to focus on a domain. But it doesn't reliably store factual knowledge in a way you can trust. Fine-tuned models still hallucinate — they just hallucinate in your style.

More practically: your data changes. Products get updated, policies change, new documents are added. Every time your data changes, you'd need to re-run fine-tuning — a process that takes hours and costs significant compute. RAG retrieves from your current data on every query. Add a document today, it's searchable immediately.

Fine-tuning and RAG solve different problems. Fine-tuning for behaviour and style. RAG for grounding the model in specific, up-to-date facts. They're not alternatives — production systems often use both.

What RAG is

RAG stands for Retrieval Augmented Generation.

The word order matters. Retrieval comes first. Generation comes second.

Instead of sending everything to the model and hoping it finds the relevant piece — you find the relevant piece first, then send only that.

Here's the idea with a concrete example.

You have 500 pages of product documentation. A user asks: "What's the return policy for electronics?"

Without RAG: send all 500 pages to the model, hope it finds the answer somewhere.

With RAG:

Search your 500 pages for the sections most relevant to "return policy for electronics"
Find 3–4 paragraphs that actually contain that information
Send only those paragraphs to the model, along with the question
The model reads what you gave it and answers from that

The model never searches anything. It just reads what you put in front of it. You are the one doing the retrieval — and then giving the model exactly what it needs.

That's RAG. Retrieve the relevant pieces. Generate the answer from them.

Two phases: ingest and query

Every RAG system has two distinct phases. Understanding this split is the key to understanding how it works.

Phase 1 — Ingest (once per document)

Before any user can ask questions, you prepare your data.

You take your documents — PDFs, text files, markdown, whatever — and process them:

Split each document into small chunks (a few hundred words each)
Embed each chunk — convert it into a list of numbers called a vector that captures its meaning
Store those vectors in a database built for vector search

This is a one-time operation per document. Once ingested, a document is ready to be retrieved.

Phase 2 — Query (every user request)

When a user asks a question:

Embed the question — convert it into a vector using the same embedding model used during ingest
Search the vector database for stored chunks that are semantically similar to the question vector
Retrieve the top 3–5 most relevant chunks (more on why this range in a moment)
Inject those chunks into the prompt alongside the question
Generate — the model reads the context and answers

Step by step, every time.

Why 3–5 chunks? Retrieve just 1 and you might miss the answer — the single most similar chunk may not contain everything the question needs, especially if the answer spans related sections. Retrieve 20 and you've recreated the "lost in the middle" problem — relevant details buried in a wall of context. 3–5 is the sweet spot: enough coverage to find the answer, focused enough that the model can use it. You'll tune this number based on your chunk size and your data.

One constraint matters here: the embedding model used at query time must be the same one used during ingest. Every embedding model has its own internal "map" of meaning. A vector produced by one model lives in a completely different space than a vector from another model — similarity scores between them are meaningless. If you embed your documents with Gemini's embedding model and then embed questions with OpenAI's, your similarity search will return garbage. Same model, both ways, always.

How the prompt changes

Without RAG, a prompt looks like this:

System: You are a helpful assistant.
User: What's the return policy for electronics?

The model guesses from its training data.

With RAG, the prompt looks like this:

System: You are a helpful assistant. Answer ONLY based on the provided context.
        If the context lacks enough information, say so clearly.

Context:
---
[Chunk 1: "Electronics purchased at full price may be returned within 30 days
with original packaging and receipt. Items must be in original condition..."]

[Chunk 2: "Exceptions to the standard return policy: laptops and tablets
cannot be returned after the seal is broken unless defective..."]

[Chunk 3: "To initiate a return, contact customer support with your order number.
Refunds are processed within 5–7 business days..."]
---

User: What's the return policy for electronics?

Now the model isn't guessing. It's reading actual policy text and summarising it for the user.

Notice the system prompt instruction: Answer ONLY based on the provided context. This is critical. Without it, the model might supplement your retrieved context with its training knowledge — mixing real data with guesses. That instruction keeps it grounded in what you provided.

And when the answer isn't in the retrieved chunks? The model says: "I don't have enough information to answer this." That's not a failure — that's the system working correctly. A confident wrong answer is far more dangerous than an honest "I don't know." RAG gives you the second option.

RAG is not a search engine

A keyword search finds documents that contain your words. RAG finds documents that are semantically related to your question — even when they share no keywords.

Search for "fix broken leg" — a keyword search returns results with those exact words. A RAG embedding model understands that "broken leg," "bone fracture," and "orthopedic injury" are close in meaning — and ranks results by semantic similarity, not by word overlap.

This is what makes retrieval in RAG powerful. It understands meaning. How it does that — through embeddings — is what the next article covers in detail.

Two types of questions RAG answers

Before building, be clear which problem you're solving. RAG works for two different use cases that look similar from the outside.

Document questions — "what does the policy say about X?"
You're searching for information inside documents. Product specs, how-to guides, FAQs, policies, uploaded notes. Semantic retrieval finds the relevant section.

Record questions — "what happened last week?"
You're looking up structured data. Order history, user activity, database records. Semantic search isn't the right tool here — structured queries are. This is where a database agent (covered later in the series) takes over.

RAG handles documents. A SQL agent handles records. A complete AI backend needs both — but they're separate problems.

The architecture after adding RAG

Before RAG, the architecture is simple:

User → API → LLM
         ↓
   PostgreSQL (chat history)

After adding RAG, two new components appear:

User → API → LLM
         ↓        ↑
   PostgreSQL   Context (retrieved chunks)
                    ↑
              Vector DB ← Embedding Model
                    ↑
              Ingest pipeline (split → embed → store)

Embedding model — converts text to vectors. For Gemini users, Google provides an embedding API. Same provider, no extra dependency.

Vector database — stores and searches vectors by similarity. Pinecone is the most common choice. Free tier covers everything you need to learn.

Ingest pipeline — when a document is uploaded, it runs through: split → embed → store. One operation, runs once per document.

On every user query, the API: embeds the question → searches the vector DB → retrieves the top chunks → injects them into the prompt → calls the LLM.

A mistake worth knowing before you start

When I first built the ingest pipeline, I chunked documents into 2,000-word pieces — roughly half a page of text.

Retrieval looked fine. Top chunks came back, all semantically relevant. But responses were vague. The model had the right section but couldn't extract specific answers cleanly.

The problem: each chunk was too large. A 2,000-word chunk gets embedded as one vector representing the average meaning of the whole section — not any specific fact inside it. When retrieved, you get a chunk that contains the answer somewhere inside 400 other words. The model has to work harder and often glosses over the detail.

Dropping to 300-word chunks fixed it. Retrieval precision jumped. Specific questions started getting specific answers.

Chunk size is one of the biggest practical levers in RAG quality. It's easy to get wrong the first time — and it fails silently. We'll go deep on exactly why in upcoming articles.

What's next

RAG needs two things: a way to measure semantic similarity so you can find relevant chunks, and a database that can search by that similarity. Both depend on embeddings.

What is an embedding? Why does converting text into a list of numbers let you search by meaning? Why does "cricket" end up close to "sports" in vector space even though they share no characters?

That's what the next article is about.

Building something where the model needs to answer from your own data? Drop in the comments what the use case is — I'm curious what people are solving.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

The System Prompt Is Not a Description — It's a Contract

Sham Prakash K — Thu, 01 Oct 2026 14:30:00 +0000

My first system prompt was this:

You are a helpful and friendly AI assistant. You are knowledgeable about many topics 
and always try to give thorough, well-explained answers. You are patient and 
understanding. You never refuse to help with reasonable questions. You always 
maintain a positive and encouraging tone in all your responses.

It did nothing. The model behaved exactly the same with it as without it. I'd written five sentences that described what an LLM already does by default.

This article is about writing system prompts that actually change how the model behaves — with real examples from the app we've been building.

What a system prompt actually is

When you call the Gemini API, your request has three parts:

System prompt — instructions the model reads before anything else
Conversation history — the previous messages
User message — what the user just typed

The system prompt is processed first, before any user input. The model uses it to understand its role, constraints, and expected behaviour for the entire conversation.

Think of it as the briefing you give an employee before their first day. The more specific and well-structured the briefing, the less they have to guess.

What breaks a system prompt

Too vague

You are a helpful assistant.

This tells the model nothing it doesn't already know. It'll just do what it would do anyway.

Describing what the model IS instead of what it should DO

You are an expert travel planner who knows everything about destinations worldwide.

Describing the model's identity doesn't constrain its behaviour. The model can still say anything.

Too long with no structure

A wall of text is hard for the model to parse. Instructions buried in paragraph 4 often get ignored or deprioritised. Important rules need to stand out.

Contradictory instructions

Be concise. Always give thorough, detailed answers with examples.

When instructions conflict, the model picks one — and you won't know which.

What actually works

Good system prompts have clear building blocks. Not every prompt needs all of them, but knowing what each one does helps you write deliberately.

Persona — who the model is in this context

Not a description of the model's general nature, but a specific role with scope:

You are a database assistant. You help users query business data 
using natural language.

Persona alone isn't enough, but it sets the frame for everything that follows.

Constraints — what the model will not do

This is where system prompts earn their keep. Hard limits that override the model's defaults:

Only write SELECT queries — no INSERT, UPDATE, DELETE, DROP, or DDL.

Answer ONLY based on the provided context. If the context lacks enough 
information, say so clearly.

If the user asks about any destination outside India, politely decline.

Constraints are the most powerful thing in a system prompt. They make the model predictable. Without them, the model will try to be helpful in ways you didn't intend.

Output format — how to structure the response

ALWAYS return query results as a full markdown table showing ALL rows 
and ALL columns — do NOT summarize, abbreviate, or omit rows.

Answer only YES or NO, nothing else.

Output format instructions are surprisingly effective. When you tell the model exactly what shape the response should be, it follows it consistently.

Few-shot examples — show the model exactly what you want

Instructions tell the model what to do. Examples show it. When format consistency matters, one or two examples in the system prompt are more reliable than a paragraph of instructions.

You are a helpful assistant for a Java backend engineer learning AI development.
Be concise and practical.

Example:
User: How do I check if a string is empty in Java?
Assistant: Use `str == null || str.isEmpty()`. Prefer `str.isBlank()` in Java 11+ to also catch whitespace-only strings.

The model picks up the response style — length, tone, format — from the example and applies it consistently. One example is usually enough. Two if the format is unusual or strict.

Sequence — the order things should happen

For tool-calling agents, the order of operations matters:

For DATABASE questions, ALWAYS follow this sequence:
1. Call listTables to see what tables are available
2. Call getTableSchema for every table you need — never guess column names
3. Write a safe SELECT query and call executeQuery

Without this, the model might try to write a query without checking the schema first — and guess column names that don't exist.

Real examples from this app

Here's how these principles look in practice, using the actual system prompts from the app we've been building.

The simplest case — lean and specific

public static final String CHAT_AI_SYSTEM =
    "You are a helpful assistant for a Java backend engineer learning AI development. " +
    "Be concise and practical.";

Two sentences. But they're specific. The model knows the audience (Java backend engineers learning AI) and the output style (concise and practical). It won't give long theoretical explanations when a code snippet would do.

Single-purpose validator — extremely tight

public static final String INDIA_VALIDATION_SYSTEM =
    "You are a geography validator. Answer only YES or NO, nothing else.";

This prompt is used to check whether a destination is in India before the travel agent runs. The output constraint (only YES or NO, nothing else) is absolute. No explanation, no uncertainty — just the answer the code needs to branch on.

RAG assistant — context-bound

public static final String RAG_SYSTEM =
    "You are a helpful assistant. Answer ONLY based on the provided context. " +
    "If the context lacks enough information, say so clearly.";

The key instruction is ONLY based on the provided context. Without this, the model uses its general knowledge to fill in gaps — which means it might answer with information that isn't in your documents. For a RAG system, that's a hallucination problem.

Complex agent — structured with rules

public static final String DATABASE_AGENT_SYSTEM =
    "You are a database and knowledge assistant. " +
    "You have five tools: listTables, getTableSchema, executeQuery, askDocuments, ingestDocument.\n" +
    "For DATABASE questions, ALWAYS follow this sequence:\n" +
    "1. Call listTables to see what tables are available\n" +
    "2. Call getTableSchema for every table you need — never guess column names\n" +
    "3. Write a safe SELECT query and call executeQuery\n" +
    "DATABASE RULES:\n" +
    "- Only write SELECT queries — no INSERT, UPDATE, DELETE, DROP, or DDL\n" +
    "- Always check the schema before querying — column names must come from getTableSchema, not guesses\n" +
    "- ALWAYS return query results as a full markdown table showing ALL rows and ALL columns\n" +
    "- If the user asks something the data cannot answer, say so honestly";

This prompt is longer — but it's structured. The sequence is numbered. The rules are bulleted. Important words are capitalised (ALWAYS, ONLY). The model can scan this prompt and find what applies to the current situation.

Dynamic system prompts

System prompts don't have to be static strings. In Spring AI you can build the system prompt at call time and inject runtime context — current date, user preferences, session data — before sending the request.

@PostMapping("/chat-ai")
public String chat(@RequestBody Map<String, String> request) {
    String userId = request.get("userId");
    String systemPrompt = String.format(
        "You are a helpful assistant for a Java backend engineer learning AI development. " +
        "Be concise and practical. Today's date is %s.",
        LocalDate.now()
    );

    return chatClient.prompt()
        .system(systemPrompt)   // ← override the default system prompt per request
        .user(request.get("message"))
        .advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
        .call()
        .content();
}

.system(systemPrompt) on the prompt overrides the defaultSystem(...) you set in the ChatClient builder for that one call. The builder default is still used for everything else.

This is also how production applications inject per-user context: user tier, preferences, retrieved memory facts, or feature flags — all assembled into the system prompt at request time rather than stored as a static constant.

Keep the static parts in Prompts.java as constants and string-format in the runtime values. That way the static structure stays readable and the dynamic parts are clearly marked.

How to test if your prompt is working

Write test cases the same way you'd write unit tests. For each instruction in your prompt, write a user message that should trigger it:

Instruction	Test input	Expected behaviour
India-only	"Plan a trip to Paris"	Politely declines
SELECT only	"Delete all orders"	Refuses
YES or NO only	"Is Mumbai in India?"	Returns `YES`
Concise	"What is Spring Boot?"	Short answer, no long essay

If the model doesn't behave as expected on your test inputs, the instruction is either too vague, missing, or buried where the model deprioritises it. Move critical rules earlier in the prompt and make them more explicit.

Practical rules I follow

Lead with constraints, not with personality. The model's personality is fine by default. Your constraints are what matters.

Use ALL CAPS for non-negotiable rules. ALWAYS, NEVER, ONLY. It works.

Number sequences. If order matters, number the steps. Bullet points suggest optional items. Numbered lists suggest mandatory sequence.

Be specific about output format. Don't say "give a structured answer." Say "return a markdown table with all rows and columns."

Keep it under 300 tokens where possible. Every token in your system prompt runs on every API call. A bloated system prompt costs money and attention.

What's next

System prompts shape how the model thinks. Next: structured output — making the model return JSON instead of prose, so your backend can parse and act on the response programmatically.

Have a system prompt that finally started working after you changed something specific? Drop it in the comments.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.