One user chatting with your agent is easy. Ten concurrent users is fine. A hundred concurrent users exposes every assumption you made when building for one.
The agent code itself rarely needs to change. What breaks is everything around it — conversation memory tied to a single instance, no limits on how much work one request can do, API connections opened and closed on every call, and identical questions hitting the model repeatedly when a cached answer would do.
This article covers how to design your Spring AI agent to scale horizontally: stateless services, persistent memory, connection pooling, response caching, and async request handling.
The stateless rule
The single most important principle for scaling: your application instances must be stateless.
Stateless means: any instance can handle any request. No local data that only lives in one instance's memory.
The problem with InMemoryChatMemory at scale:
User A's first message → hits Instance 1 → history stored in Instance 1
User A's second message → hits Instance 2 → no history found → agent has no context
With a load balancer distributing requests across instances, a user's conversation can land on a different instance every time. InMemoryChatMemory breaks immediately.
The fix: move conversation history out of the JVM and into a shared store.
Persistent conversation memory with Redis
Redis is the standard choice for session data at scale — fast, simple, and supported natively by Spring.
Add the dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
Configure the connection:
spring.data.redis.host=${REDIS_HOST:localhost}
spring.data.redis.port=6379
spring.data.redis.password=${REDIS_PASSWORD:}
Implement ChatMemory backed by Redis:
@Component
public class RedisChatMemory implements ChatMemory {
private static final String KEY_PREFIX = "chat:";
private static final Duration TTL = Duration.ofHours(24);
private final RedisTemplate<String, String> redis;
private final ObjectMapper objectMapper;
public RedisChatMemory(RedisTemplate<String, String> redis, ObjectMapper objectMapper) {
this.redis = redis;
this.objectMapper = objectMapper;
}
@Override
public void add(String conversationId, List<Message> messages) {
String key = KEY_PREFIX + conversationId;
messages.forEach(msg -> {
try {
String json = objectMapper.writeValueAsString(
Map.of("role", msg.getMessageType().name(),
"content", msg.getContent()));
redis.opsForList().rightPush(key, json);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
});
redis.expire(key, TTL);
}
@Override
public List<Message> get(String conversationId, int lastN) {
String key = KEY_PREFIX + conversationId;
long size = Optional.ofNullable(redis.opsForList().size(key)).orElse(0L);
long start = Math.max(0, size - lastN);
List<String> raw = redis.opsForList().range(key, start, -1);
if (raw == null) return List.of();
return raw.stream()
.map(this::deserialize)
.filter(Objects::nonNull)
.toList();
}
@Override
public void clear(String conversationId) {
redis.delete(KEY_PREFIX + conversationId);
}
private Message deserialize(String json) {
try {
Map<?, ?> map = objectMapper.readValue(json, Map.class);
String role = (String) map.get("role");
String content = (String) map.get("content");
return switch (role) {
case "USER" -> new UserMessage(content);
case "ASSISTANT" -> new AssistantMessage(content);
default -> null;
};
} catch (JsonProcessingException e) {
return null;
}
}
}
Wire it into the ChatClient:
@Bean
public ChatClient chatClient(ChatModel chatModel, RedisChatMemory redisChatMemory) {
return ChatClient.builder(chatModel)
.defaultAdvisors(
new MessageChatMemoryAdvisor(redisChatMemory)
)
.build();
}
Now any instance can handle any conversation — history lives in Redis, not in the JVM.
Connection pooling for the model API
Every call to the model goes over HTTP. Without connection pooling, each request opens a new TCP connection, completes TLS handshake, sends the request, and closes the connection. Under load, this overhead adds up.
Spring AI uses an HTTP client under the hood. Configure it to keep connections alive and reuse them:
@Configuration
public class HttpClientConfig {
@Bean
public RestClient.Builder restClientBuilder() {
HttpClient httpClient = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.executor(Executors.newVirtualThreadPerTaskExecutor())
.build();
ClientHttpRequestFactory factory =
new JdkClientHttpRequestFactory(httpClient);
return RestClient.builder()
.requestFactory(factory);
}
}
For higher throughput, use Apache HttpClient with explicit pool configuration:
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
</dependency>
@Bean
public PoolingHttpClientConnectionManager connectionManager() {
PoolingHttpClientConnectionManager manager =
new PoolingHttpClientConnectionManager();
manager.setMaxTotal(200); // total connections across all routes
manager.setDefaultMaxPerRoute(50); // connections to the model API endpoint
return manager;
}
Connection pooling is one of the cheapest performance wins — a few lines of config, meaningful latency reduction under load.
Caching repeated queries
Some questions get asked over and over:
"What are the top tourist attractions in Mysore?"
"What's the cheapest way to travel from Bangalore to Mysore?"
These don't change hour to hour. Calling the model every time wastes money and adds latency. Cache the response.
Spring's caching abstraction with Redis makes this straightforward:
Enable caching in your main class:
@EnableCaching
@SpringBootApplication
public class Application { ... }
Configure the cache manager:
@Bean
public RedisCacheManager cacheManager(RedisConnectionFactory factory) {
RedisCacheConfiguration config = RedisCacheConfiguration.defaultCacheConfig()
.entryTtl(Duration.ofMinutes(30))
.serializeValuesWith(
RedisSerializationContext.SerializationPair
.fromSerializer(new GenericJackson2JsonRedisSerializer()));
return RedisCacheManager.builder(factory)
.cacheDefaults(config)
.build();
}
Add @Cacheable to tool methods that return stable data:
@Cacheable(value = "attractions", key = "#city")
@Tool(description = "Get the top tourist attractions in a city.")
public String getAttractions(@ToolParam(description = "City name") String city) {
return attractionApi.fetch(city);
}
@Cacheable(value = "weather", key = "#city")
@Tool(description = "Get the current weather for a city.")
public String getWeather(@ToolParam(description = "City name") String city) {
return weatherApi.fetch(city);
}
The first call for "Mysore" hits the API and stores the result. The next 100 calls for "Mysore" return from Redis instantly — no model call, no tool API call.
What to cache and what not to:
- Cache: attraction lists, route options, general information that changes slowly
- Do not cache: user-specific data, prices that change frequently, anything time-sensitive
Async request handling
By default, Spring MVC handles each request on a thread from a thread pool. A 5-second model call holds that thread for 5 seconds. With 100 concurrent users and a 20-thread pool, requests queue up fast.
Switch agent calls to async processing to free threads while waiting for the model:
@PostMapping("/chat")
public CompletableFuture<ResponseEntity<String>> chat(@RequestBody ChatRequest request) {
return CompletableFuture
.supplyAsync(() -> callAgent(request), agentExecutor)
.orTimeout(30, TimeUnit.SECONDS)
.thenApply(ResponseEntity::ok)
.exceptionally(e -> {
if (e.getCause() instanceof TimeoutException) {
return ResponseEntity.status(504)
.body("Request timed out.");
}
return ResponseEntity.status(500)
.body("Something went wrong.");
});
}
Configure a dedicated thread pool for agent calls:
@Bean("agentExecutor")
public Executor agentExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(20);
executor.setMaxPoolSize(50);
executor.setQueueCapacity(100);
executor.setThreadNamePrefix("agent-");
executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
executor.initialize();
return executor;
}
With Java 21+, virtual threads handle this even more cleanly:
spring.threads.virtual.enabled=true
One line in properties — Spring Boot automatically uses virtual threads for all servlet request handling. Each request gets its own lightweight thread, and blocking on the model call costs almost nothing.
Rate limiting at the application layer
Under load, a single slow user can consume a disproportionate share of your model API quota. Enforce per-user limits at the application layer before requests reach the model.
Using Bucket4j for token bucket rate limiting:
<dependency>
<groupId>com.github.vladimir-bukhtoyarov</groupId>
<artifactId>bucket4j-core</artifactId>
<version>8.10.1</version>
</dependency>
@Component
public class UserRateLimiter {
private final Map<String, Bucket> buckets = new ConcurrentHashMap<>();
private Bucket bucketFor(String userId) {
return buckets.computeIfAbsent(userId, k ->
Bucket.builder()
.addLimit(Bandwidth.classic(10, Refill.greedy(10, Duration.ofMinutes(1))))
.build()
);
}
public boolean tryConsume(String userId) {
return bucketFor(userId).tryConsume(1);
}
}
Check in the controller:
if (!userRateLimiter.tryConsume(request.userId())) {
return ResponseEntity.status(429)
.body("You've sent too many messages. Please wait a moment.");
}
Ten requests per minute per user is a reasonable starting limit for an agent that makes multiple tool calls per request.
What to monitor under load
Once traffic scales up, these metrics tell you where the system is struggling:
Latency percentiles — not averages
Average latency hides tail problems. Watch p95 and p99 (the slowest 5% and 1% of requests). If p99 is 30 seconds and p50 is 2 seconds, a small number of users are timing out while most are fine.
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(registry);
}
@Timed(value = "agent.call", percentiles = {0.5, 0.95, 0.99})
public String callAgent(String message, String conversationId) {
// ...
}
Thread pool utilisation
If your agent thread pool is at capacity, requests queue. Watch agent.pool.active vs agent.pool.size — if active consistently approaches max, increase pool size or reduce per-request latency.
Redis hit rate for cached tools
If your cache hit rate is low, either the cache TTL is too short, keys are not being reused across users, or the queries are naturally unique. A hit rate below 20% for stable data suggests a caching problem.
Token usage per request over time
If average token count per request is growing, conversation histories are getting longer or system prompts changed. Unbounded growth here becomes a cost problem.
A scaling checklist
Before putting your agent behind a load balancer:
- [ ] Conversation history is stored in Redis or a database, not in-memory
- [ ] Application instances have no local state that affects responses
- [ ] HTTP connection pooling is configured for the model API client
- [ ] Stable tool results are cached with an appropriate TTL
- [ ] Async request handling is in place (virtual threads or dedicated pool)
- [ ] Per-user rate limiting is enforced before reaching the model
- [ ] p95 and p99 latency are being tracked
- [ ] Thread pool utilisation is monitored
What's next
The agent scales horizontally. The next step is expanding what it can do — connecting it to external systems through the Model Context Protocol (MCP), so tools don't have to live inside your Spring Boot application.
Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.
Top comments (0)