Building a Production AI Agent in Spring Boot: Conversation Memory and Context (Part 2)

Two weeks after I shipped the e-commerce agent from Part 1, my first real tester asked it a question that broke the illusion: "What was in my cart again?"

The agent had answered that exact question two messages earlier. It had called the cart tool, listed four items, and totaled the price. And now it was starting from zero, because to the model every message is a fresh conversation. Statelessness is the default contract of every LLM API. If you do not send the history back, the model does not have it.

That is the gap this part of the series closes. In Part 1, I showed how to give an agent hands with the @Tool annotation. Here I show how to give it a memory: conversation history that survives across turns, session isolation so users never see each other's context, and the context-management decisions that keep the whole thing inside your token budget.

I am a Senior Software Engineer II at BS23 in Dhaka, and I have been building production AI agents with Spring Boot and Spring AI for over a year. Everything below comes from an agent I actually run, not a toy demo.

Memory Means You Re-Send the History

There is no hidden state inside a model. When your user says "what was in my cart?", the model can only answer from what is in the request you build. So conversation memory is not a database lookup the model performs. It is a prompt-engineering problem: you keep the history, and you send the relevant slice of it with every call.

That framing changes how you design it. You are not storing memories for the model. You are storing them for yourself, and deciding each turn what to replay. The quality of the agent depends on two things:

• What you store after each turn
• What you replay into the next prompt

Spring AI 2.0 handles both with two pieces: a ChatMemory that stores messages, and an advisor that wires the store into every request. Let me show you the setup I run, then the decisions that matter.

Step 1: Create a ChatMemory Bean

The ChatMemory interface is tiny, and that is a feature:

public interface ChatMemory {
    void add(String conversationId, List<Message> messages);
    List<Message> get(String conversationId, int lastN);
    void clear(String conversationId);
}

Everything is keyed by a conversation ID. The simplest production-ready implementation is MessageWindowChatMemory, which keeps a sliding window of the last N messages per conversation:

@Bean
public ChatMemory chatMemory() {
    return MessageWindowChatMemory.builder()
            .maxMessages(30)
            .build();
}

The sliding window is the default answer to "how much should the agent remember?" It remembers the last 30 messages and silently drops everything older. That bounds your token usage and protects you from unbounded growth, at the cost of short-term memory. For a support agent that resolves a cart issue in one session, 30 messages is plenty. For a long-running assistant you revisit daily, it is not, and I will come back to that in the context-management section.

Step 2: Attach Memory with an Advisor

Spring AI's advisor chain is the clean way to plug behavior into every prompt. The MessageChatMemoryAdvisor does three things per call: it reads the history for the conversation ID, appends it to the prompt, and after the response it stores the new turn back into the ChatMemory. You never touch history manually.

@Bean
public ChatClient chatClient(ChatModel chatModel, ChatMemory chatMemory, ShoppingTools tools) {
    return ChatClient.builder(chatModel)
            .defaultSystem(SYSTEM_PROMPT)
            .defaultAdvisors(MessageChatMemoryAdvisor.builder(chatMemory).build())
            .defaultTools(tools)
            .build();
}

This is the full wiring for the agent I described in Part 1: a system prompt, a memory advisor, and a set of @Tool methods, all assembled into one ChatClient bean. Notice that the advisor is registered as a default, which means it applies to every request through this client. The chat memory documentation has the details on ordering advisors and the available options.

With that bean in place, my tester's question finally works. The agent remembers the cart contents it listed two messages ago, because those messages are replayed into the current prompt.

Step 3: Scope Memory Per Session

Here is where most tutorials stop, and where production starts. One ChatMemory bean serves every user in your application. If you do not scope it, the memory is global: user A's conversation bleeds into user B's, and your agent starts answering with someone else's cart.

The advisor scopes by conversation ID, which you pass per request:

String answer = chatClient.prompt()
        .user(userMessage)
        .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
        .call()
        .content();

The conversation ID should come from your server-side session, not from the client. In my project, the controller receives the session ID from the web layer and passes it through. Never accept a raw conversation ID from the request body and store messages under it blindly, or a user can read and poison another session. Derive it from your auth context, or generate it at session start and keep it server-side.

This is also the pattern for streaming, which I will cover in Part 3. The advisor works identically when you call .stream() instead of .call(); you pass the same conversation ID and the memory is maintained token by token.

Step 4: Pass the Session to Your Tools

The conversation ID is not just for the model. It matters for your tools, and this is the mistake that cost me a day of debugging.

Your @Tool methods are plain Java called by Spring AI's tool executor. They have no idea which user triggered them. If a tool touches per-user state, like a shopping cart, it must know the conversation ID or every user shares one cart.

The fix is ToolContext, a map you pass alongside the prompt that reaches your tool methods:

chatClient.prompt()
        .user(userMessage)
        .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
        .toolContext(Map.of("conversationId", conversationId))
        .call()
        .content();

And inside the tool:

@Tool(description = "Show the current contents of the shopper's cart with quantities, subtotals and the grand total.")
public String viewCart(ToolContext toolContext) {
    String conversationId = conversationId(toolContext);
    List<CartItem> items = cartService.getItems(conversationId);
    if (items.isEmpty()) {
        return "The cart is empty.";
    }
    // format items and total...
}

private static String conversationId(ToolContext toolContext) {
    Object id = toolContext.getContext().get("conversationId");
    if (id == null) {
        throw new IllegalStateException("conversationId missing from tool context");
    }
    return id.toString();
}

Every tool in my project that touches the cart or an order reads its conversation ID this way, and fails fast if it is missing. That one convention prevents a whole class of cross-user state bugs. The ChatClient reference documents the full toolContext and advisor APIs if you want the details.

Context Management: The Token Budget You Did Not Budget For

Memory solved the "forgot my cart" problem, and then it created a new one: cost. Every message you replay is tokens, and tokens are the only thing you pay for on a hosted model. Here is the arithmetic on my production agent, which runs on a 128K-token context model:

• System prompt: roughly 600 tokens, every call
• Tool schemas: each @Tool method becomes a JSON schema the model must see. My 9 tools cost around 5,000 tokens per call, even when the model calls none of them
• Memory window: 30 messages at 400 to 800 tokens each, roughly 12,000 to 24,000 tokens
• RAG retrieval: top-5 document chunks, about 1,000 tokens when retrieval is used

That is 18,000 to 30,000 tokens of context before the model writes a single word of the answer. On a paid API that is real money per request, and on a big model it is real latency. Context management is cost management.

The levers I actually use, in order of impact:

• Shrink the memory window to what the task needs. A checkout flow needs 10 messages, not 30. Every message you keep is a tax on every future call.
• Cut tool schema bloat. Shorten descriptions, merge rarely used tools, drop parameters the model never fills. This was the single biggest saving in my project.
• Summarize, do not replay. For long-running assistants, keep a rolling summary of old turns plus the last few raw messages, instead of replaying everything. The summary captures the decisions, the raw tail captures the immediate context.
• Retrieve less, better. For RAG, 5 tight chunks beat 20 loose ones. Prefer chunk quality over topK.
• Tune per model, not once. The same window that fits a 128K model will blow a budget on a smaller one. If you support multiple models, the window size belongs in configuration.

None of this requires a different architecture. It is configuration and discipline once you can see the token breakdown, which is why I count observability as a first-class feature and not a nice-to-have.

External Memory: RAG and Tool-Based Retrieval

Conversation memory is one kind of context. The other is external memory: your documents, catalog, and database. The agent from Part 1 knows how to call tools, so it can query live data. But "find products that feel cozy" is not a SQL query, and that is where retrieval comes in.

There are two ways to give an agent retrieval, and they differ in who decides when to look:

Advisor-based retrieval. A QuestionAnswerAdvisor sits in the advisor chain, embeds the user's question, pulls the top-K most similar chunks from a VectorStore, and injects them into every prompt. Simple and reliable, but you pay the retrieval tokens on every call, even when the user just asks for order status.

Tool-based retrieval. Register retrieval as a @Tool and let the model decide when to search. This is what I run, because it keeps retrieval out of the hot path. The vector store is a plain bean:

@Bean
public VectorStore vectorStore(EmbeddingModel embeddingModel) {
    return SimpleVectorStore.builder(embeddingModel).build();
}

And semantic search is just another tool the model can choose to call:

@Tool(description = """
        Semantic product search: finds products by meaning rather than exact keywords.
        Use this for vague or vibe-based requests like 'something cozy for winter evenings'
        or 'a gift for someone who loves the outdoors'.""")
public List<ProductView> semanticSearchProducts(
        @ToolParam(description = "Natural-language description of what the shopper wants") String query,
        @ToolParam(required = false, description = "How many results to return (default 5)") Integer maxResults) {
    int topK = (maxResults == null || maxResults < 1) ? 5 : maxResults;
    return productSearchService.semanticSearch(query, topK).stream()
            .map(ProductView::of)
            .toList();
}

The trade-off is control. With an advisor you know retrieval always happens. With a tool you save tokens, but the model has to recognize when a vague request needs semantic search, which is exactly what a good tool description is for. I use the tool form and rely on descriptions to steer the model. If your use case demands guaranteed grounding, the advisor form is the safer default.

The Production Memory Checklist

If you take nothing else from this part, take this checklist. It is the difference between a demo agent and one you can leave running:

• Pick a storage backend that matches your deployment. In-memory MessageWindowChatMemory is fine for one instance. When you scale horizontally, switch to a shared backend; Spring AI ships JDBC-backed and Redis-backed ChatMemory implementations out of the box.
• Derive the conversation ID server-side. From your session or auth context, never from raw client input.
• Set the window per model. Put maxMessages in configuration and tune it against real traffic, not a guess.
• Pass the conversation ID to every stateful tool through toolContext, or users share state.
• Purge memory when sessions end, and add a TTL for abandoned ones. Conversation history is user data; treat it like one.
• Test memory explicitly. One test: the same conversation ID sees prior turns. Another: different IDs never see each other's turns. Both are quick integration tests and both catch real bugs.
• Watch your tool schemas. They are charged on every call, used or not. A lean schema set is a monthly saving.

What Comes Next

Part 1 gave the agent tools. This part gave it a memory and a context budget. Part 3 covers the remaining production concerns: streaming responses to the browser token by token with SSE, observability for tool calls and latency, and the multi-agent pattern where one agent delegates to another.

Have you hit the wall where your agent forgets? What did you do: bigger windows, rolling summaries, or a real conversation store? I read every response.

I write about Java, Spring Boot, and AI agents every week. Subscribe, it's free. Part 3 goes out soon.

If this was useful, bookmark it. The checklist at the end is the part you will reach for again six months from now, when the agent is in production and someone asks why the context bill doubled.