Building an AI Agent Harness in Java with Spring AI

In Part 1 we learned two ideas:

• Orchestration decides the next step. It's either a loop (the model decides) or a graph (your code decides).
• Agent = Model + Harness. The model is the brain; the harness is everything around it: orchestration, tools, memory, rules and logs.

Now let's see what that looks like in real Java code.

We'll build a tiny order-support agent with Spring AI, using the same example as before:

A customer asks: "Where's my order #123? If it's late, I want a refund."

---

The map: which code is the harness?

In Spring AI:

• The model is the ChatModel: Azure OpenAI, OpenAI, Anthropic or another provider. It only decides what to say, or which tool to ask for. It runs nothing itself.
• The harness is your Spring code: ChatClient, your @Tool methods, chat memory, advisors and the checks you write.

Let's build each piece.

---

Step 1: Tools (harness: the toolbox)

@Component
public class OrderTools {

    @Tool(description = "Get the status of an order by its ID")
    public String getOrderStatus(String orderId) {
        return orderService.findStatus(orderId);   // real DB/API call
    }

    @Tool(description = "Issue a refund for an order")
    public String issueRefund(String orderId, double amount) {
        if (amount > 100) {                          // guardrail
            return "Refund needs manager approval.";
        }
        return refundService.refund(orderId, amount);
    }
}

The model never runs this code. It can only ask for it, by name. The harness runs it and passes the result back.

The description matters: it's how the model knows when to use each tool. Treat it like documentation written for the model.

---

Step 2: Wire the harness together

@Configuration
public class AgentConfig {

    @Bean
    ChatClient supportAgent(ChatClient.Builder builder, OrderTools tools) {
        ChatMemory memory = MessageWindowChatMemory.builder()
                .maxMessages(20)                     // context management
                .build();

        return builder
            .defaultSystem("You are a support agent. Be brief and polite.")
            .defaultTools(tools)                     // register the toolbox
            .defaultAdvisors(
                MessageChatMemoryAdvisor.builder(memory).build(), // memory
                new SimpleLoggerAdvisor()                         // logs
            )
            .build();
    }
}

Every line here is harness:

• defaultSystem(...): the rules and instructions
• defaultTools(tools): the toolbox
• MessageWindowChatMemory: memory, capped at the last 20 messages so the context doesn't grow forever
• SimpleLoggerAdvisor: logs of each request and response

---

Step 3: The endpoint

@RestController
class SupportController {

    private final ChatClient agent;

    SupportController(ChatClient supportAgent) {
        this.agent = supportAgent;
    }

    @PostMapping("/chat")
    String chat(@RequestParam String sessionId, @RequestBody String message) {
        return agent.prompt()
            .user(message)
            .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, sessionId))
            .call()        // the loop runs inside here
            .content();
    }
}

The sessionId keeps each customer's conversation separate in memory.

---

Step 4: What actually happens inside .call()

1. Harness → sends prompt + tool list to the model
2. Model   → "call getOrderStatus(123)"
3. Harness → runs getOrderStatus("123") → "Shipped, arriving today"
4. Harness → sends the result back to the model
5. Model   → "Your order arrives today!" (no more tool calls)
6. Harness → no tool call, so the loop stops → returns the answer

Only steps 2 and 5 are the model. Everything else is the harness.

Part of the app	Model or harness?
The LLM (ChatModel)	Model
@Tool methods	Harness: toolbox
The loop inside .call()	Harness: orchestration
MessageWindowChatMemory	Harness: memory
defaultSystem(...)	Harness: instructions
amount > 100 check	Harness: rules
SimpleLoggerAdvisor	Harness: logs

---

Step 5: Take control of the loop

Spring AI hides the loop inside .call(). You can switch that off and write it yourself, which makes the harness visible:

ChatOptions options = ToolCallingChatOptions.builder()
        .toolCallbacks(ToolCallbacks.from(orderTools))
        .internalToolExecutionEnabled(false)   // "I'll run the loop myself"
        .build();

Prompt prompt = new Prompt(userMessage, options);
ChatResponse response = chatModel.call(prompt);

while (response.hasToolCalls()) {                               // the loop
    ToolExecutionResult result =
        toolCallingManager.executeToolCalls(prompt, response);  // run tools
    prompt = new Prompt(result.conversationHistory(), options); // feed back
    response = chatModel.call(prompt);                          // ask again
}
return response.getResult().getOutput().getText();

This while loop is exactly what .call() normally does for you. Owning it lets you add things like:

• a max-turns limit, so a confused model can't loop forever
• a cost cap
• an approval pause before a risky tool such as issueRefund

---

Bonus: the same agent as a graph

Everything above is a loop: the model decides which tool to call. What if the refund decision must never be the model's?

Spring AI doesn't include a graph engine, but a simple graph is just plain Java where your code chooses the path and the model only writes text:

public String handle(String orderId) {
    Order order = orderService.find(orderId);           // node: plain code

    if (!order.isLate()) {                               // edge: YOUR code decides
        return chatClient.prompt()
            .user("Write a short, friendly tracking update: " + order.tracking())
            .call()
            .content();                                  // node: LLM writes text
    }

    String offer = chatClient.prompt()
        .user("Draft a polite refund offer for order " + orderId)
        .call()
        .content();                                      // node: LLM writes text

    approvalService.requestApproval(orderId, offer);     // human-in-the-loop
    return "Your refund request is waiting for manager approval.";
}

No tools are registered here, so the model can't refund anything. It can only write the messages.

For bigger graphs with saved state, checkpoints and pause/resume, use a graph library such as LangGraph4j, or a workflow engine you already trust.

---

Cheat sheet

• ChatModel is the model. Everything else you write is the harness.
• .call() runs the tool loop for you. Turn it off with internalToolExecutionEnabled(false) to own the loop.
• @Tool methods are the toolbox. The model asks; your code runs them.
• Memory, system prompt, advisors and checks in your tools are the rest of the harness.
• Need a graph? Plain Java if/else for small flows; a graph library for big ones.

---

Next in this series: who builds the harness when you build agents on Azure: Copilot Studio vs Foundry Agent Service vs your own code.

What does your Spring AI harness look like? Let me know in the comments 👇

Code & Learn