DEV Community: Rıfat Çakır

Mocking Spring AI @Tool calls via record/replay

Rıfat Çakır — Wed, 12 Aug 2026 12:36:06 +0000

The first question about testing a Spring AI agent is "how do I avoid calling the model." The second one, the moment tools enter the picture, is scarier:

When my test makes the agent call refundCustomer(...) or sendEmail(...) or deleteOrder(...) — does the real method run? Will my test suite issue a real refund, send a real email, or write to a real database, on every CI run?

With most testing approaches the honest answer is "you have to make sure it doesn't, manually, and hope nobody forgets." This is about making the safe answer the default.

Two clarifications up front. First, this isn't a hand-written mock or a stubbed ChatModel — it records a real model-and-tool exchange once and replays it deterministically. Second, unlike Ruby's VCR or WireMock, which record at the HTTP socket, this intercepts at Spring AI's own ToolCallingManager bean — above the network layer, where Spring AI actually runs your tools.

How tool calling actually works in Spring AI

A tool call isn't one request — it's a loop:

Your code sends a prompt, with tools registered.
The model replies "I want to call getOrderStatus("ORD-4471")."
Spring AI's ToolCallingManager runs your real @Tool method and feeds the result back to the model.
The model produces the final answer using that result.

So a naive recording of just the final answer is useless — you've lost the middle. And if you replay the model turns but let step 3 run for real every time, your side effects fire on every replay, forever. Both failure modes are exactly what you don't want in a test.

How the isolation actually works (no magic): the library replaces the Spring-managed ToolCallingManager with a thin wrapper, VcrToolCallingManager. On a cassette hit it returns the recorded (tool name, arguments) → result directly and never reaches the dispatcher that would invoke your @Tool; on a miss it delegates to the real manager, lets the tool run once, and records the arguments/result pair. That is the entire trick — interception at the same bean Spring AI already uses to run tools, not bytecode magic or a proxy around your method.

TL;DR — the default is already safe

Tool isolation is on by default; you don't configure anything to get it:

spring:
  ai:
    test:
      vcr:
        enabled: true
        mode: RECORD_OR_REPLAY
        tool:
          mode: REPLAY_FROM_CASSETTE   # default — full isolation

@SpringBootTest
class RefundAgentTest {

    @Autowired ChatClient.Builder chatClientBuilder;
    @Autowired RefundTools refundTools;   // has the real @Tool methods

    @Test
    void agentRefundsUsingTheTool() {
        String answer = chatClientBuilder.build().prompt()
            .user("Refund order ORD-4471, it arrived damaged.")
            .tools(refundTools)
            .call().content();

        assertThat(answer).contains("refund");
        // On replay: the recorded getRefundStatus/issueRefund result is injected.
        // The real @Tool body NEVER runs — no charge, no email, no DB write.
    }
}

First run records two fixtures — the model's tool request and, separately, the tool's recorded arguments/result pair. Every run after replays both, offline, and the real tool method's body never executes.

One caveat worth stating here, not buried at the end: this interception lives on the Spring-managed ToolCallingManager bean, so it only applies inside a Spring context. Use @SpringBootTest and you get isolation; a hand-built ChatClient.builder(model) outside the context has no such bean, nothing is wrapped, and the real @Tool runs.

Configuring model replay vs. tool mocking

There are two independent switches, and that separation is the whole design:

spring.ai.test.vcr.mode — governs the model call (record/replay the conversation).
spring.ai.test.vcr.tool.mode — governs the tool invocation:
- REPLAY_FROM_CASSETTE (default): on a cassette hit, the recorded (tool name, arguments) → result is returned directly. The real @Tool body never executes. A side-effecting tool fires at most once, ever, per distinct arguments — when it was first recorded.
- EXECUTE_REAL: the real tool runs on every call. Use it for the one test that specifically wants to assert the real method was invoked, with the right arguments, the right number of times:

@Test
@VcrTool(mode = VcrToolMode.EXECUTE_REAL)   // opt-in, this test only
void theRealRefundMethodIsInvokedOnce() { ... }

@VcrTool is the same escape-hatch shape as @Vcr — you loosen isolation for exactly one test without weakening it for every other test in the same run.

Testing the agent loop vs. unit-testing domain logic

With isolation on, this verifies the model calls the right tool with the right arguments, and your code handles the tool's result correctly. That's a contract test of the agent loop, and it's the part that's genuinely hard to test any other way.

It does not test your @Tool method's own business logic — the refund calculation, the DB write, the email formatting. That's a plain unit test of that method, with no model involved. Two concerns, two tests.

When the tool changes: stale fixtures

The tool fixture is keyed by the tool name and the arguments the model called it with. So if a signature change alters the call — a new required parameter, a renamed tool — the key changes, and in REPLAY_ONLY (your CI mode) that is a loud cache miss, not a silent pass: the build fails and tells you to re-record. Drift in the call is caught.

What is not caught automatically: if the method's behaviour changes while the name and arguments stay identical — same call, different real result — replay keeps returning the old recorded result until you re-record. This is true of every record/replay tool, VCR included, and it is why fixtures are committed and reviewed in pull requests: a stale result is a visible diff when you re-record, not a hidden runtime surprise.

Try it

A complete, offline example — a tool that records a side effect, proven not to fire on replay under isolation and to fire under EXECUTE_REAL — is in the companion repo (ToolSideEffectIsolationTest, ToolCallingRecordReplayTest). Every test runs with no Docker and no network.

Independent, community project (not affiliated with Spring/Broadcom), Apache-2.0, 0.1.0, tested against Java 21 · Spring Boot 4.0.0 · Spring AI 2.0.0.

Your Spring AI tests are slow, flaky, and cost money. Here's how to make them deterministic.

Rıfat Çakır — Tue, 11 Aug 2026 10:07:48 +0000

You wire up Spring AI, the ChatClient fluent API feels great, your feature works. Then you sit down to write a test — and every good option is bad.

A test that calls a real model is:

Slow. A hosted call is a second or two; run it on every branch and it adds up. Run inference locally (Ollama + Testcontainers) and a cold call is ~47 s. Either way it's not a loop you run on save.
Expensive. Every run of every test is billable tokens — times every developer, times every CI job.
Unrepeatable. The same prompt can answer differently tomorrow. A test that asserts on model output is flaky by construction.
Untestable in CI. No GPU, no model container, and putting a provider API key in a pipeline to run unit tests is a security problem, not a testing strategy.

The usual workarounds all hurt: Mockito means hand-building Spring AI's nested ChatResponse → Generation → AssistantMessage graph and asserting against a response you wrote; WireMock/MockWebServer means owning each provider's exact wire JSON, SSE frames, and tool-call envelopes, and rewriting it all when you switch providers; the real model is the four problems above, accepted rather than solved.

There's a well-worn answer from the HTTP world — Ruby's VCR, Python's vcrpy: record the real interaction once, replay it deterministically after. The catch is those work at the HTTP layer, and Spring AI's value is the abstraction above HTTP. So I built the same idea where Spring AI actually lives.

TL;DR — the whole integration

One dependency:

<dependency>
    <groupId>io.github.rifatcakir</groupId>
    <artifactId>spring-ai-test-tools</artifactId>
    <version>0.1.0</version>
    <scope>test</scope>
</dependency>

One property (src/test/resources/application-test.yml):

spring:
  ai:
    test:
      vcr:
        enabled: true
        mode: RECORD_OR_REPLAY   # REPLAY_ONLY in CI

Your test doesn't change at all — you write it exactly as you would against a real model:

@SpringBootTest
class OrderStatusTest {

    @Autowired ChatClient.Builder chatClientBuilder;

    @Test
    void answersAQuestionAboutTheOrder() {
        String answer = chatClientBuilder.build().prompt()
            .user("What's the status of order ORD-4471?")
            .call().content();

        assertThat(answer).contains("shipped");
    }
}

First run reaches a real model and writes src/test/resources/llm-cache/{sha256}.json — you commit that file. Every run after replays it in under a millisecond, offline.

Record once. Replay forever.

FIRST RUN          slow · costs tokens · needs network
  Your test ──▶ ChatClient ──▶ Real LLM  ──writes──▶  cassette.json  (committed)

EVERY RUN AFTER    instant · $0 · fully offline
  Your test ──▶ ChatClient ◀──reads──  cassette.json                 (~0.8 ms)

The advisor attaches to every ChatClient.Builder in the context via ChatClientBuilderCustomizer — so nothing under test, and nothing in production, knows the cache exists. In CI you seal it with mode: REPLAY_ONLY: now a cache miss is a loud failure, not a silent call to a live model. The cache key is an exact SHA-256 over the canonical request; there is no fuzzy matching, ever. (This is why Spring AI's production semantic cache doesn't solve the testing problem — it matches on similarity thresholds, which is exactly backwards for a test.)

What you actually get

The point isn't a benchmark number — it's what disappears:

No network call on replay. Zero HTTP requests (asserted by a request counter in the suite) — no latency, no timeouts, no rate limits, no flakiness.
No tokens. Zero spend, every run, forever.
Runs in a keyless, GPU-less CI. The thing that was impossible becomes the default.
Deterministic. The same committed response, the same assertion, every run.

And yes, replay is ~0.8 ms (median over 200 timed iterations in a real Spring context) versus a warm hosted call of ~1–2 s or a local cold call of ~47 s — but treat that as a side effect. The real win is that the network, the cost, and the rate limits are simply gone.

When this is the wrong tool

Up front, because senior engineers rightly distrust silver bullets — this sits above the HTTP layer, so it cannot test that layer:

Retry/backoff, timeouts, a 429 with Retry-After, connection pooling, a body arriving malformed mid-stream → that's WireMock/MockWebServer, and they're the right tool.
Anything that isn't a model call → Mockito, as always.
Proving the integration really works against a real provider → a genuine integration test before you ship. This doesn't replace that; it replaces running it on every commit.

It isn't just plain text

Each of these is verified against a real model, not assumed:

Tool / function calling. A @Tool call's name and arguments are part of the cache key, and on replay the recorded tool result is injected without re-running the real method — so a test can't accidentally write to your database or send an email. When you do want the real method to run, there's an opt-in mode.
Streaming. A Flux<ChatResponse> replays chunk-for-chunk — not a single-chunk fake — tool-call fragments included.
Structured output. An .entity(MyDto.class) call's target schema is part of the cache key, so two output types with the same prompt never collide.
Embeddings. EmbeddingModel calls cache independently of chat; a replayed vector is exactly, not approximately, what was recorded.
Spring AI's own evaluators. RelevancyEvaluator / FactCheckingEvaluator run deterministically in CI (the judge call itself is recorded), or live on demand for a drift check.

Try it

Repo: https://github.com/rifatcakir/spring-ai-test-tools
Docs: https://rifatcakir.github.io/spring-ai-test-tools
Runnable examples: https://github.com/rifatcakir/spring-ai-test-tools-example

Independent, community project (not affiliated with Spring/Broadcom), Apache-2.0, currently 0.1.0 and early — tested against Java 21 · Spring Boot 4.0.0 · Spring AI 2.0.0. If you try it, issues and feedback are genuinely wanted.