DEV Community

Rıfat Çakır
Rıfat Çakır

Posted on • Originally published at Medium

Your Spring AI tests are slow, flaky, and cost money. Here's how to make them deterministic.

#java #springboot #ai #testing

You wire up Spring AI, the ChatClient fluent API feels great, your feature works. Then you sit down to write a test — and every good option is bad.

A test that calls a real model is:

  • Slow. A hosted call is a second or two; run it on every branch and it adds up. Run inference locally (Ollama + Testcontainers) and a cold call is ~47 s. Either way it's not a loop you run on save.
  • Expensive. Every run of every test is billable tokens — times every developer, times every CI job.
  • Unrepeatable. The same prompt can answer differently tomorrow. A test that asserts on model output is flaky by construction.
  • Untestable in CI. No GPU, no model container, and putting a provider API key in a pipeline to run unit tests is a security problem, not a testing strategy.

The usual workarounds all hurt: Mockito means hand-building Spring AI's nested ChatResponse → Generation → AssistantMessage graph and asserting against a response you wrote; WireMock/MockWebServer means owning each provider's exact wire JSON, SSE frames, and tool-call envelopes, and rewriting it all when you switch providers; the real model is the four problems above, accepted rather than solved.

There's a well-worn answer from the HTTP world — Ruby's VCR, Python's vcrpy: record the real interaction once, replay it deterministically after. The catch is those work at the HTTP layer, and Spring AI's value is the abstraction above HTTP. So I built the same idea where Spring AI actually lives.

TL;DR — the whole integration

One dependency:

<dependency>
    <groupId>io.github.rifatcakir</groupId>
    <artifactId>spring-ai-test-tools</artifactId>
    <version>0.1.0</version>
    <scope>test</scope>
</dependency>
Enter fullscreen mode Exit fullscreen mode

One property (src/test/resources/application-test.yml):

spring:
  ai:
    test:
      vcr:
        enabled: true
        mode: RECORD_OR_REPLAY   # REPLAY_ONLY in CI
Enter fullscreen mode Exit fullscreen mode

Your test doesn't change at all — you write it exactly as you would against a real model:

@SpringBootTest
class OrderStatusTest {

    @Autowired ChatClient.Builder chatClientBuilder;

    @Test
    void answersAQuestionAboutTheOrder() {
        String answer = chatClientBuilder.build().prompt()
            .user("What's the status of order ORD-4471?")
            .call().content();

        assertThat(answer).contains("shipped");
    }
}
Enter fullscreen mode Exit fullscreen mode

First run reaches a real model and writes src/test/resources/llm-cache/{sha256}.json — you commit that file. Every run after replays it in under a millisecond, offline.

Record once. Replay forever. 

FIRST RUN          slow · costs tokens · needs network
  Your test ──▶ ChatClient ──▶ Real LLM  ──writes──▶  cassette.json  (committed)

EVERY RUN AFTER    instant · $0 · fully offline
  Your test ──▶ ChatClient ◀──reads──  cassette.json                 (~0.8 ms)
Enter fullscreen mode Exit fullscreen mode

The advisor attaches to every ChatClient.Builder in the context via ChatClientBuilderCustomizer — so nothing under test, and nothing in production, knows the cache exists. In CI you seal it with mode: REPLAY_ONLY: now a cache miss is a loud failure, not a silent call to a live model. The cache key is an exact SHA-256 over the canonical request; there is no fuzzy matching, ever. (This is why Spring AI's production semantic cache doesn't solve the testing problem — it matches on similarity thresholds, which is exactly backwards for a test.)

What you actually get

The point isn't a benchmark number — it's what disappears:

  • No network call on replay. Zero HTTP requests (asserted by a request counter in the suite) — no latency, no timeouts, no rate limits, no flakiness.
  • No tokens. Zero spend, every run, forever.
  • Runs in a keyless, GPU-less CI. The thing that was impossible becomes the default.
  • Deterministic. The same committed response, the same assertion, every run.

And yes, replay is ~0.8 ms (median over 200 timed iterations in a real Spring context) versus a warm hosted call of ~1–2 s or a local cold call of ~47 s — but treat that as a side effect. The real win is that the network, the cost, and the rate limits are simply gone.

When this is the wrong tool

Up front, because senior engineers rightly distrust silver bullets — this sits above the HTTP layer, so it cannot test that layer:

  • Retry/backoff, timeouts, a 429 with Retry-After, connection pooling, a body arriving malformed mid-stream → that's WireMock/MockWebServer, and they're the right tool.
  • Anything that isn't a model call → Mockito, as always.
  • Proving the integration really works against a real provider → a genuine integration test before you ship. This doesn't replace that; it replaces running it on every commit.

It isn't just plain text

Each of these is verified against a real model, not assumed:

  • Tool / function calling. A @Tool call's name and arguments are part of the cache key, and on replay the recorded tool result is injected without re-running the real method — so a test can't accidentally write to your database or send an email. When you do want the real method to run, there's an opt-in mode.
  • Streaming. A Flux<ChatResponse> replays chunk-for-chunk — not a single-chunk fake — tool-call fragments included.
  • Structured output. An .entity(MyDto.class) call's target schema is part of the cache key, so two output types with the same prompt never collide.
  • Embeddings. EmbeddingModel calls cache independently of chat; a replayed vector is exactly, not approximately, what was recorded.
  • Spring AI's own evaluators. RelevancyEvaluator / FactCheckingEvaluator run deterministically in CI (the judge call itself is recorded), or live on demand for a drift check.

Try it

Independent, community project (not affiliated with Spring/Broadcom), Apache-2.0, currently 0.1.0 and early — tested against Java 21 · Spring Boot 4.0.0 · Spring AI 2.0.0. If you try it, issues and feedback are genuinely wanted.

Top comments (0)