Spring AI Evals: how I test agent behavior

When building an AI agent, there is usually a moment when the prompt seems to work. But as development continues, this can quickly get out of control. Today the agent can call the right tool. Tomorrow, after a small system prompt change, it can stop calling it. Later, it can start calling it when it should not.

So I treat evals as regular regression tests for AI behavior.

In my project the example is small: a Spring AI agent must create a Markdown note through the create_note tool. The task looks simple, but there are already several places where the agent can fail:

• not call the tool when the user asks to create a note;
• call the tool when the user only asks to show or explain something;
• create an extra file;
• write a note without a Markdown heading;
• add extra advice or irrelevant content.

So I split the checks into a few layers.

Success Criteria

First I define what correct behavior means.

The agent should:

• call create_note when the user asks to create a new note;
• not call create_note when the user asks to read/delete/rename/explain;
• create only note.txt;
• put a Markdown heading on the first line;
• write short, relevant, and readable content.

This matters: without clear success criteria, an eval turns back into a subjective check.

Tool Definition

The tool is intentionally narrow. It does one thing: creates note.txt.

@Tool(
    name = "create_note",
    description = "Create a new Markdown note in note.txt. Use this only when the user asks to create a new note."
)
public String createNote(String title, String body) {
    var notePath = this.notesDirectory.resolve("note.txt");

    var content = "# " + normalize(title, "New Note")
            + System.lineSeparator()
            + System.lineSeparator()
            + normalize(body, "No content provided.")
            + System.lineSeparator();

    try {
        Files.createDirectories(this.notesDirectory);
        Files.writeString(notePath, content, CREATE, TRUNCATE_EXISTING);
        return "Created note.txt.";
    }
    catch (IOException ex) {
        return "Failed to create note.txt: " + ex.getMessage();
    }
}

Here name and description matter. They are not just documentation. For the model, they are a signal for when the tool fits and when it does not.

Agent Instructions

Next, I define the process goals clearly in the system prompt.

return this.chatClient.prompt()
        .system("""
            When the user asks to create a new note, use the create_note tool.
            The tool must create note.txt, put a Markdown heading on the first line,
            add the body text, and create no extra files.
            Keep the note concise and do not add unrelated suggestions.
            """)
        .user(message)
        .call()
        .content();

I am not trying to write a "beautiful prompt" here. I am fixing a contract:

• when to use the tool;
• which file must appear;
• what must not appear in the result.

Deterministic Checks

The first eval layer checks tool invocation. These are fast and clear checks: was the tool called or not?

@ParameterizedTest
@CsvSource(value = {
    "note-create-01|true|Create a new note about my plans for the week",
    "note-create-02|true|Make a markdown note with the title Retro and the body: what to improve",
    "note-no-trigger-01|false|Show me the existing notes",
    "note-no-trigger-02|false|Read the note.txt file",
    "note-no-trigger-03|false|Delete the note.txt file",
    "note-no-trigger-04|false|Draft the text of a note, but do not create a file"
}, delimiter = '|')
void evaluatesWhetherAgentCallsNoteTool(String id, boolean shouldTrigger, String prompt) {
    doReturn("Created note.txt.").when(this.noteTool).createNote(anyString(), anyString());
    clearInvocations(this.noteTool);

    var response = this.chatController.chat(prompt);

    assertThat(response).as(id).isNotBlank();

    if (shouldTrigger) {
        verify(this.noteTool, atLeastOnce()).createNote(anyString(), anyString());
    }
    else {
        verify(this.noteTool, never()).createNote(anyString(), anyString());
    }
}

This test does not argue with the model about text quality. It checks one concrete behavior.

If the test fails, the reason is clear:

• false negative: the user asked to create a note, but the tool was not called;
• false positive: the user did not ask to create a file, but the tool was called.

I like starting here because failures are easy to understand.

Tool Unit Test

I also test the tool itself without the model. This is a normal unit test, and it should be stable.

@Test
void createNoteWritesOnlyNoteFileWithMarkdownHeading() throws IOException {
    var noteTool = new NoteTool(this.tempDir);

    var result = noteTool.createNote("Meeting Notes", "Discuss the evaluation setup.");

    var notePath = this.tempDir.resolve("note.txt");
    var content = Files.readString(notePath);
    var lines = Files.readAllLines(notePath);

    try (var fileStream = Files.list(this.tempDir)) {
        var files = fileStream.toList();

        assertThat(result).isEqualTo("Created note.txt.");
        assertThat(notePath).exists();
        assertThat(content).isNotBlank();
        assertThat(lines.getFirst()).startsWith("#");
        assertThat(files).containsExactly(notePath);
    }
}

This separates two problems:

• if the unit test breaks, the problem is in the tool implementation;
• if the eval test breaks, the problem is in agent behavior, prompt, or model output.

Rubric-Based Grading

Deterministic checks are good at catching facts, but not everything can be checked with a simple assert.

For example, "the note is relevant", "there is no extra information", and "the text is easy to read" are qualitative requirements. For them, I use rubric-based grading.

In Spring AI this fits well into Evaluator: it receives an EvaluationRequest, runs a grader prompt inside, and returns an EvaluationResponse with pass, score, feedback, and metadata.

For the grader I keep the model fixed, ideally with a versioned model name instead of a floating alias. For example, I prefer gpt-5.5-2026-04-23 over gpt-5.5. If the provider updates a model alias, the grader can become stricter or more lenient without any code change. Then a score change may come from grader drift, not from agent behavior.

I also prefer low temperature for grading. The grader should not be creative; it should apply the same rubric in the same way.

For important rubric evals, I can also run the grader more than once. A single pass is useful, but a pass rate is more honest when the grader is also a model.

class NoteStyleEvaluator implements Evaluator {

    private static final List<String> REQUIRED_CHECK_IDS =
            List.of("title", "relevance", "simplicity", "readability");

    private final ChatClient chatClient;

    private final int minimumScore;

    NoteStyleEvaluator(ChatClient.Builder chatClientBuilder, int minimumScore) {
        this.chatClient = chatClientBuilder.build();
        this.minimumScore = minimumScore;
    }

    @Override
    public EvaluationResponse evaluate(EvaluationRequest evaluationRequest) {
        var rubricResult = gradeNote(evaluationRequest);

        if (rubricResult == null) {
            return new EvaluationResponse(false, 0.0f, "No rubric result", Map.of());
        }

        var pass = rubricResult.overallPass()
                && rubricResult.score() >= this.minimumScore
                && hasRequiredPassingChecks(rubricResult.checks());

        var metadata = Map.<String, Object>of(
                "overallPass", rubricResult.overallPass(),
                "checks", rubricResult.checks());

        return new EvaluationResponse(pass, rubricResult.score(), rubricResult.toString(), metadata);
    }

    private StyleRubricResult gradeNote(EvaluationRequest evaluationRequest) {
        var graderPrompt = """
            Read the original user request and the note.txt content,
            then grade the note against these requirements:

            - it has a clear title;
            - the content is relevant to the original user request;
            - there is no clearly unnecessary information;
            - the text is easy to read.

            Return only a structured JSON object with these fields:
            - overall_pass: boolean
            - score: integer from 0 to 100
            - checks: exactly four items with ids title, relevance, simplicity, readability.

            Each check must include id, pass, and notes.

            original user request:
            %s

            note.txt content:
            %s
            """.formatted(
                evaluationRequest.getUserText(),
                evaluationRequest.getResponseContent()
            );

        return this.chatClient.prompt()
                .user(graderPrompt)
                .call()
                .entity(StyleRubricResult.class);
    }

    private boolean hasRequiredPassingChecks(List<StyleRubricCheck> checks) {
        if (checks == null || checks.size() != REQUIRED_CHECK_IDS.size()) {
            return false;
        }

        var checkIds = checks.stream()
                .map(StyleRubricCheck::id)
                .toList();

        return checkIds.containsAll(REQUIRED_CHECK_IDS)
                && checks.stream().allMatch(check ->
                        check.pass() && check.notes() != null && !check.notes().isBlank());
    }

    record StyleRubricResult(
            @JsonProperty("overall_pass") boolean overallPass,
            int score,
            List<StyleRubricCheck> checks) {
    }

    record StyleRubricCheck(
            String id,
            boolean pass,
            String notes) {
    }
}

Here the model is used as a grader, but not in free-form mode. I ask for structured output so the result can be checked automatically.

The main benefit of Spring AI eval abstractions is that grading becomes a normal Java contract:

• EvaluationRequest keeps the original user prompt and generated output;
• Evaluator contains the grading logic;
• EvaluationResponse returns a machine-checkable result;
• metadata keeps details for each rubric check.

This makes grading useful for CI: there is pass, score, feedback, and a list of checks.

Rubric Eval Test

@ParameterizedTest
@CsvSource(value = {
    "weekly-plan-01|Create a new note about my plans for the week. Include a clear title and only a concise list of concrete weekly goals.|80",
    "weekly-plan-02|Create a markdown note for my weekly planning. Mention only work, health, and learning goals. Do not add extra categories.|80"
}, delimiter = '|')
void gradesCreatedNoteWithStyleRubric(String id, String prompt, int minimumScore) throws IOException {
    var agentResponse = this.chatController.chat(prompt);

    assertThat(agentResponse).as(id).isNotBlank();
    assertThat(this.notePath).as(id).exists();

    var noteContent = Files.readString(this.notePath);

    var evaluator = new NoteStyleEvaluator(this.chatClientBuilder, minimumScore);

    int graderRuns = 3;
    int requiredPassingRuns = 2;

    var results = IntStream.range(0, graderRuns)
            .mapToObj(run -> evaluator.evaluate(new EvaluationRequest(prompt, noteContent)))
            .toList();

    long passingRuns = results.stream()
            .filter(EvaluationResponse::isPass)
            .count();

    assertThat(passingRuns)
            .as(id + ": " + results)
            .isGreaterThanOrEqualTo(requiredPassingRuns);
}

This test checks more than "the tool was called". It checks the full outcome:

Here I still run the agent once. Only the grader runs multiple times against the same note.txt content.

This only measures grader stability. Agent stability is a separate metric: for that, I would run the whole agent flow multiple times and track agent pass rate separately.

prompt -> agent run -> note.txt -> rubric result -> score

Saving Results Between Versions

I also save eval results between runs. This helps me compare behavior after prompt, model, or rubric changes.

Pass/fail is not always enough. A test can still pass while the score slowly gets worse. That is why I keep the score, feedback, and rubric metadata as artifacts.

var resultPath = Path.of("build", "eval-results", id + ".json");

var resultJson = objectMapper.writeValueAsString(Map.of(
        "id", id,
        "prompt", prompt,
        "minimumScore", minimumScore,
        "graderRuns", graderRuns,
        "requiredPassingRuns", requiredPassingRuns,
        "passingRuns", passingRuns,
        "results", results
));

Files.createDirectories(resultPath.getParent());
Files.writeString(resultPath, resultJson);

This preserves the pass/fail result, score, feedback, and metadata needed to compare the current run with previous versions.

Eval Task

I keep AI evals separate from regular tests. They call the model API, so they are slower, more expensive, and less deterministic.

tasks.named('test') {
    useJUnitPlatform {
        excludeTags 'eval'
    }
}

tasks.register('evalTest', Test) {
    description = 'Runs AI agent evaluation tests.'
    group = 'verification'

    useJUnitPlatform {
        includeTags 'eval'
    }

    shouldRunAfter test
}

Run them separately:

./gradlew test
./gradlew evalTest

What This Gives Me

This makes agent development more predictable and easier to control.

I can change the system prompt, the @Tool description, model options, or the tool itself, and then check concrete signals instead of asking "does it feel better?":

• did false positives increase;
• did false negatives appear;
• is the expected artifact created;
• does the output pass the rubric;
• did the quality drop below the minimum score.

The most useful part is that the eval suite can grow gradually. Found a new miss - add a new eval case. The agent failed on a real user prompt - that prompt becomes a regression test. This way, behavior drift becomes visible instead of staying hidden in manual testing.