Baseline for Spring AI Evals: catching silent degradation

Part 1: https://dev.to/lbobylev/spring-ai-evals-how-i-test-agent-behavior-2pgj

In the first part, the eval suite checked two questions:

• did the agent create note.txt;
• did the result pass rubric-based grading.

This is already better than manual checking, but this approach has a blind spot. A test can pass while quality slowly gets worse.

For example, the project has this eval case:

"weekly-plan-01|Create a new note about my plans for the week. Include a clear title and only a concise list of concrete weekly goals.|80"

If the grader returns score = 91, the test passes. If the score becomes 82 after a system prompt change, the test still passes because the minimum score is 80. But the agent behavior got worse.

That is exactly where a baseline helps.

What a baseline is

A baseline is the last accepted good result for an eval case.

Important: it is not just the previous run. The previous run could be bad, random, or produced during an experiment. A baseline is a result that I explicitly consider acceptable and want to use as the comparison point.

In this project, it is convenient to store baselines by eval case id:

src/test/resources/eval-baselines/weekly-plan-01.json
src/test/resources/eval-baselines/weekly-plan-02.json
src/test/resources/eval-baselines/weekly-plan-03.json

And write the current results of each run separately:

build/eval-results/weekly-plan-01.json
build/eval-results/weekly-plan-02.json
build/eval-results/weekly-plan-03.json

This separates two things:

• src/test/resources/eval-baselines - the accepted comparison point, which can be kept in git;
• build/eval-results - artifacts for a concrete run, which are useful to attach to a CI job.

What to compare

For agent evals, I compare signals:

• pass - whether the rubric passed;
• score - the numeric grader score;
• checks - which rubric checks passed or failed;
• feedback - the grader explanation;
• noteContent - the actual created note.txt, so the failure can be understood quickly.

In the simple version below, the regression gate uses only score and pass. The per-check results, feedback, and noteContent are retained for diagnosis.

A useful next extension is to gate on individual rubric checks too. For example, if simplicity passed in the baseline, the current run should not be allowed to fail simplicity just because the average score is still high enough.

Minimal result model

In the current project, NoteStyleRubricTests receives a single EvaluationResponse. For a baseline mechanism, I would first wrap the result in a custom record that is easy to save as JSON.

record NoteEvalResult(
        String id,
        String prompt,
        int minimumScore,
        boolean pass,
        float score,
        String feedback,
        Map<String, Object> metadata,
        String noteContent) {
}

If the grader runs once, this is enough. If I later add multiple grader runs, the record can be extended with averageScore, passingRuns, graderRuns, and the list of all responses.

Saving the current result

First, the current result should always be saved to build/eval-results. Even if the baseline comparison fails, the artifact remains on disk and can be inspected in CI.

private final ObjectMapper objectMapper = new ObjectMapper()
        .enable(SerializationFeature.INDENT_OUTPUT);

private void writeEvalResult(NoteEvalResult result) throws IOException {
    var resultPath = Path.of("build", "eval-results", result.id() + ".json");

    Files.createDirectories(resultPath.getParent());
    Files.writeString(resultPath, this.objectMapper.writeValueAsString(result));
}

This saves a snapshot of the current run.

Comparing with the baseline

Now we can load the baseline for the same id and check that quality did not drop beyond the allowed threshold.

private void assertNoRegressionAgainstBaseline(NoteEvalResult current) throws IOException {
    var baselinePath = Path.of(
            "src",
            "test",
            "resources",
            "eval-baselines",
            current.id() + ".json");

    if (Files.notExists(baselinePath)) {
        return;
    }

    var baseline = this.objectMapper.readValue(
            Files.readString(baselinePath),
            NoteEvalResult.class);

    var allowedScoreDrop = 5.0f;

    assertThat(current.score())
            .as(current.id() + " score regressed from baseline")
            .isGreaterThanOrEqualTo(baseline.score() - allowedScoreDrop);

    assertThat(current.pass())
            .as(current.id() + ": " + current.feedback())
            .isTrue();
}

There is an important detail here: if the baseline file does not exist, the test does not fail. This is useful when adding a new eval case. First the case is added, then after several runs I accept the result and add the baseline JSON.

How this looks in the test

The current gradesCreatedNoteWithStyleRubric can be extended without changing the core idea of the test.

@ParameterizedTest(name = "{0}")
@CsvSource(value = {
        "weekly-plan-01|Create a new note about my plans for the week. Include a clear title and only a concise list of concrete weekly goals.|80",
        "weekly-plan-02|Create a markdown note for my weekly planning. Mention only work, health, and learning goals. Do not add extra categories.|80",
        "weekly-plan-03|Write note.txt as a short weekly plan with a title and simple bullet points. Keep it limited to concrete weekly planning items.|80"
}, delimiter = '|')
void gradesCreatedNoteWithStyleRubric(String id, String prompt, int minimumScore) throws IOException {
    var agentResponse = this.chatController.chat(prompt);

    assertThat(agentResponse)
            .as(id)
            .isNotBlank();
    assertThat(this.notePath)
            .as(id)
            .exists();

    var noteContent = Files.readString(this.notePath);

    assertThat(noteContent)
            .as(id)
            .isNotBlank();

    var evaluator = new NoteStyleEvaluator(this.chatClientBuilder, minimumScore);
    var evaluationResponse = evaluator.evaluate(new EvaluationRequest(prompt, noteContent));

    var result = new NoteEvalResult(
            id,
            prompt,
            minimumScore,
            evaluationResponse.isPass(),
            evaluationResponse.getScore(),
            evaluationResponse.getFeedback(),
            evaluationResponse.getMetadata(),
            noteContent);

    writeEvalResult(result);
    assertNoRegressionAgainstBaseline(result);

    assertThat(evaluationResponse.isPass())
            .as(id + ": " + evaluationResponse.getFeedback())
            .isTrue();
    assertThat(evaluationResponse.getScore())
            .as(id)
            .isGreaterThanOrEqualTo(minimumScore);
}

The order is intentional:

• first the agent run;
• then grading;
• then writing the current result artifact;
• then comparison with the baseline;
• then the usual assertions against the minimum score.

Even if the baseline comparison fails, build/eval-results/<id>.json has already been written.

Baseline JSON example

The file src/test/resources/eval-baselines/weekly-plan-01.json can look like this:

{
  "id" : "weekly-plan-01",
  "prompt" : "Create a new note about my plans for the week. Include a clear title and only a concise list of concrete weekly goals.",
  "minimumScore" : 80,
  "pass" : true,
  "score" : 92.0,
  "feedback" : "StyleRubricResult[overallPass=true, score=92, checks=[...]]",
  "metadata" : {
    "overallPass" : true,
    "checks" : [
      {
        "id" : "title",
        "pass" : true,
        "notes" : "The note has a clear Markdown title."
      },
      {
        "id" : "relevance",
        "pass" : true,
        "notes" : "The note stays focused on weekly planning."
      },
      {
        "id" : "simplicity",
        "pass" : true,
        "notes" : "The note does not add unrelated advice."
      },
      {
        "id" : "readability",
        "pass" : true,
        "notes" : "The note is short and easy to read."
      }
    ]
  },
  "noteContent" : "# Weekly Plan\n\n- Work: finish the evaluation setup.\n- Health: schedule regular workouts.\n- Learning: review Spring AI evals.\n"
}

I do not assert exact noteContent. It is stored in the baseline as diagnostic information. If the score drops, I want to quickly see what the agent started writing.

When to update the baseline

The baseline must not be updated automatically after every run.

If it is, it stops being the "last good result" and becomes just the "latest result". Then the regression disappears on the next run because the bad result becomes the new normal.

The right workflow is:

1. Change the prompt, tool description, model options, or rubric.
2. Run ./gradlew evalTest.
3. Inspect failures and JSON artifacts in build/eval-results.
4. If the degradation is real, fix the prompt/tool/rubric.
5. If the new behavior is better or the change is intentional, update the baseline JSON manually.
6. Commit the baseline change together with the reason.

A baseline update should be an explicit engineering decision.

Handling nondeterminism

A single grader run can be noisy. For a simple demo this is fine, but for a more serious eval suite it is better to use several grader runs instead of one score.

Then the baseline stores an aggregate result:

record NoteEvalAggregateResult(
        String id,
        String prompt,
        int minimumScore,
        int graderRuns,
        long passingRuns,
        double averageScore,
        List<NoteEvalSingleRun> runs,
        String noteContent) {

    double passRate() {
        if (this.graderRuns == 0) {
            return 0.0;
        }
        return (double) this.passingRuns / this.graderRuns;
    }
}

record NoteEvalSingleRun(
        boolean pass,
        float score,
        String feedback,
        Map<String, Object> metadata) {
}

And the comparison becomes softer:

private void assertNoRegressionAgainstBaseline(NoteEvalAggregateResult current) throws IOException {
    var baselinePath = Path.of(
            "src",
            "test",
            "resources",
            "eval-baselines",
            current.id() + ".json");

    if (Files.notExists(baselinePath)) {
        return;
    }

    var baseline = this.objectMapper.readValue(
            Files.readString(baselinePath),
            NoteEvalAggregateResult.class);

    var allowedScoreDrop = 5.0;
    var allowedPassRateDrop = 0.20;

    assertThat(current.averageScore())
            .as(current.id() + " average score regressed")
            .isGreaterThanOrEqualTo(baseline.averageScore() - allowedScoreDrop);

    assertThat(current.passRate())
            .as(current.id() + " pass rate regressed")
            .isGreaterThanOrEqualTo(baseline.passRate() - allowedPassRateDrop);
}

This better reflects the nature of AI evals. One run can be accidentally weaker, but if average score or pass rate drops consistently, that is a regression.

What the baseline gives in this project

For the current Spring AI agent, the baseline answers questions that a regular minimumScore does not see:

• the note still passes the rubric, but did it become worse;
• did the agent start adding extra advice;
• did readability get worse after a system prompt change;
• did the grader start complaining about simplicity more often;
• did one eval case get worse after fixing another one.

In the end, the eval suite checks current quality, while the baseline shows the direction of change. These are different signals, and agent behavior needs both.