A user reports that your agent gave them wrong information. You check the logs — the tool calls look correct, the model returned a response, no exceptions were thrown. Everything ran, but the answer was bad.
This is the hardest problem in AI backend development. The agent doesn't crash when it's wrong. It confidently returns a plausible-sounding answer that happens to be incorrect, and your standard monitoring missed it entirely.
Evaluating AI output means building systems that can catch wrong answers before they reach users — or at least before they become a pattern. This article covers how to approach it practically.
Why traditional testing doesn't work
Unit tests verify deterministic behaviour. Call a function with input X, expect output Y. AI models are not deterministic — the same prompt can produce slightly different answers on different runs, and many correct answers exist for the same question.
You can't write:
assertEquals("24°C", agent.ask("What's the weather in Mysore?"));
The agent might say "It's 24 degrees and sunny" or "Currently 24°C with clear skies" — both correct, neither matching the expected string.
What you need instead: evaluation criteria, not expected strings. The question isn't "did the agent return exactly this?" but "did the agent return something that satisfies these conditions?"
The three types of checks
1. Structural checks — does the response have the right shape?
These are the easiest to automate. They don't require understanding the content.
@Test
void agentResponseShouldNotBeEmpty() {
String response = agent.chat("Plan a trip to Mysore");
assertThat(response).isNotBlank();
assertThat(response.length()).isGreaterThan(50);
}
@Test
void agentShouldMentionTemperature() {
String response = agent.chat("What's the weather in Mysore?");
assertThat(response).containsAnyOf("°C", "degrees", "temperature", "warm", "hot", "cool");
}
@Test
void agentShouldNotHallucinate() {
String response = agent.chat("What is the weather in Mysore?");
assertThat(response).doesNotContain("I don't know");
assertThat(response).doesNotContain("I cannot access");
}
These catch obvious failures: blank responses, refusals when the tool should have answered, missing key information.
2. Semantic checks — is the content correct?
These require either a known ground truth or a scoring mechanism.
For known ground truth (the answer is a fact you can verify):
@Test
void agentWeatherShouldMatchActualData() {
// inject a mock tool that always returns "Mysore: 24°C, clear"
String response = agentWithMockTools.chat("What's the weather in Mysore?");
assertThat(response).contains("24");
}
Using mock tools with fixed return values turns semantic checks into structural ones — you control the data, so you know what the answer should contain.
3. Consistency checks — does the agent behave the same way across runs?
Run the same prompt multiple times and compare outputs:
@Test
void agentShouldBeConsistentOnFactualQuestions() {
Set<String> responses = new HashSet<>();
for (int i = 0; i < 5; i++) {
responses.add(agentWithMockTools.chat("What's the weather in Mysore?"));
}
// All responses should mention 24°C since the mock always returns it
responses.forEach(r ->
assertThat(r).contains("24")
);
}
High variance on factual questions (where tool data is fixed) indicates a prompt problem — the model isn't reliably grounding its answer in the tool result.
Building a test dataset
Ad-hoc tests catch known problems. A test dataset catches regressions — problems you didn't know to look for.
Structure each test case with: input, expected properties, and what would count as failure.
public record EvalCase(
String input,
List<String> mustContain,
List<String> mustNotContain,
String description
) {}
A starter dataset for a travel agent:
List<EvalCase> evalDataset = List.of(
new EvalCase(
"What's the weather in Mysore?",
List.of("24", "°C"),
List.of("I cannot", "I don't know"),
"weather query should return temperature"
),
new EvalCase(
"How do I get from Bangalore to Mysore?",
List.of("bus", "train", "₹"),
List.of("I cannot"),
"travel query should return route and cost"
),
new EvalCase(
"Plan a 2-day trip to Mysore within ₹3,000",
List.of("₹", "Mysore", "day"),
List.of("I cannot", "unable"),
"trip plan should be structured and within budget context"
),
new EvalCase(
"What is the capital of France?",
List.of(),
List.of(),
"out-of-scope question — agent should handle gracefully"
)
);
Run the dataset:
@Test
void evalDatasetShouldPass() {
List<String> failures = new ArrayList<>();
for (EvalCase evalCase : evalDataset) {
String response = agentWithMockTools.chat(evalCase.input());
for (String must : evalCase.mustContain()) {
if (!response.toLowerCase().contains(must.toLowerCase())) {
failures.add(String.format(
"[%s] missing '%s' in: %s",
evalCase.description(), must, response.substring(0, Math.min(100, response.length()))
));
}
}
for (String mustNot : evalCase.mustNotContain()) {
if (response.toLowerCase().contains(mustNot.toLowerCase())) {
failures.add(String.format(
"[%s] found forbidden '%s' in: %s",
evalCase.description(), mustNot, response.substring(0, Math.min(100, response.length()))
));
}
}
}
assertThat(failures)
.withFailMessage("Eval failures:\n" + String.join("\n", failures))
.isEmpty();
}
This runs the full dataset as one test. Any failure prints the case description and the offending response. Add cases as you discover new failure modes in production.
Mock tools for evaluation
The hardest part of evaluating an agent in a test environment is that its tools call external APIs. Those APIs are slow, cost money, and return different values over time.
The solution: inject mock tools during evaluation.
Define an interface for your tools:
public interface TravelToolsPort {
String getWeather(String city);
String getAttractions(String city);
String getTicketPrice(String from, String to);
}
Create a fixed mock implementation:
public class MockTravelTools implements TravelToolsPort {
@Override
@Tool(description = "Get the current weather for a city in India.")
public String getWeather(@ToolParam(description = "City name") String city) {
return switch (city.toLowerCase()) {
case "mysore" -> "Mysore: 24°C, clear skies";
case "bangalore" -> "Bangalore: 22°C, partly cloudy";
default -> "Weather data not available for '" + city + "'.";
};
}
@Override
@Tool(description = "Get the top tourist attractions in a city.")
public String getAttractions(@ToolParam(description = "City name") String city) {
return switch (city.toLowerCase()) {
case "mysore" -> "Top attractions: Mysore Palace, Chamundi Hills, Brindavan Gardens";
default -> "No attraction data for '" + city + "'.";
};
}
@Override
@Tool(description = "Get the cheapest ticket price between two cities.")
public String getTicketPrice(
@ToolParam(description = "Departure city") String from,
@ToolParam(description = "Destination city") String to) {
if (from.equalsIgnoreCase("bangalore") && to.equalsIgnoreCase("mysore")) {
return "Cheapest: KSRTC bus ₹180, journey 3 hours";
}
return "No direct route found between " + from + " and " + to + ".";
}
}
Wire the mock into a test-only ChatClient bean:
@TestConfiguration
public class EvalConfig {
@Bean
@Primary
public ChatClient evalChatClient(ChatModel chatModel) {
MockTravelTools mockTools = new MockTravelTools();
return ChatClient.builder(chatModel)
.defaultSystem("You are a travel planning assistant for India.")
.defaultTools(mockTools)
.build();
}
}
Now eval tests run without hitting any external API. They're fast, free, and deterministic.
Tracking pass rate over time
A single test run tells you if something is broken now. A history of runs tells you if quality is drifting.
Log each eval run to a table:
@PostMapping("/eval/run")
public EvalResult runEval() {
int passed = 0;
int failed = 0;
List<String> failures = new ArrayList<>();
for (EvalCase evalCase : evalDataset) {
String response = evalChatClient.prompt()
.user(evalCase.input())
.call()
.content();
boolean pass = evalCase.mustContain().stream()
.allMatch(s -> response.toLowerCase().contains(s.toLowerCase()))
&& evalCase.mustNotContain().stream()
.noneMatch(s -> response.toLowerCase().contains(s.toLowerCase()));
if (pass) passed++;
else {
failed++;
failures.add(evalCase.description());
}
}
EvalResult result = new EvalResult(
Instant.now(), passed, failed,
(double) passed / (passed + failed), failures
);
evalResultRepository.save(result);
return result;
}
Run this endpoint after every deployment. If pass rate drops, a recent change broke something. If it drops gradually over days, your prompt or tool data is drifting against user expectations.
What to do when an agent gives a wrong answer in production
When a user reports a bad answer, the investigation follows the same path every time:
- Find the conversation ID in the logs
- Replay the tool calls from the log — did they return correct data?
- If tool data was correct, the model misused it — update the system prompt or tool description
- If tool data was wrong, fix the tool
- Add a new eval case covering this exact failure so it can't silently regress
Step 5 is the most important one. Every production failure is an opportunity to make your eval dataset stronger.
LLM-as-a-judge
String-based checks (mustContain, mustNotContain) are fast and cheap but limited. They can't tell whether an answer is helpful, accurate in tone, or logically sound. They only check whether certain words appear.
LLM-as-a-judge fixes this: you send the question and the agent's answer to a second model and ask it to score the response. The judge model reads the full answer and reasons about its quality — the way a human reviewer would.
@Service
public class LlmJudge {
private final ChatClient judgeClient;
public LlmJudge(ChatModel chatModel) {
this.judgeClient = ChatClient.builder(chatModel)
.defaultSystem("""
You are an evaluator for an AI travel assistant.
Given a user question and the assistant's response,
score the response on a scale of 1 to 5:
5 — complete, accurate, directly answers the question
4 — mostly correct, minor gaps
3 — partially answers but missing key information
2 — vague or unhelpful despite having the right data
1 — wrong, hallucinated, or refused without reason
Reply with ONLY a JSON object:
{"score": <1-5>, "reason": "<one sentence>"}
""")
.build();
}
public JudgeResult score(String question, String answer) {
String prompt = "Question: " + question + "\n\nAnswer: " + answer;
String raw = judgeClient.prompt()
.user(prompt)
.call()
.content();
try {
BeanOutputConverter<JudgeResult> converter =
new BeanOutputConverter<>(JudgeResult.class);
return converter.convert(raw);
} catch (Exception e) {
return new JudgeResult(0, "failed to parse judge response");
}
}
}
public record JudgeResult(int score, String reason) {}
Use it in your eval dataset runner alongside string checks:
for (EvalCase evalCase : evalDataset) {
String response = agentWithMockTools.chat(evalCase.input());
// existing string checks
boolean structuralPass = evalCase.mustContain().stream()
.allMatch(s -> response.toLowerCase().contains(s.toLowerCase()));
// LLM judge score
JudgeResult judgeResult = llmJudge.score(evalCase.input(), response);
log.info("[eval] case='{}' structural={} judgeScore={} reason='{}'",
evalCase.description(), structuralPass,
judgeResult.score(), judgeResult.reason());
if (judgeResult.score() < 3) {
failures.add(String.format("[%s] judge score %d: %s",
evalCase.description(), judgeResult.score(), judgeResult.reason()));
}
}
What LLM-as-a-judge catches that string checks miss:
- An answer that contains "24°C" but buries it in a confusing paragraph — string check passes, judge scores it 3/5
- An answer that invents a detail not in the tool data — structurally fine, judge flags the hallucination
- An answer that refuses when it should answer — structural check might not have a
mustNotContainfor this specific refusal phrasing, but the judge scores it 1/5
The cost trade-off: every judge call is a full model API call. For a dataset of 20 cases, that's 20 extra calls per eval run. Use string checks as the first pass — only run the judge on cases that pass string checks, or on a random sample of production traffic.
Which model to use as judge? The judge should be at least as capable as the agent, ideally more capable. If your agent uses a smaller model, judge with a larger one. The judge's output is only as reliable as its own reasoning ability.
Online evaluation — scoring live production traffic
Everything so far runs offline: in a test suite before deployment. Online evaluation runs in production, scoring real responses as they happen.
Why bother? Your eval dataset covers cases you anticipated. Real users ask things you never thought of. Online evaluation catches quality problems that your test cases missed.
The pattern: after the agent produces a response, score it before (or after) returning it to the user.
@PostMapping("/chat")
public ResponseEntity<String> chat(@RequestBody ChatRequest request) {
String requestId = UUID.randomUUID().toString();
long start = System.currentTimeMillis();
String response = callAgent(request.message(), request.conversationId());
// score asynchronously — don't block the user
CompletableFuture.runAsync(() ->
scoreOnline(requestId, request.message(), response)
);
return ResponseEntity.ok(response);
}
private void scoreOnline(String requestId, String question, String answer) {
try {
// fast structural check first
boolean hasContent = answer.length() > 30
&& !answer.toLowerCase().contains("i cannot")
&& !answer.toLowerCase().contains("i don't know");
if (!hasContent) {
log.warn("[eval:online] requestId={} structural=FAIL answer='{}'",
requestId, answer.substring(0, Math.min(80, answer.length())));
onlineEvalRepository.save(new OnlineEvalResult(
requestId, question, answer, false, 0, "structural fail"));
return;
}
// LLM judge for a sample (10% of traffic)
if (Math.random() < 0.10) {
JudgeResult judgeResult = llmJudge.score(question, answer);
log.info("[eval:online] requestId={} judgeScore={} reason='{}'",
requestId, judgeResult.score(), judgeResult.reason());
onlineEvalRepository.save(new OnlineEvalResult(
requestId, question, answer, true,
judgeResult.score(), judgeResult.reason()));
}
} catch (Exception e) {
log.warn("[eval:online] scoring failed for requestId={}: {}", requestId, e.getMessage());
}
}
Key design decisions in this pattern:
Score asynchronously. Never make the user wait for the evaluation. Fire it off with CompletableFuture.runAsync() and return the response immediately.
Sample for LLM judging. Running the judge on 100% of traffic doubles your model costs. Start with 10% — enough to detect quality trends without doubling the bill. Increase the sample rate temporarily after a deployment to validate the new version.
Always do structural checks. Structural checks are free (just string operations). Run them on 100% of traffic. Only invoke the judge on a sample.
Store results. Every scored response goes to a table. After a week of production traffic, you can query: "what's the average judge score this week vs last week?" and "which question types score lowest?"
@Query("SELECT AVG(judgeScore) FROM OnlineEvalResult " +
"WHERE judgeScore > 0 AND createdAt > :since")
double averageScoreSince(@Param("since") Instant since);
Combining offline and online:
| Offline eval | Online eval | |
|---|---|---|
| When | Before every deploy | On every live request |
| What it catches | Regressions vs known cases | Unknown failure patterns |
| Cost | Fixed (N cases × model call) | Proportional to traffic |
| Action | Block deploy if pass rate drops | Alert if average score drops |
The two complement each other. Offline eval is your gate — it prevents known regressions from reaching production. Online eval is your monitor — it catches new failure modes that your test dataset didn't anticipate.
Limits of automated evaluation
Automated evals catch structural and factual problems. They don't catch:
Tone and helpfulness — the answer is technically correct but unhelpful or rude. Requires human review.
Subtle hallucinations — the agent makes up a plausible detail not in the tool data. Hard to catch without comprehensive ground truth.
Edge case combinations — real users ask things your eval dataset never anticipated. Eval coverage grows over time, it's never complete.
Prompt injection in user input — a user sends "ignore previous instructions and...". Structural checks won't catch this; requires specific adversarial test cases.
The right mental model: automated evaluation is a regression net, not a quality guarantee. It tells you the agent isn't obviously broken. Human review for a sample of real conversations tells you whether it's actually good.
What's next
The agent is evaluated and monitored. The last pieces of the production puzzle are cost and efficiency — how to reduce what you spend on the model without reducing what users get. The next article covers cost optimisation: prompt caching, smaller models for simpler tasks, and reducing token waste.
Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.
Top comments (0)