A common interview mistake is to evaluate an AI agent by asking whether it produced a good answer once. That tests a demo, not a system.
A stronger answer explains how to define success, build an evaluation set, compare outputs against evidence, and monitor behavior in production. The interviewer is usually testing whether you can turn an open ended AI problem into a measurable engineering process.
Before naming metrics, clarify the agent's job. An agent that retrieves documents, calls tools, and completes a workflow has more evaluation dimensions than a single turn chatbot.
The interviewer is likely looking for four things:
- Can you define task success from the user's perspective?
- Can you separate intermediate behavior from the final answer?
- Can you design reliable test cases and review procedures?
- Can you detect regressions when prompts, models, tools, or retrieval settings change?
You do not need to list every possible metric. Connect each check to a specific failure mode.
Start with the intended outcome. An illustrative practice example is an agent that answers questions from an approved document set, selects a tool, and completes a multi step workflow.
Possible failures include:
- Unsupported claims
- Missing required information
- Incorrect tool selection
- Invalid tool arguments
- Repeated actions
- Failure to stop
- Unnecessary latency or cost
- Sensitive data exposure
This shows that evaluation begins with system behavior, not a metric dashboard.
Use real or carefully constructed examples covering normal requests, ambiguous requests, edge cases, and known incidents. Freeze a version of the set so that changes can be compared consistently.
For retrieval augmented generation, bind each expected claim to an approved source where possible. For tool use, record the expected tool, arguments, order of operations, and stopping condition.
The final answer matters, but it is not enough. Inspect the trajectory as well.
For every check, define a pass condition:
- Task completion passes only when all required steps are completed
- Tool use passes only when the selected tool and arguments match the case requirements
- Groundedness passes only when every factual claim maps to a retrieved source
- Citation alignment passes only when the cited source supports the associated claim
- Safety passes only when the response avoids the defined prohibited behavior
- Efficiency passes when the trajectory stays within the case's step, latency, or cost limit
For subjective qualities, use a rubric with clear criteria and illustrative examples. Human review can validate the rubric, while automated checks can run more frequently.
Freeze the reference documents and review important claims against them. Ask:
- Is the claim supported by the retrieved source?
- Does the source actually answer the question?
- Did the agent add details that are not present?
- Did it ignore conflicting or newer evidence?
- Can another reviewer reproduce the judgment?
If a factual claim cannot be mapped to an approved source, mark it unsupported and record the failure category. This makes evaluation more concrete than asking whether an answer feels good.
Offline tests do not cover every production condition. Log inputs, outputs, tool calls, errors, latency, and cost in a privacy aware way. Sample interactions for review and track new failure patterns.
When the system changes, rerun the frozen evaluation set and compare claim level, trajectory level, and task level results. Investigate regressions before accepting the new baseline.
A weak answer says, “I would use accuracy and user feedback.” That is too vague because it does not define accuracy, identify the reference, or explain how feedback maps to a failure mode.
A stronger answer says what success means, which examples represent the task, how outputs are compared to sources, and how regressions are caught.
This article was prepared with AI assistance and reviewed before publishing. The original lives on the free question bank:
Top comments (0)