Asked in a GenAI Engineer interview.
What they are really testing:
Whether you can turn a vague idea of a "good answer" into a repeatable evaluation pipeline that also covers retrieved context and agent memory, not just a single accuracy number.
A strong answer:
- Start with a golden dataset of representative queries, reference answers or rubrics, and the expected source chunks or memory items. Include adversarial cases such as stale facts, conflicting memories, and questions that should be refused.
- Evaluate each component separately. For retrieval, measure context precision and recall against labeled relevant chunks. For generation, measure faithfulness (claims supported by the supplied context), answer relevance, and correctness against references.
- For agents with memory, check what actually entered the context window: was the right memory retrieved, was it recent enough, and did truncation or summarization drop a key fact?
- Use LLM-as-judge for scale, but calibrate it against human labels on a sample, pin the judge model and prompt version, and report agreement rates. Prefer deterministic checks (schema validation, exact match, citation presence) wherever they apply.
- Run offline evals in CI on every change to prompts, models, or chunking, then score a sample of live traffic online and track drift, user feedback, and escalation rates.
- Tie metrics to business outcomes such as task success rate and cost per resolved task, so a higher faithfulness score is not mistaken for a better product.
Common mistakes:
- Relying on one aggregate score or on a handful of manual spot checks, which hides regressions in specific query types.
- Using the same model as both generator and judge without calibration, which inflates scores through shared biases.
- Evaluating only the final answer, so retrieval misses, stale memory, and context overflow get misdiagnosed as model hallucinations.
How would you structure the evaluation for an agent that remembers past sessions, and which failure would you check for first?
Originally shared on LinkedIn.
An AI interview question every day, leadership for tech people and hands-on AI learning:
Top comments (0)