Originally published on AI Tech Connect.
What to measure, and why final-answer accuracy lies Most teams start evaluating an agent the way they evaluated a single LLM call: run the task, look at the final answer, mark it right or wrong. For a one-shot summariser that is reasonable. For an agent it is dangerous, because it scores only the last token of a journey that may have gone badly wrong on the way there. An agent run is a chain of decisions. At each step the agent chooses a tool, decides what arguments to pass, reads the result, and decides what to do next. A modest task — refund a customer, reconcile two ledgers, answer a policy question with citations — can involve a dozen such decisions. The failures are compositional: a single wrong argument early on cascades, a missing tool call leaves a gap the model papers over with a…
Top comments (0)