The operating problem
A 98% answer accuracy on a retrieval eval doesn't mean the agent completed the business task. I've watched teams ship agents that pass every eval and still fail in production. Accuracy is computed over token matches or rubric scores, not over API side effects, schema conformance, or downstream state changes. A customer-support agent can return the correct balance but fail to call the account-lookup tool with the right customer_id. The response is correct in isolation and useless in the workflow. Platform teams and governance leaders can't defend spend or set thresholds with model metrics alone.
The failure isn't hidden; it's unmeasured. A bank's support agent answers balance inquiries correctly but escalates 40% of Salesforce tickets because it lacks account context. An IT helpdesk agent passes every accuracy test but leaks PII in 3% of ServiceNow interactions. A procurement agent drafts purchase orders with 95% model accuracy, but 20% of those orders require manual rework in SAP because the emitted JSON uses vendor_id as a string while SAP expects an integer. In each case, technical metrics look fine, but the business process suffers. You need a scorecard that ties agent performance to business outcomes, not model outputs.
The architecture that holds up
A layered KPI taxonomy works because each layer has different failure modes and owners. Technical health covers p95 latency, tool-call error rate, and uptime. Task success covers completion rate, first-time resolution, and schema conformance. Business impact covers cost per resolution, rework rate, and revenue influence. Risk covers policy adherence, PII leakage, and hallucination rate in regulated outputs. Without layer separation, a single dashboard mixes leading and lagging indicators and hides who should act.
Layered KPI Scorecard for AI Agents
Explore how technical health, task success, business impact, and risk metrics interlock to form a holistic benchmark.
Not all agents are the same. A retrieval agent that answers questions from a knowledge base needs different KPIs than a transaction agent that updates records, and both differ from a decision support agent that recommends actions. Map each agent to the business process it serves, then select KPIs per task type. For retrieval, track answer relevance, citation precision, and groundedness. For transaction, track API success rate, idempotency violations, and data integrity checks. For decision support, track recommendation acceptance rate, override rate, and downstream error rate. This prevents the common mistake of using one KPI set for all agents, which hides poor performance in high-risk tasks.
KPI Selection by Agent Task Type
Compare retrieval, transaction, and decision support agents to select appropriate KPIs and governance focus.
Where teams usually fail
The recurring failure modes are: using model accuracy as a proxy for task success; excluding tool-call and rework costs from unit economics; measuring only completed tasks and ignoring abandonment and handoff quality; and treating compliance as a pre-deployment gate instead of a continuous control.
Consider the procurement agent scenario. Business operations measures cycle time and error rate in downstream SAP entries. They find that 95% model accuracy still causes 20% rework. The agent drafts purchase orders correctly, but the format doesn't match SAP's expected schema. The fix wasn't better prompt engineering; it was a schema validation step and a cost_per_clean_order metric that included rework. That's the difference between a model metric and a business metric. And that's where most teams get stuck: they measure the model, not the outcome.
How to measure progress
Cost per resolution is the metric that forces cross-functional accountability. Calculate it as (compute + tool calls + human review + rework) / resolved tasks. Don't amortize fixed costs into the first pilot; track variable cost per resolution separately. Compare against fully loaded human cost, not just handle time. If your agent costs $2.40 per resolved ticket and a human agent costs $1.80, you have a problem, even if the agent's success rate is higher. This single number aligns platform teams, finance, and business operations.
Cost-per-Resolution Breakdown
See how compute, tool calls, human review, and rework add up to total cost per resolved task, and compare against human baseline.
Escalation rate and handoff quality are leading indicators of user trust and operational load. Track escalation rate by intent, not aggregate. A 10% escalation on password reset is different from 10% on wire transfer. Track not just how often the agent escalates, but what context it passes to the human. A clean handoff with full conversation history reduces human handling time. A messy handoff forces the human to start over, erasing any efficiency gain. For compliance, measure policy adherence rate, PII leakage incidents, explainability score, and change management logs. These need continuous monitoring, not a one-time pre-deployment check. Link to Red Cards in Agentic AI for handling policy violations, and to Agentic AI Governance: Managing Model Drift for drift detection.
Benchmarking cadence matters. Run a pre-deployment baseline on a golden set, then a shadow deployment with 5% traffic and no side effects, then post-deployment continuous evaluation with drift detection on prompt embeddings and tool schemas. Review thresholds monthly for high-risk agents, quarterly for low-risk. This isn't a one-time exercise; it's an operating rhythm.
What to build next
The dashboard should have four tabs, one per layer, with owners and thresholds. The CFO sees cost per resolution and rework rate; the support director sees escalation rate and first-time resolution; the governance lead sees policy adherence and PII leakage. If cost per resolution drops 30% but compliance incidents rise from 0.2% to 0.4%, the governance board must decide whether the tradeoff is acceptable. Don't bury that in a model accuracy chart.
Build the operating model around continuous evaluation. Instrument agents to emit structured traces: tool calls, input/output hashes, schema validation results, and human handoff context. Store them in an append-only audit log. Run drift detection on prompt embeddings and tool schemas weekly. Assign owners per layer and set alert thresholds: p95 latency > 2s, escalation > 15%, PII leakage > 0.1%. Review monthly for high-risk agents, quarterly for low-risk. That turns the scorecard from a report into a control loop. Link to Beyond Black Boxes for explainability and audit instrumentation. And when an agent drifts, you'll catch it before it becomes a headline. That's the difference between a pilot and a production system.
Top comments (0)