Six months into our AI deployment, usage was climbing. Daily active users up, query volume up, sessions per user up. I was reporting these numbers to the executive team as evidence the deployment was working.
Then our head of finance asked a question that stopped me cold: "Can you show me one decision we made better because of AI?"
I could not. I had numbers showing people were using the tool. I had no evidence the tool was making anything better.
This is a trap that is easy to fall into because activity metrics are easy to collect and feel meaningful. They are not meaningless, but they measure inputs, not outcomes. An employee who generates three AI-assisted drafts per week that all require substantial rewrites is counted the same as an employee whose AI-assisted drafts go to approval with minor edits. Usage treats them identically. Value does not.
After that conversation, I rebuilt our measurement approach from scratch around four questions.
The first is time-to-first-draft for document categories where we could track it. Not total time to completion, because that includes editing and review cycles that the AI does not control. Specifically the gap between "work assigned" and "reviewable draft exists." This captures the productivity gain we were actually targeting, and it is measurable from existing project management data without any new instrumentation.
The second is revision rounds before approval. For any document going through a formal review process, how many rounds of changes were required? We expected AI-assisted drafts to require fewer rounds. For some document types that was true. For others we discovered the AI was introducing specific types of errors that reviewers consistently flagged, resulting in more revision rounds than unassisted drafts. We would never have found this without measuring it.
The third is the proportion of AI outputs used with minor modification versus substantially rewritten versus discarded. This is the metric I wish we had built from day one. It captures whether the AI is genuinely reducing work or shifting where work happens. Tracking it required a brief survey in our document management workflow, two questions added to the existing review process. We found that roughly 55% of AI outputs were used with light editing, 30% required substantial rewriting, and 15% were discarded and the work redone from scratch. That 15% represents pure time cost with no return. Understanding where it concentrated, which document types, which users, which query patterns, let us address the specific failure modes rather than treating the system as a monolith.
The fourth is trust calibration over time. Every quarter I run a simple survey: for the work you used AI for this week, how confident were you that the outputs were accurate before you verified them? And after verification, how often was that confidence justified? The gap between expected and realized accuracy tells you whether users are developing appropriate skepticism or whether trust is miscalibrated in either direction.
The finance question forced a discipline that the initial deployment lacked. Activity metrics feel good to report. Outcome metrics are harder to collect and less flattering when the numbers are honest. But they are the only ones that tell you whether the investment is actually working.
We reduced our reported metrics from eight numbers to four. The four we kept are all outcomes. None of them look as impressive as the usage numbers. All of them are more useful.
Top comments (0)