Most organizations measuring their AI deployments are measuring activity, not value. Usage rates, query volumes, active user counts, sessions per week. These metrics tell you whether people are using the tool. They tell you almost nothing about whether the tool is producing outcomes that justify the investment.
I want to describe the measurement framework I have built up through trial and error, because the path I took through wrong measurements to useful ones was longer than it needed to be and cost real decision-making quality along the way.
The measurement I started with and why it was wrong
We started with a simple question: are people using the tool? We tracked daily active users, weekly query volumes, and the percentage of employees who had used the tool at least once in the past thirty days.
These numbers looked good and climbed consistently in the first six months. I reported them to the executive team as evidence that the deployment was succeeding. Then a member of the finance team did something I had not done: she asked several employees directly what they were using the tool for.
The answers were instructive. Several people were using the tool to generate first drafts of documents they were going to rewrite almost completely anyway. Several were using it for tasks where the AI output was marginally useful but not substantially better than what they would have produced without it. A few were using it to look productive when they were not sure what to work on.
Usage was high. Value delivered was ambiguous.
The problem with activity metrics is that they measure the top of a funnel without measuring anything about the funnel's output. High usage of a tool that produces outputs requiring significant rework is not the same as high usage of a tool that reduces total work time. Lumping both together in a "users" metric obscures the difference.
The measurement shift that changed how I understood the deployment
The shift that produced useful information was moving from measuring tool activity to measuring workflow outcomes.
Instead of asking "how many people used the AI tool," I started asking "for the specific workflows where we deployed AI assistance, how have the time and quality characteristics of those workflows changed?"
This sounds obvious in retrospect. The reason I had not started there was that measuring workflow outcomes requires you to have baseline measurements before the deployment, which requires forethought that most deployments do not have.
We solved this retroactively by using historical data as a proxy baseline. For document workflows where we had timestamps, we could compare before-and-after cycle times. For tasks that produced outputs with trackable quality indicators, like proposals with win rates or reports with feedback scores, we could compare quality metrics before and after.
The specific metrics that proved most informative:
Time-to-first-draft for document categories where we could track this. Not the time saved in total but specifically the time from when the work was assigned to when a reviewable draft existed. This was meaningful because it captured the actual bottleneck we had been trying to address and was not confounded by how long editing took after the draft existed.
Error rates in specific high-volume tasks. For the operational workflows where we had automated AI assistance, we tracked error rates and compared them to the pre-AI baseline. This was the metric that caught a significant problem early: for one specific task type, the AI-assisted error rate was actually higher than the manual error rate, not lower. We would not have found this without measuring it.
Revision rounds required before approval. For documents going through internal review, we tracked whether AI-assisted drafts required fewer, the same, or more revision rounds before approval. The expectation was that AI drafts would require fewer revisions. For some document types this was true. For others, the AI introduced specific types of errors that reviewers consistently flagged, resulting in more revision rounds than unassisted drafts.
The metric that I wish I had built from the start
The most informative single metric I have found for enterprise AI deployments is not one that most organizations track: the proportion of AI outputs that are used with minor modification versus substantially rewritten versus discarded.
This metric captures whether the AI is actually reducing work or just shifting where the work happens. A tool where 70% of outputs require substantial rewriting before use is a tool that is consuming time in a different place, not a tool that is reducing total time spent.
Tracking this requires either user surveys or workflow instrumentation that records what happens to AI-generated content after it is produced. The instrumentation approach is more accurate but requires technical work upfront. The survey approach is faster to implement but suffers from self-reporting bias.
For any organization that is serious about understanding whether their AI deployment is delivering value, I would treat this metric as essential and design the measurement infrastructure before deploying, not after.
What to do when the measurements disappoint
The measurement framework I described will sometimes produce results that are worse than expected. This is the value of actually measuring rather than relying on impressions and activity metrics. The question is what to do with disappointing results.
The instinct is to adjust the measurement to make the numbers look better. Redefine "used with minor modification" to include outputs that were substantially edited. Focus reporting on the metrics that look good and de-emphasize the ones that do not.
The correct response is the opposite: treat disappointing measurements as diagnostic information and use them to identify specifically what is producing the disappointing result.
In my experience, disappointing AI deployment measurements trace to one of four specific causes. The AI is being used for the wrong tasks, ones where its capabilities are poorly matched to what the work actually requires. The underlying data quality is insufficient for the AI to produce good outputs on the tasks where it is being used. The prompting and configuration have not been tuned to the specific use cases of the organization. Or the deployment is in workflows where the speed improvement is real but the quality bar is high enough that the speed gain is offset by increased editing time.
Each of these has a specific solution. None of them is solved by not measuring.
The organizations that get sustained value from AI deployments are not the ones whose first measurements were impressive. They are the ones that measured honestly, identified the gaps between expectation and reality, and used that information to make specific changes. The measurement is not the destination. It is the navigation system that tells you whether you are going the right direction and how far you have to go.
Top comments (0)