The question nobody can answer in the room
There is a moment in most AI reviews where someone asks the only question that matters, and the room goes quiet. Not "is it clever," not "did the demo land," but "how do we know it works." What usually follows is a confident anecdote, a screenshot of a good run, and a change of subject. I have sat in that quiet more times than I would like, and I have learned to treat it as the single most important signal in the meeting. If a team cannot answer "how do we know" with a number and a method, they are not measuring their system. They are admiring it.
This is the evaluation gap: the distance between a thing that demos well and a thing you can actually trust in production. It is where a large share of enterprise AI quietly dies, not because the model was weak, but because nobody built the instrument that would have told them it was drifting until a customer did.
A demo is one run, production is the distribution
Start with the mental model, because everything downstream depends on it. A demo is a single run. It is one input, chosen by the person showing it, on a day the system happened to behave. Production is a distribution: thousands of runs, the inputs nobody rehearsed, the long tail of the strange and the adversarial. Judging an AI system by its demo is like judging a bridge by the one car that crossed it while the cameras were on.
The reason this matters so much for AI, more than for ordinary software, is that the failures are not loud. A traditional bug throws an error. A model that is subtly wrong returns a fluent, confident, plausible answer that is simply incorrect, and it does so without raising its hand. You do not get a stack trace. You get quiet erosion. The only way to see erosion is to measure the distribution, not the anecdote.
The three altitudes of measurement
I think about measurement at three altitudes, and a serious team runs all three.
The first is the offline evaluation: a fixed set of graded cases that you own. This is your golden set, a curated collection of inputs paired with what a good answer looks like, including the hard cases, the edge cases, and the ones that burned you before. You run every change against it and you watch the score move. The discipline here is ownership. If your only benchmark is a public leaderboard, you are measuring someone else's problem. The cases that matter are the ones drawn from your actual users, your actual data, your actual failure history.
The second altitude is online monitoring: what the system is doing right now, on live traffic, not what it did last week in a test harness. Offline evals tell you whether a change is safe to ship. Online monitoring tells you whether the world has changed underneath you since you shipped it. Both are necessary and neither substitutes for the other. A system that passed every offline case can still degrade in production because the inputs shifted, a dependency changed, or users found a path you never imagined.
The third altitude is the business outcome: did the number you promised actually move. Not tokens generated, not tickets touched, not "engagement," but the specific result the project existed to produce. Cost per resolved case. Cycle time to a finished draft. Revenue retained. This is the altitude executives care about and the one AI teams most often skip, because it is the hardest to attribute and the least flattering. It is also the only one that pays for the other two.
Keeping the judge trustworthy
Most teams quickly discover they cannot grade every output by hand, so they reach for a model to grade the model. Using an LLM as a judge is a genuinely useful technique, and it is also where a lot of measurement quietly goes wrong, because a judge that is biased or vague gives you numbers that feel rigorous and are not.
A few rules keep the judge trustworthy. Give it a precise rubric, not a vibe: define what a passing answer contains, not "rate this out of ten." Calibrate it against human labels on a sample, so you know its agreement rate before you trust it at scale. Watch for the well-documented tilts, such as favouring longer answers or preferring the first option presented, and control for them. And never let the same model both do the work and grade the work unchecked, because you have then asked it to mark its own homework. The judge is a measuring instrument. An instrument you never calibrate is decoration.
An evaluation is never finished
The last idea is the one that changes how you staff and budget this work. An evaluation is not a milestone you complete before launch. It is a living asset that decays. Models get updated. User behaviour shifts. The distribution of inputs moves, sometimes slowly, sometimes overnight when a new use case arrives. A golden set that was representative six months ago is now describing a world that no longer exists.
So the eval suite has to be tended, not shipped and forgotten. Every real incident becomes a new test case, so the system can never regress on the same failure twice. Every new capability arrives with the cases that prove it and the cases that bound it. This is unglamorous, continuous work, and it is precisely the work that separates the teams who ship on evidence from the teams who ship on hope and find out later.
What a leader insists on
I will not sign off on an AI system reaching production without these, and I have learned to ask for them early, because retrofitting measurement onto a live system is far harder than building it in.
A golden set that we own, drawn from our real users and our real failures, not a public benchmark. Distributional evaluation, many runs and adversarial inputs, not a single happy path. Online monitoring that watches live behaviour and alerts on drift, not a quarterly spot check. A judge, where we use one, that has been calibrated against human labels and audited for bias. And a business metric wired in from day one, so we are always answering the executive's real question and not a proxy for it.
None of this is exotic. It is the AI equivalent of a test suite, staffed and funded like one. The teams that treat it as optional are the teams that get surprised.
Where this goes
Capability is arriving faster than most organisations can absorb, and the temptation is to spend all of the attention on the model and none on the instrument that tells you whether the model is doing its job. That is exactly backwards. The model is a commodity that will be better next quarter no matter what you do. The evaluation harness, tuned to your data, your users, and your definition of a good outcome, is the durable asset, and it is the thing that lets you adopt the next better model with confidence instead of fear.
The interesting question in an AI review was never "is it impressive." It was always "how do we know it works, and how would we know if it stopped." A team that can answer that has closed the evaluation gap. A team that cannot has not yet started.
I am still refining my own version of this, and I would genuinely like to compare notes. If you are running AI against real systems, what are you measuring beyond the demo, and what has caught you by surprise.
I write a field-note series on shipping AI that actually works in production. The original of this piece, and the rest, live on my LinkedIn. Originally published on LinkedIn.
Top comments (0)