The pilot-to-production gap in AI projects gets discussed as if it were mysterious, or organisational, or a matter of change management. It is mostly none of those. It is five missing engineering artefacts, and they are the same five almost every time.
A demo runs on curated data, handles the happy path, and is judged by a sympathetic audience in a meeting. A production system handles malformed input, upstream schema changes, adversarial users, latency budgets, cost ceilings, model deprecations, and the specific edge case that represents four percent of volume and eighty percent of your support tickets.
Here is what to build from the first commit so the second thing is reachable from the first.
1. An evaluation harness
This is the one that matters most and gets skipped most.
You need a held-out set of representative examples and an automated method of scoring output against them. Not a notebook where someone eyeballs twenty results — a scored, repeatable, version-controlled comparison you can run on every change.
Without it, every subsequent decision about prompts, model versions, retrieval strategy, chunking, or thresholds is made on anecdote. Teams end up in an endless loop of subjective adjustment where "is it better now?" is genuinely unanswerable, because these systems fail in ways that are invisible without measurement.
Build the evaluation set before building the system. It forces you to define what correct means, which is frequently where the real disagreement surfaces — and it is far better to surface it in week one than in month five.
2. A real data connection
Pilots run on CSVs. Production runs on a pipeline.
The gap is not the extraction — it is everything the manual extract silently removed. Nulls someone dropped. Encoding problems someone fixed. Records that failed a filter nobody documented. Schema variations across the three years of history the sample did not span.
Connect to the actual source early, even at reduced volume. Discovering that eleven percent of production records have a field format the pilot never saw is a week-two problem or a month-five problem, and it is exactly the same problem either way.
3. Explicit failure behaviour
What does the system do when the model returns malformed output? When the provider times out? When input exceeds the context limit? When retrieval returns nothing relevant?
A demo has no answer because these did not happen. A production system needs a defined answer for each, and the answer is often "escalate to a human" — which requires a path for that to happen, and someone on the other end of it.
Define the confidence threshold below which the system declines to act. Systems without a decline path produce confident wrong output on inputs they should have refused, and that failure mode is more damaging than an outright error because nothing signals it.
4. A cost model
Token costs at a hundred requests a day tell you nothing about ten thousand.
Instrument cost per request from the beginning, broken down by component — retrieval, generation, re-ranking, any evaluation running in production. Then project to realistic volume before committing to an architecture.
This routinely changes design decisions. A pattern that makes three model calls per request is fine at demo volume and can be untenable at scale, and discovering that after the architecture is settled is expensive. Caching strategy, model tier selection, and whether to re-rank at all are all cost decisions disguised as quality decisions.
5. An owner
Not an engineering artefact, and it kills more pilots than the other four combined.
Systems built by a team that then disperses have no one to handle the model deprecation notice, the gradual quality drift, the new input pattern that started appearing in March. Name the owner before launch, not after.
The drift problem specifically
One production concern has no demo equivalent and deserves separate attention.
These systems degrade without erroring. The input distribution shifts. An upstream system changes format. The provider updates the underlying model. Output quality drops and nothing raises an alarm, because from the infrastructure's perspective everything is healthy — requests are being served, latency is fine, error rate is zero.
Uptime monitoring cannot see this. You need quality monitoring: run the evaluation set on a schedule against production configuration, track the score over time, and alert on degradation. This is why the evaluation harness is not merely a development tool. It is a production control, and it is the reason skipping it early is so hard to recover from.
What this costs
Roughly thirty percent more than a demo.
That is the entire trade. Thirty percent more up front against a rewrite when the demo needs to become real — and the rewrite is where second-year AI budgets tend to disappear, usually while the team is also trying to deliver the next thing.
The wider context — engagement models, use-case selection, build versus buy, governance, and cost ranges — is at techcirkle.com/blog/ai-business-consulting. Most of what makes this work in practice is LLM integration architecture: retrieval, evaluation, caching, and fallbacks.
Frequently Asked Questions
What is an evaluation harness for an AI system?
A held-out set of representative inputs with expected outputs or scoring criteria, plus an automated way to run the system against them and produce a comparable score. It is version-controlled and runs on every change. Without one, decisions about prompts, models, retrieval, and thresholds are made on anecdote, and quality becomes unmeasurable rather than merely uncertain.
How do you detect AI model quality drift in production?
Run your evaluation set on a schedule against the production configuration and track the score over time, alerting on degradation. Standard uptime and error-rate monitoring cannot detect drift, because a degrading model still returns well-formed responses quickly. This is why the evaluation harness functions as a production control rather than only a development tool.
Why does connecting to real data early matter so much?
Because manual extracts silently remove exactly the problems production will surface — dropped nulls, fixed encodings, filtered records, schema variation across years the sample did not span. Connecting to the actual source at reduced volume surfaces these in week two rather than month five, at which point the architecture has usually settled around assumptions that no longer hold.
What should an AI system do when it is uncertain?
Decline and escalate, which requires a defined confidence threshold and a human path on the other side. Systems without a decline path produce confident wrong output on inputs they should have refused. That failure mode is more damaging than an outright error because nothing signals it — the output looks exactly like a correct one.
How much more does building a pilot properly cost?
Roughly thirty percent more than a demo, covering the evaluation harness, a real data connection, explicit failure behaviour, and cost instrumentation. The comparison is against rewriting the system when the demo needs to become production, which typically costs considerably more and happens while the team is also committed to delivering the next thing.
Top comments (0)