A model that answers correctly in a Jupyter notebook has cleared maybe 20% of the work. I've watched teams treat a good demo as the finish line, ours included in the early days, then spend the next two months finding everything the demo quietly skipped. Empty inputs. A user pasting 40 pages into a box built for one sentence. The model returning valid-looking JSON that's missing a field the code assumes is always there.
'Production-ready' turned vague the moment every framework started shipping a chatbot template. Here's the checklist we actually run before an AI feature gets near real users.
The checklist we actually use
It fails on purpose, safely. Every model call has a timeout, a retry with backoff, and a defined behavior when the provider returns a 500 or a rate limit. If OpenAI or Anthropic has a bad ten minutes, the feature degrades to a polite message instead of a stack trace.
Outputs get validated, never trusted. We schema-check every response before it touches the database or the screen. A model asked for a category that returns something off-list gets caught, not stored.
There's a logged trail. Prompt version, input, output, latency, tokens, and cost, per call. When a user says it gave them something strange on Tuesday, we can actually find Tuesday.
Cost has a ceiling. A per-user and per-day cap that trips before the invoice does. I've seen one retry loop turn a $40-a-day feature into a $900 surprise by morning.
Someone can switch it off. A flag that disables the AI path and falls back to the old flow with no redeploy. If the model starts misbehaving, the fix should take thirty seconds.
The part nobody demos
Evaluation is where 'ready' is won or lost. Before launch we build a set of 50 to 200 real inputs with known-good answers, and every prompt change runs against it. We do this because without it, 'I improved the prompt' is just a feeling. A tweak that helps one case often breaks three others, and skipping the eval means your users are the ones who find out.
We also decide, out loud, what the model may never do alone. Send money, delete records, email a customer. Those stay behind a human click. A confident wrong answer is the default failure mode of these systems, so the blast radius has to be small by design.
Ready means you can walk away
The AI features I trust in production are the ones I can stop watching. They log everything, cost a predictable amount, fall into a safe fallback, and keep a human on the actions that matter. None of that shows up in a demo, which is why demos oversell.
At Shanti Infosoft most of our launch delays come from this list, not from the model. The model was usually fine in week one. Making it safe to leave running unattended is the other six weeks. We write more about how we ship AI at https://shantiinfosoft.com.
The demo-to-production gap is the whole story in why a working AI demo is the problem, not the finish line.
What's the one item on this list your last AI launch skipped, and did it come back to bite you?
Top comments (0)