Your AI feature passed the demo.
That is not evidence it will survive Monday morning.
Demos are curated. Someone picks the happy path, soft-prompts around the edges, and never pastes the weird customer email that invents a discount, leaks a ticket ID, or loops a tool until the bill spikes. Production does all of that before lunch.
I call the missing test set smoke evals — a short, fixed pack of trap cases that must pass before any AI change leaves the sandbox. Not a research benchmark. Not a vibe check. A deploy gate.
If Ship Gate is the checklist, smoke evals are the part that actually fails the build.
Quick use case
Situation. A platform team shipped an “AI ops assistant” that could summarize incidents and suggest runbook steps. Staging looked sharp. Leadership wanted it in the on-call channel before a busy release week.
What broke. Night one, a noisy alert chain made the model invent a rollback command that did not exist in the runbook. An engineer almost ran it. There was no frozen set of “known trap” incidents, no assert that refused invented commands, and no CI job that would have blocked the merge.
The fix (smoke evals).
- Freeze ~20 cases — real-ish incidents: missing runbook steps, ambiguous alerts, PII in tickets, prompt-injection style pastes, “do nothing” noise
- Assert two things — safe (no invented commands, no secret leakage) and useful (still names the right service / severity when the answer is known)
- Wire to CI — every prompt, tool schema, or model bump runs the pack; red means no ship
- Own the failures — each failing case gets an owner and a fix path (prompt, tool contract, or human gate)
- Refresh monthly — add the newest production scare; retire cases that no longer teach
Same model. Different gate. The next invented rollback never reached the channel.
Smoke evals aren’t bureaucracy. They’re the smallest set of cases that would have caught the last three failures — run on every change.
Why demos lie (again)
A demo answers: Can we make it look smart once?
A smoke eval answers: Does it refuse the bad path, every time, on this exact change?
Those are different questions. Most teams only ask the first.
The second is cheaper than a postmortem. Twenty cases, a few dollars of inference, a red CI job. Compare that to an almost-run false rollback at 2 a.m.
The smoke-eval stack (five layers)
Build the pack like a system, not a spreadsheet someone opens once.
1. Case bank (frozen, versioned)
Keep the cases in git. Each case has:
- Input — the ticket, chat, alert, or document paste
- Expected behavior — refuse / escalate / answer with X
- Failure mode tagged — hallucination, tool misuse, PII leak, loop, overconfidence
If it isn’t in version control, it isn’t a gate. It’s a wish.
2. Dual assert: safe and useful
Safe-only evals train the model to say nothing. Useful-only evals ship confident nonsense.
Every case should check both:
- Safe — no invented SKUs, commands, prices, or secrets
- Useful — when the ground truth is known, the answer still helps
A pack that only measures “didn’t explode” will ship a polite brick.
3. CI as the gate
Run the pack on:
- Prompt / system-message changes
- Tool schema changes
- Model version bumps
- Retriever / RAG index refreshes that affect answers
Red build = no merge. No “we’ll check it later in staging” exception for the pack that exists because staging lied.
4. Failure ownership
A failing case without an owner becomes noise. Assign:
- Who owns the failure mode
- Whether the fix is prompt, tool contract, retrieval, or human override
- When it must be green again
Smoke evals without ownership rot into ignored red.
5. Monthly refresh from production scars
Every real incident that almost (or did) hurt customers becomes a candidate case. Add it. Keep the pack near twenty so it stays fast. Retire cases that no longer teach.
The pack should smell like your last three scares — not a generic LLM leaderboard.
How this maps to what you already have
- Ship Gate — gate #3 (eval set) is where smoke evals live; the other four gates still matter
- 5-layer agent stack — traces tell you which case failed and why; guardrails + rollback catch what the pack missed
Smoke evals don’t replace architecture. They make the architecture earn the next deploy.
The test
Ask one question before the next AI ship:
Do we have ~20 versioned trap cases that assert safe and useful — and does CI fail if they fail?
If either half is no, you are still shipping on demo confidence.
Originally published on sheikhwasim.com. I’m Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on.
Connect: LinkedIn · Practical AI Notes · X @anciwasim
Top comments (0)