DEV Community: Wasim Sheikh

Ship Gate: The Pre-Deploy Checklist Most AI Features Skip

Wasim Sheikh — Thu, 17 Sep 2026 15:35:12 +0000

Most AI features don't fail in the model.

They fail at the gate.

Someone demos a shiny prompt. Leadership loves it. It ships. Then a customer pastes something weird, an agent calls the wrong tool, or a "helpful" answer invents an offer that never existed. The postmortem is always the same: we optimized for the demo, not for production.

I call the missing step Ship Gate — a short, boring checklist you run before any AI feature leaves the sandbox. Not a 40-page risk memo. A gate you can clear in an afternoon.

Quick use case

Situation. A product team shipped an AI email summarizer into customer-facing replies. Staging looked clean. Leadership wanted it live before the quarter closed.

What broke. Day one, the model invented a discount a customer had never been offered. Support scrambled. There was no eval set of "known trap" emails, no human gate on outbound AI text, and no kill switch short of a full deploy rollback.

The fix (Ship Gate).

Threat model — named "hallucinated commercial offers" as a top failure mode with an owner
Data boundary — outbound copy restricted to template fields; free-form claims blocked
Eval set — trap cases that fail the build if the model invents offers or leaks PII
Human override — outbound AI copy held until a human approves (or stricter template mode)
Observability — one bad reply reconstructable in minutes: prompt, context, output, cost

Then they added the ops layer every Ship Gate implies: canary traffic, a one-click kill switch, and a documented rollback owner before the flag flipped on.

Ship Gate isn't bureaucracy. It's the minimum checklist so an AI feature earns production — instead of learning in front of customers.

Why demos lie

Demos are curated. You pick the happy path. You soft-prompt around the edge cases. You never paste a raw customer email with secrets still in it.

Production is the opposite. Users paste junk. Models invent confident nonsense. Tools fire with the wrong args. Logs capture more than you think.

If your only test was "it looked good in the meeting," you didn't ship AI — you shipped a vibe.

Ship Gate exists to make that gap visible before customers feel it.

The five gates

Run these in order. Fail any one and you don't ship. Pass all five and you can sleep.

1. Threat model (ten minutes, not a committee)

Before code, write three sentences:

What can go wrong? (wrong answer, tool misuse, data leak, prompt injection, cost runaway)
Who gets hurt? (customer, employee, your brand, a regulator)
What's the blast radius? (one chat vs. every tenant)

If you can't name the top three failure modes, you're not ready to build — you're ready to be surprised.

2. Data boundary

Answer out loud:

What data can the model see?
What can it write?
What must never leave this boundary (PII, secrets, customer documents, internal tickets)?

Then enforce it in code, not in a slide. Strip secrets before the model. Scope tools to the minimum. Prefer retrieval over stuffing the whole corpus into context. If the model doesn't need a field, don't give it the field.

3. Eval set before polish

Pick 20–50 real-ish cases: happy path, hostile prompts, empty input, long paste, "ignore previous instructions," the ticket that always breaks things.

Score each on three axes:

Correct enough for the job
Safe enough (no leak, no unsafe action)
Useful format (what the user actually needs)

Ship Gate rule: you don't tune the UI until the eval set is green enough that you'd trust a teammate to use the feature unsupervised.

4. Human override

Every autonomous or semi-autonomous path needs an escape hatch:

Confirm before irreversible actions (send email, delete, charge, deploy)
Log who approved what
Make "stop / undo / escalate" obvious

If the only recovery path is "hope the model was right," you don't have a product. You have a liability with a chat box.

5. Observability that a human can read

When it breaks at 2 a.m., can you answer:

What did the user ask?
What context did the model get?
Which tools ran, with which args?
What did we return?
Cost and latency for that turn?

If the answer is "we'd have to dig through JSON for an hour," fix the logs before you grow the feature. Ship Gate isn't complete until a tired engineer can reconstruct a single bad turn.

A one-page Ship Gate card

Gate	Pass criteria
Threat model	Top 3 failure modes named + owner
Data boundary	See / write / never-leave documented + enforced
Eval set	≥20 cases; safe + useful bar met
Human override	Irreversible actions confirmed; stop path clear
Observability	One bad turn reconstructable in <5 minutes

No green across the board → no ship.

The test

Ask one question before any AI feature goes live:

Which of the five gates is still red — and who owns fixing it?

If nobody owns the red gate, you don't have a launch plan. You have a calendar date.

I'm Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production.

Canonical: https://sheikhwasim.com/insights/ship-gate-pre-deploy-checklist/

The 5-Layer Stack Behind Agents That Ship

Wasim Sheikh — Mon, 14 Sep 2026 15:59:29 +0000

Originally published on my site: https://sheikhwasim.com/insights/agent-architecture-five-layer-stack/

Most "AI agents" fail for the same reason.

Someone wires a language model to a handful of tools, demos a happy path, and calls it architecture. It works in the slide. It collapses the first time production is messy timeouts, bad inputs, spend spikes, irreversible side effects.

Here's the five-layer stack that actually ships.

Quick use case

Situation. A team built an internal "support triage agent." Demo day looked sharp: read a ticket, call search, suggest a reply, update CRM status.

What broke. In production, one vague ticket sent the agent into a 40-step tool loop. It wrote the wrong CRM status, and spend spiked past the monthly budget in an afternoon. Chat logs showed answers — not which tool ran, with what inputs, or why it kept going. The CRM write couldn't be cleanly reversed.

The fix (mapped to the stack).

Orchestrator — hard max steps + stop when confidence is low
Tools — typed CRM update with auth scope + idempotency key
Traces — every plan / tool call / result / cost on one correlation ID
Guardrails — daily spend cap + deny list for irreversible tools without a human gate
Rollback — compensating "revert status" action when the write was wrong

Same model. Different system. That's the difference between a demo and something teams can trust.

Layer 1 — Orchestrator

The orchestrator decides the next step. It is not the model dumping text forever.

Good orchestrators:

Tools are clear APIs, not mystery side effects.

Ship-ready tools have:

Auth and scoped credentials
Typed inputs and validated outputs
Timeouts, rate limits, and idempotency keys
Explicit success / failure contracts

A tool that "usually works" is a liability. Prefer boring interfaces over clever ones.

Layer 3 — Traces

Every step logged so you can see what it did — and why.

Traces are how you debug, audit, and improve:

Prompt / plan / tool call / result / decision
Timing and cost per step
Correlation IDs across services
Redaction for secrets and PII

If you can't replay the path, you can't trust the system in an enterprise.

Layer 4 — Guardrails

Budget, permissions, PII, and stop conditions — before it runs wild.

Guardrails belong in the control plane, not in a polite system prompt:

If it's wrong, you reverse it. Idempotent actions beat clever guesses.

Design for undo:

Prefer reversible writes
Snapshot before mutate
Compensating transactions when undo isn't free
Clear "blast radius" for every tool

An agent that can't roll back isn't a system. It's a demo with confidence.

The test

Ask one question of any agent stack:

Can it show traces, and can it roll back?

If either answer is no, keep it out of production.

Closing

I'm Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production.

Follow for practical AI architecture that ships.
Site: https://sheikhwasim.com · Notes: https://practicalainotes.substack.com/ · X: https://x.com/anciwasim

Spend and token caps
Allow / deny lists for tools and data
Human gates for irreversible actions
Policy checks on inputs and outputs

Prompts are guidance. Guardrails are enforcement.

Layer 5 — Rollback

Choose among plan act verify → stop
Bound loop length and retry policy
Separate "thinking" from "committing"
Fail closed when confidence or evidence is weak

If your system can't explain why it took the next action, you don't have an orchestrator — you have a chat session with plugins.

DEV Community: Wasim Sheikh

Ship Gate: The Pre-Deploy Checklist Most AI Features Skip

Quick use case

Why demos lie

The five gates

1. Threat model (ten minutes, not a committee)

2. Data boundary

3. Eval set before polish

4. Human override

5. Observability that a human can read

A one-page Ship Gate card

The test

The 5-Layer Stack Behind Agents That Ship

Quick use case

Layer 1 — Orchestrator

Layer 3 — Traces

Layer 4 — Guardrails

The test

Closing

Layer 5 — Rollback

Layer 2 — Tools