DEV Community: Wasim Sheikh

Boundary Contract: Make Every Agent Tool Fail Boring

Wasim Sheikh — Tue, 06 Oct 2026 22:12:34 +0000

Cross-post of Insights #8 — canonical: https://sheikhwasim.com/insights/boundary-contract-tools-fail-boring/

Most agent failures I see in production don't start in the model. They start at the edge of a tool.

A call times out. An API returns an empty list. A search gives back three results out of thirty. The model gets something shaped like data, so it does what models do: it writes a fluent, confident answer on top of it.

The transcript reads fine. The customer gets the wrong answer. Nobody notices until a human asks why.

I call the missing control a Boundary Contract: every tool the agent can call returns a typed outcome, not just a payload. Ok, empty, partial, timeout, rejected. The agent has to handle each one by name. It can't turn a broken call into the truth.

Clever prose can wait. The boundary has to be boring first.

Quick use case

Situation. A field-service company shipped a scheduling agent. A customer asks for a repair visit. The agent checks the technician calendar, offers the next open slot, and books it. Staging looked sharp.

What broke. On a busy Monday the calendar service started timing out under load. The tool wrapper caught the timeout and returned an empty list, because that's what the original developer thought "safe" meant. The agent read "no slots" and told customers the next opening was three weeks out. Some cancelled. Some called a competitor. The logs showed a successful tool call, a polite reply, and no error anywhere. The dashboard stayed green all morning.

The fix (Boundary Contract).

Typed outcomes on every tool. The calendar tool now returns ok, empty, partial, timeout, or rejected, plus the payload. "Empty" and "timed out" stopped looking the same.
Input checked before the call. A bad zip code or a missing service type gets rejected with a reason, before it reaches the calendar API and comes back as a mystery.
A named policy per outcome. timeout means retry once, then tell the customer "I can't see the calendar right now" and offer a callback. Never "no slots."
Partial results are surfaced. If one region's calendar answered and another didn't, the agent says so instead of quietly offering only what came back.
The outcome goes in the trace. Every run logs which outcome each tool returned, so "how many timeouts did we hide this week" has a real answer.

Same model. Same prompt. The difference was that a broken call could no longer pass as an answer.

Why "it returned something" isn't enough

A tool payload answers: What came back?

A Boundary Contract answers: What actually happened at the edge, and what is the agent allowed to say about it?

Most stacks only fund the first question. They wrap errors into empty results to "keep the agent from crashing," and the model fills the gap with confidence. That isn't resilience. It's a lie with good grammar.

The Boundary Contract (five layers)

Build it into the tool layer, not into a line of the system prompt that says "be careful with errors."

1. Input contract

Each tool declares what valid input looks like: required fields, allowed values, sane ranges. Invalid input is rejected before the call, with a reason the agent can repeat or fix. A tool that accepts anything will eventually do anything.

2. Outcome types, not just payloads

Every response carries one of a small, fixed set of outcomes. Five is enough for most tools: ok, empty, partial, timeout, rejected. The important part is that empty and failed are never the same value.

3. Time budget, named

Each tool has an explicit timeout and retry rule written down next to it. When the budget runs out, the outcome says timeout. It doesn't silently become empty, and it doesn't hang until the whole run dies.

4. Outcome policy in the orchestrator

For each outcome, decide what the agent may do and say. partial means disclose the gap. timeout means fall back or hand off to a human. rejected means ask a clarifying question. Write these as rules the orchestrator enforces, so the model doesn't improvise them per ticket.

5. Outcome in the trace

Log the outcome for every tool call, not only the payload. Freeze one happy path and one timeout path as Golden Traces, and add "timeout must never produce a confident answer" to your Smoke Evals.

How this maps to what you already have

5-layer agent stack. Boundary Contract lives in Tools, enforced by Guardrails, recorded in Traces.
Receipt Gate. Receipt Gate checks a write landed before the next one fires. Boundary Contract makes sure the agent knows when a read didn't land.
Permission Envelope. The envelope decides whether a tool may run. The contract decides what its result means.
Cheap Twin. A partial or timeout outcome is a good escalate line on the checklist.
Ship Gate. Inject a timeout in canary and prove the agent says "I can't see that right now" instead of making something up.

The test

Pick the tool your agent calls most. Then ask:

If this tool timed out right now, would the agent tell the user something true, and would the trace show it was a timeout?

Run it for real: block the endpoint in staging and read the reply. If the agent still answers confidently, the boundary isn't a contract yet. It's a hope.

Closing

Agents don't need to be clever at the edges. They need to be honest there. Every tool should fail in a way that's boring, typed, and visible, so the clever parts of the system are built on something true.

I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept, production.

Follow for practical AI architecture that ships.

Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Practical AI Notes · X: @anciwasim

Clean Exit, Wrong Record: Grade the Agent on the Database, Not the Last Line

Wasim Sheikh — Sun, 04 Oct 2026 21:16:50 +0000

Cross-post of an Insights piece. Canonical: https://sheikhwasim.com/insights/clean-exit-wrong-record/

An agent can do the careful work, close the ticket, and still be wrong.

On October 3, Microsoft and Hugging Face published ThinkingBox. It grades agents on the records they leave behind, not the sentences they generate. Their walkthrough is an ordinary support case: a delayed kitchen appliance, nine tool calls, the policy read correctly, the ticket closed as resolved, and a polite closer. The required end state was on hold. The customer never got a real answer. A grader watching tool calls would pass it. The database wouldn't.

That's not a leaderboard story. It's the bug already sitting in production agents that treat a clean exit as proof of work.

Receipt Gate says the last write has to leave a skim-worthy receipt before the next one fires. ThinkingBox is the colder half of the same rule: the receipt has to match the record. The model's last line doesn't get a vote.

Quick use case

Situation. A support team shipped a delivery-exception agent. A stuck shipment comes in. The agent can read the order, check the carrier, open or update a ticket, and reply to the customer. The eval was a golden transcript: the right tools fired, the reply was polite, and the last line said the case was resolved. Staging looked done.

What broke. Put that published retail task on their stack. Tools return success. The ticket lands in solved. The customer gets the closer. The carrier exception is still open, so the required end state is hold, not solved. Nothing in the eval looks at that field. The run is green because the trajectory is tidy. The record isn't.

The ThinkingBox writeup is blunt about how often that tidy exit is a costume. In a common-set ablation (121,680 valid trials across 12 models), 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error. The checks still found wrong field values in 77.61% of them, unintended extra effects in 43.30%, and missing required effects in 25.36%. Those findings overlap. A green trace is not a passed state.

The fix.

Name the end state in fields: status, required effects, forbidden extras. Not "the reply sounds resolved."
Read the record back after the last mutating tool, from the system of record, not from the model's summary.
Latch the customer-visible step on that readback. No send until the fields match.
Receipt the diff: what the record shows, what it was supposed to show, pass or fail in one skim.
Repeat the golden task from a clean start until a one-off pass stops counting as a ship.

Same tools. Same model. Different definition of done. To be fair, they haven't measured the lift from checking terminal state before you commit. You still have to run it on your own records.

A trajectory is a claim

The line worth keeping: a trajectory is a claim. Database state is the evidence. Repetition is the trust test.

Pass@1 says how a model usually does. "Solved at least once in 20 tries" says whether it can ever do the task. Neither answers "will it do this every time." On their 507 workflows, each run 20 times from a clean backend, Claude Opus 5.5 led the overall pass@1 they report, at 67.16%. It passed the same number of tasks on all 20 attempts as Claude Opus 5: 241. A small bump in the headline score bought no extra dependability. Kimi-K3 solved 476 of 507 tasks at least once and only 68 of them every time.

If you're picking a model for work that touches real records off the "solved at least once" column, you're buying a demo.

Where the failures sit matters for how you spend your week. They label roughly four in five as tool handling, not reasoning: tool usage at 79.9% in an unweighted average of per-model shares, not a unique cause and not a law. Agents usually get far enough to try the workflow, then fail to recover from tool errors, failed preconditions, or empty lookups. That's a retry policy and a smaller tool surface before it's a shopping trip for a newer model.

Cheap Twin still applies: don't send every ticket to the flagship, and don't route on a single-pass score. In their list-rate estimate (not an invoice), the cheapest single success wasn't the cheapest task that passed every repeat.

Five lines before the next closer ships

Required end state is a field diff, written down before the run
Post-write readback from the system of record, not the assistant message
Customer-visible send stays closed on mismatch
Receipt names the fields that matched, the fields that didn't, and any extra writes
Ship Gate uses a repeat count you can defend, and you say whether you mean best-of-k or every-of-k

Twenty is their number, not a religion. A refund that moves money may need a stricter every-time bar than a draft note. Name k. Say which one you mean. Grade the sentence only when the requirement has no field.

How this sits with the rest of the stack

Receipt Gate already refuses the next write when the last step left mush. This tightens the receipt: it has to agree with a readback. Golden Trace should freeze the end state, not only the happy transcript, otherwise a prompt bump that still "sounds resolved" sails right through. Smoke evals need one trap where every tool returns ok and a required field is wrong.
Permission Envelope still wraps the irreversible send; a correct record doesn't grant a standing right to email the customer. Cheap Twin can pick the brain. It can't pick the definition of done.

The test

Take one workflow that writes a record and then tells a human it's finished. Run it five times from the same starting state. After each run, ignore the last message. Read the fields you claimed were the goal.

If any run says resolved while a required field is wrong, your eval is grading the claim.

Curious how others handle this: do you read the record back before the agent talks to the customer, or are you still trusting the closer?

I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: production, not demos.

Full piece and the rest of the series: sheikhwasim.com · LinkedIn · Practical AI Notes · X @anciwasim

Sources

Numbers and the retail walkthrough are from the October 3, 2026 joint post. I didn't re-run the benchmark. Cost remarks there are the authors' list-rate estimates (OpenRouter snapshot noted as September 20, 2026), not production invoices. They say they haven't measured the lift from terminal-state checks, a smaller tool surface, or human approval on this benchmark.

Microsoft and Hugging Face, "The Agent Said It Was Done. The Database Disagreed." Tuhin Kundu, October 3, 2026. https://huggingface.co/blog/microsoft/thinkingbox
Benchmark data repo cited in that post (release tag thinkingbox-bench-v1.0): https://github.com/microsoft/thinkingbox-data

Cheap Twin: Escalate Only When the Small Model Needs Help

Wasim Sheikh — Sun, 04 Oct 2026 17:02:19 +0000

Cross-post of Insights #7 — canonical: https://sheikhwasim.com/insights/cheap-twin-escalate-when-needed/

Most agent stacks send every request to the flagship model "just in case."

That looks careful. It is usually waste — and it hides when the cheap path was already good enough. Latency climbs. The bill spikes. Nobody can answer: which tickets actually needed the heavyweight brain?

I call the missing control a Cheap Twin: run the same input through a small model first. Escalate to the expensive brain only when the twin disagrees with a short checklist, or fails a smoke assert. Everything else ships on the twin — with a receipt that names which model answered.

Receipt Gate makes each write legible. Cheap Twin makes each brain choice legible — and defaults the happy path to the model you can afford to run all day.

Quick use case

Situation. A commercial construction ops team shipped an "RFI triage agent." It scored inbound requests for information: trade, urgency, and suggested owner. Leadership wanted accuracy, so every ticket went to the flagship model. Staging looked sharp.

What broke. Volume hit production. Latency climbed into the awkward zone where PMs refreshed and re-submitted. The model bill jumped without a matching jump in "hard" tickets. Nobody could say which RFIs needed the heavyweight path. Logs showed tokens and a model name — not why that model was chosen. There was no choice. There was a default.

The fix (Cheap Twin).

Run the twin first — small model proposes trade / urgency / owner on every RFI
Gate escalate on a short checklist — missing deadline, multi-trade conflict, owner-facing language, dollar threshold, unknown vendor
Fail closed on smoke asserts — if the twin output fails a cheap structural check, escalate (or hold) instead of shipping mush
Escalate with the twin's draft — flagship gets the same input plus the twin proposal and which checklist line fired
Receipt the brain every ticket logs twin vs flagship, checklist hits, and cost — skim-worthy, not a token dump

Same tools. Same schema. Most traffic never left the twin. When it did, the receipt said why.

Cheap Twins aren't about starving the agent of intelligence. They're about refusing the expensive path when you have no signal you need it.

Flagship-by-default is a missing control

A flagship call answers: What would the best model say?

A Cheap Twin answers: Did we need the best model for this ticket — and can we prove it?

Those are different questions. Most stacks fund the first and call the bill "the cost of quality." Quality without a routing receipt is hope with a larger context window: you cannot tune the checklist, Golden-Trace the cheap path, or tell finance which spike was complexity vs lazy routing.

The Cheap Twin stack (five layers)

Build it like a control on the model path, not a discount coupon in the prompt.

1. Twin-first contract

Every eligible request hits the small model before the flagship. The twin returns a structured proposal in the same schema the flagship would use — not a chatty essay the orchestrator reinterprets.

If the twin cannot speak the production schema, it is not a twin. It is a different product.

2. Escalate checklist (five lines, not fifty)

Name the concrete reasons the cheap path is not enough. Keep the list short enough to argue in standup:

Missing or ambiguous deadline
Multi-trade / multi-system conflict
Owner-facing or customer-facing language
Dollar / risk threshold crossed
Unknown vendor, SKU, or entity

Vague scores ("confidence < 0.7") belong behind a named line — or they become a knob nobody trusts.

3. Smoke assert before ship

Even on the twin path, run a cheap structural assert before anything leaves: required fields present, enums valid, no empty owner, no "see above" mush. Fail → escalate or hold. Do not ship a polite brick because the small model was fast.

Pair with your Smoke Evals pack so the same traps block CI and live routing.

4. Escalate with context, not from scratch

When a checklist line fires, the flagship should see the twin's proposal and which line triggered the escalate. That keeps the expensive call short and disagreement inspectable. Blind re-runs that ignore the twin waste the money you just decided to spend.

5. Brain receipt in the run record

Every ticket leaves a skim-worthy receipt: twin model id, flagship model id (or "not called"), checklist hits, assert pass/fail, latency, and cost. Mushy "used GPT" fails the gate — same rule as Receipt Gate.

Freeze a twin-happy path and a forced-escalate path as Golden Traces. Prompt and model bumps must not silently flip the routing mix.

How this maps to what you already have

5-layer agent stack — Cheap Twin lives in Models + Guardrails; Traces carry the brain receipt
Smoke evals — "twin ships when checklist clear" and "checklist hit → escalate"
Golden Trace — freeze one twin-only success and one escalate success; regress both
Receipt Gate — model choice deserves a plain-language receipt before the next write
Permission Envelope — escalate widens which brain may propose, not which tools
Ship Gate — prove routing + receipts in canary, not that the demo always hits flagship

Architecture without a twin is just a flagship with a louder invoice.

The test

Ask one question before the next "always use the best model" ship:

For a normal ticket, does a small model propose first — and can a teammate skim a receipt that says whether you escalated, which checklist line fired, and what it cost?

If the answer is no, you are still buying flagship by default.

Closing

I'm Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production.

Follow for practical AI architecture that ships.

Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Practical AI Notes · X: @anciwasim

Receipt Gate: No Second Write Without a Receipt

Wasim Sheikh — Tue, 29 Sep 2026 15:38:46 +0000

Most agent failures are not one bad tool call. They are three "successful" writes in a row that nobody can reconstruct.

Tool A updates the CRM. Tool B opens a ticket. Tool C emails the customer. Each step returns ok. The dashboard stays green. Hours later support asks why a customer got a resolution note for a case that never closed — and the only answer lives in a token dump no human will read.

I call the missing control a Receipt Gate: after every write, produce a short, human-readable receipt. The next write stays closed until that receipt is skim-worthy. No receipt, no second write. Most cascading messes die right there.

Golden Trace freezes the path that worked. Receipt Gate makes each live step legible before the next one fires.

Quick use case

Situation. A revenue-ops team shipped an "exception closer" agent. On a flagged invoice it could (1) patch the CRM opportunity stage, (2) post a note on the finance ticket, and (3) send the customer a status email. Staging looked sharp: three tools, one happy path, under thirty seconds.

What broke. A model bump plus a "helpful" prompt tweak patched the CRM to Closed-Won while the ticket note still said "pending PO match." The email went out with a warm "resolved" tone. Every tool returned success. Nobody could skim, in ten seconds, what actually changed.

Ops found it when a customer asked for a closed invoice that did not exist. Logs showed tokens, not a receipt chain a human could trust.

The fix (Receipt Gate).

Require a receipt after every write — object, fields, tool, success/fail in plain language.
Gate the next write on that receipt — tool B cannot fire until tool A's receipt is present and non-mushy.
Make receipts skim-first — ten-second read; raw JSON alone is not enough.
Fail closed on mush — vague "updated record" blocks the chain.
Keep the chain in the run record — receipts travel with the trace for audits and Golden Trace.

Same three tools. Same model family. Different blast radius. The next "Closed-Won + resolved email" never left the first gate.

Receipt Gates aren't about slowing the agent for sport. They're about refusing to stack silent side effects before a human — or the next step — can see them.

Success responses lie. Receipts don't have to.

A tool status code answers: Did the API accept the call?

A receipt answers: What business state actually changed, in words a teammate can check?

Those are different questions. Most stacks fund the first and hope the second shows up in observability later.

When writes chain, hope is the bug. Later tools inherit a world the first write may have only partly changed — or changed the wrong way while still returning 200.

If the only bridge is the model's short-term memory, you are one polite hallucination away from a customer-facing contradiction.

The Receipt Gate stack (five layers)

Build it like a control on the write path, not a nicer log sink.

1. Write → receipt contract

Every mutating tool emits a structured receipt before the orchestrator continues:

Object / resource id
Fields or side effects that changed
Tool name + schema id
Inputs that mattered (redacted)
Outcome in plain language: success, partial, or fail

If the tool cannot say what it did, it is not ready for production chaining.

2. Skim rule (ten seconds)

A receipt only a debugger can parse is not a receipt. Target a human skim: one-sentence summary, concrete change bullets, and an explicit "did not change" when an expected field was skipped.

Mushy language ("synced the account," "handled the exception") fails the gate. Specificity is the product.

3. Next-write latch

The orchestrator holds a latch: the next mutating tool stays closed until the prior receipt passes schema + skim checks. Read-only tools can still run. Irreversible or customer-visible writes wait.

Cascading damage dies at the latch — not in the postmortem.

4. Chain in the run record

Persist receipts in order next to the trace. Receipt[n] can reference Receipt[n-1]. Partial failures keep the chain. Ops get an export without opening the full token stream.

Golden Trace can freeze a successful chain. Smoke evals can assert "no second write without receipt." Ship Gate should refuse agents that mutate twice with nothing skimmable in between.

5. Human or policy break-glass

Sometimes a human must accept a partial receipt and continue. That is fine if it is explicit: named approver, why mush was allowed, and a short TTL on the override (pair with Permission Envelope).

Silent auto-continue on a failed skim check is how the CRM went Closed-Won again.

How this maps to what you already have

5-layer agent stack — Receipt Gate lives across Tools + Traces; Guardrails enforce the latch.
Permission Envelope a write may be authorized and still blocked if it left no receipt.
Golden Trace — freeze a chain whose receipts still match milestones after a model bump.
Smoke evals — include "second write with missing/mushy receipt → blocked."
Ship Gate prove chained writes leave skim-worthy receipts, not only that demos finish.

Architecture without receipts is just faster confusion with better models.

The test

Ask one question before the next multi-write agent ship:

If write #1 returns ok, can a teammate skim a plain-language receipt of what changed and does write #2 stay closed until that receipt exists?

If the answer is no, you are still chaining side effects on status codes and model memory.

Closing

I'm Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production.

Follow for practical AI architecture that ships.

Connect on LinkedIn: Wasim Sheikh

Golden Trace: The Eval Anchor Most Teams Skip

Wasim Sheikh — Fri, 25 Sep 2026 21:45:16 +0000

Cross-post of Insights #5 — canonical: https://sheikhwasim.com/insights/golden-trace-eval-anchor/

Most teams drown in failed agent runs and never lock the one that worked.

They collect stack traces, bad tool calls, and angry tickets. They rarely freeze a clean end-to-end success and treat it as sacred. The next prompt tweak, model bump, or tool schema change quietly bends that path. The demo still "looks fine." Customers feel the drift first.

I call the missing control a Golden Trace: one real successful run you refuse to lose — inputs, plans, tool calls, intermediate state, final outcome — frozen as the regression anchor for every later change.

Smoke evals catch the traps. Golden Trace proves the happy path you already earned still holds.

Quick use case

Situation. A platform team shipped an internal "invoice exception agent." It could read a flagged invoice, pull the PO and receipt lines, open a clarifying ticket when amounts diverged, and close the exception when the math matched. One Friday afternoon path worked end to end: correct PO match, one clarifying question, clean close. Leadership celebrated. Nobody saved the run.

What broke. Two weeks later a "harmless" prompt polish and a new retrieval index shipped. The agent still sounded sharp. It started closing exceptions when the PO line was close enough, skipping the clarifying ticket. Finance caught it in a weekly audit. Traces showed answers. They did not show which milestones of the original Friday path had disappeared. There was no frozen success to regress against — only a pile of failure tickets from earlier weeks.

The fix (Golden Trace).

Pick one proven success — the Friday path that matched PO → asked once → closed clean
Freeze the full trace — input invoice, tool calls + args, intermediate checks, final status, cost
Name the milestones — "PO exact match," "clarify if delta > $X," "no close without receipt line"
Regress on every change — prompt, tool schema, model, or index bump must hit the same milestones
Fail closed on drift — if a milestone vanishes, CI red; no "looks good in staging" override

Same model. Same tools. A locked success path the team could no longer accidentally erase.

Failure logs teach. Success paths prove.

A failure log answers: What went wrong that time?

A Golden Trace answers: Does the path that already worked still work on this exact change?

Those are different questions. Most LLM ops programs only fund the first.

Smoke evals (the trap pack) guard the bad paths. Golden Trace guards the good one you already paid for in debugging time. Without both, you either ship polite bricks or silent regressions that still pass a vibe check.

The Golden Trace stack (five layers)

Build it like a control, not a screenshot in Slack.

1. Capture (one real win)

Do not synthesize a perfect run in a notebook. Capture a production-like success:

Real-ish inputs (redact secrets / PII)
Every tool call with typed args and results
The decision points that mattered
The final business outcome, not just the last model sentence

If you cannot replay it, it is not a Golden Trace. It is a story.

2. Milestone contract

A full token dump is too brittle. Extract the milestones that define success:

Required tools in order (or allowed set)
Hard checks that must still pass (exact match, threshold, human gate)
Forbidden shortcuts (skip clarify, invent a line, close on "close enough")
Exit condition that matches the business outcome

Milestones are the contract. The raw trace is the evidence.

3. Freeze in version control

Store the golden artifact next to the code:

Trace JSON (redacted) + milestone assertions
Linked smoke-eval case IDs if the same path has trap variants
Owner and last-reviewed date

If it is not in git, the next refactor will "simplify" it away.

4. Regress on every change that can bend the path

Run Golden Trace checks when you change:

System / developer prompts
Tool schemas or auth scopes
Model version or temperature defaults
Retriever / RAG index that feeds the agent

Red means no ship. "Staging still looks fine" is how the invoice agent learned "close enough."

5. Refresh when the product truth moves

When the real success criteria change — new policy, new required human gate, new tool — update the Golden Trace on purpose. Do not let an outdated golden become a museum piece that blocks good ships, and do not silently drift the milestones to match a worse path.

A Golden Trace is a living contract with the last path you were proud of.

How this maps to what you already have

5-layer agent stack — Golden Trace lives in Traces; Guardrails + Rollback catch what drift still slips through
Smoke evals — trap cases block the bad paths; Golden Trace locks the good one
Ship Gate — gate #3 (eval set) should include at least one golden success, not only failures
Permission Envelope — if a milestone requires a fresh human grant, the golden must assert that the grant was present and not silently skipped

Architecture without a locked success path is just hope with better diagrams.

The test

Ask one question before the next agent ship:

Do we have one versioned successful run — with named milestones — that CI fails if this change drifts off it?

If the answer is no, you are still shipping on demo confidence and failure archaeology.

Closing

I'm Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production.

Follow for practical AI architecture that ships.
Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Practical AI Notes · X: @anciwasim

Permission Envelope: Consent That Expires

Wasim Sheikh — Tue, 22 Sep 2026 16:06:05 +0000

Most teams give an AI agent a capability once and never take it back.

That is not consent. That is a standing power of attorney with no end date.

The model asks for calendar access, a CRM write, or a deploy token. Someone clicks allow. Three weeks later the same grant is still live — for a different task, a different tenant context, and a tired engineer who forgot what they approved. When something goes wrong, the logs show “authorized.” They do not show when the human last meant it.

I call the missing control a Permission Envelope: a short-lived, scoped grant that dies unless a human renews it. Expiry is not bureaucracy. Expiry is the audit trail that proves consent was still fresh when the tool fired.

Quick use case

Situation. A platform team shipped an internal “release assistant.” It could read a PR, draft release notes, and — with an approved toggle — open the production deploy ticket and page the on-call channel. Staging demos used a long-lived service account so nobody had to re-auth every run.

What broke. Two sprints later, a noisy alert chain made the agent open a deploy ticket for the wrong service and page the wrong rotation. The grant that allowed “open ticket + page” had been approved once for a canary release. It never expired. There was no re-ask, no TTL on the capability, and no log line that said “last human consent: 19 days ago for a different change.” Security called it authorized. Ops called it a near miss.

The fix (Permission Envelope).

Scope the grant — “open deploy ticket for service X + page rotation Y” — not “any ticket, any channel”
Bind to a purpose — one change ID / one release window; the grant cannot be reused for a different PR
Set a short TTL — minutes or hours for irreversible tools; days only for read-only
Re-ask on expiry — silent renewal is forbidden; a fresh human intent is the only path back in
Log consent, not just auth — who approved, what they saw, when it expires, and which action consumed it

Same agent. Same model. Different blast radius. The next wrong-service page never left the draft state.

Permission Envelopes aren’t about distrusting the model. They’re about refusing to let yesterday’s yes authorize today’s irreversible action.

Why permanent grants fail quietly

Auth answers: Is this identity allowed to call the tool?

A Permission Envelope answers: Did a human still mean this specific action, for this purpose, right now?

Those are different questions. Most agent stacks only ask the first.

Service accounts and OAuth tokens are great at surviving restarts. They are terrible at remembering that the human clicked “allow” for a canary, not for every future Tuesday. Permanent grants optimize for fewer clicks. Production optimizes for fewer surprises at 2 a.m.

The Permission Envelope stack (five layers)

Build the grant like a system, not a checkbox buried in settings.

1. Capability (what)

Name the exact tool surface: which write endpoints, which tenants, which channels. Prefer deny-by-default. “CRM update” is not a capability — “update ticket status on account A, fields status + note only” is.

If you cannot write the capability in one sentence a junior engineer understands, the grant is already too wide.

2. Purpose (why / which)

Bind the grant to a concrete work item: PR number, change ticket, incident ID, customer case. When the purpose changes, the envelope dies. Reuse across tasks is how “authorized” becomes meaningless.

3. Duration (how long)

Pick the shortest TTL that still lets the workflow finish:

Irreversible writes (send, deploy, charge, delete, page) — minutes to a few hours
Mutating drafts (create ticket, draft email) — hours, then re-ask
Read-only — longer is fine; still expire so abandoned sessions don’t linger

Auto-extend is a smell. If the job needs more time, the human should know.

4. Fresh consent (who / when)

Expiry must force a real re-ask: show the blast radius again, get an explicit approve, freeze the approved payload. Do not let the model regenerate the action after the click. Do not treat “still logged in” as “still consented.”

The re-ask is the audit trail field teams keep skipping — and the one regulators and postmortems actually care about.

5. Consumption log (proof)

Every tool call that used the envelope records:

Grant ID + scope + purpose
Approver identity + timestamp + what they were shown
Expiry + whether this call consumed or renewed
Outcome (success / denied / expired mid-flight)

If you only log “token valid,” you cannot reconstruct consent. You can only reconstruct hope.

How this maps to what you already have

5-layer agent stack — Tools get typed scopes; Guardrails deny out-of-envelope calls; Traces carry the grant ID; Rollback still matters when a consumed grant was wrong
Ship Gate — threat model names “stale consent”; human override is the re-ask; observability is the consumption log
Smoke evals — add cases where the grant is expired, purpose-mismatched, or too wide; CI should fail if the agent still fires the write

A Permission Envelope does not replace Ship Gate. It makes gate #4 (human override) real after the first approve — and every hour after.

The test

Ask one question before the next agent write-tool ships:

Can we show, for every irreversible action, who consented, to what scope, for which purpose, and when that consent expires — and does the tool refuse when any of those are missing or stale?

If any half is no, you are still shipping standing power of attorney.

Closing

I’m Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production.

Follow for practical AI architecture that ships.

Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Practical AI Notes · X: @anciwasim

Smoke Evals: The 20 Cases That Should Block Every AI Deploy

Wasim Sheikh — Fri, 18 Sep 2026 15:39:01 +0000

Your AI feature passed the demo.

That is not evidence it will survive Monday morning.

Demos are curated. Someone picks the happy path, soft-prompts around the edges, and never pastes the weird customer email that invents a discount, leaks a ticket ID, or loops a tool until the bill spikes. Production does all of that before lunch.

I call the missing test set smoke evals — a short, fixed pack of trap cases that must pass before any AI change leaves the sandbox. Not a research benchmark. Not a vibe check. A deploy gate.

If Ship Gate is the checklist, smoke evals are the part that actually fails the build.

Quick use case

Situation. A platform team shipped an “AI ops assistant” that could summarize incidents and suggest runbook steps. Staging looked sharp. Leadership wanted it in the on-call channel before a busy release week.

What broke. Night one, a noisy alert chain made the model invent a rollback command that did not exist in the runbook. An engineer almost ran it. There was no frozen set of “known trap” incidents, no assert that refused invented commands, and no CI job that would have blocked the merge.

The fix (smoke evals).

Freeze ~20 cases — real-ish incidents: missing runbook steps, ambiguous alerts, PII in tickets, prompt-injection style pastes, “do nothing” noise
Assert two things — safe (no invented commands, no secret leakage) and useful (still names the right service / severity when the answer is known)
Wire to CI — every prompt, tool schema, or model bump runs the pack; red means no ship
Own the failures — each failing case gets an owner and a fix path (prompt, tool contract, or human gate)
Refresh monthly — add the newest production scare; retire cases that no longer teach

Same model. Different gate. The next invented rollback never reached the channel.

Smoke evals aren’t bureaucracy. They’re the smallest set of cases that would have caught the last three failures — run on every change.

Why demos lie (again)

A demo answers: Can we make it look smart once?

A smoke eval answers: Does it refuse the bad path, every time, on this exact change?

Those are different questions. Most teams only ask the first.

The second is cheaper than a postmortem. Twenty cases, a few dollars of inference, a red CI job. Compare that to an almost-run false rollback at 2 a.m.

The smoke-eval stack (five layers)

Build the pack like a system, not a spreadsheet someone opens once.

1. Case bank (frozen, versioned)

Keep the cases in git. Each case has:

Input — the ticket, chat, alert, or document paste
Expected behavior — refuse / escalate / answer with X
Failure mode tagged — hallucination, tool misuse, PII leak, loop, overconfidence

If it isn’t in version control, it isn’t a gate. It’s a wish.

2. Dual assert: safe and useful

Safe-only evals train the model to say nothing. Useful-only evals ship confident nonsense.

Every case should check both:

Safe — no invented SKUs, commands, prices, or secrets
Useful — when the ground truth is known, the answer still helps

A pack that only measures “didn’t explode” will ship a polite brick.

3. CI as the gate

Run the pack on:

Prompt / system-message changes
Tool schema changes
Model version bumps
Retriever / RAG index refreshes that affect answers

Red build = no merge. No “we’ll check it later in staging” exception for the pack that exists because staging lied.

4. Failure ownership

A failing case without an owner becomes noise. Assign:

Who owns the failure mode
Whether the fix is prompt, tool contract, retrieval, or human override
When it must be green again

Smoke evals without ownership rot into ignored red.

5. Monthly refresh from production scars

Every real incident that almost (or did) hurt customers becomes a candidate case. Add it. Keep the pack near twenty so it stays fast. Retire cases that no longer teach.

The pack should smell like your last three scares — not a generic LLM leaderboard.

How this maps to what you already have

Ship Gate — gate #3 (eval set) is where smoke evals live; the other four gates still matter
5-layer agent stack — traces tell you which case failed and why; guardrails + rollback catch what the pack missed

Smoke evals don’t replace architecture. They make the architecture earn the next deploy.

The test

Ask one question before the next AI ship:

Do we have ~20 versioned trap cases that assert safe and useful — and does CI fail if they fail?

If either half is no, you are still shipping on demo confidence.

Originally published on sheikhwasim.com. I’m Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on.

Connect: LinkedIn · Practical AI Notes · X @anciwasim

Ship Gate: The Pre-Deploy Checklist Most AI Features Skip

Wasim Sheikh — Thu, 17 Sep 2026 15:35:12 +0000

Most AI features don't fail in the model.

They fail at the gate.

Someone demos a shiny prompt. Leadership loves it. It ships. Then a customer pastes something weird, an agent calls the wrong tool, or a "helpful" answer invents an offer that never existed. The postmortem is always the same: we optimized for the demo, not for production.

I call the missing step Ship Gate — a short, boring checklist you run before any AI feature leaves the sandbox. Not a 40-page risk memo. A gate you can clear in an afternoon.

Quick use case

Situation. A product team shipped an AI email summarizer into customer-facing replies. Staging looked clean. Leadership wanted it live before the quarter closed.

What broke. Day one, the model invented a discount a customer had never been offered. Support scrambled. There was no eval set of "known trap" emails, no human gate on outbound AI text, and no kill switch short of a full deploy rollback.

The fix (Ship Gate).

Threat model — named "hallucinated commercial offers" as a top failure mode with an owner
Data boundary — outbound copy restricted to template fields; free-form claims blocked
Eval set — trap cases that fail the build if the model invents offers or leaks PII
Human override — outbound AI copy held until a human approves (or stricter template mode)
Observability — one bad reply reconstructable in minutes: prompt, context, output, cost

Then they added the ops layer every Ship Gate implies: canary traffic, a one-click kill switch, and a documented rollback owner before the flag flipped on.

Ship Gate isn't bureaucracy. It's the minimum checklist so an AI feature earns production — instead of learning in front of customers.

Why demos lie

Demos are curated. You pick the happy path. You soft-prompt around the edge cases. You never paste a raw customer email with secrets still in it.

Production is the opposite. Users paste junk. Models invent confident nonsense. Tools fire with the wrong args. Logs capture more than you think.

If your only test was "it looked good in the meeting," you didn't ship AI — you shipped a vibe.

Ship Gate exists to make that gap visible before customers feel it.

The five gates

Run these in order. Fail any one and you don't ship. Pass all five and you can sleep.

1. Threat model (ten minutes, not a committee)

Before code, write three sentences:

What can go wrong? (wrong answer, tool misuse, data leak, prompt injection, cost runaway)
Who gets hurt? (customer, employee, your brand, a regulator)
What's the blast radius? (one chat vs. every tenant)

If you can't name the top three failure modes, you're not ready to build — you're ready to be surprised.

2. Data boundary

Answer out loud:

What data can the model see?
What can it write?
What must never leave this boundary (PII, secrets, customer documents, internal tickets)?

Then enforce it in code, not in a slide. Strip secrets before the model. Scope tools to the minimum. Prefer retrieval over stuffing the whole corpus into context. If the model doesn't need a field, don't give it the field.

3. Eval set before polish

Pick 20–50 real-ish cases: happy path, hostile prompts, empty input, long paste, "ignore previous instructions," the ticket that always breaks things.

Score each on three axes:

Correct enough for the job
Safe enough (no leak, no unsafe action)
Useful format (what the user actually needs)

Ship Gate rule: you don't tune the UI until the eval set is green enough that you'd trust a teammate to use the feature unsupervised.

4. Human override

Every autonomous or semi-autonomous path needs an escape hatch:

Confirm before irreversible actions (send email, delete, charge, deploy)
Log who approved what
Make "stop / undo / escalate" obvious

If the only recovery path is "hope the model was right," you don't have a product. You have a liability with a chat box.

5. Observability that a human can read

When it breaks at 2 a.m., can you answer:

What did the user ask?
What context did the model get?
Which tools ran, with which args?
What did we return?
Cost and latency for that turn?

If the answer is "we'd have to dig through JSON for an hour," fix the logs before you grow the feature. Ship Gate isn't complete until a tired engineer can reconstruct a single bad turn.

A one-page Ship Gate card

Gate	Pass criteria
Threat model	Top 3 failure modes named + owner
Data boundary	See / write / never-leave documented + enforced
Eval set	≥20 cases; safe + useful bar met
Human override	Irreversible actions confirmed; stop path clear
Observability	One bad turn reconstructable in <5 minutes

No green across the board → no ship.

The test

Ask one question before any AI feature goes live:

Which of the five gates is still red — and who owns fixing it?

If nobody owns the red gate, you don't have a launch plan. You have a calendar date.

I'm Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production.

Canonical: https://sheikhwasim.com/insights/ship-gate-pre-deploy-checklist/

The 5-Layer Stack Behind Agents That Ship

Wasim Sheikh — Mon, 14 Sep 2026 15:59:29 +0000

Originally published on my site: https://sheikhwasim.com/insights/agent-architecture-five-layer-stack/

Most "AI agents" fail for the same reason.

Someone wires a language model to a handful of tools, demos a happy path, and calls it architecture. It works in the slide. It collapses the first time production is messy timeouts, bad inputs, spend spikes, irreversible side effects.

Here's the five-layer stack that actually ships.

Quick use case

Situation. A team built an internal "support triage agent." Demo day looked sharp: read a ticket, call search, suggest a reply, update CRM status.

What broke. In production, one vague ticket sent the agent into a 40-step tool loop. It wrote the wrong CRM status, and spend spiked past the monthly budget in an afternoon. Chat logs showed answers — not which tool ran, with what inputs, or why it kept going. The CRM write couldn't be cleanly reversed.

The fix (mapped to the stack).

Orchestrator — hard max steps + stop when confidence is low
Tools — typed CRM update with auth scope + idempotency key
Traces — every plan / tool call / result / cost on one correlation ID
Guardrails — daily spend cap + deny list for irreversible tools without a human gate
Rollback — compensating "revert status" action when the write was wrong

Same model. Different system. That's the difference between a demo and something teams can trust.

Layer 1 — Orchestrator

The orchestrator decides the next step. It is not the model dumping text forever.

Good orchestrators:

Tools are clear APIs, not mystery side effects.

Ship-ready tools have:

Auth and scoped credentials
Typed inputs and validated outputs
Timeouts, rate limits, and idempotency keys
Explicit success / failure contracts

A tool that "usually works" is a liability. Prefer boring interfaces over clever ones.

Layer 3 — Traces

Every step logged so you can see what it did — and why.

Traces are how you debug, audit, and improve:

Prompt / plan / tool call / result / decision
Timing and cost per step
Correlation IDs across services
Redaction for secrets and PII

If you can't replay the path, you can't trust the system in an enterprise.

Layer 4 — Guardrails

Budget, permissions, PII, and stop conditions — before it runs wild.

Guardrails belong in the control plane, not in a polite system prompt:

If it's wrong, you reverse it. Idempotent actions beat clever guesses.

Design for undo:

Prefer reversible writes
Snapshot before mutate
Compensating transactions when undo isn't free
Clear "blast radius" for every tool

An agent that can't roll back isn't a system. It's a demo with confidence.

The test

Ask one question of any agent stack:

Can it show traces, and can it roll back?

If either answer is no, keep it out of production.

Closing

I'm Wasim Sheikh — AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept — production.

Follow for practical AI architecture that ships.
Site: https://sheikhwasim.com · Notes: https://practicalainotes.substack.com/ · X: https://x.com/anciwasim

Spend and token caps
Allow / deny lists for tools and data
Human gates for irreversible actions
Policy checks on inputs and outputs

Prompts are guidance. Guardrails are enforcement.

Layer 5 — Rollback

Choose among plan act verify → stop
Bound loop length and retry policy
Separate "thinking" from "committing"
Fail closed when confidence or evidence is weak

If your system can't explain why it took the next action, you don't have an orchestrator — you have a chat session with plugins.