Rehearse the Rollback, Not the Launch: A 45-Minute Exit Drill for AI Features

Every AI feature I ship gets a launch checklist. Almost none of them get an exit checklist. That asymmetry is the real bug.

Launch day is loud and social. Rollback day is quiet and 2 a.m. So we rehearse the loud thing and improvise the quiet one. Bad trade.

This post is a drill, not a policy doc. It takes 45 minutes. It prints three numbers and a pass/fail line you can paste into a PR. A checklist in a wiki is a wish. A script is a rehearsal.

The question launch checklists never ask

Your launch checklist answers "does it work?" Mine did, for months. The question that actually bites later is different.

How many seconds until the feature is dark? How many until a human knows? How many until it is back?

Call them T-dark, T-inform, and T-restore. If you cannot state all three, you do not have an exit. You have a hope.

Numbers beat adjectives. "Fast rollback" means nothing in an incident channel. "T-dark was 40 seconds, T-inform was 6 minutes" starts a real conversation.

Why I rehearse on a free tier instead of staging

A rollback drill is deliberately destructive. You kill a working feature and watch what screams. Doing that against production quarterly is defensible. Doing it weekly is not.

So I keep a disposable twin: one small server, one model endpoint, one feature flag, nothing else. The account operator describes MonkeyCode's free tier as 10M tokens plus a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use those two things as the drill target for one specific reason. The drill burns tokens and reboots boxes. I would rather it burn a free allocation than a metered one. Quotas, models, and longevity change — verify current terms yourself before you plan around them. I am not promising any of it stays the same.

Who should not use this approach: teams whose alerting pages a real human on shared infra. A drill that fires false pages is worse than no drill. Run the twin in an isolated project, or skip this entirely and rehearse on a scheduled maintenance window instead.

Step 1 — Stand up a twin you are allowed to break (15 min)

Nothing fancy here. The twin needs exactly four things, no more:

1. A server running your feature behind a flag.
2. A model endpoint you can point at without touching production keys.
3. A probe command that fails when the feature is genuinely dead.
4. A log file, so the numbers survive the terminal scroll.

Write down which of those four is missing. That missing item is your first real finding, and you have not run anything yet.

Step 2 — The 40-line drill script

This is the whole artifact. Save it as drill.sh, read it once, then edit the three env vars.

#!/usr/bin/env bash
# drill.sh - rehearse the exit for ONE AI feature. Time-box: 45 min.
set -euo pipefail

FLAG_URL="${FLAG_URL:?set FLAG_URL to your flag toggle endpoint}"
HEALTH_URL="${HEALTH_URL:?set HEALTH_URL to a human-readable status endpoint}"
PROBE_CMD="${PROBE_CMD:?set PROBE_CMD to a command that fails when the feature is dead}"
LOG="drill-$(date -u +%Y%m%dT%H%M%SZ).log"

now() { date -u +%s; }
say() { printf '%s %s\n' "$(date -u +%H:%M:%SZ)" "$*" | tee -a "$LOG"; }

say "drill start"
t0=$(now)
curl -fsS -X POST "$FLAG_URL" -d '{"enabled":false}' >/dev/null
t1=$(now)
say "T-dark: $((t1 - t0))s"

if eval "$PROBE_CMD"; then
  say "FAIL: feature still answering after flag off"
  exit 1
fi
say "PASS: probe now fails closed"

sleep 30   # human notification window; scripts cannot measure this

t2=$(now)
curl -fsS "$HEALTH_URL" | tee -a "$LOG"
say "T-inform: $((t2 - t1))s to operator-visible state"

t3=$(now)
curl -fsS -X POST "$FLAG_URL" -d '{"enabled":true}' >/dev/null
if eval "$PROBE_CMD"; then
  say "T-restore: $((t3 - t2))s"
else
  say "FAIL: restore broken, twin is now dark"
  exit 2
fi

say "drill end. paste T-dark / T-inform / T-restore into the PR."

Run it:

chmod +x drill.sh
export FLAG_URL="http://twin.local/flags/ai-summary"
export HEALTH_URL="http://twin.local/healthz"
export PROBE_CMD='curl -fsS http://twin.local/summarize | grep -q "\\[\\]" && exit 1 || exit 0'
./drill.sh

That probe is intentionally ugly. It is also the part people get wrong.

Step 3 — Keep one failure fixture, not zero

Here is the failure that taught me to stop trusting status codes.

My first drill passed. The flag was off. The endpoint returned HTTP 200. Lovely green dashboard.

The body was {"choices":[]}. An empty completion list, served with a happy status code. A status-only probe calls that healthy. My probe called it healthy too, and I only caught it because I read the log by accident.

The fix is a probe that asserts content, not code:

# bad: 200 means nothing here
curl -fsS "$URL" >/dev/null

# good: assert a non-empty result and a token field
curl -fsS "$URL" | jq -e '.choices[0].text | length > 0' >/dev/null

Commit that fixture. Keep a saved response that returns 200 with an empty body, and run your probe against the fixture in CI. A probe validated only against a healthy response is decoration.

Step 4 — The pass/fail table you actually enforce

Fill this in after one drill. Do not negotiate it afterwards.

Number	Your value	Pass	Fail-closed action
T-dark	____ s	< 60s	Add a kill switch at the edge; flags are not fast enough
Probe after flag off	____	Must fail	Probe is lying; rewrite it against the fixture
T-inform	____ s	< 5 min	Nobody is watching the dashboard; move it to a page
T-restore	____ s	< 120s	You cannot safely ship; keep the feature dark
Data written while dark	____	Zero or queued	You are losing writes; stop and design a buffer

Two rules make this table real. A missing number fails. An unmeasured cell is a no, not a maybe. And one drill is not evidence — run it twice, a week apart, before you quote the numbers to anyone.

The 45-minute boundary, and when to walk away

Time-box the drill. If you hit 45 minutes and the twin still cannot toggle, stop.

That is not a failed drill. It is a successful early warning. A feature you cannot turn off in a sandbox will not turn off cleanly in production.

Walk away and fix the toggle first. Come back when curl on your own laptop can dark the feature in under a minute. No amount of documentation substitutes for that one command working.

What this does not cover

This drill measures mechanical exit speed, not judgment. It will not tell you whether to roll back. It will not survive a model provider whose failure mode is slow degradation rather than a hard stop. And it says nothing about cost rollback — turning a feature off rarely un-spends the tokens already used.

If your team has no feature flag at all, this drill is not your next step. Add the flag first. Then rehearse.

The twin does not need to be good. It needs to be disposable. Mine is a free server and a free model allocation, and I break it on purpose every other Friday.

So — when you filled in the table, which cell did you leave blank? That blank is the one I would fix next.