AI Feature Launch Checklist: Staged Rollout and Rollback Triggers in Python

Launching an LLM feature isn't the finish line. In some ways it's the start of the riskiest phase. Real users send inputs your golden set never imagined. Provider models get updated underneath you. Costs track usage in ways the spreadsheet didn't predict. And when something goes wrong, the output is confidently wrong in front of a customer.

If you're the engineer or tech lead who owns the flag, you've probably sat through the "are we ready?" meeting where everyone nods and nobody can point to a number that would make you pull the plug. This post is the checklist we use before, during, and after shipping an AI feature — adapted for people who write the feature flag and the kill switch — plus about 100 lines of Python that turns your rollback triggers into a decision table you can run against a metrics snapshot.

About the numbers: every threshold, metric and stage percentage below is made-up example data for a fictional ticket-summarizer feature. They're picked to make the arithmetic easy to follow, not taken from any real product. Set your own numbers with your on-call and PM before launch.

What "ready" actually means

A launch checklist that lives only in a PM doc gets ignored on the day. Split readiness by function so every team owns its own items, and keep the engineering half next to the code.

Product, UX and evals

• PRD approved; non-goals and autonomy level confirmed
• Every UX state built: loading, success, partial/low-confidence, failure/fallback, disabled by admin, unsupported input
• AI-generated content labeled, with limitations explained in the product
• Feedback capture (thumbs, reason codes) live and flowing to a triage queue
• Holdout or experiment set up if you're measuring outcome impact
• Final eval on the held-out golden set meets every blocking threshold (record the run ID)
• Failures tagged by mode; slice minimums met; red-team findings fixed or signed off
• Model version pinned; prompt version tagged and linked to the eval run

Engineering & reliability

• Feature flag and staged rollout configured
• Kill switch tested in staging, with time-to-take-effect measured and written down
• Non-AI fallback works end to end with the provider unavailable
• Timeouts, retries, and fallback model configured
• Rate limits and per-user/per-tenant spend caps set; provider quota confirmed for peak
• Logging of inputs, outputs, and versions in line with privacy requirements
• Dashboards live and alerts routed to on-call

That kill-switch line is the one teams fake. "We have a kill switch" and "we flipped it in staging and know it takes 47 seconds to drain in-flight requests" are very different levels of ready. Measure it. Write the number in the launch channel before you flip the canary.

Safety, cost and support

Also confirm: privacy checklist complete, vendor terms match the contract, admin opt-in/opt-out works, guardrails have tests, cost model re-run with actual dogfooding token counts, spend alerts configured, support trained on known limitations, and an incident channel with an escalation path. Close with a go/no-go table: one named approver each for Product, Engineering, ML/Data, Privacy/Legal, and Support.

Launch day is boring on purpose

Confirm the flag targets only the intended audience. Watch error rate, latency, spend and feedback for the first couple of hours. Spot-check 10–20 live outputs against your rubric. Send a test alert. Post status at set times (T+1h, T+4h, end of day). The goal is a quiet day where every number already has a pre-agreed action.

Rollback triggers: numbers, not feelings

The most expensive rollback is the one that gets debated during an incident. Define triggers as numbers ahead of time, and for each one name who can act without asking for permission.

Trigger	Example threshold	Action	Who decides
Confirmed critical failure (e.g. fabricated commitment)	Any confirmed instance with customer impact	Kill switch → incident process	On-call or PM, no approval needed
Cross-tenant or permission data exposure	Any confirmed instance	Kill switch immediately; notify Security & Legal	Anyone on the response rota
Sampled audit pass rate	Below 85% for N periods	Pause rollout; investigate	PM + ML lead
Negative feedback rate	Above 8% for 24h	Investigate; pause expansion	PM
Error/timeout rate	Above 3% for N min	Degrade to fallback	On-call
p95 latency	Above 4s for N min	Fallback or smaller model	On-call
Spend	Above 130% of daily forecast	Investigate; tighten limits	PM / budget owner

Not every trigger means "turn it off." Many mean degrade: move to a fallback model, or to a non-AI path. That only works if your PRD defined a degradation ladder and engineering built it. If the only button you have is off, every incident becomes a binary argument.

Rollback procedure

1. Act: flip the kill switch or roll the flag back a stage; confirm the fallback is showing.
2. Communicate: post what triggered it, when, the scope, and the current state.
3. Preserve evidence: save example inputs/outputs, versions, and timestamps (within privacy rules).
4. Assess impact: which users or tenants were affected, and did bad output reach customers?
5. Notify: Support (macro ready), customer-facing teams, and Legal/Privacy if needed.
6. Fix and verify: add failing cases to the golden set, fix, pass a full regression.
7. Re-enable with sign-off, starting smaller.
8. Post-incident review within a set number of business days, with owners.

Step 6 compounds. Every incident should leave your test set stronger.

Encode the decision table in Python

Spreadsheets and wiki tables go stale the moment the launch channel starts moving. A short script makes every threshold visible, diffable, and runnable against a metrics snapshot. Save this as rollout_monitor.py (standard library only, Python 3.8+):

# rollout_monitor.py -- staged rollout + numeric rollback triggers for an LLM feature.
# All numbers are made-up example data for a fictional ticket-summarizer feature.

from dataclasses import dataclass

# Rollout stages: flag percentage of eligible traffic
STAGES = [
    ("canary", 1),
    ("early", 5),
    ("mid", 25),
    ("wide", 50),
    ("full", 100),
]

@dataclass
class Snapshot:
    """One monitoring window of made-up production metrics."""
    label: str
    stage: str                      # current stage name
    audit_pass_pct: float           # sampled audit pass rate
    neg_feedback_pct: float         # negative thumbs / total feedback
    error_pct: float                # error + timeout rate
    p95_latency_s: float            # end-to-end p95 seconds
    spend_vs_forecast_pct: float    # daily spend as % of forecast
    critical_failure: bool = False  # confirmed critical (e.g. fabricated commitment)
    data_exposure: bool = False     # confirmed cross-tenant / permission leak

# Thresholds agreed before launch (example numbers only)
TRIGGERS = {
    "critical_failure":  "any confirmed instance → KILL",
    "data_exposure":     "any confirmed instance → KILL",
    "audit_pass":        ("<", 85.0, "PAUSE"),       # below 85% → pause expansion
    "neg_feedback":      (">", 8.0,  "PAUSE"),       # above 8% → pause expansion
    "error_rate":        (">", 3.0,  "DEGRADE"),     # above 3% → fallback model
    "p95_latency":       (">", 4.0,  "DEGRADE"),     # above 4s → fallback / smaller model
    "spend":             (">", 130.0, "TIGHTEN"),    # above 130% of forecast → tighten limits
}

ACTION_PRIORITY = {"KILL": 4, "PAUSE": 3, "TIGHTEN": 2, "DEGRADE": 1}

def stage_pct(name: str) -> int:
    for n, pct in STAGES:
        if n == name:
            return pct
    raise KeyError(name)

def next_stage(name: str):
    names = [n for n, _ in STAGES]
    i = names.index(name)
    return STAGES[i + 1] if i + 1 < len(STAGES) else None

def evaluate(snap: Snapshot):
    """Return (action, reasons) for one monitoring window."""
    reasons = []

    if snap.critical_failure:
        reasons.append("critical_failure: confirmed customer-impacting hallucination")
    if snap.data_exposure:
        reasons.append("data_exposure: confirmed cross-tenant leak")

    if reasons:
        return "KILL", reasons

    checks = [
        ("audit_pass", snap.audit_pass_pct, *TRIGGERS["audit_pass"]),
        ("neg_feedback", snap.neg_feedback_pct, *TRIGGERS["neg_feedback"]),
        ("error_rate", snap.error_pct, *TRIGGERS["error_rate"]),
        ("p95_latency", snap.p95_latency_s, *TRIGGERS["p95_latency"]),
        ("spend", snap.spend_vs_forecast_pct, *TRIGGERS["spend"]),
    ]
    actions = []
    for name, value, op, thresh, action in checks:
        hit = value < thresh if op == "<" else value > thresh
        if hit:
            reasons.append(f"{name}: {value:g} {op} {thresh:g} → {action}")
            actions.append(action)

    if not actions:
        nxt = next_stage(snap.stage)
        if nxt is None:
            return "HOLD_FULL", ["all clear; already at full"]
        return "ADVANCE", [
            f"all clear; eligible to advance {snap.stage}→{nxt[0]} ({nxt[1]}%)"
        ]

    worst = max(actions, key=lambda a: ACTION_PRIORITY[a])
    return worst, reasons

# Four fictional monitoring windows during a staged rollout
WINDOWS = [
    Snapshot("T+2h canary", "canary", 92.0, 3.5, 1.2, 2.1, 95.0),
    Snapshot("T+1d early", "early", 88.0, 4.0, 3.8, 5.2, 110.0),
    Snapshot("T+3d mid", "mid", 81.0, 9.5, 1.5, 2.4, 140.0),
    Snapshot(
        "T+5d mid (incident)",
        "mid",
        90.0,
        4.0,
        1.0,
        2.0,
        100.0,
        critical_failure=True,
    ),
]

if __name__ == "__main__":
    print("Stage plan:", " → ".join(f"{n} {p}%" for n, p in STAGES))
    print()
    for snap in WINDOWS:
        action, reasons = evaluate(snap)
        pct = stage_pct(snap.stage)
        print(f"[{snap.label}] stage={snap.stage} ({pct}%) → {action}")
        for r in reasons:
            print(f"  - {r}")
        print()

Run it against four fictional monitoring windows for the ticket summarizer:

$ python3 rollout_monitor.py
Stage plan: canary 1% → early 5% → mid 25% → wide 50% → full 100%

[T+2h canary] stage=canary (1%) → ADVANCE
  - all clear; eligible to advance canary→early (5%)

[T+1d early] stage=early (5%) → DEGRADE
  - error_rate: 3.8 > 3 → DEGRADE
  - p95_latency: 5.2 > 4 → DEGRADE

[T+3d mid] stage=mid (25%) → PAUSE
  - audit_pass: 81 < 85 → PAUSE
  - neg_feedback: 9.5 > 8 → PAUSE
  - spend: 140 > 130 → TIGHTEN

[T+5d mid (incident)] stage=mid (25%) → KILL
  - critical_failure: confirmed customer-impacting hallucination

Read that output the way you'd read a pager:

• T+2h canary → ADVANCE. Metrics are inside every threshold. Eligible to move from 1% to 5% — permission to consider advancing, not a robot flipping the flag.
• T+1d early → DEGRADE. Error rate 3.8% and p95 latency 5.2s both trip. Route traffic to the fallback (or a smaller/faster model) while you investigate. Do not keep expanding.
• T+3d mid → PAUSE. Audit pass 81%, negative feedback 9.5%, spend at 140% of forecast. PAUSE beats TIGHTEN in the priority ladder, so expansion stops; spend gets tightened as a follow-up.
• T+5d mid (incident) → KILL. A confirmed customer-impacting hallucination short-circuits every numeric check. Kill switch first, debate later.

The priority ladder (KILL > PAUSE > TIGHTEN > DEGRADE) is the part most homemade runbooks leave implicit. Without it, two people reading the same dashboard can pick two different "obvious" next steps. Put the ladder in code, review it like a config change, and paste the output into the launch channel when a window looks bad.

After launch: monitoring and drift

Cover quality (sampled audit pass rate, negative feedback and reason codes, format/refusal/empty checks), usage, reliability (error, timeout, fallback activation), latency (p50/p95, time to first token), cost (spend vs. forecast, tokens per request), and outcome vs. holdout.

Cadence: daily checks for the first two weeks, weekly audits and golden-set additions for the next several, then monthly drift reviews and a quarterly cost/model review.

The monthly drift review is the step most teams skip, because nothing looks broken. Run it anyway — AI quality can decline with no change on your side:

• Input drift: longer inputs, new languages, new use cases?
• Output drift: length, refusal rate, format errors changing?
• Quality drift: audit pass rate trending down in any slice?
• Cost & latency drift: tokens per request creeping up?
• External changes: provider model update/deprecation, regulation, or vendor-term change?

End every review with written decisions and owners, even if the decision is "no action." If you already keep rollout_monitor.py next to the feature, add a drift snapshot once a month and diff it against launch baselines the same way you'd diff a performance regression.

If you'd rather not maintain the script

Not everyone on the launch call wants to run Python. We packaged the same launch, rollback, and monitoring checklist into the AI Feature Launch & Rollback Playbook ($19, editable docs plus a Notion-ready CSV). It covers pre-launch readiness by function, numeric rollback triggers, the rollback procedure, post-launch monitoring, and the monthly drift review: AI Feature Launch & Rollback Playbook.

Disclosure: we make and sell that playbook. You don't need it for anything in this article. The script above will do the decision-table job on its own.

If you're earlier in the process, the free AI Feature PRD Template is where the failure-mode table, degradation ladder, and rollout plan should live — those are what make a fast, calm rollback possible later.

A PM-focused version of this guide first appeared on the AI Spec Kit Substack.

What does your team's kill switch actually do when you flip it in staging, and how long does it take to drain? We'd like to hear the number.

---

This article was written with the help of AI tools, then reviewed and edited before publishing.