DEV Community

Taylor Zhu
Taylor Zhu

Posted on

Fifteen-Minute Rollback or No Merge: A Reversibility Checklist

If an AI-assisted change cannot be reversed in fifteen minutes, it is not production-ready. Green CI does not prove that. A passing demo does not prove that either.

You need four named proofs before merge: who owns the blast radius, what switch stops the new path, which command restores the old path, and what you will watch for the first hour. Missing any one of those is a fail. Not a comment. A fail.

Why agent diffs skip the unmerge clock

Agents optimize for "make the tests pass." They do not feel pager duty. They will add a consumer, a cron, a new index, or a side-effecting webhook because the prompt said implement. You review the happy path. Production reviews the undo path.

The cheap-code problem is not style. It is volume. When a patch is cheap to generate, reversible design becomes the scarce resource. Your gate has to encode that scarcity, or a Friday deploy will teach it to you.

This article is a copy-paste production-readiness checklist for that gap. It is not a model bake-off. It is not a claim that any assistant understands your traffic.

Four proofs, not an essay

Treat reversibility as a contract the PR must attach. Keep it short. If someone needs a narrative, the change is too large.

1. Owner of record

A human pager target. Not "the AI." Not a Slack channel with fifty people. One primary, one backup. If both are out, the gate fails.

2. Blast radius

What user journeys, tables, queues, and regions can this break? Write names, not "backend." If you cannot list them, you do not understand the change.

3. Stop switch

A feature flag, config kill, traffic weight, or process you can flip without a rebuild. Schema expansions that old binaries cannot ignore fail this proof unless you document an expand/contract sequence in the same packet.

4. Restore command

The exact command or runbook step that returns traffic or data to the prior behavior. "Revert the PR" is acceptable only when the change is stateless and any migration is additive and unused. If you wrote data, reverting git is not a restore command.

Copy-paste packet

The YAML below is a proposed template, not a completed incident. Swap team names and metric names for yours. Leave the structure.

# .reversibility.yml — proposed production gate
apiVersion: reversibility/v1
change:
  id: "pr-${PULL_REQUEST_NUMBER}"
  summary: "Add retry wrapper around billing capture"
  generated_by: "agent-assisted"
owner:
  primary: "oncall-payments"
  backup: "oncall-platform"
  timezone: "America/Los_Angeles"
blast_radius:
  journeys: ["checkout/capture", "checkout/refund-lookup"]
  datastores: ["payments.captures"]
  queues: ["captures.outgoing"]
  regions: ["us-west-2"]
stop_switch:
  type: "feature_flag"
  name: "payments.retry_wrapper"
  default: false
  disable_command: "flagctl disable payments.retry_wrapper --env prod"
restore:
  unmerge_minutes: 15
  data_written: false
  command: "kubectl rollout undo deploy/payments-api"
  verifies: "GET /internal/flag/payments.retry_wrapper returns false"
watch_first_hour:
  - "capture_success_rate"
  - "capture_latency_p99"
  - "retry_storm_count"
limits:
  max_files: 40
  max_migration_files: 0
Enter fullscreen mode Exit fullscreen mode

If data_written is true, command cannot be git revert alone. The validator below enforces that.

Fail the build when the packet is theater

Run this locally or in CI. It deploys nothing. It fails closed on missing fields, empty blast-radius lists, a rebuild-shaped stop switch, and the git-revert lie. Treat it as an unexecuted starting point until you wire it to your flag CLI.

#!/usr/bin/env python3
"""Fail closed if .reversibility.yml cannot be undone in 15 minutes."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit  = 2  # labeled example; fix locally if you paste this

REQUIRED = [
    ("owner", "primary"),
    ("owner", "backup"),
    ("stop_switch", "name"),
    ("stop_switch", "disable_command"),
    ("restore", "command"),
    ("restore", "verifies"),
]

BANNED_OWNERS = {"ai", "agent", "bot", "copilot", "nobody"}
REBUILD_HINTS = ("docker build", "terraform apply", "helm upgrade", "migrate up")


def fail(msg: str) -> None:
    print(f"reversibility: FAIL: {msg}", file=sys.stderr)
    raise SystemExit(1)


def main(path: Path) -> None:
    if not path.is_file():
        fail(f"missing {path}")
    data = yaml.safe_load(path.read_text()) or {}
    for section, key in REQUIRED:
        value = (data.get(section) or {}).get(key)
        if not value or not str(value).strip():
            fail(f"{section}.{key} is empty")

    owner = str(data["owner"]["primary"]).strip().lower()
    if owner in BANNED_OWNERS:
        fail("owner.primary must be a human pager target")

    radius = data.get("blast_radius") or {}
    for key in ("journeys", "datastores", "regions"):
        items = radius.get(key) or []
        if not items:
            fail(f"blast_radius.{key} must list real names")

    restore = data["restore"]
    minutes = int(restore.get("unmerge_minutes") or 0)
    if minutes <= 0 or minutes > 15:
        fail("restore.unmerge_minutes must be 1..15")

    command = str(restore["command"]).lower()
    disable = str(data["stop_switch"]["disable_command"]).lower()
    if any(h in disable for h in REBUILD_HINTS):
        fail("stop_switch requires a rebuild; that is not a kill switch")

    data_written = bool(restore.get("data_written"))
    if data_written and "git revert" in command:
        fail("data_written=true cannot use git revert as the restore command")

    watch = data.get("watch_first_hour") or []
    if len(watch) < 2:
        fail("watch_first_hour needs at least two metrics")

    print("reversibility: PASS")


if __name__ == "__main__":
    main(Path(".reversibility.yml"))
Enter fullscreen mode Exit fullscreen mode

Wire it as a required check. A suggested snippet, not a vendor-specific workflow:

python3 check_reversibility.py
# exit 1 blocks merge; do not allow a skip label for agent-assisted PRs
Enter fullscreen mode Exit fullscreen mode

Decision table you can paste into the PR template

Condition Gate Why
Owner is a model, bot, or rotating channel Fail Nobody gets paged
Blast radius is "all services" or blank Fail No one can scope the incident
Stop switch needs a rebuild or migration Fail You cannot stop it in minutes
data_written: true and restore is git revert Fail Git does not unwrite rows
unmerge_minutes > 15 Fail You chose a change that is too large
Additive, unused column + flag default off Pass Expand/contract is intact
Stateless config with kubectl rollout undo Pass Old ReplicaSet is still present
Dual-write with a documented drain command Pass You can stop the new writer first

Print the table in the PR body. Reviewers should argue with the row, not with vibes.

How teams fake the four proofs

You will see these. Reject them.

  • Owner set to a bot. The packet compiles. The pager does not.
  • Blast radius "platform." That is a department, not a journey.
  • Stop switch that ships in the same binary as the bug. You cannot disable what you must rebuild to change.
  • Restore command that is a wiki URL. A link is not a command. If the wiki is down, you are down twice.
  • unmerge_minutes: 15 next to a blocking backfill. Arithmetic is not a runbook.
  • Watch list copied from another service. You will alert on the wrong graph while the real queue melts.

A useful review question: "If this ships at 16:55, what do I type at 17:02?" If the author hesitates, the packet is incomplete.

Fill the packet without making the agent invent prod

Do not ask an assistant to guess your flag name. Feed it inventory you already trust: service list, flag catalog, dashboard UIDs, on-call schedule. Then ask it only to draft the YAML against that inventory.

Proposed prompt, not a magic spell:

Using ONLY the inventory pasted below, fill .reversibility.yml.
If a field is not in the inventory, write UNKNOWN and stop.
Do not invent flag names, metric names, or pager targets.
Inventory:
- services: ...
- flags: ...
- dashboards: ...
- oncall: ...
Enter fullscreen mode Exit fullscreen mode

If the draft contains UNKNOWN, the gate should fail. Unknown is information. A plausible fake name is not.

Keep the change small enough for a fifteen-minute undo. Split schema expand from schema contract. Split dual-write from cutover. Agents will gladly one-shot all three. You should not.

Run the gate before shared CI is the bottleneck

You can execute the validator on a laptop against a branch. That is the whole point: catch a missing restore command before reviewers debate style.

When you want the same check on a throwaway remote box while an assistant fills the YAML, you do not need a dedicated cluster. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are one way to draft the packet and run check_reversibility.py on a machine you can delete. They do not replace your on-call roster, your flag service, or your production metrics. If those are missing, skip the product and fix the roster.

What this will not save you from

  • Wrong physics. A fifteen-minute undo does not repair a unique constraint you already enforced on live rows.
  • Silent correctness. Rollback can restore an old bug you were trying to leave.
  • Multi-region drift. If only one region has the flag default, your packet is lying about blast radius.
  • Vendor runtimes you cannot roll back. If the change is a managed resource with no undo API, this checklist should fail until you have a compensating action.
  • Load you never measured. watch_first_hour is useless if the metrics do not exist yet.

The script does not talk to Kubernetes, feature flags, or billing. It only refuses empty theater. You still have to run the disable command in a staging environment once, on purpose, and paste the output into the PR.

Who should not use this

Skip this approach if you ship only documentation, or if every change is already behind a global dark-launch switch with a tested disable path. Skip it if your "production" is a single developer database with no users. Skip it if you will add a skip label on every agent PR; a gate that is optional is not a gate.

Do not use a reversibility packet as permission to merge unbounded agent diffs. The packet is a shrinking device. If you cannot name the undo, shrink the change until you can.

Ship the owner, the radius, the switch, and the restore command. Then merge. If you cannot, keep the branch. Fifteen minutes is the test. Everything else is a story about tests that were green.

Top comments (0)