Opinion: AI Server Changes Need a Fault Drill, Not Just a Rollback Plan

A rollback plan tells you how to undo an AI change, but not what breaks first when the change stays in place. Most production incidents do not begin with a deliberate rollback; they begin with an unexpected failure mode that the author never tested. I now treat a passing fault drill as a precondition for reviewing any AI-generated server patch. The drill runs on a disposable server before a human reads a single line of the diff.

Why a rollback plan is not enough

A rollback plan answers a question about the past: how do we return the system to a known state? A fault drill answers a question about the future: what happens when this change meets a condition the author did not imagine? The second question decides whether you get paged at 3 a.m. A change with a perfect rollback can still fail in a way that nobody notices until the data is gone.

Free model access changes the economics of this argument, because generation stops being the bottleneck and verification starts. When a draft is nearly free, the cheapest verification is the one that breaks the change on purpose. A rollback plan is documentation; a fault drill is evidence. Documentation tells you what should happen, while evidence tells you what actually happens on a real service manager.

The fault drill in five steps

The workflow assumes two cheap resources: a model that generates failure hypotheses from a diff, and a server that can be destroyed after the drill. MonkeyCode's free model access covers the first, and its free server option covers the second, so a drill costs almost nothing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Any ephemeral VM or container host works if you prefer a different provider.

1. Generate failure modes before you apply anything

Ask the model to enumerate failure modes for the diff, and forbid it from proposing fixes, because fixes are a distraction at this stage. The prompt below is the one I use, and it produces a catalog that the drill can test. Treat the model's list as a hypothesis set, not a safety analysis. The list is useful because it is cheap and broad, and it is dangerous only if you trust it without running the drill.

You are reviewing a systemd unit change. Here is the diff:
<diff>

List the five most likely failure modes of this change on a running server.
For each failure mode, state: (1) the trigger, (2) the observable symptom,
(3) whether a plain restart recovers the service, (4) whether data can be
lost. Do not propose fixes.

2. Provision a disposable server and apply the change

Provision a server that mirrors the target's OS and service manager, apply the change, and confirm that the service starts cleanly. The server must be disposable, because the next step will deliberately break it and you should not care. A shared staging box is the wrong target, since a drill that corrupts another team's state teaches the wrong lesson.

3. Run the fault drill

Run fault-drill.sh against the unit and a sentinel file that represents data the service must not lose. The script applies each fault, waits a few seconds, records the service state, attempts a manual restart, and checks the sentinel. Each fault is applied to a restored system, so one bad result does not contaminate the next row.

4. Read the table like a reviewer

Each output row classifies one failure mode along three axes: service state after the fault, recovery after a manual restart, and sentinel integrity. A row with down, manual-fail, and lost is a reject, while a row with active, manual-ok, and intact is boring. Boring is the highest compliment a failure mode can earn, because boring means the incident is already solved by existing machinery.

5. Decide, and write the verdict into the review

Approve the change only when every tested failure mode is boring or recoverable, and document the verdict next to the diff. If any fault loses the sentinel, the change fails review regardless of how clean the diff looks. A clean diff is a statement about intent, while a drill result is a statement about behavior, and behavior is what runs in production.

The artifact: fault-drill.sh

The script is intentionally small, because a drill you cannot read in one sitting is a drill you will not run. It takes a unit name, a sentinel path, and a list of faults, and it prints a classification table. Run it only on the disposable server, and restore the unit between faults so the results stay independent. Treat the script as a starting point, not a certification suite, and adjust the fault list and the fill size to your unit and disk before you trust a verdict.

#!/usr/bin/env bash
# fault-drill.sh — classify failure modes of a systemd unit on a disposable server
# Usage: ./fault-drill.sh <unit> <sentinel> [fault ...]
set -euo pipefail

UNIT="$1"
SENTINEL="$2"
shift 2

CONFIG="$(systemctl show -p FragmentPath --value "$UNIT")"
CONFIG_BAK="${CONFIG}.drill-bak"

restore() {
  [[ -f "$CONFIG_BAK" ]] && mv "$CONFIG_BAK" "$CONFIG"
  rm -f /tmp/drill-fill
  systemctl daemon-reload 2>/dev/null || true
  systemctl reset-failed "$UNIT" 2>/dev/null || true
}

printf '%-14s %-8s %-12s %s\n' "FAULT" "STATE" "RECOVERY" "SENTINEL"
for fault in "$@"; do
  case "$fault" in
    kill)      systemctl kill --signal=SIGKILL "$UNIT" || true ;;
    stop)      systemctl stop "$UNIT" || true ;;
    restart)   systemctl restart "$UNIT" || true ;;
    corrupt)   cp "$CONFIG" "$CONFIG_BAK"; printf 'garbage\n' >> "$CONFIG" ;;
    fill-disk) dd if=/dev/zero of=/tmp/drill-fill bs=1M count=256 status=none || true ;;
    revoke)    cp "$CONFIG" "$CONFIG_BAK"; chmod 000 "$CONFIG" ;;
    *)         echo "unknown fault: $fault" >&2; exit 2 ;;
  esac

  sleep 3

  if systemctl is-active --quiet "$UNIT"; then state="active"; else state="down"; fi

  if systemctl start "$UNIT" 2>/dev/null; then recovery="manual-ok"; else recovery="manual-fail"; fi

  if [[ -s "$SENTINEL" ]]; then sentinel="intact"; else sentinel="lost"; fi

  printf '%-14s %-8s %-12s %s\n' "$fault" "$state" "$recovery" "$sentinel"
  restore
done

Example invocation against a unit that has already been changed on the disposable server:

./fault-drill.sh myapp.service /var/lib/myapp/data.db kill stop restart corrupt fill-disk revoke

Sample output for a change that survives the boring faults but loses data when the disk fills:

FAULT         STATE    RECOVERY     SENTINEL
kill          active   manual-ok    intact
stop          down     manual-ok    intact
restart       active   manual-ok    intact
corrupt       down     manual-fail  intact
fill-disk     down     manual-ok    lost
revoke        active   manual-fail  intact

Note what the table does that a diff review cannot: it separates the failure modes a restart absorbs from the ones that destroy data. The fill-disk row is the reason this change fails review even though every other fault was boring. The revoke row is the subtle one, because the service still reports active while a restart fails, so the next reboot turns a healthy unit into a permanently down one.

Reading the results like a reviewer

Apply the verdict of the worst row, not the average, because one lost row dominates any number of intact rows. Data loss is the only outcome that a rollback plan cannot undo, which makes it the natural veto for the whole drill.

Worst row observed	Verdict
active + manual-ok + intact	boring; approve and move on
down + manual-ok + intact	recoverable; add an alert for the trigger
active/down + manual-fail + intact	fragile; require a guard or a fix before approval
any row with lost	reject the change

Limitations and who should skip this

The drill is a heuristic, not a chaos-engineering platform, and it tests a handful of faults rather than the full failure space. It does not exercise load, latency, multi-node coordination, or the behavior of a degraded dependency. The free model's failure-mode list is a hypothesis generator, and every hypothesis must be confirmed by the drill before it counts as evidence.

Skip this workflow if you have no disposable environment, because drilling production or shared staging is worse than not drilling at all. Skip it if your change touches multiple hosts, because a single-node drill will give you false confidence. Skip it if your team cannot act on a lost row, because a drill without a rejection rule is just theater.

A rollback plan is the floor, not the ceiling, for reviewing AI-generated server changes. The ceiling is a fault drill that shows you which failure modes are boring and which ones lose data. Free model access makes the hypothesis list cheap, and a free disposable server makes the evidence cheap. The only expensive part is the discipline to reject a change that fails one row. If your team runs a drill like this, publish the boring failure modes you catalog; they are worth more than the impressive ones.