Reproduce the Pager on a Scratch Box Before You Unfreeze Prod

I freeze production writes the moment the pager fires, and I keep that freeze until a scratch box reproduces the same alert fingerprint. Model-suggested patches do not get a live shell. They get a replica, a failing test, and a human ACK. If the replica cannot fail the same way, I escalate with evidence instead of guessing on prod. Does that feel slower than pasting kubectl at three in the morning?

The live shell is the wrong first destination

When the pager lights up, I want a command that feels like progress, not another pretty graph. I have watched myself paste a confident one-liner into production because a coding model sounded sure of itself. Have you done that, then spent the next hour explaining a second outage to people who were asleep? The postmortem always starts the same way: we treated a suggestion as a runbook.

The useful question is whether I can make this alert fail on a box that is allowed to break. Until that replica exists, production stays read-only for me, including shells that look harmless. The freeze is not a mood I announce in chat; it is a lock file, a fingerprint, and a named unfreeze rule. If I cannot point at those three objects, I am still in observe-only, and that is fine.

An “agent” sitting beside the pager is usually a loop with better copy. Loops are good at drafting probes. Loops are not on-call authority, and I will not pretend otherwise during a SEV.

The replica-first contract

I keep four objects in the incident channel, and I refuse writes until all four exist:

• alert_id plus a fingerprint hash of the symptom, never the essay the model wrote
• a freeze record that names who holds the write lock
• a scratch host that is allowed to die without a customer notice
• a failing probe that matches the fingerprint on that host

If any object is missing, the only legal production commands are observe-only. Want a patch? Build it against the replica until the probe goes green. Want to unfreeze? Show me that red-to-green transition, then get a human ACK. Why would I skip the replica just because the prompt included the word urgent?

Freeze and unfreeze rule

• Freeze trigger: first page of a mutating class (latency, error-budget, data-loss, authz), or any model-suggested shell that is not a read.
• Allowed under freeze: logs, metrics, traces, kubectl get, unlocked SELECT, and copying artifacts onto the scratch box.
• Forbidden under freeze: restarts, scale, config apply, schema changes, feature flags, secret rotation, and anything that changes blast radius.
• Unfreeze: a human types UNFREEZE <alert_id> <replica_test_id> <ack_name> after the replica matches and the probe is green. No bot may emit that token.

Is the model allowed to draft the unfreeze line for me? Yes. Can it send that token into the lock path? No, and I will not wire that up.

First commands I actually run

I keep a wrapper so the first ten minutes look the same every night. The wrapper refuses to continue when freeze.lock is missing, which surprises people who wanted to “just look around with apply.”

#!/usr/bin/env bash
# replica-gate.sh — labeled example: observe-only on prod while freeze.lock exists
set -euo pipefail
ALERT_ID="${1:?alert_id}"
LOCK="${LOCK:-/var/oncall/freeze.lock}"
REPLICA_HOST="${REPLICA_HOST:?set REPLICA_HOST to a scratch box}"

fingerprint() {
  printf '%s' "$ALERT_ID" | sha256sum | awk '{print $1}'
}

if [[ ! -f "$LOCK" ]]; then
  echo "no freeze.lock; refusing to proceed without an explicit freeze" >&2
  exit 2
fi

if ! grep -q "writes=forbidden" "$LOCK"; then
  echo "freeze.lock does not forbid writes; fix the lock before any shell" >&2
  exit 3
fi

FP="$(fingerprint)"
HOLDER="$(awk -F= '/^holder=/{print $2}' "$LOCK")"
echo "alert_id=$ALERT_ID fingerprint=$FP freeze_holder=$HOLDER"

# legal first commands: reads only
kubectl --context prod get deploy,po,endpoints -A | head
# ship the symptom pack to the replica, never a write to prod
ssh "$REPLICA_HOST" "mkdir -p /incidents/$ALERT_ID && printf '%s\\n' '$FP' > /incidents/$ALERT_ID/fingerprint"

The lock file is equally boring, which is the point. I want something a tired person can grep without interpreting a novel.

# /var/oncall/freeze.lock  (example)
alert_id=ALT-1842
holder=emery
class=latency
frozen_at=2026-09-14T02:11:08Z
writes=forbidden
unfreeze_requires=replica_match,human_ack

If you cannot name the replica host next to that file, you are not ready to unfreeze. Why would a model know your blast radius if you have not even named the box that is allowed to break?

Unfreeze is a separate script, not a second thought in the same shell history.

#!/usr/bin/env bash
# unfreeze.sh — labeled example: humans only, replica test id required
set -euo pipefail
ALERT_ID="${1:?alert_id}"
TEST_ID="${2:?replica_test_id}"
ACK_NAME="${3:?ack_name}"
LOCK="${LOCK:-/var/oncall/freeze.lock}"
PROBE_LOG="${PROBE_LOG:-/incidents/$ALERT_ID/probe.log}"

if [[ ! -f "$LOCK" ]]; then
  echo "nothing frozen" >&2
  exit 1
fi
if [[ "$ACK_NAME" == "model" || "$ACK_NAME" == "bot" ]]; then
  echo "bots cannot unfreeze" >&2
  exit 2
fi
if ! grep -q "PASS $TEST_ID" "$PROBE_LOG" 2>/dev/null; then
  echo "replica test $TEST_ID is not green; stay frozen" >&2
  exit 3
fi

printf 'UNFREEZE %s %s %s\n' "$ALERT_ID" "$TEST_ID" "$ACK_NAME" | tee -a /var/oncall/unfreeze.log
mv "$LOCK" "$LOCK.released.$(date -u +%Y%m%dT%H%M%SZ)"

Scratch lane: draft the probe, do not draft prod

I need a place that can run the failing probe without sharing a kernel with customers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode's free model access and free server option as that scratch lane: the model drafts the probe, the free server runs it, and production stays frozen.

I do not treat those free options as capacity planning, an SLO, or a benchmark. I treat them as a disposable workbench. If the workbench is slow or missing, I still have the freeze. The incident does not earn mutate rights on prod because the scratch lane hiccuped.

A session on the scratch box gets a tight prompt. Notice the model only sees the fingerprint and a redacted log slice, not cluster-admin.

# inputs I paste into the coding session (redacted example)
alert_id: ALT-1842
fingerprint: 9c1e...
symptom: p99 checkout > 2s, error rate 4% on /pay
forbidden: kubectl apply, helm, terraform, DROP, restart
required_artifact: a probe that exits 1 when the fingerprint matches

#!/usr/bin/env bash
# probes/pay_p99.sh — labeled example, not a production measurement
set -euo pipefail
URL="${1:?base url of the replica app}"
python3 - "$URL" <<'PY'
import sys, time, urllib.request
url = sys.argv[1].rstrip("/") + "/pay"
lat, err = [], 0
for _ in range(30):
    t = time.time()
    try:
        urllib.request.urlopen(url, timeout=3)
        lat.append((time.time() - t) * 1000)
    except Exception:
        err += 1
        lat.append(3000.0)
lat.sort()
p99 = lat[int(0.99 * (len(lat) - 1))]
# fingerprint match: high latency or errors on the replica only
sys.exit(0 if p99 < 2000 and err == 0 else 1)
PY

When that probe exits 1 on the replica, I have a contract instead of a vibe. The model may propose a patch against the replica checkout. I apply the patch on the scratch box only. I rerun the probe. Green on replica is necessary. It is not sufficient. A human still has to unfreeze, using the script above, with a real name.

Escalation that prefers evidence over heroics

If the replica cannot reproduce in fifteen minutes, I do not “just try it on prod to see.” I escalate with a structured note. Reproduction failure is a first-class outcome, not a personal failure, and I say that in the channel so nobody improvises a restart.

Escalation note I paste:

1. alert_id and fingerprint
2. freeze holder and freeze start time
3. replica host and probe path
4. what matched (logs, metrics) and what did not (cannot trip the probe)
5. the ask: owner of the service, not a more aggressive command

Who gets the page next? The service owner named in the catalog, not the model, not the loudest person in Slack. If the catalog has no owner, I stay frozen and I say that out loud. Unowned services do not get emergency writes from me, even when the suggested patch looks tiny.

Decision table I keep above the fold

Signal	Replica result	Production action	Next human step
Pager, mutating class	Not started	Freeze writes, observe-only	Name replica host
Probe cannot fail	No match	Stay frozen	Escalate “cannot reproduce”
Probe fails like pager	Match	Stay frozen, patch replica	Review diff, rerun probe
Probe green on replica	Match then green	Still frozen	Human UNFREEZE with names
Unfreeze token present	Green	Apply the same diff in a change window	Watch fingerprint for 15 min
Fingerprint returns	Green then red	Re-freeze immediately	Escalate, do not loop patches

Would I let a coding loop walk that table by itself? Not on my pager. The table is the control plane. The model is a clerk that fills in the probe column.

A twenty-minute drill you can run tomorrow

This is a labeled drill, not a war story with fake graphs, and it should feel slightly annoying.

1. Pick a non-prod service and create a synthetic alert class drill-latency.
2. Write freeze.lock with your name and the drill alert_id.
3. Point REPLICA_HOST at a throwaway VM or any scratch server you already control.
4. Ask a coding model for a probe that exits 1 on the synthetic symptom only.
5. Break the replica on purpose, confirm the probe is red, patch the replica, confirm green.
6. Practice the unfreeze line out loud. Then delete the lock. Do not touch prod.

If step 4 wanders into kubectl apply -f for the live cluster, the drill already paid for itself. You found the hole before the pager did. Can you finish the drill without once opening a write to production? That is the only pass condition I care about.

Limitations, and who should skip this

This runbook assumes you can isolate a scratch box and that alert text is allowed to leave the production network. If you are in a sealed environment, keep the replica inside the same trust boundary and do not paste dumps into any external model. If the outage is the data itself, reproduction can be harmful; freeze and escalate instead of cloning customer records onto a free box.

Free model access and a free server are convenience, not an incident SLO. I do not know their quotas, hardware, or uptime, and I will not invent numbers to sound current. If your company already has a staging cluster, use that and keep the same freeze file. If you cannot freeze writes because your platform has no lock, this article will not save you; build the lock first.

Do not use this approach when the page is a simple disk-full on a single node with a documented runbook. Do not use it to delay a well-rehearsed failover. The replica gate is for the messy middle: model-drafted remediations, unclear blast radius, and the urge to type something heroic before anyone has a failing test.

What I want next time I am paged

I want the first artifact to be a red probe on a box that can die, not a green dashboard after I rolled the dice. I want the unfreeze line to include a human name that will still answer email tomorrow. I want the model in the room as a drafter of tests, not as an operator with root.

Steal the freeze file and the decision table. Run the twenty-minute drill on a service nobody will miss. Keep the model off the live shell until a human says UNFREEZE.