I will not accept a production page until the handoff packet names a rollback file. That sounds stubborn on a noisy night, and it should, because unowned mitigations become the next incident. Alerts without a five-minute diagnostic budget turn into guesswork, and guesswork is how freeze rules get skipped. So the packet comes first, freeze comes second, and any assistant-generated command stays off production.
Speed from agents is not permission to skip the packet
People keep asking whether an agent should call production APIs without ever leaving the editor window. Have you watched a tool-calling demo and felt your pager twitch at the same time? I have, and that uneasy feeling is exactly why this handoff packet exists before diagnosis. Fast patches are useful on a scratch box, but they are reckless while the customer path stays unfrozen.
The public conversation around agents and tool calling is loud right now, and on-call is the quiet counter-argument. An assistant that can draft a command is not an assistant that may run that command. I want command generation isolated from the cluster that is actually paging me right now.
Fill the handoff packet before you type a diagnostic
I keep a YAML packet next to the service, and I refuse to start diagnosis until every required field has a value. The packet is not a novel, and it is not a postmortem; it is a small transfer object. If I get hit by a second alert, the backup on-call should continue from this file alone.
# proposed example: oncall-handoff.packet.yaml
apiVersion: oncall.packet/v1
kind: HandoffPacket
metadata:
alert_id: example-ALERT-0000
service: checkout-api
opened_at: 2026-09-23T02:14:00Z
spec:
diagnostic_budget_seconds: 300
rollback_file: deploy/rollback/checkout-api.last-good.txt
rollback_file_sha256: pending
customer_visible: true
freeze_scope: checkout-api-canary
unfreeze_owner: primary-oncall
escalation_target: payments-secondary
first_commands:
- kubectl get deploy checkout-api -o wide
- kubectl logs deploy/checkout-api --tail=200 --since=10m
- curl -sS https://status.internal.example/checkout-api/healthz
notes: proposed packet; do not treat hostnames as real inventory
Does that YAML look heavy for a single page, or does it look lighter than a wrong restart? Every field exists because I have watched pages drift when the rollback path lived only in someone's head. I treat missing fields as a failed start, not as optional documentation I can fill after mitigation.
Fields I fill in a fixed order
- I copy
alert_idfrom the page so the packet and the pager cannot silently diverge later. - I point
rollback_fileat a real artifact so mitigation is a file path, not a remembered command. - I set
diagnostic_budget_secondsto three hundred so the first five minutes cannot quietly become an hour. - I name
freeze_scopeas a concrete surface, because freezing the whole company is not a strategy. - I name
unfreeze_owneras one person, so thawing production is never an implied group decision at night. - I name
escalation_targetas a different person, so a spent budget already has a next human.
Those six fields are the contract I accept with the rotation, and I will not bargain them down during the incident. If a field is unknown, I page the service owner instead of inventing a placeholder that looks complete. A pretty packet with fake owners is worse than a refused page, because it launders uncertainty into action.
The rollback file itself is boring on purpose, and boring is what I want under a paging alert. I keep last-good image, replica count, and config identity in one text file beside the deploy path. If that file is missing, the validator below refuses the page before I touch a shell.
# proposed example: deploy/rollback/checkout-api.last-good.txt
image: checkout-api:sha-example
replicas: 3
configmap: checkout-api-2026-09-01
First commands stay read-only even when a restart looks obvious
I map each alert to commands that cannot change cluster state, and I paste them from the packet. If a command needs apply, delete, restart, or scale, it does not belong in the first_commands list. Ask yourself this: would I still run that command if the unfreeze owner were offline tonight?
# proposed read-only first commands for a latency page
kubectl get deploy checkout-api -o jsonpath='{.spec.replicas}{"\n"}'
kubectl describe pod -l app=checkout-api | sed -n '1,80p'
kubectl get events --field-selector involvedObject.name=checkout-api --sort-by=.lastTimestamp
curl -sS -o /tmp/healthz.json -w "%{http_code}\n" https://status.internal.example/checkout-api/healthz
sha256sum deploy/rollback/checkout-api.last-good.txt
I record the output hashes in the packet notes before I argue about causes with anyone. Why bother hashing terminal output during a live page, when the logs are already scrolling away? Because the next person should see the same evidence I saw, not a vanished buffer.
A freeze annotation is not a first command, even when the dashboard is screaming and the room wants motion. I will sketch it with --dry-run=client after the budget expires, and I still will not apply it from an assistant session. Human ownership stays on the freeze, and the scratch box stays a scratch box.
# not a first command; only after budget expiry and named freeze_scope
# proposed freeze annotation, human-owned, never assistant-owned
kubectl annotate deploy checkout-api oncall.packet/frozen=true --dry-run=client -o yaml
Freeze only what the packet names, then keep unfreeze as a second signature
The question is not whether freezing feels dramatic; the question is which named surface the packet is allowed to freeze. I freeze only the freeze_scope listed in the file, and I do not freeze adjacent services because the dependency graph looks scary. Unfreeze requires the named unfreeze_owner, a current packet, and a rollback file whose checksum still matches.
If any of those three checks fail, production stays frozen, and I escalate instead of improvising a thaw. Is that slower than a hopeful restart on a red dashboard at two in the morning? Yes, and that slower path is the trade I want when the customer path is still burning.
| Packet state | Customer visible? | Action I take | Who may unfreeze |
|---|---|---|---|
| rollback_file missing | yes | refuse the page, page the service owner | nobody yet |
| budget still open | yes | run first_commands only | unfreeze_owner after budget |
| budget expired, cause unknown | yes | freeze freeze_scope, escalate | escalation_target after review |
| cause known, rollback_file present | no | optional freeze, then planned change | unfreeze_owner |
| assistant drafted a patch | either | keep patch on scratch server | never the assistant |
That table is the whole policy I will defend on the incident bridge without extra rows. If a situation is not a row in the table, I do not invent a fifth path while the alert is still firing. Improvisation belongs in the postmortem, and it does not belong in the unfreeze signature.
The five-minute clock is a hard stop, not a suggestion
I start a timer when the packet validates, and I do not extend it because the logs look interesting. Interesting logs after minute five belong in the escalation note, not in a longer local debug session. If the first commands did not name a cause, I freeze freeze_scope and I stop typing.
I also stop asking an assistant for one more hypothesis once the budget is gone. Extra hypotheses are cheap; extra mutations during freeze are how you write the next page. The clock exists to protect the freeze rule, not to make me look decisive on the bridge.
Where an isolated assistant belongs in this loop
I still want help drafting the packet, especially when a new alert type shows up without a template. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode as a coding assistant with free model access and a free server option, kept off production. The product is in the loop only as a scratch drafting aid, never as the unfreeze owner.
The useful loop is narrow, and I will not pretend it replaces the on-call human holding the pager. I paste a redacted alert description into the isolated session and ask for candidate read-only commands, then I reject anything that mutates state. The free server is a scratch place to run the validator below, not a jump host into the cluster that is paging.
# proposed prompt I keep next to the packet (unexecuted example)
Redact customer data. Propose only read-only diagnostic commands.
Each command must name the binary, the resource, and the output I should save.
Do not propose kubectl apply, delete, rollout restart, or scale.
Do not propose unfreeze steps. Return a YAML fragment for first_commands only.
If you want a scratch box for that drafting loop, MonkeyCode's free server option is one isolated place I will try. I still copy the YAML back into the repository by hand, because the packet is an owned artifact, not a chat transcript. No generated fragment lands in git until I have read every command against the mutating-word list.
A validator I run before I accept the rotation
This is a proposed Python check, not a production metric, and it fails closed when required fields are empty. Run it on the scratch server or on your laptop; do not run it as a cluster controller. I want a refuse-the-page exit code, not a dashboard that argues with the freeze owner.
#!/usr/bin/env python3
"""Proposed handoff packet validator. Example only; not production incident data."""
from __future__ import annotations
import hashlib
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError:
print('install pyyaml in the scratch environment, not on the frozen host')
sys.exit(2)
MUTATING = re.compile(
r'\b(apply|delete|restart|scale|cordon|drain|replace|patch)\b',
re.I,
)
REQUIRED = (
'alert_id',
'rollback_file',
'diagnostic_budget_seconds',
'freeze_scope',
'unfreeze_owner',
'escalation_target',
)
def load_packet(path: Path) -> dict:
data = yaml.safe_load(path.read_text())
if not isinstance(data, dict):
raise ValueError('packet must be a mapping')
spec = data.get('spec') or {}
meta = data.get('metadata') or {}
return {**meta, **spec}
def main() -> int:
packet_path = Path(sys.argv[1] if len(sys.argv) > 1 else 'oncall-handoff.packet.yaml')
pkt = load_packet(packet_path)
errors: list[str] = []
for key in REQUIRED:
if not pkt.get(key):
errors.append(f'missing {key}')
budget = pkt.get('diagnostic_budget_seconds')
if not isinstance(budget, int) or budget <= 0 or budget > 900:
errors.append('diagnostic_budget_seconds must be an int between 1 and 900')
rollback = Path(str(pkt.get('rollback_file', '')))
if not rollback.is_file():
errors.append(f'rollback_file does not exist: {rollback}')
else:
digest = hashlib.sha256(rollback.read_bytes()).hexdigest()
print(f'rollback_file sha256={digest}')
commands = pkt.get('first_commands') or []
if not commands:
errors.append('first_commands must contain at least one read-only command')
for cmd in commands:
if MUTATING.search(str(cmd)):
errors.append(f'mutating first command: {cmd}')
if pkt.get('unfreeze_owner') == pkt.get('escalation_target'):
errors.append('unfreeze_owner and escalation_target must be different people')
if errors:
print('REFUSE THE PAGE')
for item in errors:
print(f'- {item}')
return 1
print('packet complete; freeze_scope may be applied by the named owner')
return 0
if __name__ == '__main__':
raise SystemExit(main())
python3 validate_handoff_packet.py oncall-handoff.packet.yaml
# expected for a complete packet: prints sha256 and a single success line
# expected for a broken packet: prints REFUSE THE PAGE and exits 1
If the validator prints REFUSE THE PAGE, I do not start first commands, and I do not ask an assistant to skip the check. The backup on-call can run the same script on their laptop and should get the same refusal. That repeatability is the only reason I bother checking the handoff packet in code at all.
Escalation is a packet transfer, not a feeling
When the budget clock hits zero, I stop proposing new hypotheses and I freeze the named scope. I paste the packet, the command output hashes, and the validator result into the escalation note. Then I page escalation_target and I stay on the call as a reader, not as a second unfreeze owner.
What do I refuse to send during that handoff, even if the channel is already busy? I refuse secret-mixed chat logs, assistant-only patches, and freezes of services the packet never named. Escalation moves the packet, and it does not dump my anxiety onto the next engineer.
Limitations, and who should not copy this
This workflow assumes you can name a rollback file before the page, which baby services and weekend prototypes often cannot. If you have no deploy artifact, no service owner, and no secondary, the validator will refuse every page, and that refusal is correct. Do not use this contract as a reason to skip paging humans who actually know the service.
I also will not run assistant-drafted commands against production because a free server made the script look tidy. The isolated environment has no authority over freeze or unfreeze, and it should stay that way. Teams with a formal incident commander already owning freeze policy should not bolt this packet on as a second source of truth.
If your alerts are purely informational, a five-minute budget is theater, and you should fix routing instead of filling YAML. This article is a proposed on-call contract, not a claim about latency numbers, token quotas, or model rankings I did not measure. Use it where ownership is real, and ignore it where the service cannot even name a rollback file.
Top comments (0)