DEV Community

Emery Huang
Emery Huang

Posted on

Keep the Assistant on a Scratch Host Until a Human Diff Unfreezes Prod

I will not feed a live page to an assistant until a scratch host is named, isolated, and written into the freeze note. Production stays frozen while that host tries to reproduce the symptom with copied fixtures, not live credentials. A generated patch remains a suggestion until a human pastes a diff and signs the unfreeze line. If this feels fussy, ask yourself who you want holding the deploy key at 2 a.m.

Why I separate the model from the page

The failure mode I am writing against is not a dumb suggestion, but a useful suggestion aimed at the wrong machine. Someone pastes the alert, the tool offers a patch, and the same shell session still has production environment variables loaded. Would you trust a completion that can also see your current kube context and your deploy role? I would not, because a single wrong directory is enough to turn a drill into an outage.

I want the assistant where it can be wrong in public, on a host that cannot reach the production network path. That host can be small, temporary, and boring, as long as its name is in the note before the first prompt. The page still needs a human owner, because a model does not carry the pager and does not join the bridge. When the scratch host and the production host share a name, I stop and rename one of them before I continue.

Alerts I will actually accept

I accept a page only when these four lines are filled, and I bounce it back when any line is blank. Each line is a fact I can check, not a vibe about how severe the chart looks. A blank line is a bounce, not a debate, because the bridge should not invent missing facts while the chart is red. Would you rather argue about severity, or spend those minutes filling the four lines I listed below?

  1. The alert name, the firing time, and the service identifier are copied from the page, not retyped from memory.
  2. A symptom query is named, and that query is read-only, so the first action cannot mutate state.
  3. A scratch host name is reserved, and that name is different from every production host in the rotation sheet.
  4. Two clocks are written down: one for scratch reproduction, and one for a human diff review.

First commands I run before any prompt

I start on the production host with commands that only read, and I paste their output into the note before I open a chat. The goal is a symptom packet, not a fix, because the fix conversation can wait until the packet is boring and complete. If a command would change a file, restart a process, or roll a deployment, it does not belong in this first pass. I would rather look slow on the bridge than explain a write command I cannot undo after the fact.

What the probe is allowed to do

# Unexecuted example. Run only on a host you already own.
set -eu
hostname
date -u +%Y-%m-%dT%H:%M:%SZ
git -C /srv/app status --short || true
git -C /srv/app rev-parse --short HEAD || true
# Read-only symptom probe. Replace with your service's real query.
curl -fsS --max-time 5 http://127.0.0.1:8080/healthz || echo "healthz_failed"
Enter fullscreen mode Exit fullscreen mode

I copy that output into a local file called symptom.txt, and I do not upload secrets that sometimes hide in environment dumps. A health check failure is enough context for a first prompt, and a stack trace can be trimmed before it leaves the host. Would you paste a database URL into a chat window just to save one tired minute? I hope your answer is no, because that saved minute is cheaper than rotating a leaked credential later.

What I refuse to paste into the prompt

I keep the prompt smaller than the note, because the note can hold more context than a chat window should see. I paste the alert name, the trimmed health output, and the question I want answered, and I leave credentials out. A model that asks for a token, a kubeconfig, or a production hostname is asking me to break the freeze. I close that chat and continue with the local editor, because convenience is not an exception to the note.

Prompt questions that stay useful

  • Can you rewrite this gate so a missing freeze note exits before any append, and show the diff only?
  • Can you list which lines in this health output are symptoms, and which lines are just timestamps?
  • Can you draft a freeze-note template with scratch host, reviewer, and both clocks, without inventing a service name?

Escalation clocks that are not one vague minute

I do not use one vague escalate-if-needed line, because that line is how pages stall while everyone waits for someone else. I set a reproduction clock at fifteen minutes, and I set a human-diff clock at thirty minutes from the page time. If the scratch host cannot replay the symptom by the first clock, I call the service owner instead of prompting harder. If the human diff is still missing at the second clock, I call a second engineer and keep production frozen.

Those numbers are a starting proposal for a small team, not a benchmark I measured across companies. Change them in your runbook if your bridge already has a tighter rule, and write the chosen numbers into the note. The point is that both clocks exist before the assistant is opened, so the model cannot become the escalation path. Ask yourself who you call when the suggestion looks confident and the graph still looks wrong.

Freeze and unfreeze, with the assistant in the rule

Freeze means no deploy, no config write, and no apply of generated code on any host whose name appears in the production inventory. Unfreeze means a named human has reviewed a diff that was produced on the scratch host and copied back as text. I do not unfreeze on a model claim that tests passed, because I did not watch those tests myself. The freeze note must name the scratch host, the symptom file, and the person who may sign the unfreeze line.

What the scratch gate checks

Here is the gate I want in the scratch session before anyone applies a patch, even on the scratch host. It is a proposal, not a tool I have certified, and you should read every line before you trust it. The script exits if it sees a production marker, because a scratch workflow that runs on prod is just an outage with extra steps. I keep it short so a tired reviewer can actually read it on a bridge call.

#!/usr/bin/env bash
# Proposal only. Not executed in this article. No network calls.
set -euo pipefail

: "${SCRATCH_HOST:?set SCRATCH_HOST}"
: "${FREEZE_NOTE:?set FREEZE_NOTE to a local note path}"
: "${HUMAN_REVIEWER:?set HUMAN_REVIEWER}"

if [[ "${PRODUCTION:-0}" == "1" || "${DEPLOY_ENV:-}" == "prod" ]]; then
  echo "refusing: production marker is set" >&2
  exit 2
fi

this_host="$(hostname -s)"
if [[ "${this_host}" != "${SCRATCH_HOST}" ]]; then
  echo "refusing: hostname ${this_host} is not ${SCRATCH_HOST}" >&2
  exit 3
fi

if [[ ! -f "${FREEZE_NOTE}" ]]; then
  echo "refusing: freeze note missing" >&2
  exit 4
fi

grep -q "scratch_host=${SCRATCH_HOST}" "${FREEZE_NOTE}" || {
  echo "refusing: freeze note does not name this scratch host" >&2
  exit 5
}

printf '%s host=%s reviewer=%s action=scratch_ok\n' \
  "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${this_host}" "${HUMAN_REVIEWER}" \
  >> "${FREEZE_NOTE}"
echo "scratch gate passed; still frozen until a human diff is signed"
Enter fullscreen mode Exit fullscreen mode

A three-step drill before the rotation

I treat the script as untrusted until this drill passes on a host that has no production route. The drill is a proposal you can run in a shell, and I have not published timing numbers from it. If a step fails, fix the script or the note, and do not skip ahead to a model prompt. Would you trust a gate you have never watched fail on purpose during a quiet drill?

  1. Export PRODUCTION=1 on the scratch host and confirm the gate script exits with status 2 before you unset the variable.
  2. Point SCRATCH_HOST at a wrong name and confirm the script exits before it appends a line to the freeze note.
  3. Run the script with matching names and confirm it appends one scratch_ok line and still prints that production remains frozen.

A small decision table for the bridge

I keep this table in the runbook so the bridge does not invent a new policy while the graph is red. Read the row, do the action, and write the time next to the row you used. If two rows seem to match, I take the stricter row and I stay frozen until the note is updated. Would you rather argue about a row for ten minutes, or follow the stricter one and argue after the page is quiet?

Situation What I do Freeze state
Symptom file is missing Run the read-only probe, save output, do not prompt Stay frozen
Scratch host name matches a prod host Rename the scratch host and update the note Stay frozen
Reproduction fails before the first clock Call the service owner with the symptom file Stay frozen
Patch exists but no human diff Wait, or call the second engineer at the second clock Stay frozen
Human diff is signed and tests were watched Unfreeze only the named service, then watch the same probe Unfreeze that service

Where a free assistant fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I want a second editor only for the scratch-host script, the freeze note, and the trimmed symptom summary. That is the only job I will give an assistant during a page, and I will not hand it the deploy role. MonkeyCode is relevant here as an open-source assistant only when free model access and a free server are still offered. I am not naming a token quota, a model, or hardware, because this draft does not cite a current pricing page.

Check the project docs yourself before you plan a shift around any free tier you might have heard about. Treat a missing or unclear limit as a reason to stay on tools you already run locally. Would you bet a page on a free tier whose current cap you have not opened and read? I would not, so the runbook names the local editor as the fallback when the docs are silent.

The free server option matters only as a candidate scratch host, and only after you confirm it cannot see production credentials or production networks. I would use it to rewrite the gate script, to draft the freeze note, or to replay a fixture I already copied by hand. I would not point it at a live cluster, and I would not paste deploy keys into the prompt to save a step. If the free server and the production VPC share a route, it is not a scratch host, no matter how convenient the signup felt.

Remove the product name and the workflow still stands: isolate the editor, freeze production, demand a human diff, and escalate on two clocks. The assistant is a text worker on a disposable machine, not an on-call engineer and not a change-management system. If the free access disappears tomorrow, I still want the same gate script and the same table in the runbook. That is the test I use before I mention a tool in a page procedure at all.

Limits, and who should not copy this

This approach assumes you can spare a separate host, a local note file, and a second human before the second clock expires. It is a poor fit for a solo operator who has no one to sign the diff, because the unfreeze rule then becomes theater. It is also a poor fit for a data-loss page that needs a vendor bridge in the first minutes. A scratch replay can waste the only useful minutes when the vendor already holds the timeline.

I have not measured mean time to recovery with this gate, and I will not pretend a blog draft is an incident report. Do not use this script as a security boundary against a hostile insider, because it only checks a few environment markers and a hostname. Do not store customer data in the symptom file, and do not treat a model transcript as an audit log. If your regulator wants a signed change record, keep using that system and let this note point at it.

What I would do before the next page

I would copy the gate script into a private drill repo and point it at a host that is already empty. I would then fail the script on purpose with a production marker, and I would keep the failure in the drill note. After that drill passes, I would add the decision table and pick both clocks with the people who carry the pager. The page gets safer when the freeze note names a host a human can still refuse, not because a tool was free.

Top comments (0)