48-Hour Field Notes: The Dry Run That Read the Wrong Database

The ticket said the migration had been dry-run. The log said nothing was pending. The column on the shared Postgres instance was still the old one.

That gap is cheap to misread. A laptop shell resolved DATABASE_URL to a sqlite file under /tmp. The server shell resolved the same name to the host written on the ticket. Both commands exited 0. Only one of them had rehearsed the change.

These notes are a 48-hour lab window for that miss: what to try, what breaks, and what is worth repeating. Replay them only on machines you administer. They are not a customer postmortem, and they report no timings, token totals, or uptime. The snippets are a harness to copy. Until you run them, they are unexecuted examples, not measurements.

The false green

A dry run is a fire drill. If the alarm is wired to the break room, the drill can look tidy while the server room never hears it.

File order creates the false green. A repo root may load .env, then .env.local, then a leftover export in a laptop profile. The process on the server may load a unit file written by the deploy. Same key. Different winner. An assistant that starts in the repo often trusts the first file it can parse.

SQLite makes the miss quiet. A URL with no host becomes a file, the driver creates that file, and SELECT 1 succeeds. You get a green log and an untouched server. Postgres never gets a vote.

Hour 0 is the green log. Do not merge on it. Hour 1 is one question: which host did this shell actually parse?

Refuse the wrong room

The artifact does not migrate. It exits non-zero unless the URL in the current shell is a non-loopback host you named before the command ran.

#!/usr/bin/env bash
# Unexecuted lab harness. Copy it, then run it on a host you administer.
# Refuses a dry-run shell unless DATABASE_URL matches the ticket host.
set -euo pipefail

expected_host="${EXPECTED_DB_HOST:?set EXPECTED_DB_HOST}"
url="${DATABASE_URL:?DATABASE_URL is empty in this shell}"

python3 - "$url" "$expected_host" <<'PY'
import sys
from urllib.parse import urlparse

url, expected = sys.argv[1], sys.argv[2]
parsed = urlparse(url)
scheme = (parsed.scheme or "").split("+", 1)[0]
host = parsed.hostname or ""

if scheme in {"sqlite", ""} or url.startswith("sqlite:"):
    sys.stderr.write("refusing: sqlite or file URL in a server dry-run\n")
    sys.exit(2)
if host in {"", "localhost", "127.0.0.1", "::1"}:
    sys.stderr.write("refusing: loopback host %r\n" % host)
    sys.exit(3)
if host != expected:
    sys.stderr.write("refusing: host %r != expected %r\n" % (host, expected))
    sys.exit(4)
sys.stdout.write("host=%s scheme=%s\n" % (host, scheme))
PY

postgresql+psycopg:// is why the scheme is split on +. urlparse keeps the driver suffix inside the scheme, so the check should see postgresql, not a private driver name, and it should still reject sqlite. Host comparison stays exact. A wiki nickname is not a hostname.

Run the same command in both shells. Diff the host lines. Do not print the raw URL. Userinfo often holds a password, and a transcript that looks successful is a poor vault.

EXPECTED_DB_HOST=db.internal.example ./scripts/assert_db_host.sh

Exit 2 is a file database. Exit 3 is loopback. Exit 4 is a real host that is still the wrong one. Exit 0 means this shell matches the ticket host. It does not mean the migration is safe, the backup exists, or you are allowed to run either.

Then compare shells. The only datum this method claims is string equality.

# Lab-only. Use a host you administer. Never echo the full URL.
local_host=$(EXPECTED_DB_HOST=db.internal.example ./scripts/assert_db_host.sh)
remote_host=$(ssh [e-mail] 'EXPECTED_DB_HOST=db.internal.example ./scripts/assert_db_host.sh')
test "$local_host" = "$remote_host"

A failing test is the stop sign. If the lines differ, the dry run you already trusted was a rehearsal in the wrong room. Fix the shell that will execute the change. Do not edit the ticket prose until the lines match.

A small fixture locks the refusal so a later cleanup cannot quietly accept sqlite. This test is also unexecuted until you run it.

import os
import subprocess

def test_refuses_sqlite():
    env = os.environ.copy()
    env["EXPECTED_DB_HOST"] = "db.internal.example"
    env["DATABASE_URL"] = "sqlite:////tmp/app.db"
    proc = subprocess.run(
        ["./scripts/assert_db_host.sh"],
        env=env,
        capture_output=True,
        text=True,
    )
    assert proc.returncode == 2
    assert "sqlite" in proc.stderr

Where the assistant is allowed to sit

Drafting the guard is a fair job for a coding assistant. Running it in a second shell you control is a fair job for a remote dev box. MonkeyCode can sit in that pairing if you want free model access and a free server option in one workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Only those two availability claims are used here. This note names no model, no token quota, no machine size, and no expiry. Older posts and chat replies go stale. If a number conflicts with the current project page, keep the page and drop the number.

Keep the assistant on the harness. Ask it to review the exit codes, then read the diff yourself. Run the script in the server shell with your own hands. If the model proposes accepting localhost so a laptop demo stays green, reject the edit. That patch is the original bug in a cleaner shirt.

Use the free server as a shell whose files you can list, pointed at a lab database you are allowed to drop. Run the guard first. Run the vendor dry-run only after exit 0, in that same shell. A dry run before the guard is how the false green gets built.

What broke while the harness was still wrong

The first draft printed DATABASE_URL. The host was right and the lab still failed, because the secret landed in scrollback and in the assistant transcript. The cut worth repeating prints scheme and host only.

The second break was an empty variable that looked like success. With nounset off, echo "$DATABASE_URL" exits 0 when the name is unset. The log is a blank line, and a later tool may fall through to a default file. ${DATABASE_URL:?...} fails closed. Leave it closed.

The third break was quote stripping. A rewrite inlined the URL into the Python text instead of passing argv. A password containing a quote became a syntax error, or became part of the program. Arguments stay arguments. The heredoc is quoted, <<'PY', so the shell does not expand the body.

The fourth break was a hostless DSN. A UNIX socket or a cloud connector leaves hostname empty, and the script exits 3. That stop is the result. Do not ask the model to skip the check so the command can proceed. Extend the parser only when a person can name the endpoint without pasting a secret into the prompt.

What to repeat, and who should walk away

Repeat the two-shell host diff. Repeat a human read of any edit to the guard. Repeat the rule that the assistant never receives the migration command, a production URL, or a dump.

Do not repeat a pass that lets the model rewrite .env until the diff looks calm. Calm text can still point at /tmp/app.db.

Walk away if you do not administer the server. Walk away if the database holds customer data, or if a free shared shell is the only place a secret could be pasted. Free availability is not an isolation proof.

Walk away if you need an audit trail the current docs do not describe. Walk away if the driver has no hostname and nobody can say what the host means. A refusal you immediately disable is theater.

This check is also the wrong tool for backups, migration locks, and snapshot restores. Two instances can share a nickname in a ticket and still be different machines. Take the expected host from the connection string you intend to change, not from memory.

Before you try the pairing

Read the current MonkeyCode docs and confirm what free model access and the free server option include today, including logs and environment handling. Where this note and the docs disagree, the docs win.

Then run the harness against a lab URL you can revoke. If the local line and the server line differ, you have the failure this window is about. Fix the shell before you fix the story.