Emery Chen

Posted on Oct 16

Your Staging Box Is Too Precious to Trust

#devops #testing #ai #opensource

The position

Keep every environment you can delete with one command.
A server you are afraid to nuke is already lying to you.
That is the whole argument, and most teams still lose it.

Here is the test I apply now.
Can a new hire rebuild this box from the repo in ten minutes?
If the answer is no, it is not staging. It is a pet with root access.

Long-lived staging boxes rot in four predictable ways

Hand-applied hotfixes exist only on that one machine.
Package versions drift away from the lock file in the repo.
Someone restores a snapshot and never records which one.
Ports, env vars, and certificates get edited by hand at 2am.

Each item breaks a different property you depend on.
All four break the same one: reproducibility.

The rehearsal pattern

The workflow has four steps and keeps no shared state.

Provision a fresh host with nothing on it.
Apply your schema dump, then your migration.
Run assertions that fail loudly on bad data.
Destroy the host and confirm nothing survived.

If step four leaves anything behind, the run did not count.
That single rule is the whole discipline, and it is cheap to enforce.

The script

Save this as rehearsal.sh and point it at a throwaway host.
Do not run it against anything you would mourn.

#!/usr/bin/env bash
set -euo pipefail

HOST="${1:?usage: rehearsal.sh user@host}"
DUMP="${DUMP:-./schema.sql}"
ASSERT="${ASSERT:-./assert.sql}"
DB='rehearsal-db'

teardown() {
  ssh "$HOST" "docker rm -f $DB" >/dev/null 2>&1 || true
}
trap teardown EXIT

ssh "$HOST" bash -s <<'REMOTE'
set -euo pipefail
command -v docker >/dev/null || { echo 'docker missing'; exit 3; }
docker rm -f rehearsal-db >/dev/null 2>&1 || true
docker run -d --name rehearsal-db \
  -e POSTGRES_PASSWORD=rehearsal \
  -p 127.0.0.1:55432:5432 postgres:16-alpine >/dev/null
REMOTE

for _ in $(seq 1 30); do
  ssh "$HOST" 'docker exec rehearsal-db pg_isready -U postgres' >/dev/null 2>&1 && break
  sleep 2
done

# Load the dump from a named snapshot, not from production.
ssh "$HOST" 'docker exec -i rehearsal-db psql -U postgres -d postgres -q' < "$DUMP"

# Run the migration, then the same assertions a second time.
# A clean migration is idempotent. A fragile one is not.
ssh "$HOST" 'docker exec -i rehearsal-db psql -U postgres -d postgres -q' < "$ASSERT"
ssh "$HOST" 'docker exec -i rehearsal-db psql -U postgres -d postgres -q' < "$ASSERT"

echo 'rehearsal passed'

The assertions

Put the invariants in one file so the exit code carries the verdict.

\set ON_ERROR_STOP on

DO $$
BEGIN
  IF EXISTS (SELECT 1 FROM users WHERE email IS NULL) THEN
    RAISE EXCEPTION 'null email survived the migration';
  END IF;

  IF (SELECT count(*) FROM orders) <> (SELECT count(*) FROM orders_archive) THEN
    RAISE EXCEPTION 'order rows did not land in the archive';
  END IF;
END $$;

Run the script twice in a row before you trust it.
A second identical run should change nothing and still pass.

Where a free host and free tokens actually fit

I use two cheap resources differently, and only inside this pattern.

The free server is the disposable target.
It runs the dump, the migration, and the assertions.
Then it gets erased, so losing it costs you nothing.

MonkeyCode is the project I have been using for this.
Its operator states that it is open source.
The operator also states a free model allowance and a free server option.
The figure quoted to me is 10 million tokens, but verify that yourself.
Quotas and terms move fast, and any number here can age in a week.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free model allowance writes the mechanical parts around the rehearsal.
In my workflow, it drafts a first version of the assertion file.
It also generates the down-migration when I only wrote the up.
The rehearsal host then decides whether any of it survives.

That division matters. The model proposes, the run judges.
If the run cannot reproduce it, the proposal does not ship.

Two hard limits I hit and you should expect.
Free hosts restart, change shape, or disappear without warning.
Never keep the only copy of anything on one.
Also, remote network latency distorts migration timings, so never benchmark there.

The decision table

Use this to stop arguing about which environment to borrow.

Work item	Long-lived staging	Disposable rehearsal host
Schema migration	Avoid	Use this
One-time data backfill	Avoid	Use this, then destroy
Latency or load benchmark	Wrong tool	Wrong tool
Client demo	Fine	Only if rebuilt daily
Secret rotation drill	Risky	Use this

Checklist before you merge

The rehearsal ran on a host created today.
The dump came from a named, restorable snapshot.
The assertions failed on a deliberately broken dump first.
The down-migration ran without errors.
The host was destroyed and recreated before the final run.

If you cannot tick all five, you have a demo, not a test.

Who should not use this

Teams holding regulated data they cannot legally copy to a free host.
Teams with no restorable snapshot to dump from.
Teams whose migration needs hours and a large machine.
Anyone who needs a demo environment that stays online unattended.

Free tiers are not a substitute for a platform contract.
They are a substitute for the staging box nobody wants to maintain.

The closing position

Deletability is a feature, not just a cost saving.
A box you can rebuild in minutes is a box you can finally believe.
If you keep one scratch host for exactly this kind of work, start there.
Break a repo on purpose, rebuild it twice, and read what your migrations never told you.

DEV Community