DEV Community

Taylor Zhu
Taylor Zhu

Posted on

The Lab Host Is Not Prod: A Fail-Closed Promotion Checklist

#testing #ai #devops #programming

You can prototype on a free model and a free server. You cannot promote that host. Treat the lab as disposable, pin a promotion contract, and fail closed when the evidence files are missing.

That is the rule. Everything below is a copy-paste checklist for the cutover, not another merge ritual.

The leak you will actually ship

AI-assisted services rarely blow up because a generated loop is ugly. They blow up because the lab leaked.

You scaffold a handler. You run it on the same box that hosts the assistant. You paste the lab URL into a client. You leave a debug token in .env. A teammate then deploys "the same thing."

Unit tests will not catch that. A PR checklist about tools or retries will not catch it either. Promotion is a different gate. If the process that built the service still owns the runtime, you do not have production. You have a shared notebook with a port.

What counts as a lab

Call it a lab when all of these are true:

  • The coding assistant, the app process, and scratch data can share a host.
  • Model access is convenience-first, not contract-first.
  • Secrets are whatever unblocked the demo.
  • Nobody can name the production identity of the service.

Free model access and a free server option are useful in that phase. They become a problem only when you skip the cutover and ship the box you happened to be using.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want a scratch environment with free model access and a free server option, MonkeyCode is an open-source project some teams park that work on. The checklist does not depend on it. Strip the product name out and the gates still stand.

Copy this promotion contract

Keep one file in the repo: PROMOTION.yaml. Deploy jobs should refuse to start unless every gate is pass and every evidence path exists on disk. Do not type pass by hand. Generate the files, then let a checker read both.

# PROMOTION.yaml — proposal for a local clone, not an executed audit
version: 1
service: billing-adapter
lab:
  allowed_hosts:
    - "127.0.0.1"
    - "lab.internal.example"
  must_not_appear_in:
    - deploy/
    - infra/
    - clients/
gates:
  - id: HOST_SPLIT
    question: "Is production a different host than the lab?"
    evidence: evidence/host-split.txt
    fail_closed: "Refuse deploy if a lab hostname appears under deploy/, infra/, or clients/."
  - id: SECRET_MATERIAL
    question: "Were lab tokens, cookies, and debug keys rotated out?"
    evidence: evidence/secret-scan.json
    fail_closed: "Refuse deploy if any lab secret pattern still matches."
  - id: MODEL_PIN
    question: "Is every model call pinned to a named contract, not the lab default?"
    evidence: evidence/model-pin.md
    fail_closed: "Refuse deploy if the client still reads LAB_MODEL or an unset MODEL_ID."
  - id: DATA_CLASS
    question: "Is production data blocked from the lab host?"
    evidence: evidence/data-class.md
    fail_closed: "Refuse deploy if the lab can still reach prod datastores."
  - id: OBSERVABILITY
    question: "Can you name the prod service in logs, traces, and alerts without the lab project name?"
    evidence: evidence/obs-names.txt
    fail_closed: "Refuse deploy if service.name still equals the assistant project."
  - id: BLAST_RADIUS
    question: "Is there a one-command rollback that does not SSH into the lab?"
    evidence: evidence/rollback.md
    fail_closed: "Refuse deploy if rollback docs mention the lab host."
Enter fullscreen mode Exit fullscreen mode

Six gates. Six files. If a file is missing, the answer is no.

Fail-closed checker you can run locally

The script below is a proposal. Run it in a clone you control. It does not open a network connection. It fails if evidence is missing, if a gate is not pass, or if a forbidden lab host string appears under the listed trees.

#!/usr/bin/env python3
"""promotion_gate.py — fail closed when lab evidence is missing."""
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("install pyyaml in your local venv before running", file=sys.stderr)
    sys.exit(2)

REQUIRED_STATUS = {"pass"}


def load_contract(path: Path) -> dict:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict) or "gates" not in data:
        raise ValueError("PROMOTION.yaml must contain a gates list")
    return data


def scan_forbidden_hosts(root: Path, hosts: list[str], trees: list[str]) -> list[str]:
    hits: list[str] = []
    for tree in trees:
        base = root / tree
        if not base.exists():
            continue
        for path in base.rglob("*"):
            if not path.is_file():
                continue
            text = path.read_text(encoding="utf-8", errors="ignore")
            for host in hosts:
                if host and host in text:
                    hits.append(f"{path}: contains {host}")
    return hits


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", type=Path, default=Path("."))
    parser.add_argument("--config", type=Path, default=Path("PROMOTION.yaml"))
    args = parser.parse_args()

    contract = load_contract(args.config)
    failures: list[str] = []

    for gate in contract["gates"]:
        evidence = args.root / gate["evidence"]
        if not evidence.is_file():
            failures.append(f"{gate['id']}: missing {evidence}")
            continue
        marker = evidence.read_text(encoding="utf-8", errors="ignore")
        if "STATUS: pass" not in marker:
            failures.append(f"{gate['id']}: {evidence} has no STATUS: pass line")

    lab = contract.get("lab", {})
    hits = scan_forbidden_hosts(
        args.root,
        lab.get("allowed_hosts", []),
        lab.get("must_not_appear_in", []),
    )
    failures.extend(hits)

    if failures:
        print("PROMOTION FAILED")
        for item in failures:
            print(f"- {item}")
        return 1

    print(json.dumps({"result": "ok", "gates": [g["id"] for g in contract["gates"]]}))
    return 0


if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode

Wire it so deploy never starts without the checker:

python3 -m pip install pyyaml
python3 promotion_gate.py --root . --config PROMOTION.yaml
Enter fullscreen mode Exit fullscreen mode 
.PHONY: promotion-gate
promotion-gate:
    python3 promotion_gate.py --root . --config PROMOTION.yaml
Enter fullscreen mode Exit fullscreen mode

If your pipeline cannot see evidence/, that is a fail, not a skip.

What each evidence file has to prove

Write short files. A novel in evidence/ is how teams hide a missing gate.

HOST_SPLIT

Prove the production runtime is a different machine, cluster, or account than the lab. A useful evidence/host-split.txt looks like this:

STATUS: pass
lab_host: 127.0.0.1
prod_runtime: ecs://billing-adapter-prod
checked_trees: deploy/, infra/, clients/
command: rg -n "127.0.0.1|lab.internal.example" deploy infra clients
Enter fullscreen mode Exit fullscreen mode

If rg prints a line, you do not have a split. You have a rename.

SECRET_MATERIAL

Rotate anything the lab touched. Demo cookies, assistant tokens, copied Authorization headers, default DB passwords. Then record the scan, not the secret.

rg -n "LAB_|DEBUG_TOKEN|sk-live|BEGIN OPENSSH" --glob '!.git' . \
  && echo "still dirty" && exit 1
printf 'STATUS: pass\nscanner: rg\npatterns: LAB_, DEBUG_TOKEN\n' > evidence/secret-scan.json
Enter fullscreen mode Exit fullscreen mode

Adjust the patterns for your repo. Do not paste live credentials into the evidence file. The file is a receipt, not a backup.

MODEL_PIN

Lab defaults are not a contract. Pin the caller so production cannot silently follow whatever the assistant used that afternoon.

STATUS: pass
caller: src/llm_client.py
required_env:
  - MODEL_ID
  - MODEL_ENDPOINT
  - MAX_OUTPUT_TOKENS
forbidden_env:
  - LAB_MODEL
unset_means: fail closed, do not fall back
Enter fullscreen mode Exit fullscreen mode

If the client still does os.getenv("MODEL_ID", "whatever-the-lab-used"), the gate is red. A default is a second lab.

DATA_CLASS

Keep production records off the lab host. That includes logs that embed payloads, copied fixtures, and "just one customer row."

STATUS: pass
prod_datastores:
  - postgres://prod-billing
lab_may_read:
  - fixtures/synthetic/
lab_may_not_read:
  - customers
  - invoices
egress_check: no prod DSN in lab compose files
Enter fullscreen mode Exit fullscreen mode

If your lab compose file still contains the prod DSN "for realism," throw the compose file away. Realism is not a reason to mix classes.

OBSERVABILITY

Rename the service before the first prod request. Lab project names in service.name, dashboard folders, or alert titles will page the wrong people.

rg -n "service.name|OTEL_SERVICE_NAME|job:" deploy infra \
  | tee evidence/obs-names.txt
# then append:
# STATUS: pass
# prod_name: billing-adapter
# lab_name_absent: yes
Enter fullscreen mode Exit fullscreen mode

BLAST_RADIUS

Rollback must not require SSH into the machine where you vibe-coded the demo. If the runbook says "just restart the lab process," you have no blast-radius control.

STATUS: pass
rollback: kubectl rollout undo deploy/billing-adapter
does_not_use: ssh lab.internal.example
owner_alert: #billing-oncall
Enter fullscreen mode Exit fullscreen mode

Decision table: promote, rebuild, or throw away

Signal Promote Rebuild in prod-shaped infra Throw the lab away
Lab hostname in clients No Yes, after stripping If clients cannot be found
Model id unset at runtime No Yes, after pinning If nobody can name the contract
Prod data reached the lab No Only after rotation and review Default when customer data moved
Rollback needs lab SSH No Yes If no other runtime exists
Evidence file missing No No Yes, until the file exists

Read the table left to right. Missing evidence is not "unknown." Missing evidence is "do not promote."

A 20-minute dry run

Label this unexecuted until you run it on your clone.

  1. Create PROMOTION.yaml and empty evidence/ files without STATUS: pass.
  2. Run python3 promotion_gate.py. Confirm it exits 1.
  3. Put http://127.0.0.1:8080 into clients/demo.env. Run it again. Confirm the host scan fails.
  4. Remove the lab URL. Fill each evidence file with a STATUS: pass line and the commands you actually ran.
  5. Run the checker again. Only then allow the deploy job to see the artifact.

If step 2 passes, the checker is wrong. A first run with empty evidence must fail.

Limitations

This checklist does not prove load, correctness, or latency. It does not replace threat modeling, IAM review, or a real staging environment. String search will miss encoded hostnames, image layers, and secrets injected at runtime. STATUS: pass is only as honest as the person who wrote it.

Free servers and free model access also have limits you cannot inventory from a YAML file: capacity, tenancy, retention, and how long the option remains available. Do not treat a lab uptime number as a production SLO. Do not treat a convenience default as a vendor contract.

Who should not use this

Skip this approach if you already promote through a locked-down path that forbids third-party assistants on any host that can see production data. Skip it if you need a compliance certificate; a repo checklist is not one. Skip it if the service never leaves the laptop. And skip it if you were about to point production DNS at the same process that runs the assistant.

In that last case the fix is not a better YAML file. The fix is a different host.

If you need a disposable lab before you run this checker, MonkeyCode's free model access and free server option are one way to keep that scratch work off the production cluster. Use it as a lab. Then make the checker fail until the lab disappears from deploy/.

Top comments (0)