Cut Over the Model Path or Don't Ship: A Fail-Closed Inference Checklist

If production can still reach the model endpoint you used while drafting, you did not cut over. A green deploy does not prove that. It only proves the process started.

Treat the inference path as a promotion surface. Pin the base URL, the model identity, the timeout, and the fallback. If any of those still point at a lab or shared drafting host, fail the gate. Do not “try prod first and fall back to the sandbox.” That pattern turns an outage into a data leak.

This is not a vibe-coding essay. It is a copy-paste checklist plus a small policy checker you can run in CI. Use it when an AI feature is about to leave the branch.

The failure you are actually shipping

You added a chat box, a summarizer, or a “explain this diff” button. During development the client pointed at whatever answered quickly. That is normal. It is also how lab hosts become silent production dependencies.

The bug is not “we used a model.” The bug is an unnamed path. Someone pasted a base URL into a helper. A generated client baked it into a default. A retry wrapper treats HTTP 429 from prod as a reason to call the drafting host. None of that shows up in a unit test that mocks complete().

You need evidence that prod traffic cannot reach the lab path. You also need evidence that when prod is unhealthy, the feature fails closed instead of wandering.

What “cut over” means here

Cut over is not “we created a prod API key.” Cut over means all four of these are true at once:

1. The production base URL is an allowlisted origin, not an environment default inherited from .env.local.
2. The model identifier is an explicit production contract, not default, latest, or an empty string.
3. Timeouts, token ceilings, and retry counts are set in prod config and are smaller than “keep hammering.”
4. Fallback is deny-by-default: no lab host, no alternate vendor you have not reviewed, no “best effort” second try against the drafting server.

If you cannot show those four, you are still in the sandbox. Ship the feature flag off.

Six gates, each with a receipt

Every gate needs an artifact. A Slack “looks good” is not a receipt.

Gate 1 — Named production origin

Pass: INFERENCE_BASE_URL in the production secret store matches a committed allowlist. The value is HTTPS, has no path wildcards, and is not a personal tunnel.

Fail closed: any prod manifest that still contains localhost, 127.0.0.1, ngrok, trycloudflare, or a hostname tagged lab, dev, sandbox, or draft.

Receipt: the commit SHA of inference-allowlist.json plus the secret-store version id.

Gate 2 — Model identity is not a moving alias

Pass: production sets INFERENCE_MODEL_ID to a pinned id your vendor or self-hosted gateway documents. The same id appears in the runbook.

Fail closed: latest, auto, empty, or a name that only exists on the drafting server.

Receipt: a one-line contract in the repo: model id, max tokens, and who owns rotation.

Gate 3 — Lab host denylist in CI

Pass: CI greps deployable config (Helm, Terraform, Docker Compose prod overlay, sealed secrets templates) and fails on denylisted hosts.

Fail closed: denylist bypassed with “temporary” comments, or the check only scans src/ and ignores deploy/.

Receipt: the CI job log URL for the merge commit.

Gate 4 — Budgets are numbers, not folklore

Pass: production config sets request timeout, connect timeout, max retries, and a per-request token ceiling. Retries do not change the destination.

Fail closed: unlimited retries, exponential backoff with no cap, or a retry that swaps base_url.

Receipt: the config snippet and a test that asserts the client is constructed once with prod settings.

Gate 5 — Fallback cannot re-enter the lab

Pass: on timeout, 5xx, or quota errors, the feature returns a user-visible failure and an internal metric. No second client.

Fail closed: “if prod fails, call the free server so the demo still works.”

Receipt: a failing integration test that stubs prod errors and asserts the lab host is never dialed.

Gate 6 — Traffic identity

Pass: production requests send a stable X-Service / User-Agent and an environment tag prod. Logs can answer “which app, which model id, which base URL” without reading source.

Fail closed: the drafting client and the prod client are the same binary with the same defaults.

Receipt: one redacted log line from a staging call that already uses the prod origin.

Copy-paste checklist

Use this in the PR. Check a box only when the artifact exists.

• [ ] inference-allowlist.json lists the single prod origin (or the exact set of regional origins).
• [ ] inference-denylist.json lists lab, draft, and tunnel hosts.
• [ ] Prod overlay sets INFERENCE_BASE_URL and INFERENCE_MODEL_ID; neither is blank.
• [ ] Client construction test fails if base_url is missing.
• [ ] Error-path test fails if any HTTP call target is on the denylist.
• [ ] Timeout, retry cap, and token ceiling are set in prod, not only in a README.
• [ ] Feature flag default is off until Gate 1–5 receipts are linked.
• [ ] Runbook names the owner of model-id rotation.
• [ ] Rollback is “flag off,” not “point DNS back at the sandbox.”

If a box depends on “we will add it after launch,” the gate failed.

Artifact: a policy file CI can reject

Commit this as inference-policy.json. Keep it boring. Boring is reviewable.

{
  "allow_base_urls": [
    "https://inference.prod.example.internal"
  ],
  "deny_host_substrings": [
    "localhost",
    "127.0.0.1",
    "ngrok",
    "trycloudflare",
    "lab.",
    "sandbox.",
    "draft.",
    "dev-inference"
  ],
  "require_env": [
    "INFERENCE_BASE_URL",
    "INFERENCE_MODEL_ID",
    "INFERENCE_TIMEOUT_MS",
    "INFERENCE_MAX_RETRIES",
    "INFERENCE_MAX_OUTPUT_TOKENS"
  ],
  "max_retries": 1,
  "forbid_fallback_base_url": true
}

Replace the allow URL with yours. Do not add the drafting host “just for staging” in the same file that production loads. Staging gets its own overlay.

Artifact: a fail-closed checker

The script below is a proposal you can run locally and in CI. It does not call any model. It only inspects env and text files you pass as deploy roots. Label it unproven against your repo until you execute it once and keep the log.

#!/usr/bin/env python3
"""Fail closed if prod config can still reach a lab inference host."""

from __future__ import annotations

import json
import os
import sys
from pathlib import Path
from urllib.parse import urlparse

POLICY = Path("inference-policy.json")
SCAN_ROOTS = [Path("deploy"), Path("k8s"), Path("infra"), Path(".")]
SCAN_SUFFIXES = {".yml", ".yaml", ".json", ".tf", ".env", ".toml"}


def load_policy() -> dict:
    if not POLICY.exists():
        print("FAIL: inference-policy.json missing")
        sys.exit(2)
    return json.loads(POLICY.read_text())


def host_of(url: str) -> str:
    parsed = urlparse(url if "://" in url else f"https://{url}")
    return (parsed.hostname or "").lower()


def denied(host: str, needles: list[str]) -> str | None:
    for needle in needles:
        if needle.lower() in host:
            return needle
    return None


def check_env(policy: dict) -> list[str]:
    errors = []
    for key in policy["require_env"]:
        if not os.environ.get(key):
            errors.append(f"missing env {key}")
    base = os.environ.get("INFERENCE_BASE_URL", "")
    if base:
        host = host_of(base)
        hit = denied(host, policy["deny_host_substrings"])
        if hit:
            errors.append(f"INFERENCE_BASE_URL host matches denylist '{hit}': {host}")
        allowed_hosts = {host_of(u) for u in policy["allow_base_urls"]}
        if host not in allowed_hosts:
            errors.append(f"INFERENCE_BASE_URL host not allowlisted: {host}")
    model = os.environ.get("INFERENCE_MODEL_ID", "")
    if model.lower() in {"", "latest", "auto", "default"}:
        errors.append(f"INFERENCE_MODEL_ID is not pinned: {model!r}")
    try:
        retries = int(os.environ.get("INFERENCE_MAX_RETRIES", "99"))
    except ValueError:
        retries = 99
        errors.append("INFERENCE_MAX_RETRIES is not an int")
    if retries > int(policy["max_retries"]):
        errors.append(f"retries {retries} exceed policy max {policy['max_retries']}")
    return errors


def check_files(policy: dict) -> list[str]:
    errors = []
    needles = policy["deny_host_substrings"]
    for root in SCAN_ROOTS:
        if not root.exists():
            continue
        for path in root.rglob("*"):
            if not path.is_file() or path.suffix.lower() not in SCAN_SUFFIXES:
                continue
            if path.name == POLICY.name:
                continue
            # Skip local-only samples; prod overlays must still pass.
            if ".local." in path.name or path.name.endswith(".example"):
                continue
            text = path.read_text(errors="ignore")
            lower = text.lower()
            for needle in needles:
                if needle.lower() in lower and "prod" in path.parts:
                    errors.append(f"{path}: denylist hit {needle!r}")
    return errors


def main() -> int:
    policy = load_policy()
    errors = check_env(policy) + check_files(policy)
    if errors:
        print("FAIL-CLOSED: inference cutover incomplete")
        for item in errors:
            print(f" - {item}")
        return 1
    print("PASS: inference path looks cut over (config scan only)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Wire it so merge is impossible when it exits 1:

# .github/workflows/inference-cutover.yml
name: inference-cutover
on:
  pull_request:
    paths:
      - "deploy/**"
      - "k8s/**"
      - "infra/**"
      - "inference-policy.json"
      - ".github/workflows/inference-cutover.yml"
jobs:
  fail-closed:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Require prod env in the CI job (from secrets, not the PR)
        env:
          INFERENCE_BASE_URL: ${{ secrets.INFERENCE_BASE_URL }}
          INFERENCE_MODEL_ID: ${{ secrets.INFERENCE_MODEL_ID }}
          INFERENCE_TIMEOUT_MS: ${{ secrets.INFERENCE_TIMEOUT_MS }}
          INFERENCE_MAX_RETRIES: ${{ secrets.INFERENCE_MAX_RETRIES }}
          INFERENCE_MAX_OUTPUT_TOKENS: ${{ secrets.INFERENCE_MAX_OUTPUT_TOKENS }}
        run: python3 scripts/check_inference_cutover.py

Secrets belong in the store, not in the PR description. If CI has no prod URL, the job must fail. A skipped check is an open gate.

Decision table

Symptom	What you might tell yourself	Fail-closed action
Drafting host still in prod overlay	“Staging needs it”	Split overlays. Prod overlay cannot parse the lab hostname.
Model id is latest	“We want improvements automatically”	Pin an id. Rotate with a ticket and a replay test.
Prod 429, client calls lab	“Users should still get an answer”	Return an error. Page the owner. Never change base_url in retry.
Unit tests mock the SDK	“Coverage is high”	Add one test that inspects the constructed URL.
Tunnel URL in a hotfix	“Only for this incident”	Incident flag off. Do not hot-patch origin.
Free drafting server is fast today	“We can launch on it”	Launch is a contract plus capacity you control. Speed is not a contract.

Where a free drafting path belongs

Drafting against a shared or free inference endpoint is fine. Shipping that endpoint is not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need a place to sketch clients, prompts, and the checker itself, MonkeyCode’s free model access and free server option can stay on the drafting side of the line. Keep that origin on the denylist for production overlays. Generate the policy file there if you want. Then run the checker against the repo you actually deploy. The product mention ends here because the gate does not care which editor you used. It cares whether prod can still dial the lab.

Who should not use this approach

Do not use this checklist as a substitute for load tests, eval suites, or a real vendor contract. It will not tell you the model is accurate. It will not size GPUs. It will not prove GDPR. It only proves you did not leave a drafting path wired into the live service.

Skip it if you have no production inference origin yet. In that case the honest gate is “do not ship the feature,” not “scan a policy file that allowlists a wish.”

Also skip it if your app does not call a model at runtime. Generated code that never performs inference is a different review: dependency pins, tests, and side effects. This article is only for the path that answers complete() after deploy.

Limitations

The scanner is string-based. Obfuscated URLs, runtime service discovery, and hosts injected by a sidecar can evade it. Pair it with egress policy in the mesh or network layer. If the cluster can resolve the lab hostname, add a network deny.

The allowlist does not prove capacity. A pinned origin can still 429. Fail closed on that too: user-visible error, metric, page. Do not reopen the sandbox to absorb overflow.

Aliases will drift. Re-run the job when you rotate model ids. If nobody owns rotation, you do not have a contract. You have a default.

Ship rule

Name the production origin. Pin the model id. Deny the lab host in CI. Cap retries without changing destination. Link the job log on the PR. If you cannot, leave the flag off.

That is the whole method. The drafting environment can be free, shared, or noisy. Production cannot inherit it.