If You Cannot Replay the Call, Do Not Ship

You should not ship that AI feature yet. First replay a failed model call from disk. A one-time demo is not shipping evidence.

Cheap inference changed the wrong incentive for teams. You now generate features faster than frozen contracts. That gap is the real failure mode today.

The position

Here is a hard position with no hedge. You do not need another productized chat box. You need a replayable inference contract in git.

If reconstruction fails, that feature stays unfinished. Shipping without a replay file is simply reckless.

Teams still treat spare tokens as launch fuel. That is the wrong use of spare capacity. Spare capacity should prove that failure reconstruction works.

Why live demos lie

A live demo hides three missing production files. You rarely keep the exact request body. You rarely freeze the tool schema version.

You also skip storing the raw model output. Production then breaks, and you argue from memory. Memory is not a production-grade debugging artifact.

Cheap models make this worse, not better. You can retry until a screenshot looks fine. That screenshot still cannot rebuild the failed call.

What replayable actually means

A model call is replayable with four pieces. Drop one piece, and you start guessing.

1. Frozen request schema plus one example payload
2. Versioned tool or function definitions, even if empty
3. Raw response body, never a human summary
4. A pass or fail rule your CI can enforce

Guessing is not an architecture review at all. Guessing is how silent prompt drift ships.

Do not trust the model to remember identifiers. Force every id to round-trip from the fixture. If the model invents order_id, you failed.

Freeze one contract first

Do not start with ten clever prompts. Start with one path that can hurt users. Refunds, permissions, and deletions qualify immediately.

Write the contract as data, not slides. Keep it in git beside application code. Treat a contract change like any API change.

Fixture format

Use one boring JSON file per scenario. Label this fixture as a proposed contract, not a measured benchmark.

{
  "id": "checkout_refund_v3",
  "contract_version": "2026-09-01.1",
  "request": {
    "messages": [
      {
        "role": "system",
        "content": "Extract refund decisions as JSON only. Never invent ids."
      },
      {
        "role": "user",
        "content": "Order 4412, item arrived broken, paid 48 USD."
      }
    ],
    "response_format": { "type": "json_object" },
    "temperature": 0
  },
  "tools": [],
  "expect": {
    "http_status": 200,
    "required_keys": ["decision", "order_id", "amount_usd", "reason_code"],
    "decision_enum": ["refund", "deny", "review"],
    "max_bytes": 2048
  }
}

This file is the feature, not the UI. If the file cannot fail a test, delete the UI.

Temperature zero is a gate default only. It is not a quality claim about the model. You are testing contract obedience, not prose style.

Run the gate on spare capacity

You need compute that is safe to burn. You also need a server you can discard. That is the honest use of free inference.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access and a free server option. Point the replay harness at that spare capacity before you add UI. The same gate works against any OpenAI-compatible MODEL_BASE_URL you already control.

Pin environment variables. Do not hardcode hostnames into fixtures. Keep secrets out of the contract file.

export MODEL_BASE_URL="https://your-endpoint.example/v1"
export MODEL_NAME="${MODEL_NAME}"
export MODEL_API_KEY="${MODEL_API_KEY}"
python replay_gate.py fixtures/checkout_refund_v3.json

If the command fails, you do not ship. If the command passes, you still do not celebrate. You only earned the right to open a pull request.

A replay harness you can run

Label this script as an unexecuted local proposal. Run it yourself before you trust the output. It fails CI when the frozen contract breaks.

#!/usr/bin/env python3
"""replay_gate.py — fail CI when a frozen inference contract breaks."""

from __future__ import annotations

import json
import os
import sys
import urllib.error
import urllib.request
from typing import Any


def load_fixture(path: str) -> dict[str, Any]:
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)


def post_chat(payload: dict[str, Any]) -> tuple[int, dict[str, Any] | str]:
    base = os.environ["MODEL_BASE_URL"].rstrip("/")
    url = f"{base}/chat/completions"
    body = dict(payload)
    body["model"] = os.environ.get("MODEL_NAME", body.get("model", ""))
    raw = json.dumps(body).encode("utf-8")
    req = urllib.request.Request(
        url,
        data=raw,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ.get('MODEL_API_KEY', '')}",
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            return resp.status, json.loads(resp.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        return exc.code, exc.read().decode("utf-8", errors="replace")


def extract_content(response: dict[str, Any] | str) -> str:
    if isinstance(response, str):
        return response
    choices = response.get("choices") or []
    if not choices:
        return ""
    return str(choices[0].get("message", {}).get("content") or "")


def validate(
    fixture: dict[str, Any],
    status: int,
    response: dict[str, Any] | str,
) -> list[str]:
    errors: list[str] = []
    expect = fixture["expect"]
    if status != expect["http_status"]:
        errors.append(f"status {status} != {expect['http_status']}")
    content = extract_content(response)
    if len(content.encode("utf-8")) > expect["max_bytes"]:
        errors.append("response exceeded max_bytes")
    try:
        parsed = json.loads(content)
    except json.JSONDecodeError:
        errors.append("content is not JSON")
        return errors
    for key in expect["required_keys"]:
        if key not in parsed:
            errors.append(f"missing key: {key}")
    decision = parsed.get("decision")
    if decision not in expect["decision_enum"]:
        errors.append(f"decision {decision!r} not in enum")
    order = str(parsed.get("order_id", ""))
    if "4412" not in order:
        errors.append("order_id did not round-trip")
    return errors


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: python replay_gate.py <fixture.json>", file=sys.stderr)
        return 2
    fixture = load_fixture(sys.argv[1])
    status, response = post_chat(fixture["request"])
    errors = validate(fixture, status, response)
    report = {
        "id": fixture["id"],
        "contract_version": fixture["contract_version"],
        "ok": not errors,
        "errors": errors,
    }
    print(json.dumps(report, indent=2))
    return 0 if not errors else 1


if __name__ == "__main__":
    raise SystemExit(main())

Save the fixture next to the script. Run it in CI on every prompt change. Keep the raw JSON report as a build artifact.

Break the schema in a branch on purpose. Confirm the job goes red immediately. If it stays green, your gate is fake.

Promotion decision table

Paste this table into the pull request. Reviewers must point at a row. Chat opinions do not override the table.

Signal	Promote	Hold	Kill the feature
Fixture replays twice in a row	Yes, if schema stays frozen	Extra keys appeared	JSON parse failed
order_id round-trips	Yes	Format drifted	Model invented an id
decision stays in the enum	Yes	New value needs a spec	Free-text novel showed up
Byte size stays under cap	Yes	Size climbs each run	Chain-of-thought dumped
HTTP status stays stable	Yes	Intermittent 429	401 or schema 400

Do not vote in a hallway thread. The table is the review. If nobody can cite a row, reject the PR.

Wire it to git

Add a short job. Keep the workflow boring. Path filters stop docs-only burns.

# .github/workflows/replay-gate.yml
name: replay-gate
on:
  pull_request:
    paths:
      - "fixtures/**"
      - "prompts/**"
      - "replay_gate.py"
jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Replay frozen contracts
        env:
          MODEL_BASE_URL: ${{ secrets.MODEL_BASE_URL }}
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
          MODEL_NAME: ${{ secrets.MODEL_NAME }}
        run: |
          python replay_gate.py fixtures/checkout_refund_v3.json

Prompt edits must trip this gate. Tool-definition edits must trip it too. Copy changes in the user message count as contract changes.

Version the fixture when copy changes. Do not silently edit checkout_refund_v3 in place. Bump contract_version so failures stay explainable.

Record the failure, not the vibe

When the gate fails, store three artifacts. Keep them next to the fixture id.

• The exact request bytes you sent
• The exact response bytes you got
• The validator error list from CI

Do not store a Slack paraphrase of the bug. Future you cannot replay a paraphrase. Future you can replay a file.

Redact before you commit anything. Refund text can carry names and addresses. A fixture with personal data is a new incident.

What this will not catch

Replay is not an eval suite. It will not rank writing quality. It will not prove fairness across users.

It also will not prove production latency. A free endpoint is not your paid region. Treat timeouts as signals, never as benchmarks.

Fixtures go stale when product copy changes. You must refresh the user message on purpose. Silent fixture rot is still a contract bug.

Tool calling needs extra frozen fields. If you add tools later, version them. Never replay against an unbound tool list.

This gate does not replace authz checks in your app. The model can return refund while your ledger should deny. Enforce money movement in deterministic code.

Who should not use this

Skip this gate if you have no user impact. Toy weekend agents do not need CI theater.

Skip it if you cannot store raw prompts safely. Then you do not have a fixture program. You have a leak waiting for a commit.

Skip it for medical, legal, or credit decisions. A JSON enum is not a compliance program. Those paths need a real review process.

Do not use a discarded server as production. It is a burn-down lab for contracts. Promotion still happens on your controlled runtime.

Do this on one path

Pick one live AI path only. Write one fixture with a kill rule. Run the harness until it fails on purpose.

Then change one required key in the branch. Watch CI go red for that key. That red build is the feature you actually shipped.

Spare tokens are for that red build. They are not for a louder demo. If you already have free model access, point MODEL_BASE_URL at it and keep the fixture in git.