Your CI Gate Should Not Call a Hosted Model: Record, Replay, Then Merge

A model call inside a required check is an uncontrolled dependency. The lane that blocks merge should read fixtures from disk. Only the lane that cannot block merge should touch the network.

That single split removes most of the flakiness people blame on "AI in CI". It also makes review cheaper, because every model answer your gate saw is a file you can diff.

Below is the adapter, the contract test, the two-job workflow, and the triage order I use when the gate goes red.

Why the live call breaks the gate

Three failure shapes show up almost immediately once a model call sits behind a required check.

Latency variance. A 900 ms call and a 14 s call in the same job produce different timeouts on different days. Your retry logic then hides a real prompt regression behind a slow afternoon.

Sampling drift. Even at low temperature, output shape changes between runs. A gate that asserts on prose will pass on Monday and fail on Tuesday with no code change in between.

Outage equals red. When the provider is down, your PR shows as failing. The author sees a red check and starts editing code that was never broken.

There is a fourth problem that gets less attention: secrets. If the blocking job holds a model key, that key is now in the same process as pull-request code you do not control. Moving live calls out of the blocking lane is a security decision, not only a stability one.

The anti-pattern looks like this:

# do not do this in a required check
try:
    return call_model(prompt)
except TimeoutError:
    return load_cassette(prompt)   # silent fallback hides drift

A silent fallback is worse than no fallback. You cannot tell a replayed answer from a live one in the logs, so nobody knows which runs actually tested anything.

Two lanes, one adapter

Before writing code, agree on which lane does what.

Lane	Trigger	Network	Blocks merge	Purpose
gate	every PR	none	yes	prove your code handles the recorded contract
live-drift	nightly schedule	yes	no	detect provider or model drift, open an issue
record	manual, local	yes	n/a	write or refresh one cassette, reviewed in the PR

If a lane cannot block merge, it cannot hold a secret that PR code can reach. Put the model key on live-drift only, and keep the gate job at contents: read.

Build the adapter in five steps

1. Make the call hashable

Hash every input that can change the answer, including the template identity. If you hash only the prompt text, an edit to the system instruction can silently reuse a stale cassette.

# ci/model_gate.py
from __future__ import annotations

import hashlib
import json
import os
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Callable

ADAPTER_VERSION = "3"  # bump when hashing or request shape changes
CASSETTE_DIR = Path(os.environ.get("CASSETTE_DIR", "ci/cassettes"))
MODE = os.environ.get("MODEL_MODE", "replay")  # replay | record | live


class MissingCassette(RuntimeError):
    pass


@dataclass(frozen=True)
class Call:
    template_id: str
    template_version: int
    prompt: str
    temperature: float = 0.0
    max_tokens: int = 512


def cassette_key(call: Call) -> str:
    blob = json.dumps(
        {"adapter": ADAPTER_VERSION, **asdict(call)},
        sort_keys=True,
        separators=(",", ":"),
    )
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:20]

2. Fail closed in replay mode

Missing fixture means missing fixture. Raise, do not call out.

Transport = Callable[[Call], dict]


def run(call: Call, transport: Transport | None = None) -> dict:
    path = CASSETTE_DIR / f"{cassette_key(call)}.json"

    if MODE == "replay":
        if not path.exists():
            raise MissingCassette(
                f"no cassette for {call.template_id} v{call.template_version} "
                f"({path.name}); record it with MODEL_MODE=record"
            )
        return json.loads(path.read_text())["response"]

    if MODE == "record":
        if path.exists():
            return json.loads(path.read_text())["response"]
        if transport is None:
            raise ValueError("record mode needs a transport")
        response = transport(call)
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(
            json.dumps(
                {"request": asdict(call), "response": response},
                indent=2,
                sort_keys=True,
            )
            + "\n"
        )
        return response

    if transport is None:
        raise ValueError("live mode needs a transport")
    return transport(call)

Store the request inside the cassette. Reviewers need to see what was asked, not only what came back.

3. Assert contracts, not prose

Your gate exists to prove the shape your code consumes. Validate structure and bounds, then stop.

# ci/contracts.py
PATCH_PLAN = {
    "type": "object",
    "required": ["files", "reason"],
    "properties": {
        "files": {"type": "array", "items": {"type": "string"}, "maxItems": 20},
        "reason": {"type": "string", "minLength": 10},
    },
    "additionalProperties": False,
}

# tests/test_patch_plan_contract.py
import pytest
from jsonschema import ValidationError, validate

from ci.contracts import PATCH_PLAN
from ci.model_gate import Call, run


def test_plan_is_mergeable(cassette_prompt):
    call = Call(template_id="patch_plan", template_version=4, prompt=cassette_prompt)
    result = run(call)
    try:
        validate(result, PATCH_PLAN)
    except ValidationError as exc:
        pytest.fail(f"recorded output breaks the contract: {exc.message}")

Do not assert on wording. The moment you test prose, you have a test that fails for reasons you cannot reproduce.

4. Split the workflow

The gate job gets no key. The nightly job gets the key and is allowed to fail.

# .github/workflows/model-gate.yml
name: model-gate
on:
  pull_request:
  schedule:
    - cron: "17 3 * * *"

jobs:
  gate:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    permissions:
      contents: read
    env:
      MODEL_MODE: replay
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r ci/requirements.txt
      - run: pytest tests/ -q

  live-drift:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    continue-on-error: true
    env:
      MODEL_MODE: live
      MODEL_BASE_URL: ${{ secrets.MODEL_BASE_URL }}
      MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r ci/requirements.txt
      - run: pytest tests/ -q -m live

This is also where a free endpoint earns its place. The nightly lane makes a small, bounded number of real calls, so it is the right consumer of a free model allowance. MonkeyCode's free model access and free server option fit that lane: you can run the drift job without opening a paid budget line first, and the free server option gives the schedule somewhere to live.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability terms for free offerings change. Check the current terms on the project page before you let a nightly job depend on them, and keep a continue-on-error job non-blocking until you have watched it for a week.

5. Keep the fixtures small

One cassette per template version. When you bump template_version, the old file stops matching and you should delete it in the same commit. If ci/cassettes/ grows without bound, nobody reviews it, and an unreviewed fixture is a test with a comment that lies.

When the gate goes red, triage in this order

1. Read the exception type. MissingCassette is a fixture problem. ValidationError is a contract problem. A KeyError on response means a hand-edited or truncated cassette.
2. MissingCassette: did you change the prompt? Record once with MODEL_MODE=record, then read the cassette diff. If the new answer surprises you, that is the review you wanted.
3. ValidationError: fix the fixture, not the assertion. The recorded output never satisfied the contract. Either the prompt is underspecified or the contract is wrong. Decide which, in the PR.
4. Contract passes but production breaks. Your schema is looser than your consumer. Tighten maxItems, add additionalProperties: false, and add a bound you actually enforce downstream.

Track one number: cassette miss rate per week. Above roughly one in ten PR runs means your templates are churning, and the gate is measuring your edit speed rather than your code.

Limits worth stating out loud

Replay proves your code handles a known answer set. It cannot prove the model still behaves. That is the nightly lane's job, and it needs a human to read the failure.

Cassettes go stale when a provider swaps a model behind an endpoint name. Pin the model identifier in the request and re-record deliberately, not on a schedule that erases the diff.

A single recorded answer is not an evaluation. If you need quality numbers, you need multiple samples and a scoring rule, which is a different pipeline with different costs.

And the drift job is a smoke alarm, not a gate. Once it blocks merges, you have rebuilt the problem this article removes.

Who should not do this

Skip the replay lane if your prompts are assembled at runtime from unbounded user input, because the cassette space explodes and you will spend more time pruning than testing.

Skip it if you need a per-PR quality score on model output. Replay gives you a contract check, and a contract check is not a quality signal.

And skip it if nobody on the team owns the prompts. Record/replay makes prompt changes visible, which is only useful if someone is accountable for them.

Start with one template. Record it, wire the gate, and watch the miss rate for a week before you add the second. If you want to stand up the nightly lane before you commit a budget, the free model access and free server option in the MonkeyCode project are a reasonable place to try the loop, terms permitting.