CI Treats Free-Model Output Like an Integer. It's Actually a Flaky Dependency.

Same commit. Same prompt file. Same fixture. Different verdict.

If you have wired a free model into GitLab CI, you already know that feeling.
The job was green at 10:00. It is red at 02:47. Nobody changed the code.

That is not a mystery. It is a category error.

A CI step is treated as a deterministic transform: inputs in, status out. A model call is a remote service with retries, latency, routing, sampling, and silent behavior change. When I started treating the model call as a flaky dependency instead of a deterministic step, the failures got easier to reason about.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow below uses MonkeyCode's free model access and free server option as the concrete environment, but the pattern also works with any model endpoint you can call from CI.

Stop gating raw text

The problem with pass/fail checks is that they collapse several different failures into one red job.

A free model step can fail in at least four distinct ways:

• The transport failed: timeout, DNS, rate limit, or endpoint unavailable.
• The shape changed: the response used to be a JSON object and now it is a markdown blob.
• The content drifted: the same prompt now returns a different recommendation.
• The latency moved: the call still succeeds, but it no longer fits the pipeline budget.

Each failure needs a different response. Retrying a schema break is waste. Caching an endpoint outage is useful. Blocking a merge because the model is temporarily slow is usually wrong.

So I stopped storing one giant model_check: green signal.

I now store a small, reviewable fixture per prompt.

The artifact: record normalized fixtures first

A fixture is not the full model response. It is the fields CI actually needs.

That keeps the file small, makes diffs readable, and prevents the fixture from leaking prompt history into a place where nobody will read it.

# record_fixture.py
import hashlib
import json
import pathlib
import sys

prompt_path = pathlib.Path(sys.argv[1])
response_path = pathlib.Path(sys.argv[2])

prompt = prompt_path.read_text()
response = response_path.read_text()

record = {
    "input_hash": hashlib.sha256(prompt.encode()).hexdigest(),
    "prompt": prompt,
    "output_text": response.strip(),
}

print(json.dumps(record, indent=2))

Run it once after a scheduled model call:

python record_fixture.py \
  prompt.txt \
  model-output.txt \
  > fixtures/latest.json

The input_hash is not just for caching. It gives the replay layer a stable lookup key later.

Split the gate into three layers

Instead of one brittle check, I use three separate gates.

1. Schema gate

This catches shape drift before the rest of the pipeline tries to parse the model output.

{
  "type": "object",
  "required": ["input_hash", "prompt", "output_text"],
  "additionalProperties": true,
  "properties": {
    "output_text": {
      "type": "string",
      "minLength": 1
    }
  }
}

# schema_gate.py
import json
import pathlib
import sys

from jsonschema import validate

schema = json.loads(pathlib.Path(sys.argv[1]).read_text())

for path in pathlib.Path(sys.argv[2]).glob("*.json"):
    validate(json.loads(path.read_text()), schema)

print("schema gate passed")

This is fast. It should run on every merge request.

2. Semantic drift gate

A response can pass the schema and still be wrong for the repo.

I use a token-level similarity check, not a strict string equality check. That avoids failing the pipeline because the model added one harmless word.

# drift_gate.py
import json
import pathlib
import sys
from difflib import SequenceMatcher

old = json.loads(pathlib.Path(sys.argv[1]).read_text())
new = json.loads(pathlib.Path(sys.argv[2]).read_text())

old_text = old["output_text"].lower().strip()
new_text = new["output_text"].lower().strip()

ratio = SequenceMatcher(None, old_text, new_text).ratio()
print(f"drift_ratio={ratio:.3f}")

if ratio < 0.75:
    raise SystemExit("semantic drift above project threshold")

The threshold is a starting point, not a law. You tune it with your own fixture history.

3. Replay gate

This is where the free server option becomes useful.

Instead of calling a live model from every pipeline, the merge-request pipeline can call a small replay service. The service returns the last known good response for an input_hash.

# replay_server.py
import json
import pathlib
from http.server import BaseHTTPRequestHandler, HTTPServer

fixtures = {}

for path in pathlib.Path("fixtures").glob("*.json"):
    item = json.loads(path.read_text())
    fixtures[item["input_hash"]] = item["output_text"]

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers["Content-Length"])
        body = json.loads(self.rfile.read(length))
        text = fixtures.get(body.get("input_hash"), "")

        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps({"output_text": text}).encode())

HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()

Run this on the free server. The CI job no longer depends on the live model being awake, fast, or stable.

Live model calls move to a scheduled refresh job. That job updates fixtures/latest.json and opens a drift issue if the semantic score drops too far.

GitLab CI jobs

Here is the split I use.

model-contract:
  image: python:3-slim
  script:
    - pip install jsonschema
    - python schema_gate.py schemas/model-output.schema.json fixtures
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

model-drift:
  image: python:3-slim
  script:
    - python drift_gate.py fixtures/baseline.json fixtures/latest.json
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      allow_failure: true

The drift job is allowed to fail as a warning. The schema job is not.

A shape break blocks the merge. A content change opens a review conversation. That distinction matters.

Decision table

This is the policy I keep next to the fixtures.

Observation	First action	Merge policy
Schema gate fails	Stop, no retry	Block merge
Drift ratio drops 5-20%	Save new fixture, open issue	Allow merge with warning
Drift ratio drops more than 20%	Freeze live refresh	Block until human review
Live endpoint unavailable	Replay last known good fixture	Allow contract-only changes
Call exceeds latency budget	Use cached fixture	Allow merge, schedule refresh

The percentages are project-specific. The order is what matters: stable contract first, semantic review second, live endpoint last.

What this does not solve

Replay is not reality.

A fixture can hide a model improvement. It can also hide a model regression that only appears on new inputs. The contract gate only catches changes you expected. The drift gate only catches changes you can measure.

This approach adds files, review work, and a scheduled job. It is not worth doing on a repo with one prompt and one caller.

Who should skip it?

• Teams with a strict live-audit requirement and no room for replay.
• Products where output uniqueness is the point, such as creative text generation.
• Small repos that do not yet have a stable fixture corpus.
• Prompts that contain secrets or user data that should never be stored in a repo.

Bottom line

A free model call is not an integer step. It is a service with shape, content, latency, and availability risk.

Treat it that way. Record a normalized fixture. Split schema from semantics. Replay the last known good response during merge-request pipelines, and let a scheduled job refresh live behavior.

If you are already using MonkeyCode's free model access, the free server option is a natural place to run that replay service. Keep live calls scheduled, keep fixtures reviewable, and stop letting one flaky response decide whether a merge is safe.