Most CI pipelines assume a function called with the same input twice returns the same output. That assumption breaks the moment an LLM call enters your test suite. Ask GPT-4 or Claude the same question twice and you can get two different (both correct) answers. Teams shipping LLM-backed features often respond to this by writing almost no tests for the LLM-touching code paths, or by asserting on exact string output and then disabling the test the first time it flakes. Neither is sustainable once the feature is in production and a prompt change or model upgrade can silently break behavior.
The fix isn't a clever assertion library. It's splitting what you're actually testing into three tiers with different determinism guarantees, and mapping each tier to a different CI job.
Tier 1: Contract tests (deterministic, run on every PR)
A contract test never calls a live model. It asserts on the shape of what your pipeline produces, not the content. If your orchestrator expects the LLM to return {title, tags, price_usd, full_content}, a contract test feeds a canned response through your parsing/validation layer and checks it doesn't throw, that required fields are present, and that types match.
# test_contract.py
import json
import jsonschema
import pytest
PRODUCT_SCHEMA = {
"type": "object",
"required": ["title", "tags", "price_usd", "full_content"],
"properties": {
"title": {"type": "string", "minLength": 5},
"tags": {"type": "array", "minItems": 1},
"price_usd": {"type": "number", "minimum": 1},
"full_content": {"type": "string", "minLength": 200},
},
}
@pytest.mark.parametrize("fixture_file", [
"fixtures/well_formed.json",
"fixtures/missing_price.json",
"fixtures/empty_content.json",
])
def test_pipeline_handles_llm_output(fixture_file):
payload = json.load(open(fixture_file))
result = parse_and_validate(payload) # your own code
if fixture_file == "fixtures/well_formed.json":
jsonschema.validate(result, PRODUCT_SCHEMA)
else:
assert result.rejected_reason is not None
These fixtures are static JSON files checked into the repo. They run in milliseconds, need no API key, and catch the bug class that actually causes outages: your parser choking on a field the model omitted, a null where you expected a string, a schema drift after a prompt edit. This tier runs on every commit and should block merges.
Tier 2: Replay tests (deterministic, run on every PR)
Between "fake JSON fixture" and "call the real API" sits cassette-based replay: record a real model response once, save it to disk, and replay it on every subsequent CI run instead of hitting the network. This is the same idea as VCR for HTTP tests, applied to LLM calls.
# conftest.py
import json, os, hashlib
CASSETTE_DIR = "tests/cassettes"
def llm_call_with_cassette(prompt, real_call_fn):
key = hashlib.sha256(prompt.encode()).hexdigest()[:16]
path = f"{CASSETTE_DIR}/{key}.json"
if os.path.exists(path):
return json.load(open(path))
if os.environ.get("CI") and not os.environ.get("RECORD_CASSETTES"):
raise RuntimeError(f"Missing cassette for prompt hash {key}; run with RECORD_CASSETTES=1 locally")
response = real_call_fn(prompt)
json.dump(response, open(path, "w"))
return response
Cassettes are committed to the repo like any other fixture. When you deliberately change a prompt, you re-record locally with RECORD_CASSETTES=1, review the diff in the cassette file as part of the PR (this is often where a reviewer catches a regression before it ships), and commit the new cassette. CI never touches the network for these tests, so they're fast and free, but they still exercise your real parsing and business logic against real model output, not a hand-written fixture.
Tier 3: Live smoke tests (non-deterministic, run nightly, not on PRs)
This tier calls the actual model with real API keys and real cost. Its job is to answer one question: has the model's behavior drifted since we last checked, in a way our replay cassettes wouldn't reveal? Because output varies, you can't assert equality. Instead assert on properties: is the response valid JSON, is the schema satisfied, is a semantic similarity score against a reference answer above a threshold (using embeddings, not string match), does token usage stay within a cost budget.
def test_live_semantic_similarity(embed_fn, cosine_sim):
reference = load_reference_embedding("golden_answer.json")
live_output = call_real_llm(PROMPT)
similarity = cosine_sim(embed_fn(live_output["full_content"]), reference)
assert similarity > 0.80 # loose bound; catches total drift, not phrasing
This is where CircleCI's workflow model earns its keep. Define two workflows in .circleci/config.yml: one gated on every PR that runs tiers 1 and 2 only, and a scheduled workflow that runs tier 3 nightly (or on demand via an approval job), so a flaky live call never blocks a merge.
version: 2.1
jobs:
fast-tests:
docker: [{ image: cimg/python:3.12 }]
steps:
- checkout
- run: pip install -r requirements.txt
- run: pytest tests/test_contract.py tests/test_replay.py -v
live-smoke-tests:
docker: [{ image: cimg/python:3.12 }]
steps:
- checkout
- run: pip install -r requirements.txt
- run:
name: Enforce per-run cost ceiling
command: python scripts/check_budget.py --max-usd 2.00
- run: pytest tests/test_live_smoke.py -v --reruns 1
workflows:
on-pr:
jobs:
- fast-tests
nightly-live-check:
triggers:
- schedule:
cron: "0 6 * * *"
filters:
branches: { only: main }
jobs:
- live-smoke-tests
Two details matter here. First, --reruns 1 on the live job acknowledges that a single API timeout shouldn't page anyone; a persistent two-run failure is a real signal. Second, the check_budget.py step runs before the pytest call and estimates cost from a token count on the fixed test prompts, failing the job outright if a prompt change would blow past a dollar ceiling — this catches the scenario where someone widens a prompt's context window and 10x's the nightly bill before it happens, not after the invoice arrives.
Why the split, not just "more retries"
Teams that skip this split usually end up with one of two failure modes: PRs blocked for minutes waiting on flaky network calls to a rate-limited API, or tests so loosely asserted they'd pass even if the pipeline returned garbage. Separating contract correctness (tier 1), regression detection against known-good real output (tier 2), and live-model drift detection (tier 3) lets each tier use the assertion style that actually fits its determinism guarantee, and lets CircleCI schedule each at the cadence its cost and speed profile deserves. The PR-blocking path stays fast and free; the expensive, flaky, real-money path runs once a day where a human can triage it without holding up a release.
Top comments (5)
Your cassette key is sha256(prompt), so a prompt edit forces a re-record and lands as a reviewable diff. That part is right. What is missing from the key is the model identifier, so if the provider moves the weights behind an alias, every cassette still matches and Tier 2 stays green while the real behaviour has moved. Hashing prompt + model + temperature + top_p costs one line and turns a silent provider update into a loud missing-cassette failure.
That also takes pressure off Tier 3, which is currently the only thing standing between you and a model change, and it is checking similarity > 0.80 against a single golden. I would rather derive that bound than pick it: run the same prompt 30 times against the current model, take the empirical 1st percentile of pairwise similarity, and set the threshold there. Ours came out around 0.91 on a summarisation path, which meant 0.80 would have slept through a real regression.
Great topic. Testing non-deterministic LLM outputs is painful—we've hit it building autonomous pipelines. A contract-based approach makes sense: define acceptable output contracts (schema, token bounds, tone markers) rather than chasing exact output matches. Pair that with:
The contract layer also decouples test stability from model changes—you're testing behavior, not token sequences.
The tier that quietly pays for itself is Tier 1 — contract tests on shape, not content. The outages we've actually caused were never "the model said something wrong," they were the parser choking on a field the model dropped after a prompt edit, or a schema drift that turned a string into a null. A JSON-schema check over canned fixtures catches that class in milliseconds with no API key, and it blocks the merge before the flaky stuff even runs.
One addition worth making to Tier 2 replay: pin the cassettes to a model version, not just the prompt. We got bitten when a provider silently rolled a model and our "deterministic" replays kept passing against a recording of the old behavior — green CI, changed reality. Tagging each cassette with the model id it was recorded against, and failing loudly when the live model drifts off it, closed that gap. How are you handling cassette invalidation when a prompt changes — regenerate the whole set, or diff only the affected paths?
I like the three-tier separation. One additional layer I’d consider is testing the orchestration itself, not just the model output.
In many production LLM systems, failures come from retrieval, tool selection, permission checks, or function-calling rather than the model’s text generation. Even with stable model behavior, a change in tool routing or retrieval results can produce a completely different outcome.
To me, the contract should cover the entire agent workflow, not only the LLM response. That makes the tests much closer to the real production failure modes.
Contract-based testing is the right instinct here, since asserting on invariants like schema, required fields, and value ranges survives the non-determinism that exact-match assertions choke on. The piece I always add is a tolerance budget: run the same case a handful of times and fail CI only if the contract breaks past a threshold, so one unlucky sample doesn't red the whole build. Are you pinning a seed for repeatability, or accepting a pass rate?