Why the Same Prompt Test Passes Locally and Fails in CI

#ai #debugging #llm #machinelearning

“It passes on my machine.” For a test that calls a model, this almost always means the two environments are sending different requests, or one of them is not sending a request at all. Six causes cover nearly all of it, and each has a check that distinguishes it from the others.

The symptom, precisely

Before diagnosing, pin down which symptom you have, because they have disjoint cause sets:

Fails every time in CI, passes every time locally. This is deterministic and therefore environmental. It is the easy one, and everything in the next section applies.
Fails sometimes in CI, never locally. Usually concurrency, rate limiting, or a local cache silently serving the same response every time so you never see the variance.
Fails sometimes in both, more in CI. This is ordinary flakiness plus an environmental amplifier; run the procedure in telling a flake from a regression first, because you may be chasing a regression.

The first move: diff the request

Do not start with the failure message. Start by printing the exact request in both environments and comparing them byte for byte. Almost every cause below shows up as a visible difference in this dump, and the ones that do not are eliminated by it.

# conftest.py — dump the outgoing request shape in both environments
import json, os, pytest

@pytest.fixture(autouse=True)
def dump_env(request):
    if os.environ.get("DUMP_TEST_ENV"):
        print(json.dumps({
            "test": request.node.nodeid,
            "model": os.environ.get("MODEL", "<unset>"),
            "base_url": os.environ.get("OPENAI_BASE_URL", "<default>"),
            "api_key_set": bool(os.environ.get("OPENAI_API_KEY")),
            "api_key_suffix": (os.environ.get("OPENAI_API_KEY") or "")[-4:],
            "temperature": os.environ.get("TEMPERATURE", "<unset>"),
            "seed": os.environ.get("SEED", "<unset>"),
            "tz": os.environ.get("TZ", "<unset>"),
            "vcr_mode": os.environ.get("VCR_RECORD_MODE", "<unset>"),
        }, indent=2))

Print the last four characters of the key, never the key. A test that passes locally on a personal key and fails in CI on a service account key with different model access is an extremely common version of this, and the suffix is enough to see it without leaking anything into a build log.

Six causes and how to tell them apart

The model is different. Your shell exports MODEL from a dotfile; CI does not, so the code falls back to a default that was chosen two years ago. Check: the model field in the dump differs, or the model id in the response differs. Fix: fail loudly on an unset model rather than defaulting — a missing environment variable should raise, not silently substitute.
The credential has different access. Different key, different organisation, different enabled models, sometimes a different region. Check: the key suffix differs, or the error is a 404 or 403 on the model rather than an assertion failure. Fix: use the same model-access tier for CI, and assert on the served model id at the start of the suite.
Nothing was ever sent locally. A cassette, a mock or an HTTP cache served your local run, and CI has no such file, so CI is the only environment that actually calls the provider. Check: your local run passes with the network disabled. Fix: see the next section — this is the single most common cause and it usually means a file is gitignored.
Sampling is unpinned. No seed, a nonzero temperature, and an assertion tight enough to depend on the draw. Check: run locally twenty times; if it fails locally too, the environment is innocent. Fix: pin what you can and loosen the assertion, because pinning is not sufficient — OpenAI documents seed as a best-effort mechanism and points at system_fingerprint as the signal that the backend changed.
The clock or locale differs. CI runs in UTC with a C locale; your laptop does not. A prompt that interpolates datetime.now(), a date format, or a currency symbol is a different prompt in the two places. Check: the rendered prompt differs on a date or a separator. Fix: inject the clock as a fixture and freeze it; never let a prompt read the ambient time.
Concurrency and rate limits. Covered below, because it is the one that produces intermittent rather than deterministic failure.

The recorded-response trap

If you use VCR.py, the behaviour is governed by record_mode, and its four documented values do very different things when a cassette is missing. once, the default, replays an existing cassette and records a new one only if the file does not exist — which means a missing cassette in CI silently becomes a live recording against the provider, and a passing local run turns into a live, billed, flaky CI run. none replays and errors on any request it has no recording for. all always records; new_episodes replays what it has and records the rest.

# tests/conftest.py — CI must never record, and must never call out
import os, pytest, vcr

RECORD_MODE = "all" if os.environ.get("UPDATE_CASSETTES") else "none"

my_vcr = vcr.VCR(
    cassette_library_dir="tests/cassettes",
    record_mode=RECORD_MODE,
    match_on=["method", "scheme", "host", "port", "path", "query", "body"],
    filter_headers=["authorization", "api-key", "x-api-key"],
)

@pytest.fixture
def cassette(request):
    with my_vcr.use_cassette(f"{request.node.name}.yaml"):
        yield

Two details in there matter more than the rest. filter_headers keeps your key out of a file you are about to commit, and it is not optional. And including body in match_on is what makes a prompt change fail loudly instead of quietly replaying the wrong recording — without it, any request to the same URL matches, so a test whose prompt you just edited will happily replay the old response and pass. Then check that tests/cassettes is not gitignored, which is the actual bug about half the time.

Rate limits and concurrency

CI runs your suite with more parallelism than you do, in one burst, from one IP, often on a shared organisation key that other pipelines are also using. Your laptop runs a handful of requests spread over a minute. The provider’s per-minute request and token limits are therefore reached in CI and never locally.

The distinguishing evidence is the status code and the headers, not the assertion. A 429 with a retry-after header, or a burst of failures clustered in time across unrelated tests, is a rate limit. Log the status code of every model call in the test harness; without that you will read the resulting assertion failures as content problems. The general mechanism is in how provider rate limits work.

The fix is to bound the concurrency of the suite rather than to retry into the limit: pytest-xdist’s -n, Vitest’s maxConcurrency and fileParallelism, or a semaphore in the client. Retrying a rate-limited burst without backoff makes the burst longer.

A separate key per environment is what makes this diagnosable at all: when CI has its own credential, a 429 in CI cannot be caused by production traffic and the two spend lines are separable. Multigrid issues per-environment keys against one API with their own limits and spend caps, so an eval run that loops cannot quietly consume the budget the live application is using.

A checklist that ends in a fix

Run the test locally with the network disabled. If it still passes, you have cause three — a recording or mock is serving it, and CI is the only environment making a real call.
Dump the request shape in both environments and diff it. Model, base URL, key suffix, temperature, seed, timezone.
Compare the served model id from the response, not the requested one. An alias can resolve differently for two keys.
Look at the HTTP status of the failing call. A 4xx or 5xx is an environment problem; a 200 with a failing assertion is a content problem and belongs in the provider-versus-your-code split.
Run the test twenty times locally. If it fails locally too, CI was never the variable and you have an ordinary flaky test.
Once fixed, make the environment difference impossible rather than merely corrected: raise on an unset model variable, set record_mode to none in CI, freeze the clock, and commit the cassettes.

Record modes, plugin flags and rate-limit header names are all vendor and library surfaces that change. Confirm the current spelling in the VCR.py and provider documentation before adopting the snippets above.