A contract test that only ever runs against a recording tells you what you believed on the day you recorded it. The provider is a moving service on somebody else’s roadmap, and the only way to learn it moved is to ask it.
Mocks freeze; providers do not
Every mock, fixture and cassette in your repository is a snapshot of a past response. That is exactly what makes them good for unit tests: they are fast and they never change, so a failure is always your code. It is also what makes them useless for the question this page is about. A frozen fixture cannot go red when the thing it copies changes, because it does not know the thing exists.
The result is a familiar and unpleasant shape. Your suite is green. Your deploys are green. A provider ships a change, and the first system to notice is production — usually at whatever hour the change rolled out to your region, and usually as an exception in a parser rather than as anything that names the provider. The debugging starts from the wrong end, because everything you own passed.
The fix is small and structural: take the assertions you already wrote for the candidate-provider suite and run them against the real endpoint on a timer. You are not testing your code. You are monitoring an external dependency, and framing it that way answers most of the design questions that follow.
The change classes, ordered by damage
Not every change is a breakage, and a suite that treats them alike becomes noise. These are the classes, worst first:
- A type change on a field you read. An integer becomes a string, a string becomes an object, a scalar becomes an array. Worst because your code often keeps running — concatenating instead of adding, truncating instead of iterating — and produces wrong output rather than an error.
- A removed or renamed field. Loud in a typed language, silent in an untyped one where a missing key reads as undefined and flows onward. A schema check catches both.
- A new enum value. A new
finish_reasonor a new error code. Your switch statement falls through to a default that was written for “impossible”. This is the class most likely to be intended as non-breaking by the provider and most likely to break you anyway. - A changed default. A parameter you never sent because the default was what you wanted. Nothing in the response shape changes; behaviour does. Only an invariant test catches it — assert the property you relied on, not the parameter.
- A model substitution behind a stable name. Same contract, different weights. Not a contract change at all, which is why it belongs to silent model updates and to your evaluation suite rather than here — but a scheduled run that records the
modelfield will often see it first, because the resolved version string moves. - An added field. Not a breakage. Report it, do not fail on it. This is the most common change by a wide margin and failing on it is how a monitor loses its audience.
Running it as a monitor, not a gate
The tempting move is to add the live suite to the pull-request pipeline. Resist it. A test that calls a third-party API over the public internet fails for reasons that have nothing to do with the change under review: a rate limit, a regional outage, a slow response, an expired key. Attach that to the merge button and within a fortnight the team has learned to re-run the pipeline without reading it, which costs you the signal you built the thing for.
So: a separate scheduled job, on its own cadence, with its own notification path. Hourly is generous for most teams and daily is defensible; the useful question is how long you are willing to be broken in production before something tells you, and the schedule should be shorter than that. Run it against production credentials and the production model names, because a contract verified for a model you do not use is not verified.
Two design details matter more than the cadence. Retry once on anything that looks like transport — a timeout, a 5xx, a 429 — and only report a failure if the retry also fails, which removes most of the noise for the cost of a few lines. And record each run’s outcome somewhere durable, because the question you will actually be asked during an incident is “when did this start failing”, and a job that only alerts cannot answer it. Emitting the result as a metric alongside the rest of your LLM telemetry gives you that history for free.
What a failure should do
Route it like a dependency alert, not like a broken build. The person who should see it is whoever owns the integration, and the message should contain the assertion that failed and the actual response body, because the first question is always “what did it return instead”. A message that says only that a job failed makes somebody reproduce it by hand, which is several minutes of an incident spent recovering information the job already had.
Separate the schema-failure alert from the additive-change report. They are different urgencies and they should not share a channel. A removed field wakes somebody up; a new field is a weekly digest, and reading it is how you find out about a capability before your competitors do. If you keep the recorded key set as a committed baseline, the diff between runs is a small artefact worth putting into that digest verbatim.
One thing the scheduled run should not do is auto-open a ticket per failure. A provider incident produces a red run every hour until it is fixed, and twenty-four identical tickets is how a monitor gets muted. Deduplicate on the assertion name.
Keeping the cost trivial
Contract assertions do not need long completions. Every request in the suite can use a short prompt, temperature zero and a small max_tokens, because nothing you assert on depends on the length or quality of the reply. That keeps the recurring spend small enough that nobody has to approve it, which is the practical condition for the job surviving its first budget review. The arithmetic for a multi-provider version, with the assumptions written out, is in running the same suite against every provider.
The one test that cannot be made cheap is the truncation assertion, which by design asks for more output than it allows. It is still tiny — the cap is what bounds the cost, and the cap is eight tokens.
Keep the whole thing in the same repository as the code it protects. A monitor that lives in a separate infrastructure project drifts out of sync with the parser it is supposed to be guarding, and the drift is invisible until the day it matters.
Top comments (0)