Did the model get worse, or is it just you? How to actually tell when an LLM API silently changes

#llm #devops #ai #monitoring

You ship a feature on top of gpt-x or claude-y. It works. Three weeks later your users say the outputs feel worse — vaguer, sloppier, failing on the same prompts that used to pass. You didn't change your code. Did the model change, or are you imagining it?

This question eats an astonishing number of engineering hours, and it almost always dissolves into a vibes argument: half the thread says "it's clearly worse", the other half says "you have no evidence, it's confirmation bias." Both sides are right, because nobody has instrumented the one thing that would settle it. Here is how to settle it.

Why your gut is useless here (and so is a single comparison)

Two things move at once when a hosted model "feels worse":

The provider's side — a new checkpoint, a routing change, an inference optimization (batching, speculative decoding, quantization), or an infrastructure bug. You are not told when these happen. Often the provider doesn't announce it, and sometimes it isn't even a model change — it's an infra regression that only looks like the model got dumber.
Your side — a prompt tweak, a temperature change, a new library version, a different context length, or just which prompts you happened to hit today.

From a single account, on a handful of anecdotal prompts, these two are indistinguishable. That's the trap. You need two things to escape it: an objective, dated signal, and a way to tell "it changed for me" from "it changed for everyone."

Step 1 — Fix a canary suite

Pick a small, frozen set of prompts — 20 to 100 — that exercise the capabilities you actually depend on. Keep it small and cheap; this is a smoke detector, not a benchmark. Freeze it: same prompts, same parameters (temperature, top_p, max_tokens), same model string, forever. The moment you change the suite you lose your baseline.

For each canary, define a cheap scalar you can compute automatically:

JSON/schema validity rate (did it return parseable, valid output?)
Output length (a sudden shift often precedes a quality change)
A pass/fail against a fixed assertion (does the function it wrote compile?)
A rubric score from a cheap grader

You are not trying to measure "quality" in the abstract. You are trying to detect change. A boring, stable scalar is exactly what you want.

Step 2 — Record a baseline while things are good

Run the suite on a schedule — hourly, daily, whatever your budget allows — and store (timestamp, prompt_id, metric). Do this before you suspect anything. The baseline is the whole game: without "what normal looked like last month" you have nothing to compare today against. Estimate the mean μ₀ and standard deviation σ₀ of your metric per prompt from a stable window.

Step 3 — Run a change-point test, not a threshold

A fixed threshold ("alert if success rate < 90%") is noisy and late. What you want is a sequential change-point detector that accumulates small, consistent shifts and fires with a timestamp. The classic is CUSUM (Page, 1954). It's a few lines:

def cusum_alarm(values, mu0, sigma0, k=0.5, h=5.0):
    """Two-sided CUSUM on standardized observations.
    k = slack (in std devs), h = alarm threshold.
    Returns the index where a sustained shift is detected, or None."""
    s_pos = s_neg = 0.0
    for i, x in enumerate(values):
        z = (x - mu0) / sigma0          # standardize against the baseline
        s_pos = max(0.0, s_pos + z - k) # catches upward shifts
        s_neg = max(0.0, s_neg - z - k) # catches downward shifts
        if s_pos > h or s_neg > h:
            return i                     # dated change-point
    return None

Feed it your per-prompt metric over time. Instead of "feels worse", you now get "json_success_rate on prompt #14 stepped down starting Tuesday" — a claim with a date on it that you can put in a ticket. (h=5.0, k=0.5 are reasonable starting points for standardized data; tune on your own history. Bayesian Online Change-Point Detection is a fancier alternative if you want posterior probabilities.)

This alone puts you ahead of ~95% of "the model got worse" threads, which never get past anecdotes.

Step 4 — The part one account can never answer: "is it just me?"

Here's the ceiling you hit: even with a perfect change-point signal, you still can't tell a provider-side change from your own drift, because from one seat they look identical. Your CUSUM fired — but was it the model, or the LlamaIndex bump you did last week, or a flaky region?

The only way to separate them is correlation across independent observers. If your canary fires on Tuesday, and several other organizations watching the same model on the same metric also fire around Tuesday, that agreement is the signal that it's provider-side — not your harness. One observer has N=1 forever; that's why these threads never converge. This is a genuinely different kind of detection: not "monitor my app" (every observability tool does that), but "did this change for everyone, privately confirmed across the fleet."

That cross-observer angle is what SEISMOGRAPH is built around: a lightweight probe runs your canary suite, a change-point detector flags shifts locally, and only privacy-preserving distributional features — never your prompts or outputs — are shared, so a public "model weather" alert only fires when multiple independent observers agree. A single org's signal stays private.

Does this actually work? A worked example

The method isn't hypothetical. Take the September 2025 Anthropic incident: for a stretch, claude-sonnet-4 degraded because of infrastructure bugs — notably a context-window routing error — not a model update (Anthropic said so explicitly in their postmortem). Exactly the "is it the model or the infra?" fog this whole article is about.

Replaying a canary suite over that period with the CUSUM method above: a seeded backtest flags it 38 days before the postmortem. Not a live catch — a reproducible backtest, seeded and open — but it shows the signal was sitting in cheap distributional metrics well before the official acknowledgement.

Do it yourself

Probe (open source, Apache-2.0): pip install seismograph-probe
Live "model weather" dashboard: https://driftdefense.dev?utm_source=devto&utm_medium=article&utm_campaign=evergreen_howto
Code + method: https://github.com/Tania-coder/SEISMOGRAPH

You don't need any of that to start — the four steps above work with a cron job, a CSV, and the ten lines of CUSUM. But if you want the "is it just me?" half answered, that needs more than one observer, and that's the part worth federating.

FAQ

Did GPT / Claude / Gemini actually get worse recently?
Sometimes yes, sometimes it's your setup, and often it's a provider-side infra change that isn't a "worse model" at all. The point of this article is that you shouldn't answer from vibes — instrument it and get a dated signal.

How can I prove a hosted model changed?
You can't prove it from one account, but you can get strong objective evidence: a frozen canary suite + a baseline + a change-point test gives you a dated shift. Provider-side vs. local drift is only separable with agreement across multiple independent observers.

Isn't this just LLM observability / evals?
Related but different. Evals score quality at a point in time; observability watches your traffic. This is about detecting change over time in a hosted dependency you don't control — and, uniquely, correlating that change across organizations to rule out "it's just me."

What metrics should I track?
Start with the cheapest ones that move when quality moves: schema/JSON validity, output length, and a single fixed pass/fail per canary. Add a rubric score later if you want resolution.