DEV Community: Tatiana Radchenko 🇩🇰

Did the model get worse, or is it just you? How to tell when an LLM API silently changes

Tatiana Radchenko 🇩🇰 — Fri, 24 Jul 2026 12:30:34 +0000

You ship a feature on top of gpt-x or claude-y. It works. Three weeks later your users say the outputs feel worse — vaguer, sloppier, failing on the same prompts that used to pass. You didn't change your code. Did the model change, or are you imagining it?

This question eats an astonishing number of engineering hours, and it almost always dissolves into a vibes argument: half the thread says "it's clearly worse", the other half says "you have no evidence, it's confirmation bias." Both sides are right, because nobody has instrumented the one thing that would settle it. Here is how to settle it.

Why your gut is useless here (and so is a single comparison)

Two things move at once when a hosted model "feels worse":

The provider's side — a new checkpoint, a routing change, an inference optimization (batching, speculative decoding, quantization), or an infrastructure bug. You are not told when these happen. Often the provider doesn't announce it, and sometimes it isn't even a model change — it's an infra regression that only looks like the model got dumber.
Your side — a prompt tweak, a temperature change, a new library version, a different context length, or just which prompts you happened to hit today.

From a single account, on a handful of anecdotal prompts, these two are indistinguishable. That's the trap. You need two things to escape it: an objective, dated signal, and a way to tell "it changed for me" from "it changed for everyone."

Step 1 — Fix a canary suite

Pick a small, frozen set of prompts — 20 to 100 — that exercise the capabilities you actually depend on. Keep it small and cheap; this is a smoke detector, not a benchmark. Freeze it: same prompts, same parameters (temperature, top_p, max_tokens), same model string, forever. The moment you change the suite you lose your baseline.

For each canary, define a cheap scalar you can compute automatically:

JSON/schema validity rate (did it return parseable, valid output?)
Output length (a sudden shift often precedes a quality change)
A pass/fail against a fixed assertion (does the function it wrote compile?)
A rubric score from a cheap grader

You are not trying to measure "quality" in the abstract. You are trying to detect change. A boring, stable scalar is exactly what you want.

Step 2 — Record a baseline while things are good

Run the suite on a schedule — hourly, daily, whatever your budget allows — and store (timestamp, prompt_id, metric). Do this before you suspect anything. The baseline is the whole game: without "what normal looked like last month" you have nothing to compare today against. Estimate the mean mu0 and standard deviation sigma0 of your metric per prompt from a stable window.

Step 3 — Run a change-point test, not a threshold

A fixed threshold ("alert if success rate < 90%") is noisy and late. What you want is a sequential change-point detector that accumulates small, consistent shifts and fires with a timestamp. The classic is CUSUM (Page, 1954). It's a few lines:

def cusum_alarm(values, mu0, sigma0, k=0.5, h=5.0):
    """Two-sided CUSUM on standardized observations.
    k = slack (in std devs), h = alarm threshold.
    Returns the index where a sustained shift is detected, or None."""
    s_pos = s_neg = 0.0
    for i, x in enumerate(values):
        z = (x - mu0) / sigma0          # standardize against the baseline
        s_pos = max(0.0, s_pos + z - k) # catches upward shifts
        s_neg = max(0.0, s_neg - z - k) # catches downward shifts
        if s_pos > h or s_neg > h:
            return i                     # dated change-point
    return None

Feed it your per-prompt metric over time. Instead of "feels worse", you now get "json_success_rate on prompt #14 stepped down starting Tuesday" — a claim with a date on it that you can put in a ticket. (h=5.0, k=0.5 are reasonable starting points for standardized data; tune on your own history. Bayesian Online Change-Point Detection is a fancier alternative if you want posterior probabilities.)

This alone puts you ahead of ~95% of "the model got worse" threads, which never get past anecdotes.

Step 4 — The part one account can never answer: "is it just me?"

Here's the ceiling you hit: even with a perfect change-point signal, you still can't tell a provider-side change from your own drift, because from one seat they look identical. Your CUSUM fired — but was it the model, or the LlamaIndex bump you did last week, or a flaky region?

The only way to separate them is correlation across independent observers. If your canary fires on Tuesday, and several other organizations watching the same model on the same metric also fire around Tuesday, that agreement is the signal that it's provider-side — not your harness. One observer has N=1 forever; that's why these threads never converge. This is a genuinely different kind of detection: not "monitor my app" (every observability tool does that), but "did this change for everyone, privately confirmed across the fleet."

That cross-observer angle is what SEISMOGRAPH is built around: a lightweight probe runs your canary suite, a change-point detector flags shifts locally, and only privacy-preserving distributional features — never your prompts or outputs — are shared, so a public "model weather" alert only fires when multiple independent observers agree. A single org's signal stays private.

Does this actually work? A worked example

The method isn't hypothetical. Take the September 2025 Anthropic incident: for a stretch, claude-sonnet-4 degraded because of infrastructure bugs — notably a context-window routing error — not a model update (Anthropic said so explicitly in their postmortem). Exactly the "is it the model or the infra?" fog this whole article is about.

Replaying a canary suite over that period with the CUSUM method above: a seeded backtest flags it 38 days before the postmortem. Not a live catch — a reproducible backtest, seeded and open — but it shows the signal was sitting in cheap distributional metrics well before the official acknowledgement.

Do it yourself

Probe (open source, Apache-2.0): pip install seismograph-probe
Live "model weather" dashboard: https://driftdefense.dev?utm_source=devto&utm_medium=article&utm_campaign=evergreen_howto
Code + method: https://github.com/Tania-coder/SEISMOGRAPH

You don't need any of that to start — the four steps above work with a cron job, a CSV, and the ten lines of CUSUM. But if you want the "is it just me?" half answered, that needs more than one observer, and that's the part worth federating.

FAQ

Did GPT / Claude / Gemini actually get worse recently?
Sometimes yes, sometimes it's your setup, and often it's a provider-side infra change that isn't a "worse model" at all. The point of this article is that you shouldn't answer from vibes — instrument it and get a dated signal.

How can I prove a hosted model changed?
You can't prove it from one account, but you can get strong objective evidence: a frozen canary suite + a baseline + a change-point test gives you a dated shift. Provider-side vs. local drift is only separable with agreement across multiple independent observers.

Isn't this just LLM observability / evals?
Related but different. Evals score quality at a point in time; observability watches your traffic. This is about detecting change over time in a hosted dependency you don't control — and, uniquely, correlating that change across organizations to rule out "it's just me."

What metrics should I track?
Start with the cheapest ones that move when quality moves: schema/JSON validity, output length, and a single fixed pass/fail per canary. Add a rubric score later if you want resolution.

Your LLM didn't get worse. It changed — and nobody told you.

Tatiana Radchenko 🇩🇰 — Thu, 02 Jul 2026 11:22:37 +0000

It's 2am. Your JSON parse errors are up 12%. Latency: normal. Uptime: 100%. Your prompt didn't change. Your code didn't change.

Is it you, or did the model silently change underneath you?

Every team building on third-party LLM APIs eventually hits this moment. And the frustrating part: your monitoring stack can't answer the question. Latency, error rates, uptime — all green. The thing that changed is behavior, and behavior emits no infrastructure signal.

I built SEISMOGRAPH — an open-source, privacy-preserving early-warning network for exactly this failure mode. This post covers what silent drift is, how to detect it without leaking a single prompt, and a reproducible backtest where the detector flags a real, publicly documented provider incident 38 days before the official postmortem.

What "silent drift" actually is

Provider APIs do not broadcast behavioral changes. An endpoint that returns 200 can still start producing subtly different outputs: degraded JSON fidelity, shifted response-length distributions, changed reasoning patterns.

This isn't hypothetical. In September 2025, Anthropic published a detailed postmortem describing three infrastructure bugs that silently degraded Claude output quality across August–September 2025. The most long-lived one: a context-window routing error, introduced on August 5, that misrouted a fraction of Claude Sonnet 4 requests to servers configured for the 1M-token context window. It started at ~0.8% of traffic. On August 29 a load-balancer change escalated it to ~16%. The public postmortem landed on September 17.

Note what this incident was not: it was not a model update, and not intentional degradation — Anthropic was explicit about that. It was infrastructure. Which is precisely the point: the provider's own uptime and latency dashboards stayed green the whole time. The signal only existed at the semantic layer.

So the question I wanted to answer in Phase 0 of the project: would a lightweight behavioral canary have caught it, and how early?

The detection idea: canaries + change-point statistics

The approach is deliberately boring:

A canary suite — a fixed, content-addressed set of prompts (≤200, temperature 0) — runs on a schedule against the API you depend on. Deterministic prompts against a deterministic-ish endpoint give you a behavioral baseline. Every suite version is immutably hash-addressed, so a baseline can never silently mutate either.

Each response is reduced to distributional features: JSON parse success rate, output length, result counts. Not the text — the shape of the behavior.

Those features feed a Page-CUSUM change-point detector per (model_tuple, metric_name) stream:

S+(n) = max(0, S+(n-1) + z(n) - k)    # upward shifts
S-(n) = max(0, S-(n-1) - z(n) - k)    # downward shifts
Alert when S+ or S- > h

CUSUM accumulates small standardized deviations from a learned baseline (mu0, sigma0 from the first 30 observations). That's what makes it good at this job: a 0.8% fault doesn't produce a visible spike on any single day, but it produces a persistent small negative bias — and CUSUM integrates exactly that.

The privacy boundary (the part that makes federation possible)

One org's canary is a smoke detector. A network of canaries across independent orgs is an early-warning system — if orgs are willing to participate. They're only willing if participation leaks nothing.

So the probe enforces a hard perimeter. Raw prompts and outputs never leave your infrastructure. What gets transmitted:

SHA-256 hash of each response (not the response)
DP-noised aggregates: json_success_rate, avg_output_length (Laplace mechanism, ε = 2.0 per flush)
the content-addressed canary suite version hash
a pseudonymous Ed25519 public key — batches are signed, org identity is not disclosed

And the alerting side has a matching rule: a single organization's signal is never promoted to a public alert. A quorum scorer requires ≥ 2 independent orgs to agree on the same (model, metric) drift before anything goes public. That one gate filters probe bugs, network hiccups, and Sybil injection in a single move — and it means your private fleet data stays private fleet data.

The proof: a reproducible backtest

Here's the Phase 0 experiment (seeded, fully reproducible — SEED=42):

Reconstruct the Aug–Sep 2025 incident timeline from the public postmortem as synthetic daily probe traffic:

Phase	Dates	json_success_rate
Baseline	Jul 1 – Aug 4	0.990
Phase 1 (0.8% misrouting)	Aug 5 – Aug 28	0.982
Phase 2 (16% misrouting)	Aug 29 – Sep 17	0.840

Feed it through the exact CUSUM detector from the codebase (h=5.0, k=0.5). Watch when it fires:

Date        Phase          rate    S-      note
2025-08-03  Stable         0.9902  0.000
2025-08-04  Stable         0.9881  0.000
2025-08-05  Phase1(0.8%)   0.9855  0.598   [bug introduced]
2025-08-06  Phase1(0.8%)   0.9857  1.142
2025-08-07  Phase1(0.8%)   0.9786  3.309
2025-08-08  Phase1(0.8%)   0.9877  3.396
2025-08-09  Phase1(0.8%)   0.9816  4.889
2025-08-10  Phase1(0.8%)   0.9777  7.278   <<< FIRST ALERT

First alert: August 10 — five days into the subtle 0.8% window.

19 days before the escalation made the problem visible to users
38 days before the official postmortem

You can re-run this yourself: python scripts/anthropic_backtest.py in the repo regenerates the full report and asserts the result.

What this does and doesn't prove

I want to be precise here, because this is where drift-detection claims usually get slippery:

This is a backtest on synthetic data, not a live catch. The timeline and magnitudes come from the public postmortem; real probe noise may differ, and real lead time could be shorter or longer.
It's a single simulated observer. A real public alert requires the ≥2-org quorum.
The simulation omits live DP noise, which adds variance and may delay an alert by a few days.

What it does prove: a metric as cheap as "did the JSON parse?" — measured consistently, at temperature 0, against a fixed suite — carries enough signal to surface a 0.8%-of-traffic infrastructure fault weeks before it becomes user-visible. Conventional monitoring is structurally blind to this class of failure. A $0.10/day canary is not.

Architecture in one pass

your infra                          public
┌─────────────────────────┐
│ probe SDK               │
│  canary suite (≤200,    │
│  temp 0, hash-addressed)│
│  → features + DP noise  │
│  → Ed25519 signature    │
└──────────┬──────────────┘
           ▼
   ingestion gateway (FastAPI)
   schema validation, signature check
           ▼
   CUSUM per (model, metric)  →  local alert (private)
           ▼
   quorum scorer (≥2 orgs)    →  public drift alert
           ▼
   "model weather" dashboard

Stack: Python probe SDK (OpenTelemetry GenAI semantic conventions), FastAPI gateway, SQLAlchemy storage, CUSUM + Bayesian online change-point detection, 127 tests, ruff-clean CI. Apache-2.0.

Try it

The public "model weather" dashboard (no login) shows live drift status for four production models:

seismograph-weather.onrender.com/dashboard (free host — first load may take ~30s)

Run the probe against your own endpoint:

pip install seismograph-probe

Or clone the whole stack — gateway, dashboard, the federated quorum demo (scripts/demo_simulation.py shows two orgs independently confirming a drift and the public dashboard flipping only when quorum is met):

github.com/Tania-coder/SEISMOGRAPH

If you've been burned by a silent model change — I'd like to hear the story. That failure mode is exactly what this network exists to catch, and every additional independent observer makes the signal stronger.

SEISMOGRAPH is open source (Apache-2.0), archived at DOI 10.5281/zenodo.21045517. Built by Tatiana Radchenko.

SEISMOGRAPH now has a live public dashboard

Tatiana Radchenko 🇩🇰 — Wed, 24 Jun 2026 12:04:48 +0000

When I wrote about detecting silent LLM provider drift, the dashboard was localhost-only — you had to take my word for it. Now it's live and you can touch it:

👉 https://seismograph-weather.onrender.com/dashboard

It shows real-time "model weather" for 4 production models (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Claude 3 Haiku) — no login. The same engine flagged the Claude 3.5 Sonnet degradation 38 days before the official postmortem on a reproducible backtest.

Open-source, Apache-2.0, 107 tests.

pip install seismograph-probe

Repo: https://github.com/Tania-coder/SEISMOGRAPH — building in public, feedback welcome.

I've been building SEISMOGRAPH for 3 weeks. Here's what shipped today

Tatiana Radchenko 🇩🇰 — Tue, 23 Jun 2026 20:42:47 +0000

tl;dr: pip install seismograph-probe — a Python probe that detects silent LLM API drift using CUSUM change-point detection, with privacy-preserving signal aggregation. 103 tests passing. Dashboard live. Open source.

Three weeks ago I asked a question I couldn't answer:

"Did GPT-4 just change underneath me, or is it my prompt?"

No latency spike. No downtime. Just subtly different outputs from the same prompts, same parameters, same everything. I spent days debugging something that wasn't my fault.

So I built a detector.

Today I'm shipping it publicly.

What's actually working right now
This isn't a concept post. Here's what's live:
The probe SDK — on PyPI today
pip install seismograph-probe

from probe.sdk import ProbeSDK

sdk = ProbeSDK(provider="openai", model="gpt-4-turbo")

result = sdk.run_canary_suite()

print(result.drift_score) # 0.0 stable → 1.0 significant shift

The probe runs ≤200 canary prompts at temperature=0 daily. These are semantically stable tasks — deterministic questions, structured reasoning, format-adherence checks. The goal is a reliable behavioral baseline, not a capability benchmark.

Privacy boundary: raw prompts and model outputs never leave your machine. The probe extracts SHA-256 feature hashes, distributional stats, and DP-noised aggregates. That's all that transmits.
CUSUM change-point detection — running
The correlation engine uses CUSUM (Cumulative Sum) — a sequential statistical test that's sensitive to gradual drift, not just threshold crossings.

When I backtest against a known LLM behavioral shift event (Aug–Sep 2025):

Day 0: CUSUM statistic: 0.12 (stable baseline)

Day 11: First elevation detected

Day 19: Alert threshold crossed ← SEISMOGRAPH fires

Day 57: Public postmortem published

38-day lead time. That's the number I keep coming back to.
Ingestion gateway — deployed
FastAPI gateway with:

Ed25519-signed batch verification (unsigned batches rejected atomically)
Pydantic v2 schema validation
SQLAlchemy ORM + SQLite (ClickHouse migration planned for Phase 2)
Bearer token auth on audit export endpoint
Public dashboard — live at localhost, hosted version coming
Dark-mode model weather dashboard. Polls /v1/weather every 60 seconds. Shows per-model drift status across your fleet.

GET /v1/weather

→ [{ "model": "gpt-4-turbo", "status": "STABLE", ... },

{ "model": "claude-3-5-sonnet", "status": "STABLE", ... }]
Test suite — 103/103 passing
Not "it works on my machine." 103 tests across probe SDK, storage layer, gateway, CUSUM detector, privacy boundary, and auth. Zero ruff violations across 22 Python files.
Provider ToS compliance — checked
Before adding any provider to the canary suite, I verify it doesn't violate their Terms of Service. Done for: OpenAI ✅, Anthropic ✅, Google Gemini ✅, Mistral ✅, Cohere ✅. Documented in docs/PROVIDER_TOS_CHECKS.md.

What's NOT done yet (being honest)
No hosted gateway yet. The gateway runs locally. Public ingestion endpoint is Phase 1.
No Bayesian online detector yet. CUSUM is running. BayesianOnlineDetector.update() is deferred — it's on the backlog.
No federation yet. Right now it's single-org. The cross-observer agreement scoring that makes it genuinely valuable is Phase 2.
No cloud dashboard. localhost:8000 only for now.

This is Phase 0: I'm proving the detection logic works before scaling it.

The architecture in one diagram
Your app

│ (gen_ai.* OTel spans)

▼

ProbeSDK

│ SHA-256 hashes + DP-noised stats only

│ Ed25519-signed batch

▼

Ingestion Gateway (FastAPI)

│ signature check → schema validation → store

▼

SQLite / ClickHouse

│

▼

CUSUM Detector ──► DriftAlert

│

▼

/v1/weather dashboard

OTel-native throughout. If you're already emitting gen_ai.* spans, the adapter plugs straight in.

Why this matters (and why it has to be federated)
A single organization's drift signal is almost useless. Your outputs change because your users change. Your prompts change. Your context windows change.

But if 15 independent organizations running the same canary suite all see correlated semantic drift on the same day — that's a model change. That's the signal you can act on.

Single-org signal = private fleet data (yours only).
Multi-org correlated signal = public drift alert.

That's the design. Federation is Phase 2. The local probe is shippable today.

Try it / follow along
GitHub: github.com/Tania-coder/SEISMOGRAPH
PyPI: pypi.org/project/seismograph-probe

If you've been burned by a silent model change — I want to hear about it. Open an issue, or find me on Twitter @tatyanti.

The probe is Apache 2.0. The gateway will be too.

Tatiana Radchenko · AI Infrastructure · Aarhus, Denmark
Building in public. Phase 0 of 3.