Your LLM didn't get worse. It changed — and nobody told you.

It's 2am. Your JSON parse errors are up 12%. Latency: normal. Uptime: 100%. Your prompt didn't change. Your code didn't change.

Is it you, or did the model silently change underneath you?

Every team building on third-party LLM APIs eventually hits this moment. And the frustrating part: your monitoring stack can't answer the question. Latency, error rates, uptime — all green. The thing that changed is behavior, and behavior emits no infrastructure signal.

I built SEISMOGRAPH — an open-source, privacy-preserving early-warning network for exactly this failure mode. This post covers what silent drift is, how to detect it without leaking a single prompt, and a reproducible backtest where the detector flags a real, publicly documented provider incident 38 days before the official postmortem.

What "silent drift" actually is

Provider APIs do not broadcast behavioral changes. An endpoint that returns 200 can still start producing subtly different outputs: degraded JSON fidelity, shifted response-length distributions, changed reasoning patterns.

This isn't hypothetical. In September 2025, Anthropic published a detailed postmortem describing three infrastructure bugs that silently degraded Claude output quality across August–September 2025. The most long-lived one: a context-window routing error, introduced on August 5, that misrouted a fraction of Claude Sonnet 4 requests to servers configured for the 1M-token context window. It started at ~0.8% of traffic. On August 29 a load-balancer change escalated it to ~16%. The public postmortem landed on September 17.

Note what this incident was not: it was not a model update, and not intentional degradation — Anthropic was explicit about that. It was infrastructure. Which is precisely the point: the provider's own uptime and latency dashboards stayed green the whole time. The signal only existed at the semantic layer.

So the question I wanted to answer in Phase 0 of the project: would a lightweight behavioral canary have caught it, and how early?

The detection idea: canaries + change-point statistics

The approach is deliberately boring:

A canary suite — a fixed, content-addressed set of prompts (≤200, temperature 0) — runs on a schedule against the API you depend on. Deterministic prompts against a deterministic-ish endpoint give you a behavioral baseline. Every suite version is immutably hash-addressed, so a baseline can never silently mutate either.

Each response is reduced to distributional features: JSON parse success rate, output length, result counts. Not the text — the shape of the behavior.

Those features feed a Page-CUSUM change-point detector per (model_tuple, metric_name) stream:

S+(n) = max(0, S+(n-1) + z(n) - k)    # upward shifts
S-(n) = max(0, S-(n-1) - z(n) - k)    # downward shifts
Alert when S+ or S- > h

CUSUM accumulates small standardized deviations from a learned baseline (mu0, sigma0 from the first 30 observations). That's what makes it good at this job: a 0.8% fault doesn't produce a visible spike on any single day, but it produces a persistent small negative bias — and CUSUM integrates exactly that.

The privacy boundary (the part that makes federation possible)

One org's canary is a smoke detector. A network of canaries across independent orgs is an early-warning system — if orgs are willing to participate. They're only willing if participation leaks nothing.

So the probe enforces a hard perimeter. Raw prompts and outputs never leave your infrastructure. What gets transmitted:

• SHA-256 hash of each response (not the response)
• DP-noised aggregates: json_success_rate, avg_output_length (Laplace mechanism, ε = 2.0 per flush)
• the content-addressed canary suite version hash
• a pseudonymous Ed25519 public key — batches are signed, org identity is not disclosed

And the alerting side has a matching rule: a single organization's signal is never promoted to a public alert. A quorum scorer requires ≥ 2 independent orgs to agree on the same (model, metric) drift before anything goes public. That one gate filters probe bugs, network hiccups, and Sybil injection in a single move — and it means your private fleet data stays private fleet data.

The proof: a reproducible backtest

Here's the Phase 0 experiment (seeded, fully reproducible — SEED=42):

Reconstruct the Aug–Sep 2025 incident timeline from the public postmortem as synthetic daily probe traffic:

Phase	Dates	json_success_rate
Baseline	Jul 1 – Aug 4	0.990
Phase 1 (0.8% misrouting)	Aug 5 – Aug 28	0.982
Phase 2 (16% misrouting)	Aug 29 – Sep 17	0.840

Feed it through the exact CUSUM detector from the codebase (h=5.0, k=0.5). Watch when it fires:

Date        Phase          rate    S-      note
2025-08-03  Stable         0.9902  0.000
2025-08-04  Stable         0.9881  0.000
2025-08-05  Phase1(0.8%)   0.9855  0.598   [bug introduced]
2025-08-06  Phase1(0.8%)   0.9857  1.142
2025-08-07  Phase1(0.8%)   0.9786  3.309
2025-08-08  Phase1(0.8%)   0.9877  3.396
2025-08-09  Phase1(0.8%)   0.9816  4.889
2025-08-10  Phase1(0.8%)   0.9777  7.278   <<< FIRST ALERT

First alert: August 10 — five days into the subtle 0.8% window.

• 19 days before the escalation made the problem visible to users
• 38 days before the official postmortem

You can re-run this yourself: python scripts/anthropic_backtest.py in the repo regenerates the full report and asserts the result.

What this does and doesn't prove

I want to be precise here, because this is where drift-detection claims usually get slippery:

1. This is a backtest on synthetic data, not a live catch. The timeline and magnitudes come from the public postmortem; real probe noise may differ, and real lead time could be shorter or longer.
2. It's a single simulated observer. A real public alert requires the ≥2-org quorum.
3. The simulation omits live DP noise, which adds variance and may delay an alert by a few days.

What it does prove: a metric as cheap as "did the JSON parse?" — measured consistently, at temperature 0, against a fixed suite — carries enough signal to surface a 0.8%-of-traffic infrastructure fault weeks before it becomes user-visible. Conventional monitoring is structurally blind to this class of failure. A $0.10/day canary is not.

Architecture in one pass

your infra                          public
┌─────────────────────────┐
│ probe SDK               │
│  canary suite (≤200,    │
│  temp 0, hash-addressed)│
│  → features + DP noise  │
│  → Ed25519 signature    │
└──────────┬──────────────┘
           ▼
   ingestion gateway (FastAPI)
   schema validation, signature check
           ▼
   CUSUM per (model, metric)  →  local alert (private)
           ▼
   quorum scorer (≥2 orgs)    →  public drift alert
           ▼
   "model weather" dashboard

Stack: Python probe SDK (OpenTelemetry GenAI semantic conventions), FastAPI gateway, SQLAlchemy storage, CUSUM + Bayesian online change-point detection, 127 tests, ruff-clean CI. Apache-2.0.

Try it

The public "model weather" dashboard (no login) shows live drift status for four production models:

seismograph-weather.onrender.com/dashboard (free host — first load may take ~30s)

Run the probe against your own endpoint:

pip install seismograph-probe

Or clone the whole stack — gateway, dashboard, the federated quorum demo (scripts/demo_simulation.py shows two orgs independently confirming a drift and the public dashboard flipping only when quorum is met):

github.com/Tania-coder/SEISMOGRAPH

If you've been burned by a silent model change — I'd like to hear the story. That failure mode is exactly what this network exists to catch, and every additional independent observer makes the signal stronger.

---

SEISMOGRAPH is open source (Apache-2.0), archived at DOI 10.5281/zenodo.21045517. Built by Tatiana Radchenko.

Comments

[Sol]

The 2am JSON parse error spike from a silent model change is one of the harder failure classes — it looks like a code regression until you rule out everything else. We've been documenting similar incident patterns across teams shipping OpenAI/Anthropic features: the failure kind, the first signal the team saw, and how long it took to isolate the real cause. Model/version mismatches consistently show up as among the slowest to diagnose, because there's no error code pointing at a provider-side change.

What was your first clear signal that it was a model change rather than a bug in your own code? And did the provider changelog give you any useful signal, or did you trace it yourself?

[Tatiana Radchenko 🇩🇰]

Good question. To be upfront: I didn't personally live through this exact incident as an end user — the 38-day number comes from a reproducible, seeded backtest (SEED=42) against the public Anthropic postmortem timeline, not a live catch. I'm explicit about that in the post because backtest vs. live-catch claims get blurred a lot in this space.
That said, the pattern you're describing is exactly the gap a dedicated canary closes. Today, "do the error rates look weird" is the first signal most teams get, and it's slow — easy to rationalize away as your own bug first. A fixed suite (frozen prompts, temp 0, hash-addressed) removes that ambiguity: if json_success_rate drifts on prompts that haven't changed, it's not your code, it's the provider. And changelogs don't help in real time — Anthropic's postmortem here landed 38 days after the bug shipped — so in practice teams trace it themselves the hard way, same as you're describing.
Curious what your data shows, though — how long do model/version mismatches take to isolate compared to the other failure classes you've documented?