DEV Community: Tatiana Radchenko 🇩🇰

Your LLM didn't get worse. It changed — and nobody told you.

Tatiana Radchenko 🇩🇰 — Thu, 02 Jul 2026 11:22:37 +0000

It's 2am. Your JSON parse errors are up 12%. Latency: normal. Uptime: 100%. Your prompt didn't change. Your code didn't change.

Is it you, or did the model silently change underneath you?

Every team building on third-party LLM APIs eventually hits this moment. And the frustrating part: your monitoring stack can't answer the question. Latency, error rates, uptime — all green. The thing that changed is behavior, and behavior emits no infrastructure signal.

I built SEISMOGRAPH — an open-source, privacy-preserving early-warning network for exactly this failure mode. This post covers what silent drift is, how to detect it without leaking a single prompt, and a reproducible backtest where the detector flags a real, publicly documented provider incident 38 days before the official postmortem.

What "silent drift" actually is

Provider APIs do not broadcast behavioral changes. An endpoint that returns 200 can still start producing subtly different outputs: degraded JSON fidelity, shifted response-length distributions, changed reasoning patterns.

This isn't hypothetical. In September 2025, Anthropic published a detailed postmortem describing three infrastructure bugs that silently degraded Claude output quality across August–September 2025. The most long-lived one: a context-window routing error, introduced on August 5, that misrouted a fraction of Claude Sonnet 4 requests to servers configured for the 1M-token context window. It started at ~0.8% of traffic. On August 29 a load-balancer change escalated it to ~16%. The public postmortem landed on September 17.

Note what this incident was not: it was not a model update, and not intentional degradation — Anthropic was explicit about that. It was infrastructure. Which is precisely the point: the provider's own uptime and latency dashboards stayed green the whole time. The signal only existed at the semantic layer.

So the question I wanted to answer in Phase 0 of the project: would a lightweight behavioral canary have caught it, and how early?

The detection idea: canaries + change-point statistics

The approach is deliberately boring:

A canary suite — a fixed, content-addressed set of prompts (≤200, temperature 0) — runs on a schedule against the API you depend on. Deterministic prompts against a deterministic-ish endpoint give you a behavioral baseline. Every suite version is immutably hash-addressed, so a baseline can never silently mutate either.

Each response is reduced to distributional features: JSON parse success rate, output length, result counts. Not the text — the shape of the behavior.

Those features feed a Page-CUSUM change-point detector per (model_tuple, metric_name) stream:

S+(n) = max(0, S+(n-1) + z(n) - k)    # upward shifts
S-(n) = max(0, S-(n-1) - z(n) - k)    # downward shifts
Alert when S+ or S- > h

CUSUM accumulates small standardized deviations from a learned baseline (mu0, sigma0 from the first 30 observations). That's what makes it good at this job: a 0.8% fault doesn't produce a visible spike on any single day, but it produces a persistent small negative bias — and CUSUM integrates exactly that.

The privacy boundary (the part that makes federation possible)

One org's canary is a smoke detector. A network of canaries across independent orgs is an early-warning system — if orgs are willing to participate. They're only willing if participation leaks nothing.

So the probe enforces a hard perimeter. Raw prompts and outputs never leave your infrastructure. What gets transmitted:

SHA-256 hash of each response (not the response)
DP-noised aggregates: json_success_rate, avg_output_length (Laplace mechanism, ε = 2.0 per flush)
the content-addressed canary suite version hash
a pseudonymous Ed25519 public key — batches are signed, org identity is not disclosed

And the alerting side has a matching rule: a single organization's signal is never promoted to a public alert. A quorum scorer requires ≥ 2 independent orgs to agree on the same (model, metric) drift before anything goes public. That one gate filters probe bugs, network hiccups, and Sybil injection in a single move — and it means your private fleet data stays private fleet data.

The proof: a reproducible backtest

Here's the Phase 0 experiment (seeded, fully reproducible — SEED=42):

Reconstruct the Aug–Sep 2025 incident timeline from the public postmortem as synthetic daily probe traffic:

Phase	Dates	json_success_rate
Baseline	Jul 1 – Aug 4	0.990
Phase 1 (0.8% misrouting)	Aug 5 – Aug 28	0.982
Phase 2 (16% misrouting)	Aug 29 – Sep 17	0.840

Feed it through the exact CUSUM detector from the codebase (h=5.0, k=0.5). Watch when it fires:

Date        Phase          rate    S-      note
2025-08-03  Stable         0.9902  0.000
2025-08-04  Stable         0.9881  0.000
2025-08-05  Phase1(0.8%)   0.9855  0.598   [bug introduced]
2025-08-06  Phase1(0.8%)   0.9857  1.142
2025-08-07  Phase1(0.8%)   0.9786  3.309
2025-08-08  Phase1(0.8%)   0.9877  3.396
2025-08-09  Phase1(0.8%)   0.9816  4.889
2025-08-10  Phase1(0.8%)   0.9777  7.278   <<< FIRST ALERT

First alert: August 10 — five days into the subtle 0.8% window.

19 days before the escalation made the problem visible to users
38 days before the official postmortem

You can re-run this yourself: python scripts/anthropic_backtest.py in the repo regenerates the full report and asserts the result.

What this does and doesn't prove

I want to be precise here, because this is where drift-detection claims usually get slippery:

This is a backtest on synthetic data, not a live catch. The timeline and magnitudes come from the public postmortem; real probe noise may differ, and real lead time could be shorter or longer.
It's a single simulated observer. A real public alert requires the ≥2-org quorum.
The simulation omits live DP noise, which adds variance and may delay an alert by a few days.

What it does prove: a metric as cheap as "did the JSON parse?" — measured consistently, at temperature 0, against a fixed suite — carries enough signal to surface a 0.8%-of-traffic infrastructure fault weeks before it becomes user-visible. Conventional monitoring is structurally blind to this class of failure. A $0.10/day canary is not.

Architecture in one pass

your infra                          public
┌─────────────────────────┐
│ probe SDK               │
│  canary suite (≤200,    │
│  temp 0, hash-addressed)│
│  → features + DP noise  │
│  → Ed25519 signature    │
└──────────┬──────────────┘
           ▼
   ingestion gateway (FastAPI)
   schema validation, signature check
           ▼
   CUSUM per (model, metric)  →  local alert (private)
           ▼
   quorum scorer (≥2 orgs)    →  public drift alert
           ▼
   "model weather" dashboard

Stack: Python probe SDK (OpenTelemetry GenAI semantic conventions), FastAPI gateway, SQLAlchemy storage, CUSUM + Bayesian online change-point detection, 127 tests, ruff-clean CI. Apache-2.0.

Try it

The public "model weather" dashboard (no login) shows live drift status for four production models:

seismograph-weather.onrender.com/dashboard (free host — first load may take ~30s)

Run the probe against your own endpoint:

pip install seismograph-probe

Or clone the whole stack — gateway, dashboard, the federated quorum demo (scripts/demo_simulation.py shows two orgs independently confirming a drift and the public dashboard flipping only when quorum is met):

github.com/Tania-coder/SEISMOGRAPH

If you've been burned by a silent model change — I'd like to hear the story. That failure mode is exactly what this network exists to catch, and every additional independent observer makes the signal stronger.

SEISMOGRAPH is open source (Apache-2.0), archived at DOI 10.5281/zenodo.21045517. Built by Tatiana Radchenko.

SEISMOGRAPH now has a live public dashboard

Tatiana Radchenko 🇩🇰 — Wed, 24 Jun 2026 12:04:48 +0000

When I wrote about detecting silent LLM provider drift, the dashboard was localhost-only — you had to take my word for it. Now it's live and you can touch it:

👉 https://seismograph-weather.onrender.com/dashboard

It shows real-time "model weather" for 4 production models (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Claude 3 Haiku) — no login. The same engine flagged the Claude 3.5 Sonnet degradation 38 days before the official postmortem on a reproducible backtest.

Open-source, Apache-2.0, 107 tests.

pip install seismograph-probe

Repo: https://github.com/Tania-coder/SEISMOGRAPH — building in public, feedback welcome.

I've been building SEISMOGRAPH for 3 weeks. Here's what shipped today

Tatiana Radchenko 🇩🇰 — Tue, 23 Jun 2026 20:42:47 +0000

tl;dr: pip install seismograph-probe — a Python probe that detects silent LLM API drift using CUSUM change-point detection, with privacy-preserving signal aggregation. 103 tests passing. Dashboard live. Open source.

Three weeks ago I asked a question I couldn't answer:

"Did GPT-4 just change underneath me, or is it my prompt?"

No latency spike. No downtime. Just subtly different outputs from the same prompts, same parameters, same everything. I spent days debugging something that wasn't my fault.

So I built a detector.

Today I'm shipping it publicly.

What's actually working right now
This isn't a concept post. Here's what's live:
The probe SDK — on PyPI today
pip install seismograph-probe

from probe.sdk import ProbeSDK

sdk = ProbeSDK(provider="openai", model="gpt-4-turbo")

result = sdk.run_canary_suite()

print(result.drift_score) # 0.0 stable → 1.0 significant shift

The probe runs ≤200 canary prompts at temperature=0 daily. These are semantically stable tasks — deterministic questions, structured reasoning, format-adherence checks. The goal is a reliable behavioral baseline, not a capability benchmark.

Privacy boundary: raw prompts and model outputs never leave your machine. The probe extracts SHA-256 feature hashes, distributional stats, and DP-noised aggregates. That's all that transmits.
CUSUM change-point detection — running
The correlation engine uses CUSUM (Cumulative Sum) — a sequential statistical test that's sensitive to gradual drift, not just threshold crossings.

When I backtest against a known LLM behavioral shift event (Aug–Sep 2025):

Day 0: CUSUM statistic: 0.12 (stable baseline)

Day 11: First elevation detected

Day 19: Alert threshold crossed ← SEISMOGRAPH fires

Day 57: Public postmortem published

38-day lead time. That's the number I keep coming back to.
Ingestion gateway — deployed
FastAPI gateway with:

Ed25519-signed batch verification (unsigned batches rejected atomically)
Pydantic v2 schema validation
SQLAlchemy ORM + SQLite (ClickHouse migration planned for Phase 2)
Bearer token auth on audit export endpoint
Public dashboard — live at localhost, hosted version coming
Dark-mode model weather dashboard. Polls /v1/weather every 60 seconds. Shows per-model drift status across your fleet.

GET /v1/weather

→ [{ "model": "gpt-4-turbo", "status": "STABLE", ... },

{ "model": "claude-3-5-sonnet", "status": "STABLE", ... }]
Test suite — 103/103 passing
Not "it works on my machine." 103 tests across probe SDK, storage layer, gateway, CUSUM detector, privacy boundary, and auth. Zero ruff violations across 22 Python files.
Provider ToS compliance — checked
Before adding any provider to the canary suite, I verify it doesn't violate their Terms of Service. Done for: OpenAI ✅, Anthropic ✅, Google Gemini ✅, Mistral ✅, Cohere ✅. Documented in docs/PROVIDER_TOS_CHECKS.md.

What's NOT done yet (being honest)
No hosted gateway yet. The gateway runs locally. Public ingestion endpoint is Phase 1.
No Bayesian online detector yet. CUSUM is running. BayesianOnlineDetector.update() is deferred — it's on the backlog.
No federation yet. Right now it's single-org. The cross-observer agreement scoring that makes it genuinely valuable is Phase 2.
No cloud dashboard. localhost:8000 only for now.

This is Phase 0: I'm proving the detection logic works before scaling it.

The architecture in one diagram
Your app

│ (gen_ai.* OTel spans)

▼

ProbeSDK

│ SHA-256 hashes + DP-noised stats only

│ Ed25519-signed batch

▼

Ingestion Gateway (FastAPI)

│ signature check → schema validation → store

▼

SQLite / ClickHouse

│

▼

CUSUM Detector ──► DriftAlert

│

▼

/v1/weather dashboard

OTel-native throughout. If you're already emitting gen_ai.* spans, the adapter plugs straight in.

Why this matters (and why it has to be federated)
A single organization's drift signal is almost useless. Your outputs change because your users change. Your prompts change. Your context windows change.

But if 15 independent organizations running the same canary suite all see correlated semantic drift on the same day — that's a model change. That's the signal you can act on.

Single-org signal = private fleet data (yours only).
Multi-org correlated signal = public drift alert.

That's the design. Federation is Phase 2. The local probe is shippable today.

Try it / follow along
GitHub: github.com/Tania-coder/SEISMOGRAPH
PyPI: pypi.org/project/seismograph-probe

If you've been burned by a silent model change — I want to hear about it. Open an issue, or find me on Twitter @tatyanti.

The probe is Apache 2.0. The gateway will be too.

Tatiana Radchenko · AI Infrastructure · Aarhus, Denmark
Building in public. Phase 0 of 3.