DEV Community: Tatiana Radchenko 🇩🇰

My drift board learned to say "I don't know". Here is what it still cannot say.

Tatiana Radchenko 🇩🇰 — Wed, 30 Sep 2026 21:05:55 +0000

Archival copy of record, committed before this post: https://github.com/Tania-coder/SEISMOGRAPH/blob/main/docs/reports/2026-09-22-weather-report-02.md

Provenance. Every figure below is recomputed from two committed
snapshots of the public board by scripts/weather_window_stats.py,
which runs in the project's test gate. The snapshots are
docs/evidence/weather-2026-09-09T173824Z.json and
docs/evidence/weather-2026-09-22T104310Z.json; the update block uses
a third and a fourth, docs/evidence/weather-2026-09-30T193201Z.json
and docs/evidence/weather-2026-09-30T205449Z.json. Nothing here is
arithmetic done once in prose.

Update, 2026-09-30. This report was written on 2026-09-22 and is
published eight days later. The body below is unchanged, and its
figures are as of the 2026-09-22 read. Before publishing I read the
board again, at 19:32 UTC, and recomputed with the same instrument:

mistral has written no row since 2026-09-19T09:31:44Z. Its newest row is now 274.01 h old, eleven days, and the board publishes STALE, which agrees. It still prints a sample count of 10 over a window that opens on 2026-09-02. Why it stopped is still not diagnosed.
google is collecting at 0.8926 of its nominal rate, newest row 8.26 h old, STABLE, which agrees. Its saturation bound is now at most 41.6%.

None of the findings below changed.

Later the same evening. The cause was found: the API key had
expired. I had created it with an expiry date and forgotten. With a new
key the leg wrote a row at 2026-09-30T20:48:38Z, and the next read
publishes STABLE. That is limitation 2 below, happening live: the
newest row is 0.10 h old, but the ten samples span 363.20 h and
the leg collects at 0.2974 of its nominal rate. Fresh, and still a
fifteen-day average that nobody should read as recent. The logs also
showed that the probe saw every request fail and threw the reason away
instead of writing it down. That is the next fix.

SEISMOGRAPH is an open-source early-warning network for silent drift in
third-party LLM APIs -- the 2am question of whether the model changed
underneath you or your prompt is simply worse today. It runs a fixed
canary suite against provider endpoints on a schedule, ships only
hashes and differentially-private aggregates, and publishes a public
board.

The same two things, up front, because everything below depends on
them:

There is exactly one observer, and it is me. A public drift alert requires cross-observer agreement from three independent observers. At one observer, no public alert can fire by construction. That is a property of the design, not a gap in it.
The live board's baseline was re-established on 2026-08-04. There is no continuous history before that date. Separately, on historical replay of a known incident, a seeded backtest flags it 38 days before the postmortem. That is a synthetic replay, not a live catch, and I do not describe it as one.

Report #1
ended with a list of limitations. The sixth read:

No staleness signal. mistral's 56-hour-old row is visible on the
board but not flagged, because "stale after N hours" is a judgement
that needs its own defence. The timestamps are published; the reader
concludes.

That one is now closed, and closing it turned out to be the smaller
half of the story.

The same leg, eighteen days apart

mistral/mistral-small-latest	2026-09-09 17:38 UTC	2026-09-22 10:43 UTC
`sample_count`	10	10
`window_end`	2026-09-02T09:38:39Z	2026-09-19T09:31:44Z
age of newest row	176.00 h	73.19 h
published `status`	STABLE	STALE
status agrees with age	no	yes

Nothing about that leg got better. A third status did.

The board's status used to read one table: DRIFTING if a
quorum-verified alert existed, STABLE otherwise. At one observer no
alert can exist, so STABLE was not a finding. The green light was
produced by the system's own inability to raise an alarm. A leg whose
probe had been dead for a week published it with a healthy-looking
count of ten.

There are now three states, and a leg whose newest sample is older than
30 hours publishes STALE regardless of what the alert table says. The
threshold is not a guess in a report: the probe fires twice a day and
was measured running 2.5 to 4.5 hours behind its own cron slot, so 30
hours is one missed slot plus that observed lateness. Below it, "merely
late" and "already stopped" are not separable, and I would rather the
board say nothing than say the wrong one.

STALE does not hide or alter the leg's published metrics, and it does
not attempt to say why the leg stopped. It is a statement about the
data, not about the provider.

What the counts still cannot see

Here is the part I got wrong in Report #1, and the instrument is what
showed me.

Report #1 treated "the mistral row is 56 hours old" and "the mistral
leg is losing runs" as one defect. They are two, they are independent,
and only one of them is now visible on the board.

Recomputed from the snapshots:

	2026-09-09	2026-09-22
window span	112.19 h	407.88 h
mean interval between samples	12.47 h	45.32 h
nominal interval (twice daily)	12.00 h	12.00 h
collection rate	0.9627	0.2648

On 2026-09-09 that leg was stale but collecting normally -- 12.47
hours between samples against a 12.00 hour schedule. Today it is
flagged stale and collecting at a quarter of its own cadence: ten
rows spread across seventeen days. Its window_start is still
2026-09-02T09:38:39Z, the frozen timestamp from the September outage,
because ten rows in seventeen days is not enough to evict it from a
last-ten window.

So the published "recent average output length" for that leg covers
seventeen days, and the google leg's covers five. The board does
not say so. Both legs print 10 / 10 / 10.

This is the same shape as Report #1's finding, one level up. A run that
exhausts its retries writes no row at all, so a count can never witness
its own losses. Adding a third status fixed the age of the newest
row. It did not make the rate visible, and the rate is what tells you
whether a number means anything.

The cadence figures above are not prose. nominal interval is parsed
out of the cron line in the probe's own workflow file, so if I change
the schedule and forget a report, the test suite fails instead of the
report quietly lying. That parse refuses anything it cannot interpret
rather than guessing.

The second way a stream can look stable

There is a failure mode in this design that no staleness flag reaches,
and it is worse than a dark leg because it is not dark.

The probe clamps each output length to 320 characters before averaging,
because the differential-privacy noise is calibrated to that bound. On
the corpus I measure with, that clamp removes about a third of the
mean, and the signal that separates two model generations survives it
at roughly ten times the noise. I had predicted the clamp would destroy
that signal. It did not, and the prediction was wrong for an
interesting reason: what matters is not the mean length but the
fraction of records above the cap, which depends on spread.

Take a corpus with a tight spread and a mean above the cap. Every
record saturates. Both legs then report exactly 320, a difference of
exactly 0.00 where the unclamped difference would have been -141.6
characters -- and with the DP noise applied, the stream emits -3.12.

That number is the dangerous one. It does not read as "not measured".
It reads as "almost perfectly stable".

A saturated stream is not a stable model. It is an unmeasured one, and
in every field the board publishes, the two are identical. I do not
have this measured on the live legs; from the published means I can
only bound it -- at most 41.9% of google's records and at most 28.6% of
mistral's can be at the cap. Near-total saturation is ruled out on both.
The exact fraction is not measured, and I am not going to state it as
though it were.

The rule I took from this, and now apply: a number that appears in a
published artefact has an instrument that recomputes it in the test
gate. Prose recording a measurement is a report of a measurement, not
the measurement. Every figure in this report satisfies that rule; the
script that produces them is in the repository and runs in the gate.

Where a benchmark comes in, and why not yet

The feedback I keep getting on the 50-prompt canary suite is the
correct feedback: don't build your own benchmark, build the
infrastructure around the ones that exist and let the operator pick.
That is the direction, and it is decided: only a suite registered in a
pinned public registry can ever enter cross-observer correlation, while
private corpora stay fleet-only. Naive pluggability would make the
observer count one per suite and quietly destroy the quorum that
makes a public alert mean anything.

What the clamp finding adds is a prerequisite I did not see before.
External benchmarks have wildly different output profiles -- a
multiple-choice suite sits far below the cap, a code or essay suite may
sit entirely above it. Running one of those through this probe today
could produce exactly the -3.12 shape above. So before any external
suite enters the network, its stream has to be classified interpretable
by the saturation instrument. That is a direction and a constraint, not
a shipping date, and no external benchmark has been run through this
probe yet.

I was wrong about the Google leg

Worth recording because the correction only happened because someone
asked the right question.

The google leg authenticates with a key in a format the provider is
migrating to. Two auth paths were tested; both failed; I wrote down
that the observer could not be restored, and started reasoning about
which of three bad options to take.

Then I was asked what had actually been tested, as against what was
being claimed. Four paths, with the response body read rather than the
status code alone:

	native endpoint	OpenAI-compatible layer
provider-specific key header	200	400, missing or invalid Authorization header
`Authorization: Bearer`	401	400

The key is live. The account is live. What is incompatible is the auth
scheme -- the compatible layer demands a header the new key format is
rejected in, and my provider code speaks only that layer. The leg needs
code, not a new credential, and the model tuple and board history are
preserved.

Two tested paths were generalised into a property of the world. To be
precise about what is still inferred: the 200 was against a model
listing, not against a completion. That a generation call with the same
header also succeeds is expected and not yet measured.

Limitations, stated rather than buried

One observer. No public alert can fire. Everything above is single-organisation fleet data.
The collection rate is still not on the board. STALE flags the age of the newest row. A leg collecting at a quarter of its cadence with a fresh newest row would publish STABLE and be within its rights.
The censoring described in Report #1 is unfixed. Runs are discarded when prompts exhaust retries on 429; 429 correlates with provider load; provider load is when a provider is most likely to shift behaviour. The surviving samples systematically exclude the periods most worth measuring.
The window is last-10-by-id, not time-bounded. Publishing the bounds makes its age visible; it does not change what it selects. The mistral leg above is the consequence.
The live saturation fraction is bounded, not measured. See above.
Why the mistral leg stopped on 2026-09-19 is not diagnosed. An earlier stoppage on that leg was an account-level rate limit. This one has not been investigated, and I am not assuming it is the same cause.
The 30-hour threshold is reasoned, not optimised. It was chosen from the measured scheduler lateness, not swept against a cost of false flags.
Overlapping windows. The last-10 window slides; two successive reads share rows, so a difference between them is not a difference between independent samples.

The board is at https://seismograph-weather.onrender.com/dashboard and
the probe is pip install seismograph-probe. If you run agents or
evals against a third-party API on a schedule, you already have the
data this network needs; a second observer is the only thing that turns
any of this into an alert anyone can act on.

I gave my drift monitor a denominator. The first thing it exposed was a hole in my own data collection

Tatiana Radchenko 🇩🇰 — Fri, 04 Sep 2026 18:23:38 +0000

SEISMOGRAPH is an open-source early-warning network for silent drift in
third-party LLM APIs -- the 2am question of whether the model changed
underneath you or your prompt is simply worse today. It runs a fixed
canary suite against provider endpoints on a schedule, ships only hashes
and differentially-private aggregates, and publishes a public board.

Two things about its current state, up front, because everything below
depends on them:

There is exactly one observer, and it is me. This is a one-person project, and that matters for reading everything below. A public drift alert requires cross-observer agreement from three independent observers. At one observer, no public alert can fire by construction. That is a property of the design, not a gap in it: a single organisation seeing a wobble is not evidence that a provider changed.
The live board's baseline was re-established on 2026-08-04. There is no continuous history before that date. Separately, on historical replay of a known incident, a seeded backtest flags it 38 days before the postmortem. That is a synthetic replay, not a live catch, and I do not describe it as one.

Last week I shipped a small read-side change: /v1/weather now
publishes, alongside each metric, the number of samples it rests on and
the time bounds of the window those samples came from. Publishing a rate
without its base is a defect; I had been doing it.

This is what the denominator found.

The board, read 2026-09-04

	google/gemini-3.5-flash-lite	mistral/mistral-small-latest
status	STABLE	STABLE
`sample_count`	10	10
`json_sample_count`	10	10
`length_sample_count`	10	10
`window_start`	2026-08-28T01:34:25Z	2026-08-28T17:27:23Z
`window_end`	2026-09-04T09:46:39Z	2026-09-02T09:38:39Z
window span	176.2 h	112.2 h
mean inter-sample interval	19.6 h	12.5 h

The probe is scheduled twice a day. Ten samples should therefore span
about 108 hours. The google leg's ten samples span 176.2 hours: it
is emitting at 61% of its own cadence. And the mistral leg's window
ends on 2026-09-02 -- as of the read at 2026-09-04 18:00 UTC that
row is 56 hours old, four scheduled slots ago. Both figures are
recomputable from the bounds in the table by anyone reading this.

Every count on that table reads 10. Both legs. The whole time.

Why the counts could never have found it

The probe runs a 50-prompt canary suite and flushes one aggregated row
per run. If any single prompt exhausts its retries, the runner discards
the entire suite: flushing at a reduced n would change the
differential-privacy sensitivity of the stream (MAX/n), so a partial
batch is not a smaller sample, it is a differently-calibrated one.

The consequence is that a lost run writes no row at all.

A class of loss that removes rows entirely is invisible to any count,
however many counts you publish. sample_count answers how many rows
survived into this metric. Only the window bounds answer how long it
took to collect them -- and it was the second question that found the
defect. I did not anticipate this when I scoped the change.

Then it moved legs

The interesting part is what happened between two reads 48 hours apart.

leg	2026-09-02	2026-09-04
google window span	196.2 h	176.2 h
google `avg_output_length`	131.060	130.398
mistral window span	112.2 h	112.2 h
mistral `avg_output_length`	89.42253	89.42253

google partially recovered. mistral froze -- byte-identical values,
byte-identical bounds, because not one new row arrived. The leg I was
about to describe as the clean one is the leg that is currently dark.

Both still publish 10 / 10 / 10.

What the run logs actually say

I reconciled the board against the scheduler for the window
2026-08-24T05:58Z to 2026-09-01T10:11Z:

16 scheduled runs fired.
9 succeeded, 7 failed, 0 were cancelled at the run level.
10 rows reached the board from the google leg.
So 6 runs produced no google row.

Ten rows from nine run-level successes is not a typo. A run is marked
failed if any provider leg fails, and a failed run can still carry a
successful google leg -- which is the first hint that run-level status is
a poor proxy for per-leg collection.

That kills the comfortable explanations. The scheduler ran. Nothing hit
a job timeout. The rows were lost inside runs that happened.

Per-run detail, from the emission logs:

run	leg	outcome
#118	google	`40/50 prompts completed`, 10 failed ids scattered mid-suite
#122	google	`49/50 prompts completed`, one failed id
#126	mistral	`0/50 prompts completed`, all fifty failed, job dead in 75 s

The emission policy is printed by the probe itself: pacing: 4500 ms between prompts, <= 2 retries on 429/503.

So the mechanism on the google leg is not a budget running out. It is
one prompt out of fifty losing three attempts to a transient 429, and
taking the other forty-nine with it. In run #122 the suite was 98%
complete and published nothing.

The mistral leg is a different failure entirely: zero completions in 75
seconds is not rate-limit turbulence, it is a hard condition at the
endpoint. I do not yet know which, and that is its own finding -- the
run log records the failed prompt ids and nothing else. No status code,
no response body, no retry spend. I built a drift detector whose own
collection failures are not diagnosable from its own logs. That gets
fixed before the collection logic does.

One more thing worth stating because it invalidates arithmetic I had
been doing: scheduled runs fire 2.5 to 4.5 hours after their cron
slot. Any reasoning that treats missing samples as exact multiples of
the 12-hour interval is unsound.

The part that matters: the surviving samples are biased, not just sparse

This is the finding I would most want a reader to take away, and it is
uncomfortable.

A run is discarded when prompts exhaust retries on 429. 429 correlates
with provider load. And provider load under stress is precisely the
condition under which a provider is most likely to shift behaviour --
it is the mechanism this project exists to detect.

So the samples that survive systematically exclude the periods most
worth measuring. The google leg's drift readings are computed over a
censored sample, censored by a variable correlated with the thing
being measured. This is not a footnote about data quality. It changes
what the number means.

For completeness: across the two reads neither leg shows a drift
signal. google's normalised JSON validity moved +0.0016 in raw units
(+0.12 sd of the per-batch DP noise) and its average output length
-0.66 characters (-0.33 sd); mistral moved by exactly zero because it
has no new data. Read those as "nothing to report from a sample I
have just told you is compromised", not as "the providers are stable".

Limitations, stated rather than buried

One observer. No public alert can fire. Everything above is single-organisation fleet data.
The censoring above. The google leg's numbers are not a random sample of provider behaviour.
Overlapping windows. The last-10 window slides; two successive reads share rows, so a difference between them is not a difference between independent samples. The sd figures are indicative.
The window is last-10-by-id, not time-bounded. Publishing the bounds makes its age visible; it does not change what it selects.
Counts do not say why a row was excluded. Enough for a denominator, not enough for a diagnosis.
No staleness signal. mistral's 56-hour-old row is visible on the board but not flagged, because "stale after N hours" is a judgement that needs its own defence. The timestamps are published; the reader concludes.
Publishing volume is not authenticated. Sample counts on the public payload are not yet quorum-gated, so forged volume would raise apparent support for an average. Open, tracked, undefended.

Why publish this

Because the honest version is more useful than the polished one, and
because the bottleneck is structural rather than technical: at one
observer this network is correct and silent. It becomes an early-warning
system at three.

If you run a canary suite against a provider API -- any provider, any
suite -- and would consider contributing observations, or just want to
tell me the collection design is wrong, the code and the board are open.

Engine: SEISMOGRAPH (Apache-2.0). Service: Drift Defense.

Board: https://seismograph-weather.onrender.com/dashboard
Repo: https://github.com/Tania-coder/SEISMOGRAPH
Probe: pip install seismograph-probe
DOI: https://doi.org/10.5281/zenodo.21045517

All figures above are measured, dated, and reproducible from the public
endpoint and the public Actions history at the time of writing.

Did the model get worse, or is it just you? How to tell when an LLM API silently changes

Tatiana Radchenko 🇩🇰 — Fri, 24 Jul 2026 12:30:34 +0000

You ship a feature on top of gpt-x or claude-y. It works. Three weeks later your users say the outputs feel worse — vaguer, sloppier, failing on the same prompts that used to pass. You didn't change your code. Did the model change, or are you imagining it?

This question eats an astonishing number of engineering hours, and it almost always dissolves into a vibes argument: half the thread says "it's clearly worse", the other half says "you have no evidence, it's confirmation bias." Both sides are right, because nobody has instrumented the one thing that would settle it. Here is how to settle it.

Why your gut is useless here (and so is a single comparison)

Two things move at once when a hosted model "feels worse":

The provider's side — a new checkpoint, a routing change, an inference optimization (batching, speculative decoding, quantization), or an infrastructure bug. You are not told when these happen. Often the provider doesn't announce it, and sometimes it isn't even a model change — it's an infra regression that only looks like the model got dumber.
Your side — a prompt tweak, a temperature change, a new library version, a different context length, or just which prompts you happened to hit today.

From a single account, on a handful of anecdotal prompts, these two are indistinguishable. That's the trap. You need two things to escape it: an objective, dated signal, and a way to tell "it changed for me" from "it changed for everyone."

Step 1 — Fix a canary suite

Pick a small, frozen set of prompts — 20 to 100 — that exercise the capabilities you actually depend on. Keep it small and cheap; this is a smoke detector, not a benchmark. Freeze it: same prompts, same parameters (temperature, top_p, max_tokens), same model string, forever. The moment you change the suite you lose your baseline.

For each canary, define a cheap scalar you can compute automatically:

JSON/schema validity rate (did it return parseable, valid output?)
Output length (a sudden shift often precedes a quality change)
A pass/fail against a fixed assertion (does the function it wrote compile?)
A rubric score from a cheap grader

You are not trying to measure "quality" in the abstract. You are trying to detect change. A boring, stable scalar is exactly what you want.

Step 2 — Record a baseline while things are good

Run the suite on a schedule — hourly, daily, whatever your budget allows — and store (timestamp, prompt_id, metric). Do this before you suspect anything. The baseline is the whole game: without "what normal looked like last month" you have nothing to compare today against. Estimate the mean mu0 and standard deviation sigma0 of your metric per prompt from a stable window.

Step 3 — Run a change-point test, not a threshold

A fixed threshold ("alert if success rate < 90%") is noisy and late. What you want is a sequential change-point detector that accumulates small, consistent shifts and fires with a timestamp. The classic is CUSUM (Page, 1954). It's a few lines:

def cusum_alarm(values, mu0, sigma0, k=0.5, h=5.0):
    """Two-sided CUSUM on standardized observations.
    k = slack (in std devs), h = alarm threshold.
    Returns the index where a sustained shift is detected, or None."""
    s_pos = s_neg = 0.0
    for i, x in enumerate(values):
        z = (x - mu0) / sigma0          # standardize against the baseline
        s_pos = max(0.0, s_pos + z - k) # catches upward shifts
        s_neg = max(0.0, s_neg - z - k) # catches downward shifts
        if s_pos > h or s_neg > h:
            return i                     # dated change-point
    return None

Feed it your per-prompt metric over time. Instead of "feels worse", you now get "json_success_rate on prompt #14 stepped down starting Tuesday" — a claim with a date on it that you can put in a ticket. (h=5.0, k=0.5 are reasonable starting points for standardized data; tune on your own history. Bayesian Online Change-Point Detection is a fancier alternative if you want posterior probabilities.)

This alone puts you ahead of ~95% of "the model got worse" threads, which never get past anecdotes.

Step 4 — The part one account can never answer: "is it just me?"

Here's the ceiling you hit: even with a perfect change-point signal, you still can't tell a provider-side change from your own drift, because from one seat they look identical. Your CUSUM fired — but was it the model, or the LlamaIndex bump you did last week, or a flaky region?

The only way to separate them is correlation across independent observers. If your canary fires on Tuesday, and several other organizations watching the same model on the same metric also fire around Tuesday, that agreement is the signal that it's provider-side — not your harness. One observer has N=1 forever; that's why these threads never converge. This is a genuinely different kind of detection: not "monitor my app" (every observability tool does that), but "did this change for everyone, privately confirmed across the fleet."

That cross-observer angle is what SEISMOGRAPH is built around: a lightweight probe runs your canary suite, a change-point detector flags shifts locally, and only privacy-preserving distributional features — never your prompts or outputs — are shared, so a public "model weather" alert only fires when multiple independent observers agree. A single org's signal stays private.

Does this actually work? A worked example

The method isn't hypothetical. Take the September 2025 Anthropic incident: for a stretch, claude-sonnet-4 degraded because of infrastructure bugs — notably a context-window routing error — not a model update (Anthropic said so explicitly in their postmortem). Exactly the "is it the model or the infra?" fog this whole article is about.

Replaying a canary suite over that period with the CUSUM method above: a seeded backtest flags it 38 days before the postmortem. Not a live catch — a reproducible backtest, seeded and open — but it shows the signal was sitting in cheap distributional metrics well before the official acknowledgement.

Do it yourself

Probe (open source, Apache-2.0): pip install seismograph-probe
Live "model weather" dashboard: https://driftdefense.dev?utm_source=devto&utm_medium=article&utm_campaign=evergreen_howto
Code + method: https://github.com/Tania-coder/SEISMOGRAPH

You don't need any of that to start — the four steps above work with a cron job, a CSV, and the ten lines of CUSUM. But if you want the "is it just me?" half answered, that needs more than one observer, and that's the part worth federating.

FAQ

Did GPT / Claude / Gemini actually get worse recently?
Sometimes yes, sometimes it's your setup, and often it's a provider-side infra change that isn't a "worse model" at all. The point of this article is that you shouldn't answer from vibes — instrument it and get a dated signal.

How can I prove a hosted model changed?
You can't prove it from one account, but you can get strong objective evidence: a frozen canary suite + a baseline + a change-point test gives you a dated shift. Provider-side vs. local drift is only separable with agreement across multiple independent observers.

Isn't this just LLM observability / evals?
Related but different. Evals score quality at a point in time; observability watches your traffic. This is about detecting change over time in a hosted dependency you don't control — and, uniquely, correlating that change across organizations to rule out "it's just me."

What metrics should I track?
Start with the cheapest ones that move when quality moves: schema/JSON validity, output length, and a single fixed pass/fail per canary. Add a rubric score later if you want resolution.

Your LLM didn't get worse. It changed — and nobody told you.

Tatiana Radchenko 🇩🇰 — Thu, 02 Jul 2026 11:22:37 +0000

It's 2am. Your JSON parse errors are up 12%. Latency: normal. Uptime: 100%. Your prompt didn't change. Your code didn't change.

Is it you, or did the model silently change underneath you?

Every team building on third-party LLM APIs eventually hits this moment. And the frustrating part: your monitoring stack can't answer the question. Latency, error rates, uptime — all green. The thing that changed is behavior, and behavior emits no infrastructure signal.

I built SEISMOGRAPH — an open-source, privacy-preserving early-warning network for exactly this failure mode. This post covers what silent drift is, how to detect it without leaking a single prompt, and a reproducible backtest where the detector flags a real, publicly documented provider incident 38 days before the official postmortem.

What "silent drift" actually is

Provider APIs do not broadcast behavioral changes. An endpoint that returns 200 can still start producing subtly different outputs: degraded JSON fidelity, shifted response-length distributions, changed reasoning patterns.

This isn't hypothetical. In September 2025, Anthropic published a detailed postmortem describing three infrastructure bugs that silently degraded Claude output quality across August–September 2025. The most long-lived one: a context-window routing error, introduced on August 5, that misrouted a fraction of Claude Sonnet 4 requests to servers configured for the 1M-token context window. It started at ~0.8% of traffic. On August 29 a load-balancer change escalated it to ~16%. The public postmortem landed on September 17.

Note what this incident was not: it was not a model update, and not intentional degradation — Anthropic was explicit about that. It was infrastructure. Which is precisely the point: the provider's own uptime and latency dashboards stayed green the whole time. The signal only existed at the semantic layer.

So the question I wanted to answer in Phase 0 of the project: would a lightweight behavioral canary have caught it, and how early?

The detection idea: canaries + change-point statistics

The approach is deliberately boring:

A canary suite — a fixed, content-addressed set of prompts (≤200, temperature 0) — runs on a schedule against the API you depend on. Deterministic prompts against a deterministic-ish endpoint give you a behavioral baseline. Every suite version is immutably hash-addressed, so a baseline can never silently mutate either.

Each response is reduced to distributional features: JSON parse success rate, output length, result counts. Not the text — the shape of the behavior.

Those features feed a Page-CUSUM change-point detector per (model_tuple, metric_name) stream:

S+(n) = max(0, S+(n-1) + z(n) - k)    # upward shifts
S-(n) = max(0, S-(n-1) - z(n) - k)    # downward shifts
Alert when S+ or S- > h

CUSUM accumulates small standardized deviations from a learned baseline (mu0, sigma0 from the first 30 observations). That's what makes it good at this job: a 0.8% fault doesn't produce a visible spike on any single day, but it produces a persistent small negative bias — and CUSUM integrates exactly that.

The privacy boundary (the part that makes federation possible)

One org's canary is a smoke detector. A network of canaries across independent orgs is an early-warning system — if orgs are willing to participate. They're only willing if participation leaks nothing.

So the probe enforces a hard perimeter. Raw prompts and outputs never leave your infrastructure. What gets transmitted:

SHA-256 hash of each response (not the response)
DP-noised aggregates: json_success_rate, avg_output_length (Laplace mechanism, ε = 2.0 per flush)
the content-addressed canary suite version hash
a pseudonymous Ed25519 public key — batches are signed, org identity is not disclosed

And the alerting side has a matching rule: a single organization's signal is never promoted to a public alert. A quorum scorer requires ≥ 2 independent orgs to agree on the same (model, metric) drift before anything goes public. That one gate filters probe bugs, network hiccups, and Sybil injection in a single move — and it means your private fleet data stays private fleet data.

The proof: a reproducible backtest

Here's the Phase 0 experiment (seeded, fully reproducible — SEED=42):

Reconstruct the Aug–Sep 2025 incident timeline from the public postmortem as synthetic daily probe traffic:

Phase	Dates	json_success_rate
Baseline	Jul 1 – Aug 4	0.990
Phase 1 (0.8% misrouting)	Aug 5 – Aug 28	0.982
Phase 2 (16% misrouting)	Aug 29 – Sep 17	0.840

Feed it through the exact CUSUM detector from the codebase (h=5.0, k=0.5). Watch when it fires:

Date        Phase          rate    S-      note
2025-08-03  Stable         0.9902  0.000
2025-08-04  Stable         0.9881  0.000
2025-08-05  Phase1(0.8%)   0.9855  0.598   [bug introduced]
2025-08-06  Phase1(0.8%)   0.9857  1.142
2025-08-07  Phase1(0.8%)   0.9786  3.309
2025-08-08  Phase1(0.8%)   0.9877  3.396
2025-08-09  Phase1(0.8%)   0.9816  4.889
2025-08-10  Phase1(0.8%)   0.9777  7.278   <<< FIRST ALERT

First alert: August 10 — five days into the subtle 0.8% window.

19 days before the escalation made the problem visible to users
38 days before the official postmortem

You can re-run this yourself: python scripts/anthropic_backtest.py in the repo regenerates the full report and asserts the result.

What this does and doesn't prove

I want to be precise here, because this is where drift-detection claims usually get slippery:

This is a backtest on synthetic data, not a live catch. The timeline and magnitudes come from the public postmortem; real probe noise may differ, and real lead time could be shorter or longer.
It's a single simulated observer. A real public alert requires the ≥2-org quorum.
The simulation omits live DP noise, which adds variance and may delay an alert by a few days.

What it does prove: a metric as cheap as "did the JSON parse?" — measured consistently, at temperature 0, against a fixed suite — carries enough signal to surface a 0.8%-of-traffic infrastructure fault weeks before it becomes user-visible. Conventional monitoring is structurally blind to this class of failure. A $0.10/day canary is not.

Architecture in one pass

your infra                          public
┌─────────────────────────┐
│ probe SDK               │
│  canary suite (≤200,    │
│  temp 0, hash-addressed)│
│  → features + DP noise  │
│  → Ed25519 signature    │
└──────────┬──────────────┘
           ▼
   ingestion gateway (FastAPI)
   schema validation, signature check
           ▼
   CUSUM per (model, metric)  →  local alert (private)
           ▼
   quorum scorer (≥2 orgs)    →  public drift alert
           ▼
   "model weather" dashboard

Stack: Python probe SDK (OpenTelemetry GenAI semantic conventions), FastAPI gateway, SQLAlchemy storage, CUSUM + Bayesian online change-point detection, 127 tests, ruff-clean CI. Apache-2.0.

Try it

The public "model weather" dashboard (no login) shows live drift status for four production models:

seismograph-weather.onrender.com/dashboard (free host — first load may take ~30s)

Run the probe against your own endpoint:

pip install seismograph-probe

Or clone the whole stack — gateway, dashboard, the federated quorum demo (scripts/demo_simulation.py shows two orgs independently confirming a drift and the public dashboard flipping only when quorum is met):

github.com/Tania-coder/SEISMOGRAPH

If you've been burned by a silent model change — I'd like to hear the story. That failure mode is exactly what this network exists to catch, and every additional independent observer makes the signal stronger.

SEISMOGRAPH is open source (Apache-2.0), archived at DOI 10.5281/zenodo.21045517. Built by Tatiana Radchenko.

SEISMOGRAPH now has a live public dashboard

Tatiana Radchenko 🇩🇰 — Wed, 24 Jun 2026 12:04:48 +0000

When I wrote about detecting silent LLM provider drift, the dashboard was localhost-only — you had to take my word for it. Now it's live and you can touch it:

👉 https://seismograph-weather.onrender.com/dashboard

It shows real-time "model weather" for 4 production models (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Claude 3 Haiku) — no login. The same engine flagged the Claude 3.5 Sonnet degradation 38 days before the official postmortem on a reproducible backtest.

Open-source, Apache-2.0, 107 tests.

pip install seismograph-probe

Repo: https://github.com/Tania-coder/SEISMOGRAPH — building in public, feedback welcome.

I've been building SEISMOGRAPH for 3 weeks. Here's what shipped today

Tatiana Radchenko 🇩🇰 — Tue, 23 Jun 2026 20:42:47 +0000

tl;dr: pip install seismograph-probe — a Python probe that detects silent LLM API drift using CUSUM change-point detection, with privacy-preserving signal aggregation. 103 tests passing. Dashboard live. Open source.

Three weeks ago I asked a question I couldn't answer:

"Did GPT-4 just change underneath me, or is it my prompt?"

No latency spike. No downtime. Just subtly different outputs from the same prompts, same parameters, same everything. I spent days debugging something that wasn't my fault.

So I built a detector.

Today I'm shipping it publicly.

What's actually working right now
This isn't a concept post. Here's what's live:
The probe SDK — on PyPI today
pip install seismograph-probe

from probe.sdk import ProbeSDK

sdk = ProbeSDK(provider="openai", model="gpt-4-turbo")

result = sdk.run_canary_suite()

print(result.drift_score) # 0.0 stable → 1.0 significant shift

The probe runs ≤200 canary prompts at temperature=0 daily. These are semantically stable tasks — deterministic questions, structured reasoning, format-adherence checks. The goal is a reliable behavioral baseline, not a capability benchmark.

Privacy boundary: raw prompts and model outputs never leave your machine. The probe extracts SHA-256 feature hashes, distributional stats, and DP-noised aggregates. That's all that transmits.
CUSUM change-point detection — running
The correlation engine uses CUSUM (Cumulative Sum) — a sequential statistical test that's sensitive to gradual drift, not just threshold crossings.

When I backtest against a known LLM behavioral shift event (Aug–Sep 2025):

Day 0: CUSUM statistic: 0.12 (stable baseline)

Day 11: First elevation detected

Day 19: Alert threshold crossed ← SEISMOGRAPH fires

Day 57: Public postmortem published

38-day lead time. That's the number I keep coming back to.
Ingestion gateway — deployed
FastAPI gateway with:

Ed25519-signed batch verification (unsigned batches rejected atomically)
Pydantic v2 schema validation
SQLAlchemy ORM + SQLite (ClickHouse migration planned for Phase 2)
Bearer token auth on audit export endpoint
Public dashboard — live at localhost, hosted version coming
Dark-mode model weather dashboard. Polls /v1/weather every 60 seconds. Shows per-model drift status across your fleet.

GET /v1/weather

→ [{ "model": "gpt-4-turbo", "status": "STABLE", ... },

{ "model": "claude-3-5-sonnet", "status": "STABLE", ... }]
Test suite — 103/103 passing
Not "it works on my machine." 103 tests across probe SDK, storage layer, gateway, CUSUM detector, privacy boundary, and auth. Zero ruff violations across 22 Python files.
Provider ToS compliance — checked
Before adding any provider to the canary suite, I verify it doesn't violate their Terms of Service. Done for: OpenAI ✅, Anthropic ✅, Google Gemini ✅, Mistral ✅, Cohere ✅. Documented in docs/PROVIDER_TOS_CHECKS.md.

What's NOT done yet (being honest)
No hosted gateway yet. The gateway runs locally. Public ingestion endpoint is Phase 1.
No Bayesian online detector yet. CUSUM is running. BayesianOnlineDetector.update() is deferred — it's on the backlog.
No federation yet. Right now it's single-org. The cross-observer agreement scoring that makes it genuinely valuable is Phase 2.
No cloud dashboard. localhost:8000 only for now.

This is Phase 0: I'm proving the detection logic works before scaling it.

The architecture in one diagram
Your app

│ (gen_ai.* OTel spans)

▼

ProbeSDK

│ SHA-256 hashes + DP-noised stats only

│ Ed25519-signed batch

▼

Ingestion Gateway (FastAPI)

│ signature check → schema validation → store

▼

SQLite / ClickHouse

│

▼

CUSUM Detector ──► DriftAlert

│

▼

/v1/weather dashboard

OTel-native throughout. If you're already emitting gen_ai.* spans, the adapter plugs straight in.

Why this matters (and why it has to be federated)
A single organization's drift signal is almost useless. Your outputs change because your users change. Your prompts change. Your context windows change.

But if 15 independent organizations running the same canary suite all see correlated semantic drift on the same day — that's a model change. That's the signal you can act on.

Single-org signal = private fleet data (yours only).
Multi-org correlated signal = public drift alert.

That's the design. Federation is Phase 2. The local probe is shippable today.

Try it / follow along
GitHub: github.com/Tania-coder/SEISMOGRAPH
PyPI: pypi.org/project/seismograph-probe

If you've been burned by a silent model change — I want to hear about it. Open an issue, or find me on Twitter @tatyanti.

The probe is Apache 2.0. The gateway will be too.

Tatiana Radchenko · AI Infrastructure · Aarhus, Denmark
Building in public. Phase 0 of 3.