DEV Community: Tatiana Radchenko 🇩🇰

I gave my drift monitor a denominator. The first thing it exposed was a hole in my own data collection

Tatiana Radchenko 🇩🇰 — Fri, 04 Sep 2026 18:23:38 +0000

SEISMOGRAPH is an open-source early-warning network for silent drift in
third-party LLM APIs -- the 2am question of whether the model changed
underneath you or your prompt is simply worse today. It runs a fixed
canary suite against provider endpoints on a schedule, ships only hashes
and differentially-private aggregates, and publishes a public board.

Two things about its current state, up front, because everything below
depends on them:

There is exactly one observer, and it is me. This is a one-person project, and that matters for reading everything below. A public drift alert requires cross-observer agreement from three independent observers. At one observer, no public alert can fire by construction. That is a property of the design, not a gap in it: a single organisation seeing a wobble is not evidence that a provider changed.
The live board's baseline was re-established on 2026-08-04. There is no continuous history before that date. Separately, on historical replay of a known incident, a seeded backtest flags it 38 days before the postmortem. That is a synthetic replay, not a live catch, and I do not describe it as one.

Last week I shipped a small read-side change: /v1/weather now
publishes, alongside each metric, the number of samples it rests on and
the time bounds of the window those samples came from. Publishing a rate
without its base is a defect; I had been doing it.

This is what the denominator found.

The board, read 2026-09-04

	google/gemini-3.5-flash-lite	mistral/mistral-small-latest
status	STABLE	STABLE
`sample_count`	10	10
`json_sample_count`	10	10
`length_sample_count`	10	10
`window_start`	2026-08-28T01:34:25Z	2026-08-28T17:27:23Z
`window_end`	2026-09-04T09:46:39Z	2026-09-02T09:38:39Z
window span	176.2 h	112.2 h
mean inter-sample interval	19.6 h	12.5 h

The probe is scheduled twice a day. Ten samples should therefore span
about 108 hours. The google leg's ten samples span 176.2 hours: it
is emitting at 61% of its own cadence. And the mistral leg's window
ends on 2026-09-02 -- as of the read at 2026-09-04 18:00 UTC that
row is 56 hours old, four scheduled slots ago. Both figures are
recomputable from the bounds in the table by anyone reading this.

Every count on that table reads 10. Both legs. The whole time.

Why the counts could never have found it

The probe runs a 50-prompt canary suite and flushes one aggregated row
per run. If any single prompt exhausts its retries, the runner discards
the entire suite: flushing at a reduced n would change the
differential-privacy sensitivity of the stream (MAX/n), so a partial
batch is not a smaller sample, it is a differently-calibrated one.

The consequence is that a lost run writes no row at all.

A class of loss that removes rows entirely is invisible to any count,
however many counts you publish. sample_count answers how many rows
survived into this metric. Only the window bounds answer how long it
took to collect them -- and it was the second question that found the
defect. I did not anticipate this when I scoped the change.

Then it moved legs

The interesting part is what happened between two reads 48 hours apart.

leg	2026-09-02	2026-09-04
google window span	196.2 h	176.2 h
google `avg_output_length`	131.060	130.398
mistral window span	112.2 h	112.2 h
mistral `avg_output_length`	89.42253	89.42253

google partially recovered. mistral froze -- byte-identical values,
byte-identical bounds, because not one new row arrived. The leg I was
about to describe as the clean one is the leg that is currently dark.

Both still publish 10 / 10 / 10.

What the run logs actually say

I reconciled the board against the scheduler for the window
2026-08-24T05:58Z to 2026-09-01T10:11Z:

16 scheduled runs fired.
9 succeeded, 7 failed, 0 were cancelled at the run level.
10 rows reached the board from the google leg.
So 6 runs produced no google row.

Ten rows from nine run-level successes is not a typo. A run is marked
failed if any provider leg fails, and a failed run can still carry a
successful google leg -- which is the first hint that run-level status is
a poor proxy for per-leg collection.

That kills the comfortable explanations. The scheduler ran. Nothing hit
a job timeout. The rows were lost inside runs that happened.

Per-run detail, from the emission logs:

run	leg	outcome
#118	google	`40/50 prompts completed`, 10 failed ids scattered mid-suite
#122	google	`49/50 prompts completed`, one failed id
#126	mistral	`0/50 prompts completed`, all fifty failed, job dead in 75 s

The emission policy is printed by the probe itself: pacing: 4500 ms between prompts, <= 2 retries on 429/503.

So the mechanism on the google leg is not a budget running out. It is
one prompt out of fifty losing three attempts to a transient 429, and
taking the other forty-nine with it. In run #122 the suite was 98%
complete and published nothing.

The mistral leg is a different failure entirely: zero completions in 75
seconds is not rate-limit turbulence, it is a hard condition at the
endpoint. I do not yet know which, and that is its own finding -- the
run log records the failed prompt ids and nothing else. No status code,
no response body, no retry spend. I built a drift detector whose own
collection failures are not diagnosable from its own logs. That gets
fixed before the collection logic does.

One more thing worth stating because it invalidates arithmetic I had
been doing: scheduled runs fire 2.5 to 4.5 hours after their cron
slot. Any reasoning that treats missing samples as exact multiples of
the 12-hour interval is unsound.

The part that matters: the surviving samples are biased, not just sparse

This is the finding I would most want a reader to take away, and it is
uncomfortable.

A run is discarded when prompts exhaust retries on 429. 429 correlates
with provider load. And provider load under stress is precisely the
condition under which a provider is most likely to shift behaviour --
it is the mechanism this project exists to detect.

So the samples that survive systematically exclude the periods most
worth measuring. The google leg's drift readings are computed over a
censored sample, censored by a variable correlated with the thing
being measured. This is not a footnote about data quality. It changes
what the number means.

For completeness: across the two reads neither leg shows a drift
signal. google's normalised JSON validity moved +0.0016 in raw units
(+0.12 sd of the per-batch DP noise) and its average output length
-0.66 characters (-0.33 sd); mistral moved by exactly zero because it
has no new data. Read those as "nothing to report from a sample I
have just told you is compromised", not as "the providers are stable".

Limitations, stated rather than buried

One observer. No public alert can fire. Everything above is single-organisation fleet data.
The censoring above. The google leg's numbers are not a random sample of provider behaviour.
Overlapping windows. The last-10 window slides; two successive reads share rows, so a difference between them is not a difference between independent samples. The sd figures are indicative.
The window is last-10-by-id, not time-bounded. Publishing the bounds makes its age visible; it does not change what it selects.
Counts do not say why a row was excluded. Enough for a denominator, not enough for a diagnosis.
No staleness signal. mistral's 56-hour-old row is visible on the board but not flagged, because "stale after N hours" is a judgement that needs its own defence. The timestamps are published; the reader concludes.
Publishing volume is not authenticated. Sample counts on the public payload are not yet quorum-gated, so forged volume would raise apparent support for an average. Open, tracked, undefended.

Why publish this

Because the honest version is more useful than the polished one, and
because the bottleneck is structural rather than technical: at one
observer this network is correct and silent. It becomes an early-warning
system at three.

If you run a canary suite against a provider API -- any provider, any
suite -- and would consider contributing observations, or just want to
tell me the collection design is wrong, the code and the board are open.

Engine: SEISMOGRAPH (Apache-2.0). Service: Drift Defense.

Board: https://seismograph-weather.onrender.com/dashboard
Repo: https://github.com/Tania-coder/SEISMOGRAPH
Probe: pip install seismograph-probe
DOI: https://doi.org/10.5281/zenodo.21045517

All figures above are measured, dated, and reproducible from the public
endpoint and the public Actions history at the time of writing.

Did the model get worse, or is it just you? How to tell when an LLM API silently changes

Tatiana Radchenko 🇩🇰 — Fri, 24 Jul 2026 12:30:34 +0000

You ship a feature on top of gpt-x or claude-y. It works. Three weeks later your users say the outputs feel worse — vaguer, sloppier, failing on the same prompts that used to pass. You didn't change your code. Did the model change, or are you imagining it?

This question eats an astonishing number of engineering hours, and it almost always dissolves into a vibes argument: half the thread says "it's clearly worse", the other half says "you have no evidence, it's confirmation bias." Both sides are right, because nobody has instrumented the one thing that would settle it. Here is how to settle it.

Why your gut is useless here (and so is a single comparison)

Two things move at once when a hosted model "feels worse":

The provider's side — a new checkpoint, a routing change, an inference optimization (batching, speculative decoding, quantization), or an infrastructure bug. You are not told when these happen. Often the provider doesn't announce it, and sometimes it isn't even a model change — it's an infra regression that only looks like the model got dumber.
Your side — a prompt tweak, a temperature change, a new library version, a different context length, or just which prompts you happened to hit today.

From a single account, on a handful of anecdotal prompts, these two are indistinguishable. That's the trap. You need two things to escape it: an objective, dated signal, and a way to tell "it changed for me" from "it changed for everyone."

Step 1 — Fix a canary suite

Pick a small, frozen set of prompts — 20 to 100 — that exercise the capabilities you actually depend on. Keep it small and cheap; this is a smoke detector, not a benchmark. Freeze it: same prompts, same parameters (temperature, top_p, max_tokens), same model string, forever. The moment you change the suite you lose your baseline.

For each canary, define a cheap scalar you can compute automatically:

JSON/schema validity rate (did it return parseable, valid output?)
Output length (a sudden shift often precedes a quality change)
A pass/fail against a fixed assertion (does the function it wrote compile?)
A rubric score from a cheap grader

You are not trying to measure "quality" in the abstract. You are trying to detect change. A boring, stable scalar is exactly what you want.

Step 2 — Record a baseline while things are good

Run the suite on a schedule — hourly, daily, whatever your budget allows — and store (timestamp, prompt_id, metric). Do this before you suspect anything. The baseline is the whole game: without "what normal looked like last month" you have nothing to compare today against. Estimate the mean mu0 and standard deviation sigma0 of your metric per prompt from a stable window.

Step 3 — Run a change-point test, not a threshold

A fixed threshold ("alert if success rate < 90%") is noisy and late. What you want is a sequential change-point detector that accumulates small, consistent shifts and fires with a timestamp. The classic is CUSUM (Page, 1954). It's a few lines:

def cusum_alarm(values, mu0, sigma0, k=0.5, h=5.0):
    """Two-sided CUSUM on standardized observations.
    k = slack (in std devs), h = alarm threshold.
    Returns the index where a sustained shift is detected, or None."""
    s_pos = s_neg = 0.0
    for i, x in enumerate(values):
        z = (x - mu0) / sigma0          # standardize against the baseline
        s_pos = max(0.0, s_pos + z - k) # catches upward shifts
        s_neg = max(0.0, s_neg - z - k) # catches downward shifts
        if s_pos > h or s_neg > h:
            return i                     # dated change-point
    return None

Feed it your per-prompt metric over time. Instead of "feels worse", you now get "json_success_rate on prompt #14 stepped down starting Tuesday" — a claim with a date on it that you can put in a ticket. (h=5.0, k=0.5 are reasonable starting points for standardized data; tune on your own history. Bayesian Online Change-Point Detection is a fancier alternative if you want posterior probabilities.)

This alone puts you ahead of ~95% of "the model got worse" threads, which never get past anecdotes.

Step 4 — The part one account can never answer: "is it just me?"

Here's the ceiling you hit: even with a perfect change-point signal, you still can't tell a provider-side change from your own drift, because from one seat they look identical. Your CUSUM fired — but was it the model, or the LlamaIndex bump you did last week, or a flaky region?

The only way to separate them is correlation across independent observers. If your canary fires on Tuesday, and several other organizations watching the same model on the same metric also fire around Tuesday, that agreement is the signal that it's provider-side — not your harness. One observer has N=1 forever; that's why these threads never converge. This is a genuinely different kind of detection: not "monitor my app" (every observability tool does that), but "did this change for everyone, privately confirmed across the fleet."

That cross-observer angle is what SEISMOGRAPH is built around: a lightweight probe runs your canary suite, a change-point detector flags shifts locally, and only privacy-preserving distributional features — never your prompts or outputs — are shared, so a public "model weather" alert only fires when multiple independent observers agree. A single org's signal stays private.

Does this actually work? A worked example

The method isn't hypothetical. Take the September 2025 Anthropic incident: for a stretch, claude-sonnet-4 degraded because of infrastructure bugs — notably a context-window routing error — not a model update (Anthropic said so explicitly in their postmortem). Exactly the "is it the model or the infra?" fog this whole article is about.

Replaying a canary suite over that period with the CUSUM method above: a seeded backtest flags it 38 days before the postmortem. Not a live catch — a reproducible backtest, seeded and open — but it shows the signal was sitting in cheap distributional metrics well before the official acknowledgement.

Do it yourself

Probe (open source, Apache-2.0): pip install seismograph-probe
Live "model weather" dashboard: https://driftdefense.dev?utm_source=devto&utm_medium=article&utm_campaign=evergreen_howto
Code + method: https://github.com/Tania-coder/SEISMOGRAPH

You don't need any of that to start — the four steps above work with a cron job, a CSV, and the ten lines of CUSUM. But if you want the "is it just me?" half answered, that needs more than one observer, and that's the part worth federating.

FAQ

Did GPT / Claude / Gemini actually get worse recently?
Sometimes yes, sometimes it's your setup, and often it's a provider-side infra change that isn't a "worse model" at all. The point of this article is that you shouldn't answer from vibes — instrument it and get a dated signal.

How can I prove a hosted model changed?
You can't prove it from one account, but you can get strong objective evidence: a frozen canary suite + a baseline + a change-point test gives you a dated shift. Provider-side vs. local drift is only separable with agreement across multiple independent observers.

Isn't this just LLM observability / evals?
Related but different. Evals score quality at a point in time; observability watches your traffic. This is about detecting change over time in a hosted dependency you don't control — and, uniquely, correlating that change across organizations to rule out "it's just me."

What metrics should I track?
Start with the cheapest ones that move when quality moves: schema/JSON validity, output length, and a single fixed pass/fail per canary. Add a rubric score later if you want resolution.

Your LLM didn't get worse. It changed — and nobody told you.

Tatiana Radchenko 🇩🇰 — Thu, 02 Jul 2026 11:22:37 +0000

It's 2am. Your JSON parse errors are up 12%. Latency: normal. Uptime: 100%. Your prompt didn't change. Your code didn't change.

Is it you, or did the model silently change underneath you?

Every team building on third-party LLM APIs eventually hits this moment. And the frustrating part: your monitoring stack can't answer the question. Latency, error rates, uptime — all green. The thing that changed is behavior, and behavior emits no infrastructure signal.

I built SEISMOGRAPH — an open-source, privacy-preserving early-warning network for exactly this failure mode. This post covers what silent drift is, how to detect it without leaking a single prompt, and a reproducible backtest where the detector flags a real, publicly documented provider incident 38 days before the official postmortem.

What "silent drift" actually is

Provider APIs do not broadcast behavioral changes. An endpoint that returns 200 can still start producing subtly different outputs: degraded JSON fidelity, shifted response-length distributions, changed reasoning patterns.

This isn't hypothetical. In September 2025, Anthropic published a detailed postmortem describing three infrastructure bugs that silently degraded Claude output quality across August–September 2025. The most long-lived one: a context-window routing error, introduced on August 5, that misrouted a fraction of Claude Sonnet 4 requests to servers configured for the 1M-token context window. It started at ~0.8% of traffic. On August 29 a load-balancer change escalated it to ~16%. The public postmortem landed on September 17.

Note what this incident was not: it was not a model update, and not intentional degradation — Anthropic was explicit about that. It was infrastructure. Which is precisely the point: the provider's own uptime and latency dashboards stayed green the whole time. The signal only existed at the semantic layer.

So the question I wanted to answer in Phase 0 of the project: would a lightweight behavioral canary have caught it, and how early?

The detection idea: canaries + change-point statistics

The approach is deliberately boring:

A canary suite — a fixed, content-addressed set of prompts (≤200, temperature 0) — runs on a schedule against the API you depend on. Deterministic prompts against a deterministic-ish endpoint give you a behavioral baseline. Every suite version is immutably hash-addressed, so a baseline can never silently mutate either.

Each response is reduced to distributional features: JSON parse success rate, output length, result counts. Not the text — the shape of the behavior.

Those features feed a Page-CUSUM change-point detector per (model_tuple, metric_name) stream:

S+(n) = max(0, S+(n-1) + z(n) - k)    # upward shifts
S-(n) = max(0, S-(n-1) - z(n) - k)    # downward shifts
Alert when S+ or S- > h

CUSUM accumulates small standardized deviations from a learned baseline (mu0, sigma0 from the first 30 observations). That's what makes it good at this job: a 0.8% fault doesn't produce a visible spike on any single day, but it produces a persistent small negative bias — and CUSUM integrates exactly that.

The privacy boundary (the part that makes federation possible)

One org's canary is a smoke detector. A network of canaries across independent orgs is an early-warning system — if orgs are willing to participate. They're only willing if participation leaks nothing.

So the probe enforces a hard perimeter. Raw prompts and outputs never leave your infrastructure. What gets transmitted:

SHA-256 hash of each response (not the response)
DP-noised aggregates: json_success_rate, avg_output_length (Laplace mechanism, ε = 2.0 per flush)
the content-addressed canary suite version hash
a pseudonymous Ed25519 public key — batches are signed, org identity is not disclosed

And the alerting side has a matching rule: a single organization's signal is never promoted to a public alert. A quorum scorer requires ≥ 2 independent orgs to agree on the same (model, metric) drift before anything goes public. That one gate filters probe bugs, network hiccups, and Sybil injection in a single move — and it means your private fleet data stays private fleet data.

The proof: a reproducible backtest

Here's the Phase 0 experiment (seeded, fully reproducible — SEED=42):

Reconstruct the Aug–Sep 2025 incident timeline from the public postmortem as synthetic daily probe traffic:

Phase	Dates	json_success_rate
Baseline	Jul 1 – Aug 4	0.990
Phase 1 (0.8% misrouting)	Aug 5 – Aug 28	0.982
Phase 2 (16% misrouting)	Aug 29 – Sep 17	0.840

Feed it through the exact CUSUM detector from the codebase (h=5.0, k=0.5). Watch when it fires:

Date        Phase          rate    S-      note
2025-08-03  Stable         0.9902  0.000
2025-08-04  Stable         0.9881  0.000
2025-08-05  Phase1(0.8%)   0.9855  0.598   [bug introduced]
2025-08-06  Phase1(0.8%)   0.9857  1.142
2025-08-07  Phase1(0.8%)   0.9786  3.309
2025-08-08  Phase1(0.8%)   0.9877  3.396
2025-08-09  Phase1(0.8%)   0.9816  4.889
2025-08-10  Phase1(0.8%)   0.9777  7.278   <<< FIRST ALERT

First alert: August 10 — five days into the subtle 0.8% window.

19 days before the escalation made the problem visible to users
38 days before the official postmortem

You can re-run this yourself: python scripts/anthropic_backtest.py in the repo regenerates the full report and asserts the result.

What this does and doesn't prove

I want to be precise here, because this is where drift-detection claims usually get slippery:

This is a backtest on synthetic data, not a live catch. The timeline and magnitudes come from the public postmortem; real probe noise may differ, and real lead time could be shorter or longer.
It's a single simulated observer. A real public alert requires the ≥2-org quorum.
The simulation omits live DP noise, which adds variance and may delay an alert by a few days.

What it does prove: a metric as cheap as "did the JSON parse?" — measured consistently, at temperature 0, against a fixed suite — carries enough signal to surface a 0.8%-of-traffic infrastructure fault weeks before it becomes user-visible. Conventional monitoring is structurally blind to this class of failure. A $0.10/day canary is not.

Architecture in one pass

your infra                          public
┌─────────────────────────┐
│ probe SDK               │
│  canary suite (≤200,    │
│  temp 0, hash-addressed)│
│  → features + DP noise  │
│  → Ed25519 signature    │
└──────────┬──────────────┘
           ▼
   ingestion gateway (FastAPI)
   schema validation, signature check
           ▼
   CUSUM per (model, metric)  →  local alert (private)
           ▼
   quorum scorer (≥2 orgs)    →  public drift alert
           ▼
   "model weather" dashboard

Stack: Python probe SDK (OpenTelemetry GenAI semantic conventions), FastAPI gateway, SQLAlchemy storage, CUSUM + Bayesian online change-point detection, 127 tests, ruff-clean CI. Apache-2.0.

Try it

The public "model weather" dashboard (no login) shows live drift status for four production models:

seismograph-weather.onrender.com/dashboard (free host — first load may take ~30s)

Run the probe against your own endpoint:

pip install seismograph-probe

Or clone the whole stack — gateway, dashboard, the federated quorum demo (scripts/demo_simulation.py shows two orgs independently confirming a drift and the public dashboard flipping only when quorum is met):

github.com/Tania-coder/SEISMOGRAPH

If you've been burned by a silent model change — I'd like to hear the story. That failure mode is exactly what this network exists to catch, and every additional independent observer makes the signal stronger.

SEISMOGRAPH is open source (Apache-2.0), archived at DOI 10.5281/zenodo.21045517. Built by Tatiana Radchenko.

SEISMOGRAPH now has a live public dashboard

Tatiana Radchenko 🇩🇰 — Wed, 24 Jun 2026 12:04:48 +0000

When I wrote about detecting silent LLM provider drift, the dashboard was localhost-only — you had to take my word for it. Now it's live and you can touch it:

👉 https://seismograph-weather.onrender.com/dashboard

It shows real-time "model weather" for 4 production models (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Claude 3 Haiku) — no login. The same engine flagged the Claude 3.5 Sonnet degradation 38 days before the official postmortem on a reproducible backtest.

Open-source, Apache-2.0, 107 tests.

pip install seismograph-probe

Repo: https://github.com/Tania-coder/SEISMOGRAPH — building in public, feedback welcome.

I've been building SEISMOGRAPH for 3 weeks. Here's what shipped today

Tatiana Radchenko 🇩🇰 — Tue, 23 Jun 2026 20:42:47 +0000

tl;dr: pip install seismograph-probe — a Python probe that detects silent LLM API drift using CUSUM change-point detection, with privacy-preserving signal aggregation. 103 tests passing. Dashboard live. Open source.

Three weeks ago I asked a question I couldn't answer:

"Did GPT-4 just change underneath me, or is it my prompt?"

No latency spike. No downtime. Just subtly different outputs from the same prompts, same parameters, same everything. I spent days debugging something that wasn't my fault.

So I built a detector.

Today I'm shipping it publicly.

What's actually working right now
This isn't a concept post. Here's what's live:
The probe SDK — on PyPI today
pip install seismograph-probe

from probe.sdk import ProbeSDK

sdk = ProbeSDK(provider="openai", model="gpt-4-turbo")

result = sdk.run_canary_suite()

print(result.drift_score) # 0.0 stable → 1.0 significant shift

The probe runs ≤200 canary prompts at temperature=0 daily. These are semantically stable tasks — deterministic questions, structured reasoning, format-adherence checks. The goal is a reliable behavioral baseline, not a capability benchmark.

Privacy boundary: raw prompts and model outputs never leave your machine. The probe extracts SHA-256 feature hashes, distributional stats, and DP-noised aggregates. That's all that transmits.
CUSUM change-point detection — running
The correlation engine uses CUSUM (Cumulative Sum) — a sequential statistical test that's sensitive to gradual drift, not just threshold crossings.

When I backtest against a known LLM behavioral shift event (Aug–Sep 2025):

Day 0: CUSUM statistic: 0.12 (stable baseline)

Day 11: First elevation detected

Day 19: Alert threshold crossed ← SEISMOGRAPH fires

Day 57: Public postmortem published

38-day lead time. That's the number I keep coming back to.
Ingestion gateway — deployed
FastAPI gateway with:

Ed25519-signed batch verification (unsigned batches rejected atomically)
Pydantic v2 schema validation
SQLAlchemy ORM + SQLite (ClickHouse migration planned for Phase 2)
Bearer token auth on audit export endpoint
Public dashboard — live at localhost, hosted version coming
Dark-mode model weather dashboard. Polls /v1/weather every 60 seconds. Shows per-model drift status across your fleet.

GET /v1/weather

→ [{ "model": "gpt-4-turbo", "status": "STABLE", ... },

{ "model": "claude-3-5-sonnet", "status": "STABLE", ... }]
Test suite — 103/103 passing
Not "it works on my machine." 103 tests across probe SDK, storage layer, gateway, CUSUM detector, privacy boundary, and auth. Zero ruff violations across 22 Python files.
Provider ToS compliance — checked
Before adding any provider to the canary suite, I verify it doesn't violate their Terms of Service. Done for: OpenAI ✅, Anthropic ✅, Google Gemini ✅, Mistral ✅, Cohere ✅. Documented in docs/PROVIDER_TOS_CHECKS.md.

What's NOT done yet (being honest)
No hosted gateway yet. The gateway runs locally. Public ingestion endpoint is Phase 1.
No Bayesian online detector yet. CUSUM is running. BayesianOnlineDetector.update() is deferred — it's on the backlog.
No federation yet. Right now it's single-org. The cross-observer agreement scoring that makes it genuinely valuable is Phase 2.
No cloud dashboard. localhost:8000 only for now.

This is Phase 0: I'm proving the detection logic works before scaling it.

The architecture in one diagram
Your app

│ (gen_ai.* OTel spans)

▼

ProbeSDK

│ SHA-256 hashes + DP-noised stats only

│ Ed25519-signed batch

▼

Ingestion Gateway (FastAPI)

│ signature check → schema validation → store

▼

SQLite / ClickHouse

│

▼

CUSUM Detector ──► DriftAlert

│

▼

/v1/weather dashboard

OTel-native throughout. If you're already emitting gen_ai.* spans, the adapter plugs straight in.

Why this matters (and why it has to be federated)
A single organization's drift signal is almost useless. Your outputs change because your users change. Your prompts change. Your context windows change.

But if 15 independent organizations running the same canary suite all see correlated semantic drift on the same day — that's a model change. That's the signal you can act on.

Single-org signal = private fleet data (yours only).
Multi-org correlated signal = public drift alert.

That's the design. Federation is Phase 2. The local probe is shippable today.

Try it / follow along
GitHub: github.com/Tania-coder/SEISMOGRAPH
PyPI: pypi.org/project/seismograph-probe

If you've been burned by a silent model change — I want to hear about it. Open an issue, or find me on Twitter @tatyanti.

The probe is Apache 2.0. The gateway will be too.

Tatiana Radchenko · AI Infrastructure · Aarhus, Denmark
Building in public. Phase 0 of 3.