Your LLM provider can quietly downgrade your model, and nothing will tell you

In October 2026, Chinese AI communities went through repeated waves of "degradation panic." Users on paid tiers discovered their models had been quietly swapped or throttled — no notice, no dashboard indicator, no changelog. People invented their own detection rituals: asking the model to draw a pelican riding a bicycle as SVG, candy-counting questions, "who is the current prime minister of Japan" fingerprint tests, even typing error messages into support chats to check for hidden account flags.

One-off fingerprint websites exist. But nobody was doing continuous canary monitoring with a real baseline and cost accounting. So I built model-canary: a zero-dependency Python watchdog that probes your endpoint on a schedule and tells you when something changed.

How it works

Four canary probes, each catching a different failure mode:

probe	kind	what it catches
identity-svg	identity	Pelican-on-a-bicycle SVG — the community gold standard. A silently swapped model draws a recognizably different picture.
candy-arithmetic	reasoning	"3 candies, take 1" — a degraded model fumbles even trivial reasoning.
instruction-repeat	instruction	Repeat "blue" five times — catches instruction-following decay.
factual-anchor	factual	Stable factual question — signal is drift against your baseline, not absolute correctness.

pip install model-canary

model-canary init --endpoint https://api.openai.com/v1 --model gpt-6.1-sol
model-canary run        # probes run, baseline written on first run
model-canary check      # exit code tells you the verdict
model-canary report     # markdown drift report

Put run + check on cron (hourly is plenty). check exits non-zero when something's wrong:

exit	verdict	meaning
0	ok / warn / baselining	fine, or still collecting baseline samples
2	degraded	high-confidence signal: model swapped, quality dropped, or account flagged
3	probe_failed	transport broke — never reported as degradation
4	throttled	provider rate-limited you — also not degradation

Every run logs token usage too, so report shows exactly what the watchdog itself costs. A monitor you can't afford to run is a monitor you'll turn off.

Design decisions that matter

Fail-open on alerting. A dead network is not a degraded model. Transport failures get their own verdict and exit code, and throttling gets its own too (exit 4). Only explicit account-level markers (like subscription_sharing_user_not_eligible) raise an account_flag. I match structured error.code/error.type fields, not substrings — matching "capacity" as a substring would flag half the internet's 503s.

No single-sample decisions. run --samples 5 (default) takes multiple samples; check looks at a window of recent runs and needs k-of-n confirmation. temperature=0 doesn't guarantee determinism on most providers, and some routers send a fraction of traffic to smaller models — one sample can't catch either case.

Baseline hygiene. The baseline needs n≥5 samples before it's trusted (before that, check reports baselining). Baselines carry model/endpoint/prompt-hash metadata — if you switch models, you get baseline_stale, not a false drift alarm. The representative sample is the medoid, not the last run. Rebuilding a baseline excludes runs that already showed drift, so a degraded period can't poison its own reference.

Atomic ledger. The ledger is append-only JSONL with O_APPEND + flock, single-write records, and the baseline is written to tmp + os.replace. A crashed or overlapping cron run can't corrupt either.

The review that caught real bugs

I had Claude do a full code review before publishing. It found three critical issues I'd missed, all false-positive factories:

1. Guaranteed fingerprint false positive. The baseline sample was truncated to 500 chars while run records kept 4000. With SequenceMatcher, any SVG longer than ~1320 chars mathematically could not score above the 0.55 threshold — every real pelican SVG would have been flagged as "model swapped." Fixed: identical normalization on both sides, autojunk=False, and a dynamic threshold derived from baseline sample similarity instead of a hardcoded 0.55.
2. Rate limits reported as degradation. The old code appended a HIGH error_signal whenever an error body contained "rate_limit" or "capacity" — a normal 429 would page you at 3am as "model degraded." Now throttling is its own verdict.
3. In-stream errors treated as success. Some relays return HTTP 200 with {"error": ...} inside the SSE stream. The old code read choices, got empty text, and reported quality/fingerprint/length drift. Now a stream error raises immediately, and a stream without [DONE] or with a non-stop finish reason is a probe failure, not a degradation.

All 21 tests pass, including regression tests for each of these.

Limitations, stated plainly

• Probes cost tokens every run. Hourly, not minutely.
• Fingerprint heuristics are statistical — alerts mean "go look," not "verdict."
• If a provider degrades everyone identically, the identity probe won't trip it. That's what the reasoning probes are for.
• Your API key is read from env and never written to disk.

MIT licensed. If you're paying for API access through any channel you don't fully control, a canary is cheaper than the alternative: finding out from your users.

Repo: https://github.com/hahahahahahahahah6/model-canary

Comments

[Max Quimby]

The fail-open design is the part I'd underline for anyone copying this pattern. Treating transport failures and throttling as their own verdicts rather than folding them into "degraded" is exactly right — the fastest way to get a canary ignored is to have it cry wolf every time the network hiccups, and matching structured error.code instead of substrings saves you from flagging half the 503s on earth.

One thing I've had to add in practice: providers also run silent A/B tests, so a single degraded window isn't always a permanent swap — sometimes you're just on the B arm for an hour. A requirement like "N consecutive degraded windows across M minutes before alerting" cut our false positives a lot without hiding real swaps. The other sharp edge is baseline drift: the identity-SVG probe is great, but models get legitimately updated, so you need a cheap "accept new baseline" path or your canary slowly turns into a nag. Have you thought about versioning the baseline so an intentional upgrade doesn't read as an incident?

[Zhang Enquan]

Worth adding a fifth probe, and it's the one no quality canary catches: token count on a fixed payload.

I measured a vocabulary swap from the other direction - same prompts, tokenizer changed underneath - and the median token count moved 0.54%, the batch 0.97%. Nothing a pelican SVG, a candy question or a factual anchor would flag. But the provider bills per token, so a silent swap that shifts counts by 1% shifts your bill by 1% while every quality probe stays green. Cost is the quietest degradation signal there is, and it's the only one that arrives as an invoice before it arrives as a bug report.

Second thing, and it applies to whichever probes you keep: grade per call, not per batch. My 10k run passed at the aggregate - 5% off - and failed at the row - 40.5% off by more than 20%. A canary that averages a week of probes will report the survivor, not the incident.

[Arhan Canli]

Your k-of-n rule is where I'd put the measurement. With a baseline of 5 samples and non-deterministic outputs, the alert rate on an endpoint that never changed is a number you can get: run it against a stable model for a week and count the warns and degraded verdicts. If that count isn't zero, it tells you how much to trust a real alert, and it sets k.

On the medoid: with 5 samples the medoid is one of five noisy answers, so the factual-anchor drift could be measured against the spread of the baseline samples (distance between baseline samples versus distance from the new run), not against the medoid alone.

Did you run it against an endpoint where you know the model didn't change, and what did the false-alarm count look like?

[IteraCompute]

Nice design. One more cheap check that pairs well with the canary: log the provider's declared serving precision and the price next to each run. On OpenRouter, for example, GLM-5.3 endpoints (checked 2026-10-27) declare fp8, fp4/nvfp4, or nothing at all, and the cheapest ones are fp4: around $0.07 input / $1.40 output per 1M tokens, versus $1.40 / $4.40 on Z.ai's own API. If a router quietly moves you from an fp8 host to an fp4 or "unknown" one, your bill drops and your canary scores may drift on the same day. With precision and $/1M in the run log, that shows up as one correlated change instead of a mystery regression.