Catch Silent LLM Drift with a 100-Line Probe

A free LLM can return 200 OK while the model behind it has already changed—no changelog, no error, just worse labels. I catch that silent swap with a ~100-line drift probe: fixed inputs, temperature 0, and exact-output logging, which is what flagged an accuracy drop from 92% to 78% when no code or prompt had changed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why free LLM endpoints drift without a changelog

Free model endpoints typically have no SLA on behavior. The provider can:

• Swap the underlying model version without notice
• Route traffic to different hardware (quantized vs. full precision)
• Change default sampling parameters
• Load-balance across multiple model snapshots

None of these show up in a health check. All of them change output. This is closer to concept drift than to downtime: the endpoint is up; the decision surface inside it is different.

I hit this on a small GitHub-issue classifier. It maps comments into four labels—bug, thanks, feature, other—runs 20–50 comments per day at under 500 tokens each, and uses a free tier (MonkeyCode's free model access). It posts label suggestions to a dashboard, not a customer-facing SLA. One Tuesday, a job that had sat at 92% accurate for three weeks fell to 78%. I had changed nothing. Wrong labels filled the dashboard within a day.

OpenAI's production guidance covers uptime, retries, and evals. It still will not tell you a free endpoint swapped the weights. A health check asks "is the endpoint up?" I needed a probe that asked "is the model behaving like last week?"

Build a drift probe with fixed inputs and temperature 0

The idea is simple. Keep a fixed set of inputs with known expected labels, run it on a schedule, and record three things each run: accuracy against those labels, the output distribution, and the exact strings for later diffs.

I use ten short, unambiguous comments that map cleanly to one class:

# drift_probe.py
import json
import os
import urllib.request
from datetime import date

PROBES = [
    ("The build failed again.", "bug"),
    ("Thanks for the quick fix!", "thanks"),
    ("Can you add pagination?", "feature"),
    ("This is not related to the issue.", "other"),
    ("The error message is misleading.", "bug"),
    ("Great work on the release!", "thanks"),
    ("Please support Python 3.12.", "feature"),
    ("Just bumping this thread.", "other"),
    ("The test suite is flaky.", "bug"),
    ("Appreciate the detailed response.", "thanks"),
]

The call hits an OpenAI-compatible /chat/completions body. I set temperature to 0 so sampling noise does not masquerade as drift. If the model is stable, outputs should be nearly identical run to run.

def call(base_url, api_key, model, messages):
    body = json.dumps({
        "model": model,
        "messages": messages,
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{base_url}/chat/completions",
        data=body,
        headers={
            "content-type": "application/json",
            "authorization": f"Bearer {api_key}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        data = json.loads(resp.read())
    return data["choices"][0]["message"]["content"].strip().lower()

def main():
    base_url = os.environ["LLM_BASE_URL"]
    api_key = os.environ["LLM_API_KEY"]
    model = os.environ["LLM_MODEL"]

    outputs = []
    correct = 0
    for text, expected in PROBES:
        actual = call(base_url, api_key, model, [
            {"role": "system", "content": "Classify the text. Reply with exactly one word: bug, thanks, feature, or other."},
            {"role": "user", "content": text},
        ])
        outputs.append({"text": text, "expected": expected, "actual": actual})
        if expected in actual:
            correct += 1

    accuracy = correct / len(PROBES)
    record = {
        "date": date.today().isoformat(),
        "model": model,
        "accuracy": accuracy,
        "outputs": outputs,
    }

    with open("drift_history.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

    print(f"{date.today().isoformat()} accuracy={accuracy:.2f}")

if __name__ == "__main__":
    main()

Compare that to a typical uptime ping:

Check	Question it answers	What it misses
Health / 200 OK	Is the API reachable?	Model swap, quantization, sampling change
Latency / rate-limit	Is it fast and allowed?	Wrong labels at the same speed
Drift probe	Is output still the same?	Patterns not in the probe set

I schedule it daily (cron or CI). Cost is a few hundred tokens per day. History appends to drift_history.jsonl.

What the probe caught: accuracy, distribution, and diffs

A month of daily runs produced a clear signal:

Week	Accuracy	Notable output changes
1	0.90	Baseline
2	0.90	None
3	0.80	"This is not related to the issue." → "other" became "feature"
4	0.90	Back to baseline

Week 3 was real. The model briefly tagged the "other" probe as "feature", which in production meant unrelated comments got the feature label. The dip lasted about four days, then silently corrected itself. Without the probe I would have spent hours debugging my code. With it, one file showed the model's output had changed, not the pipeline.

Accuracy is the bluntest signal. Two others caught issues accuracy can miss.

Output distribution. If the model classifies everything as "bug", accuracy on the "bug" probes stays high while production becomes useless. I store the full output set so I can count labels:

from collections import Counter

def distribution(record):
    return Counter(r["actual"] for r in record["outputs"])

A healthy run spreads across all four labels. A collapsed distribution—90% one label—is a drift signal even when accuracy looks fine.

Exact output diffing. When accuracy drops, I want the specific input that moved. Because the probe stores exact strings, I can compare two dates:

import json

with open("drift_history.jsonl") as f:
    records = [json.loads(line) for line in f]

by_date = {r["date"]: r for r in records}
w1 = {o["text"]: o["actual"] for o in by_date["2026-08-01"]["outputs"]}
w3 = {o["text"]: o["actual"] for o in by_date["2026-08-15"]["outputs"]}

for text, actual_w1 in w1.items():
    actual_w3 = w3.get(text)
    if actual_w1 != actual_w3:
        print(f"{text!r}: {actual_w1} -> {actual_w3}")

That turns "the model feels different" into a date, an accuracy number, and a one-line diff.

Respond when it fires—and know the limits

Drift detection without a response plan is just a log. My plan for this project:

1. Pause the pipeline. Stop posting label suggestions until I understand the drift.
2. Check the endpoint's status page and docs. A model version change is often announced there even without a direct ping.
3. Re-run the probe against a backup model. If the backup's accuracy is stable, switch routing to it.
4. Adjust expectations only after verification. If the drift is permanent and the new behavior is actually correct, update the probe's expected labels.

The probe is the trigger. The response is still judgment.

It does not catch:

• Drift on inputs I did not test. Ten comments vs. thousands of real ones. If the failure pattern is not in the probe, the probe stays quiet.
• Subtle quality drops. A summary that is 10% worse is hard to see with exact-string diffs. This design fits discrete, checkable outputs.
• Latency or reliability. Pair it with a health check for the full picture.

Who should not rely on this: teams with a customer-facing SLA on model output, apps where a wrong classification is costly, and anyone who needs a guarantee the model is identical tomorrow. Those cases need a paid, pinned model version.

Who should use it: developers on a zero budget, maintainers of small tools on free endpoints, and anyone who wants to hear about a model change before users do.

Free models are not static. They change without a changelog, and availability checks will not tell you. A drift probe—fixed inputs, temperature 0, daily runs, exact output logging—turns a vague feeling into a date, a number, and a diff.

Try it this week. Point the script at any OpenAI-compatible endpoint (I pointed mine at MonkeyCode's free model access), let it append to drift_history.jsonl for seven days, then diff day 1 against day 7. If accuracy or label distribution moved, you have evidence instead of a hunch. If you run a similar probe, tell me what it caught—I want the failure modes I missed.

A free server option is enough to reproduce the setup.