The Free LLM Changed Overnight. My 40-Line Sentinel Caught It.

Wednesday. 9:14 AM. A teammate pasted a PR summary into Slack. "Something feels off."

The code was right. The format was right. The tone was wrong. I opened the provider changelog. Nothing. I opened the model card. Nothing. Then I looked at the last 300 responses. The distribution had shifted. Nobody changed a prompt that week.

Free endpoints don't tell you when they move

Free model endpoints rarely pin a version. Providers swap models to manage cost and load. The API contract stays identical. The weights underneath change without notice.

A test suite that checks schema and keywords will pass. Users notice the difference before your assertions do. You need a behavior detector, not another unit test.

My current pipeline uses free model access from the open source MonkeyCode project. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Deployment runs on the project's free server option, which keeps the watch loop cheap. The published allowance is 10 million tokens at the time of writing; re-check the project page before you build on that number.

Why unit tests miss a model swap

Your tests assert exact things. JSON shape. Required fields. Banned phrases. A model swap changes style first.

Sentence length shifts. Repetition rises. Code fences appear where none used to be. None of that fails a schema test. None of it fails a keyword test. This is distribution drift, and distribution drift needs a detector, not a validator.

A 40-line output sentinel

The detector computes three cheap features per response: token count, type-token ratio, and code-fence ratio. It builds a baseline from the first 50 warmup responses. Then it converts new responses into z-scores against that baseline.

A z-score above 3 means the distribution moved. The whole thing has no dependencies.

import json
import re
from statistics import mean, pstdev

class OutputSentinel:
    def __init__(self, warmup=50, z_threshold=3.0):
        self.warmup = warmup
        self.z_threshold = z_threshold
        self.warmup_feats = []
        self.baseline = None

    def _features(self, text):
        tokens = re.findall(r"\S+", text)
        words = re.findall(r"[A-Za-z']+", text)
        if not tokens:
            return None
        fences = sum(1 for t in tokens if t.startswith("`"))
        return {
            "length": len(tokens),
            "ttr": len(set(words)) / max(1, len(words)),
            "code_ratio": fences / len(tokens),
        }

    def _zscore(self, feat):
        zs = []
        for key in ("length", "ttr", "code_ratio"):
            z = (feat[key] - self.baseline[key]["mean"]) / max(1e-9, self.baseline[key]["std"])
            zs.append(abs(z))
        return max(zs)

    def add(self, text):
        feat = self._features(text)
        if feat is None:
            return None, None
        if self.baseline is None:
            self.warmup_feats.append(feat)
            if len(self.warmup_feats) >= self.warmup:
                self.baseline = {
                    key: {
                        "mean": mean(f[key] for f in self.warmup_feats),
                        "std": pstdev(f[key] for f in self.warmup_feats),
                    }
                    for key in ("length", "ttr", "code_ratio")
                }
            return None, None
        return self._zscore(feat) > self.z_threshold, self._zscore(feat)

The hot path is a dictionary lookup and a running mean. It can sit in front of a completion callback for years. When it fires, it prints one line.

sentinel = OutputSentinel()

def on_completion(response):
    text = response["text"]
    alert, z = sentinel.add(text)
    if alert:
        print(json.dumps({"event": "distribution_shift", "z": round(z, 2)}))

Reading the signal

A z-score tells you something moved. What moved? That is the next step.

Signal	Likely cause	First move
length z-score spikes	new model variant is more verbose	sample 20 outputs, diff them against the warmup baseline
ttr drops toward 1	repetitive generation mode	add a repetition guard at the application layer
code_ratio jumps	routing changed or the prompt drifted	replay the same 20 prompts against both output sets

Do not tune the threshold on day one. Take the 99th percentile of your own history and start there. A threshold that never fires today will still catch next month's swap.

Deploying the watch loop

A sentinel needs a home. A cron job is enough.

1. Log every completion as one JSONL line: timestamp, prompt hash, output text.
2. Replay the last 300 lines through the sentinel at startup.
3. Set the z threshold from your own 99th percentile, not from a blog post.
4. Run the script every hour on the free server option; the loop costs almost nothing.
5. When it fires, copy the recent outputs into a diff and read them before changing anything.

Five steps. No new infrastructure. The JSONL file is your audit trail.

What it will not catch

The sentinel misses a swap that preserves these three features. A well-tuned swap keeps length, diversity, and formatting stable.

It can also be fooled by a workload change. New task types shift the distribution honestly. That is not a bug in the detector. It is a reason to log prompt hashes and split the baseline per task type.

Volume matters too. Fewer than 50 warmup responses and the baseline is noise.

Who should skip this

Teams with a locked model version do not need a sentinel. The contract is pinned, so drift is impossible.

Teams that process fewer than 50 completions a week will chase phantom signals. And anyone without a JSONL log should build that first. The sentinel is useless without history.

The lesson

A free model is a moving target. Treat it like one. Test the shape, watch the distribution, and keep the receipts.

The sentinel above is plain Python, no dependencies, and it runs anywhere — including the free server option I already use. If you want a free endpoint to practice against, MonkeyCode's free model access is a reasonable place to start. Just re-check the published allowance first. Free tiers move. Now your detector can too.