Your Eval Set Is Rotting: A Theme-Fingerprint Drift Check

Your evaluation set has an expiration date. Last month it helped you pick a model. This month it is lying to you. Production prompts drift. Eval prompts stay frozen. The score climbs while real quality sinks.

I watched this happen to a support bot. The benchmark said 84%. The users said worse. Both were right. The eval set still measured the old product. The users had moved on.

Could a leaderboard catch this? No. Leaderboards measure your old questions. They never see your current users. You need a signal that compares production reality with eval reality.

My fix is a theme fingerprint. No vector database. No GPU. Just filtered words and a Jaccard score. Run it weekly. Let it tell you when the golden set has expired.

The fingerprint in three steps

1. Lowercase every prompt and extract word tokens.
2. Drop stopwords and words shorter than four characters.
3. Turn the survivors into a set. Compare eval and production sets.

The math is boring. The insight is not. A Jaccard score below 0.6 means your eval set no longer represents production. Trusting its numbers is then a habit, not a measurement.

The actual implementation

import re
from collections import Counter

STOP = set("""
the a an and or but for with on at to of in
is are was were be do does did have has had
it its this that these those i you we they
he she them their your my our not no so if
then than from by as
""".split())

def token_bag(prompts):
    bag = Counter()
    for prompt in prompts:
        words = re.findall(r"[a-z']+", prompt.lower())
        for word in words:
            if word not in STOP and len(word) > 3:
                bag[word] += 1
    return set(bag)

def jaccard(left, right):
    if not left or not right:
        return 0.0
    return len(left & right) / len(left | right)

That is the whole detector. Feed it two lists of prompts. One list from your eval set. One list from last week's production logs. Compare the fingerprints.

Turning it into a weekly check

This is where free infrastructure changes the habit. MonkeyCode is an open-source project with two relevant pieces: free model access for small jobs and a free server option for recurring tasks. I use the free model access to label and cluster support transcripts. I run the fingerprint comparison on the free server every Sunday. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free token allowance, reported at the time of writing as ten million tokens, covers hundreds of weekly runs; the repository is the source of truth for current terms.

The weekly job is a short shell command, not a ceremony:

python drift_check.py \
  --eval data/eval_prompts.jsonl \
  --prod data/production_prompts.jsonl \
  --threshold 0.6

The job writes one line of JSON to a log. That line feeds a dashboard, a Slack bot, or a quiet file.

The decision table

Stop treating the Jaccard score as a number. Treat it as a verdict.

Jaccard	Verdict	Action
>= 0.75	Fresh	Trust the eval score for model shopping
0.60–0.74	Aging	Add production prompts to a holdout set
0.40–0.59	Stale	Rebuild the eval set before comparing models
< 0.40	Rotting	Stop comparing models entirely

The last row is the real trick. A rotting eval set does not just mislead you. It actively sells you the wrong model, because the score difference you see is noise plus drift.

Reproduce it in one command

You do not need real logs to verify the logic. Generate two synthetic sets and check the behavior.

import random

def synth(count, seed, topic_pool):
    rng = random.Random(seed)
    return [
        f"how do i {rng.choice(topic_pool)} my order {rng.randint(1, 9999)}"
        for _ in range(count)
    ]

base = ["refund", "login", "billing"]
shifted = ["refund", "login", "billing", "export", "sync", "timezone"]

eval_prompts = synth(300, 1, base)
prod_same = synth(300, 2, base)
prod_shifted = synth(300, 3, shifted)

a = token_bag(eval_prompts)
print("same topics:", round(jaccard(a, token_bag(prod_same)), 2))
print("drifted topics:", round(jaccard(a, token_bag(prod_shifted)), 2))

Run it and the gap appears immediately. The same-topics case scores high. The shifted case drops below the threshold. That drop is the early warning.

Reading the weekly line

The job writes one line like this:

{"date": "2026-08-28", "jaccard": 0.51, "eval_n": 400, "prod_n": 1200, "verdict": "stale"}

Train your team to read the verdict column first. A "stale" verdict is not a bug report. It is a trigger to rebuild the eval set before you spend a single token on model comparison. A "fresh" verdict means the model shopping can proceed.

Where this breaks

A keyword fingerprint ignores synonyms. "Payment" and "billing" never overlap, so a pure rename in your docs looks like drift. Short prompts produce noisy bags. Tiny samples do, too. Use at least a few hundred prompts per side.

The check also assumes your eval set represents one stable product. If you ship a major feature every week, the fingerprint will scream constantly. That is accurate, but it means the threshold needs tuning.

Who should skip this? Teams with strict data residency rules. If no transcript may leave the machine, run the job locally and keep the free server out of the path. The fingerprint still works; only the scheduling changes.

The one decision that matters

Do not ask which model is better this week. Ask whether your eval set still describes this week. The fingerprint answers the second question cheaply. The answer decides whether the first question is even worth asking.

Clone the repo, confirm the current free terms, then point the drift check at your own logs. The first run tells you how long your trust in that 84% should last.