An eval set is a snapshot of what your product needed the day you built it, and products, users, models, and even the golden answers themselves drift after that. So treat the set as a living asset: version it, feed it with real production failures every week, retire cases that no longer represent real traffic, and re-check the labels on a schedule. A 100 percent pass rate on a set nobody has touched in six months is a warning sign, whatever the dashboard says.
How does an eval set rot?
Quietly, and from several directions at once. Product policy changes while the golden answers still encode the old rule. A new user segment arrives asking unfamiliar questions. Eval cases leak into the prompt as few-shot examples. Some labels were wrong on day one. And the set gets easy, because every failure was fixed and no hard cases replaced them.
A refund window moving from 14 days to 30 is enough to turn a dozen gold answers into confidently wrong ones. The dashboard rarely tells you which mode you're in:
| Rot mode | What the dashboard shows | What is actually happening |
|---|---|---|
| Policy change | Pass rate holds steady | Gold answers encode a rule the product dropped |
| New user segment | Pass rate holds steady | The set no longer resembles real traffic |
| Few-shot leakage | Pass rate climbs | The prompt is graded on cases it memorized |
| Labels wrong on day one | Pass rate holds steady | You're optimizing toward somebody's mistake |
| Set gone too easy | Pass rate near 100% | Every hard case was fixed, none replaced |
We hit the leakage one ourselves. Pass rate sat at 98 percent for two months while user complaints climbed. Someone had lifted six of the "hard" eval cases into the system prompt as examples of good output. The eval was grading the prompt on its own homework.
Feeding it from production
Every week, pull four buckets out of the logs and label twenty to thirty cases from them. Every case carries a source tag and a date. Ambiguous ones get two labellers, and disagreements get discussed rather than averaged. That's the whole maintenance habit, and it takes about an hour once somebody owns it.
- Schema-validation failures. Anything the parser rejected outright.
- Thumbs-downs. The user has already told you; believe them.
- Escalations, either to the bigger model or to a human.
- A plain random sample, so the set doesn't become a museum of failures.
Keep a separate "hard set" of the cases that failed at least once in production. It stays small and it runs on every prompt change, which is how it catches the regressions the big set is too diluted to notice.
Retiring and re-labelling
Once a quarter, re-label a random 10 percent of the set blind, without showing the stored gold answer. If agreement with the stored labels drops below roughly 90 percent, audit the whole slice that sample came from. Retire cases tied to removed features, and archive them instead of deleting.
Features come back, and so do their bugs.
Bake in a rule that a case cannot enter the set without a named labeller. "Generated by the model and looked fine" is how you end up with an eval set that agrees with the model by construction.
The numbers I watch
Four of them. Pass rate sliced by the date a case was added. A per-case flakiness score, from running each case three times, so you know which failures are noise. The labeller agreement rate from the quarterly re-label. And the age distribution of the whole set, which tells you at a glance whether anyone is still feeding it.
The first one earns its keep fastest. If cases from the last month pass at 80 percent while cases from a year ago pass at 99, the product has drifted away from the old set and the old set is flattering you.
This is also my favourite question to put to vendors selling "agents." Real ones can show you their eval set and how it changed over the last quarter, and it's one of the fastest ways to tell a real agent from agent-washing. At Shanti Infosoft the eval set is a deliverable with a version number, handed over alongside the code, because a client who inherits a model without its eval set inherits something they can't safely change. That handover closes every machine learning engagement we take on.
When was the last time a case in your eval set was re-labelled by a human who didn't see the original answer?
Sagar Jain leads engineering as technical co-founder of Shanti Infosoft, a CMMI Level 5 team of 80+ engineers.
Top comments (0)