Your Golden Dataset Is Rotting: The Eval Oracle Nobody Re-Validates

We talk about agents drifting. We almost never talk about the thing we measure them against drifting. But your golden dataset — the fixtures, expected outputs, and "known good" traces your evals grade against — is code that ships to production and then never gets a code review again. It rots. And when your oracle rots, a green dashboard stops meaning your agent is correct. It means your agent still agrees with a snapshot of the world you took eight months ago.

This is the failure mode senior teams keep rediscovering the hard way: the agent is fine, the harness is fine, and the test oracle is the thing that's wrong. Nobody re-validates it, because passing tests feel like the end of the story instead of a claim that also decays.

Why oracles rot

A golden dataset encodes assumptions about the world at the moment you captured it. A few months later:

• The API you called changed its response shape, so your "expected output" is now describing an endpoint that no longer exists.
• The correct answer changed. "Current CEO," "latest stable version," "the recommended flag" — these have expiration dates baked in.
• A human labeled the golden answer under a policy that has since been updated, so your eval now enforces last quarter's rules.
• The expected string was itself generated by a model, and you never checked whether it was actually right. You froze a plausible guess and called it truth.

None of these trip an alarm. Your suite is green the whole way down, because green means "matches the fixture," not "matches reality."

The tier lens: which evidence actually expires?

This is where it helps to stop ranking evidence by cost (cheap vs expensive) and start ranking it by independence — how forgeable the signal is, and by extension how well it holds up over time. agent-eval sorts evidence into three tiers on exactly that axis:

• Tier 1 — externally observable proof the agent can't forge. Valid JSON, the file exists, the URL resolves, it compiled, the tests passed, it finished inside the timeout, the output is non-empty.
• Tier 2 — statistical signal against a baseline the agent didn't author. Embedding similarity to the task, length and repetition checks, whether the diff actually changed anything.
• Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

Now re-read the rot list through that lens. Tier 1 checks barely rot: "is this valid JSON" and "does this file exist" mean the same thing today as they did last year. The oracle for Tier 1 is the world, and the world re-validates itself every run. Tier 2 rots slowly and legibly: your baseline embedding shifts as the task definition shifts, which is a signal you can watch. It's Tier 3 and hand-frozen golden strings that rot fastest and most silently — because their oracle is an opinion or a snapshot, and opinions and snapshots don't refresh themselves.

That's the argument for two structural rules a lot of teams learn late:

1. Tier 1 + Tier 2 are your real-time gate — deterministic, roughly free, fast enough to block a run in the hot path. Tier 3 is offline-only — metered, slow, non-deterministic, so it can't sit in the latency budget of a live request.
2. Tier 1 + Tier 2 may run over the agent's full trajectory. Tier 3 may not. A model judging another model's reasoning is circular — judge and judged share a substrate, so there's no independent ground truth in the loop. Tier 3 gets to inspect artifacts the judged agent didn't get to write, and nothing more.

A rotting golden dataset is what happens when you let a frozen Tier 3 opinion masquerade as a Tier 1 fact.

Ship the 80%, then re-validate the tail

Most real failures — stale output, a crash, a malformed response, a hallucinated file path, an empty answer — are caught at Tier 1 + Tier 2 alone, and those checks age gracefully. Reserve the judge for the ~20% subjective tail, and label its output as "opinion, not evidence." That framing also tells you where to point your re-validation budget: not at the whole suite, but at the fixtures whose correctness is an opinion or a snapshot. Those are the ones with an expiration date.

Here's the cheap version: give every golden fixture a freshness contract, and let Tier 1 checks re-prove the world-facing ones on every run.

type Tier = 1 | 2 | 3;

interface GoldenFixture {
  id: string;
  expected: unknown;
  oracleTier: Tier;        // how the "truth" was established
  capturedAt: string;      // ISO timestamp
  maxAgeDays: number;      // expiration for snapshot/opinion oracles
  reverify?: () => Promise<boolean>; // Tier 1 re-proof against the world
}

async function checkOracleHealth(f: GoldenFixture): Promise<string[]> {
  const problems: string[] = [];

  // Tier 1 fixtures can re-prove themselves: does the URL/file still resolve?
  if (f.reverify && !(await f.reverify())) {
    problems.push(`${f.id}: world-facing oracle no longer holds`);
  }

  // Snapshot/opinion oracles (frozen strings, judge labels) expire.
  if (f.oracleTier >= 2) {
    const ageDays = (Date.now() - Date.parse(f.capturedAt)) / 86_400_000;
    if (ageDays > f.maxAgeDays) {
      problems.push(
        `${f.id}: oracle is ${Math.round(ageDays)}d old (max ${f.maxAgeDays}), re-validate before trusting`,
      );
    }
  }
  return problems;
}

Run that as a meta-eval — an eval on your evals. When it goes red, you don't touch the agent. You go re-validate the oracle. The point isn't the specific thresholds; it's making "this expected value has an expiration date" a first-class property instead of a tribal assumption.

You can't re-validate what you can't see

Here's the part teams miss: to re-validate a fixture, you need to know how the agent actually produced the output you're grading, not just the final string. That's where the eval half and the trace half meet. agent-eval scores and gates the output — the tiers, the drift, the hallucination checks above. AgentLens captures the trace of how the agent got there: every model call and tool step, the resolved inputs, the raw outputs. Two halves of one workflow.

The trace is what makes Tier 1 + Tier 2 possible in the first place — those tiers need trajectory data the agent didn't get to author, and AgentLens is where that unforgeable record lives. It's also what makes rot diagnosable: when your meta-eval flags a stale fixture, the trace tells you whether the expected value was a real observation or a model's frozen guess, and which tool response it was pinned to. Without the trace you're re-guessing the oracle. With it, you're auditing it.

The takeaway

An eval is a claim about the world, and claims expire. Rank your evidence by independence, not price: Tier 1 and Tier 2 hold up because their oracle is reality; Tier 3 and frozen golden strings are opinions with a shelf life. Gate on the durable tiers in real time, keep the judge offline and clearly labeled as opinion, and run a meta-eval that treats oracle freshness as a real signal. Pair agent-eval for the scoring with AgentLens for the trace, and a red fixture stops being a mystery. Because the most dangerous eval isn't the one that fails. It's the one that passes against a world that no longer exists.