Your Agent Said It Worked. Go Check the World, Not the Sentence.

Your agent said it created the ticket. The eval passed. The ticket does not exist.

This is the failure mode that action-taking agents introduce and that output-grading evals sail right past. When an agent's job is to say something, grading the text is grading the job. When an agent's job is to do something — write a file, open a PR, charge a card, send an email — the text it emits is a claim about a side effect, not the side effect itself. An agent that has learned to produce confident, well-formatted "Done! I created issue #4213" strings will pass any judge you point at its output, forever, whether or not #4213 is real.

If you take one thing from this post: for action agents, the artifact under eval is the world, not the sentence.

Claims are cheap, effects are load-bearing

A model-as-judge reading "I created the ticket and assigned it to the on-call" has exactly zero independent information about whether a ticket exists. Judge and agent share a substrate; the judge is just a second language model agreeing that the sentence sounds like success. That is circular, and it is the whole reason agent-eval ranks evidence on an independence axis — independent to corruptible — rather than a cost axis of cheap to expensive.

• Tier 1 — externally observable proof the agent can't forge. The ticket exists when you GET /issues/4213 and get a 200. The file exists on disk. The PR is open. The row is in the database. None of this can be hallucinated into being.
• Tier 2 — statistical signal vs a baseline the agent didn't author. The created ticket's title actually embeds-similar to the task you gave it (not a real ticket for the wrong thing). The diff changed the file it claimed to change.
• Tier 3 — model-as-judge. Was the ticket well-written? That's an opinion, a signal, never a verdict — and it only earns a seat after Tiers 1 and 2 have confirmed the ticket is real.

For side-effecting agents, Tier 1 is not the nice-to-have. It's the whole game. Most of your production failures — the API call 500'd, the write hit a read-only mount, the agent retried and created the ticket twice, the "sent" email bounced — are caught here for ~$0, deterministically, fast enough to sit in the hot path and block the run. Tier 3 can't do any of that: it's offline-only, metered, and non-deterministic. You do not want a slow model opinion standing between your user and a retry.

Verify the effect, don't grade the claim

Here's the shape of a Tier 1 side-effect check. Note what it does not do: it never reads the agent's own summary of what happened.

type EffectCheck<T> = {
  name: string;
  // Observe the world independently. The agent does not get to write this.
  observe: () => Promise<T | null>;
  // Assert the observed state matches the task, not the agent's story.
  expect: (observed: T) => boolean;
};

async function verifyEffect<T>(check: EffectCheck<T>) {
  const observed = await check.observe();
  if (observed === null) {
    return { tier: 1, pass: false, reason: `${check.name}: effect not found in world` };
  }
  if (!check.expect(observed)) {
    return { tier: 1, pass: false, reason: `${check.name}: effect exists but wrong shape` };
  }
  return { tier: 1, pass: true };
}

// The agent claimed it opened issue #4213 for the task "flaky login test".
const ticketExists = await verifyEffect({
  name: "github-issue",
  observe: () => gh.issues.get({ owner, repo, issue_number: 4213 })
                    .then(r => r.data).catch(() => null),
  expect: (issue) =>
    issue.state === "open" &&
    embedSimilar(issue.title, "flaky login test") > 0.75, // Tier 2 riding along
});

The observe function is the entire point. It calls GitHub, not the agent. A hallucinated issue number returns null and fails at Tier 1 before any judge is ever invoked. This is how you "ship the 80%": stale, crashed, wrong-shape, and hallucinated-effect failures all die here, cheaply, leaving only the genuinely subjective tail — is this a good ticket? — for the metered judge, clearly labeled opinion, not evidence.

The trace is what makes this debuggable

There's a second problem hiding in the code above. When verifyEffect returns effect not found, why? Did the agent call issues.create and get rate-limited? Did it call the wrong endpoint? Did it call the right one, get a 201 back with issue #4299, and then hallucinate #4213 into its summary? The pass/fail tells you the world is wrong. It doesn't tell you where the agent went off the rails.

That's what AgentLens is for. It captures the trace of how the agent got to its claim — every model step and tool call, the resolved inputs, and the raw outputs the agent actually received. So when Tier 1 goes red, you replay the trajectory: issues.create returned 201 with number: 4299, and the agent's final message said 4213. Now you know it's a summarization bug, not a permissions bug, and you fix the right thing.

The pairing runs deeper than debugging, though. Tier 1 and Tier 2 need something to score against, and it has to be data the agent didn't author. The trace is exactly that. The raw 201 response body sitting in AgentLens is unforgeable ground truth: agent-eval reads the real returned issue number from the trace and compares it to what the agent claimed, and the lie surfaces instantly. agent-eval scores and gates the output; AgentLens captures the trajectory that makes the score both debuggable and trustworthy. They're two halves of one loop — you can't do independent evals on trajectory data you didn't independently capture.

The rule

Stop asking "did the agent say it worked?" Ask "did the world change the way the task required?" For action agents those are different questions, and only one of them is load-bearing. Verify the effect at Tier 1, corroborate its shape at Tier 2, and reserve the judge for the 20% where taste actually matters — with the trace underneath so a red gate points you at a fix instead of a shrug.

Your agent's summary is a hypothesis. Go check.

Comments

[Reid Marlow]

This is the line I wish more agent demos crossed. The summary is just a claim, and the useful verifier is usually the cheapest one in the stack. For code I want the test result, the changed files, and the command transcript before I care what the model thought happened.