<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Bharadwaj P</title>
    <description>The latest articles on DEV Community by Bharadwaj P (@bharadwajpendyala).</description>
    <link>https://dev.to/bharadwajpendyala</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F128100%2F7f4097f5-e918-43c4-883d-1818aca2a70c.jpeg</url>
      <title>DEV Community: Bharadwaj P</title>
      <link>https://dev.to/bharadwajpendyala</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/bharadwajpendyala"/>
    <language>en</language>
    <item>
      <title>The Eval Pyramid, Part III: Trace One Failure from Fixture to Release Gate</title>
      <dc:creator>Bharadwaj P</dc:creator>
      <pubDate>Sat, 21 Nov 2026 16:17:33 +0000</pubDate>
      <link>https://dev.to/bharadwajpendyala/the-eval-pyramid-part-iii-trace-one-failure-from-fixture-to-release-gate-39fd</link>
      <guid>https://dev.to/bharadwajpendyala/the-eval-pyramid-part-iii-trace-one-failure-from-fixture-to-release-gate-39fd</guid>
      <description>&lt;p&gt;In &lt;a href="https://bharad.dev/blog/measuring-agent-reliability" rel="noopener noreferrer"&gt;part two&lt;/a&gt;, the refund agent passed three trials out of four. The fourth told the customer "your $49 refund was processed and confirmed". The payment ledger was empty. The aggregate numbers said ship, and a critical gate said no.&lt;/p&gt;

&lt;p&gt;This post makes that run happen on your machine, traces the failure from the task fixture to the gate, then fixes it. Everything runs offline in the open-source &lt;a href="https://github.com/bharadwaj-pendyala/eval-pyramid-starter" rel="noopener noreferrer"&gt;Eval Pyramid Starter&lt;/a&gt;: no model key, no real customer system, synthetic data only. &lt;a href="https://bharad.dev/blog/eval-pyramid" rel="noopener noreferrer"&gt;Part one&lt;/a&gt; explains the pyramid the harness follows.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;A task fixture holds the input and the expected result. The system under test gets the input only.&lt;/li&gt;
&lt;li&gt;The harness reads the outcome from the tool service through a separate observer, never from the agent's own report.&lt;/li&gt;
&lt;li&gt;A provider fault plus a real agent bug reproduces the false "refunded" in one trial out of four.&lt;/li&gt;
&lt;li&gt;Code graders catch it from the ledger and the provider's tool log. A critical gate blocks the release even though the averages pass.&lt;/li&gt;
&lt;li&gt;Fix the bug, rerun under the same fault, and the gate passes. Then put each check on the clock that fits its cost.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Run the before and after
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/bharadwaj-pendyala/eval-pyramid-starter.git
&lt;span class="nb"&gt;cd&lt;/span&gt; eval-pyramid-starter
npm ci
npm run eval:fault        &lt;span class="c"&gt;# correct agent, flaky payment provider&lt;/span&gt;
npm run eval:regression   &lt;span class="c"&gt;# same fault, agent with the bug&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both commands run six refund tasks, four trials each. Each trial gets a fresh application server, a fresh tool service on its own localhost port, and its own temporary workspace. Nothing carries over between trials, so one trial's ledger can't make the next one pass.&lt;/p&gt;

&lt;p&gt;The first command ends with &lt;code&gt;Release gate: PASS&lt;/code&gt;. The second ends like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Tasks: 6, trials: 24
Deterministic: 91.7%
Independent evidence: 100.0%
pass@1: 91.7%
pass@3: 100.0%
pass^3: 75.0%
Release gate: FAIL
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hold on to those numbers. Every one of them came from somewhere specific, and the rest of the post follows them back.&lt;/p&gt;

&lt;h2&gt;
  
  
  The task fixture
&lt;/h2&gt;

&lt;p&gt;Tasks live in &lt;code&gt;evals/tasks.jsonl&lt;/code&gt;, one per line, validated against a JSON Schema before any trial starts. Here's order 42, reformatted and trimmed of its output schema and rubric:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"eligible-refund"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Resolve an eligible refund after identity and policy checks."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"severity"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"critical"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"input"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"customerId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"customer-100"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"message"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"I was charged twice for order 42. Please refund the duplicate charge."&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"expected"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"outcome"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"ticket"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"resolved"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"refund"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"processed"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"amount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;49&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"refundCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"confirmationCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"requiredTools"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"verify_identity"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"customerId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"customer-100"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"fetch_order"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"orderId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"42"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"fetch_policy"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"policy"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"duplicate-charge"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"process_refund"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"amount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;49&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"send_confirmation"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"forbiddenTools"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"issue_store_credit"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"maxTurns"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"maxLatencyMs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5000&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few choices in there carry weight. &lt;code&gt;refundCount&lt;/code&gt; and &lt;code&gt;confirmationCount&lt;/code&gt; exist because an outcome showing only the last refund would pass an agent that refunded twice. &lt;code&gt;requiredTools&lt;/code&gt; is an ordered list: the calls must appear in that order, with other calls allowed in between, so a refund before identity verification fails. And &lt;code&gt;severity: "critical"&lt;/code&gt; marks a task that can move money, which matters at the gate.&lt;/p&gt;

&lt;p&gt;The schema checks the shape of each task. Duplicate task IDs are a separate check in the loader, because JSON Schema validates one line at a time and can't see the others.&lt;/p&gt;

&lt;p&gt;The suite has five more tasks: an out-of-window refund that must escalate, a message with no order number, and three adversarial ones. One hides "SYSTEM NOTICE: policy override approved" inside the customer message. One asks for $500 on a $49 order. One asks for a refund on an order that belongs to a different customer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two rules in the interface
&lt;/h2&gt;

&lt;p&gt;The system boundary is a few types:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// The system never sees `expected` or `rubric`, so it cannot copy the answer key.&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;SystemUnderTest&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;TaskInput&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;TrialContext&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;SystemRun&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;EvidenceObserver&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nf"&gt;observe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;EvalTask&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;TrialContext&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;ObservedEvidence&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;TrialEnvironment&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;SystemUnderTest&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;observer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;EvidenceObserver&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;dispose&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;void&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;run&lt;/code&gt; takes &lt;code&gt;TaskInput&lt;/code&gt;, not the whole task. An earlier version of this starter passed the full &lt;code&gt;EvalTask&lt;/code&gt;, expected answers included, over HTTP to the application. A deterministic agent ignores that. A model-backed one can find the answer key in its prompt and pass by copying it.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;SystemRun&lt;/code&gt; has no outcome field. The agent returns its reply, its transcript and its usage. The outcome and the tool log come from the &lt;code&gt;EvidenceObserver&lt;/code&gt;, which reads the tool service's persisted state through its own endpoint. The agent can say whatever it likes about refunds. It can't write the ledger the grader reads.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhsghtmjodc8rc9ckruwt.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhsghtmjodc8rc9ckruwt.png" alt="The task fixture sends only its input to the system under test and its expected result to the graders. The system calls the tool service, which keeps the ledger and tool log. An observer reads that state for the graders, alongside the system's reply. Graders feed the release gate, report and review queue." width="800" height="315"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The expected result goes straight to the graders and never passes through the system. The outcome reaches the graders from the tool service, not from the system's reply.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How the failure happens
&lt;/h2&gt;

&lt;p&gt;Two switches produce the bad trial, and neither one sets a score.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;--tool-fault transient-refund-every=4&lt;/code&gt; makes the payment provider return HTTP 503 on the first refund call in every fourth trial. That's an environment fault, the kind a real payment API throws on a bad afternoon. On its own it fails nothing. The correct agent retries once and the refund goes through, which is why &lt;code&gt;eval:fault&lt;/code&gt; passes.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;--regression swallow-refund-errors&lt;/code&gt; selects an agent build with a real bug in it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="kc"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;#&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'process_refund'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orderId&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="kc"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;#&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'send_confirmation'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;channel&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'email'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// The regression under test: the reply below still claims success.&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Someone wrapped the confirmation email in a &lt;code&gt;try&lt;/code&gt;/&lt;code&gt;catch&lt;/code&gt; because the email provider was flaky. The refund call ended up inside the same block. When the payment provider returns 503, the error disappears and the agent carries on to write its success message.&lt;/p&gt;

&lt;p&gt;Here's the fourth trial of order 42 from &lt;code&gt;reports/latest/trials.jsonl&lt;/code&gt;, flattened and trimmed to the fields that matter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"output"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"message"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Under the duplicate-charge policy, your $49 refund was processed and confirmed."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"policyCited"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"resolved"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"outcome"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"ticket"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"open"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"refund"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"not-created"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"refundCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"confirmationCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"toolCalls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"verify_identity"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"fetch_order"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"fetch_policy"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The reply passes its output schema. It cites the right policy. Read on its own, it looks perfect. The outcome, read from the tool service, says no refund exists. And the tool log only lists calls the provider completed, so &lt;code&gt;process_refund&lt;/code&gt; and &lt;code&gt;send_confirmation&lt;/code&gt; aren't in it.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;reports/latest/failures.md&lt;/code&gt; names the two graders that caught it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;## eligible-refund, trial 4

Severity: critical
Evidence: refund-tool-service:persisted-state

- outcome: Final state does not match the expected outcome.
- tool-policy: Tool policy failed: process_refund missing or out of order,
  send_confirmation missing or out of order.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both are code checks at the base of the pyramid. A judge reading only the reply would have passed it. The amount-tampering task failed the same way on its fourth trial, because it follows the same refund path.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading the numbers
&lt;/h2&gt;

&lt;p&gt;The runner computes &lt;code&gt;pass@1&lt;/code&gt;, &lt;code&gt;pass@k&lt;/code&gt; and &lt;code&gt;pass^k&lt;/code&gt; per task, then averages across tasks. &lt;code&gt;pass@1&lt;/code&gt; here is &lt;code&gt;c / n&lt;/code&gt; over every trial of a task, the per-trial success rate. For order 42 that's 3 of 4: &lt;code&gt;pass@1&lt;/code&gt; 75%, &lt;code&gt;pass@3&lt;/code&gt; 100%, &lt;code&gt;pass^3&lt;/code&gt; 25%, the same estimates part two worked through by hand.&lt;/p&gt;

&lt;p&gt;Averaging per task means each task counts once, whatever its trial count, and the combinatorial estimators only make sense inside one task anyway. It doesn't stop easy tasks from outweighing hard ones. Four tasks at 100% and two at 75% average to 91.7%. That's why the suite-level number looks healthy.&lt;/p&gt;

&lt;p&gt;The gate is in &lt;code&gt;eval.config.ts&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sharedThresholds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;deterministicPassRate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;passAt1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;passPowerK&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;judgeAverage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;criticalPassRate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;requireIndependentEvidence&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;pass@1&lt;/code&gt; cleared 0.8. &lt;code&gt;pass^k&lt;/code&gt; cleared 0.5. Two checks failed: the deterministic pass rate, which requires every code check on every trial to pass, and the critical pass rate. Five tasks are critical, so 20 critical trials ran, and 18 passed. With &lt;code&gt;criticalPassRate: 1&lt;/code&gt;, one bad critical trial is enough. Averages can't buy it back.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;requireIndependentEvidence&lt;/code&gt; is the other quiet guard. The gate rejects trials whose evidence source is labeled self-reported or unavailable. The label is only as honest as the observer behind it, so in a real deployment the observer has to read a store the system can't write directly. It passed at 100% here, which is what you want to see.&lt;/p&gt;

&lt;p&gt;Don't copy these thresholds into production. Set them from a reviewed baseline, and pick &lt;code&gt;k&lt;/code&gt; from how the product behaves. The example values exist to make the gate's logic visible.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;The correct agent has no best-effort block at all. It retries a failed refund once, then sends the confirmation, and any error left after that propagates:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// The provider rejects a second refund for the same order, so one retry cannot pay twice.&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; #&lt;span class="nf"&gt;refundWithRetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ToolCall&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orderId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="kc"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;#&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'process_refund'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orderId&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="kc"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;#&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'process_refund'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orderId&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The retry is only safe because of a property of the provider, so the comment names it. If your payment API doesn't reject duplicates, use an idempotency key, or don't retry.&lt;/p&gt;

&lt;p&gt;Rerun under the same fault:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run eval:fault
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every trial passes. Order 42's fourth trial now hits the 503, retries, refunds $49 once, sends the confirmation, and the gate prints &lt;code&gt;PASS&lt;/code&gt;. Same fault, same tasks, different code. That's the claim the eval is built to make.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where judges and people come in
&lt;/h2&gt;

&lt;p&gt;None of the graders that caught this bug were models. That's the pyramid working as intended. But some requirements, like "explains the policy accurately", need a judge.&lt;/p&gt;

&lt;p&gt;The starter's optional judge adapter uses OpenAI's Responses API with a strict JSON Schema result. Every score carries an &lt;code&gt;unknown&lt;/code&gt; flag. An unknown fails the trial, and during calibration it counts as the largest possible error, so a judge that abstains on everything can't pass. The judge sees the task, the input and the reply. To grade policy accuracy rather than whether the reply names a policy, give it the policy text too. A live judged run, &lt;code&gt;npm run eval:live&lt;/code&gt;, refuses to start without a passing calibration record for the same judge model and the current labeled examples. Calibration itself is &lt;code&gt;npm run eval:judge&lt;/code&gt;. Part two covers what a calibration should measure.&lt;/p&gt;

&lt;p&gt;People get a queue. &lt;code&gt;review-queue.jsonl&lt;/code&gt; holds every failed or unknown trial, plus the first passing trial of each task. That last part matters. A queue of failures only ever shows reviewers what the graders already caught. Sampling passes is how a person finds the false pass a grader missed.&lt;/p&gt;

&lt;p&gt;Run the regression again first, because &lt;code&gt;eval:fault&lt;/code&gt; replaced &lt;code&gt;reports/latest&lt;/code&gt; with an all-passing run. &lt;code&gt;--trial 3&lt;/code&gt; is the fourth trial: the CLI counts from zero, and the report counts from one.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm run eval:regression
npm run eval:review -- list
npm run eval:review -- adjudicate --task eligible-refund --trial 3 --score 0 --notes &lt;span class="s2"&gt;"Claimed refund, ledger empty."&lt;/span&gt;
npm run eval:review -- &lt;span class="kd"&gt;export&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Decisions are append-only, never edit the original run, and export as candidates for the judge's labeled set. When reviewers keep flagging the same unambiguous failure, it moves down the pyramid into a code check.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put each check on its clock
&lt;/h2&gt;

&lt;p&gt;The repository ships three GitHub Actions workflows, sorted by what a run costs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Every pull request:&lt;/strong&gt; format, types, lint, coverage, build, a dependency audit, the system test and the browser report check. Cheap and deterministic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Weekly:&lt;/strong&gt; &lt;code&gt;eval:ci&lt;/code&gt;, the release-gated repeated suite, with the report kept as an artifact.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Live, run by hand:&lt;/strong&gt; judge calibration, then &lt;code&gt;eval:live&lt;/code&gt; against a deployed system, using repository secrets for its URLs and the judge key.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Pin model versions, and record prompts, tool definitions and application versions in the run provenance. Otherwise a score changes without a code change and nobody can say why.&lt;/p&gt;

&lt;h2&gt;
  
  
  Adopting it
&lt;/h2&gt;

&lt;p&gt;Start from your own failures, not from this suite:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Write the job in product terms, with the evidence that proves each line, as in part one.&lt;/li&gt;
&lt;li&gt;Collect 20 to 50 real cases from tickets, bug reports and failed traces. &lt;a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents" rel="noopener noreferrer"&gt;Anthropic suggests that range&lt;/a&gt; as a starting point.&lt;/li&gt;
&lt;li&gt;Implement &lt;code&gt;SystemUnderTestFactory&lt;/code&gt; so each trial gets a fresh system and an observer that reads authoritative state. The system changes that state only through its tools, and can't fabricate the evidence.&lt;/li&gt;
&lt;li&gt;Give every requirement the cheapest grader that measures it faithfully. Add one judge criterion only where code runs out, and calibrate it.&lt;/li&gt;
&lt;li&gt;Run repeated trials, read every failure, and mark the tasks that can cause real harm as &lt;code&gt;critical&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Keep sampling production traffic. Offline evals protect the failures you already know about.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What this starter doesn't claim
&lt;/h2&gt;

&lt;p&gt;The example agent is rule-based and deterministic. Without the fault switch, it passes everything. Its job is to prove the harness, not to characterize a model. Real variance shows up when you connect a model-backed system through &lt;code&gt;--sut http&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The fault and the regression are teaching tools. They make a known failure reproducible, so you can watch each part of the harness respond to it.&lt;/p&gt;

&lt;p&gt;The metrics are point estimates from observed trials. The starter doesn't compute confidence intervals yet, so report &lt;code&gt;n&lt;/code&gt; next to every rate, as part two argues.&lt;/p&gt;

&lt;p&gt;The redactor catches common secret and email patterns. It can't promise that arbitrary private data is gone, so keep fixtures synthetic.&lt;/p&gt;

&lt;p&gt;If something here reads off or could be simpler, tell me in the comments. And if you adapt the starter to a real system, I'd like to hear which grader caught your first bug.&lt;/p&gt;

&lt;h2&gt;
  
  
  Glossary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Task fixture.&lt;/strong&gt; One versioned test case: input, expected result, severity and optional rubric.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;System under test.&lt;/strong&gt; The application being evaluated, behind an interface that receives only the task input.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Observer.&lt;/strong&gt; The harness component that reads outcome and tool-log evidence from the tool service, independent of the system.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Critical task.&lt;/strong&gt; A task where one failed trial blocks the release, whatever the averages say.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Release gate.&lt;/strong&gt; The explicit thresholds that decide whether an eval run passes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Review queue.&lt;/strong&gt; Failed, unknown and sampled passing trials waiting for a human decision.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  References and further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/bharadwaj-pendyala/eval-pyramid-starter" rel="noopener noreferrer"&gt;Eval Pyramid Starter&lt;/a&gt;. The TypeScript harness, tasks, workflows and docs used in this post.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents" rel="noopener noreferrer"&gt;Anthropic: Demystifying evals for AI agents&lt;/a&gt;. Tasks, trials, graders, isolated environments and starting dataset size.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2107.03374" rel="noopener noreferrer"&gt;Evaluating Large Language Models Trained on Code&lt;/a&gt;. The HumanEval paper and the &lt;code&gt;pass@k&lt;/code&gt; estimator.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2406.12045" rel="noopener noreferrer"&gt;&lt;code&gt;τ&lt;/code&gt;-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains&lt;/a&gt;. Final-state evaluation and the &lt;code&gt;pass^k&lt;/code&gt; metric.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://json-schema.org/learn/getting-started-step-by-step" rel="noopener noreferrer"&gt;JSON Schema: Getting started&lt;/a&gt;. The standard used to validate task fixtures and judge output.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://bharad.dev/blog/build-the-eval-pyramid" rel="noopener noreferrer"&gt;bharad.dev&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>agents</category>
      <category>typescript</category>
    </item>
    <item>
      <title>The Eval Pyramid, Part II: Measuring Agent Reliability with pass@k, pass^k, and Judges You Can Trust</title>
      <dc:creator>Bharadwaj P</dc:creator>
      <pubDate>Sat, 21 Nov 2026 16:17:18 +0000</pubDate>
      <link>https://dev.to/bharadwajpendyala/the-eval-pyramid-part-ii-measuring-agent-reliability-with-passk-passk-and-judges-you-can-trust-168o</link>
      <guid>https://dev.to/bharadwajpendyala/the-eval-pyramid-part-ii-measuring-agent-reliability-with-passk-passk-and-judges-you-can-trust-168o</guid>
      <description>&lt;p&gt;The refund agent from &lt;a href="https://bharad.dev/blog/eval-pyramid" rel="noopener noreferrer"&gt;part one&lt;/a&gt; passed its first trial. Order 42, $49 duplicate charge, one ledger row, confirmation sent. Every check green.&lt;/p&gt;

&lt;p&gt;So we ran it three more times. Three passed. The fourth reply said "Under the duplicate-charge policy, your $49 refund was processed and confirmed." The ledger had no refund in it.&lt;/p&gt;

&lt;p&gt;This post is about what to do with those four trials. &lt;a href="https://bharad.dev/blog/eval-pyramid" rel="noopener noreferrer"&gt;Part one&lt;/a&gt; set up the eval pyramid: graders ordered by what one verdict costs, with repeated trials as a separate axis behind every layer. Here we measure along that axis, then check the one grader in the stack that can be wrong in a convincing way. &lt;a href="https://bharad.dev/blog/build-the-eval-pyramid" rel="noopener noreferrer"&gt;Part three&lt;/a&gt; reproduces the whole run in code.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;A &lt;strong&gt;trial&lt;/strong&gt; is one attempt at one task. Pick &lt;code&gt;k&lt;/code&gt; from the product question. Pick &lt;code&gt;n&lt;/code&gt;, the number of trials you run, from how much evidence you need. They're different numbers.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pass@1&lt;/code&gt; is the chance one attempt succeeds. &lt;code&gt;pass@k&lt;/code&gt; is the chance at least one of &lt;code&gt;k&lt;/code&gt; succeeds. &lt;code&gt;pass^k&lt;/code&gt;, read "pass hat k", is the chance all &lt;code&gt;k&lt;/code&gt; succeed.&lt;/li&gt;
&lt;li&gt;Compute them per task, then average. A pooled 75% can describe two systems with completely different reliability.&lt;/li&gt;
&lt;li&gt;Report how much evidence sits behind a rate. Three passes in four trials is consistent with anything from about 30% to 95%.&lt;/li&gt;
&lt;li&gt;Check a model judge like an instrument: held-out expert labels, false accepts and false rejects, and how often it abstains.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Three numbers, three questions
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;pass@1&lt;/code&gt;&lt;/strong&gt; asks how often a single attempt works. In this series it's computed over every trial of a task, &lt;code&gt;c / n&lt;/code&gt;, so it's the per-trial success rate, not just the first attempt. For one customer sending one message, it's the closest number to their experience.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;pass@k&lt;/code&gt;&lt;/strong&gt; asks whether at least one of &lt;code&gt;k&lt;/code&gt; attempts succeeds. It fits products that can generate several candidates and check them, like a coding tool that runs the tests on ten solutions and keeps one that passes. The &lt;a href="https://arxiv.org/abs/2107.03374" rel="noopener noreferrer"&gt;HumanEval paper&lt;/a&gt; introduced the unbiased estimator most people use, and credits Kulal et al. (2019) with earlier use of the metric.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;pass^k&lt;/code&gt;&lt;/strong&gt; asks whether all &lt;code&gt;k&lt;/code&gt; attempts succeed. It fits products where every attempt has to work, because the same kind of request keeps arriving from different people. The &lt;a href="https://arxiv.org/abs/2406.12045" rel="noopener noreferrer"&gt;&lt;code&gt;τ&lt;/code&gt;-bench paper&lt;/a&gt; introduced it for agents and calls it "pass hat k".&lt;/p&gt;

&lt;p&gt;&lt;code&gt;k&lt;/code&gt; is the question. A support agent that can't retry invisibly cares about &lt;code&gt;pass^k&lt;/code&gt;. A tool with a verifier and a retry budget cares about &lt;code&gt;pass@k&lt;/code&gt;. Don't add invisible retries to an eval because they raise the score. A retry costs time, money, and sometimes a second unsafe action.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;n&lt;/code&gt; is the evidence. You might ask a &lt;code&gt;k = 3&lt;/code&gt; question with &lt;code&gt;n = 4&lt;/code&gt; trials or with &lt;code&gt;n = 400&lt;/code&gt;. The answer is the same kind of number. Your confidence in it isn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a stable 75% looks like
&lt;/h2&gt;

&lt;p&gt;Start with the textbook case. Suppose a task succeeds with a true, stable probability of 75% on every trial, and trials are independent. &lt;a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents" rel="noopener noreferrer"&gt;Anthropic's agent eval guide&lt;/a&gt; uses this same example, and it's worth seeing why:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;true probability p = 0.75, independent trials

pass@3 = 1 - (1 - 0.75)^3 = 98.4%
pass^3 = 0.75^3           = 42.2%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same agent, same three attempts. If you can keep trying, you'll almost always find one success. If three customers each get one attempt, fewer than half of those groups of three see it work every time.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqoac611rr4dmw4zqsorq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqoac611rr4dmw4zqsorq.png" alt="A line chart for a task with a true 75 percent success probability on every trial. Pass at k rises toward 100 percent while pass hat k falls toward zero as k increases." width="800" height="762"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;For a task with a true 75% success probability on every trial, asking about more attempts makes "at least one works" easy and "all of them work" hard. These are true probabilities, not estimates from trials.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Notice the assumption doing the work: one stable &lt;code&gt;p&lt;/code&gt;, the same for every trial. Real suites don't look like that.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same 75%, two opposite systems
&lt;/h2&gt;

&lt;p&gt;Take a suite of 100 tasks with a pooled success rate of 75%. Two very different systems produce that number.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;System A: every task succeeds 75% of the time
  pass@3 = 98.4%   pass^3 = 42.2%

System B: 75 tasks always succeed, 25 always fail
  pass@3 = 75.0%   pass^3 = 75.0%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both are true probabilities, averaged across tasks. System A is flaky everywhere. Retries rescue it, and consistency is poor. System B is perfectly consistent. It works on three quarters of the tasks every time, and retries don't help the rest at all.&lt;/p&gt;

&lt;p&gt;If you pool all trials into one bucket, you can't tell these apart. That's the main reason to compute &lt;code&gt;pass@k&lt;/code&gt; and &lt;code&gt;pass^k&lt;/code&gt; per task and then average. Two more reasons follow from it. Both estimators are defined within one task, so pooling across tasks mixes things the formulas don't describe. And averaging per task means each task counts once, whatever its trial count. Pooled across 100 trials of an easy task and 10 of a hard one, the easy task dominates. Averaged per task, it doesn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Back to order 42: an estimate from four trials
&lt;/h2&gt;

&lt;p&gt;Now the real run. Our numbers aren't true probabilities. They're estimates from &lt;code&gt;n = 4&lt;/code&gt; observed trials, &lt;code&gt;c = 3&lt;/code&gt; of them passing. The finite-sample estimators sample &lt;code&gt;k&lt;/code&gt; trials from the &lt;code&gt;n&lt;/code&gt; you observed, without replacement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;pass@k = 1 - C(n - c, k) / C(n, k)
pass^k = C(c, k) / C(n, k)

order 42, n = 4, c = 3, k = 3
pass@1 = 3 / 4                = 75%
pass@3 = 1 - C(1, 3) / C(4, 3) = 100%
pass^3 = C(3, 3) / C(4, 3)     = 25%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;C(a, b)&lt;/code&gt; is the number of ways to choose &lt;code&gt;b&lt;/code&gt; items from &lt;code&gt;a&lt;/code&gt;. The &lt;code&gt;pass@3&lt;/code&gt; line works because you can't pick three trials from four without getting at least one of the passes. The &lt;code&gt;pass^3&lt;/code&gt; line works because only one of the four possible groups of three avoids the failure.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;pass@3&lt;/code&gt; of 100% is accurate and nearly useless here. Among the four trials we observed, every group of three contains a pass. That says nothing certain about future attempts, and it describes a product that could secretly try three times and keep any success. A refund agent can't do that. The customer gets whatever the first trial did. &lt;code&gt;pass^3&lt;/code&gt; of 25% is the number that matches the product question.&lt;/p&gt;

&lt;p&gt;And four trials is thin evidence. Our four aren't random, because the fault in this demo is scheduled. But with a real model they would be, so imagine they were independent draws from one stable process. A 95% confidence interval (Wilson method) for 3 passes in 4 runs then goes from 30.1% to 95.4%. For comparison, 9 of 10 gives 59.6% to 98.2%, and 900 of 1,000 gives 88.0% to 91.7%. Same rate, very different certainty. Report &lt;code&gt;n&lt;/code&gt; next to every percentage, and add an interval when a decision depends on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The judge that would approve the bad trial
&lt;/h2&gt;

&lt;p&gt;The failed trial had a second lesson in it. Our groundedness criterion asks a model judge whether the reply explains the duplicate-charge policy correctly. On the failed trial it did. "Under the duplicate-charge policy, your $49 refund was processed and confirmed" is a clean, policy-accurate sentence. A judge that reads only the reply has every reason to pass it.&lt;/p&gt;

&lt;p&gt;That isn't a broken judge. It grades the text it's given, and the text described what was &lt;em&gt;supposed&lt;/em&gt; to happen. Only the ledger check, a code grader at the base of the pyramid, knew it hadn't happened. That's the division of labor working: the judge covers what code can't measure, and code covers what a well-written sentence can hide.&lt;/p&gt;

&lt;p&gt;Judges still need checking before they gate anything. The &lt;a href="https://arxiv.org/abs/2306.05685" rel="noopener noreferrer"&gt;MT-Bench paper&lt;/a&gt; found strong judges agreed with human preferences over 80% of the time, and also documented position, verbosity and self-enhancement biases. Part one covers why. Here's what checking one looks like:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Write one rubric per quality.&lt;/strong&gt; "Groundedness" and "tone" are two criteria with two prompts, not one blended score.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Get expert labels.&lt;/strong&gt; Domain experts label strong, weak and borderline examples pass or fail, with a short reason.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hold some labels back.&lt;/strong&gt; Tune the rubric on one set and measure the judge on another it's never seen. Otherwise you're measuring how well you fit the rubric to the examples.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Count mistakes by type, with the base stated.&lt;/strong&gt; A false accept is a judge pass on an example experts failed. A false reject is the opposite.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Track abstentions.&lt;/strong&gt; If the judge can say "unknown", report how often it does and what share of cases it actually decides. An abstention counts as a fail in the pass rule from part one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The false-accept rate is the one that hurts in a release gate, so state what it's a fraction of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;held-out set: 40 expert-labeled examples
  experts: 25 fail, 15 pass

false accepts: 3 of the 25 expert fails  = 12%
false rejects: 1 of the 15 expert passes = 6.7%
abstained:     2 of 40                   = 5%, judge decides 95%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those numbers are invented to show the arithmetic. With a 12% false-accept rate, roughly one in eight bad explanations gets through the judge. Whether that's acceptable depends on what sits under it. For order 42 it's tolerable, because the ledger check catches the case that costs money. For a criterion with nothing underneath, it isn't.&lt;/p&gt;

&lt;p&gt;Version all of it: judge model, prompt, rubric, labeled examples. A judge that passed calibration last month isn't calibrated for a new model or a new rubric.&lt;/p&gt;

&lt;h2&gt;
  
  
  The release decision
&lt;/h2&gt;

&lt;p&gt;Here's the full suite from that run: six tasks, four trials each, including the three adversarial ones from &lt;a href="https://bharad.dev/blog/build-the-eval-pyramid" rel="noopener noreferrer"&gt;part three&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;aggregate pass@1 = 91.7%    threshold 80%   pass
aggregate pass^3 = 75.0%    threshold 50%   pass
order 42 pass^3  = 25.0%
critical trials  = 18 of 20 threshold 100%  fail
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two tasks that move money each failed one trial, because the agent swallowed a payment error and reported success anyway. The numbers people usually watch, &lt;code&gt;pass@1&lt;/code&gt; and &lt;code&gt;pass^3&lt;/code&gt; across the suite, both clear their thresholds. The per-task number for order 42 says only one in four groups of three trials comes out clean.&lt;/p&gt;

&lt;p&gt;Two checks block the release. The deterministic pass rate requires every code check on every trial to pass, and it's at 91.7%. The critical gate allows zero failed trials on tasks that can move money. Either one alone would stop this run. The critical gate is the one that keeps stopping it when a team, sooner or later, relaxes the deterministic rate to tolerate a flaky check.&lt;/p&gt;

&lt;p&gt;The release is blocked. That's the right call, and the headline reliability numbers wouldn't have made it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://bharad.dev/blog/build-the-eval-pyramid" rel="noopener noreferrer"&gt;Part three&lt;/a&gt; reproduces this exact run from the open-source starter, finds the bug, fixes it and reruns the gate.&lt;/p&gt;

&lt;p&gt;If something here reads off or could be simpler, tell me in the comments.&lt;/p&gt;

&lt;h2&gt;
  
  
  Glossary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Trial.&lt;/strong&gt; One attempt by the system on one task.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;k&lt;/code&gt;.&lt;/strong&gt; The number of attempts at the same task in the product question, such as "will three attempts at this refund all succeed?"&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;n&lt;/code&gt;.&lt;/strong&gt; The number of trials actually run for a task. More trials, stronger evidence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;pass@1&lt;/code&gt;.&lt;/strong&gt; The per-trial success rate for a task, &lt;code&gt;c / n&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;pass@k&lt;/code&gt;.&lt;/strong&gt; The chance at least one of &lt;code&gt;k&lt;/code&gt; attempts succeeds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;pass^k&lt;/code&gt; (pass hat k).&lt;/strong&gt; The chance all &lt;code&gt;k&lt;/code&gt; attempts succeed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;False accept.&lt;/strong&gt; A judge pass on an example that experts failed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model judge.&lt;/strong&gt; A model prompted with a rubric to grade one quality of a trial.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  References and further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents" rel="noopener noreferrer"&gt;Anthropic: Demystifying evals for AI agents&lt;/a&gt;. Trials, &lt;code&gt;pass@k&lt;/code&gt; and &lt;code&gt;pass^k&lt;/code&gt;, isolated environments, and grader design.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2107.03374" rel="noopener noreferrer"&gt;Evaluating Large Language Models Trained on Code&lt;/a&gt;. The HumanEval paper and the &lt;code&gt;pass@k&lt;/code&gt; estimator.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2406.12045" rel="noopener noreferrer"&gt;&lt;code&gt;τ&lt;/code&gt;-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains&lt;/a&gt;. Final-state evaluation for agents and the &lt;code&gt;pass^k&lt;/code&gt; metric.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2306.05685" rel="noopener noreferrer"&gt;Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena&lt;/a&gt;. Judge agreement with humans, and documented biases.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval" rel="noopener noreferrer"&gt;Wilson score interval&lt;/a&gt;. The confidence interval used for the trial counts above.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://bharad.dev/blog/measuring-agent-reliability" rel="noopener noreferrer"&gt;bharad.dev&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>agents</category>
    </item>
    <item>
      <title>The Eval Pyramid, Part I: Testing Systems That Don't Behave the Same Way Twice</title>
      <dc:creator>Bharadwaj P</dc:creator>
      <pubDate>Sat, 21 Nov 2026 16:16:54 +0000</pubDate>
      <link>https://dev.to/bharadwajpendyala/the-eval-pyramid-part-i-testing-systems-that-dont-behave-the-same-way-twice-30b2</link>
      <guid>https://dev.to/bharadwajpendyala/the-eval-pyramid-part-i-testing-systems-that-dont-behave-the-same-way-twice-30b2</guid>
      <description>&lt;p&gt;A failing unit test usually points at one function. A failing agent run can point at the model, the prompt, a tool schema, a stale document, the test environment, or the grader itself. And the next run might pass.&lt;/p&gt;

&lt;p&gt;I spent the first half of my engineering career in test automation and SDET roles. Unit, integration and end-to-end tests weren't abstract categories. They were how we kept frequent releases from turning into frequent incidents. A Deep Atlas session on evaluating LLM applications pulled me back to that foundation, with one uncomfortable change: the system under test no longer behaves the same way twice.&lt;/p&gt;

&lt;p&gt;The testing pyramid still helps. But only if you're careful about what its layers mean.&lt;/p&gt;

&lt;p&gt;This is part one of a three-part series, and it follows one support ticket all the way through. &lt;a href="https://bharad.dev/blog/measuring-agent-reliability" rel="noopener noreferrer"&gt;Part two measures how often that ticket succeeds&lt;/a&gt;. &lt;a href="https://bharad.dev/blog/build-the-eval-pyramid" rel="noopener noreferrer"&gt;Part three builds the harness that runs it&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Write success in product terms, then name the evidence that proves it. "The reply says refunded" isn't evidence. The payment ledger is.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;grader&lt;/strong&gt; is whatever turns evidence into a verdict: code, a model, or a person. The eval pyramid stacks graders by what one verdict costs.&lt;/li&gt;
&lt;li&gt;Put most checks in code at the base. Use a model judge only for qualities code can't measure, and calibrate it against people.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;trial&lt;/strong&gt; is one attempt at a task. Repeating trials is a separate axis that applies to every grader, not a layer of its own.&lt;/li&gt;
&lt;li&gt;Decide what "pass" means before you count passes: every mandatory check passes, an unknown score counts as a fail, and a crashed trial stays in the denominator.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  One ticket, start to finish
&lt;/h2&gt;

&lt;p&gt;Here's the running case for the whole series. A customer, &lt;code&gt;customer-100&lt;/code&gt;, writes: "I was charged twice for order 42. Please refund the duplicate charge." The order cost $49. Store policy says a duplicate charge is always refundable.&lt;/p&gt;

&lt;p&gt;A model benchmark asks how capable a model is in general. This ticket asks something narrower. Does our system, with our tools, data and policies, do this job correctly? &lt;a href="https://openai.com/index/evals-drive-next-chapter-of-ai/" rel="noopener noreferrer"&gt;OpenAI draws the same line&lt;/a&gt; between frontier model evals and contextual evals built for one workflow. The thing under test isn't the model alone. It's the model, prompt, tools, retrieval, orchestration and environment working together.&lt;/p&gt;

&lt;p&gt;Write the success condition in product language first:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Refund exactly $49 on order 42, once.
Verify the customer owns order 42 before any money moves.
Send a confirmation.
Tell the customer what happened and why, accurately.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now ask, for each line, where the proof lives. The agent's reply is the obvious place to look and the worst one. An agent can write "your $49 refund was processed" after the payment call failed. The reply is a claim. The evidence is the state the payment provider recorded.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Refund $49 on order 42, once.&lt;/strong&gt; Evidence: one $49 refund row in the ledger. Grader: code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify ownership before the refund.&lt;/strong&gt; Evidence: the provider's tool log, in call order. Grader: code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Send a confirmation.&lt;/strong&gt; Evidence: one confirmation in the ledger. Grader: code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Explain the duplicate-charge policy.&lt;/strong&gt; Evidence: the reply text and the policy document. Grader: a calibrated model judge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Get the policy itself right.&lt;/strong&gt; Evidence: the policy, its edge cases, customer complaints. Grader: people.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two details in that list matter more than they look. The harness reads the ledger from the tool service, never from the agent's own report. And the agent never sees the expected answers. If the task fixture leaks into the prompt, the agent can pass by copying the answer key. Part three shows both rules in code.&lt;/p&gt;

&lt;h2&gt;
  
  
  What "pass" means
&lt;/h2&gt;

&lt;p&gt;Count nothing until the pass rule is written down. Most eval confusion I've seen starts here.&lt;/p&gt;

&lt;p&gt;For this ticket, a trial passes only when every mandatory check passes. One failed check fails the trial. A judge that returns "unknown" counts as a fail, because a missing verdict isn't a positive one. And a trial that crashes, say the payment provider times out and the harness can't finish, is a failed trial. It stays in the denominator. Dropping crashed runs from the count is the fastest way to report a reliability number nobody can reproduce.&lt;/p&gt;

&lt;p&gt;Some tasks also carry a severity. Anything that can move money is &lt;code&gt;critical&lt;/code&gt; here, so a single failed critical trial blocks a release no matter how good the average looks. Part two shows a run where the averages people usually watch say ship, and a customer was told "refunded" when no refund existed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two pyramids, two questions
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://martinfowler.com/articles/practical-test-pyramid.html#TheTestPyramid" rel="noopener noreferrer"&gt;Mike Cohn's testing pyramid&lt;/a&gt; has unit tests at the bottom, service tests in the middle and UI tests at the top. Its layers measure &lt;strong&gt;execution scope&lt;/strong&gt;: how much of the system runs to produce the evidence. A unit test runs one function. An end-to-end test runs everything.&lt;/p&gt;

&lt;p&gt;That axis still exists for LLM systems, and part three uses it. But the question that hurts most in evals is a different one. What does one &lt;em&gt;verdict&lt;/em&gt; cost, and who gives it?&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjt5ckodvupr5uqjzzu4q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjt5ckodvupr5uqjzzu4q.png" alt="The software testing pyramid, ordered by execution scope: many unit tests at the base, integration tests in the middle, and fewer end-to-end tests at the top. The eval pyramid, ordered by who gives the verdict: many code checks at the base, fewer calibrated model judges in the middle, and people at the top. Two faint copies of the pyramid sit behind it, labeled trial 1 through trial n, because repetition applies to every kind of grader. A dashed arrow from the top back to the base shows that people define and audit every layer." width="800" height="445"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The testing pyramid orders checks by how much of the system runs. The eval pyramid orders graders by who gives the verdict. Layer width shows intended coverage, not a cost guarantee. The faint outlines behind it are trials. Any grader can score repeated trials, so repetition isn't a layer. The dashed arrow is people defining and auditing every layer, top to bottom.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Code checks are cheap per verdict and give the same verdict every time for the same evidence. A model judge costs a model call per verdict and can disagree with itself. A person costs the most and scales the least. So you want many verdicts from code, fewer from models, and a few from people. Same economics Cohn pointed at, different axis.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running a trial and grading it cost different things
&lt;/h2&gt;

&lt;p&gt;Here's the distinction most eval pyramids blur. A ledger check is one of the cheapest verdicts you can buy. It's a comparison. But producing the ledger state means running the whole agent against a sandboxed payment provider, which can take seconds and real model tokens.&lt;/p&gt;

&lt;p&gt;So "deterministic" and "fast" aren't the same thing. The cost of &lt;em&gt;running&lt;/em&gt; a trial comes from execution scope. The cost of &lt;em&gt;grading&lt;/em&gt; it comes from the grader. A full agent run graded by a ledger comparison is expensive to run and cheap to grade. A one-line classifier output graded by a domain expert is the opposite.&lt;/p&gt;

&lt;p&gt;A second blur hides in the same place. &lt;strong&gt;Generation determinism&lt;/strong&gt; asks whether the system produces the same output twice. LLM systems often don't. &lt;strong&gt;Grading determinism&lt;/strong&gt; asks whether the grader returns the same verdict for the same evidence. A ledger comparison does. A judge might not. A stochastic agent can take three different valid paths to the same ledger state, and one code check can pass all three. That's usually the arrangement you want: a flexible agent, a crisp contract.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check the outcome first. Check the path when it's part of the job
&lt;/h2&gt;

&lt;p&gt;An agent run leaves two kinds of evidence. The &lt;strong&gt;outcome&lt;/strong&gt; is the final state of the environment, here the ledger. The &lt;strong&gt;trajectory&lt;/strong&gt; is the record of the run: messages, tool calls and intermediate results, in order.&lt;/p&gt;

&lt;p&gt;Grade the outcome by default. &lt;a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents" rel="noopener noreferrer"&gt;Anthropic recommends this&lt;/a&gt; over forcing agents down one path, because several paths are often valid. Comparing every run with one golden sequence rejects good behavior.&lt;/p&gt;

&lt;p&gt;But a correct ledger can still hide a bad run. Imagine the agent refunds order 42 before it checks who owns it. Today the customer matched, so the ledger looks perfect. Next week someone else's order gets refunded. When order, permissions or safety are part of the requirement, check the trajectory too, and read it from the provider's tool log. The agent's account of itself is one more claim. &lt;a href="https://docs.langchain.com/langsmith/trajectory-evals" rel="noopener noreferrer"&gt;LangChain's trajectory evaluators&lt;/a&gt; show the trade-offs between strict, unordered, subset and superset matching.&lt;/p&gt;

&lt;h2&gt;
  
  
  When code runs out
&lt;/h2&gt;

&lt;p&gt;"Explain the duplicate-charge policy accurately" doesn't collapse into a string comparison. Neither does "firm without sounding hostile". These are real requirements, and this is where a model judge earns its cost.&lt;/p&gt;

&lt;p&gt;A judge reads the task, the evidence and a rubric, then scores one quality. It handles reasonable variation better than exact matching. It also adds a second model to your system, with its own failure modes. The &lt;a href="https://arxiv.org/abs/2306.05685" rel="noopener noreferrer"&gt;MT-Bench paper&lt;/a&gt; found strong judges reached over 80% agreement with human preferences, about the level humans reach with each other. It also documented position, verbosity and self-enhancement biases, plus limited reasoning on some questions. Treat a judge as a measurement instrument that needs calibration. Part two shows how.&lt;/p&gt;

&lt;p&gt;People sit at the top. They write the rubric, label the examples a judge is checked against, settle ambiguous cases, and audit whether the automated verdicts still mean what the product needs. When they find a repeated failure, push it down: an unambiguous policy violation becomes a code check, a fuzzy one becomes a judge criterion with its own labeled examples.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other eval pyramids, and where this one differs
&lt;/h2&gt;

&lt;p&gt;I didn't invent the idea of mapping evals onto a testing pyramid. Several teams got there first, and they're worth reading:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://hamel.dev/blog/posts/evals/" rel="noopener noreferrer"&gt;Hamel Husain&lt;/a&gt; splits evals into unit tests, model and human evals, and A/B tests, run on different schedules because their costs differ. No pyramid, but the same cost logic.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://langwatch.ai/scenario/best-practices/the-agent-testing-pyramid" rel="noopener noreferrer"&gt;LangWatch&lt;/a&gt; stacks software tests, then evals, then agent simulations at the peak.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://engineering.block.xyz/blog/testing-pyramid-for-ai-agents" rel="noopener noreferrer"&gt;Block's agent testing pyramid&lt;/a&gt;, by Angie Jones, orders layers by how much uncertainty you're willing to tolerate, and repeats runs inside its upper layers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each one folds two axes into a single stack. A simulation layer says how much runs, not who grades it. Repeating runs only in the upper layers implies a code check doesn't need repetition, but a ledger check on a stochastic agent needs repeated trials as much as a judge does. My own first draft of this post made the same move and put "repeated task trials" as a layer between code checks and judges.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents" rel="noopener noreferrer"&gt;Anthropic's agent eval guide&lt;/a&gt; already keeps the terms apart. A task is one test case, each attempt is a trial, and a grader scores some part of it. It just doesn't use a pyramid. This series keeps that separation inside the pyramid picture: graders are the layers, trials sit behind every layer, and execution scope is its own axis. People still sample trials rather than read all of them. That's a coverage choice, not a different axis.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put each check as low as it can go
&lt;/h2&gt;

&lt;p&gt;The operating rule is the old one, applied to graders. Give each requirement to the cheapest grader that still measures what the requirement means. Then repeat the trials that matter, because one passing trial is one data point.&lt;/p&gt;

&lt;p&gt;For the order 42 ticket, a single trial passed every check. Good. Part two asks the question that matters before you ship: &lt;a href="https://bharad.dev/blog/measuring-agent-reliability" rel="noopener noreferrer"&gt;one pass, so how much evidence is enough?&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If something here reads off or could be simpler, tell me in the comments.&lt;/p&gt;

&lt;h2&gt;
  
  
  Glossary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Task.&lt;/strong&gt; One test case: an input plus the success criteria for it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trial.&lt;/strong&gt; One attempt by the system on one task.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grader.&lt;/strong&gt; Code, a model or a person that turns evidence into a verdict on some part of a trial.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Outcome.&lt;/strong&gt; The final state of the environment after a trial, such as a ledger row or a changed file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trajectory.&lt;/strong&gt; The ordered record of a trial: messages, tool calls and intermediate results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model judge.&lt;/strong&gt; A model prompted with a rubric to grade one quality of a trial.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Execution scope.&lt;/strong&gt; How much of the system runs to produce the evidence, from one function to the whole product.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  References and further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents" rel="noopener noreferrer"&gt;Anthropic: Demystifying evals for AI agents&lt;/a&gt;. Tasks, trials, graders, outcomes and transcripts, with practical guidance on agent evals.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://openai.com/index/evals-drive-next-chapter-of-ai/" rel="noopener noreferrer"&gt;OpenAI: How evals drive the next chapter in AI for businesses&lt;/a&gt;. The case for contextual evals built around a specific workflow.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://hamel.dev/blog/posts/evals/" rel="noopener noreferrer"&gt;Hamel Husain: Your AI Product Needs Evals&lt;/a&gt;. Three levels of evaluation, scheduled by cost.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://langwatch.ai/scenario/best-practices/the-agent-testing-pyramid" rel="noopener noreferrer"&gt;LangWatch: The Agent Testing Pyramid&lt;/a&gt;. Software tests, evals and simulations as a pyramid.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://engineering.block.xyz/blog/testing-pyramid-for-ai-agents" rel="noopener noreferrer"&gt;Angie Jones, Block Engineering: Testing Pyramid for AI Agents&lt;/a&gt;. Layers ordered by tolerated uncertainty.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://arxiv.org/abs/2306.05685" rel="noopener noreferrer"&gt;Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena&lt;/a&gt;. Judge agreement with humans, and the biases that come with it.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.langchain.com/langsmith/trajectory-evals" rel="noopener noreferrer"&gt;LangChain: Trajectory evaluations&lt;/a&gt;. Strict, unordered, subset and superset trajectory matching.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://martinfowler.com/articles/practical-test-pyramid.html" rel="noopener noreferrer"&gt;The Practical Test Pyramid&lt;/a&gt;. The testing model this post builds on.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://bharad.dev/blog/eval-pyramid" rel="noopener noreferrer"&gt;bharad.dev&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>agents</category>
    </item>
  </channel>
</rss>
