This is Part 3 of a 5-part series on evaluating and hardening production AI agents. See also: Part 1: Moving Beyond LLM-as-a-Judge, Part 2: Deterministic State Evaluations, Part 4: Curating Golden Trajectories, and Part 5: Building an Agent CI/CD Pipeline.
In AI engineering discussions, teams frequently report evaluation metrics with unfounded precision:
"Our customer support agent achieved an 85% task completion rate on our benchmark suite."
When you inspect the benchmark under the hood, you discover that the suite consists of 20 test cases, and the model completed 17 of them ($17 / 20 = 85.0\%$).
To anyone trained in actuarial science or biostatistics, this claim is an illusion.
A point estimate without a confidence interval is not a measurement—it is an anecdote. When deploying autonomous agents that touch live databases, issue payments, or interact with customers, reporting point averages on small sample sizes creates false security.
Here is why most AI evaluation benchmarks fail basic statistics, how to compute robust confidence intervals for Bernoulli trials, and how to determine the exact sample size needed to detect regressions.
The Small-Sample Illusion
Every evaluation of an agent on a deterministic task is a Bernoulli trial: the agent either achieves the goal state ($X = 1$) or it fails ($X = 0$).
When engineering teams calculate a confidence interval, they often rely on the standard normal approximation (the Wald interval):
$$\hat{p} \pm z_{1-\alpha/2} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$
The Wald interval makes a critical assumption: that the sampling distribution is approximately normal. For small sample sizes ($n < 100$) or when success rates approach the boundaries ($p > 0.85$ or $p < 0.15$), the Wald interval breaks down catastrophically. It produces intervals that spill outside $[0, 1]$ and drastically underestimates uncertainty.
Enter the Wilson Score Interval
For binomial proportions in high-stakes systems, we use the Wilson score interval. The Wilson interval inverts the score test and performs reliably across all sample sizes and probability ranges:
$$w = \frac{\hat{p} + \frac{z^2}{2n} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$
Where:
- $\hat{p}$ is the observed sample proportion ($x / n$)
- $n$ is the total number of test runs
- $z$ is the standard normal quantile ($z = 1.96$ for a 95% confidence interval)
What the Numbers Actually Say
Let us apply the Wilson score interval to typical benchmark claims:
-
Case A: 17 successes out of 20 runs ($\hat{p} = 85.0\%$)
- Point estimate: 85.0%
- 95% Wilson Confidence Interval: [63.9%, 94.8%]
- Reality: The true production capability could be a failing 64% or a strong 95%. Your test suite cannot distinguish between a mediocre model and a production-grade model.
-
Case B: 85 successes out of 100 runs ($\hat{p} = 85.0\%$)
- Point estimate: 85.0%
- 95% Wilson Confidence Interval: [76.9%, 90.6%]
- Reality: The uncertainty window narrows, but still spans a 14-percentage-point margin of error.
-
Case C: 425 successes out of 500 runs ($\hat{p} = 85.0\%$)
- Point estimate: 85.0%
- 95% Wilson Confidence Interval: [81.6%, 87.9%]
- Reality: Only at $n = 500$ do we have statistical proof that the true reliability exceeds 80%.
How to Calculate Wilson Score Intervals in Python
Here is a lightweight, zero-dependency helper to compute exact confidence bounds for evaluation suites:
import math
def wilson_score_interval(successes: int, total: int, confidence: float = 0.95) -> tuple[float, float]:
"""
Compute the Wilson score confidence interval for a binomial proportion.
"""
if total == 0:
return (0.0, 0.0)
# z-value for two-tailed standard normal distribution
# 0.90 -> 1.645, 0.95 -> 1.960, 0.99 -> 2.576
z_map = {0.90: 1.645, 0.95: 1.95996, 0.99: 2.57583}
z = z_map.get(confidence, 1.95996)
p_hat = successes / total
denominator = 1.0 + (z**2) / total
centre_adjusted_probability = p_hat + (z**2) / (2 * total)
adjusted_standard_deviation = math.sqrt((p_hat * (1 - p_hat) + (z**2) / (4 * total)) / total)
lower_bound = (centre_adjusted_probability - z * adjusted_standard_deviation) / denominator
upper_bound = (centre_adjusted_probability + z * adjusted_standard_deviation) / denominator
return max(0.0, lower_bound), min(1.0, upper_bound)
Whenever your CI evaluation pipeline completes, report both the point estimate and the interval:
successes, total = 82, 100
low, high = wilson_score_interval(successes, total, confidence=0.95)
print(f"Agent Task Success: {successes/total:.1%} [95% CI: {low:.1%} – {high:.1%}]")
# Agent Task Success: 82.0% [95% CI: 73.4% – 88.2%]
Sample Sizing: How Many Tests Do You Actually Need?
When architecting an evaluation suite, the fundamental engineering constraint is: How many test runs are required to detect a regression?
If your baseline agent has an 85% success rate, and a new prompt or model fine-tune drops performance to 78%, you need sufficient statistical power to detect that 7% drop without false alarm.
To detect a difference between two proportions ($\delta = p_1 - p_2$) with significance level $\alpha$ (typically 0.05) and statistical power $1 - \beta$ (typically 0.80):
$$n \approx \frac{\left(z_{1-\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)}\right)^2}{\delta^2}$$
Practical Rule of Thumb for Engineering Teams
- To reliably detect a 10% shift (e.g., 85% vs. 75%): You need roughly ~180 runs per variant.
- To reliably detect a 5% shift (e.g., 85% vs. 80%): You need roughly ~750 runs per variant.
If your benchmark suite only evaluates 30 test cases on a pull request, you literally cannot detect a 10% regression. An apparent drop from 26/30 (86.7%) to 23/30 (76.7%) is statistically indistinguishable from chance ($p > 0.30$).
Pass@k vs. Pass^k: Disentangling Temperature from Capability
Agents are non-deterministic systems. When evaluating models at temperature $T > 0$, single-run evaluations conflate sampling noise with model capability.
We separate evaluation into two distinct metrics:
Pass@k (Exploration / Ceiling):
Does the agent solve the task at least once across $k$ independent attempts?
This measures capability ceiling—whether the model can formulate a viable trajectory.Pass^k (Reliability / Production Readiness):
Does the agent solve the task every single time across $k$ consecutive runs?
$$P(\text{pass}^k) = p^k$$
For enterprise workflows, Pass^3 or Pass^5 is the real deployment gate. If an agent cannot complete a workflow 3 times out of 3 without failure, it is not ready for unattended operation.
The Next Step
Now that we have established deterministic state testing and statistical sample sizing, where do we get the actual test cases?
In Part 4, we explore Curating Golden Trajectories: how to curate adversarial edge-case datasets, simulate tool failures, and stress-test agent reasoning before users do.
Need Actuarial Rigor in Your AI Testing?
Most AI consulting groups give generic prompt advice. We approach AI evaluation from applied statistics and systems engineering.
If you need to establish statistical benchmarks, design sample-efficient evaluation pipelines, or prepare your models for production compliance:
- Embed our team via Hacker-in-Residence
- Read about our Model Red-Teaming & Reliability Services
- Browse the latest engineering breakdowns on Artists of Intelligence
Frequently asked questions
Why does an 85% score on 20 AI agent test cases mean little?
A point estimate without a confidence interval is an anecdote. For 17 successes out of 20 runs, the 95% Wilson confidence interval is 63.9% to 94.8%, so the true capability could be a failing 64% or a strong 95%. The suite cannot tell a mediocre model from a production-grade one.
Why use Wilson score intervals instead of the Wald interval for agent evaluations?
The Wald interval assumes a roughly normal sampling distribution, which breaks down for sample sizes under 100 or success rates above 0.85 or below 0.15. It can spill outside 0 to 1 and underestimates uncertainty. The Wilson interval performs reliably across all sample sizes and probability ranges.
How many test runs do I need to detect a regression in an AI agent?
As a rule of thumb, detecting a 10% shift such as 85% versus 75% needs roughly 180 runs per variant, and a 5% shift such as 85% versus 80% needs roughly 750. A 30-case suite cannot detect a 10% regression, since 26/30 versus 23/30 is statistically indistinguishable from chance.
What is the difference between Pass@k and Pass^k for evaluating AI agents?
Pass@k asks whether the agent solves the task at least once across k attempts, measuring capability ceiling. Pass^k asks whether it solves the task every time across k consecutive runs, equal to p to the power k. For enterprise workflows, Pass^3 or Pass^5 is the real deployment gate.
How narrow does the confidence interval get with 85% success at larger sample sizes?
At 85 successes out of 100 runs, the 95% Wilson interval is 76.9% to 90.6%, a margin of about 14 percentage points. At 425 out of 500 it narrows to 81.6% to 87.9%, and only at n = 500 is there statistical proof that true reliability exceeds 80%.
This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack.
Top comments (0)