TL;DR
- If you pick an agent's action-gate threshold by sweeping a grid and keeping the argmax, the EV you report for that threshold is biased upward. It is a best-of-K number.
- In a seeded simulation with +1 / -9 payoffs (a costly-error action class), choosing the best of 21 thresholds on 100 held-out actions reported an average EV of +0.166 per action while the chosen threshold's true EV was only +0.045. The overstatement, 0.121, is larger than the best EV that exists in the whole problem (+0.0985).
- The bias fades with sample size (0.018 at n=1,000, 0.002 at n=5,000), but the damage to the choice of threshold fades more slowly, and in costly-error classes a bad choice is expensive.
- Three fixes: report on data the selection never saw, recalibrate the score and solve the break-even threshold instead of searching for it, and treat a wandering threshold as a signal that you need more labels, not a finer grid.
- All numbers below come from a synthetic generator (illustrative, NumPy, fixed seeds). They are not measurements of any production system.
A reader of my earlier post on action gates (thanks to @arhancanli) pointed out that the "best EV on held-out trajectories" I quoted is a best-of-21 number, so it runs high. They were right, and the effect is much bigger than a footnote once you move to per-action-class thresholds with expensive failures. This post reproduces the effect with my own generator (so the numbers are not comparable to theirs), shows where it bites, and ends with a recipe you can paste into an evaluation harness.
What is an action gate threshold, and why is it a statistic?
An action gate sits between an agent's proposed tool call and its execution. A scorer (a verifier model, a self-reported confidence, a rules engine that emits a number) outputs a score r in [0, 1]. The gate executes the action if r >= t and abstains (or escalates to a human) otherwise.
For a given action class the economics are simple. Executing a good action pays +a, executing a bad one costs -b, abstaining pays zero. If your score were a perfectly calibrated probability, executing is worth it exactly when
p * a - (1 - p) * b > 0 => p > b / (a + b)
With a = 1 and b = 9 the break-even probability is 0.9. Gate quality is then summarized as the expected value per proposed action:
EV(t) = mean( payoff_i * 1[r_i >= t] )
That sample mean is a statistic. Evaluate it at 21 thresholds on a finite set of labeled trajectories, keep the largest, and you have computed the maximum of 21 correlated noisy estimates. The maximum of noisy estimates is biased high. This is the winner's curse, the same mechanism that inflates the best of many hyperparameter configurations, the best of many prompts, or the best of N sampled completions.
Two separate things go wrong, and they are worth keeping apart:
- Reporting bias. The EV you write in the dashboard for the chosen threshold is too optimistic.
- Selection error. The threshold you chose is not the best one, so you ship a gate that is worse than the one you could have had.
Most discussions fix (1) and forget (2).
How big is the winner's curse for a costly-error action class?
I built a generator where the truth is known exactly, so every bias can be measured instead of argued about.
- A latent difficulty
z ~ Normal(1.0, 1.5)per proposed action. - The action is good with probability
p = sigmoid(z). - The scorer is overconfident: it emits
r = sigmoid(1.6 * z). That is realistic, since raw model confidences tend to run high (Guo et al., "On Calibration of Modern Neural Networks", arXiv 1706.04599, is the standard reference). - Payoffs are
+1and-9. - The true
EV(t)is computed by numerical integration overz, not by sampling, so the ground truth carries no noise.
The oracle gate threshold in raw-score terms is 0.9711 (the score at which the true success probability equals 0.9), with a true EV of +0.0985 per proposed action and a pass rate of about 21%. Plugging 0.90 straight into the raw score, which is what you do if you trust the number as a probability, yields a true EV of only +0.0066: among actions with raw score above 0.90 the mean score is 0.966, but the true success rate is 0.902.
The curve above is the true EV across 21 thresholds from 0.50 to 0.99. It is deeply negative for a lenient gate, crosses zero near 0.90, peaks near 0.97, and drops again at 0.99 because too many good actions are refused.
Now the experiment. For each sample size n, draw n labeled actions, evaluate EV at the 21 thresholds, take the argmax, and record what you would report versus what is true. Repeat 3,000 times (1,000 for n=5,000).
| Held-out actions (n) | Mean reported EV | Mean true EV of chosen threshold | Overstatement | Runs choosing a money-losing threshold |
|---|---|---|---|---|
| 100 | +0.166 | +0.045 | +0.121 | 19.4% |
| 300 | +0.121 | +0.074 | +0.047 | 5.0% |
| 1,000 | +0.105 | +0.087 | +0.018 | 0.4% |
| 5,000 | +0.098 | +0.095 | +0.002 | 0.0% |
Read the first row twice. With 100 labeled actions in the class, the average reported EV is 68% higher than the best EV that is physically achievable (0.166 versus 0.0985). A dashboard built this way would show a gate beating the theoretical optimum. And roughly one run in five picks a threshold whose true EV is negative, meaning the gate loses money on average while its own report says it earns about 0.17 per action.
At n=5,000 the problem looks solved. But per-action-class thresholds slice your data. An agent with 30 tool types and a few thousand logged trajectories has a handful of classes with hundreds of labels and several with dozens. The high-stakes classes (payments, deletions, outbound messages) are exactly the ones where failures are rare, so labeled failures are scarcer still.
Why does the chosen threshold wander so much?
With +1 / -9 payoffs, the EV curve is steep on the lenient side and fairly flat near the top. A handful of labeled failures near the optimum shifts which grid point wins.
At n=100, the 10th to 90th percentile of the chosen threshold ran from 0.87 to 0.99. At n=300 it narrowed to 0.92 to 0.99, at n=1,000 to 0.94 to 0.99, and at n=5,000 it settled on 0.97. Anything from "gate on a score of 0.87" to "gate on 0.99" is a different product: in this generator the first passes about 45% of proposed actions and the second about 11%.
If you retrain the scorer weekly and re-sweep thresholds on a small replay set, the chosen threshold will drift week to week even when nothing changed. Teams read that drift as a model change. It is mostly selection noise.
Fix 1: How do I report an honest EV? Select on one slice, report on another
The cheapest repair for reporting bias is to separate the data that picks the threshold from the data that grades it. Pick t* on half A, then compute EV(t*) on half B.
In the simulation this removes reporting bias: the honest estimate on the held-out half was off by +0.002, -0.002, -0.0002 and +0.0007 at the four sample sizes, i.e. noise around zero.
But it costs you selection quality, because you chose the threshold with half the data. Mean regret versus the oracle threshold (EV lost per action) was 0.094 at n=100, versus 0.053 for the naive argmax on all the data. Splitting buys an honest report at the price of a worse gate. Use it when the report is what matters (a go/no-go decision, a number someone will sign), then refit on all the data for the final choice and quote the split estimate as the honest figure.
Fix 2: How do I choose the threshold without searching? Recalibrate, then solve
The grid search is the root cause. It uses the data to rank 21 candidates, and every candidate's score is noisy. If you know the payoffs, the optimal decision is determined by a probability, and fitting one smooth calibration curve uses all the data at once.
The recipe:
- Fit a two-parameter logistic map from the raw score to the observed outcome:
p_hat = sigmoid(w0 + w1 * logit(r)). This is Platt-style recalibration (with a single slope parameter it becomes temperature scaling). - Solve
p_hat = b / (a + b)forr. Thatris your threshold.
No grid, no argmax, no best-of-K.
In the simulation this was the clear winner on selection error:
| n | Argmax on all data (regret) | Select on half, report on half (regret) | Recalibrate + solve (regret) |
|---|---|---|---|
| 100 | 0.0535 | 0.0938 | 0.0168 |
| 300 | 0.0246 | 0.0389 | 0.0058 |
| 1,000 | 0.0115 | 0.0175 | 0.0018 |
| 5,000 | 0.0031 | 0.0068 | 0.0003 |
Regret is the EV lost per action relative to the oracle threshold (0.0985). At n=100 the recalibrated gate loses about a third as much as the naive argmax, and at n=1,000 about six times less.
The caveat matters: the simulation's truth is exactly a logistic function of the score, so a logistic recalibrator is correctly specified here. Real scorers are not that kind. If the true calibration curve is non-monotone or has a cliff, a two-parameter logistic can be wrong in a systematic way that no amount of data repairs. Use isotonic or binned calibration with a monotonic constraint when you have the labels for it, check the fit with a reliability diagram, and keep the logistic as the low-data fallback.
Fix 3: What should I do when the threshold keeps moving? Treat it as a labeling signal
Selection noise is itself a measurement. Resample your held-out set with replacement a few hundred times, redo the whole selection pipeline on each resample, and look at the spread of the chosen threshold and of the EV at the chosen threshold. This is the bootstrap done around the selection step, not around the final number.
- If the 10th to 90th percentile of the chosen threshold covers a range that matters to the product (in the n=100 case here, 0.87 to 0.99), you do not have enough labels in that class to set a threshold. Fall back to a conservative default and route more of that class to human review until labels accrue.
- Report the EV from the bootstrap lower quantile, not the mean, for classes where the downside is costly.
- Pool across related classes with a hierarchical prior (or a shared calibration slope with per-class offsets) before you give each class its own threshold.
What does a reproducible core look like?
This is the essential part of the simulation, trimmed for readability. It uses only NumPy.
import numpy as np
A, B = 1.0, 9.0
GRID = np.linspace(0.50, 0.99, 21)
sig = lambda x: 1 / (1 + np.exp(-x))
logit = lambda p: np.log(p / (1 - p))
def draw(rng, n, slope=1.6):
z = rng.normal(1.0, 1.5, n)
good = rng.random(n) < sig(z) # truth
r = sig(slope * z) # overconfident raw score
return r, np.where(good, A, -B)
def ev_curve(r, pay):
return np.array([(pay * (r >= t)).mean() for t in GRID])
def naive(r, pay): # best-of-21: biased report, noisy choice
c = ev_curve(r, pay); k = c.argmax()
return GRID[k], c[k]
def split_report(r, pay): # honest report for the chosen threshold
h = len(r) // 2
k = ev_curve(r[:h], pay[:h]).argmax()
return GRID[k], ev_curve(r[h:], pay[h:])[k]
def recalibrated_threshold(r, pay, iters=25):
x = logit(np.clip(r, 1e-6, 1 - 1e-6)); y = (pay > 0).astype(float)
X = np.stack([np.ones_like(x), x], 1); w = np.zeros(2)
for _ in range(iters): # logistic regression by Newton steps
p = sig(X @ w)
H = (X * (p * (1 - p))[:, None]).T @ X + 1e-6 * np.eye(2)
w += np.linalg.solve(H, X.T @ (y - p))
return float(sig((logit(B / (A + B)) - w[0]) / w[1]))
To use it on real logs, replace draw with your labeled trajectories (score, realized payoff) per action class. Keep the label definition strict: the payoff must be the outcome of executing the action, which you only observe for actions that actually passed the gate. That is a separate and well-known selection problem (you never see outcomes for abstained actions), so add a small randomized exploration slice, or use counterfactual labels from a sandbox replay, before you trust any threshold fitted on production logs.
Where does this fit with the rest of agent safety evaluation?
Gates are one layer, and published evaluations suggest the layers beneath them are not yet reliable on their own. Agent-SafetyBench (Zhang et al., arXiv 2412.14470) evaluated 16 LLM agents across 349 environments and 2,000 test cases and reported that none scored above 60% on its safety metric. More recent work such as JANUS (Xiong et al., arXiv 2607.19913, July 2026) trains guard models to forecast delayed risk from partial action sequences and intervene before an unsafe action executes, reporting a 15.9 percentage point improvement in protection over baselines.
Whatever guard sits behind the score, the thresholding step is where a numeric score becomes a policy, and it is the step most often tuned on the least data. A better guard model does not repair a biased threshold, and a biased threshold can erase a better guard model's gains.
FAQ
Is the winner's curse different from overfitting?
It is the same family. Overfitting is fitting noise in the training data. The winner's curse is the specific bias in the reported value of whichever candidate won a noisy comparison. With 21 grid points and a single threshold parameter, the model is not overfit in the classical sense, yet the reported number is still biased.
Does a bigger grid make it worse?
Generally yes, because you take the maximum over more noisy estimates. A finer grid on a small set also makes the chosen threshold jumpier. If you want fine resolution, smooth the curve first (calibration plus a closed-form threshold) instead of searching it.
Does this matter when payoffs are symmetric?
Much less in principle. The bias scales with the noise in the estimate, and a +1 / -1 class has a lower-variance payoff than a +1 / -9 class. I did not run the symmetric case in this post, so I make no numeric claim for it.
How many labels per class do I need?
There is no universal number. In this generator, a few hundred labels brought the money-losing-choice rate to about 5%, and about a thousand to under 0.5%. Your scorer, payoffs and base rates move that, so run the bootstrap-around-selection check on your own data instead of borrowing these figures.
Can I just use a conservative threshold and skip all this?
A conservative threshold is the right fallback for low-data classes. In the simulation, moving from the optimum (0.97) to 0.99 cut the true EV from +0.097 to +0.073 and the pass rate from about 21% to 11%. The safety is real but so is the cost, which is why you want to measure the trade-off instead of guessing it.
Are these numbers from a real agent?
No. They come from a synthetic generator with a known truth, chosen so the bias can be measured exactly. Treat the magnitudes as illustrative and the mechanism as general.
Top comments (0)