Twenty green ticks are satisfying. They are also a surprisingly small sample to turn into the word “always.” 🙂
Microsoft and Hugging Face's October 22 ThinkingBox post reports a useful contrast. After post-training, the number of benchmark tasks passing every recorded attempt rose from 38 to 107. Tasks solved at least once moved the other way, from 453 to 446. Each task was attempted 20 times. These are the authors' results, not a model run I reproduced. [1]
The distinction is worth keeping: finding a working path occasionally and repeating a working path are different properties. Neither observation makes future failure impossible.
Computed illustration, independently checked against SciPy's exact interval. Each row assumes independent attempts on one fixed task with an unchanged success probability; these are not confidence intervals fitted to ThinkingBox's campaign.
First, separate the denominators
An average success rate counts successful attempts. An at-least-once measure counts tasks with any success in their repeated trials. An all-attempts measure counts tasks with no observed failures. ThinkingBox reports all three because they answer different questions. [1]
A task that passes once and fails the remaining attempts contributes to coverage. It does not contribute to the all-pass count. That is why a system can gain repeatability while losing some breadth.
I would keep those columns together when choosing an agent for a repeated workflow. Selecting only the most flattering column discards part of the result. 💡
A streak is evidence, with uncertainty attached
Imagine one fixed task with an unchanged success probability of 0.9 on each independent attempt. The probability that all 20 attempts pass is:
0.9 ** 20 = 0.1215766546
So an all-green streak occurs about 12.16% of the time in this assumed scenario, even though the underlying failure probability is 10%. This is arithmetic under a model, not an estimate of ThinkingBox's agents.
Now suppose all you observed was 20 successes from 20 attempts. I checked a two-sided 95% exact binomial interval:
from scipy.stats import binomtest
interval = binomtest(20, 20).proportion_ci(
confidence_level=0.95, method="exact"
)
print(interval.low, interval.high)
The result was approximately 0.83157 to 1.0. SciPy uses the Clopper–Pearson exact method for this option. [2] 🧪
For the special case where every attempt passes, the lower endpoint also has a compact independent check:
lower = 0.025 ** (1 / n)
With n = 20, that gives the same endpoint. With 100 successes in 100 attempts, it is about 0.96378; with 1,000 in 1,000, about 0.99632. More observations narrow the uncertainty. A perfect observed fraction alone does not tell you how much evidence sits behind it.
The assumption doing the heavy lifting
⚠️ That interval assumes independent Bernoulli attempts with a fixed success probability. Shared infrastructure failures, persistent state or a changed agent can violate the setup. It is also a per-task illustration: pooling unrelated tasks and treating every attempt as one interchangeable coin flip needs justification.
A frequentist 95% confidence interval describes the long-run coverage of the interval procedure under its assumptions. It is not a 95% posterior probability that this particular task's unknown rate lies in the displayed range.
The calculation above does not overturn the reported ThinkingBox improvement. It explains why its careful wording, observed 20/20, matters. The data says what happened in the recorded attempts. A guarantee would have to say what must happen next. ✅
References
- Microsoft and Hugging Face: ThinkingBox post-training, October 22, 2026.
- SciPy: exact proportion confidence intervals.
Asset credit: original plot from computed binomial intervals. No benchmark figures were copied.
AI disclosure
AI was used to draft the article.
Top comments (0)