Robot Policy Evaluation: Why 90% vs 92% Proves Little

Abstract

When evaluating robot control policies, many practitioners draw direct conclusions from simple success‑rate percentages. For instance, given Policy A with 90 % success and Policy B with 92 % success, people frequently claim Policy B performs better. Nevertheless, purely comparing percentage figures without sample size, confidence intervals, paired experimental design and statistical power analysis often produces unreliable judgments. Drawing on Clopper‑Pearson exact confidence intervals, Wilson score intervals, McNemar’s paired testing and hierarchical episode‑within‑task structure, this article lays out a complete practical workflow for robot policy evaluation, covering pre‑experiment planning and post‑hoc result checking. For engineering teams running robot‑simulation benchmarks mixed with LLM‑based agent workloads, an API gateway such as 4sapi can help standardize telemetry collection and multi‑backend request orchestration.

1. The Pitfall: Percentages Without Sample Sizes Lack Evidentiary Weight

Statements such as “Policy A achieves 90 % success; Policy B achieves 92 % success” are ubiquitous in robotics papers and technical reports. However, these two numbers alone cannot support the conclusion that Policy B is stronger. Valid interpretation must account for roll‑out count, task composition, random seeds, paired‑group configuration and statistical power.

The RoboLab v4 benchmark illustrates this concrete risk. Each policy runs only 10 episodes per task. Under this setup, when a policy reaches a 90 % success rate, its 95 % confidence interval spans approximately 19 percentage points. Even expanding to 100 roll‑outs, the interval width still sits near six percentage points. Authors explicitly classify 10‑episode runs as coarse‑grained indicators and warn that fine‑grained policy comparison remains untrustworthy. This warning generalizes across most high‑cost robot benchmarks: reported numbers may print with high numerical precision, yet real statistical certainty can be very low.

Five core practical take‑aways are summarized below:

1. Always report raw counts in the form k/n, rather than percentages in isolation.
2. Confidence intervals quantify plausible ranges for true underlying success probability; they do not serve as guarantees for future performance.
3. Comparing independent samples: 90/100 versus 92/100 yields a 95 % confidence interval for difference roughly from −5.9 pp to +9.9 pp. This interval crosses zero, so we cannot rule out that Policy A could actually be better.
4. If testing two policies on identical tasks and seeds, prefer paired comparison instead of fully independent sampling.
5. To reliably resolve a true difference of ±2 pp near the 90 % success level usually requires thousands of roll‑out trials. Detecting a real gap between 90 % and 92 % generally demands even larger sample sizes.

2. Confidence‑Interval Methods for Bernoulli Roll‑out Data

Robot roll‑out outcomes follow Bernoulli distributions: each episode ends in binary success or failure. Two widely adopted interval approaches are Clopper‑Pearson exact interval and Wilson score interval.

2.1 Clopper‑Pearson Exact Confidence Interval

Given n independent Bernoulli trials with k successes, the two‑sided (1-\alpha) Clopper‑Pearson interval computes bounds using Beta‑distribution quantiles:
[
\text{lower}=B_{\text{Beta}}(\alpha/2;\ k,\ n-k+1),\quad
\text{upper}=B_{\text{Beta}}(1-\alpha/2;\ k+1,\ n-k)
]
Special boundary handling is required for edge cases where (k=0) or (k=n). This method guarantees coverage probability no less than the nominal confidence level, which makes it “exact”, but it tends to produce comparatively conservative, wider intervals.

Example calculation: for 70 total roll‑outs and 63 successful episodes, observed success = 63/70 = 90.0 %. The 95 % Clopper‑Pearson interval becomes approximately 80.48 % – 95.88 %. The corresponding Wilson interval reads 82.77 % – 95.07 %. This example is purely for statistical demonstration and does not come from RoboLab published experimental data.

Sample Python implementation using scipy.stats.beta:

from scipy.stats import beta

def clopper_pearson(k: int, n: int, alpha: float = 0.05) -> tuple[float, float]:
    if k <= 0:
        return 0.0, 1.0 - beta.ppf(alpha, 1, n)
    if k >= n:
        return beta.ppf(alpha, n, 1), 1.0
    lower = beta.ppf(alpha / 2, k, n - k + 1)
    upper = beta.ppf(1 - alpha / 2, k + 1, n - k)
    return float(lower), float(upper)

2.2 Wilson Score Interval for Default Reporting

The normal‑approximation Wald interval performs poorly for small samples or proportions close to 0 or 1. The Wilson score interval comes from inverting the score test. It delivers better statistical coverage and narrower interval width and is recommended as the default for benchmark dashboards. Clopper‑Pearson can be kept as a conservative alternative option.

Important practical reminder: No interval technique can magically turn small‑sample noisy measurements into strong evidence. Even well‑computed intervals will remain wide when available data are scarce.

3. Practical Comparison: 90/100 against 92/100

Take two independent groups:

• Policy A: 90 successes out of 100 roll‑outs
• Policy B: 92 successes out of 100 roll‑outs

Point estimate for difference: +2.0 percentage points. The 95 % confidence interval for difference in independent‑proportions test runs from −5.93 pp up to +9.93 pp. Since zero lies inside this range, these 100‑versus‑100 independent samples cannot reliably order the two policies. The interval admits possibilities that A is better, B is better, or the two are effectively equivalent.

Simple visual overlap between two separate single‑group confidence intervals is not a formal statistical test for difference. Direct interval calculation on the difference of proportions is the correct workflow.

4. Paired Evaluation Design for Robot Benchmarks

Robot policy experiments gain substantial statistical power when adopting paired‑outcome design. When Policy A and Policy B run on identical tasks, scene layouts, camera noise, and random seeds, each trial yields four possible joint outcomes:

1. A succeeds, B succeeds
2. A fails, B fails
3. A succeeds, B fails (A exclusive win)
4. A fails, B succeeds (B exclusive win)

Only the latter two discordant cases carry information for comparing relative strength. Concordant outcomes (both succeed or both fail) do not contribute evidence for superiority of either algorithm. Statistical tools such as the McNemar test or paired bootstrap focus precisely on these discordant cells.

Paired designs cannot fix all experimental defects. If task‑level confounding factors exist, or if correlation between paired runs becomes weak, the power gain diminishes. Researchers must still report full joint contingency tables instead of only aggregated per‑policy success percentages.

Hierarchical structure: episodes nested inside tasks

Robot benchmark datasets commonly contain multi‑level hierarchy: multiple roll‑out episodes belong to one high‑level task definition. Episodes within the same task are not fully independent observations. Treating every episode as an independent sample artificially inflates effective sample size and produces over‑optimistic p‑values.

Valid analytical strategies for nested data include:

1. Compute differences per task, then perform bootstrap resampling over tasks.
2. Multi‑level hierarchical bootstrap: resample tasks first, then resample episodes inside each sampled task.
3. Report both macro‑averaged (task‑weighted) and micro‑averaged (episode‑weighted) success metrics.

Blindly applying naive bootstrap across raw episodes risks ignoring task‑level correlation and yields misleading results.

5. Estimation Precision versus Difference‑Detection Power

Two distinct statistical questions must not be conflated:

1. Estimate the true success rate of one single policy within ±2 pp absolute error.
2. Detect whether two different policies differ by a true gap of 2 pp.

These two goals demand very different sample sizes. To estimate a single proportion near 90 % to ±2 pp 95 % confidence requires roughly 1 000 roll‑outs. To reliably detect a true 2 pp difference between two competing policies under independent‑sample setup needs about 3 200 total roll‑outs. This number increases further when working with paired designs and low discordant rates.

Cohen’s h effect‑size metric helps quantify differences between two proportions. Power calculation should be completed before running experiments, not after collecting results. Post‑hoc power computation on already‑observed data is widely discouraged within statistics practice.

6. Sample‑Size Planning Starting from Decision Requirements

Sample‑size planning should originate from practical engineering decisions:

1. Define the minimal meaningful effect (minimum detectable effect, MDE), for example a 5‑percentage‑point gap rather than arbitrarily targeting ±2 pp.
2. Choose whether evaluation will use independent‑group or paired design.
3. Clarify whether comparisons run across multiple tasks or multiple capability categories.
4. Set target statistical significance threshold and statistical power level.
5. Compute required sample quantity, allocate roll‑out budget across tasks and perturbation conditions.
6. Execute benchmark and report confidence intervals together with raw k/n counts.

If the required sample size exceeds available compute budget, practitioners may increase MDE thresholds, adopt stronger paired designs, reduce redundant comparisons, or accept higher uncertainty and treat outcomes as exploratory evidence.

7. Common Pitfalls in Robot‑Policy Benchmarking

Frequent statistical mistakes seen in robotics evaluation:

• Publishing percentages without attaching underlying raw k/n counts.
• Judging difference superiority purely from visual overlap of individual‑group confidence intervals.
• Treating nested per‑episode roll‑outs as fully independent samples without hierarchical consideration.
• Running comparisons until p‑value crosses significance threshold (p‑hacking via repeated intermediate checking).
• Applying post‑hoc power analysis on already‑collected experimental data.
• Ignoring that paired designs rely on discordant cases and overall success rates alone cannot tell the full story.

8. Conclusion

A measured gap of two percentage‑points between 90 % and 92 % success rate rarely delivers strong evidence for algorithmic superiority. Benchmark conclusions depend heavily on raw sample counts, confidence‑interval ranges, paired‑versus‑independent setup, hierarchical task‑episode nesting, and pre‑computed statistical power. Simply reading off aggregated percentage numbers creates substantial risk of drawing wrong engineering conclusions. Robot researchers should report raw trial counts, confidence intervals, and full contingency tables, and complete sample‑size planning before launching expensive physical or simulated robot roll‑out campaigns.

International access: https://4sapi.com
Domestic access: https://4sapi.cn

Comments

[Ahmet Özel]

The paired design point is the one that changes conclusions most often in practice, and it generalises well beyond robotics - the same argument applies to any LLM eval where people compare two aggregate scores from separately sampled runs. Running both policies on the identical episode set and testing the disagreements is what turns a two-point gap into evidence, because the variance between tasks usually dwarfs the variance between policies. The hierarchical structure deserves the emphasis too: episodes within a task are not independent, so treating N episodes as N samples inflates your effective sample size and narrows the interval to something you have not earned. That is how a benchmark ends up reporting significance for a difference that vanishes on the next seed.