Ward Ed

Posted on Oct 20

Do 66 Terminal Tasks and a 45% Private Weight Settle AI Rankings? Reading Artificial Analysis Index v4.3

#ai #machinelearning #benchmark #llm

TL;DR

Artificial Analysis Intelligence Index v4.3 (launch article dated September 7, 2026) raised the weight of evaluations with private tasks or answers from 40% to 45%. In the published weight table, four of the ten evaluations carry that 45%: AA-Briefcase (15), AutomationBench-AA (5), AA-Omniscience (10 + 5) and CritPt (10).
Private test sets attack one problem, gaming and contamination. They do not touch another: sampling noise. Terminal-Bench 4.0 has 66 tasks, each run three times, with average pass@1 reported.
With 66 tasks, one model's score has a standard error of roughly 3.5 to 6 points, depending on how task difficulty is distributed. The reported 7.1 point gap between the top two (59.1 vs 52.0) is only about 1.2 to 1.5 standard deviations of the simulated gap (4.5 to 5.8 points).
In an illustrative simulation (assumptions below), the top-two order survived a fresh 66-task draw in 88% to 93% of runs. The Fable 5.1 vs Opus 5 order (3.0 points) survived only 69% to 74%. The 19.2 point gap to GPT-5.6 Sol survived in about 100% of runs.
Practical rule: treat a gap as a ranking only when you can say how many tasks produced it. A short script below reproduces every number.

What exactly changed in Index v4.3?

The launch article lists three changes relative to v4.2. Terminal-Bench moved from 2.1 to 4.0. AutomationBench-AA replaced the tau-cubed Banking evaluation at its 5% weight. And because AutomationBench-AA uses a held-out test set that Artificial Analysis runs in collaboration with Zapier, the share of weight on private evaluations rose from 40% to 45%. Category weights stayed at Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%.

The page's weight table (version label v4.3.2 when I read it on October 20, 2026) lets you check the 45% yourself:

Category	Evaluation	Private	Weight
Agents	AA-Briefcase	Yes	15
Agents	GDPval-AA	No	10
Agents	AutomationBench-AA	Yes	5
Coding	Terminal-Bench 4.0	No	10
Coding	SciCode	No	10
General	AA-Omniscience Accuracy	Yes	10
General	AA-Omniscience Non-hallucination	Yes	5
General	GDP.pdf	No	10
General	AA-LCR	No	5
Scientific	HLE (Humanity's Last Exam)	No	10
Scientific	CritPt	Yes	10

Private rows sum to 15 + 5 + 10 + 5 + 10 = 45. Public rows sum to 55. The table has 11 rows because AA-Omniscience is scored in two parts, which is why the article counts 10 evaluations.

One inference to flag: the article says private weight was 40% in v4.2, and AutomationBench-AA (private) took over a 5% slot. So the retired tau-cubed Banking slot must have been public. The article does not say this in so many words.

Look at the coding category. It carries 20 points and none of them is private. The one coding benchmark upgraded in this release, Terminal-Bench 4.0, is also a public one, and it is the one with the small task count.

Does a private test set make a ranking more reliable?

It makes one kind of error less likely. If questions and answers never leave the evaluator, a lab cannot train on them by accident, and a model cannot memorize them from a scraped copy. That is the stated reason: the launch article says the changes add more private test sets "to prevent gaming".

It does not shrink the interval around a score. The sampling error of a pass rate depends on the number of tasks, whether or not those tasks are secret. A private set of 40 tasks is a good instrument for detecting leakage and a blunt one for ranking models that differ by a few points.

There is also a transparency cost that is easy to overlook. With a public benchmark, anyone can re-run it, rebuild the paired per-task table and compute a proper paired interval. With a private one, only the operator can. For the 45% of the index that is private, readers have to trust the operator's variance reporting, because nobody else can measure it. That is a trade, not a flaw, but it belongs on the list when you decide how much a 2-point index gap means.

How noisy is a 66-task benchmark?

The launch article says all 66 Terminal-Bench 4.0 tasks are run three times and the average pass@1 is reported. Reported scores for four models: GPT-6 Astra (max) 59.1%, Claude Fable 5.1 (max with fallback) 52.0%, Claude Opus 5 (max) 49.0%, GPT-5.6 Sol (max) 39.9%. The article itself states Astra's 19.2 point lead over Sol.

Two sources of noise matter, and they answer different questions.

Run noise. If you re-ran the same 66 tasks, how much would the score move? Averaging three runs reduces this.
Task-sampling noise. If someone built a different 66 tasks from the same pool of terminal work, how much would the score move? Averaging runs does nothing here, and for hard agentic tasks it is typically the larger term.

Bounds for a pass rate p over 66 tasks with 3 runs each:

Lower bound (every task has the same pass probability, so only run noise exists): SE = sqrt(p(1-p) / (66 x 3))
Upper bound (each task is effectively deterministic, so all variance is task sampling): SE = sqrt(p(1-p) / 66)

import numpy as np
N_TASKS, RUNS = 66, 3

def se_bounds(p):
    hi = np.sqrt(p*(1-p)/N_TASKS)
    lo = np.sqrt(p*(1-p)/(N_TASKS*RUNS))
    return lo*100, hi*100

for name, p in [("Astra", .591), ("Fable 5.1", .520), ("Opus 5", .490), ("Sol", .399)]:
    lo, hi = se_bounds(p)
    print(f"{name}: SE {lo:.1f} to {hi:.1f} pts, 95% half-width {1.96*lo:.1f} to {1.96*hi:.1f}")

Output from my run:

Astra: SE 3.5 to 6.1 pts, 95% half-width 6.8 to 11.9
Fable 5.1: SE 3.6 to 6.1 pts, 95% half-width 7.0 to 12.1
Opus 5: SE 3.6 to 6.2 pts, 95% half-width 7.0 to 12.1
Sol: SE 3.5 to 6.0 pts, 95% half-width 6.8 to 11.8

Real tasks are heterogeneous: some are nearly always solved, some nearly never, which pushes the true SE toward the upper bound. The chart at the top of this article uses the upper bound, which is why the bars for neighboring models overlap heavily.

Overlapping intervals are a conservative test, though. The models are scored on the same 66 tasks, so a hard task is hard for everyone, and that shared difficulty cancels out in the gap. A paired comparison is tighter than two separate intervals suggest. How much tighter depends on how correlated task difficulty is across models, and that is exactly the number outsiders cannot see.

Which of the reported gaps would survive a fresh task draw?

I simulated it. The setup is deliberately simple and every assumption is labeled, because the per-task results are not public.

Each model has a per-task pass probability drawn from a Beta distribution whose mean equals the reported score. The concentration parameter kappa controls spread: kappa = 1 gives many near-0 and near-1 tasks, kappa = 3 is milder.
Task difficulty is correlated across models through a Gaussian copula with correlation rho (I tried 0.5 and 0.8).
Each model runs each task three times (binomial draws), and the score is the mean.
I redraw 66 tasks 3,000 times and count how often the observed order matches the order of the reported scores.

import numpy as np
from scipy.stats import norm, beta
rng = np.random.default_rng(20261020)
RUNS = 3

def sim(pa, pb, rho, kappa, reps=3000, n=66):
    z1 = rng.standard_normal((reps, n))
    z2 = rho*z1 + np.sqrt(1-rho**2)*rng.standard_normal((reps, n))
    qa = beta.ppf(norm.cdf(z1), pa*kappa, (1-pa)*kappa)
    qb = beta.ppf(norm.cdf(z2), pb*kappa, (1-pb)*kappa)
    sa = rng.binomial(RUNS, qa)/RUNS
    sb = rng.binomial(RUNS, qb)/RUNS
    gap = (sa.mean(axis=1) - sb.mean(axis=1))*100
    return (gap > 0).mean(), gap.std()

Results for 66 tasks (share of fresh draws that keep the reported order):

Pair (reported gap)	rho 0.5, kappa 1	rho 0.5, kappa 3	rho 0.8, kappa 1	rho 0.8, kappa 3
Astra vs Fable 5.1 (7.1)	88.3%	91.4%	93.0%	93.2%
Fable 5.1 vs Opus 5 (3.0)	68.5%	71.3%	74.1%	73.0%
Astra vs Sol (19.2)	99.9%	100.0%	100.0%	100.0%

The standard deviation of the gap in these runs was 4.5 to 5.8 points. So:

A 19-point gap is a ranking. It held under every setting I tried.
A 7-point gap is probably a ranking, not a certainty. Between 7% and 12% of fresh draws flipped it, and with weaker difficulty correlation than I assumed it would flip more often.
A 3-point gap is closer to a coin flip with a thumb on the scale. The order held about 70% of the time, better than chance but far from what a leaderboard row implies.

These are not estimates of the true probabilities. They show the order of magnitude you should expect at n = 66.

How many tasks would a 7-point gap need?

Same simulation, Astra vs Fable 5.1, varying the task count (kappa = 1):

Tasks	order kept (rho 0.5)	order kept (rho 0.8)
66	88.5%	93.6%
100	94.0%	97.3%
150	96.7%	98.7%
200	98.1%	99.7%
300	99.8%	100%
600	100%	100%

Each row used 2,000 redraws, so expect about 1 point of simulation noise. Reaching 95% takes somewhere between 75 and 110 tasks under these assumptions, so a 7-point gap is only slightly under-powered at 66. A 3-point gap would need far more: required task count grows with the inverse square of the gap, so halving the gap roughly quadruples the tasks.

What does this do to the index itself?

The headline index scores in the launch article are integers: 53 for both Claude Fable 5.1 and GPT-6 Astra, 51 for Claude Opus 5, 50 for Claude Fable 5, 48 for Muse Spark 1.3, 47 for GPT-5.6 Sol. Among open-weight models the article lists GLM-5.3 Flash at 42, Qwen3.8 2.4T A95B at 40 and DeepSeek V4 Pro 0813 (max) at 36. These are launch-day numbers, and the live leaderboard has moved since.

Terminal-Bench carries 10% of the index. If component scores enter on a 0 to 100 scale (my assumption, since the article does not give the aggregation formula), a Terminal-Bench gap standard deviation of about 5 points moves the index by about 0.5 points from that component alone. Every other evaluation adds its own noise. I could not size the private ones, because the article gives no task counts or per-task results for them beyond the 657 tasks of AutomationBench-AA.

The practical reading: a 2-point index gap between two open-weight models, such as 42 versus 40, is in the range where component sampling noise could plausibly account for the difference. A 6-point gap, such as 42 versus 36, is much harder to explain that way.

A checklist before you quote a leaderboard gap

Find the task count behind each component. If the page does not say, treat the number as unsized.
Separate run noise from task noise. "Averaged over three runs" fixes the first one only.
Convert to a standard error. For a pass rate, sqrt(p(1-p)/n) is the upper bound. It takes ten seconds.
Ask for paired results. If the operator publishes per-task outcomes, a paired test beats overlapping bars.
Note the version. This index moved from v4.2 to v4.3 and the page now reads v4.3.2. Scores across versions are not a time series.
Discount private weight honestly. It protects against gaming. It does not buy precision, and it is not independently auditable.

FAQ

Is the Artificial Analysis Index v4.3 unreliable?
No. Adding private test sets is a reasonable design move. The point here is narrower: a published gap needs a task count before you read it as a ranking, and Terminal-Bench 4.0 has 66 tasks.

What percentage of the Index v4.3 is based on private test sets?
45%, up from 40% in v4.2, according to the launch article. In the weight table that is AA-Briefcase (15), AutomationBench-AA (5), AA-Omniscience Accuracy (10), AA-Omniscience Non-hallucination (5) and CritPt (10).

How many tasks does Terminal-Bench 4.0 have in this index?
66, each run three times, with the average pass@1 reported, per the launch article.

Do the simulation numbers prove the rankings are wrong?
No. They illustrate behavior under stated assumptions (Beta task difficulty, Gaussian copula correlation of 0.5 to 0.8, three binomial runs). Per-task results are not public, so the true correlation is unknown. The takeaway is the order of magnitude: about 5 points of gap noise at 66 tasks.

What is the difference between run noise and task-sampling noise?
Run noise is variation when the same tasks are re-run. Task-sampling noise is variation from which tasks happened to be in the set. Averaging more runs reduces only the first.

Can I rely on overlapping error bars to decide if two models differ?
Only as a conservative screen. When both models are scored on the same tasks, a paired analysis is tighter, which is why publishing per-task outcomes would help readers more than a wider table.

DEV Community