Build a Capability Profile, Not a Leaderboard: A Multi-Axis Eval Harness in Python

Single scores hide the shape of what a model can do. Here is a ~120-line harness that reports per-axis accuracy, bootstrap confidence intervals and cost.

Two models. One scores 0.675 on your eval, the other 0.713. A leaderboard ranks them, a slide deck calls it "roughly equal", and someone picks the higher number.

Now look at the same two models axis by axis. Model A is clearly better at arithmetic and string manipulation. Model B is clearly better at long-context and multi-step work. These differences are statistically separable, not noise. If your product is a calculator, B is the wrong choice. If it is a document agent, A is. The headline number told you nothing about either decision.

This post builds a small, dependency-light harness that reports a capability profile (per-axis accuracy with confidence intervals and cost) instead of a single score. Everything below was run as shown; the output is real.

The engineering problem

Most internal eval pipelines end with a single aggregate: mean accuracy over a mixed test set. That aggregate has three problems:

1. It hides trade-offs. Strength on one axis cancels weakness on another.
2. It hides uncertainty. With 50–100 items per axis, per-axis accuracy can move by ±10 points from sampling alone.
3. It hides cost. A model that is 5% better but needs 3× the tokens on hard items is not 5% better for your budget.

Why existing approaches fall short

Public leaderboards optimise for ranking, so they report one number per benchmark. Some report per-category scores, but usually without intervals, and rarely with cost. Large eval frameworks such as lm-evaluation-harness, HELM and OpenAI Evals are excellent at running tasks. This post is about the reporting layer on top: what you compute once you have per-item results. You can feed any framework's per-item outputs into the functions here.

The design

Four ideas, each a few lines of code:

• Tag every item with an axis. An axis is a capability you care about separately: arithmetic, retrieval, tool use, long context, and so on.
• Bootstrap a 95% interval per axis. Resample items with replacement and take percentiles of the mean. There are no distributional assumptions, and it works for 0/1 scores.
• Record cost per item. Tokens, seconds or currency. Report accuracy per unit of cost alongside raw accuracy.
• Compare profiles, not points. Two models differ on an axis only if their intervals do not overlap. This is a deliberately conservative screen. For formal testing, use a paired test on per-item results.

items ──► model(item) ──► (answer, cost) ──► per-axis 0/1 + cost
                                               │
                         bootstrap CI ◄────────┤
                                               ▼
                       profile table · separability · acc/cost

The code

Save as profile_eval.py. The only dependency is numpy.

"""profile_eval.py: report capability profiles with confidence intervals, not one score.

Requires only numpy. Run:  python profile_eval.py
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable
import numpy as np


@dataclass(frozen=True)
class Item:
    axis: str          # capability axis, e.g. "arithmetic"
    prompt: str
    answer: str
    cost: float = 1.0  # relative cost of attempting this item (tokens, seconds, ...)


Model = Callable[[Item], tuple[str, float]]   # returns (answer, cost actually spent)


def bootstrap_ci(x: np.ndarray, n_boot: int = 2000, alpha: float = 0.05,
                 rng: np.random.Generator | None = None) -> tuple[float, float]:
    """Percentile bootstrap confidence interval for the mean of 0/1 scores."""
    rng = rng or np.random.default_rng(0)
    idx = rng.integers(0, len(x), size=(n_boot, len(x)))
    means = x[idx].mean(axis=1)
    return float(np.quantile(means, alpha / 2)), float(np.quantile(means, 1 - alpha / 2))


def evaluate(model: Model, items: list[Item], seed: int = 0) -> dict[str, dict]:
    """Score a model per axis: accuracy, 95% CI, mean cost and accuracy per unit cost."""
    rng = np.random.default_rng(seed)
    by_axis: dict[str, list[tuple[int, float]]] = {}
    for it in items:
        out, spent = model(it)
        by_axis.setdefault(it.axis, []).append((int(out.strip() == it.answer), spent))
    report = {}
    for axis, rows in sorted(by_axis.items()):
        correct = np.array([r[0] for r in rows], dtype=float)
        cost = np.array([r[1] for r in rows], dtype=float)
        lo, hi = bootstrap_ci(correct, rng=rng)
        report[axis] = dict(n=len(rows), acc=correct.mean(), lo=lo, hi=hi,
                            cost=cost.mean(), acc_per_cost=correct.mean() / cost.mean())
    return report


def headline(report: dict[str, dict]) -> float:
    """The single number a leaderboard would print: the unweighted mean over axes."""
    return float(np.mean([r["acc"] for r in report.values()]))


def separable(a: dict, b: dict, axis: str) -> bool:
    """True if the two models' 95% CIs on this axis do not overlap."""
    return a[axis]["hi"] < b[axis]["lo"] or b[axis]["hi"] < a[axis]["lo"]


def print_profile(name: str, rep: dict[str, dict]) -> None:
    print(f"\n{name}   headline score = {headline(rep):.3f}")
    print(f"{'axis':<14}{'n':>4}{'acc':>7}{'95% CI':>16}{'cost':>7}{'acc/cost':>10}  profile")
    for axis, r in rep.items():
        bar = "#" * int(round(r["acc"] * 20))
        print(f"{axis:<14}{r['n']:>4}{r['acc']:>7.2f}   [{r['lo']:.2f}, {r['hi']:.2f}]"
              f"{r['cost']:>7.1f}{r['acc_per_cost']:>10.3f}  {bar}")


# --------------------------------------------------------------------------- demo
def make_suite(n_per_axis: int = 60, seed: int = 1) -> list[Item]:
    """A synthetic suite with four axes so the script runs with no downloads."""
    rng = np.random.default_rng(seed)
    items = []
    for _ in range(n_per_axis):
        a, b = rng.integers(10, 99, size=2)
        items.append(Item("arithmetic", f"{a}+{b}", str(a + b)))
        w = "".join(rng.choice(list("abcdefgh"), size=6))
        items.append(Item("string_ops", f"reverse {w}", w[::-1]))
        seq = rng.integers(1, 9, size=5)
        items.append(Item("long_context", "sum " + " ".join(map(str, seq)), str(seq.sum()), cost=4.0))
        x = int(rng.integers(2, 9))
        items.append(Item("multi_step", f"(({x}*3)+4)*2", str(((x * 3) + 4) * 2), cost=2.0))
    return items


def simulated_model(skill: dict[str, float], seed: int) -> Model:
    """A stand-in 'model' that answers correctly with a per-axis probability."""
    rng = np.random.default_rng(seed)
    def run(it: Item) -> tuple[str, float]:
        ok = rng.random() < skill[it.axis]
        return (it.answer if ok else "wrong"), it.cost * (1.5 if it.axis == "multi_step" else 1.0)
    return run


if __name__ == "__main__":
    suite = make_suite()
    model_a = simulated_model({"arithmetic": .95, "string_ops": .90, "long_context": .40, "multi_step": .55}, seed=7)
    model_b = simulated_model({"arithmetic": .70, "string_ops": .65, "long_context": .75, "multi_step": .70}, seed=8)
    ra, rb = evaluate(model_a, suite), evaluate(model_b, suite)
    print_profile("model_a", ra)
    print_profile("model_b", rb)
    print("\nAxes where the two models are statistically separable (95% CIs do not overlap):")
    for axis in ra:
        if separable(ra, rb, axis):
            better = "model_a" if ra[axis]["acc"] > rb[axis]["acc"] else "model_b"
            print(f"  {axis:<14} -> {better}")
    print(f"\nHeadline gap: {abs(headline(ra) - headline(rb)):.3f}  (a leaderboard would call this a near tie)")

The output

model_a   headline score = 0.675
axis             n    acc          95% CI   cost  acc/cost  profile
arithmetic      60   0.93   [0.87, 0.98]    1.0     0.933  ###################
long_context    60   0.37   [0.25, 0.48]    4.0     0.092  #######
multi_step      60   0.52   [0.38, 0.63]    3.0     0.172  ##########
string_ops      60   0.88   [0.80, 0.95]    1.0     0.883  ##################

model_b   headline score = 0.713
axis             n    acc          95% CI   cost  acc/cost  profile
arithmetic      60   0.67   [0.55, 0.78]    1.0     0.667  #############
long_context    60   0.77   [0.67, 0.87]    4.0     0.192  ###############
multi_step      60   0.78   [0.67, 0.88]    3.0     0.261  ################
string_ops      60   0.63   [0.52, 0.75]    1.0     0.633  #############

Axes where the two models are statistically separable (95% CIs do not overlap):
  arithmetic     -> model_a
  long_context   -> model_b
  multi_step     -> model_b
  string_ops     -> model_a

Headline gap: 0.037  (a leaderboard would call this a near tie)

The headline scores differ by 0.037, which most leaderboards would call a near-tie. But on all four axes the 95% intervals do not overlap, and the winners split two and two. That is a precise statement of a trade-off a single number erases. Add the acc/cost column and the picture sharpens again. Model B's long-context advantage costs four units per attempt. Whether that is worth it depends on your budget, not on the leaderboard.

Plugging in a real model

The Model type is just a function from Item to (answer, cost). An adapter for a chat API looks roughly like this. It is untested pseudocode, so adapt it to your client library:

def api_model(client, model_name):
    def run(item):
        resp = client.generate(model=model_name, prompt=item.prompt, max_tokens=64)
        return resp.text.strip(), resp.usage.total_tokens  # cost = tokens
    return run

For free-form answers, replace the exact-match check in evaluate with a task-specific scorer such as numeric tolerance, unit tests for code, or a rubric grader. Keep the scorer per axis, because what counts as "correct" differs by capability.

Practical notes

• Items per axis matter more than total items. At 60 items an interval is roughly ±10 points; at 200 it is roughly ±6. Budget your eval set per axis.
• Choose axes by decisions, not by datasets. An axis should map to a decision you would make differently, such as which model to route a request to. Otherwise it is decoration.
• Version the suite. Store items and the axis map with a hash, so profiles from different runs are comparable. A later post in this track builds that registry.
• Psychometrics has done this before. Item response theory estimates item difficulty and model ability jointly. It is the natural next step, and a later post in this track covers it.

Takeaway

Stop shipping one number. Ship a table with an interval and a cost on every row. It takes about 120 lines, it changes real decisions, and it makes your eval harder to game.

Companion repository

The script above is the first file of the Praneeth's Xpedition companion repo, which will hold one runnable experiment per engineering post. Next in this track: Measuring Skill-Acquisition Efficiency: an ARC-style few-shot evaluation loop.

Praneeth's Xpedition by dspraneeth07