I Stopped Measuring My LLM With a Single Number. Here's the 4-Metric Scorecard I Use Now

A model can win MMLU and still fail your users. A "fast" model can quietly double your bill because retries aren't counted. A "cheap" model can drop success rate from 94% to 71% the day your prompts get a little longer. I've shipped enough of these failures to stop trusting any single number.

For the last six months I've been grading every new model release on the same four-axis scorecard. It costs me about \$12 a month in API spend to run, takes ninety minutes on a Sunday, and has saved me from at least three production outages. Here's the whole thing, code included.

Why one number lies

Leaderboards exist because they're easy to compare. They're also where good measurement goes to die.

Three real examples from the past year, all from code I shipped:

1. I switched a customer-support classifier from a "high-MMLU" model to a smaller one that scored three points lower on benchmarks. Accuracy went up in production. The benchmark measured generic reasoning; my users asked short, domain-specific questions the smaller model had clearly been over-fine-tuned on.
2. I picked a model advertised as "fastest in class." First-token latency was great. The catch: it timed out on 6% of long-context requests, and my retry logic silently tripled the cost on those.
3. I chose a model because it was 40% cheaper per token. Two weeks later I noticed the agent loop running 1.8× more steps per task because it had to re-plan when it hallucinated tool schemas. Net cost went up.

Each of these looked fine on a one-dimensional benchmark. Each was a regression in production. The single number was hiding the dimension that mattered.

The four metrics that actually matter

For any LLM application that touches users or money, I grade the model on:

• Capability — does it solve the task at all? (judge-graded accuracy on a fixed eval set)
• Reliability — does it solve it consistently? (variance across runs, format-compliance rate)
• Cost-per-success — what does one successful completion cost? (price per token × expected retries × 1/pass-rate)
• P95 tail — what's the worst-case latency your users will feel? (not the median)

Capability without reliability gives you a model that's brilliant on Tuesday and broken on Wednesday. Reliability without capability gives you a model that's consistently wrong. Either without cost-per-success quietly drains your budget. And cost without p95 tail is how you end up apologizing to a customer who waited 40 seconds for a chatbot reply.

The reason these four beat a single score is that they're partially independent. A model can improve on one while regressing on another. The scorecard surfaces that.

The scorecard, code

Here's the harness I run every Sunday. It's a single Python file, ~150 lines, runs against any OpenAI-compatible endpoint.

import asyncio
import json
import statistics
import time
from dataclasses import dataclass, field

import httpx

@dataclass
class EvalCase:
    prompt: str
    reference: str
    judge_prompt: str
    expects_json: bool = False

@dataclass
class ModelScore:
    name: str
    capability: float = 0.0
    reliability: float = 0.0
    cost_per_success: float = 0.0
    p95_latency_ms: float = 0.0
    raw: dict = field(default_factory=dict)

async def call_model(client, base, key, model, prompt, json_mode=False):
    t0 = time.perf_counter()
    r = await client.post(
        f"{base}/chat/completions",
        headers={"Authorization": f"Bearer {key}"},
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0,
            "response_format": {"type": "json_object"} if json_mode else None,
        },
        timeout=60,
    )
    r.raise_for_status()
    dt = (time.perf_counter() - t0) * 1000
    return r.json(), dt

async def grade(model, eval_set, base, key, judge_model, runs=3):
    scores = ModelScore(name=model)
    latencies = []
    successes = 0
    total_cost = 0.0
    attempts = 0
    async with httpx.AsyncClient() as client:
        for case in eval_set:
            case_results = []
            for _ in range(runs):
                try:
                    resp, dt = await call_model(
                        client, base, key, model,
                        case.prompt, case.expects_json,
                    )
                    latencies.append(dt)
                    attempts += 1
                    total_cost += resp.get("usage", {}).get(
                        "total_tokens", 0
                    ) * 0.000003  # set per-model
                    content = resp["choices"][0]["message"]["content"]
                    judge_resp, _ = await call_model(
                        client, base, key, judge_model,
                        case.judge_prompt.format(
                            ref=case.reference, out=content
                        ),
                    )
                    ok = (
                        "PASS" in judge_resp["choices"][0]["message"]["content"]
                        and case.expects_json is False
                    ) or (
                        "PASS" in judge_resp["choices"][0]["message"]["content"]
                        and _looks_like_json(content)
                    )
                    case_results.append(ok)
                    if ok:
                        successes += 1
                except Exception:
                    case_results.append(False)
            # reliability = fraction of runs that pass per case
            scores.raw.setdefault(case.prompt[:40], case_results)

    pass_rate = successes / max(1, attempts)
    scores.capability = sum(
        sum(r) / len(r) for r in scores.raw.values()
    ) / max(1, len(scores.raw))
    scores.reliability = statistics.mean(
        [1 - (r.count(False) / len(r)) for r in scores.raw.values() if r]
    ) if scores.raw else 0
    scores.cost_per_success = (
        total_cost / max(1, successes)
    ) if successes else float("inf")
    scores.p95_latency_ms = (
        statistics.quantiles(latencies, n=20)[18]
        if len(latencies) >= 20
        else max(latencies, default=0)
    )
    return scores

def _looks_like_json(s):
    try:
        json.loads(s)
        return True
    except Exception:
        return False

def render_report(scores):
    rows = [
        "| Model | Capability | Reliability | Cost/success | p95 ms |",
        "|---|---|---|---|---|",
    ]
    for s in scores:
        rows.append(
            f"| {s.name} | {s.capability:.2f} | "
            f"{s.reliability:.2f} | ${s.cost_per_success:.4f} | "
            f"{int(s.p95_latency_ms)} |"
        )
    return "\n".join(rows)

Run it against three candidate models, render the table, and you have a real comparison.

A recent run from my eval set, three models, on a 40-case classification + JSON task with three runs per case:

| Model              | Capability | Reliability | Cost/success | p95 ms |
|--------------------|-----------|-------------|--------------|--------|
| flagship-large     | 0.86      | 0.95        | $0.0211      |  3,840 |
| flagship-mini      | 0.83      | 0.96        | $0.0048      |  1,210 |
| small-cheap        | 0.71      | 0.82        | $0.0019      |    980 |

If I'd picked on capability alone, large wins. On cost, small-cheap wins. But the scorecard surfaces: small-cheap's reliability is a deal-breaker (18% of runs fail), and flagship-mini matches flagship-large's reliability at a quarter of the cost. That's the decision I made, and it's the one I'd never have made from a single benchmark number.

How I keep the eval set honest

The eval set is the part that breaks if you don't maintain it. Three rules I follow:

• Never let the eval set leak into training prompts. If a model has seen a near-identical task during RLHF, its eval score is meaningless. Refresh ~20% of cases every month from real (redacted) production traffic.
• Keep the judge honest. A judge model that agrees with your humans 92% of the time can still be 60% accurate where the gate actually decides. I keep a held-out 50-case set where a human grades the judge, not the model.
• Pin the temperature at 0 for the model under test, sample three times for reliability. A model that's "73% accurate" with temperature 1 and one run is unreliable. Three runs at temp 0 lets you separate capability from luck.

What I learned

The first thing I learned is that the leaderboard was never wrong — it just wasn't asking the question I needed answered. Single-number benchmarks are useful for ranking research models. They're nearly useless for picking production models because production cares about reliability, cost, and latency in ways benchmarks don't measure.

The second thing is that the four metrics are partially orthogonal. That's why a scorecard beats a weighted score. If you collapse them into one weighted number, you can quietly regress on one axis while the headline moves the right way. The whole point is to see all four at once.

The third thing is that this is cheap to run. Three models, forty cases, three runs per case, judged by a small judge model — total cost around \$12 and ninety minutes of wall time. Compared to the cost of one wrong model decision in production, the math isn't close.

If you're picking a model this week, run the scorecard before you trust the leaderboard. And if you already shipped one, run it anyway — I guarantee at least one axis is hiding something.