The eval schema I stopped migrating

Every time I added a benchmark suite to EvalBench, I used to touch the database. New suite, new columns, a migration, a change to the aggregation query, and usually a change to the frontend so it knew the new metric existed. By the third suite that pattern was clearly wrong. The suites had almost nothing in common at the task level, but the thing I stored about each task was always the same handful of facts plus a bag of numbers.

So I collapsed all of it into one row shape. Adding a suite is now a three-file change, and none of those files is the schema, the aggregator, or the dashboard.

What the three suites actually disagree about

EvalBench runs three suites against multiple providers. Structured-output reliability checks whether a model returns JSON that validates against a schema, and how many retries it needs. Latency and cost runs pairwise judge scoring. RAG measures recall@k, nDCG@k, and faithfulness across swappable chunking strategies.

These have no shared task type. A structured task carries a JSON schema and an expected value. A RAG task carries a corpus and a set of relevant document ids. The scoring code is completely different. If you let each suite define its own storage, you get three tables, three aggregation paths, and a frontend with three special cases. Every new suite is a fourth of everything.

But look at what you store per task after scoring, and the disagreement disappears. You always know which run and suite and domain it belongs to, which model produced it, how long it took, what it cost, whether the model refused. And then some suite-specific numbers. Structured emits first_attempt_valid and retries_to_valid. RAG emits recall_at_k and faithfulness. Different keys, same type: a float.

One record, one open column

That observation is the whole design. The stored record is fixed except for one field:

class MetricRecord(BaseModel):
    id: str
    run_id: str
    suite: str
    domain: str
    model: str
    provider: str
    model_family: str
    task_id: str
    latency_ms: float
    prompt_tokens: int
    completion_tokens: int
    cost_usd: float
    error: str | None
    refused: bool
    metrics: dict[str, float]
    created_at: datetime

Everything above metrics is shared across every suite that will ever exist. The metrics dict is the open part. Structured fills it with schema-validity numbers, RAG fills it with retrieval numbers, and the storage layer does not care which. It is one table with a JSON column for metrics, so a new suite adds new keys to that dict and never touches the schema.

The suite interface is small on purpose. A suite declares its metric keys and how to display them, then implements three methods:

class Suite(ABC):
    name: str
    metric_keys: list[str]
    display_metrics: list[dict]

    @abstractmethod
    def load_tasks(self, domain: str) -> list[Task]: ...

    @abstractmethod
    def build_prompt(self, task: Task) -> list[dict]: ...

    @abstractmethod
    def evaluate(
        self, task: Task, raw_output: str, judge: Judge
    ) -> dict[str, float]: ...

evaluate returns a plain dict[str, float]. That return value becomes the metrics column verbatim. The runner does the wrapping:

metrics = suite.evaluate(task, raw_output, judge)
# ...
return MetricRecord(
    id=str(uuid.uuid4()),
    run_id=run_id,
    suite=suite.name,
    # shared fields the runner fills for every suite ...
    metrics=normalized_metrics,
    created_at=datetime.now(timezone.utc),
)

The suite author never constructs a MetricRecord. They return a dict of numbers and the runner attaches the shared columns. That split is what keeps a suite from inventing its own storage.

The three files

Adding a suite means writing the suite class, registering it, and dropping in task data. The registry is explicit, no auto-discovery:

register_suite(StructuredSuite())
register_suite(LatencyCostSuite())
register_suite(RagSuite())

One more line there and the suite is live. The dashboard picks it up because the API serves each suite's display_metrics list, so the frontend renders whatever labels and formats the suite declared. No frontend change, no new endpoint. The three files are the suite, the one register line, and the task data. The schema, the aggregator, and the UI are untouched.

Where "just a dict of floats" would have bitten me

An open metrics bag is convenient right up until the aggregator has to compute confidence intervals over it. A proportion like schema-validity wants a Wilson interval. A count like retries wants an interval that cannot go below zero. A latency tail wants an order-statistic interval. If the aggregator guesses the type from the number itself, it guesses wrong. I know because it already did once: a retry count published a confidence interval of -0.035 to 0.285 for a quantity that cannot be negative.

The fix is that display_metrics carries a support field, and the aggregator routes on it:

display_metrics = [
    {
        "key": "schema_valid",
        "label": "Schema valid",
        "format": "percent",
        "support": "proportion",
        "higher_is_better": True,
    },
    {
        "key": "retries_to_valid",
        "label": "Retries to valid",
        "format": "number",
        "support": "non_negative",
        "higher_is_better": False,
    },
]

support is the statistical claim. format is a display concern. The old bug came from routing interval math off format, which is a display string. Now the aggregator refuses to guess:

def _support_by_metric(suite: Suite) -> dict[str, str]:
    declared: dict[str, str] = {}
    for metadata in suite.display_metrics:
        key = metadata.get("key")
        support = metadata.get("support")
        if support not in _VALID_SUPPORTS:
            raise ValueError(...)
        declared[key] = support

    missing = [key for key in suite.metric_keys if key not in declared]
    if missing:
        raise ValueError(
            f"suite {suite.name!r} is missing declared support "
            f"for metrics: {sorted(missing)}"
        )
    return declared

If a new suite adds a metric key and forgets to declare its support, this raises instead of silently defaulting. A silent default on missing metadata is exactly how the negative interval shipped the first time, so the freedom of the open dict is bounded by a hard requirement: every key you emit, you also classify.

A contract test enforces the same thing across every registered suite, so the requirement is not a convention I have to remember. Each suite's declared display keys must be a subset of its metric keys, and each must declare a valid support. The open column is free. The classification is not optional.

What I would do differently

The one thing I gave up is a typed metrics column. metrics: dict[str, float] means a typo in a metric key is a runtime discovery, caught by the aggregator or a test, not by the type checker at the point of the mistake. For a personal eval platform that tradeoff is fine, because the suite count is small and the contract test covers it. If this were a shared library other people wrote suites against, I would want the metric keys to be a typed enum per suite so a bad key failed at definition time. The open dict bought me additive suites. It cost me a compile-time guarantee I decided I did not need yet.

The takeaway I keep coming back to: find the row shape every producer already agrees on, hide what they disagree about behind one open field, and force them to say enough about that field that the consumer never guesses. The bug taught me the last part.