Benchmarking a web-access API — search, scraping, extraction, document parsing, browser actions — is easy to do once and hard to do twice. Target pages change. The provider ships an update. Your harness gains a retry. Three months later the numbers move, and nothing in the output says which of those three moved them.
Reproducibility here does not mean identical numbers on a second run; against the live web, you will not get them. It means every published number carries enough structure to say what was measured, over which population, on what date. Five layers produce that property.
Layer 1: A versioned task contract
The unit of comparison is a task, not a call. Keep the corpus as data, separate from harness code: each task carries a stable task_id, the capability it exercises, its input — a query, a URL, a target schema — and the pass criteria that decide whether a response is usable. Every system under test receives the identical list.
Two rules make that list trustworthy. Hold it fixed across systems within a run: comparing one scraper on easy pages against another on pages behind commercial bot defenses ranks the sample, not the tools. And version it — when you add, remove, or reword a task, bump the corpus version and re-run everything, because numbers produced under v1 and v2 are different measurements that share a metric name.
Keep the corpus version distinct from the run identifier; one says what was asked, the other when. And build the corpus around what breaks: scanned PDFs, sites that refuse datacenter IPs, queries needing synthesis across sources. Easy tasks compress every system toward the same score.
Layer 2: A normalized observation schema
Record one row per attempt, before aggregating anything. A harness that computes a mean inside the request loop loses the ability to answer any question nobody anticipated.
A workable attempt record holds run_id, corpus_version, task_id, system, capability, observed_at, latency_ms, billed_usd, outcome, and outcome_reason. Write observed_at as an RFC 3339 date-time with an explicit offset; a bare local timestamp turns ambiguous as soon as runs cross machines or zones.
outcome deserves a small closed vocabulary — usable, unusable, error — because the distinction that matters is not the HTTP status. A 200 OK carrying an empty extraction is a failed task and a billed call. Folding that into a success rate is the most common way a benchmark flatters what it measures.
The OpenTelemetry metrics data model is worth borrowing from even if you never emit an OTLP point. It requires every data point to carry the time window it covers, and sums to declare whether they are delta or cumulative rather than leaving a reader to guess. Benchmark rows deserve the same discipline.
Layer 3: Named denominators and failure accounting
Every rate has a denominator, and in a benchmark of web tools the denominators differ inside a single table. A latency median covers attempts that produced a timing. An error rate covers all attempts. A cost-per-usable-result covers usable results only, while failed-but-billed calls stay in the numerator.
So publish the denominator as a field, not a footnote. This normalizer turns synthetic attempt observations into long-form metric rows, each carrying the name and size of its own denominator. The values illustrate the shape; they are not measurements of any real service.
from collections import Counter, defaultdict
from statistics import median
RUN_ID = "2026-08-11-synthetic"
CORPUS_VERSION = "demo-v1"
OUTCOMES = ("usable", "unusable", "error")
# Synthetic. Not measurements of any real service.
OBSERVATIONS = [
{"task": "t1", "system": "alpha", "capability": "search", "outcome": "usable",
"latency_ms": 620, "billed_usd": 0.002},
{"task": "t2", "system": "alpha", "capability": "search", "outcome": "unusable",
"latency_ms": 700, "billed_usd": 0.002},
{"task": "t3", "system": "alpha", "capability": "search", "outcome": "error",
"latency_ms": None, "billed_usd": 0.002},
{"task": "t1", "system": "beta", "capability": "search", "outcome": "usable",
"latency_ms": 910, "billed_usd": 0.003},
{"task": "t2", "system": "beta", "capability": "search", "outcome": "usable",
"latency_ms": 990, "billed_usd": 0.003},
]
def metric_rows(observations):
groups = defaultdict(list)
for obs in observations:
assert obs["outcome"] in OUTCOMES, obs["outcome"]
groups[(obs["system"], obs["capability"])].append(obs)
rows = []
for (system, capability), group in sorted(groups.items()):
counts = Counter(o["outcome"] for o in group)
attempts = len(group)
assert sum(counts.values()) == attempts # no attempt escapes an outcome bucket
timed = [o["latency_ms"] for o in group if o["latency_ms"] is not None]
billed = sum(o["billed_usd"] for o in group)
usable = counts["usable"]
for key, value, denominator, size in [
("error_rate", counts["error"] / attempts, "attempts", attempts),
("latency_p50_ms", median(timed) if timed else None, "timed_attempts", len(timed)),
("cost_per_usable_usd", billed / usable if usable else None, "usable_results", usable),
]:
rows.append({
"run_id": RUN_ID, "corpus_version": CORPUS_VERSION,
"system": system, "capability": capability,
"metric_key": key, "value": value,
"denominator": denominator, "denominator_size": size,
"attempts": attempts,
# every key present, so absent never reads as zero
"outcome_counts": {name: counts[name] for name in OUTCOMES},
})
return rows
if __name__ == "__main__":
rows = metric_rows(OBSERVATIONS)
indexed = {(r["system"], r["metric_key"]): r for r in rows}
latency = indexed[("alpha", "latency_p50_ms")]
assert (latency["denominator_size"], latency["attempts"]) == (2, 3)
assert round(indexed[("alpha", "cost_per_usable_usd")]["value"], 6) == 0.006
assert round(indexed[("beta", "cost_per_usable_usd")]["value"], 6) == 0.003
print(f"{len(rows)} rows, every denominator named")
The script prints 6 rows, every denominator named. The latency assertion pins alpha's median to two timed attempts against three attempts: the errored call produced no timing, so it cannot join the latency population. The cost assertions show alpha billing less per attempt than beta and more per usable result, because a paid failure sits in the numerator, not the denominator.
Long form — one row per system, capability, and metric — earns its verbosity. Metric sets differ by capability, so a wide table degenerates into a sparse union of columns.
Layer 4: Snapshots with provenance
A result set that cannot be traced back to its inputs is an assertion, not a measurement.
Hash the corpus file and the raw observation file, and record both digests in the published output. Make the serializer deterministic: fixed field order, stable sort, no wall-clock timestamp inside the artifact. A rebuild from the same inputs then produces byte-identical output, and a diff between two runs shows only what changed.
Repeat the provenance fields on every row rather than once in a header. Rows get filtered, joined, and pasted into other people's notebooks; a slice carrying its own run_id and corpus_version survives that trip, and one that does not becomes an orphan number.
The W3C's Data on the Web Best Practices covers publication: provide machine-readable metadata, state provenance and licensing, supply version information and a version history, and keep superseded versions reachable rather than rewriting numbers in place. Someone cited the old figures.
Layer 5: Limitations you state before anyone asks
The last layer is presentational, and the one most often dropped.
Start by naming the operator. A first-party benchmark — one run by a party with a commercial interest in the outcome — is not disqualified by that fact, but a reader is entitled to weigh it. One concrete example is a published first-party benchmark methodology for web-access services, operated by NativePort: it keeps one versioned corpus per capability, puts a run date on every scorecard, labels each cost denominator, publishes raw per-metric values beneath each composite, and lists what its runner does not observe — uptime, SLA conformance, throughput ceilings — as claims it therefore does not make.
Three habits then earn trust cheaply. Publish weak results, including unflattering ones. Treat absent as absent and never as zero, because "not measured" and "measured at zero" are opposite facts that a null-to-zero coercion merges. And put the run date beside the number everywhere, since provider behavior and site defenses shift underneath you.
Hugging Face's dataset card documentation models where this belongs: a card ships with the data, carries machine-readable metadata in its header, and reserves explicit sections for collection process and limitations. Caveats belong in the artifact, not in a post that consumers of your table will never open.
The underlying decision
None of this needs a large harness. It needs one decision, made before the first run: that the output is a dataset with a contract, not a table in a slide.
References
- Dataset Cards — Hugging Face Hub documentation on shipping a card alongside data, with machine-readable metadata and dedicated sections for collection process and limitations.
- Metrics Data Model — OpenTelemetry specification requiring data points to carry their time window and to declare aggregation temporality explicitly.
- Data on the Web Best Practices — W3C Recommendation covering metadata, provenance, licensing, versioning, and version history for published datasets.
-
RFC 3339 — the internet date/time format used for the
observed_atfield, including explicit UTC offsets.
Disclosure
I work on NativePort, and its public methodology is cited once above as an example of first-party benchmark disclosure. AI assisted with drafting; a human reviewed the text, the code example, and every cited source. All values shown here are synthetic.
Top comments (0)