A benchmark can declare ten rooms tested while publishing nine room records. That difference deserves a check, but it does not establish a counting mistake. The declaration and the observed records may describe different parts of the testing process.
StagingBench's published results provide a concrete example. Across 12 tools, the declared room counts total 120, while the published room records total 119. Exactly one tool declares ten rooms and has nine records. A note in the results explains that a room was refused and produced no output.
Start with the two meanings
A declared count states what the benchmark reports about its test coverage. An observed count states how many room records are present. An attempted room can belong in the former without producing an image that can be scored and included in the latter.
Keeping those values separate makes an audit useful. If you overwrite the declaration with the record count, you lose the discrepancy that needs explanation. If you create a room record merely to satisfy the declaration, you risk implying that an output exists when it does not.
Read the counts and sources
The following Python snippet uses only the standard library. Run it in a directory containing rooms_tested.csv and sources.csv, two flat exports from results.json: per tool, the declared rooms_tested and the number of room records, and the sources entries with their checked dates. The first input needs the columns tool, rooms_tested_declared and room_records. The second needs tool and checked, with dates written as YYYY-MM-DD.
The mismatch message uses an anonymous label. The source summary prints each tool identifier with its source count and oldest check date. Parsing dates also checks that every source date has the expected form.
import csv
from collections import defaultdict
from datetime import date
def read_csv(path):
with open(path, newline="", encoding="utf-8") as handle:
return list(csv.DictReader(handle))
rooms = read_csv("rooms_tested.csv")
sources = read_csv("sources.csv")
print("Room mismatches")
for row in rooms:
declared = int(row["rooms_tested_declared"])
observed = int(row["room_records"])
if declared != observed:
print(f"one tool: declared {declared}, {observed} records")
dates_by_tool = defaultdict(list)
for row in sources:
dates_by_tool[row["tool"]].append(date.fromisoformat(row["checked"]))
print("Sources by tool")
for row in sorted(rooms, key=lambda row: row["tool"]):
tool = row["tool"]
dates = dates_by_tool[tool]
oldest = min(dates).isoformat() if dates else "none"
print(f"{tool}: {len(dates)} sources, oldest {oldest}")
all_dates = [day for dates in dates_by_tool.values() for day in dates]
print("Source totals")
print(f"{len(sources)} sources")
print(f"oldest date {min(all_dates).isoformat()}")
print(f"newest date {max(all_dates).isoformat()}")
Read the output
For the published results, the room mismatch section is:
Room mismatches
one tool: declared 10, 9 records
The source totals section is:
Source totals
38 sources
oldest date 2026-10-19
newest date 2026-11-10
The script also prints the source summary for every tool. That summary helps identify where supporting references are concentrated and how far back each tool's recorded checks begin. A source count measures the number of entries, not the quality or completeness of their evidence.
Investigate the difference
The refused run explains why one attempted room has no corresponding output record. That is a coherent account of the mismatch, rather than evidence by itself that the data is wrong. The refusal remains part of the testing history even though it supplies no staged image for measurement.
The audit should retain that explanation beside the discrepancy. It should not silently change ten to nine or insert an empty measurement to make the totals agree. Readers need to distinguish a room that was attempted without output from a room that produced a measurable result.
Know what this check establishes
This snippet compares the declaration with the supplied room_records value. It does not independently count image records. In a production pipeline, calculate that observed value directly from the room results, then compare it with the declaration. Otherwise, two summary fields could agree while both contain the same counting mistake.
The methodology supplies the context for interpreting what the benchmark measures. A pipeline can combine that context with count checks, date validation and documented exceptions. Run those checks before publication so that unexplained differences receive attention and explained differences remain visible in the published record.
Top comments (0)