A benchmark consumer can parse a valid number and still make an invalid comparison. The risk appears when a field preserves its name while the measurement procedure behind it changes. In these staging records, drift_score can hold either an aligned architecture score or a raw fallback after alignment fails. A numeric sort cannot recover that distinction once a consumer has discarded the status fields.
My recommendation is to enforce comparability before sorting, averaging, or presenting a rank. The score object should enter the application as a measurement with a procedure and a status, rather than as an isolated floating point value.
Read the dependency chain
The recorded alignment method is SIFT matching with a ratio test and a RANSAC similarity transform. A successful record includes a transformation matrix mapping the staged image into the original working frame. That gives the architecture comparison a registered basis.
A failed record preserves diagnostic information instead. The benchmark reports alignment.ok: false, gives a failure reason, and leaves framing.pass null. The architecture block then records aligned: false and says that it falls back to the raw, unaligned score. A caller can retain a numerical diagnostic without pretending that registration succeeded, but the downstream consumer must preserve the distinction.
Real records at the failure boundary
The MeltFlex room 01 record reports 20 inliers and the reason "too few inliers." Its raw score is 48.5. The top level drift_score and architecture.drift_score both also contain 48.5, with architecture.aligned false and framing unknown.
The REimagineHome room 04 record reports 10 inliers, again with "too few inliers." The raw score, top level score and architecture score all contain 40.1, and its framing decision is also unknown.
The test log states an acceptance minimum of 25 inliers, so the shortfalls are 5 for MeltFlex and 15 for REimagineHome. Those calculations explain the recorded failures; they do not establish any relationship between inlier shortfall and the amount of architectural change.
| Record | Inliers | Raw fallback | Architecture aligned |
|---|---|---|---|
| MeltFlex room 01 | 20 | 48.5 | false |
| REimagineHome room 04 | 10 | 40.1 | false |
Why a mixed ranking has no stable meaning
Alignment accounts for differences in image registration before architecture is measured. A raw comparison leaves that source of difference unresolved, so its result can reflect a different mixture of effects from an aligned result.
Sorting the shared field would put those procedures on the same apparent scale. A reader might interpret the order as evidence about architectural preservation, even though the preprocessing condition differs. In these failed records the reported and raw values are equal because the reported value is the fallback. That equality is a consequence of the recorded procedure, not evidence that alignment would have made no difference.
Put eligibility beside the measurement
A downstream importer should retain alignment.ok, alignment.reason, architecture.aligned, raw_drift_score and the framing result. Check status before including an architecture score in a comparison intended for aligned measurements. Keep failed records visible with their diagnostic values and reasons, outside that ranking.
An accepted alignment is also only a prerequisite. It does not establish that framing passed. Avoid a convenient aggregate that silently substitutes a raw result whenever an aligned measurement is unavailable.
The public StagingBench records expose the information needed to preserve this boundary, and its methodology gives the wider measurement context. When registration fails, the application should communicate the failed dependency as clearly as the number it retained.
Top comments (0)