DEV Community

StagingBench Team
StagingBench Team

Posted on AI-assisted

Raw versus aligned drift: measuring the gap in StagingBench

A staging comparison can produce different drift measurements depending on whether the images have first been aligned. StagingBench, an independent test of AI virtual staging tools on real room photographs, provides both raw_drift and aligned_architecture_drift in its CSV.

That makes a direct paired calculation possible. For each eligible output, subtract aligned_architecture_drift from raw_drift. A positive gap means the aligned value is lower. A negative gap means it is higher.

The supplied file contains 120 rows from 12 tools applied to 10 rooms. Of those rows, 94 have alignment_ok set to True and enter this comparison. The remaining 26 are excluded as unknown, rather than treated as successful measurements of aligned drift. This is still a small test of 10 rooms, not a population estimate for virtual staging.

The paired result

Among the 94 included rows, the aligned value is lower in 79 cases, higher in 15, and equal in 0. The lower share is 84.04%, rounded to the nearest hundredth of a percentage point.

The mean raw drift is 27.423404, while the mean aligned drift is 22.296809. The mean paired gap, defined as raw minus aligned, is 5.126596 drift points. These means are rounded to six decimal places for reproducibility, not as a claim about the measurement's real-world precision.

The direction is common but not universal. The rows where alignment produces a higher value belong in the calculation too. Removing them would change the question from the average effect across eligible pairs to the average among pairs that moved in the preferred direction.

A compact reproduction

This Python snippet reads the CSV and filters before extracting either score. It uses the standard csv module and prints the counts, the lower share, and the means. The CSV is part of the open dataset linked from the StagingBench site.

import csv
with open('scores.csv', newline='') as f:
    rows = list(csv.DictReader(f))
known = [r for r in rows if r['alignment_ok'] == 'True']
raw = [float(r['raw_drift']) for r in known]
aligned = [float(r['aligned_architecture_drift']) for r in known]
gaps = [r - a for r, a in zip(raw, aligned)]
n = len(gaps)
lower = sum(g > 0 for g in gaps)
print(f'Included: {n}; excluded: {len(rows) - n}')
print(f'Lower: {lower}; equal: {sum(g == 0 for g in gaps)}; higher: {sum(g < 0 for g in gaps)}')
print(f'Lower share: {100 * lower / n:.2f}%')
print(f'Mean raw: {sum(raw) / n:.6f}')
print(f'Mean aligned: {sum(aligned) / n:.6f}')
print(f'Mean gap (raw minus aligned): {sum(gaps) / n:.6f}')
Enter fullscreen mode Exit fullscreen mode

I ran this snippet against the CSV and checked its results independently with decimal arithmetic. Both calculations give the figures above after rounding.

What the subtraction means

Alignment changes the coordinate relationship used for comparison. StagingBench fits a similarity transform using matched image features, then measures architecture drift using edges. The methodology reference identifies the method documented by the project.

The average gap therefore describes a change in the measurement after alignment. It does not establish that a staging tool repaired architecture or that a photographer recovered lost detail. No image was improved by the subtraction itself.

A positive gap can show that the raw comparison and the aligned comparison disagree about the amount of drift. The scalar columns cannot locate that disagreement within the room or explain the cause of an individual change. Likewise, a negative gap does not reveal which depicted feature accounts for the increase.

The unit matters here. The reported difference is in drift points. It is not a percentage improvement in the image, a percentage of architecture preserved, or an upload success rate.

Keep the paired question intact

Both means use exactly the same eligible rows. Consequently, their difference matches the mean of the individual gaps before rounding. Comparing a raw mean from the entire file with an aligned mean from a filtered subset would mix a measurement change with a sample change.

The aggregation is by tool and room record, so these results should not be read as independent observations from a large collection of properties. Reusing the same room inputs makes paired comparisons useful while limiting how far the conclusion travels.

For this dataset, alignment usually lowers the reported drift, and sometimes raises it. Keeping that variation visible makes the average gap interpretable: it summarizes a paired measurement comparison, with a defined direction and a defined sample.

Top comments (0)