When you're building a measurement system, one of the first questions you need to answer is surprisingly simple.
How stable is the measurement itself?
Before optimizing our own Shopify store, Founder Lab, we wanted to establish a baseline.
Not just one scan.
Multiple scans.
Without changing a single byte on the store.
The result was more interesting than we expected.
The experiment
We scanned the exact same Shopify store six times.
- 1 baseline scan on July 12
- 5 rescans on July 17
- all rescans completed within 62 seconds
- absolutely nothing changed between scans
Here is what we measured.
| Scan | Score | Intent |
|---|---|---|
| Jul 12 | 78 | 6 |
| Jul 17 #1 | 76 | 4 |
| Jul 17 #2 | 76 | 4 |
| Jul 17 #3 | 78 | 6 |
| Jul 17 #4 | 77 | 5 |
| Jul 17 #5 | 77 | 5 |
Every deterministic factor stayed identical
This is where things became interesting.
Every deterministic component remained exactly the same.
| Component | Score |
|---|---|
| Visual | 11 |
| Schema | 15 |
| Trust | 15 |
| Technical | 7 |
| Price | 12 |
| Recommendation | 8 |
| Brand | 4 |
Every value above was bit-for-bit identical across all six scans.
Only one component moved.
Intent.
Intent changed between 4 and 6.
Everything else remained constant.
A simple observation
Subtract the intent score.
Every scan becomes identical.
78 - 6 = 72
76 - 4 = 72
76 - 4 = 72
78 - 6 = 72
77 - 5 = 72
77 - 5 = 72
The deterministic portion of the score never changed.
Only the model-evaluated component moved.
That immediately told us something important.
A one or two point change between two scans does not necessarily mean the store improved.
It may simply be normal model variance.
Why this matters
Many AI evaluation systems produce a single score.
People naturally compare:
78
vs
80
and conclude something improved.
Our first experiment suggests that's not always true.
If part of the score is generated by an LLM rather than deterministic rules, small movements may simply be statistical noise.
That changes how we evaluate future experiments.
Instead of trusting a single rescan, every meaningful change now requires multiple rescans before we consider it real.
The baseline
Before making any optimization, Founder Lab looked like this.
- AI Commerce Score: 78 / 100
- Score range across six scans: 76–78
- AI bot visits recorded: 0
- Technical score: 7 / 15 (our weakest deterministic factor)
What happens next
This was intentionally the most boring experiment we could run.
We changed nothing.
That was the point.
Before optimizing a system, you first need to understand how noisy the measurement itself is.
Our next experiments will focus on deterministic improvements first.
Technical issues.
Structured data.
Store architecture.
Only after that will we begin changing product copy and intent signals.
Every change will be rescanned multiple times.
Not because we expect every optimization to work.
But because we now know the measurement itself has a noise floor.
Understanding that may be just as important as improving the score itself.
Founder Lab is the live Shopify laboratory behind Atom Foundry's AI Commerce research.
Every experiment is published publicly, including the ones that fail.
Top comments (0)