When you're building a measurement system, one of the first questions you need to answer is surprisingly simple.
How stable is the measurement itself?
Before optimizing our own Shopify store, Founder Lab, we wanted to establish a baseline.
Not just one scan.
Multiple scans.
Without changing a single byte on the store.
The result was more interesting than we expected.
The experiment
We scanned the exact same Shopify store six times.
- 1 baseline scan on July 12
- 5 rescans on July 17
- all rescans completed within 62 seconds
- absolutely nothing changed between scans
Here is what we measured.
| Scan | Score | Intent |
|---|---|---|
| Jul 12 | 78 | 6 |
| Jul 17 #1 | 76 | 4 |
| Jul 17 #2 | 76 | 4 |
| Jul 17 #3 | 78 | 6 |
| Jul 17 #4 | 77 | 5 |
| Jul 17 #5 | 77 | 5 |
Every deterministic factor stayed identical
This is where things became interesting.
Every deterministic component remained exactly the same.
| Component | Score |
|---|---|
| Visual | 11 |
| Schema | 15 |
| Trust | 15 |
| Technical | 7 |
| Price | 12 |
| Recommendation | 8 |
| Brand | 4 |
Every value above was bit-for-bit identical across all six scans.
Only one component moved.
Intent.
Intent changed between 4 and 6.
Everything else remained constant.
A simple observation
Subtract the intent score.
Every scan becomes identical.
78 - 6 = 72
76 - 4 = 72
76 - 4 = 72
78 - 6 = 72
77 - 5 = 72
77 - 5 = 72
The deterministic portion of the score never changed.
Only the model-evaluated component moved.
That immediately told us something important.
A one or two point change between two scans does not necessarily mean the store improved.
It may simply be normal model variance.
Why this matters
Many AI evaluation systems produce a single score.
People naturally compare:
78
vs
80
and conclude something improved.
Our first experiment suggests that's not always true.
If part of the score is generated by an LLM rather than deterministic rules, small movements may simply be statistical noise.
That changes how we evaluate future experiments.
Instead of trusting a single rescan, every meaningful change now requires multiple rescans before we consider it real.
The baseline
Before making any optimization, Founder Lab looked like this.
- AI Commerce Score: 78 / 100
- Score range across six scans: 76–78
- AI bot visits recorded: 0
- Technical score: 7 / 15 (our weakest deterministic factor)
What happens next
This was intentionally the most boring experiment we could run.
We changed nothing.
That was the point.
Before optimizing a system, you first need to understand how noisy the measurement itself is.
Our next experiments will focus on deterministic improvements first.
Technical issues.
Structured data.
Store architecture.
Only after that will we begin changing product copy and intent signals.
Every change will be rescanned multiple times.
Not because we expect every optimization to work.
But because we now know the measurement itself has a noise floor.
Understanding that may be just as important as improving the score itself.
Founder Lab is the live Shopify laboratory behind Atom Foundry's AI Commerce research.
Every experiment is published publicly, including the ones that fail.
Top comments (2)
This is a great real-world test. I've seen similar behavior when querying DeFi pools on Solana — the same endpoint can return slightly different metrics depending on when you call it, but only one or two fields actually shift. The rest stay surprisingly stable.
For Meteora DLMM pools, the number that changes most is usually the fee/TVL ratio or the latest volume. Everything else — pool address, token pair, bin step — remains constant.
Makes me wonder: in your Shopify test, was the changing number the inventory count, the price, or something else? Would love to know if you traced it back to the source.
Thanks for sharing this — very useful insight for anyone building data pipelines.
Thanks, Anh! 😉
That's actually the interesting part...nothing in the Shopify store changed at all.
Product data, pricing, schema, technical signals, trust signals... every deterministic component stayed identical across all six scans.
The only value that moved was the Intent score, which is evaluated by an LLM rather than computed from deterministic rules.
Once we subtracted the Intent score, every scan produced exactly the same underlying score (72).
That was the key finding: before optimizing an AI evaluation system, you first need to understand its own measurement noise.