"GA4 doesn't match, so I don't trust the test."
I hear this constantly, and the instinct behind it is right. Two systems reporting the same experiment should not disagree.
But they are not reporting the same thing, and once you see why, the disagreement becomes useful rather than alarming.
They are counting different populations
An A/B testing tool counts people it bucketed into an experiment. Analytics counts people who loaded a page and fired a tag.
Those two sets overlap. They are not equal, and nothing you configure will make them equal.
A visitor with an ad blocker that eats your analytics but not your testing snippet is in one and not the other. A visitor who leaves before your tag manager finishes is in neither, or in one. A bot filter that treats the two scripts differently splits them further.
So the question is never "why do the totals differ". It is "does the difference move with the variant", which is a completely different and much more answerable question.
The identity problem underneath it
Here is the mechanism that decides what "the same visitor" even means, and almost nobody checks it before reading a result.
The visitor id lives in one domain's localStorage and cookie jar. That single fact produces this:
| Where the same person browses | Counted as |
|---|---|
example.com and shop.example.com
|
1 visitor |
| Two experiments on the same site | 1 visitor |
site-A.com and site-B.com
|
2 visitors |
| The same domain in two different organisations | 2 visitors |
Subdomains of one registrable domain share a visitor id. Two genuinely different domains cannot, because browsers give each domain its own isolated storage.
The same human arrives as two unrelated visitors, and is metered twice.
There is no cross-domain identifier, and we are not planning to add one. That is a deliberate position, not a gap in the roadmap. Stitching identity across domains means shipping something that follows people between sites, and we would rather explain the limitation than build that.
If your brand runs example.com and example.co.uk as separate registrable domains, your testing tool sees two audiences. So does your analytics, differently. That is most of the gap right there.
How to actually reconcile them
Stop comparing totals. Join on the event.
GA4 emits an experience_impression event carrying exp_variant_string. Pipe GA4 to BigQuery and join that event to your conversion events on user_pseudo_id.
That is the same join GA4 does internally, except in raw SQL, with no Custom Dimension cap, no Audience limits, and no (other) rollup swallowing your variant names.
The (other) bucket is the one that catches people out. GA4 has cardinality limits, and when you exceed them it quietly rolls values up. Your variant B stops being variant B and becomes part of a bucket, and the report still renders, and nobody notices.
For concurrent experiments this stops being optional. Custom Dimensions run out fast, and the in-product GA4 reporting cannot express "bucketed into experiment 1 variant B and experiment 2 control".
What we keep, and for how long
The ledger is the canonical store for an experiment. Every exposure and every goal that was accepted, with its payload.
Behind it sits a raw compressed event archive, and that archive is purged nightly once it passes 24 months. It is a data-retention policy, not a billing lever, and it is unaffected by your plan or your subscription state.
The ledger the results are computed from survives the purge and stays on the results page for the life of the experiment. Only the raw events age out.
If you need raw events beyond 24 months, run the async export inside the window and keep the CSV in your own storage. We would rather tell you that now than when you go looking for month 25.
The practical rule
Pick one system as the system of record for the decision, before the test starts. For an experiment, that should be the testing tool, because it is the only one that knows who was bucketed.
Use analytics for the diagnosis, not the verdict. It is excellent at telling you where a difference came from and poor at telling you whether the difference is real.
And when the numbers disagree, check the direction of the gap rather than its size. A gap that is the same shape in both arms is measurement. A gap that differs by variant is a bug, and it is the most important thing on your screen.
ABTestly is A/B testing for teams that write their variations in code rather than in a visual editor. The GA4 and BigQuery join is documented at docs.abtestly.com, and the prices are published at abtestly.com/pricing.
Top comments (1)
the identity table is the part people will skim past and it's the most important one. subdomains of one registrable domain sharing a visitor id while two genuinely different domains can't is exactly the kind of thing that makes cross-domain experiments silently meaningless. i respect the "deliberate position, not a gap in the roadmap" line too. most vendors would dress that up as an upcoming feature instead of saying the thing outright.
one quibble: "check the direction of the gap rather than its size" is sound for diagnosis, but teams under time pressure will read it as permission to eyeball. the join-on-the-event section is the actual fix and it deserves to be the headline rule. the rest of the piece argues for it already.