Sample Ratio Mismatch (SRM) occurs when the actual distribution of users between A/B test variants differs significantly from the traffic allocation you configured.
For example, a test set to 50/50 should produce roughly equal numbers of users in Control and Treatment. A small difference is normal, but a statistically significant imbalance can indicate an experiment problem.
Why does SRM matter?
SRM can signal issues with:
Traffic allocation
User assignment
JavaScript implementation
Redirects
Exposure tracking
Analytics or data pipelines
Targeting rules
If these problems affect which users enter each variant, your conversion results may become unreliable.
How can you detect SRM?
Compare the observed user distribution with the expected distribution. A Pearson chi-squared goodness-of-fit test is commonly used to determine whether the difference is statistically significant.
Use unique assigned users rather than sessions when checking the sample ratio.
What should you do if SRM occurs?
Don't immediately declare an A/B test winner. Instead:
Verify the traffic allocation.
Check when the mismatch started.
Review assignment and targeting rules.
Audit exposure and analytics tracking.
Check browsers, devices, and other audience segments.
Investigate implementation or redirect issues.
Fix the underlying problem and consider restarting the test.
Final takeaway
SRM is an important A/B testing quality check. A statistically significant mismatch can indicate that your experiment has an assignment or tracking problem, making the results difficult to trust.
Read the complete guide:
https://www.brillmark.com/sample-ratio-mismatch-ab-testing-guide/
Top comments (0)