Your A/B test has three goals and they disagree. Now what?

Every A/B testing tutorial ends the same way: run the test, wait for significance, ship the winner.

Then you run a real test and variant B converts 12% better on newsletter signups, brings in 4% less revenue per visitor, and bounce is flat. Nothing is significant except the signups. Ship it?

I spent an embarrassing amount of time on this question while building an A/B engine, and most of what I read online didn't help, because most of it assumes one metric. This post is what I ended up with. It's not novel — the statistics are decades old — but I couldn't find it written down in one place with working code, so here it is.

Why the p-value doesn't answer the question you're asking

Two problems, and the second one is the bad one.

Multiple comparisons. Three metrics at α = 0.05 means roughly a 14% chance of at least one false positive if nothing is actually different. Bonferroni fixes this, but now you need α = 0.017 per metric and your test needs to run three times as long. On a site doing 300 conversions a month that's not a fix, it's a refusal.

The p-value is answering a different question. It tells you the probability of your data assuming no difference exists. What you actually want to know is: if I ship B, how much do I expect to lose if I'm wrong? Those are not the same question and no amount of Bonferroni turns one into the other.

There's also the peeking problem — everyone checks the dashboard daily and stops when it goes green, which quietly inflates the false positive rate well past whatever α you wrote down. I'll come back to that, because Bayesian methods do not magically solve it, whatever you may have read.

Posterior first, decision second

For a conversion rate, the Beta-Binomial conjugate pair gives you the posterior in one line. With a uniform prior, after c conversions out of n visitors:

p | data ~ Beta(1 + c, 1 + n - c)

That's it. No closed-form comparison between two Betas that's worth implementing, so sample.

PHP has no Beta sampler in core, and no Gamma sampler either, so you build one. Marsaglia–Tsang, which needs a normal sampler underneath:

function normal_sample(): float {
    // Box–Muller
    $u1 = max(mt_rand() / mt_getrandmax(), 1e-12);
    $u2 = mt_rand() / mt_getrandmax();
    return sqrt(-2.0 * log($u1)) * cos(2.0 * M_PI * $u2);
}

function gamma_sample(float $shape): float {
    if ($shape < 1.0) {
        $u = max(mt_rand() / mt_getrandmax(), 1e-12);
        return gamma_sample($shape + 1.0) * pow($u, 1.0 / $shape);
    }
    $d = $shape - 1.0 / 3.0;
    $c = 1.0 / sqrt(9.0 * $d);

    while (true) {
        do {
            $x = normal_sample();
            $v = 1.0 + $c * $x;
        } while ($v <= 0.0);

        $v = $v * $v * $v;
        $u = max(mt_rand() / mt_getrandmax(), 1e-12);

        if ($u < 1.0 - 0.0331 * $x * $x * $x * $x) {
            return $d * $v;
        }
        if (log($u) < 0.5 * $x * $x + $d * (1.0 - $v + log($v))) {
            return $d * $v;
        }
    }
}

function beta_sample(float $a, float $b): float {
    $x = gamma_sample($a);
    $y = gamma_sample($b);
    return $x / ($x + $y);
}

Now the single-metric comparison:

$draws = 20000;
$wins = 0;
$loss_if_ship_b = 0.0;

for ($i = 0; $i < $draws; $i++) {
    $pA = beta_sample(1 + $cA, 1 + $nA - $cA);
    $pB = beta_sample(1 + $cB, 1 + $nB - $cB);

    if ($pB > $pA) {
        $wins++;
    }
    $loss_if_ship_b += max($pA - $pB, 0.0);
}

$prob_b_better  = $wins / $draws;
$expected_loss  = $loss_if_ship_b / $draws;

Two numbers instead of one. $prob_b_better is the intuitive one everybody quotes. $expected_loss is the one that should drive the decision: it's the average amount of conversion rate you give up, across the whole posterior, in the worlds where B is actually worse. If that number is 0.0004 and you genuinely don't care about four hundredths of a percentage point, ship B and stop thinking about it, even at 88% probability.

Pick that threshold before the test. Write it in the test config. It's the smallest effect you'd bother shipping for, and it forces a conversation about what the test is actually for.

Revenue per visitor is not a Beta

This is where I initially got it wrong. I treated revenue per visitor as a conversion-like quantity and got posteriors that were far too confident.

Revenue per visitor is zero-inflated (most visitors buy nothing) and heavy-tailed (one enterprise order distorts everything). A Beta is wrong, a Normal is wrong, a log-normal is closer but still assumes away the zeros.

I use a bootstrap instead. Resample the observed per-visitor revenue values with replacement, $draws times, take the mean of each resample. That's your posterior-ish distribution, and it inherits whatever ugly shape your real data has without you having to name it:

function bootstrap_means(array $values, int $draws): array {
    $n = count($values);
    $out = [];
    for ($d = 0; $d < $draws; $d++) {
        $sum = 0.0;
        for ($i = 0; $i < $n; $i++) {
            $sum += $values[mt_rand(0, $n - 1)];
        }
        $out[] = $sum / $n;
    }
    return $out;
}

It's O(draws × n) and it will hurt on large samples. Precompute a cumulative array and sample indices in blocks, or bootstrap on a fixed random subsample of 5,000 visitors per arm — the extra Monte Carlo noise is small compared to the sampling noise you're already living with.

Combining metrics without lying to yourself

Now the actual question. Three metrics, three posteriors, one decision.

The standard answer is an OEC — Overall Evaluation Criterion, from Kohavi's work at Microsoft. One weighted composite, agreed in advance, that the test is scored against. The two things people get wrong:

1. Weight relative uplift, not raw values. A conversion rate lives in [0,1], revenue per visitor might be €3.40, bounce rate is a percentage. Summing those with weights is meaningless. Convert each to relative uplift first, so everything is dimensionless.

2. Compute the composite inside the loop. This one matters and it's easy to miss. If you compute P(conv_B > conv_A), P(rev_B > rev_A) separately and then combine the summary numbers, you've thrown away the correlation between metrics. The draws are paired: draw i represents one coherent hypothetical world. Score the composite in that world, then aggregate.

$draws  = 20000;
$w      = ['conv' => 0.5, 'rev' => 0.4, 'bounce' => 0.1]; // must sum to 1
$rev_A  = bootstrap_means($revenue_A, $draws);
$rev_B  = bootstrap_means($revenue_B, $draws);

$oec_positive = 0;
$oec_loss     = 0.0;

for ($i = 0; $i < $draws; $i++) {
    $pA = beta_sample(1 + $cA, 1 + $nA - $cA);
    $pB = beta_sample(1 + $cB, 1 + $nB - $cB);
    $bA = beta_sample(1 + $bounceA, 1 + $nA - $bounceA);
    $bB = beta_sample(1 + $bounceB, 1 + $nB - $bounceB);

    $u_conv   =  ($pB - $pA) / $pA;
    $u_rev    =  ($rev_B[$i] - $rev_A[$i]) / $rev_A[$i];
    $u_bounce = -(($bB - $bA) / $bA);   // lower bounce is better, so flip

    $oec = $w['conv'] * $u_conv
         + $w['rev']  * $u_rev
         + $w['bounce'] * $u_bounce;

    if ($oec > 0) {
        $oec_positive++;
    }
    $oec_loss += max(-$oec, 0.0);
}

$prob_b_better  = $oec_positive / $draws;
$expected_loss  = $oec_loss / $draws;  // in relative-uplift units

Ship B when $expected_loss is below your threshold of caring. Not when $prob_b_better crosses 95%.

Fix the weights before you look at any data. Otherwise you will — I promise you will — nudge them until your preferred variant wins. If you can't decide the weights before the test, you don't have a hypothesis, you have a hope.

Guardrails are not goals

Some metrics don't belong in the composite at all. Page load time, JS error rate, checkout failures — these are guardrails, and no amount of signup uplift should be allowed to buy a regression in them.

They get a veto, not a weight:

if ($p95_load_B > $p95_load_A * 1.10) {
    return Decision::BLOCKED;   // no composite, no discussion
}

Mixing a guardrail into the OEC is how you ship a variant that converts 8% better and breaks checkout for Safari.

The peeking problem is still there

You will read that Bayesian methods let you monitor continuously without penalty. That's half true and the half that's false will burn you.

Expected loss is a decision-theoretic quantity, so stopping when it falls under your threshold isn't the same statistical sin as stopping when a p-value dips below 0.05. But with tiny samples the posterior is wide and jumpy, and "expected loss happens to be low right now" is a thing that occurs by chance early in a test.

The pragmatic fix I settled on: a hard minimum before any decision is offered at all. Both arms need a floor of visitors and a floor of conversions — conversions are what actually carry the information, and 50,000 visitors with 11 conversions tells you almost nothing. Below the floor the UI shows the posterior and refuses to say anything else.

Not elegant. Works.

Performance, since this runs on shared hosting

20,000 draws × 4 Beta samples × 2 arms is a lot of mt_rand() calls in PHP, and it is nowhere near free on a €4/month host.

What made it survivable:

• Run it on a scheduled job, not on dashboard load. Cache the decision payload.
• Aggregate counts ($nA, $cA, …) in a summary table updated incrementally. Never scan the raw events table to compute a posterior.
• Drop to 10,000 draws. The Monte Carlo error on $prob_b_better at 10k draws is about ±0.5 percentage points, which is invisible next to the uncertainty in your data.
• Bail out early when the result isn't close. If the two 99% intervals don't overlap, you don't need the simulation.

On my setup, the full three-metric run at 20k draws takes [X] ms for [N] visitors per arm, down from [Y] ms before the summary table. Replace these with your real numbers or delete the paragraph — made-up benchmarks are worse than no benchmarks.

What I'd tell past me

The statistics were the easy part. The hard part was accepting that "which variant won?" is a badly formed question, and that the honest output of an A/B test is a distribution plus a decision rule you committed to in advance — not a green checkmark.

The engine got simpler once I stopped trying to produce certainty and started producing an expected cost of being wrong.

---

I'm the author of Opti-Behavior, a self-hosted analytics plugin, and the decision engine described here is what's running inside its A/B testing module — so take my enthusiasm for this approach with the appropriate grain of salt. The maths is standard and framework-agnostic; port it wherever you like. Happy to argue about the weighting scheme in the comments, it's the part I'm least sure about.