Why Your Link A/B Test Gets Uneven Traffic, and How to Fix It

You set a redirect test to split 50/50. A day later the dashboard says 62/38. Before you call it a winner, check the plumbing. In link tests, uneven traffic is often an assignment or caching bug, not random variation.

This post covers the common causes and a server-side pattern that keeps the split honest.

Random assignment does not guarantee even observed traffic

Math.random() < 0.5 gives each request an equal chance of variant A. It does not guarantee that the first 100 requests split 50/50. With small samples, variance is normal. With thousands of exposures, a 60/40 split is a signal.

Two things to separate:

• Assignment: which variant the system chose.
• Exposure: which variant actually reached the user and was logged.

Most 'uneven traffic' bugs live in exposure or caching, not in the random function.

The usual suspects

1. Permanent redirects

A 301 tells browsers and CDNs to cache the redirect. The first variant can stick for every later request. Use 302 or 307 for tests, and set Cache-Control: no-store on the redirect response.

2. CDN or edge cache

If your redirect is cached at the edge, the first request decides the variant for everyone in that cache region. Make the redirect response uncacheable, or move assignment to the edge with a key that varies per user.

3. Assignment after the redirect

If you assign the variant on the destination page with JavaScript, you miss users who block JS, bots, prefetchers, and link previews. You also log exposure late. Assign before the redirect and log there.

4. Unstable assignment keys

A new cookie per request, a session ID that resets, or a hash of IP + user agent can reshuffle users. Mobile IPs change, UA strings change, and cookie blockers delete state. Use a first-party cookie with a generated ID, and fall back to a deterministic hash only when necessary.

5. Bots and prefetch

Slack, email scanners, and browsers prefetch links. They can hit both variants or hit one repeatedly. Filter known bots from analysis, and log user-agent plus a bot flag at exposure time.

6. Multiple entry points

If one campaign link uses the splitter and another points directly to variant B, your traffic is not a clean 50/50. Route every test link through the same splitter.

7. Race conditions

Parallel requests from the same user can assign twice before the cookie is written. This is rare, but it happens with prefetch plus navigation. A deterministic hash of the stable cookie ID prevents double assignment.

A server-side pattern that holds up

Use a stable anonymous ID, hash it with the test name, and map the hash to a bucket. Set the cookie if missing, redirect with 302, and disable caching.

const crypto = require('crypto');

function bucket(key, buckets = 2) {
  const hash = crypto.createHash('sha256').update(key).digest();
  return hash.readUInt32BE(0) % buckets;
}

app.get('/test/:testName', (req, res) => {
  const cookieName = `ab_${req.params.testName}`;
  let anonId = req.cookies[cookieName];

  if (!anonId) {
    anonId = crypto.randomUUID();
    res.cookie(cookieName, anonId, {
      httpOnly: true,
      sameSite: 'lax',
      maxAge: 1000 * 60 * 60 * 24 * 30
    });
  }

  const variant = bucket(`${req.params.testName}:${anonId}`, 2);
  const target = variant === 0
    ? 'https://example.com/landing-a'
    : 'https://example.com/landing-b';

  res.set('Cache-Control', 'no-store');
  logExposure({ test: req.params.testName, variant, anonId, ua: req.get('user-agent') });
  res.redirect(302, target);
});

Notes:

• The cookie name includes the test name so different tests do not collide.
• no-store is not optional. Without it, a CDN can freeze the split.
• Log exposure before the redirect. Destination-page analytics cannot see users who never load the page.
• If you need to run this at the edge, use the same hash and set a first-party cookie or use an existing stable ID.

What about fairness? SHA-256 modulo 2 is close enough for practical tests. For more than two variants, use modulo N. For very large tests, consider a uniform hash range and map to buckets to avoid modulo bias, though the bias is negligible for small N.

Check for sample ratio mismatch

Sample ratio mismatch (SRM) means the observed split differs from the planned split. A quick chi-square check catches broken assignment early.

function srmWarning(counts, expectedRatio = [0.5, 0.5]) {
  const total = counts.reduce((a, b) => a + b, 0);
  let chi = 0;

  for (let i = 0; i < counts.length; i++) {
    const expected = total * expectedRatio[i];
    chi += Math.pow(counts[i] - expected, 2) / expected;
  }

  // 1 degree of freedom, p < 0.001
  return chi > 10.828;
}

const counts = [620, 380];
if (srmWarning(counts)) {
  console.warn('Sample ratio mismatch: check redirect caching, bot traffic, and assignment keys.');
}

If you see SRM, do not 'fix' it by rerunning the test and stopping when the split looks even. Fix the cause, then start a fresh test.

Practical checklist

• Use 302 or 307, never 301, for test redirects.
• Send Cache-Control: no-store on the redirect response.
• Assign from a stable first-party ID, not a fresh random value per request.
• Log exposure at redirect time with variant, test name, timestamp, and a bot flag.
• Route every campaign link through one splitter.
• Exclude known bots and internal traffic before analysis.
• Monitor SRM daily; stop and debug if it appears.

When you share test links

A/B tests create more URLs: variant A, variant B, docs, Slack messages. Manual edits cause mistakes. A short link can keep the shared URL stable while the splitter handles the variant choice. LinkBit is a link shortener you can use for that kind of sharing: LinkBit.

The main fix is still server-side assignment and cache-safe redirects. Short links just make distribution tidier.