A/B Testing for a One-Person Company: AI Variants, Hash Split, Bayesian Auto-Switch

The Pain
You finish an article, come up with three candidate headlines, and pick the one that "feels best." You rework the landing page copy through a few versions and go with whatever "looks right." One channel underperforms, and you hesitate over whether to switch it out.

Stop right there. Whenever you choose by "feel," you are gambling on luck. Big companies do A/B testing: half the users see A, half see B, and the data speaks. But traditional A/B testing belongs to teams — it demands statistics expertise, tens of thousands of samples, and year-billed tools like Optimizely. How is a one-person company supposed to afford that?

What You'll Learn

• Why OPC swaps textbook significance testing for "good-enough and automatic" A/B testing
• Layer 1: one YAML experiment config + an LLM that generates 3-5 variants for you
• Layer 2: consistent hashing — traffic split with zero state and zero storage
• Layer 3: Beta-Bernoulli Bayesian convergence that auto-switches the winning variant
• The Before vs After of what you actually save, and the one experiment discipline you must keep

Opening: changing a headline by gut feel is gambling

You finish writing an article, come up with three candidate headlines, and finally pick the one that "feels best." You rework the landing page copy through several versions and choose the one that "looks right." One channel is underperforming, and you keep hesitating over whether to swap it out.

Stop. As long as you make decisions by "feel," you are gambling. How do big companies do it? A/B testing: half the users see A, half see B, and the data speaks. But traditional A/B testing is a team sport — it needs statistics expertise, tens of thousands of samples, and tools billed by the year like Optimizely. How is a one-person company supposed to afford that?

The answer: OPC does not need traditional A/B testing — it needs A/B testing that is "good enough and automatic." No chasing academic statistical significance; instead, find the better variant quickly with limited traffic, and switch to it automatically. The system has 3 layers: AI-generated experiment variants, hash-based traffic split, and Bayesian automatic convergence. Once it is built, all you do is write the config for a new experiment — splitting, collecting, judging, and switching all run themselves.

In the previous article, we built the automated data analytics engine — user behavior events land in an events table. Today's A/B system plugs straight into it.

---

1. First, get this straight: the A/B testing OPC needs is not the textbook version

The textbook A/B testing workflow: compute the sample size first, run for a fixed period, compute the p-value, and only declare significance below 0.05. This workflow is unrealistic for OPC in three ways:

1. Not enough traffic — an article gets a few thousand reads; split into two groups, each gets fewer; by the time it is "statistically significant," the content's moment has passed
2. Not enough time — a fixed period means you sit and wait, but OPC's rhythm is fast trial-and-error
3. Not enough people — there is no data scientist around to interpret the results for you

So the OPC version swaps three things:

• Decision goal: not "prove B beats A with 99% confidence," but "under current traffic, is the probability that B beats A high enough? If yes, switch"
• Stopping rule: no fixed period — judge in real time with the Bayesian posterior probability, and converge when it is enough
• Execution: auto-switch after convergence, with no human watching

In one sentence: a big-company A/B test is an academic experiment; an OPC A/B test is an automatic decision machine.

The essence of this comparison is swapping the decision tool: from "gut feel" to "posterior probability." For that to work, event instrumentation must be in place up front — the events table we built last article needs two more things recorded: which variant the user was assigned (the exposure event), and whether that user completed the target action (the conversion event). Only two instrumentation fields: experiment and variant, and they must match the names in the experiment config exactly — miss one field and the convergence judgment is wrong.

---

2. Overall architecture: the 3-layer automated A/B system

Figure 1: The 3-layer A/B system — define, split, collect, converge, switch. Data flows one way.

┌────────────────────────────────────────────────────────┐
│Layer 1: Experiment definition & AI variant generation  │
│-> experiment.yaml declares the experiment              │
│-> LLM generates N variants from the original copy      │
├────────────────────────────────────────────────────────┤
│Layer 2: Automatic traffic split                        │
│-> consistent hashing: hash(exp:user_id) % N            │
│-> group A (control) | group B (variants)               │
├────────────────────────────────────────────────────────┤
│Layer 3: Event collection & Bayesian convergence        │
│-> reuse the events table from the previous article     │
│-> Beta posterior updates -> win probability            │
│-> probability above threshold -> auto-switch winner    │
└────────────────────────────────────────────────────────┘

Each layer has a single responsibility, and data flows one way. Let's go through them layer by layer.

---

3. Layer 1: experiment definition and AI variant generation

First step: define the experiment config. One experiment file describes the experiment subject (headline / landing page / email subject), the original version, the evaluation metric, and the stop threshold:

# abtest/experiment.py
import dataclasses

@dataclasses.dataclass
class Experiment:
    name: str                 # experiment name, e.g. "landing_v1"
    target: str               # experiment subject: title | landing | email_subject
    control: str              # control-group original copy
    variants: list[str]       # variants under test
    metric: str = "conversion"  # evaluation metric; matches an event name in the events table
    traffic_ratio: float = 0.5  # share of traffic routed into the experiment
    stop_threshold: float = 0.95  # converge once win probability exceeds 95%

EXPERIMENTS: dict[str, Experiment] = {}

def register(exp: Experiment):
    EXPERIMENTS[exp.name] = exp

Where do variants come from? One person cannot squeeze out 10 headlines, but an LLM can. Write a generator that sends the original copy plus the variant requirements to the model and returns a candidate list:

# abtest/variant_generator.py
import openai

client = openai.OpenAI()

def generate_variants(exp: Experiment, n: int = 3) -> list[str]:
    """Generate n variants from the control copy; returns a candidate list"""
    prompt = (
        "You are a growth-copywriting expert. Here is a landing-page headline. "
        f"Generate {n} alternative headlines from different angles, each no "
        "more than 15 characters, output as a numbered list.\n"
        f"Original headline: {exp.control}"
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.9,
    )
    lines = [l.strip() for l in resp.choices[0].message.content.splitlines() if l.strip()]
    variants = [l.split(".", 1)[-1].strip() for l in lines if "." in l]
    return variants[:n]

After generation, confirm the candidates once by hand and write the variants into the experiment config. Note: set temperature to 0.9 so the LLM produces more "unstable" candidates — a variant test wants difference, not another copy of yourself.

---

4. Layer 2: automatic traffic split — the same user always sees the same version

Traffic splitting has exactly one hard requirement: the same user must see the same variant for the entire experiment. Otherwise the user sees A today and B tomorrow, and the data is all dirty.

With consistent hashing, you don't need to store any split records:

# abtest/traffic_split.py
import hashlib

def assign_variant(user_id: str, exp_name: str, total_groups: int) -> int:
    """Stably assign a user to a group in 0..total_groups-1; the same user always lands in the same group"""
    key = f"{exp_name}:{user_id}".encode()
    digest = int(hashlib.md5(key).hexdigest(), 16)
    return digest % total_groups

def should_serve_experiment(user_id: str, exp: Experiment) -> bool:
    """Decide whether this user enters the experiment based on the traffic ratio"""
    bucket = assign_variant(user_id, exp.name + ":gate", 100)
    return bucket < exp.traffic_ratio * 100

There is exactly one integration point — when rendering the headline, landing page, or email subject:

# abtest/middleware.py
def resolve_content(user_id: str, exp: Experiment) -> str:
    """Return the copy version this user should see"""
    if not should_serve_experiment(user_id, exp):
        return exp.control
    group = assign_variant(user_id, exp.name, len(exp.variants) + 1)
    if group == 0:
        return exp.control          # group A: control
    return exp.variants[group - 1]  # groups B/C: variants

That completes the split layer: no DB lookup, no recorded state, a pure function — any request can immediately compute which version it should see.

Figure 2: Consistent hashing keeps every user pinned to one group for the whole experiment — zero state, zero storage.

The traffic ratio can be tuned by risk — a low-risk experiment like changing a headline gets 50% of traffic; a high-risk one like changing prices starts with 10% as a trial, just set traffic_ratio to 0.1.

---

5. Layer 3: Bayesian convergence — auto-switch when the data is in

Once a user sees a variant, the behavioral events (exposure / conversion / payment_done) land in the events table from the previous article. Now comes the core: judging which variant is better.

Use the Beta-Bernoulli model: each variant's conversion rate is a Beta distribution, updated once per observed conversion. The Bayesian advantage is — with few samples the judgment is conservative; with enough samples it converges naturally, and at any moment it can tell you the probability that "B beats A":

# abtest/bayesian.py
import random

import psycopg2

DB_DSN = "postgresql://user:***@localhost:5432/analytics"
BETA_PRIOR_A = 1.0  # Beta(1,1) uninformative prior
BETA_PRIOR_B = 1.0

def fetch_counts(exp_name: str, days: int = 14) -> dict:
    """Aggregate exposure and conversion counts per variant from the events table"""
    conn = psycopg2.connect(DB_DSN)
    try:
        with conn.cursor() as cur:
            cur.execute("""
                SELECT payload->>'variant',
                       COUNT(*) FILTER (WHERE event_type = 'exposure'),
                       COUNT(*) FILTER (WHERE event_type = %s)
                FROM events
                WHERE payload->>'experiment' = %s
                  AND created_at >= NOW() - INTERVAL '%s days'
                GROUP BY payload->>'variant'
            """, ("conversion", exp_name, str(days)))
            return {r[0]: {"exposures": r[1], "conversions": r[2]} for r in cur.fetchall()}
    finally:
        conn.close()

def win_probability(a: dict, b: dict) -> float:
    """Posterior probability that B beats A, approximated by Monte Carlo sampling"""
    random.seed(42)
    wins = 0
    samples = 20000
    for _ in range(samples):
        pa = random.betavariate(BETA_PRIOR_A + a["conversions"], BETA_PRIOR_B + a["exposures"] - a["conversions"])
        pb = random.betavariate(BETA_PRIOR_A + b["conversions"], BETA_PRIOR_B + b["exposures"] - b["conversions"])
        if pb > pa:
            wins += 1
    return wins / samples

Then run the convergence check every 6 hours; when the win probability crosses the threshold, auto-switch and notify:

# abtest/auto_converge.py
from abtest.bayesian import fetch_counts, win_probability
from abtest.experiment import EXPERIMENTS

def check_and_switch(exp_name: str):
    exp = EXPERIMENTS[exp_name]
    counts = fetch_counts(exp_name)
    if exp.control not in counts or len(counts) < 2:
        return  # not enough data yet — keep waiting

    best_variant = max(
        (v for v in counts if v != exp.control),
        key=lambda v: counts[v]["conversions"] / max(counts[v]["exposures"], 1),
        default=None,
    )
    if not best_variant:
        return

    p = win_probability(counts[exp.control], counts[best_variant])
    if p >= exp.stop_threshold:
        # win probability high enough -> write the winner into the live config; switch complete
        apply_winner(exp, best_variant)
        notify(f"Experiment '{exp.name}' converged: variant '{best_variant}' wins with {p:.1%} probability — auto-switched")

How apply_winner works depends on your system — for a headline, write into the article metadata; for a landing page, update the page config; for an email subject, push it into the send queue. The core action is one thing: make the winning variant the default version. And two instrumentation points must always come in pairs: whenever resolve_content returns a version, immediately write one exposure event; the conversion event reuses the conversion instrumentation from the previous article. Miss the exposure event and the denominator in the Bayesian formula is wrong — the whole judgment collapses.

Figure 3: The convergence loop — Beta updates, win probability, threshold check every 6 hours, and the "no" branch that keeps collecting.

cron config (every 6 hours):

# crontab -e — add the following line
0 */6 * * * cd /home/opc/abtest && python3 -m abtest.auto_converge --all >> /var/log/abtest.log 2>&1

---

6. Before vs After: what you save

Stage	Traditional approach	This system
Thinking of variants	one person racking their brain for headlines	LLM generates 3-5 candidates in one shot
Splitting traffic	manually sharing links, manual stats	consistent hashing, zero state, zero storage
Judging	gut feel / Excel decisions	Bayesian posterior probability
Switching	manually editing the live config	auto-switch after convergence
Monitoring	refreshing the dashboard every day	automatic check every 6 hours, result pushed to WeCom

All you have to do: write experiment.yaml (a few dozen lines), confirm the generated variants once by hand, then wait for the notification.

---

7. Advanced thinking: why Bayesian, why automatic

1. With small samples, Bayesian is more honest than p-values. The frequentist school must accumulate enough samples before it can speak; Bayesian can give "the probability that B is better under current evidence" at any moment. With little traffic that probability hesitates — say 52% — and that is exactly right; it won't lie to you and say "significant." Once enough data arrives, the probability naturally climbs above 95%. For OPC, "don't switch below 95%" is the steadiest decision discipline.

2. "Automatic" matters more than "accurate." The output of a big-company A/B test is a report, because there is a person reading it and making decisions. OPC has no one to read reports, so the system must decide for itself. The convergence threshold lives in the config; when the time comes, the switch executes automatically — that is unattended operation. As the previous article said: OPC's time is spent only where the system cannot reach.

3. Experiment discipline: change one variable at a time. However automatic the system is, it cannot save you if you test headline, landing page, and price simultaneously — when you win, you won't know which one won. Automation presupposes a clean experiment. One variable per experiment is the bottom line you must hold at the design stage.

---

8. Summary and next steps

Today we built the 3-layer automated A/B system:

1. Layer 1: experiment.yaml defines the experiment + the LLM generates variants — one person can still have 5 candidates
2. Layer 2: consistent-hash traffic split — the same user sees exactly one version the whole time, zero state, zero storage
3. Layer 3: reuse the events table from the previous article, Beta-Bernoulli Bayesian convergence, auto-switch when the win probability exceeds 95%

Now your OPC system can "trial on its own, converge on its own, switch on its own." But there is still one problem: this system runs on a server — what happens when it goes down? When the database connection pool fills up at 3 a.m., who gets up to handle it?

Next up: From 996 to 007 — OPC's unattended operations system

Once articles, customer service, delivery, analytics, and optimization are all automated, the system itself becomes your employee. But employees get sick, and servers crash. In the next article, we build OPC's unattended operations stack: health checks, auto-restart, anomaly alerts, log rotation — let the system take care of itself, so you can finally sleep well.

---

About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.