Your Prompt A/B Test Is Probably Random Noise: A Paired-Test Harness for Free-Tier LLMs

Last week I almost shipped a worse prompt because I ran it exactly once.

I had rewritten a system prompt for a small internal tool. I fired both versions at five sample queries. The new prompt won 4–1. I was about to deploy it when a teammate asked, “Did you run it again with a different order?” I did. This time the old prompt won.

Same model. Same task. Different winner.

That’s the dirty secret of most prompt “A/B tests” I see on the internet: they’re single-run anecdotes dressed up as experiments. They ignore order effects, model nondeterminism, and small sample sizes. This week’s AI debates on DEV are full of confident claims about which assistant is better — but almost nobody shows a confidence interval.

So I built a minimal paired-test harness. It runs on a free server, uses free model access, and takes about an hour to set up. You can reuse it for any prompt comparison you care about.

The one-run trap

Let’s say you want to test whether adding “be concise” to a prompt improves answers. You call the model twice: once with, once without. If the concise version happens to get lucky on one question, you’ll see a difference that isn’t real.

LLM calls are stochastic. Even at temperature 0, batch headers, routing, and caching can flip outputs. Worse, if you run all baseline calls first and all variant calls second, you’re measuring time-of-day drift as much as prompt quality.

A paired design fixes both issues: for each test case, you run both prompts back-to-back, and you randomize the order. Then you compare outcomes within each case, not as two independent buckets.

What you need for a credible comparison

A minimal but honest benchmark needs four things:

1. A fixed dataset of 20–30 representative cases.
2. A deterministic scoring function (exact match, unit test pass/fail, or regex).
3. Pairing: same cases for both prompts, order randomized.
4. A confidence interval, because point estimates lie.

You don’t need thousands of examples. For a regression gate, 25 cases repeated a few times is enough to catch 10-point drops — if you measure the right thing.

Build the harness

Step 1: Define your task and metric

I’ll use a toy math/trivia task so the scoring is objective. You can replace prompt_a, prompt_b, and the score() function with anything: code generation, summarization quality via a rubric, or structured output conformance.

Step 2: Collect 20–30 fixed cases

Here are three to get you started. Add your own — ideally from real usage logs, not just made-up questions.

cases = [
    ('What is 17 * 23?', '391'),
    ('What is 144 / 12?', '12'),
    ('What is the capital of France?', 'Paris'),
    # add 25+ more...
]

Step 3: Run this paired script

The script below randomizes call order, scores each answer, and prints a Wilson confidence interval for the probability that prompt A wins (or ties) against prompt B.

import json
import math
import os
import random
from urllib import request

ENDPOINT = os.getenv('LLM_ENDPOINT', 'http://localhost:11434/v1/chat/completions')
KEY = os.getenv('LLM_API_KEY', '')
MODEL = os.getenv('LLM_MODEL', 'your-model')

def call_llm(prompt: str) -> str:
    payload = {
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0,
    }
    req = request.Request(
        ENDPOINT,
        data=json.dumps(payload).encode(),
        headers={
            'Content-Type': 'application/json',
            'Authorization': f'Bearer {KEY}',
        },
    )
    with request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data['choices'][0]['message']['content'].strip()

def prompt_a(q: str) -> str:
    return f'Answer the question. Be concise.\nQuestion: {q}\nAnswer:'

def prompt_b(q: str) -> str:
    return f'You are a helpful assistant. Provide a direct answer with no explanation.\nQuestion: {q}\nAnswer:'

cases = [
    ('What is 17 * 23?', '391'),
    ('What is 144 / 12?', '12'),
    ('What is the capital of France?', 'Paris'),
    # ...
]

def score(response: str, expected: str) -> bool:
    return response.strip().lower() == expected.strip().lower()

random.seed(42)
a_wins = b_wins = ties = 0

for question, expected in cases:
    # Randomize order to cancel out time-of-day effects
    if random.random() < 0.5:
        resp_a = call_llm(prompt_a(question))
        resp_b = call_llm(prompt_b(question))
    else:
        resp_b = call_llm(prompt_b(question))
        resp_a = call_llm(prompt_a(question))

    ok_a = score(resp_a, expected)
    ok_b = score(resp_b, expected)

    if ok_a and not ok_b:
        a_wins += 1
    elif ok_b and not ok_a:
        b_wins += 1
    else:
        ties += 1

n = len(cases)
p = (a_wins + 0.5 * ties) / n  # winning probability with ties as 0.5
z = 1.96
denom = 1 + z * z / n
center = (p + z * z / (2 * n)) / denom
half = (z * math.sqrt((p * (1 - p) + z * z / (4 * n)) / n)) / denom

print(f'A wins: {a_wins}, B wins: {b_wins}, ties: {ties}')
print(f'P(A wins or ties): {p:.3f}')
print(f'95% Wilson interval: [{center - half:.3f}, {center + half:.3f}]')

If the interval doesn’t cross 0.5, you have evidence that one prompt is better. If it does, treat the two prompts as equivalent until you collect more data.

Step 4: Turn it into a regression gate

You don’t want to run this only once. Schedule it nightly. If you’re using MonkeyCode’s free server option, you can host the harness there without paying for a VM. Just add a cron job:

0 6 * * * cd /home/user/prompt-harness && python run_harness.py >> results.log

Then grep the log every morning for intervals that flip direction.

MonkeyCode also offers free model access, which keeps the per-run cost at zero for a small dataset. That’s what makes this kind of repeated testing practical instead of theoretical.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and when this approach fails

This harness is simple, and that’s the point. But don’t over-trust it.

• Your 25 cases are a sample. If your real usage looks different, the verdict can flip. Build the dataset from actual prompts you’ve seen in production.
• Exact match scoring is crude. For code generation, run unit tests. For summarization, you’ll need a rubric and a human reviewer — or a stronger LLM judge, which introduces its own biases.
• Free tiers have variable latency and occasionally drop requests. The script will fail loudly; wrap it in try/except and log failures separately so a network blip doesn’t count as a wrong answer.
• If your prompt difference is tiny, you’ll need more than 30 examples. This harness is for catching obvious regressions, not for detecting a 2% improvement.

Who should not use this

If you’re building a novel eval dataset that will influence a launch decision, go hire an eval engineer. This script is not a substitute for proper test–retest reliability, inter-annotator agreement, or a multi-day sampling plan.

Use it when you need a quick, honest, free way to stop yourself from shipping a worse prompt.

The takeaway

Single-run prompt comparisons are noise. Pair your tests, randomize the order, and report a confidence interval. That’s not marketing math — it’s how you actually know which prompt is better.

If you have a similar harness, fork this pattern and replace the dataset with your own. Your future self, on the 20th iteration of “just one more tweak”, will thank you.