Learn Prompt Sensitivity by Building a Tiny A/B Tester

Before we build anything, here is the output you should be able to reproduce by the end:

Prompt A -> labels: ['positive', 'negative', 'neutral']  valid: 3/3
Prompt B -> labels: ['positive', 'neg', 'neutral']      valid: 2/3
Invalid label rate changed by 33.3 percentage points.

The single learning question: how much does a one-word change in a prompt actually change whether an LLM's output parses correctly? We will answer it with a tiny, dependency-free tester instead of vibes.

Why this matters

When I first started calling LLM APIs, I treated the prompt as a fixed string and the model as the variable. In practice it is the other way around more often than I expected: the same model can produce clean, parseable output for one phrasing and slightly-off output for another, and the failure only shows up downstream when your parser throws.

A prompt sensitivity test just means: hold the inputs constant, vary the prompt minimally, and measure the property you actually care about (here: does the output match an allowed label set?).

Prerequisites

• Python 3.11+ (standard library only)
• Access to an OpenAI-compatible chat endpoint. I ran this on MonkeyCode, which currently offers free model access and a free server option, so a student budget is not a blocker for small experiments like this one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Any OpenAI-compatible endpoint works for the code below; nothing here is specific to one provider.

The tester

Save as prompt_ab.py:

import json
import os
import urllib.request

ALLOWED = {"positive", "negative", "neutral"}

PROMPT_A = "Classify the sentiment of this review. Reply with one word: positive, negative, or neutral. Review: {text}"
PROMPT_B = "Classify the sentiment of this review. Reply with one word: positive, neg, or neutral. Review: {text}"

FIXTURES = [
    ("I loved this film, it was fantastic.", "positive"),
    ("Terrible. I want my two hours back.", "negative"),
    ("It came out on Tuesday.", "neutral"),
]

def chat(prompt: str) -> str:
    body = json.dumps({
        "model": os.environ["MC_MODEL"],
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        os.environ["MC_BASE_URL"].rstrip("/") + "/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ["MC_API_KEY"],
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read())
    return data["choices"][0]["message"]["content"].strip().lower()

def run(name: str, template: str) -> list:
    labels = []
    for text, _ in FIXTURES:
        labels.append(chat(template.format(text=text)))
    valid = sum(1 for l in labels if l in ALLOWED)
    print(f"Prompt {name} -> labels: {labels}  valid: {valid}/{len(FIXTURES)}")
    return labels

def main():
    a = run("A", PROMPT_A)
    b = run("B", PROMPT_B)
    va = sum(1 for l in a if l in ALLOWED) / len(a)
    vb = sum(1 for l in b if l in ALLOWED) / len(b)
    print(f"Invalid label rate changed by {abs(va - vb) * 100:.1f} percentage points.")

if __name__ == "__main__":
    main()

Run it with your endpoint settings:

export MC_BASE_URL="https://your-endpoint/v1"
export MC_API_KEY="your-key"
export MC_MODEL="your-model-name"
python prompt_ab.py

Expected output

Your exact labels may differ by model, but the shape should look like the opening block: Prompt B (which quietly suggests neg instead of negative) produces at least one label outside the allowed set on some models, while Prompt A stays clean. If both are 3/3, try temperature 0.7 — several models only show the drift once sampling is on.

The error input

Add this fixture and predict what happens before running:

("The acting was good but the plot was awful.", "negative")

Most small models answer mixed here — which is not in ALLOWED. That is the point: the invalid-label path is not a bug in the model, it is a contract problem between your prompt and your parser. This is exactly where I use MonkeyCode's free server for quick iteration: rerunning a three-fixture test after each prompt tweak costs nothing and takes seconds, so I can probe edge cases instead of guessing.

Common mistakes I made

1. Testing the prompt, not the property. Counting "does it look right" by eye hides the failure. Always assert against an explicit allowed set.
2. Forgetting temperature: 0. Without it you cannot tell whether a change came from the prompt or from sampling noise.
3. Normalizing after the fact. Lowercasing/stripping in chat() is fine, but if you silently map neg to negative, you have deleted the very signal this test exists to show.
4. One fixture per class. Three inputs is a smoke test, not a result. Treat any percentage from this script as a hint, not a benchmark.

What you should understand now

• A prompt is an interface contract, and small wording changes can silently break the parser downstream.
• Measuring a property (valid label rate) beats reading outputs by eye.
• Deterministic settings are a prerequisite for attributing failures to the prompt.

Limitations and who should not use this

This tester will not tell you which prompt is semantically better, only which one parses more often. Do not use it for production evaluation, for comparing models on three fixtures, or anywhere you need statistical confidence — that needs a real eval harness with dozens to hundreds of labeled examples. Free tiers also change, so do not bake today's availability into a course or CI pipeline without checking first.

Extension exercise

Add a Prompt C that asks for JSON ({"label": ...}) and count JSON-parse failures separately from invalid-label failures. Which failure mode appears first as you raise temperature?

---

Before you scroll on: predict which fixture in this article fails on your model, and why. If you get a surprising result, I'd genuinely like to see your minimal counterexample in the comments. And if you want a zero-cost sandbox for the reruns, MonkeyCode's free server option is what I used for mine.