A/B Test Prompts Without a Feature Flag: A Free-Server Workflow

A prompt change is a hypothesis.
Most teams test it by eye.
That is not a measurement.

This tutorial builds a minimal A/B test for model prompts.
It runs on a free server.
It needs one hash function and one log file.
No feature flag service is required.

This workflow uses MonkeyCode's free model access for the calls.
The free server option hosts the logger.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a hash, not a coin flip

A coin flip is not reproducible.
A hash of the request ID is.
The same request always lands in the same variant.
That makes debugging possible.

This is not true randomization.
It is deterministic assignment.
For a smoke test, that is enough.

Step 1: Define the response contract

Both prompts must return the same JSON shape.
If the shapes differ, you are comparing two different products.
Start with a minimal schema.

{
  "output": "string",
  "valid": "boolean"
}

Verify: call each prompt once with the same input.
Check that both outputs match the schema.
If they do not, fix the prompts before you start.

Step 2: Build deterministic assignment

Use SHA-256 on the request ID.
Take the first eight hex characters.
Convert them to an integer.
Use the lowest bit to pick A or B.

import hashlib

def variant(request_id: str) -> str:
    digest = hashlib.sha256(request_id.encode()).hexdigest()
    return "A" if int(digest[:8], 16) % 2 == 0 else "B"

Verify: run this on 100 fake IDs.
Count how many land in A and B.
Expect a rough 50/50 split.
Do not use random() here.

Step 3: Log before the model call

Log the decision first.
If the process crashes, you still know the request existed.
Write one JSON object per line.

import json
import time
import hashlib

def log_entry(entry):
    with open("experiment.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

entry = {
    "request_id": request_id,
    "variant": variant(request_id),
    "prompt_hash": hashlib.sha256(prompt.encode()).hexdigest()[:12],
    "timestamp": time.time(),
    "status": "pending",
}
log_entry(entry)

Verify: run one request.
Open experiment.jsonl.
You should see one line with "status": "pending".

Step 4: Handle failures without polluting data

Free endpoints can fail.
A failed A call is not evidence for B.
Do not route the same request to the other variant.

start = time.time()
try:
    response = call_model(prompt)
    entry["status"] = "ok"
    entry["latency_ms"] = int((time.time() - start) * 1000)
    entry["output_hash"] = hashlib.sha256(response.encode()).hexdigest()[:12]
    entry["passed"] = validate(response)
except Exception as exc:
    entry["status"] = "error"
    entry["error"] = str(exc)

log_entry(entry)

The helper functions call_model and validate are yours.
They keep this example readable.

Verify: force a failure.
Use an invalid model name.
The log should show "status": "error".
The request should not appear in the other variant.

Step 5: Compare with a pass rate

Define success before you start.
For code generation, use a test pass rate.
For chat, use a rubric.
The log already has a passed field.

import json
import sys

results = {"A": {"pass": 0, "total": 0}, "B": {"pass": 0, "total": 0}}

for line in sys.stdin:
    entry = json.loads(line)
    if entry["status"] != "ok":
        continue
    variant = entry["variant"]
    results[variant]["total"] += 1
    if entry["passed"]:
        results[variant]["pass"] += 1

for variant, counts in results.items():
    total = counts["total"]
    if total:
        rate = counts["pass"] / total
        print(f"{variant}: {rate:.2f} ({counts['pass']}/{total})")

Verify: feed the log into the script.
cat experiment.jsonl | python compare.py
You should see a rate for each variant.

Step 6: Deploy the logger on a free server

A free server kills background processes.
Run the logger as a service.
Use a process manager.
This tutorial assumes you already solved that problem.
If not, fix process survival first.

The server needs three things.
A place to store the log.
A way to restart the worker.
A route to call the model endpoint.

Verify: restart the server.
Send a test request.
Check that the log file still grows.

Step 7: Stop at a fixed sample size

Pick a sample size before you start.
For a smoke test, 100 calls per variant is a floor.
Do not peek every hour.
Do not stop early because one variant looks better.

This is not a clinical trial.
It is a directional signal.
Use it to decide which prompt to keep.

Limitations

Hash split is deterministic, not random.
It is reproducible, which is useful.
But it does not give you confidence intervals.

Prompt changes interact with system prompts and model versions.
You are testing one slice, not the whole space.

Who should not use this workflow?
Teams with strict statistical requirements.
Teams with fewer than 50 calls per day.
Teams testing safety-critical outputs.

The point

A prompt edit deserves a measurement.
A log file and a hash function are enough.
Run the experiment.
Keep the log.
Your next prompt change will have evidence.