DEV Community

niuniu
niuniu

Posted on

Your Prompt Experiments Are Burning API Budget. Build a Test Harness First.

#ai #tutorial #testing #productivity

Most developers I see iterating on prompts do it like this: paste a prompt into a playground, tweak a word, eyeball the output, tweak again. After an hour, you've spent real money on API calls and you still can't answer the only question that matters: did the change actually help?

The problem isn't the model. It's that there's no measurement. You wouldn't ship a code change without a test, but prompts get YOLO'd into production constantly.

This post shows a small, reusable prompt test harness in Python, plus a workflow for running it against free model access so the experimentation phase costs you nothing. You'll end up with a pass/fail report you can diff across prompt versions.

The idea: prompts are code, test them like code

A prompt test has three parts:

  1. A fixed input set — realistic cases, including the annoying edge cases.
  2. An assertion — something cheap and automatic: keyword presence, JSON validity, length bounds, regex match.
  3. A runner — executes every (prompt version × input × assertion) combination and prints a report.

Keep it dumb. Fancy LLM-as-judge scoring can come later; most prompt regressions are caught by boring assertions.

The harness

Save your test cases as JSON so non-engineers can edit them:

// cases.json
[
  {
    "name": "extracts email",
    "input": "Reach me at sara@example.com before Friday.",
    "expect": { "contains": ["sara@example.com"], "max_chars": 80 }
  },
  {
    "name": "no email present",
    "input": "Call the office tomorrow morning.",
    "expect": { "contains": ["none"], "max_chars": 80 }
  },
  {
    "name": "multiple emails",
    "input": "CC a@x.com and b@y.com on the reply.",
    "expect": { "contains": ["a@x.com", "b@y.com"], "max_chars": 120 }
  }
]
Enter fullscreen mode Exit fullscreen mode

The runner. Point it at any OpenAI-compatible endpoint via environment variables, so the same harness works against a paid API, a local model, or a free hosted one:

# harness.py
import json, os, sys, time
from openai import OpenAI

client = OpenAI(
    base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
    api_key=os.environ.get("LLM_API_KEY", "unused"),
)
MODEL = os.environ.get("LLM_MODEL", "gpt-4o-mini")

PROMPT_TEMPLATE = open(sys.argv[1]).read()  # prompt file passed as arg
cases = json.load(open("cases.json"))

def check(text, expect):
    fails = []
    for kw in expect.get("contains", []):
        if kw.lower() not in text.lower():
            fails.append(f"missing '{kw}'")
    if len(text) > expect.get("max_chars", 10**9):
        fails.append(f"too long ({len(text)} chars)")
    return fails

results = []
for c in cases:
    start = time.time()
    resp = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": PROMPT_TEMPLATE},
            {"role": "user", "content": c["input"]},
        ],
        temperature=0,
    )
    out = resp.choices[0].message.content
    fails = check(out, c["expect"])
    results.append((c["name"], not fails, fails, round(time.time() - start, 2)))

passed = sum(1 for r in results if r[1])
print(f"\n{passed}/{len(results)} passed against {MODEL}\n")
for name, ok, fails, secs in results:
    status = "PASS" if ok else "FAIL"
    print(f"[{status}] {name} ({secs}s)" + (f" -> {', '.join(fails)}" if fails else ""))
Enter fullscreen mode Exit fullscreen mode

Usage:

pip install openai
LLM_BASE_URL="https://your-endpoint/v1" \
LLM_API_KEY="your-key" \
LLM_MODEL="some-model" \
python harness.py prompt_v3.txt
Enter fullscreen mode Exit fullscreen mode

Now iterating on a prompt looks like: edit prompt_v3.txt, rerun, read the report. When v4 regresses the "no email present" case, you know which case broke, not just a vague feeling that the outputs got worse.

Where free compute fits in

The expensive part of prompt work is volume: a dozen test cases × dozens of prompt revisions × a few candidate models adds up fast on a paid API. The cheap workaround is doing this whole iteration loop against free model access, then running the harness once against your production model at the end as a final gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One option for that free tier is MonkeyCode, which offers free model access plus a free server option you can use as the runner environment — handy if you don't want the harness running on your laptop or burning CI minutes. Because the harness above is endpoint-agnostic, you set LLM_BASE_URL and LLM_MODEL to whatever is available there and the workflow doesn't change. If you're setting up a similar loop, it's a reasonable place to run the free iteration phase.

A decision table for when to trust free-tier results

Question Free model is fine Use your production model
Does this prompt version parse correctly? ✅
Did a revision regress an edge case? ✅ (relative signal)
Absolute quality bar for launch? ✅
Latency / cost estimation? ✅
Tone/nuance comparison between frontier models? ✅

The principle: use free models for differential questions ("did v4 beat v3?"), and paid/production models for absolute questions ("is this good enough to ship?").

Limitations, honestly

  • Model drift between tiers. A prompt tuned on a free model may behave differently on your production model. Always rerun the full suite on the real model before shipping — this is the gate, not a formality.
  • Assertions are shallow. contains checks won't catch subtle quality regressions. For anything user-facing, add a human review pass for the top few failures.
  • Free tiers change. Availability, rate limits, and model selection on any free offering can shift without notice. Don't build your CI pipeline to depend on one; treat it as a convenience layer.
  • Concurrency matters. If you scale this harness up, free endpoints will usually throttle you first. Keep the test suite small and sequential.

Who should skip this

If you run prompts a handful of times a week, a playground is genuinely fine — the harness is overhead. It pays off when you have multiple people editing prompts, prompts embedded in a product, or a regression you need to catch before users do.

The broader point isn't about any specific provider. It's that prompt iteration without measurement is just expensive guessing. Write the boring test harness once, run it against whatever free compute you can get, and save the paid calls for the decisions that actually need them.

Top comments (0)