Weekend Build Log: 20 Prompt Tests Before Shipping an AI Feature

Friday night. I had a two-evening plan. My side project needs one AI feature: a small summarizer for server logs. Nothing impressive. The plan looked simple. Call a model. Show three bullet points. Then I remembered last month's failure. My last AI feature broke in production. The prompt was the cause both times.

This week, the DEV feed argued about harness scores versus model scores. One post asked which one you actually benchmarked. Fair question. My answer was honest: neither. I barely tested my prompts at all.

So I cut the scope hard. No summarizer UI. No database. No auth. Instead, I built a tiny prompt regression harness. Twenty test cases. One file. That was the weekend project.

The scope

The rule was brutal. If a feature did not help test prompts, it was out.

Built	Skipped
20 prompt test cases	Database
Pass/fail runner	Dashboard
Minimal server endpoint	Auth
One model	Multi-model UI

Each cut kept the project shippable in two evenings.

How the harness works

Step 1. Write one test case per expected behavior.
Step 2. Call the model with a fixed system prompt.
Step 3. Check the reply for an expected substring.
Step 4. Print PASS or FAIL.
Step 5. Change a prompt. Rerun. Compare.

No vector store. No LLM-as-judge. No golden dataset. A tripwire, not a lab.

Choosing the 20 cases

Pick cases that fail loudly. Prefer exact formats over vague instructions. My set covered four buckets:

• Exact values: version numbers, names, dates.
• JSON shape: required keys appear in the reply.
• Banned output: words like "sorry" or "I can't".
• Length caps: summaries under 60 words.

If a case cannot fail, it is not a test. Delete it.

The code

# prompt_guard.py - minimal prompt regression harness
import httpx

API_URL = "YOUR_ENDPOINT/v1/chat/completions"
API_KEY = "YOUR_KEY"

CASES = [
    {
        "name": "version extraction",
        "system": "Return the version number only.",
        "user": "Fixed retry loop in release 2.4.1.",
        "expect": "2.4.1",
    },
    {
        "name": "json output",
        "system": "Reply with JSON only.",
        "user": "Summarize: connection drops every 10 minutes.",
        "expect": '"summary"',
    },
]

def run_case(case, model):
    payload = {
        "model": model,
        "messages": [
            {"role": "system", "content": case["system"]},
            {"role": "user", "content": case["user"]},
        ],
        "temperature": 0,
    }
    response = httpx.post(
        API_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json=payload,
        timeout=30,
    )
    response.raise_for_status()
    text = response.json()["choices"][0]["message"]["content"]
    return case["expect"].lower() in text.lower(), text

def main():
    model = "your-free-model-name"
    passed = 0
    for case in CASES:
        ok, _ = run_case(case, model)
        passed += ok
        print(("PASS" if ok else "FAIL"), case["name"])
    print(f"{passed}/{len(CASES)} passed")

if __name__ == "__main__":
    main()

Fill in the endpoint, key, and model name from your provider. Most providers ship an OpenAI-compatible chat endpoint. If yours does not, replace httpx with the provider's own client.

Where the free tier changes things

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I ran this suite on MonkeyCode's free model access. The free tier's token allowance was 10 million tokens at the time of writing, plus a free server option for small deployments. Check the current limits before you plan around them. Do not build a business on it. Treat it as a trial budget.

My harness burns about 2,000 tokens per run. That is roughly 5,000 full runs before the allowance matters. For a weekend project, that is plenty.

The free server mattered more than I expected. I added a minimal FastAPI wrapper. Now I can curl the harness from anywhere. No laptop required.

# server.py - minimal on purpose
from fastapi import FastAPI
from pydantic import BaseModel

from prompt_guard import CASES, run_case

app = FastAPI()

class RunRequest(BaseModel):
    model: str = "your-free-model-name"

@app.post("/run")
def run(req: RunRequest):
    results = [run_case(case, req.model) for case in CASES]
    passed = sum(1 for ok, _ in results if ok)
    return {"passed": passed, "total": len(CASES)}

curl -X POST https://your-free-server-url/run

The wrapper is illustrative. Deploy it with whatever tool your free server provides. Keep it stupid.

What the harness caught

Test 14 was the interesting one. The prompt said "Reply with JSON only." Temperature was zero. The output passed.

Then I raised temperature to 0.2. Test 14 failed. The model added punctuation before the JSON. Two characters broke the parser downstream.

A human would miss that in a manual check. The harness caught it in three seconds. That is the point. Not model quality. Regression detection.

The fix was boring. Set temperature back to zero. Pin the prompt in the repo. The test now guards both.

What I skipped on purpose

• Auth. Nobody else uses this yet.
• Database. Results live until the server restarts.
• Dashboard. A plain text table is enough.
• CI integration. That is next weekend.
• Multi-model comparison. The router experiment covered that.

Limitations

Twenty substring checks are not a benchmark. They miss semantic errors. The model can say the right thing in different words. The check still fails. That creates false alarms.

The token math is a rough estimate. Model, prompt length, and provider counting rules all shift it. Measure your own runs.

This approach fits one job: cheap regression detection.
Use it when your prompts are few and your outputs are structured.
Avoid it when you need semantic scoring, RAG evaluation, or enterprise CI gates. Use a real evaluation framework there.

Who should not use this

Teams with production prompts. Teams with hundreds of cases. Anyone comparing model quality across vendors. My harness is a tripwire, not a test lab.

The lesson

The model writes the code. You write the prompts. Nobody tests the prompts. That gap breaks side projects faster than model choice does.

My weekend project is now twenty tests plus a one-file server. The summarizer UI still does not exist. That was the right scope cut.

If you want to run small test suites this weekend, MonkeyCode's free tier is a reasonable place to start. Write five cases first. Run them before every prompt change. Ship the tripwire before the feature.