DEV Community

Dakota Huang
Dakota Huang

Posted on

Prompts Are Code: A Golden Test Gate for Free Model Endpoints

#ai #tutorial #testing #python

A prompt is a patch to your codebase. Treat it like one. This tutorial builds a golden test gate for free model endpoints. The gate runs generated code against real tests. It returns a pass or fail verdict. A git hook blocks bad prompts before merge. The whole workflow runs on free resources.

You will build five pieces. A golden test suite. A model runner. A Docker evaluator. An HTTP gate service. A pre-push hook. Each stage has a verification step. If a step fails, you know exactly where to look.

Why a gate, not a vibe check

Model outputs drift. The same prompt can pass today and fail tomorrow. Free endpoints change behavior without notice. A golden test gate catches regressions at the prompt level. It does not judge quality by feel. It runs tests and reads the exit code.

This workflow uses MonkeyCode's free model access and free server option. Both claims come from the operator. Verify current availability before you rely on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What you need

  • Python 3.11 or newer
  • Docker on the machine that runs tests
  • git, curl, jq
  • A free model endpoint URL
  • A server for the gate service

Check your tools first.

python3 --version
docker run --rm hello-world
git --version && curl --version && jq --version
Enter fullscreen mode Exit fullscreen mode

All commands must succeed. If Docker is missing, install it before continuing.

Stage 1: Write the golden tests

Create the project structure.

mkdir prompt-gate && cd prompt-gate
git init
mkdir -p golden_tests prompts scripts reference
Enter fullscreen mode Exit fullscreen mode

The golden test defines what correct means. Use a small, testable function. A duration parser is a good choice. It has edge cases and a clear contract.

# golden_tests/test_duration.py
from duration import parse_duration

def test_hours():
    assert parse_duration("2h") == 7200

def test_minutes():
    assert parse_duration("30m") == 1800

def test_combined():
    assert parse_duration("1h30m") == 5400

def test_invalid():
    try:
        parse_duration("bogus")
        assert False, "should raise"
    except ValueError:
        pass
Enter fullscreen mode Exit fullscreen mode

Write a reference implementation. This proves the tests are correct before any model is involved.

# reference/duration.py
import re

def parse_duration(s: str) -> int:
    m = re.fullmatch(r"(?:(\d+)h)?(?:(\d+)m)?", s.strip())
    if not m or not (m.group(1) or m.group(2)):
        raise ValueError(f"invalid duration: {s!r}")
    hours = int(m.group(1) or 0)
    minutes = int(m.group(2) or 0)
    return hours * 3600 + minutes * 60
Enter fullscreen mode Exit fullscreen mode

Verify this stage.

cp reference/duration.py .
pip install pytest -q
pytest golden_tests -q
Enter fullscreen mode Exit fullscreen mode

You must see 4 passed. If not, fix the tests before moving on.

Stage 2: Write the model runner

The runner sends a prompt to the model endpoint. It extracts code from the response. It writes the code to duration.py.

# scripts/run_gate.py
#!/usr/bin/env python3
"""Call a free model endpoint and extract Python code."""
import json
import os
import re
import sys
import urllib.request

ENDPOINT = os.environ.get("MODEL_ENDPOINT", "")
MODEL = os.environ.get("MODEL_NAME", "free-model")

def build_prompt(task_file: str) -> str:
    task = open(task_file).read()
    return (
        "Write a Python function parse_duration in duration.py. "
        "Only output the code. No explanations.\n\n" + task
    )

def call_model(prompt: str) -> str:
    payload = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=payload, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.load(resp)
    return data["choices"][0]["message"]["content"]

def extract_code(text: str) -> str:
    match = re.search(r"```

python\n(.*?)

```", text, re.DOTALL)
    return match.group(1) if match else text

def main() -> int:
    task_file = sys.argv[1]
    prompt = build_prompt(task_file)
    raw = call_model(prompt)
    code = extract_code(raw)
    open("duration.py", "w").write(code)
    print("duration.py written")
    return 0

if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode

Create the task file. This is the prompt you will gate.

# prompts/parse_duration.md

Implement parse_duration(s: str) -> int.
It parses strings like "2h", "30m", "1h30m".
Return the total duration in seconds.
Raise ValueError on invalid input.
Enter fullscreen mode Exit fullscreen mode

Run the runner. Replace the endpoint with your real MonkeyCode endpoint.

export MODEL_ENDPOINT="https://your-endpoint.example/v1/chat/completions"
export MODEL_NAME="free-model"
python3 scripts/run_gate.py prompts/parse_duration.md
Enter fullscreen mode Exit fullscreen mode

The runner assumes an OpenAI-style chat completions response. If your endpoint differs, adjust the parsing in call_model.

Verify the output.

python3 -c "import ast; ast.parse(open('duration.py').read()); print('valid python')"
Enter fullscreen mode Exit fullscreen mode

The file must be valid Python. If the model returned prose, inspect the raw response. Adjust the regex in extract_code if needed.

Stage 3: Write the Docker evaluator

The evaluator runs the golden tests in an isolated container. Isolation matters. Model output is untrusted code. Never execute it on your host.

# scripts/evaluate.py
#!/usr/bin/env python3
"""Run golden tests against generated code in a container."""
import json
import subprocess
import sys

def run_tests(workdir: str) -> dict:
    cmd = [
        "docker", "run", "--rm",
        "-v", f"{workdir}:/work",
        "-w", "/work",
        "python:3.12-slim",
        "sh", "-c", "pip install pytest -q && pytest golden_tests -q",
    ]
    result = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
    return {
        "exit_code": result.returncode,
        "passed": result.returncode == 0,
        "output": (result.stdout + result.stderr)[-2000:],
    }

if __name__ == "__main__":
    print(json.dumps(run_tests(sys.argv[1])))
Enter fullscreen mode Exit fullscreen mode

Verify this stage.

python3 scripts/evaluate.py .
Enter fullscreen mode Exit fullscreen mode

You must see "passed": true. The reference implementation is still in place. The first run pulls python:3.12-slim. That download takes a minute.

Stage 4: Deploy the HTTP gate service

A local script is not a gate. A gate must be callable from CI and from other machines. Deploy a small HTTP service on your free server.

# server.py
#!/usr/bin/env python3
"""HTTP gate: POST a task, get a verdict."""
import json
import shutil
import subprocess
import tempfile
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path

def run_gate(workdir: str) -> dict:
    try:
        subprocess.run(
            ["python3", "scripts/run_gate.py", "prompts/parse_duration.md"],
            cwd=workdir, capture_output=True, timeout=90,
        )
        result = subprocess.run(
            ["python3", "scripts/evaluate.py", workdir],
            capture_output=True, timeout=180,
        )
        return json.loads(result.stdout)
    except Exception as exc:
        return {"passed": False, "exit_code": 1, "output": str(exc)}

class GateHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"status":"ok"}')

    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length))
        with tempfile.TemporaryDirectory() as workdir:
            for name in ("scripts", "golden_tests"):
                shutil.copytree(name, Path(workdir) / name)
            prompts = Path(workdir) / "prompts"
            prompts.mkdir()
            (prompts / "parse_duration.md").write_text(body["task"])
            verdict = run_gate(workdir)
        payload = json.dumps(verdict).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(payload)

if __name__ == "__main__":
    HTTPServer(("0.0.0.0", 8080), GateHandler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Deploy it on the free server. Docker must be available where the evaluator runs. If your free server lacks Docker, run the evaluator elsewhere and point GATE_URL at it.

git clone https://your-git-host/prompt-gate.git
cd prompt-gate
export MODEL_ENDPOINT="https://your-endpoint.example/v1/chat/completions"
export MODEL_NAME="free-model"
nohup python3 server.py > gate.log 2>&1 &
Enter fullscreen mode Exit fullscreen mode

Verify the service.

curl -s http://localhost:8080/ | jq .
curl -s -X POST http://localhost:8080/ \
  -H "Content-Type: application/json" \
  -d "$(jq -Rs '{task: .}' < prompts/parse_duration.md)" | jq .
Enter fullscreen mode Exit fullscreen mode

You must see "passed": true. If the service is down, read gate.log.

Stage 5: Wire the pre-push hook

A gate only helps if it blocks bad prompts. Wire it into git. Create a check script first.

#!/usr/bin/env bash
# scripts/check_prompt.sh
set -euo pipefail
TASK_FILE="${1:?usage: check_prompt.sh <task.md>}"
GATE_URL="${GATE_URL:-http://localhost:8080/}"

payload=$(jq -Rs '{task: .}' < "$TASK_FILE")
verdict=$(curl -s -X POST "$GATE_URL" -H "Content-Type: application/json" -d "$payload")
echo "$verdict" | jq .
jq -e '.passed == true' <<< "$verdict" > /dev/null
Enter fullscreen mode Exit fullscreen mode

Install the hook.

cat > .git/hooks/pre-push <<'EOF'
#!/bin/sh
for f in $(git diff --name-only HEAD~1 HEAD | grep '^prompts/' || true); do
  scripts/check_prompt.sh "$f" || exit 1
done
EOF
chmod +x .git/hooks/pre-push scripts/check_prompt.sh
Enter fullscreen mode Exit fullscreen mode

Now test the failure path. Write a deliberately wrong prompt.

cat > prompts/bad.md <<'EOF'
Implement parse_duration(s: str) -> int.
Return the total duration in minutes, not seconds.
EOF
scripts/check_prompt.sh prompts/bad.md
Enter fullscreen mode Exit fullscreen mode

The verdict must show "passed": false. The script must exit nonzero. If it exits zero, your gate is broken. Fix it before you trust it.

What the gate cannot do

Golden tests only cover what you assert. They cannot catch style issues, security flaws, or hallucinated APIs. The gate detects regressions. It does not prevent them. Free endpoints can change quotas or behavior without notice. The gate reports the change after it happens.

Docker adds overhead. A small free server may time out on long test runs. Keep the golden suite small and fast. The service assumes one task file per project. Extend it if you need multiple prompts. The pre-push hook needs at least two commits. For the first push, run check_prompt.sh manually.

Who should not use this approach? Teams that generate multi-file features in one prompt. Teams that need legal-grade guarantees. Teams without Docker anywhere in the pipeline. Those cases need a full evaluation platform with human review.

The workflow in one line

Prompt as code. Tests as gate. Free server as judge.

The complete project is roughly 150 lines. It runs on free model access and a free server. It turns prompt review from a feeling into a verdict. Try it on your next prompt change. The gate will tell you the truth.

Top comments (0)