Run Prompt Changes Like Code: A Tiny Regression Suite on Free Model Endpoints

Most prompt changes ship without tests.
You edit one instruction.
It changes 40 edge cases.
You notice after a user complains.

This tutorial builds a small regression suite.
It treats prompt cases like unit tests.
It runs on free model endpoints.

The examples assume MonkeyCode's free model access and free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow has five stages.
Each stage ends with a check.

What the suite catches

A prompt regression suite checks contract, not taste.
It verifies shape, not style.
It catches four common failures.

• JSON output that becomes a sentence.
• A missing field in a tool call.
• An added URL that violates policy.
• A heading or bullet format change.

The suite fails closed.
A broken contract exits non-zero.
That matters for free endpoints.

Stage 1: Define the contract

Create cases.py.
Each case has an id, system prompt, user message, and checks.

CASES = [
    {
        'id': 'extract_invoice',
        'system': 'You are an API assistant. Return only valid JSON.',
        'user': 'Extract amount and due date from: Pay 450.00 by next Friday.',
        'checks': [
            {'type': 'json'},
            {'type': 'json_path', 'path': 'amount', 'text': '450'},
            {'type': 'json_path', 'path': 'due_date', 'text': 'Friday'}
        ]
    },
    {
        'id': 'support_heading',
        'system': 'You are a support agent. Answer with one H2 heading and one bullet list.',
        'user': 'Summarize two outage risks.',
        'checks': [
            {'type': 'contains', 'text': '## '},
            {'type': 'contains', 'text': '- '}
        ]
    },
    {
        'id': 'no_urls',
        'system': 'Do not include URLs in the response.',
        'user': 'Describe the fix for a failed build.',
        'checks': [
            {'type': 'not_contains', 'text': 'http'}
        ]
    }
]

Verification 1:

python3 -m py_compile cases.py && echo 'cases valid'

Stage 2: Build the harness

Create prompt_regression.py.
Install the one dependency.

python3 -m pip install requests

The script reads the endpoint from environment variables.

import argparse, json, os, sys

try:
    import requests
except ImportError:
    sys.exit('pip install requests first')

from cases import CASES

BASE_URL = os.environ['MODEL_BASE_URL']
TOKEN = os.environ['MODEL_TOKEN']
MODEL = os.environ.get('MODEL_NAME', 'free-model')

def complete(system, user):
    resp = requests.post(
        f'{BASE_URL}/chat/completions',
        headers={'Authorization': f'Bearer {TOKEN}'},
        json={
            'model': MODEL,
            'temperature': 0,
            'messages': [
                {'role': 'system', 'content': system},
                {'role': 'user', 'content': user},
            ],
        },
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()['choices'][0]['message']['content']

def check_one(text, check):
    kind = check['type']
    if kind == 'contains':
        result = check['text'] in text
    elif kind == 'not_contains':
        result = check['text'] not in text
    elif kind == 'json':
        try:
            json.loads(text)
            result = True
        except json.JSONDecodeError:
            result = False
    elif kind == 'json_path':
        data = json.loads(text)
        value = data
        for part in check['path'].split('.'):
            if isinstance(value, dict):
                value = value[part]
            else:
                value = value[int(part)]
        result = check['text'] in str(value)
    else:
        raise ValueError(f'unknown check type: {kind}')
    expected = check.get('expect', True)
    return result == expected

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--only', help='comma-separated case ids')
    args = parser.parse_args()

    cases = CASES
    if args.only:
        ids = set(args.only.split(','))
        cases = [c for c in cases if c['id'] in ids]

    report = []
    for case in cases:
        try:
            text = complete(case['system'], case['user'])
        except Exception as exc:
            report.append({'id': case['id'], 'status': 'error', 'detail': str(exc)})
            continue

        failures = []
        for check in case.get('checks', []):
            if not check_one(text, check):
                failures.append(check)

        status = 'pass' if not failures else 'fail'
        report.append({
            'id': case['id'],
            'status': status,
            'failures': failures,
            'output': text[:200],
        })

    print(json.dumps(report, indent=2))
    if any(r['status'] != 'pass' for r in report):
        sys.exit(1)

if __name__ == '__main__':
    main()

Verification 2:

python3 -m py_compile prompt_regression.py && echo 'syntax ok'

Stage 3: Run the baseline

Set your endpoint values.
Use the free endpoint from your provider console.

export MODEL_BASE_URL='https://your-free-endpoint.example'
export MODEL_TOKEN='replace-with-token'
python3 prompt_regression.py > baseline.json
echo $?

A zero exit means every case passed.
Save the report.

Verification 3:

import json
r = json.load(open('baseline.json'))
assert all(c['status'] == 'pass' for c in r), r
print('baseline passed')

Stage 4: Make a prompt change and watch it fail

Edit cases.py.
Change the first system message.

'system': 'You are a friendly assistant. Answer in a polite sentence.'

Run only that case.

python3 prompt_regression.py --only extract_invoice
echo $?

Expected exit code is 1.
The json check fails.
The output now looks like prose.

Verification 4:

import subprocess
p = subprocess.run(
    ['python3', 'prompt_regression.py', '--only', 'extract_invoice'],
    capture_output=True, text=True,
)
assert p.returncode != 0, 'expected failure'
print('drift detected')

Stage 5: Schedule it on free compute

The script needs no GPU.
A small Python runtime is enough.
You can run it on MonkeyCode's free server option as a scheduled job.

Keep the environment variables there.
Run the script daily.

cd /path/to/suite
python3 prompt_regression.py >> report.log 2>&1

Verification 5:

tail -n 20 report.log

What this suite does not do

It does not judge quality.
It checks contract.
It does not fix output.
It only reports drift.

Free endpoints can return HTTP 429.
The script does not retry.
Do not hide rate-limit failures.
Record them as errors.

Temperature zero does not guarantee exact wording.
Use contains and JSON paths.
Avoid exact string equality for natural language.

Do not use this approach when a wrong answer can cause harm.
Do not use it for regulated data.
Do not use it when the model output ships directly to users without review.

Bottom line

Prompt edits are code changes.
Treat them that way.
A ten-minute suite catches regressions before users do.
The same harness works with any free endpoint.

If you already have a free model endpoint, wire the script in before your next prompt edit.