A Regression Gate for Free Model Endpoints

The cheapest way to trust a free model endpoint is not to ask it more questions; it is to record what it returned when you last accepted its output, then make any silent change fail loudly. The endpoint does not need to be correct all the time. It needs to be predictable enough that you can tell the difference between a surprising answer you want to review and a regressive answer you should never have shipped. That distinction is what the small Python gate in this article creates.

Free model access and a free server option lower the cost of running this gate, because you can afford to exercise the endpoint after every prompt change instead of once per release. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The workflow treats the free access as a black box with unknown quota, latency, and lifespan. Nothing here assumes a particular model name, a rate limit, or a permanent free tier.

The failure mode this gate targets is silent drift. You ask a model to return JSON with three fields. Yesterday it returned status, owner, and priority. Today it returns the same words but priority is a string instead of an integer, or the owner field disappeared because the prompt was reworded. If you only catch that when your downstream parser throws, you have already spent your debugging time on the wrong problem. A regression corpus turns that drift into a diff you can inspect before you update the accepted digest.

The gate has three parts. First, keep a small JSONL file. Each line is one case: an id, a prompt, a list of required keys, a map of expected types, a latency ceiling, and the last accepted digest. Second, an adapter makes the actual HTTP call to whatever free endpoint you have been given. Third, a checker normalizes the response, parses it as JSON, compares the digest, and reports the invariant failures. You do not need a test framework, a vector database, or a paid API. A standard library plus requests is enough.

Here is a case in Python dictionary form so you can see the shape without fighting JSON escaping:

case = {
    'id': 'incident_status',
    'prompt': 'Read this incident note and return JSON with status, owner, and priority. Incident: login page returns 503 after midnight deploy.',
    'invariants': {
        'required_keys': ['status', 'owner', 'priority'],
        'types': {'status': 'str', 'priority': 'int'}
    },
    'max_latency_ms': 8000,
    'last_digest': None,
}

The first run has no last_digest, so it records the digest after you have reviewed the response once. That review is the only part of the workflow that requires human judgment. After that, the gate can run unattended.

This normalization function removes Markdown fences and surrounding whitespace so a formatting change does not look like a data change:

import hashlib

def normalize(text):
    text = text.strip()
    if text.startswith('```

'):
        text = text.strip('`').strip()
        if text.startswith('json'):
            text = text[4:].strip()
    return text.strip()

def digest(text):
    return hashlib.sha256(normalize(text).encode()).hexdigest()
```

The digest is intentionally strict. If the model adds a comma, reorders keys, or wraps the JSON in a different fence style, the digest changes even when the parsed object is still valid. That is not a bug. It is the gate asking you to look at the change, which is exactly the cheap review you want. The invariant checker then separates shape breakage from cosmetic drift:

``{% endraw %}{% raw %}`python
import json, time

TYPE_MAP = {
    'str': str,
    'int': int,
    'float': float,
    'bool': bool,
    'list': list,
    'dict': dict,
}

def check_invariants(raw, case):
    errors = []
    inv = case.get('invariants', {})
    try:
        data = json.loads(normalize(raw))
    except Exception as exc:
        return ['json parse failed: ' + str(exc)]
    for key in inv.get('required_keys', []):
        if key not in data:
            errors.append('missing key ' + key)
    for key, type_name in inv.get('types', {}).items():
        expected = TYPE_MAP.get(type_name)
        if expected is None:
            continue
        if key in data and not isinstance(data[key], expected):
            errors.append('key ' + key + ' should be ' + type_name + ', got ' + type(data[key]).__name__)
    return errors

def run_case(case, respond):
    start = time.perf_counter()
    raw = respond(case['prompt'])
    elapsed_ms = (time.perf_counter() - start) * 1000
    errors = []
    if elapsed_ms > case.get('max_latency_ms', 10000):
        errors.append('latency ' + str(round(elapsed_ms)) + 'ms')
    if case.get('last_digest') and digest(raw) != case['last_digest']:
        errors.append('digest changed')
    return {
        'id': case['id'],
        'elapsed_ms': elapsed_ms,
        'digest': digest(raw),
        'errors': errors,
        'invariant_errors': check_invariants(raw, case),
    }
```

The adapter is deliberately boring. It reads an endpoint from an environment variable, posts the prompt, and returns whichever field the API happens to put the text in. You should write the adapter once and keep it small, because every compatibility hack for a particular provider belongs there and not in the checker:

``{% endraw %}{% raw %}`python
import os, requests

def respond(prompt):
    url = os.environ.get('MODEL_ENDPOINT')
    if not url:
        raise RuntimeError('set MODEL_ENDPOINT before running the gate')
    response = requests.post(url, json={'prompt': prompt}, timeout=10)
    response.raise_for_status()
    payload = response.json()
    return payload.get('text') or payload.get('response') or response.text
```

A full corpus run is just a loop over the JSONL file. You can print a one-line report per case and stop only when a previously accepted digest changes or an invariant breaks. Over time, the corpus becomes a compact history of the failures you have already survived, which is far more useful than a large set of generic prompts.

The free server option earns its place when the model output is not a response to store but a process to run. Suppose the prompt asks for a small HTTP service, and the reply contains the code. Reading that code is not enough, because the failure may live in the port binding, the route registration, or the way the process exits. A local probe starts the command, waits for the port, fetches a path, and then terminates the process:

``{% endraw %}{% raw %}`python
import socket, subprocess, time, urllib.request

def probe_local_server(command, port, path, timeout=10):
    proc = subprocess.Popen(command, shell=True)
    try:
        deadline = time.time() + timeout
        while time.time() < deadline:
            try:
                with socket.create_connection(('127.0.0.1', port), timeout=0.5):
                    break
            except OSError:
                time.sleep(0.2)
        else:
            raise RuntimeError('server did not start')
        with urllib.request.urlopen('http://127.0.0.1:' + str(port) + path, timeout=5) as response:
            return response.status, response.read().decode()
    finally:
        proc.terminate()
```

If your free server option gives you a remote URL instead of a local process, you can swap the local probe for a remote request with the same status and body comparison. The gate does not care where the service lives; it only cares that the route you expect answers with the status and shape you are willing to accept. That separation keeps the server disposable and the test reusable.

There are real limitations. The invariant checker only catches shape drift, not semantic drift. A model can return status, owner, and priority with the correct types and still assign the incident to the wrong person. A digest change is also not always bad; transient wording changes will appear even when the parsed object is identical. For that reason, do not treat a single run as a verdict. Replay the same corpus a few times, collect the failures, and only update the accepted digest after you have looked at the actual diff. Do not put secrets, customer data, or infrastructure details in the prompts, because a free endpoint is a shared service and you should assume the input may be logged. And do not use this gate as a production safety mechanism. It is a pre-merge or pre-promotion sanity check, not a replacement for validation by the system that will actually serve users.

Who should skip this approach? If you are already paying for a model with stable structured-output guarantees and you have a schema registry, this gate adds little. If your free tier is only a temporary trial, building a corpus around it is still useful because the corpus is portable; the prompts and invariants move to any endpoint. The main reason to avoid it is if you cannot review the first accepted response. Without that one human review, the gate would bless the first hallucination as ground truth and then faithfully detect every change away from a wrong baseline.

Start with three failures from your last incident review. Convert each one into a prompt, a required shape, and a saved digest only after the corrected response looks right. Run the gate before you swap endpoints, before you rewrite the prompt, and again after you deploy a generated service. The free access makes those repetitions cheap; the gate makes them mean something.