Free Models Drift. Lock Their Behavior With This 30-Line Gate.

Last Tuesday, a review agent changed its voice. The endpoint, the prompt, and the model tier stayed the same, but a field that used to be a string became a nested object. The team noticed only because a smoke test broke; nobody had been watching the model itself. The incident started before the alert fired.

This is a composite scenario, not a reported incident.

A free model in production is a moving target. Providers can retrain, swap, or throttle models without changing the API contract. This is the hidden dependency in many 2026 AI workflows: the interface stays put while the behavior underneath drifts.

This article shows how to install a 30-line behavior lock that catches shape changes before they reach your product. The routine uses versioned prompts, a small eval set, and a reproducible gate script. It is deliberately vendor-neutral and aimed at teams standardizing on free model access.

MonkeyCode is one of the tools making that decision low-friction because its free models and free server remove the usual sign-up cost for trying AI inside a dev workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script below does not depend on MonkeyCode; point it at any OpenAI-compatible chat completion endpoint you can reach.

Why a Behavior Lock Beats Another Test Suite

Unit tests check your code against a fixed expectation. An LLM endpoint checks nothing, because the model can change between two identical requests. A prompt file is not a contract; it is a hypothesis about how the current weights interpret a string.

Free model tiers amplify this problem because the vendor rarely commits to output stability. The model you evaluate today may be a different model tomorrow. That is fine until your tooling expects summary to be a string and the new model returns an object. A behavior lock verifies the contract on a schedule instead of assuming it. It cannot stop drift, but it makes drift visible in minutes rather than days.

The Five Components of a Minimal Behavior Lock

1. Versioned prompt files in your repository, one JSONL file per prompt template generation.
2. An eval set of 10 to 50 real inputs with a required_fields map and expected types.
3. A gate script that calls the endpoint, parses the response, and validates the schema.
4. A pass-rate threshold such as MIN_PASS_RATE=0.95.
5. A human review drill with an owner, a failure record, and a re-run date.

This is not a benchmark. It is an alarm.

A 30-Line Gate You Can Run This Week

Save the file as behavior_lock.py. It reads a JSONL file where each line contains an id, a prompt, and a map of required top-level fields and their expected JSON types.

#!/usr/bin/env python3
import argparse
import json
import os
import sys
import urllib.request

def call_endpoint(url, api_key, model, prompt):
    body = json.dumps({
        'model': model,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0
    }).encode('utf-8')
    req = urllib.request.Request(url, data=body, headers={
        'Authorization': 'Bearer ' + api_key,
        'Content-Type': 'application/json'
    })
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read().decode('utf-8'))

def check_shape(sample, content):
    try:
        payload = json.loads(content)
    except json.JSONDecodeError:
        return False, 'not-json'
    for field, expected in sample['required_fields'].items():
        if field not in payload:
            return False, 'missing-' + field
        if expected == 'string' and not isinstance(payload[field], str):
            return False, 'wrong-type-' + field
        if expected == 'object' and not isinstance(payload[field], dict):
            return False, 'wrong-type-' + field
    return True, 'ok'

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--endpoint', required=True)
    parser.add_argument('--key', default=os.environ.get('AI_API_KEY'))
    parser.add_argument('--model', required=True)
    parser.add_argument('--samples', required=True)
    args = parser.parse_args()

    passed = 0
    failed = 0
    failures = []
    with open(args.samples, encoding='utf-8') as fh:
        for line in fh:
            line = line.strip()
            if not line:
                continue
            sample = json.loads(line)
            response = call_endpoint(args.endpoint, args.key, args.model, sample['prompt'])
            content = response['choices'][0]['message']['content']
            ok, reason = check_shape(sample, content)
            if ok:
                passed += 1
            else:
                failed += 1
                failures.append({'id': sample['id'], 'reason': reason})

    total = passed + failed
    rate = passed / total if total else 0.0
    print('pass_rate=' + format(rate, '.3f'))
    print('passed=' + str(passed))
    print('failed=' + str(failed))
    for item in failures:
        print('failed_id=' + item['id'] + ' reason=' + item['reason'])

    minimum = float(os.environ.get('MIN_PASS_RATE', '0.95'))
    if rate < minimum:
        sys.exit(1)

if __name__ == '__main__':
    main()

An example eval_set.jsonl line looks like this:

{"id": "review-01", "prompt": "Summarize this PR diff. Return valid JSON. Use a string field named summary and a string field named risk.", "required_fields": {"summary": "string", "risk": "string"}}

Run it with:

AI_API_KEY=your_key python behavior_lock.py \
  --endpoint https://your-openai-compatible-endpoint/v1/chat/completions \
  --model your-model \
  --samples eval_set.jsonl

Point the endpoint at the free server you are evaluating, and let the minimum pass rate default to 0.95. That threshold is a starting point, not a law. Teams handling regulated data often set it to 0.99, while exploratory work can tolerate 0.90.

Read the Signals, Not the Slogans

The gate produces numbers, but the action depends on the pattern. Use this table as a review script when the gate fails.

Signal	Likely reading	Gate action
Pass rate drops below 0.95	The model changed its output grammar	Freeze the prompt, replay the baseline, and manually review 50 outputs
A required field disappears in more than 10% of samples	Schema drift	Add stricter format instructions and pin to a known model if you can
Pass rate recovers on a re-run	Transient behavior, not a regression	Log the event for trend review and keep the current model
p95 latency triples for the same workload	Throttling or load on the free server	Run a two-day A/B test against a paid endpoint before migrating
Provider release note appears in changelog	Planned model replacement	Re-run the gate before the next sprint and decide whether the new shape is acceptable

Each row is a decision, not a report. The gate tells you that something moved; the table tells you whether to act on it.

Limitations and Who Should Skip This Lock

This gate only detects structural drift. It will not catch a model that becomes more verbose, less safe, or more creative. For prose-heavy outputs, you need an embedding fingerprint or a judge model instead of a field check. It also assumes your eval set is representative; if it contains only happy-path inputs, the pass rate will look better than reality.

Teams doing one-off prototypes do not need this. If your prompt runs in a notebook and no product depends on it, a behavior lock is overhead. Small teams running fewer than a few thousand tokens per week can start with a manual spot-check instead. The gate earns its keep once the model output feeds an automated workflow.

Run It Before You Standardize

Free models are an option, not a relationship. The real cost of adoption is the time needed to detect drift before it reaches users. The 30-line behavior lock makes that cost visible: it tells you when the model changes shape between two weeks of identical prompts.

Start with ten samples, run the gate, record the pass rate, and re-run next week. You do not need a perfect benchmark; you need an alarm that works before the incident does.