Dakota Huang

Posted on Aug 14

A Contract Probe for Free Model Endpoints That Drift

#ai #testing #api #devops

A model endpoint is stable only if you keep checking the JSON contract, not the output's tone.

The problem

A one-line prompt change or a quiet model update can turn confidence: 0.83 into confidence: high. The model response still sounds fine, so nobody looks at it. The downstream parser is the first component to fail, often in a nightly job with no human nearby.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This workflow treats MonkeyCode's free model access as a way to generate candidate response shapes and its free server option as a cheap place to keep the probe running. The same technique applies to any HTTP JSON model endpoint.

What to monitor

The probe does not monitor quality. It monitors the contract. A response must have:

reply: non-empty string
confidence: number between 0 and 1
tags: array of strings, at most 4 items

If the model starts returning Markdown fenced JSON, a nested data object, or a missing field, the probe fails closed. That is the only signal that matters here.

Probe code

This uses only the Python standard library. Replace MODEL_URL and the payload shape with your provider's actual API.

#!/usr/bin/env python3
'''Fail closed when the endpoint returns a different JSON shape.'''
import json, os, sys
from urllib import request, error

URL = os.environ.get('MODEL_URL', '')
TOKEN = os.environ.get('MODEL_TOKEN', '')
MAX_TAGS = int(os.environ.get('MAX_TAGS', '4'))

def validate(obj):
    errors = []
    if not isinstance(obj, dict):
        return ['root must be an object']

    reply = obj.get('reply')
    if not isinstance(reply, str) or not reply.strip():
        errors.append('reply: non-empty string required')

    confidence = obj.get('confidence')
    if isinstance(confidence, bool) or not isinstance(confidence, (int, float)):
        errors.append('confidence: number in 0..1 required')
    elif not 0 <= confidence <= 1:
        errors.append('confidence: out of range')

    tags = obj.get('tags')
    if not isinstance(tags, list) or not all(isinstance(t, str) for t in tags):
        errors.append('tags: string array required')
    elif len(tags) > MAX_TAGS:
        errors.append(f'tags: max {MAX_TAGS} items required')

    return errors

def main():
    if not URL:
        sys.exit(3)

    payload = json.dumps({
        'prompt': 'Return JSON with reply, confidence, and tags. No markdown fences.',
        'max_tokens': 128,
    }).encode()

    headers = {'Content-Type': 'application/json'}
    if TOKEN:
        headers['Authorization'] = f'Bearer {TOKEN}'

    req = request.Request(URL, data=payload, headers=headers)

    try:
        with request.urlopen(req, timeout=15) as resp:
            data = json.load(resp)
    except error.HTTPError as exc:
        if exc.code in (429, 500, 502, 503, 504):
            sys.exit(2)  # transient: retry on schedule, do not change contract
        sys.exit(3)      # auth or config problem
    except Exception:
        sys.exit(4)      # network or protocol failure: human should look

    errors = validate(data)
    if errors:
        print(json.dumps({'ok': False, 'errors': errors}))
        sys.exit(3)

    print(json.dumps({'ok': True, 'checked': ['reply', 'confidence', 'tags']}))

if __name__ == '__main__':
    main()

Exit-code contract

The exit code is the actual API for a scheduled probe.

0 — shape matched, keep the last known good state
2 — transient HTTP status, retry later without mutating the contract
3 — schema mismatch or bad config, fail closed
4 — network or protocol failure, alert a human

A JSON schema error is not retryable. Retrying a shape mismatch just wastes requests and hides drift.

Two-minute test plan

Run against a static mock that returns {"reply":"ok","confidence":0.9,"tags":["ai"]}.
Expect exit 0.
Remove confidence from the mock.
Expect exit 3.
Set confidence to the string high in the mock.
Expect exit 3.
Return HTTP 429 once.
Expect exit 2, not a contract change.

This separates a bad response shape from a bad network moment.

Run it from the free server

Put the probe in a directory, set MODEL_URL and MODEL_TOKEN, and schedule it.

*/15 * * * * cd /app && ./run_probe.sh >> probe.log 2>&1

run_probe.sh can respect the exit code:

#!/usr/bin/env sh
./contract_probe.py
code=$?
if [ "$code" = "2" ]; then
  sleep 20
  ./contract_probe.py
fi
if [ "$code" = "4" ]; then
  exit 4
fi

The free server option is useful as a watcher, not as a database. The probe is stateless: it prints pass/fail and error reasons, and the log or CI job stores the history.

Limitations

The contract probe is narrow on purpose.

It checks JSON shape, not truth. A valid response can still be wrong.
It does not catch prompt injection, personal data leaks, or semantic drift.
A free endpoint may have cold starts, rate limits, quota changes, or latency. Do not treat it as an SLA.
It only watches fields you decide to declare. New failure modes need an updated contract.
It should not be the only check before execution.

Who should skip this

Skip this pattern if the model output directly drives a payment, medical decision, safety control, or any action where a valid JSON shape is not enough.

Also skip it if you need guaranteed uptime, cannot risk a failed call, or cannot send the input to a free endpoint under your data policy.

What to change

Fork the probe for your own fields. Keep the contract small at first, then add one field only when that field has caused a real failure. The smaller the contract, the easier the drift is to read.

DEV Community