Taylor Wang

Posted on Sep 1

Don't Trust the Structure Either: A 48-Hour Contract Checker for Free Model Outputs

#ai #testing #python #devops

You have probably seen plenty of posts about how free model outputs drift in quality. I have written a few of those myself. But there is a quieter kind of drift that bites harder: the response looks fine, then one day the JSON key changes from level to severity and your parser explodes.

Last week I ran a tiny monitoring job on a free server using a free model endpoint. The rate limit was healthy, the server was healthy, and the response was mostly correct. Then my parser hit a KeyError because one object used severity instead of level. The model was not the problem — my unstated assumption that the output shape would stay stable was the problem.

So I built a contract checker: a deterministic validation layer that sits between the model response and everything else in the pipeline. It is boring code, and that is exactly why it works. This is a 48-hour field note about what I tried, what broke, and what I would repeat.

Why a Contract Checker?

If you are calling an LLM to produce JSON, you are integrating with an API that has no formal schema. MonkeyCode provides free model access and a free server option, and I used both to run a cron job every 30 minutes. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The experiment was simple: ask the model to summarise a synthetic error log and return JSON with four fields. Then validate every response against a contract before anything else consumes it.

The Experiment Setup

I wrote a small Python script and put it in cron. The prompt stayed identical for the whole run:

Below is a synthetic error log. Return JSON only: {'timestamp': '...', 'level': 'info|warn|error', 'summary': '...', 'confidence': 0.0-1.0}

The validator checked five things:

The raw text parses as JSON.
All four keys exist.
level is one of info, warn, error.
confidence is a float between 0 and 1.
timestamp can be parsed and is within five minutes of the server clock.

I used jsonschema because it is easy to extend, but plain if statements would work too.

import json, re
from datetime import datetime, timezone
from jsonschema import validate

SCHEMA = {
    'type': 'object',
    'required': ['timestamp', 'level', 'summary', 'confidence'],
    'properties': {
        'timestamp': {'type': 'string', 'format': 'date-time'},
        'level': {'enum': ['info', 'warn', 'error']},
        'summary': {'type': 'string', 'minLength': 1},
        'confidence': {'type': 'number', 'minimum': 0, 'maximum': 1}
    }
}

def parse_and_validate(raw: str) -> dict:
    raw = re.sub(r'^```

(?:json)?|

```$', '', raw.strip()).strip()
    data = json.loads(raw)
    validate(instance=data, schema=SCHEMA)
    ts = datetime.fromisoformat(data['timestamp'].replace('Z', '+00:00'))
    if abs((datetime.now(timezone.utc) - ts).total_seconds()) > 300:
        raise ValueError('timestamp too old')
    return data

The parse_and_validate function is the whole point. Every response must pass through it, and only the parsed dictionary gets passed to the rest of the app.

What the Validator Caught

After 48 hours of half-hourly calls, the failure log had a pattern. I am not publishing exact counts because one week on a shared free endpoint is not a benchmark. But the categories were stable, and each one teaches something.

1. Markdown Fences Append Themselves

Every so often, the model wrapped the JSON in

```json fences. A one-line regex stripped them. Without that, every fenced response would have been a parse error, and I would have blamed the model for a fixable formatter quirk.

2. Keys Get Renamed By Accident

level became severity in one response, and confidence became confidence_score in another. The schema caught both. My parser would have crashed on the first and silently ignored the second.

3. Types Drift Even When Keys Stay

A response returned confidence as the string 'high'. Another returned 95 instead of 0.95. Both are human-readable, but not machine-ready. A dashboard query would have rendered a broken chart without ever throwing an exception.

4. Timestamps Lie About Timezones

A handful of records used a local timezone while the server clock was UTC. The validator rejected them because the parsed time fell outside the five-minute window. Always assume the model will pick whatever convention feels natural in the moment.

The Retry Wrapper That Prevented a Storm

The validation code was only half the work. The retry logic around the model call mattered just as much. A 503 from a shared free endpoint is a normal event; my old approach would turn it into a storm of parallel requests. This time I used capped exponential backoff:


python
import time, random

def call_with_backoff(fn, attempts=3):
    for i in range(attempts):
        try:
            return fn()
        except Exception as exc:
            if i == attempts - 1:
                raise
            time.sleep(2 ** i + random.random())
    raise RuntimeError('unreachable')

Three attempts, no infinite loop. Transport errors get retried, but validation failures are not retried — they are logged, because the model already gave its best answer and another attempt would just burn tokens.

A Small Decision Table for Output Checks

Check	Why It Matters	Example
JSON parses	Your code depends on it	Markdown fences
Required keys	Missing key = `KeyError`	`severity` vs `level`
Value types	Wrong type breaks math	`'high'` instead of `0.8`
Allowed values	Out-of-range corrupts logic	`level: debug` not in enum
Recency	Stale output misleads	10-minute-old timestamp

Do not validate sentiment or wording. That is semantics, and your evaluator would be as subjective as the model. Keep the contract structural and deterministic.

What I Would Repeat and What I Would Change

I would repeat the boundary validator without hesitation. The 30-minute cadence was light enough to stay inside the free tier and long enough to collect useful failures. I would change one thing: add a hash of the prompt to every log row. I only thought of that around hour 40, and then I could not reconstruct which prompt variation caused a particular drift.

Limitations and Who Should Skip This

This is not a benchmark of any specific MonkeyCode model, and I am not naming models or quotas because I did not measure them. The failure modes above are examples, not statistics. If you are building a throwaway prototype where malformed JSON costs nothing, skip the validator and just parse. If you are building anything that feeds a dashboard, a queue, or an alert system, put the contract first.

Structural validation also says nothing about whether the text is true. It only protects your code from the shape of the answer. For content truth, keep a human in the loop.

The Takeaway

The next time a model returns severity where you asked for level, do not patch the parser. Add a contract checker at the boundary, log the failures, and let the model keep doing what it does well: producing text. Your code is the side that has to stay deterministic.

If you run your own 48-hour version, start with the validator above. The first few failures will be small, but each one will teach you something about your pipeline before it reaches a production queue.

DEV Community