Contract Tests for Model Output: The JSON Safety Net You're Missing

One day the model stops returning the shape you expected. You did not change your code. You did not change your prompt. But your parser starts throwing KeyError in production.

You could patch the parser again. Most teams do. The better move is to write a contract test for the one API that changes without a version bump: the model itself.

You already write contract tests for your REST endpoints. The model deserves the same treatment, especially when you run it on a free tier alongside your regular stack.

The Silent Contract Break

Large language models return text. Your application expects JSON. Between those two facts sits a fragile contract with three unnamed clauses: keys must match, types must match, and the payload must survive transport.

Each clause breaks in its own way. A model starts returning amount as a string. It adds a new field you never asked for. It wraps the JSON in markdown fences because a previous prompt told it to be chatty.

None of these break the API call. All of them break your downstream code.

A Contract Test for Text

A contract test for model output is a small script that does three things:

1. Send a fixed prompt to the model.
2. Parse the raw text response.
3. Validate the result against a schema.

Run it in CI. Run it on a schedule. Treat a failure the way you treat a broken API contract: loudly.

You can run this against any endpoint you can reach. I use MonkeyCode's free model access as a cheap target for exactly this kind of suite. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same test works against any OpenAI-compatible endpoint.

The Schema First

Start with the smallest schema your application actually needs. Not the ideal output. The contract.

For a receipt extractor, that might look like this:

{
  "type": "object",
  "required": ["vendor", "amount", "date", "items"],
  "properties": {
    "vendor": {"type": "string"},
    "amount": {"type": "number"},
    "date": {"type": "string", "pattern": "^\\d{4}-\\d{2}-\\d{2}$"},
    "items": {"type": "array", "items": {"type": "string"}}
  },
  "additionalProperties": false
}

Notice additionalProperties: false. That is the clause most people forget. It catches the day the model starts inventing fields.

The Three-Layer Test

Here is the reusable pytest scaffold. It keeps three concerns separate: calling the model, cleaning the output, and validating the shape.

import json
import os
import urllib.request

import jsonschema
import pytest

SCHEMA = {
    'type': 'object',
    'required': ['vendor', 'amount', 'date', 'items'],
    'properties': {
        'vendor': {'type': 'string'},
        'amount': {'type': 'number'},
        'date': {'type': 'string', 'pattern': '^\\d{4}-\\d{2}-\\d{2}$'},
        'items': {'type': 'array', 'items': {'type': 'string'}},
    },
    'additionalProperties': False,
}


def call_model(prompt):
    body = json.dumps({
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0,
    }).encode()
    req = urllib.request.Request(
        os.environ['MODEL_URL'],
        data=body,
        headers={
            'Authorization': f"Bearer {os.environ['MODEL_TOKEN']}",
            'Content-Type': 'application/json',
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read())
    return data['choices'][0]['message']['content']


def parse_model_output(raw):
    raw = raw.strip()
    if raw.startswith('```

'):
        raw = raw.strip('`')
        if raw.startswith('json'):
            raw = raw[4:]
    return json.loads(raw)


@pytest.mark.parametrize('fixture', ['receipt-1.txt', 'receipt-2.txt', 'receipt-3.txt'])
def test_model_output_matches_contract(fixture):
    with open(fixture) as f:
        prompt = f.read()
    raw = call_model(prompt)
    parsed = parse_model_output(raw)
    jsonschema.validate(parsed, SCHEMA)
    assert isinstance(parsed['amount'], (int, float))
```

The three fixtures are your baseline receipts. They should be boring, realistic examples, not edge cases. Boring fixtures catch drift; edge cases test cleverness.

## The Cleaning Function Is a Boundary

Note the `parse_model_output` function. It exists because models wrap JSON in prose, markdown fences, or both.

Do not let the cleaning function grow forever. If you find yourself adding a fourth rule, stop and change the prompt instead. The cleaning function should handle known noise, not absorb unknown model behavior.

## Running It on a Schedule

A contract test run once in CI is useful. A contract test run every hour is a different tool.

Put the suite on a timer. I run it on MonkeyCode's free server option when I want a long-lived observer without babysitting my laptop. The free tier of the model access keeps the cost near zero, and the server handles the scheduling.

The goal is a history of responses. When the model changes shape, the failure is a data point, not a mystery.

## When the Contract Fails

Here is the decision table I use when a model output fails validation:

| Failure | Likely cause | Move |
| --- | --- | --- |
| `additionalProperties` error | Model added a new key | Add the key to the schema or tighten the prompt; never let the parser ignore it silently |
| `amount` is a string | Type drift | Fix the prompt first, add a conversion layer only as a temporary guard |
| Date pattern mismatch | Format drift | Constrain the prompt with an example and re-run the suite |
| Markdown fences appear | Prompt leakage or model habit | Handle in cleaning, then log the frequency so it does not become normal |

## What This Test Will Not Catch

The schema validates shape. It does not validate meaning or safety.

The model can return `{'amount': 12.5, 'vendor': 'Not A Vendor', 'date': '2026-08-29', 'items': []}` and pass every assertion. Your business logic still needs its own checks.

This test also cannot detect prompt-injection attempts in the fixtures themselves. Treat this as a stability net, not a security boundary.

## Who Should Skip This

If your model output is a single sentence displayed to a user, skip the schema. A string is already a contract.

If you only call a model once in a script, skip the full suite. A one-off `json.loads` inside a try block is enough.

But if you have more than one parser, more than one prompt template, or a cron job that turns model output into database rows, write the contract test first.

The cost of this suite is tiny. The cost of a silent shape change is a page at 2 a.m.

## The First Step

Pick one model output you depend on. Write one schema, three fixtures, and a cleaning function. That is the whole project.

The model will change without telling you. At least your tests will be paying attention.