Free Model Outputs Need a Schema Gate, Not Just a JSON Parse

Free model outputs need a schema gate, not just a JSON parse.

A free model API can return valid JSON with a different shape from one call to the next.

• Parsing only proves the bytes are JSON.
• It does not prove the keys, types, and required fields match your pipeline.
• A missing line field or a severity field that changes from high to 'high' can break a downstream consumer silently.

A small contract gate should run before any model output reaches a config file, a linter, or a shell command.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option make it practical to run repeated non-production calls for this test without paying per token. That does not change the rule: untrusted output still needs a schema contract.

Reproduce the checker

Install one dependency:

pip install jsonschema

Freeze a contract for a code-review output. The example rejects unknown fields, requires four keys, and constrains severity.

import json
from jsonschema import Draft202012Validator

CONTRACT = {
  'type': 'object',
  'required': ['finding', 'severity', 'file', 'line'],
  'additionalProperties': False,
  'properties': {
    'finding': {'type': 'string', 'minLength': 1},
    'severity': {'enum': ['low', 'medium', 'high', 'critical']},
    'file': {'type': 'string', 'minLength': 1},
    'line': {'type': 'integer', 'minimum': 1}
  }
}

def check(payload: dict) -> dict:
    errors = sorted(
        Draft202012Validator(CONTRACT).iter_errors(payload),
        key=lambda e: list(e.path),
    )
    return {
        'ok': not errors,
        'errors': [
            {'path': list(e.path), 'message': e.message}
            for e in errors[:5]
        ],
    }

sample = {'finding': 'unchecked egress', 'severity': 'medium', 'file': 'run.sh', 'line': 12}
print(check(sample))

Run this against every buffered response before you use the output.

Batch checker for free endpoints

This version reads newline-delimited JSON, applies the schema, and retries once without retrying around rate limits.

import json
import time
from pathlib import Path

def run_gate(file_path: str, call_model_same_prompt) -> list:
    results = []
    for raw in Path(file_path).read_text().splitlines():
        if not raw.strip():
            continue
        payload = json.loads(raw)
        ok, errors = check(payload)
        attempts = 0
        while not ok and attempts < 1:
            time.sleep(1)
            payload = call_model_same_prompt()  # pseudocode: replace with your client
            ok, errors = check(payload)
            attempts += 1
        results.append({'ok': ok, 'errors': errors, 'attempts': attempts})
    return results

Replace call_model_same_prompt with your own client. The sleep and single retry are deliberate: pause on 429, do not hammer the endpoint.

Test plan

Use this plan to catch drift without burning quota.

• Freeze 20 identical prompts for one narrow task.
• Run them through the free model endpoint.
• Buffer each full response before parsing, not a stream.
• Check each parsed payload against the schema.
• Record pass rate, field drift, and latency per call.

Record results in a table:

Run	Valid shape	Drifted field	Notes
1	yes	none	baseline
2	no	line missing	drop this response
3	yes	severity became string	rejected by enum

Do not copy the numbers above. They are placeholders. Run your own.

Decision table

Condition	Action
Schema valid, severity low/medium	allow automatic use
Schema valid, severity high/critical	require human review
Schema invalid on first call	retry once with same prompt
Schema invalid after retry	drop response, use deterministic fallback
Same prompt returns three different shapes	stop, pin a newer schema before retrying again

The retry rule is narrow: one retry, same prompt, no silent auto-fix.

What the gate catches

• Missing required fields.
• Wrong JSON type for a field.
• Unknown fields that may hide new behavior.
• Severity values outside the allowed enum.
• Empty finding text that looks valid but is useless.

Limitations

• A schema gate checks shape, not correctness. A validly shaped finding can still be wrong.
• Free endpoints can be slow or rate-limited. The test plan should pause on 429 instead of retrying around the limit.
• Models may drift within a day. A contract that passed yesterday is not proof for today.
• This is not a sandbox. It does not stop a valid-shaped command from doing damage.
• Dry-run every generated command in a restricted environment before it gets more access.

Who should not use this

• Teams that need a certified SLA on model output should not rely on a free endpoint.
• Anyone handling secrets, production credentials, or irreversible actions should not feed those inputs to a free model.
• If you cannot review high-severity findings by a human, do not let model output reach an automated merge path.
• If the expected schema is still changing, finish the schema design before building the gate.

Bottom line

Parsing checks syntax. Schema checks contract. Add a schema gate when free model output is cheap but your downstream trust should not be.

Try the checker against your next non-production model response and log the drift before you wire it into anything important.