Free Model Servers Fail Quietly. Here's How I Catch Bad Outputs.

Your API call returned 200. The JSON parsed. Your code moved on.

Then the config file had the wrong port. The model invented a value. No exception. No timeout. Just a quiet, wrong answer.

That is the scariest failure mode of free model servers. They don't always fail loudly. Sometimes they fail silently.

I wanted to know how often that happens. So I built a probe. It sends prompts, collects responses, and validates them against a schema. You can run it on any chat-completions endpoint.

I ran it against MonkeyCode's free model access, from their free server option. My laptop stayed free.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Experiment

The probe sends 100 prompts. Each prompt asks for a JSON object with three fields: name, port, and enabled. The expected schema is strict.

{
  "type": "object",
  "required": ["name", "port", "enabled"],
  "properties": {
    "name": {"type": "string"},
    "port": {"type": "integer", "minimum": 1, "maximum": 65535},
    "enabled": {"type": "boolean"}
  }
}

The prompt is explicit. It says "Return only JSON. No markdown. No explanation."

The probe checks three things:

• Does the response parse as JSON?
• Does it match the schema?
• Is the content plausible? (For example, is the port in range?)

The Probe Code

Here's the full script. One file. No framework.

import asyncio
import json
import os

import aiohttp
import jsonschema
from jsonschema import Draft202012Validator

ENDPOINT = os.environ["LLM_ENDPOINT"]
API_KEY = os.environ["LLM_API_KEY"]
PROMPT = """Return only JSON with this exact structure:
{"name": "my-service", "port": 8080, "enabled": true}
Do not include markdown, comments, or extra text."""

SCHEMA = {
    "type": "object",
    "required": ["name", "port", "enabled"],
    "properties": {
        "name": {"type": "string"},
        "port": {"type": "integer", "minimum": 1, "maximum": 65535},
        "enabled": {"type": "boolean"},
    },
}

N = 100
TIMEOUT = 30


async def one_call(session, i):
    payload = {
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 200,
        "temperature": 0.2,
    }
    headers = {"Authorization": f"Bearer {API_KEY}"}
    try:
        async with session.post(
            ENDPOINT, json=payload, headers=headers,
            timeout=aiohttp.ClientTimeout(total=TIMEOUT),
        ) as resp:
            body = await resp.json()
            content = body["choices"][0]["message"]["content"]
            return i, True, content
    except Exception as exc:
        return i, False, f"error: {type(exc).__name__}"


async def main():
    results = []
    async with aiohttp.ClientSession() as session:
        tasks = [one_call(session, i) for i in range(N)]
        for coro in asyncio.as_completed(tasks):
            i, ok, content = await coro
            results.append((i, ok, content))

    failures = []
    for i, ok, content in results:
        if not ok:
            failures.append((i, "request_failed", content))
            continue
        try:
            data = json.loads(content)
        except json.JSONDecodeError as exc:
            failures.append((i, "invalid_json", content[:100]))
            continue
        try:
            Draft202012Validator(SCHEMA).validate(data)
        except jsonschema.ValidationError as exc:
            failures.append((i, f"schema: {exc.message}", content[:100]))
            continue
        if not (1 <= data.get("port", 0) <= 65535):
            failures.append((i, "port_out_of_range", content[:100]))
            continue

    print(f"total: {N}, failures: {len(failures)}")
    for i, kind, detail in failures[:10]:
        print(f"  [{i}] {kind}: {detail}")


if __name__ == "__main__":
    asyncio.run(main())

Run it:

export LLM_ENDPOINT="https://your-endpoint/v1/chat/completions"
export LLM_API_KEY="your-key"
python output_probe.py

What You'll Likely See

Run this on any free model server. You'll see failures. Here are the common ones, from my experience and the experience of others.

• Invalid JSON. The model wraps the answer in markdown. Or it adds trailing commas. Or it returns a sentence plus JSON.
• Missing keys. The model returns {"name": "x"} and stops. Or it renames enabled to is_enabled.
• Wrong types. The model returns "port": "8080" as a string. Or "enabled": "yes" instead of a boolean.
• Plausible but wrong. The model invents a port like 65536 or -1. The schema passes, but the value is nonsense.

The exact failure rate depends on your prompt, model, and temperature. Low temperature helps. Explicit instructions help. Neither eliminates failures.

The Guard

A schema validator is the first line of defense. But you also need a fallback for when validation fails.

Here's a small OutputGuard class. It validates, then applies a fallback policy.

import json
import jsonschema
from jsonschema import Draft202012Validator


class OutputGuard:
    def __init__(self, schema, fallback=None):
        self.validator = Draft202012Validator(schema)
        self.fallback = fallback or {}

    def parse(self, content):
        try:
            data = json.loads(content)
        except json.JSONDecodeError:
            return self.fallback, False
        try:
            self.validator.validate(data)
        except jsonschema.ValidationError:
            return self.fallback, False
        return data, True

Use it like this:

guard = OutputGuard(SCHEMA, fallback={"name": "unknown", "port": 0, "enabled": False})
data, ok = guard.parse(content)
if not ok:
    # log, alert, or use fallback
    pass

The fallback keeps your pipeline alive. But it also hides the failure. Log every failed parse. Track the rate over time.

Where This Breaks

Schema validation catches structural errors. It does not catch semantic errors.

The model can return valid JSON with a wrong port. It can return a plausible name that doesn't match your service. The schema won't blink.

For semantic checks, you need domain logic. If the port must match a known service, verify it. If the name must be in a list, check the list. The guard is a gate, not a brain.

Who should not use this approach? Teams that need guaranteed correctness. If a wrong config file can crash production, you need a human review, not a fallback.

The Takeaway

Free model servers fail quietly. A 200 with wrong JSON is worse than a 500.

Schema validation is cheap. It catches the most common failure modes. Add it to any pipeline that consumes model output.

Run the probe on your endpoint. See the failure rate for yourself. Then decide if you need a fallback, a human review, or a different model.

That 30-minute probe could save you a very confusing debugging session.