I Ran a 90-Call Structured Output Benchmark on a Free Model Server. Here's Where It Breaks.

Latency tells you when a server is slow. It does not tell you if the answer is correct. I spent weeks measuring response times on free model servers. This time I measured something else: output quality.

Can a free model server produce reliable structured output? Real apps need JSON, not prose. I built a small benchmark to find out.

The Question

Most free server tests stop at speed. They measure time-to-first-token, variance, timeouts. I have done all of that. None of it tells you whether the JSON will parse.

My question was narrower. If I ask for a flat object, does it come back valid? If I ask for nested data, does it survive? Where exactly does the free tier break?

The Experiment

Three tasks. Increasing difficulty. Thirty runs each. Ninety calls total.

Task A — flat extraction. Pull customer_name, date, and amount from a receipt text. Five fields of instruction. Nothing nested.

Task B — enum classification. Read a support ticket. Return category from a fixed list and a confidence score. The model must respect the enum.

Task C — nested order. Extract an order with order_id, customer, and an array of line_items. This is where free servers usually fall apart.

I pointed the harness at MonkeyCode's free model endpoint. I ran the client from MonkeyCode's free server option, so the network path resembles a real user's, not my laptop's.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Harness

The script is small. It reads cases from JSON, calls the endpoint, and scores every response. It writes a CSV and prints a summary.

The scoring is strict. Valid JSON is not enough. Every expected key must exist. Every value must match. Partial credit only counts when the schema holds.

# benchmark_free_server.py
import asyncio, csv, json, os, time
from dataclasses import dataclass
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url=os.getenv("LLM_BASE_URL", "https://your-endpoint/v1"),
    api_key=os.getenv("LLM_API_KEY", "demo-key"),
)

@dataclass
class Case:
    task: str
    name: str
    prompt: str
    expected: dict

def load_cases(path: str) -> list[Case]:
    with open(path) as f:
        return [Case(**c) for c in json.load(f)]

def validate(text: str, expected: dict) -> dict:
    score = {"valid_json": False, "schema_ok": False,
             "exact_match": False, "fields_correct": 0.0}
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        return score
    score["valid_json"] = True
    score["schema_ok"] = all(k in data for k in expected)
    if not score["schema_ok"]:
        return score
    correct = sum(1 for k, v in expected.items() if data.get(k) == v)
    score["fields_correct"] = correct / len(expected)
    score["exact_match"] = score["fields_correct"] == 1.0
    return score

async def run_case(case: Case) -> dict:
    t0 = time.perf_counter()
    try:
        resp = await client.chat.completions.create(
            model=os.getenv("LLM_MODEL", "free-model"),
            messages=[
                {"role": "system",
                 "content": "Return only valid JSON. No markdown fences."},
                {"role": "user", "content": case.prompt},
            ],
            temperature=0,  # drop this line if your endpoint rejects it
            max_tokens=500,
        )
        text = resp.choices[0].message.content or ""
        score = validate(text, case.expected)
        return {"task": case.task, "name": case.name,
                "latency_s": round(time.perf_counter() - t0, 2),
                "error": "", **score}
    except Exception as e:
        return {"task": case.task, "name": case.name,
                "latency_s": round(time.perf_counter() - t0, 2),
                "error": type(e).__name__, "valid_json": False,
                "schema_ok": False, "exact_match": False,
                "fields_correct": 0.0}

async def main():
    cases = load_cases("cases.json")
    results = [await run_case(c) for c in cases]
    with open("results.csv", "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=list(results[0].keys()))
        writer.writeheader()
        writer.writerows(results)
    by_task = {}
    for r in results:
        by_task.setdefault(r["task"], []).append(r)
    for task, rs in by_task.items():
        n = len(rs)
        print(f"{task}: n={n} valid={sum(r['valid_json'] for r in rs)} "
              f"exact={sum(r['exact_match'] for r in rs)} "
              f"avg_latency={sum(r['latency_s'] for r in rs)/n:.2f}s")

if __name__ == "__main__":
    asyncio.run(main())

The cases file is plain JSON. One entry per task:

[
  {
    "task": "A",
    "name": "receipt_01",
    "prompt": "Extract customer_name, date, and amount from: 'Cafe Luna, 2026-08-23, latte 4.50, pastry 3.25, total 7.75.'",
    "expected": {"customer_name": "Cafe Luna", "date": "2026-08-23", "amount": "7.75"}
  }
]

Run it with three environment variables:

export LLM_BASE_URL="https://your-endpoint/v1"
export LLM_API_KEY="your-key"
export LLM_MODEL="free-model"
python benchmark_free_server.py

The harness is not MonkeyCode-specific. Point it at any OpenAI-compatible endpoint. The conclusions will change. The method will not.

What I Saw

One afternoon. One endpoint. Ninety calls. Here is the summary.

Task	Valid JSON	Schema OK	Exact match	Avg latency
A: flat extraction	28/30	27/30	25/30	1.8s
B: enum + confidence	27/30	24/30	19/30	2.1s
C: nested order	19/30	11/30	8/30	3.4s

Task A was usable. Task B was close. Task C was a coin flip. The pattern repeated across every batch.

Where It Breaks

Three failure modes dominated. You will see the same ones.

Markdown fences. The model wrapped JSON in

```json even after the system prompt said no. My validator counted these as invalid. A strip step recovered most of them.

Truncation. Task C responses hit the token cap. The JSON stopped mid-array. No closing brace, no recovery, no warning.

Key drift. line_items became items. customer_name became name. The schema check failed even when the data was correct.

Sound familiar? If you parse LLM output in production, you have seen all three.

The Decision Table

Here is the rule I derived. Use it as a starting point, not gospel.

Task shape	Verdict	Why
Flat JSON, ≤5 fields	Use it	High exact-match rate
Enum classification	Use with validation	Values drift
Nested JSON	Add a repair step	High schema failure
Output > 300 tokens	Avoid	Truncation risk

The Repair Loop

Do not trust the first parse. Run a three-step repair loop instead.

1. Parse. If json.loads succeeds, stop.
2. Strip. Remove markdown fences and trailing commas. Retry once.
3. Ask the model to fix it. Send the error message back. One retry, no more.

python
def repair(text: str):
    candidates = [text]
    cleaned = text.strip()
    if cleaned.startswith("

```"):
        cleaned = cleaned.strip("`").removeprefix("json").strip()
        candidates.append(cleaned)
    for c in candidates:
        try:
            return json.loads(c)
        except json.JSONDecodeError:
            continue
    return None

The repair loop rescued most of Task C's fence failures. Valid JSON climbed from 19/30 to 26/30. Schema-OK barely moved. Truncation cannot be repaired.

Limitations

Read the caveats before you copy my conclusions.

This was one session, one endpoint, one afternoon. Free servers change without notice. My numbers are an observation, not a product benchmark. Run the harness yourself.

The prompt matters more than the model. Small wording changes moved exact-match rates by ten points. Your results will differ.

Do not send sensitive data to a free endpoint. Prompts may be logged. No PII, no secrets, no patient records.

Who should skip this approach? Teams with strict schema guarantees. Apps that cannot tolerate a retry. Anyone with a compliance officer in the room.

Final Take

Latency is half the story. Correctness is the other half. A free model server handles flat JSON well. It will fight you on nested structures.

Measure before you trust. The harness is above. Run it, then decide.

If you run it against your own endpoint, I want to see your numbers. Drop the CSV in the comments.