Valid JSON Is Not Conformant JSON: Five Free-Model Myths and a Pass-Rate Probe

A worker dies at 2 a.m. The payload parses fine. Nobody believes the alert at first.

That gap — parses versus conforms — is where a lot of free-model pipelines quietly break. I keep seeing the same five claims in review threads. Each one is testable in under an hour.

Here is the probe I reach for, and the mental model I would swap in.

Myth 1: "If json.loads accepts it, the model followed my schema."

No. You have two gates. Most teams install only the first one.

• Gate A — parse. Is the text a JSON value at all?
• Gate B — contract. Does that value match your schema: required keys, types, enums, patterns, no extras?

A response can pass A and fail B in a dozen ways. {"ticket_id": 4411} parses cleanly. It breaks a ^TCK-\d{4,}$ pattern. An extra "notes" key parses too. It breaks additionalProperties: false.

Valid JSON is syntax. Conformance is a contract. Only one of them is your actual problem.

Myth 2: "It returned one clean object, so the model is reliable."

What would you do with a test suite that ran once and passed? You would call it a demo, not a suite.

One call gives you n=1. That is an anecdote with curly braces. What you need is a pass rate and an interval around it. Twenty trials let you put a defensible lower bound on conformance. One trial lets you put a story in standup.

So ask the uncomfortable question: what is the lower bound, not the average?

Myth 3: "Temperature 0 means I get the same answer twice."

Test that on your own endpoint before you build on it.

Even at temperature 0, output can drift between runs. Floating-point summation order, request batching, and silent backend version changes all move tokens around. None of that lives in your prompt.

Do not argue about it in a design doc. Send the same prompt twice, diff the raw strings, write down what you saw. "Deterministic enough to be a cache key" is a measurement, not a setting.

Myth 4: "A repair loop fixes malformed output for free."

A repair loop is not free. It costs a second call, extra latency, and a second chance to fail differently.

Measure the marginal pass rate instead. If attempt one conforms 70% of the time, how much does attempt two add? If the answer is a few points, you bought latency for noise.

Set a stopping rule before you ship:

1. Attempt one, strict schema.
2. On failure, one repair call that includes the validation error verbatim.
3. On second failure, escalate. Do not loop — route to a human or a queue.

Three attempts is usually a bug wearing a retry policy as a disguise.

Myth 5: "Free access means the experiment costs nothing."

Free access removes the invoice. It does not remove wall-clock time, rate limits, or your own attention.

That is the trap. A probe that burns forty minutes on throttled calls is expensive, free or not. Budget the probe like production work:

• Cap the trials. Twenty is enough for a first read.
• Sleep between calls. Politeness is cheaper than a 429.
• Read rate-limit headers when the endpoint sends them. Guess nothing.

The probe: two gates, twenty trials, one interval

Here is the harness. Stdlib only, Python 3.10+ for the union type. It runs against any chat endpoint you can call.

I run it against the free model access I have through MonkeyCode, on the free server option — the harness needs a machine that stays awake for twenty sequential calls. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# conformance_probe.py — measure conformance instead of trusting a demo
import json
import math
import re
import time

TRIALS = 20
TICKET_RE = re.compile(r"^TCK-\d{4,}$")

PROMPT = (
    "Return one JSON object and nothing else. "
    'Keys: ticket_id (string, "TCK-" plus 4 or more digits), '
    "severity (one of: low, medium, high), "
    "summary (string, at least 10 characters). No extra keys."
)


def call_model(prompt: str) -> str:
    """Return raw assistant text. Wire this to your own client."""
    raise NotImplementedError("fill in your endpoint call")


def extract_json(text: str):
    fenced = re.search(r"```

(?:json)?\s*(\{.*?\})\s*

```", text, re.S)
    candidate = fenced.group(1) if fenced else text
    start, end = candidate.find("{"), candidate.rfind("}")
    if start == -1 or end <= start:
        return None, "no_object"
    try:
        return json.loads(candidate[start:end + 1]), None
    except json.JSONDecodeError as exc:
        return None, f"json_error:{exc.msg}"


def check(obj) -> str | None:
    """Gate B. Return a reason string, or None when conformant."""
    if not isinstance(obj, dict):
        return "not_an_object"
    extra = set(obj) - {"ticket_id", "severity", "summary"}
    if extra:
        return "extra_key:" + ",".join(sorted(extra))
    for key in ("ticket_id", "severity", "summary"):
        if key not in obj:
            return f"missing:{key}"
    if not isinstance(obj["ticket_id"], str) or not TICKET_RE.match(obj["ticket_id"]):
        return "bad:ticket_id"
    if obj["severity"] not in ("low", "medium", "high"):
        return "bad:severity"
    if not isinstance(obj["summary"], str) or len(obj["summary"]) < 10:
        return "bad:summary"
    return None


def wilson(passes: int, n: int, z: float = 1.96):
    if n == 0:
        return 0.0, 1.0
    p = passes / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return max(0.0, centre - half), min(1.0, centre + half)


def run(trials: int = TRIALS):
    parsed = conformant = 0
    reasons: dict[str, int] = {}

    for _ in range(trials):
        raw = call_model(PROMPT)
        obj, err = extract_json(raw)
        if obj is None:
            reasons[err] = reasons.get(err, 0) + 1
        else:
            parsed += 1
            err = check(obj)
            if err is None:
                conformant += 1
            else:
                reasons[err] = reasons.get(err, 0) + 1
        time.sleep(1)  # be a good citizen; free endpoints still have limits

    lo, hi = wilson(conformant, trials)
    return {
        "trials": trials,
        "parse_rate": round(parsed / trials, 3),
        "conformance_rate": round(conformant / trials, 3),
        "lower_95": round(lo, 3),
        "upper_95": round(hi, 3),
        "failures": dict(sorted(reasons.items(), key=lambda kv: -kv[1])),
    }


if __name__ == "__main__":
    print(json.dumps(run(), indent=2))

What each piece actually buys you

• extract_json separates "wrapped in prose" from "not JSON at all." Those are different bugs with different fixes.
• check is Gate B. It is hand-rolled for this schema; swap in jsonschema if you prefer, but then pin the version.
• wilson gives you a lower bound you can defend in a review, without assuming normality on a small sample.
• The failures histogram tells you which myth you actually have. Most teams get one failure shape, repeated.

Reading the numbers

Sort on the lower bound, not the point estimate. Decision table:

Lower bound (95%)	Action
≥ 0.95	Strict schema, no repair loop, ship it
0.80 – 0.95	One repair attempt, log every escalation
0.60 – 0.80	Shrink the schema. Fewer fields, plainer types
< 0.60	Wrong model for this task. Retries will not save it

That last row is the important one. Poor conformance is usually a task-shape problem, not a prompting problem. Adding retries to a badly scoped schema just costs more money for the same failure.

One more reading tip: if parse_rate is high but conformance_rate is low, your prompt is fine and your contract is disliked. If parse_rate is low, stop reading and fix extraction first.

Limitations, and who should skip this

This probe measures one schema, one prompt, one endpoint, one afternoon. It expires. Re-run it when the provider changes anything, or when your schema grows a field.

Skip it entirely if you need byte-identical output across runs, or if the output feeds an audit trail. Free-tier models belong in triage, tagging, and draft paths — not in the write path you will be asked to defend later.

Also skip it if your volume is five calls a day. At that scale, read the output yourself and save the harness for later.

And do not treat twenty trials as a guarantee. It is a first read with an honest error bar, nothing more.

What I would keep

Replace "it worked" with "it conformed 17 of 20 times, lower bound 0.62." That one sentence changes the whole design conversation.

Myths survive because nobody runs the twenty lines that would kill them. So run them.

If you want to try the harness before wiring up billing, MonkeyCode's free model access plus the free server option is one place to start — bring your own schema and your own skepticism.