DEV Community

Jordan Huang
Jordan Huang

Posted on

Valid JSON Is Not a Contract: Five Schema Myths

#ai #python #testing #webdev

Have you ever shipped model JSON because parsing succeeded? I have, and a required field quietly vanished. Parsing only proves the bytes were JSON.

The hole under a successful parse

A teammate pastes a schema into chat. The model replies with a tidy object. They drop it straight into a handler.

Then a string lands where an integer belongs. Why do we keep trusting that path? The payload looked professional inside the tab.

Cheap generation makes the habit easier to repeat. You can retry forever on a free model. That extra speed starts to feel like safety.

It still is not a real contract. Contracts live in files you can hash. Chat tabs do not count.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use MonkeyCode free model access for draft payloads. I also use the free server option for isolated retries. Neither one replaces a checked-in schema gate.

How to read this FAQ

Each myth has three parts below. You get the claim, the failure, and a better model. Then you get a gate you can run.

I am not publishing latency charts in this piece. I am not naming models or quotas either. Those claims rot. This workflow does not.

Labeled examples are local gates, not production services. Treat them as fixtures you can copy. Then tighten them for your repo.

Myth 1: If it parses, the contract held

Claim: json.loads returned a dict, so we are done.

What happens: Missing keys still parse without complaint. Wrong types still parse without complaint. Nested nulls still parse without any complaint.

Corrected model: Parsing checks syntax only, never semantics. Your contract is types, required keys, and bounds.

Would you accept any JSON from a stranger webhook? Then why accept it from a generated string?

# labeled example: local schema gate, not a hosted service
import json
import sys
from jsonschema import Draft202012Validator


def load_json(path):
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)


def extract_object(raw: str) -> dict:
    raw = raw.strip()
    if raw.startswith("```

"):
        lines = raw.splitlines()[1:]
        if lines and lines[-1].strip().startswith("

```"):
            lines = lines[:-1]
        raw = "\n".join(lines)
    return json.loads(raw)


def main(schema_path, payload_path):
    schema = load_json(schema_path)
    raw = open(payload_path, encoding="utf-8").read()
    try:
        data = extract_object(raw)
    except json.JSONDecodeError as err:
        print(f"PARSE_FAIL: {err}")
        raise SystemExit(2) from err
    validator = Draft202012Validator(schema)
    errors = sorted(validator.iter_errors(data), key=lambda e: list(e.path))
    if not errors:
        print("SCHEMA_OK")
        raise SystemExit(0)
    for err in errors:
        loc = ".".join(str(p) for p in err.path) or "$"
        print(f"SCHEMA_FAIL {loc}: {err.message}")
    raise SystemExit(1)


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])
Enter fullscreen mode Exit fullscreen mode

Run the gate like this on your laptop.

python gate_schema.py schema.json model_raw.txt
echo $?
Enter fullscreen mode Exit fullscreen mode

Exit 0 means the contract held today. Exit 1 means the schema rejected the object. Exit 2 means the text was not JSON.

No green parse, no discussion about “almost right.” The exit code is the review.

Myth 2: JSON mode enforces your schema

Claim: The vendor JSON toggle is a type system.

What happens: JSON mode mostly nudges braces and quotes. It does not load your required field list.

Did the provider promise Draft 2020-12 validation? Read the current docs before you assume. A mode flag is not JSON Schema.

Corrected model: JSON mode is a decoder hint only. The schema file in git is the law.

Keep the schema next to the prompt notes. Hash it in CI so silent edits fail.

sha256sum schema.json > schema.json.sha256
git add schema.json schema.json.sha256 prompts/order_payload.md
Enter fullscreen mode Exit fullscreen mode

If the hash changes without a review, stop the pipeline. Prompt text can drift. The schema should not drift quietly.

Myth 3: Extra fields are harmless leftovers

Claim: Additional properties will be ignored, so nobody cares.

What happens: Extra keys leak into logs and caches. Sometimes they overwrite columns in an ORM. Have you grepped for additionalProperties this month?

Corrected model: Unknown fields are data you did not request. Treat them as defects until the schema allows them.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["sku", "qty", "currency"],
  "properties": {
    "sku": { "type": "string", "minLength": 1, "maxLength": 64 },
    "qty": { "type": "integer", "minimum": 1 },
    "currency": { "type": "string", "pattern": "^[A-Z]{3}$" }
  }
}
Enter fullscreen mode Exit fullscreen mode

Would you let a client send "qty": "1"? Then do not let a model do it.

Inspect a saved payload with jq before feelings get involved.

jq 'keys' model_object.json
jq -e '.qty | type == "number"' model_object.json
Enter fullscreen mode Exit fullscreen mode

Keys you did not ask for are not Easter eggs. They are unreviewed input.

Myth 4: Pasting OpenAPI teaches the model your API

Claim: The prompt contains the spec, so objects will match.

What happens: Long specs get truncated in the context window. Nearby examples overpower the written constraints. Enums drift across retries without anyone noticing.

Corrected model: The prompt is only a suggestion. The validator is the actual teacher.

Feed validator errors back into the next attempt. Do not write a fresh vibe prompt. The error path is the lesson you wanted.

SCHEMA_FAIL qty: '1' is not of type 'integer'
SCHEMA_FAIL currency: 'usd' does not match '^[A-Z]{3}$'
SCHEMA_FAIL debug: additional properties are not allowed
Enter fullscreen mode Exit fullscreen mode

Is that uglier than “please follow the schema”? Yes. Is it specific enough to retry against? Also yes.

Paste those three lines. Leave the pep talk out. Models follow concrete failures better than manners.

Myth 5: Coercion in the handler is a friendly adapter

Claim: The app can cast strings to ints, so relax.

What happens: Coercion hides the generation bug forever. "qty": "" becomes 0 and ships. Inventory math then looks haunted.

Corrected model: Coerce at the edge of human input, maybe. Never coerce a model object silently.

If the gate fails, regenerate with the error. Do not write int(qty or 0) and move on. That line hides a broken generator.

# labeled anti-pattern: this is the bug, not the fix
def load_qty(payload):
    return int(payload.get("qty") or 0)  # do not ship this
Enter fullscreen mode Exit fullscreen mode

Want a handler? Map only after SCHEMA_OK. Mapping is translation. It is not forgiveness.

The artifact: three files and a pytest

Create three files beside the gate. Do this before the first model call.

  • schema.json holds the contract above.
  • good.json holds one object that must pass.
  • bad.txt holds messy model output that must fail. 
{
  "sku": "SKU-1044",
  "qty": 2,
  "currency": "USD"
}
Enter fullscreen mode Exit fullscreen mode 
Here you go!
Enter fullscreen mode Exit fullscreen mode


json
{"sku": "SKU-1044", "qty": "2", "currency": "usd", "debug": true}

Enter fullscreen mode Exit fullscreen mode


shell

Now run both paths and demand opposite codes.

python gate_schema.py schema.json good.json; echo good:$?
python gate_schema.py schema.json bad.txt; echo bad:$?
Enter fullscreen mode Exit fullscreen mode

I want good:0 and a non-zero bad. If both pass, your gate is theater. What would you even be testing then?

Add a pytest so CI repeats the argument.

# labeled example: fixture test, run with pytest
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).parent
GATE = ROOT / "gate_schema.py"


def run_gate(payload: str) -> int:
    proc = subprocess.run(
        [sys.executable, str(GATE), str(ROOT / "schema.json"), str(ROOT / payload)],
        check=False,
    )
    return proc.returncode


def test_good_payload_passes():
    assert run_gate("good.json") == 0


def test_messy_model_output_fails():
    assert run_gate("bad.txt") in {1, 2}
Enter fullscreen mode Exit fullscreen mode 
pytest -q test_schema_gate.py
Enter fullscreen mode Exit fullscreen mode

A passing lint on generated code is not this test. This test checks the payload contract. Keep those jobs separate.

Decision table I keep in the README

Observation Ship? Next action
Parse error No Retry extract, then regenerate
Schema fail No Paste validator errors, regenerate
Extra fields No Keep additionalProperties false
Types coerced in app No Delete the cast, fix generation
SCHEMA_OK plus fixtures Maybe Run real unit tests next

“Maybe” is the honest cell. A schema gate is necessary. It is not sufficient.

A workflow you can finish in one sitting

  1. Write schema.json first, before any prompt text.
  2. Add good.json and bad.txt before any model call.
  3. Generate a draft payload with a free model.
  4. Save the raw text to a file, not your clipboard memory.
  5. Run gate_schema.py and read the exit code.
  6. On failure, paste the SCHEMA_FAIL lines back.
  7. Repeat until the gate prints SCHEMA_OK.
  8. Only then drop the object into application tests.

Need a cheap place to burn retries? Free model access helps that loop. Need isolation from your laptop? A free server helps that loop. Still run the gate on a machine you control.

# labeled example: generate remotely, validate locally
scp user@scratch:model_raw.txt .
python gate_schema.py schema.json model_raw.txt
Enter fullscreen mode Exit fullscreen mode

Do not skip the copy step. A UI screenshot is not an artifact. Files in git are artifacts.

What this approach does not do

It does not prove business rules. Stock math still needs real code.

It does not prove idempotency. Two SCHEMA_OK objects can still double-charge.

It does not prove prompt provenance. Pin prompts separately if you care.

It does not make a free server your compliance boundary. Do not paste secrets. Do not paste customer records. Do not treat retries as an audit log.

jsonschema will not save an outdated schema. If the app changed and git did not, the gate lies. Format checkers are also optional unless you enable them.

Date-time strings are a frequent trap here. "2026-09-04" can pass a loose string type. It can still break your parser tomorrow.

Who should not use this loop

Skip this if generated types already hit a compiler. Your compiler is a stricter gate.

Skip this if payloads are regulated and legal review is required. A FAQ is not a control.

Skip this if you cannot store the schema in version control. Chat-only contracts will drift.

Skip this if you need a vendor SLA for every token. Free access is not that promise.

The mental model I want you to keep

Models emit strings. Servers execute processes. Schemas define contracts. Those are three different jobs.

Can a free model draft the string? Yes. Can a free server host the experiment? Yes. Can either one sign the contract? No.

What is your actual gate today? A parser? A vibe check? A schema file in git?

If it is only a parser, you do not have a contract yet. Put the schema in git before the next prompt.

Top comments (0)