Structured Output That Survives a Model Swap: The JSON Scaffold I Actually Ship

#prompting #json #structuredoutput #schema

This post is about getting stable structured output across model swaps. It is not about OpenAPI design in the abstract. By the end you will have a paste-ready scaffold, a repair pass, and a short list of times you should refuse to use JSON at all.

The July wave made multi-model routing ordinary. Cheap model for triage. Mid model for drafts. Flagship for hard reasoning. That architecture dies if every model returns a slightly different shape of "helpful" prose.

Schema is how you keep the employee badge the same when the employee changes.

The fail-state I still see

Return JSON with the headline and risks.
Be careful and thorough.

Results I have graded in the last month:

Markdown fences around the object
Keys in camelCase one run, Title Case the next
Invented fields "for completeness"
A preface paragraph apologizing for uncertainty
Valid JSON that fails the business schema (wrong types, missing required)

Different models, same soft prompt, different mess. Soft prompts do not survive routing.

The scaffold I ship

Constraints first. Always.

# Structured extraction, production scaffold
Constraints, in priority order:
1. Output a single JSON object. No markdown fences. No prose before or after.
2. Use only the keys defined in SCHEMA. No extra keys.
3. If a value is unknown, use null. Never invent a number, URL, or quote.
4. Strings must be plain text (no HTML). Arrays must be arrays even if length 1.
5. If the input is empty or unusable, return {"error":"unusable_input","reason":string}
   and no other keys.

SCHEMA:
{
  "headline": string | null,
  "dek": string | null,
  "risks": string[],
  "confidence": "high" | "medium" | "low"
}

TASK:
Read INPUT. Fill SCHEMA. Obey constraints.

INPUT:
"""
{{input}}
"""

That is the recipe. The dials you turn per client are the schema fields and the error object, not the poetry around them.

Model-specific residue (patch, do not rewrite)

Symptom	Patch
Model wraps

I re-test this scaffold with the 12-prompt eval jobs 3 and 12 whenever a default model changes.

The repair pass (second call, cheap)

When parse fails or schema validate fails, do not start a conversation. Run a dedicated repair:


prompt
Constraints:
1. Output a single JSON object matching SCHEMA exactly.
2. Use only facts present in DRAFT or SOURCE. No new facts.
3. If you cannot comply, return the error object from SCHEMA rules.

SCHEMA: {{same schema}}
SOURCE: """{{original input}}"""
DRAFT: """{{invalid output}}"""

Two cheap calls beat one expensive meandering call. This is especially true on Luna-class or other budget tiers.

When not to force JSON

The artifact is for a human's eyes only and voice matters more than keys (final essay, customer apology).
You do not have a schema yet because you do not know the job. Free text first, schema second.
The model must ask a clarifying question. Force a small JSON like {"need":"clarification","question":string} instead of pretending you have the answer.

JSON is a contract. Do not sign a contract for a conversation.

Tiny experiment

Take one production automation that currently ends in "parse whatever the model said." Freeze a schema with four fields. Run ten real inputs on two models. Count valid-and-true objects, not merely parseable ones. If either model is under 8/10, fix the scaffold before you touch temperature.

The scaffold survives the temperature change. The fail-state is trusting a fence-stripping regex as your only schema. The wrong reader for this post is someone who wants a 40-key mega object on day one. Start smaller. Earn complexity.