The instruction “return only JSON” may encourage a language model to produce machine-readable text, but it does not create a dependable application contract. If a response feeds a form, CRM record, payment workflow, or agent action, the system also needs a defined schema, application-level validation, explicit uncertainty states, and separate handling for different failure classes.
This guide is based on a review of official documentation from Google, Hugging Face, and Microsoft. No hands-on tests, benchmarks, or provider comparisons were conducted. The objective is not to claim that structured output makes model-generated information true. It is to explain how schemas can constrain response shape so that software can parse the result predictably and decide whether it is safe to use.
Why “return JSON” is not a contract
A prompt may request JSON and still receive an explanation, a Markdown code fence, a missing field, or an unexpected value. Even syntactically valid JSON can violate the consumer’s expectations: a number may arrive as a string, a category may fall outside an approved list, or an unknown value may be replaced with a plausible guess.
A useful contract starts by defining what the next component needs. Which fields are required? Which values are permitted? How is missing or ambiguous information represented? Which conditions must block automation? The application should enforce the same contract supplied to the model.
Distinguish JSON mode, structured output, and tool calls
| Mechanism | Purpose | Primary limitation |
|---|---|---|
| JSON mode | Requests a JSON object without necessarily requiring conformance to a particular schema | The JSON may contain missing, unexpected, or incorrectly typed fields |
| Schema-constrained output | Constrains the response to a supported schema | A structurally valid response can still contain incorrect information |
| Function or tool call | Produces structured arguments for an operation controlled by application code | Valid arguments may still describe an unauthorized or unsafe action |
Use a structured final response when the model must report data to the application. Use a tool call when the model proposes an operation such as searching, updating a record, or invoking an internal service. The application should not treat the operation as successful until the external system confirms it.
Support varies by model, provider, endpoint, SDK, and agent type. Google’s Gemini documentation states that models can be configured to generate responses adhering to a provided JSON Schema and documents schemas defined directly or through Pydantic and Zod. Hugging Face documents structured outputs through Inference Providers and advises selecting the model and provider carefully to avoid compatibility problems. Microsoft states that not all agent types support structured outputs natively and that compatible chat clients are required.
Begin with a minimal schema
Suppose an application extracts an invoice number, amount, currency, and result status from a document. Avoid adding speculative fields, hidden reasoning, or summaries that the consumer does not need. A smaller contract is easier to understand, validate, version, and maintain.
{
"title": "InvoiceExtractionV1",
"type": "object",
"additionalProperties": false,
"properties": {
"schema_version": { "const": "1.0" },
"status": {
"type": "string",
"enum": ["complete", "incomplete", "ambiguous"]
},
"invoice_number": { "type": ["string", "null"] },
"amount": { "type": ["number", "null"] },
"currency": {
"type": ["string", "null"],
"enum": ["USD", "EUR", null]
},
"missing_fields": {
"type": "array",
"items": {
"type": "string",
"enum": ["invoice_number", "amount", "currency"]
},
"uniqueItems": true
}
},
"required": [
"schema_version", "status", "invoice_number",
"amount", "currency", "missing_fields"
]
}This is an illustrative design, not a universal request body or accounting rule. Providers may support different JSON Schema features. Keywords such as const, type unions, additionalProperties, or uniqueItems should not be assumed compatible without checking the documentation and validating the selected integration.
Represent unknown values explicitly
Requiring every key while permitting null can help consumers distinguish an unknown value from a malformed response. However, absence may have several meanings:
- unknown: the source does not provide the value;
- not applicable: the field does not apply to the object;
- ambiguous: the source contains conflicting alternatives;
- refused: the model did not complete the requested task;
- incomplete: generation or transmission did not finish.
These states should not all be stored in the same content field. Refusal, transport failure, and incomplete generation concern execution of the request rather than the source document. They are better represented in an outer result envelope or pipeline status.
Separate observations, classifications, and actions
A field such as risk: "high because the customer seems suspicious" mixes an observation, an assessment, and an implied decision. A clearer contract separates those layers:
{
"observations": {
"payment_overdue_days": 47,
"amount": 125000
},
"classification": "manual_review",
"recommended_action": "queue_for_review"
}The numbers and labels above are examples, not validated thresholds. Application code must verify extracted values, apply domain rules, and determine whether the model is authorized to recommend or initiate an action.
The same distinction matters when discussing occupations. Exposure of particular tasks to AI is not equivalent to an observed job loss. The AI professions map provides broader context for examining occupations without merging task exposure, forecasts, and observed outcomes into one field.
A schema validates structure, not truth
Schema validation can check properties such as field presence, data types, enums, and expressed constraints. It cannot establish that an invoice number was copied correctly or that a classification is justified. After parsing and schema validation, the application still needs domain validation.
For an invoice workflow, domain checks might include confirming the currency against the organization’s approved reference data, reconciling a total with line items, checking dates according to the relevant process, and blocking automatic posting when the document contains conflicting totals. These are workflow examples, not universal accounting requirements.
Descriptions in a schema may help explain a field, but they should not be the only safeguard. If an amount must satisfy a business rule, enforce that rule in the schema where supported and in application code where necessary. Until all required checks pass, the response should remain untrusted and should not change external state.
Handle each failure class separately
result = call_model(input, schema)
if result.network_error:
return retry_or_queue("transport_error")
if result.refusal:
return safe_fallback("model_refusal")
if result.incomplete:
return retry_or_review("incomplete_output")
if not result.has_expected_payload:
return fail("missing_payload")
try:
data = parse_json(result.payload)
except ParseError:
return fail("invalid_json")
schema_errors = validate_schema(data, schema_v1)
if schema_errors:
return fail("schema_mismatch", schema_errors)
domain_errors = validate_business_rules(data)
if domain_errors:
return review("domain_invalid", domain_errors)
return authorize_and_choose_action(data)Do not reduce every failure to another model call. A transport failure may justify a retry with an idempotent operation identifier. An unsupported schema requires a configuration change. An ambiguous source may require more data or human review. A refusal should remain visible instead of being concealed by extracting fragments from free-form text.
Do not validate a partial stream as a final object
Streaming requires special handling. Google documents streamed structured-output chunks as valid partial JSON strings that can be concatenated into the final object. Microsoft’s documentation says streaming updates must be assembled before deserialization in its described .NET workflow. These statements concern their documented interfaces and should not be generalized automatically to every API.
Application code should wait until the stream is complete before final parsing, schema validation, domain validation, or execution. A fragment that looks complete may still be followed by more data, while an interrupted stream may contain only part of an otherwise valid object.
Version schemas with their consumers
Adding a required field, renaming a category, or changing a unit alters the contract. Keep schemas under version control, validate them in continuous integration, and include a version in the response or request metadata. Consumers should explicitly accept supported versions and reject unknown ones.
One possible migration pattern is to add a new version, update the consumer to understand both old and new versions, validate the change against an appropriate test corpus, switch generation, monitor errors, and retire the old version only when the organization’s release criteria are met. This is an editorial workflow recommendation, not a measured or universally validated sequence.
Build negative cases into the review plan
A review plan should cover more than an ideal example. Suggested cases include an empty source, missing fields, conflicting values, unsupported categories, incorrect types, extra properties, truncated output, refusal, ordinary text in place of the expected payload, prompt-injection text inside the source, domain-invalid values, and an unknown schema version.
Schema conformance alone is not a complete quality measure. Teams may also need to assess field accuracy, ambiguity handling, unsafe automated actions, and safe-failure behavior. This article provides no measured error rates or universal acceptance thresholds; those criteria must come from the use case, its evidence, and its risk controls.
The same principles apply to self-hosted systems. Deployment considerations are discussed in the local AI guide. Schema size and repeated requests can also affect usage planning; pricing references are available in the AI token pricing guide.
Documentation review scope
This article is a documentation review based on the cited official Google Gemini API, Hugging Face Inference Providers, and Microsoft Agent Framework pages. No API calls, streaming trials, model comparisons, schema compatibility tests, latency measurements, or accuracy evaluations were performed for this review.
Provider-specific statements have therefore been kept within the scope of their respective documentation. Compatibility with a particular model, provider, endpoint, schema, or SDK must be established by the implementing team before production use.
Frequently asked questions
Does JSON Schema guarantee a correct answer?
No. It can constrain structure and expressed values, but it cannot prove that the data is truthful or complete.
Should structured output be validated again?
Yes. The application still needs parsing, domain validation, authorization checks, and handling for refusal, interruption, network errors, and missing results.
Should malformed JSON be repaired automatically?
Only when the risk is understood and the repair cannot alter meaning. For consequential workflows, recording the failure and obtaining a new result is generally safer than silently inventing missing syntax or fields.
What is the central design principle?
Treat model output as untrusted input. Constrain its shape, validate its meaning in code, distinguish uncertainty from execution errors, and prevent external state changes until every required check and authorization step is complete.
Sources and research
- Structured outputs | Gemini API | Google AI for Developers
- Structured Outputs with Inference Providers · Hugging Face
- Producing Structured Outputs with agents | Microsoft Learn
AI-assisted article, checked against the primary sources cited above.
Originally published by AI Feed: How to Get Reliable JSON From a Language Model: Schema, Validation, and Error Handling.
Top comments (0)