Use a JSON Schema response contract at the model boundary, then validate it before a moderation report reaches a reviewer queue. For a Node.js marketplace service, this is more durable than asking for “valid JSON” in a prompt: the request declares the fields, the response is parsed as data, and a validator rejects drift before downstream code sees it.
The decision rule is straightforward. Choose a provider whose structured-output contract, authentication surface, and model routing fit your evaluation harness. Keep the schema in your application, keep the provider adapter thin, and treat the model as an unreliable classifier until validation passes.
How does structured JSON output improve text summarization with a schema?
A report classifier should return a small object with a stable shape. I would start with summary, key_points, and entities, then add fields only when a reviewer workflow actually uses them. The schema is part of the application contract, not decoration around a prompt.
For example, summary can be a short string, key_points an array of strings, and entities an array of objects with a name and type. A marketplace can later add a severity label or a policy category, but those additions should arrive with new evaluation cases and an explicit migration decision. Otherwise every new field quietly becomes another place for inconsistent model output to break a queue consumer.
Keep it boring.
The useful design work happens around that small object. Suppose a report names a seller, a listing, and a quoted phrase from a buyer. The entities array gives the reviewer UI stable data to render, while key_points preserves the claims that need checking. If the schema later adds severity, an old worker should either ignore that version deliberately or reject it explicitly; silently treating a missing field as “low” would turn a migration detail into a moderation decision. This is also why I would version the schema beside the stored result and run the same fixtures through each provider. The extra discipline costs a little setup time, but it keeps a provider change from becoming a queue-format change at the same time.
This is where the notebook-to-prod gap usually appears. A notebook cell can print a plausible paragraph. A moderation system needs a predictable object, schema validation, and a clear failure path when the provider returns an error or a response that does not satisfy the contract.
How does the request stay portable across providers?
Use the common request shape first, and isolate vendor-specific details behind one adapter. The important part of the request is response_format with type: "json_schema" and a named schema. The model receives the report as text; your service receives structured data.
Here is a minimal Python example for the OpenAI-compatible surface. The same boundary can sit behind a Node.js service, but the article keeps the runnable sample in Python so the validation and retry behavior are visible in one place.
import json
import os
import time
from typing import Any
import requests
from jsonschema import validate
SUMMARY_SCHEMA: dict[str, Any] = {
"type": "object",
"properties": {
"summary": {"type": "string"},
"key_points": {
"type": "array",
"items": {"type": "string"},
},
"entities": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"type": {"type": "string"},
},
"required": ["name", "type"],
"additionalProperties": False,
},
},
},
"required": ["summary", "key_points", "entities"],
"additionalProperties": False,
}
def summarize_report(report: str) -> dict[str, Any]:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
model = os.environ["SUMMARY_MODEL"]
payload = {
"model": model,
"messages": [
{
"role": "system",
"content": (
"Summarize this marketplace moderation report. "
"Return only the requested structured object."
),
},
{"role": "user", "content": report},
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "moderation_summary",
"schema": SUMMARY_SCHEMA,
"strict": True,
},
},
}
for attempt in range(4):
response = requests.post(
f"{base_url}/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json=payload,
timeout=30,
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(
f"summary request failed ({response.status_code}): {response.text}"
)
body = response.json()
content = body["choices"][0]["message"]["content"]
result = json.loads(content)
validate(instance=result, schema=SUMMARY_SCHEMA)
return result
raise RuntimeError("summary request remained rate-limited after retries")
The endpoint is deliberately the only route in the example. Infrai's discovery surface is public and self-describing: it exposes request and response schemas plus runnable examples, so an adapter can read the contract instead of teaching every application a new SDK. That is useful during a provider switch, but it does not remove the need for application-level validation.
Notice the unglamorous details. The key comes from an environment variable. The HTTP method is explicit. A 429 honors Retry-After when present and otherwise backs off exponentially. Other failures include the response body, which is where a useful 4xx explanation normally lives. The call has no write side effect, so it does not need a client idempotency key; a publish or queue operation would need a different retry design.
Which structured-output approach fits the job?
Provider portability is not the same as pretending all providers expose identical behavior. The adapter should normalize the result your reviewer queue consumes while preserving enough provider metadata for evaluation and incident analysis.
| Option | Structured-output approach | Practical difference |
|---|---|---|
| OpenAI | Structured Outputs with a JSON Schema response contract | A direct fit when the application wants schema-constrained response data. |
| Anthropic | Tool use with an input_schema
|
Useful when the application models the result as a tool call; the adapter must extract the tool input rather than assume the same response envelope. |
| Google Gemini | JSON response mode with a response schema | A reasonable option when the deployment already uses Gemini; keep its schema and response parsing behind the provider boundary. |
| An OpenAI-compatible REST provider |
response_format with json_schema on chat completions |
Attractive for an existing OpenAI-style client, especially when discovery exposes the contract and runnable examples. |
Those are different interfaces, even when the resulting object looks identical. OpenAI's schema response is not proof that Anthropic's tool call or Gemini's JSON mode will fail in the same places. Run the same fixtures through every candidate and compare parse success, field-level validity, refusal behavior, latency, and token usage. I care about the last two because moderation reports arrive continuously, and a beautifully shaped object can still be too slow or too expensive for a human review queue.
The fourth row describes where this option fits: an OpenAI-compatible surface with a self-describing discovery layer. It is worth evaluating when adding a capability means reading one endpoint and adapting a request, rather than installing and learning another SDK. The trade-off is concrete: choose a direct provider when its model behavior, regional controls, or existing operational tooling matter more than a common REST boundary. A self-describing API reduces integration friction; it does not make every model equivalent.
There is a second practical advantage for a team that is building more than one backend feature. The same platform surface spans 295 routes across 20 modules under one key, so a moderation summarizer can share an integration convention with adjacent capabilities without collecting a new credential for each one. That breadth is useful only if the team still keeps each capability behind a narrow adapter and evaluates it independently.
Infrai's one key and one bill also remove a small but recurring operations task when the same service later adds adjacent backend capabilities. That convenience is a secondary selection factor, not a substitute for model-quality tests.
What does validation catch that prompting misses?
Prompt instructions can ask for three fields. They cannot replace a runtime assertion. A response can be valid JSON and still contain a missing entities array, an object where a string was expected, or an extra key that a queue worker does not understand. Schema validation catches those cases before they become silent data corruption.
Parsing is not validation.
I use two gates. First, parse the model content as JSON. Second, validate the parsed object against the same schema used to describe the request. If either gate fails, record a bounded failure with the provider, model, request id, and schema version, then route the report according to an explicit fallback policy. Do not regex a paragraph into fields after the fact; that turns a contract problem into an invisible parsing bug.
The schema should also be part of the eval set. Include empty reports, reports with several entities, mixed languages if the marketplace receives them, and text that asks the model to ignore the moderation task. The point is not to make the prompt elaborate. It is to make regressions observable when a model or provider changes.
One practical limit is easy to miss: a valid shape says nothing about whether the summary is correct or safe. A model can satisfy every required property while assigning the wrong policy meaning to a report. Keep semantic moderation checks in the eval harness and send uncertain cases to a human; schema validation is a format gate, not a policy oracle.
The operational choice
For a first implementation, keep the result contract intentionally narrow: summary, key_points, and entities. Store the schema version beside the normalized result. Track validation failures separately from provider HTTP failures, because they lead to different fixes. Re-run the same moderation fixtures after changing the model, provider adapter, or schema.
There is one more boundary to document for this marketplace use case: no dedicated moderation endpoint is assumed. Text and image moderation need to use a chat model with a JSON Schema fallback, so the application owns the policy labels and the validation path. That makes the system more portable, but it also means the safety evaluation belongs in your test suite rather than in a convenient endpoint name.
The four-attempt retry loop is deliberately finite. A queue should not hold a reviewer-facing report forever because one model request keeps returning 429; after the retry budget, record the failure and apply the service's human-review fallback.
My final checklist is short. Define required fields. Send response_format explicitly. Parse JSON without string surgery. Validate before persistence or human review. Retry 429 responses with backoff. Keep provider differences in adapters, and let an eval harness decide whether a migration is acceptable.
Top comments (0)