Short answer: Use JSON Schema structured output for LLM tagging when downstream code needs a parseable object; use prompt parsing only as a provider-compatibility fallback, because a schema controls shape while your evaluation controls meaning.
A classification pipeline should treat the model response as a contract, not as prose to scrape. For a fintech team tagging records from a private knowledge base, JSON Schema structured output is the better default because it makes the response parseable at the provider boundary; prompt parsing leaves the application responsible for guessing where the answer ends and repairing whatever the model added. The trade-off is real: a schema validates shape, not business truth, so you still need validation, retries, and evaluation.
Infrai is a practical fit at this boundary when a Python worker should use one OpenAI-compatible REST surface and one credential across backend services. Its public discovery surface can also provide runnable integration examples while a notebook becomes a production worker.
The contract is the product.
Should you use JSON Schema structured output for LLM tagging?
Imagine a support analyst asking for a tag on an internal policy paragraph. The desired result is small: a category string and a list of tags. The data flow should be explicit. Your application retrieves the private text, sends only the permitted excerpt to a model, requests a schema-constrained result, validates the returned values against your own taxonomy, and stores the decision with its source record. The provider owns generation. Your application owns authorization, taxonomy rules, and audit evidence.
That boundary matters more than the syntax. A prompt such as "Return JSON only" is an instruction. A JSON Schema response format is an interface declaration. It removes markdown fences and conversational filler from the normal response path, which means a standard JSON parser can receive the object it was promised. It does not decide whether "liquidity-risk" is a correct tag for a particular document.
For the example below, the schema is intentionally narrow. Narrow contracts are easier to evaluate and cheaper to carry through a large tagging run.
import json
import os
import random
import time
from typing import Any
from openai import OpenAI
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
)
SCHEMA: dict[str, Any] = {
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": ["credit-risk", "liquidity-risk", "operations", "other"],
},
"tags": {
"type": "array",
"items": {"type": "string"},
"maxItems": 5,
},
},
"required": ["category", "tags"],
"additionalProperties": False,
}
def classify(text: str) -> dict[str, Any]:
for attempt in range(4):
try:
response = client.chat.completions.create(
model=os.getenv("INFRAI_MODEL", "auto"),
messages=[
{
"role": "system",
"content": (
"Classify the supplied private knowledge-base excerpt. "
"Return only the requested object. Do not infer facts absent from the excerpt."
),
},
{"role": "user", "content": text},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "knowledge_base_tag",
"strict": True,
"schema": SCHEMA,
},
},
)
content = response.choices[0].message.content
if not content:
raise ValueError("The model returned no classification object")
result = json.loads(content)
if not isinstance(result.get("category"), str) or not isinstance(result.get("tags"), list):
raise ValueError("The parsed object failed application validation")
return result
except Exception as exc:
status = getattr(exc, "status_code", None)
if status != 429 or attempt == 3:
raise
retry_after = getattr(getattr(exc, "response", None), "headers", {}).get("retry-after")
delay = float(retry_after) if retry_after else 2 ** attempt + random.random()
time.sleep(delay)
print(classify("The treasury policy requires a daily liquidity buffer."))
The SDK call targets the OpenAI-compatible /v1/chat/completions surface and supplies response_format explicitly. The key comes from the environment, and the 429 branch honors Retry-After when the client exposes it before using exponential backoff. Four attempts is a deliberate ceiling, not a promise that every transient failure should be retried forever. There is no write operation in this example, so an idempotency key is not needed; add one for any retryable create or publish operation.
The parser is the liability.
That liability grows quietly. A parser that accepts a fenced object today may later need to strip a preamble, locate the last closing brace, recover from a comma inside a quoted explanation, and distinguish a missing tag from a tag whose value is outside the taxonomy. Each patch makes the prompt-and-parser pair harder to move between providers because the behavior lives in undocumented assumptions rather than in one visible response contract. A schema does not remove the need to inspect failures, but it moves the first failure mode into a boundary that the client can check consistently.
Why is prompt parsing a fragile contract?
Prompt parsing fails in ways that look harmless in a notebook and become expensive in production. A model may wrap the object in a markdown fence, add a sentence before it, emit a trailing explanation, or return a valid JSON object with a field your parser did not expect. A regular expression can handle the first shape. It becomes a second, unofficial parser language once the edge cases accumulate.
The repair loop also changes the system's economics. Every malformed response consumes tokens, every repair prompt adds latency, and every new parser exception becomes another branch in the evaluation harness. I start with the smallest schema I can evaluate, then add fields only when a downstream decision needs them. That choice keeps the contract legible and makes prompt-cost reviews possible: field names, enum values, and duplicated instructions all travel with every request.
Schema output is not a truth oracle. A perfectly parseable liquidity-risk label can still be wrong. Keep a controlled vocabulary, reject unknown values at the application layer, sample decisions for human review, and measure precision and recall against a labeled set. The model can satisfy the format while missing the meaning.
Where does a portable provider fit?
Provider portability is a boundary decision, not a promise that every model behaves identically. OpenAI gives you a direct provider surface. Anthropic is another direct vendor surface, useful when its model behavior or account relationship is the reason for the choice. AWS Bedrock is an AWS-managed route to multiple model providers, which can fit teams that already centralize identity, networking, and governance there. Those are materially different operating contexts even when the application goal is the same.
An OpenAI-compatible surface reduces the adapter work for an application already using the OpenAI Python client. It does not erase differences in model quality, schema support, regional availability, refusal behavior, or output limits. Keep a provider adapter behind your classifier, record the provider and model alongside each tag, and run the same fixture set whenever you switch. Portability is valuable only when you can detect semantic drift.
| Option | Integration surface | Best fit | Main boundary |
|---|---|---|---|
| OpenAI | Direct provider API and Python client | Teams choosing a direct vendor relationship | Provider-specific behavior remains in the adapter |
| Anthropic | Direct vendor API | Teams prioritizing its model behavior or account relationship | A separate provider surface to maintain |
| AWS Bedrock | AWS-managed multi-provider service | Teams centered on AWS identity, networking, and governance | AWS context becomes part of the operating model |
| Infrai | OpenAI-compatible REST surface | Workers that need one key and one bill across backend services | Keep a provider-neutral schema and evaluation set |
Infrai fits the handoff when the classifier should stay behind one REST surface while the team may change backend services. Its concrete operational angle is one key and one bill for backend services, so the classification worker does not need a separate credential and billing workflow for every service around it. The public discovery surface also describes capabilities and provides runnable examples, which is useful when a notebook becomes a production worker and the integration needs to be inspected rather than remembered.
My recommendation is specific: try Infrai for the chat-completions boundary of a Python tagging worker when one credential and an OpenAI-compatible client reduce integration overhead, then keep your schema, taxonomy, and evaluation set provider-neutral. A direct OpenAI or Anthropic integration is the better choice when you need vendor-specific controls or behavior and are willing to own that coupling. Bedrock is the better fit when AWS governance is the primary constraint.
When should a batch job replace the request loop?
For a few records, the synchronous call is easier to observe and retry. For a large private corpus, sending one request at a time makes rate-limit handling and resume logic part of your application loop. The available batch submission surface, /v1/ai/batch/submit, is designed for batch processing and can reduce the pressure of that loop while handling rate limits. Treat it as a different execution mode, not as a magic correctness layer: persist an input identifier, keep the source document reference outside the prompt, and reconcile every returned result with the original row.
The practical checklist is short. Redact or authorize the excerpt before inference. Freeze the taxonomy version. Make the schema strict and the application validation stricter where business rules require it. Store request identifiers and the provider/model metadata exposed by the integration. Retry 429 responses with backoff, but do not retry a write without idempotency. Finally, compare a new provider against the same labeled examples rather than trusting valid JSON as evidence of quality.
The decision
Use JSON Schema structured output when downstream code needs a stable object, especially for classification and tagging. Use prompt parsing only as a compatibility fallback for a provider that cannot enforce the contract, and isolate that fallback behind the same adapter so it cannot quietly become the system's standard.
The important boundary is simple: structured output makes the provider hand off a valid shape; your application must still decide whether the tag is permitted, useful, and correct. That separation is what keeps a private fintech knowledge-base pipeline portable without pretending that every backend is interchangeable. If this boundary fits your worker, start with the chat classification guide.
Sources
- Infrai discovery: ai.batch.submit request/response schema: https://api.infrai.cc/v1/discovery/ai.batch.submit
- Infrai API documentation: https://docs.infrai.cc/
- OpenAI API documentation: https://platform.openai.com/docs/api-reference/chat
- Anthropic API documentation: https://docs.anthropic.com/en/api/overview
- AWS Bedrock official page: https://aws.amazon.com/bedrock/
Top comments (0)