A week after the customer service ticket extraction feature launched, the logs were full of JsonParseException. Sometimes the model wraps JSON in a Markdown code block; sometimes it prepends a line like "Sure, here are the results"; and the field name can even change from orderNo to order_no. Adding regexes, truncating strings, and tweaking prompts holds up for a couple of days before it breaks again. This article explains LangChain4j's structured output: how to push format constraints down to the protocol layer, and how to add business validation that the schema cannot handle. The code is a complete project; copy it and it runs.
1. What This Problem Actually Is: Why Does JSON Returned by LLMs Always Break
JSON parsing failures are not because the model is "disobedient"; they happen because you only ask it with prompts and do not constrain it at the protocol layer.
The five most common ways of asking a model to return JSON go wrong:
- Wrapped in a code block: it returns a json block wrapped in three backticks, and Jackson immediately reports "Unexpected character '`'".
- Prefixes and suffixes: "Sure, I've organized it for you as follows:" followed by JSON. You slice from
{to}, and it breaks again when curly braces appear inside string values. - Field drift: this time
orderNo, next timeorder_no, and the time after thatorder_number. - Type drift: the amount should be a number, but the model gives you
"399 元", and parsingBigDecimalblows up immediately. - Enums improvising: you define
REFUNDandLOGISTICS, and it returns"退款"and"refund_request".
Many people's first reaction is to change the prompt: add "must return pure JSON", add "do not explain", add few-shot examples. It works in the short term, then breaks again a couple of days later. The reason is no mystery:
- Prompts are soft constraints. When generating each token, the model can still choose any word.
- As context grows longer, earlier format requirements get diluted. Ticket conversations are often thousands of words long, so the model is more likely to "casually" add an explanation.
- Different models understand "JSON" differently. Some think code blocks count as JSON; some write numbers as strings.
- Even low temperature is not zero probability. As long as one token goes off track, the whole response is ruined.
Some people also think that setting temperature to 0 makes things stable. Temperature only affects the sampling distribution; it does not guarantee that the same input always produces the same output: server-side batching, model version canary releases, and changes in context length can all make results drift. What is more troublesome is "half-correct" output—four out of five fields are correct, parsing succeeds, and bad data is silently written to the database. A total failure at least triggers an alert; a half-correct result quietly pollutes reports.
Hand-written cleanup code is equally unreliable. Regexes can only cover the failure modes you have seen; substring(indexOf("{"), lastIndexOf("}")) truncates the JSON when } appears inside a string value; worse, swallowing parse exceptions and returning null pushes the null pointer to the business layer and doubles the debugging cost.
Cleanup code also has a hidden cost: it mixes "format problems" and "business problems" together. When parsing fails, you add a regex; when a field is wrong, you add if; six months later nobody dares to touch that code, because nobody knows which line is compensating for which model's which quirk. Structured output pulls this back into the type system: if something breaks, it is a compile error or a parse exception, and the boundary is clear.
Here the problem needs to be split into two layers:
LayerSpecific problemWho can solve itShapeIs it valid JSON, are the field names correct, are the types correct, are the enum values validJSON SchemaSemanticsIs the amount a refund amount or an order amount, is the classification correct, is the order number the one for this orderBusiness validation
The schema can guarantee that amount is a number, but it cannot guarantee that the number comes from "refund 399 yuan" rather than "original price 599 yuan". The schema can guarantee that category is one of the four enums, but it cannot guarantee that the model did not classify a logistics issue as a refund.
In one sentence: leave format to the schema, facts to validation, and failures to bounded retries.
2. How the Underlying Mechanism Actually Works: How JSON Schema Constrains LLM Output
Structured output compiles Java types into a JSON Schema, sends it to the model with the request, constrains the generation phase with the schema, and then deserializes the returned text back into an object.
The whole chain has three steps.
Step 1: Translate Java Classes into a Schema
LangChain4j uses reflection to read your return type and generates the schema field by field:
-
String→"type": "string" -
int/Integer→"type": "integer" -
BigDecimal/double→"type": "number" -
enum→"type": "string"plus theenumwhitelist -
List<T>→"type": "array"plusitems - Nested objects →
"type": "object"; recursive structures are closed off with$defsand$ref
The @Description on a field gets written into the schema's description. Don't underestimate this one line of text — it's mainly what the model relies on to decide "what should go in this field." The @JsonProperty(required = true) on a field goes into the required list.
This translation is recursive. List<TicketItem> produces an array, and the element objects get expanded in turn; self-referencing types don't recurse infinitely — once LangChain4j detects a cycle, it switches to $ref pointing at $defs. So you can put tree structures in your DTO and still get a valid schema, it's just that the deeper the nesting, the higher the odds the model fills something in wrong.
A schema generated from a ticket class looks roughly like this:
{
"type": "object",
"properties": {
"orderNo": {
"type": "string",
"description": "订单号，格式为 ORD- 加 8 位数字"
},
"category": {
"type": "string",
"enum": ["REFUND", "LOGISTICS", "INVOICE", "OTHER"]
},
"amount": {
"type": "number",
"description": "涉及金额，单位元；对话里没提金额就填 0"
}
},
"required": []
}
Note that required here is empty. Only fields marked @JsonProperty(required = true) make it in. Before sending to OpenAI, strict mode does another rewrite pass: it stuffs all the keys into required and turns the types of fields not marked required into nullable forms, such as ["string", "null"] and ["number", "null"]. So "required" under strict mode only guarantees the key is present, not that the value is non-null. I'll dig into this trap later.
Step 2: Put the Schema into the Request
The request carries the schema via ResponseFormat:
ResponseFormat.builder()
.type(ResponseFormatType.JSON)
.jsonSchema(schema)
.build();
The OpenAI adapter layer translates it into response_format: {"type": "json_schema", ...}.
If strict is enabled, the server enforces the constraint at the decoding stage: for every token generated, candidates that violate the schema get masked out outright. The model isn't "trying its best to get it right" — it "can't write it wrong." That's a whole different thing from asking it nicely in a prompt.
Capabilities vary widely across vendors:
- OpenAI's Structured Outputs supports a common subset of JSON Schema, with hard constraints in strict mode.
- Inference frameworks like vLLM use guided decoding and can also mask tokens according to the schema.
- Plenty of OpenAI-compatible services only support
json_object. It only guarantees "the whole thing is valid JSON" — field names, types, and enums are all ignored. - Some services just ignore
response_formatentirely — whatever you pass, they act like they never saw it.
json_object and json_schema often get conflated. The former only guarantees syntax; the latter governs field names and types as well. A concrete example: you ask it to extract {"amount": 399}, and json_object mode might return {"退款金额": "399元"} — perfectly valid syntax, and your DTO still can't handle it. Schema mode won't even let the key name amount be changed.
LangChain4j uses capability flags to decide which path to take. Only if the model's supportedCapabilities() includes RESPONSE_FORMAT_JSON_SCHEMA does it go with native schema; otherwise it appends the format instructions to the user message. The fallback mode isn't completely useless — it pastes the schema and field descriptions into the user message, so the structure the model sees is accurate, and most of the time it gets it right. The difference is between "most of the time" and "always."
There's a well-hidden trap here: OpenAiChatModel doesn't declare this capability by default. You have to explicitly declare .supportedCapabilities(RESPONSE_FORMAT_JSON_SCHEMA) for it to turn the capability flag on. Without that line, AiServices will still convert the result into an object, but under the hood it's "pasting a format requirement into the prompt," and stability drops off immediately. A lot of people say "I'm using AiServices, so why does it still fall over?" — and the answer is usually right here.
Step 3: Turn the Returned Text Back into an Object
After AiServices gets the response, it hands it to PojoOutputParser. The parser doesn't just readValue right away — it first tries the whole string; if that fails, it scans backwards for the last } or ], then matches braces going backwards, skipping over braces that sit inside strings. So code fences and surrounding chatter all get rescued.
This leniency is a safety net, not a design goal. If field names or types are wrong, it can't save you either — it just throws OutputParsingException.
There's another case it can't do anything about: the response gets truncated by max_tokens. The JSON is missing a closing brace, brace matching fails, and it throws an exception outright. You can catch this kind of failure early by checking finishReason — don't wait for Jackson to blow up. Setting max_tokens comfortably above your normal output size is a cheap and effective fix.
Three Things Schema Can't Cover
Semantics. Schema governs shape. Whose amount it is, whether the category is right, whether the order number belongs to this order — it doesn't concern itself with any of that.
Unknown values. The OpenAI adapter's strict mode puts all properties into required and also changes the types of fields not marked required into nullable form. The key must be present, and the value can be null. When the model encounters "the amount isn't mentioned in the conversation," it is more likely to give you null than to make up a number. So in the DTO, use wrapper types to receive it. If you use int, Jackson will silently turn null into 0, and validation sees fake data; if the business does not accept nulls, check it in validation, or add UNKNOWN to the enum and agree on 0 for the amount.
Cross-field rules. A constraint like "the refund amount cannot be greater than the order amount" can be written with conditional expressions in JSON Schema, but support varies widely among model vendors. Don't pin business rules on the schema; just write Java validation.
3. Hands-On: Writing LangChain4j Structured Output Code Step by Step
A minimal working project consists of four parts: dependencies, DTOs, the AI Service interface, and validation plus retry. The three examples below go from simple to complete, and each one is a full class.
Dependencies
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>structured-output-demo</artifactId>
<version>1.0.0</version>
<properties>
<maven.compiler.release>21</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j</artifactId>
<version>1.22.0</version>
</dependency>
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-open-ai</artifactId>
<version>1.22.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.22.4</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>2.0.20</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.16.0</version>
</plugin>
</plugins>
</build>
</project>
Example 1: AiServices Automatically Generates the Schema
This is the least effort approach. Define an interface, set the return type to a DTO, and LangChain4j handles schema generation, request sending, and result parsing.
package com.example.structured;
import dev.langchain4j.model.chat.ChatModel;
import dev.langchain4j.model.openai.OpenAiChatModel;
import dev.langchain4j.model.output.structured.Description;
import dev.langchain4j.service.AiServices;
import dev.langchain4j.service.UserMessage;
import java.math.BigDecimal;
import static dev.langchain4j.model.chat.Capability.RESPONSE_FORMAT_JSON_SCHEMA;
public class TicketExtractionDemo {
public enum Category {
REFUND, LOGISTICS, INVOICE, OTHER
}
public static class Ticket {
@Description("订单号，格式为 ORD- 加 8 位数字")
public String orderNo;
@Description("问题分类：REFUND 退款、LOGISTICS 物流、INVOICE 发票、OTHER 其他")
public Category category;
@Description("涉及金额，单位元；对话里没提金额就填 0")
public BigDecimal amount;
@Description("优先级，1 最低，5 最高；催得急或涉及投诉填 4 以上")
public Integer priority;
@Description("用户诉求摘要，不超过 30 个字")
public String summary;
}
public interface TicketAssistant {
@UserMessage("""
从下面的客服对话中抽取工单信息。
只抽对话里明确出现的内容，不要推测。
对话：
{{it}}
""")
Ticket extract(String dialogue);
}
public static void main(String[] args) {
ChatModel model = OpenAiChatModel.builder()
.apiKey(System.getenv("OPENAI_API_KEY"))
.modelName("gpt-4.1-mini")
// 这两行决定走原生 JSON Schema，而不是在 prompt 里贴格式说明
.supportedCapabilities(RESPONSE_FORMAT_JSON_SCHEMA)
.strictJsonSchema(true)
.temperature(0.0)
.build();
System.out.println("模型能力: " + model.supportedCapabilities());
TicketAssistant assistant = AiServices.builder(TicketAssistant.class)
.chatModel(model)
.build();
Ticket ticket = assistant.extract("""
用户：我上周买的耳机，订单 ORD-20261012，到现在还没发货。
客服：我帮您查一下物流。
用户：不用查了，直接退款，399 元，太慢了。
""");
System.out.println("订单号: " + ticket.orderNo);
System.out.println("分类: " + ticket.category);
System.out.println("金额: " + ticket.amount);
System.out.println("优先级: " + ticket.priority);
System.out.println("摘要: " + ticket.summary);
}
}
Let's break down the key behaviors:
-
{{it}}in@UserMessageis a placeholder corresponding to the method's only parameter. You don't need to manually concatenate strings. - The return type is
Ticket, notString. This single signature tells LangChain4j to generate a schema and deserialize the result. -
@Descriptionis not a comment; it goes into the schema. The more abstract the field name, the clearer the description must be. For example, ifprioritydoesn't say '1 to 5', the model will happily give you 10. -
temperature(0.0)reduces randomness, but don't expect it to replace the schema. -
System.out.println(model.supportedCapabilities())is for troubleshooting. If the output lacksRESPONSE_FORMAT_JSON_SCHEMA, you're in prompt fallback mode.
Example 2: Hand-Write the Schema Without Going Through the Interface Proxy
Some scenarios are not suitable for AiServices: the schema is assembled at runtime, you need to reuse a ChatModel, or you want to see exactly what request was sent. In these cases, construct ChatRequest directly.
package com.example.structured;
import dev.langchain4j.data.message.SystemMessage;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.model.chat.ChatModel;
import dev.langchain4j.model.chat.request.ChatRequest;
import dev.langchain4j.model.chat.request.ResponseFormat;
import dev.langchain4j.model.chat.request.ResponseFormatType;
import dev.langchain4j.model.chat.request.json.JsonObjectSchema;
import dev.langchain4j.model.chat.request.json.JsonSchema;
import dev.langchain4j.model.chat.response.ChatResponse;
import dev.langchain4j.model.openai.OpenAiChatModel;
import java.util.List;
import static dev.langchain4j.model.chat.Capability.RESPONSE_FORMAT_JSON_SCHEMA;
public class RawSchemaDemo {
public static void main(String[] args) {
ChatModel model = OpenAiChatModel.builder()
.apiKey(System.getenv("OPENAI_API_KEY"))
.modelName("gpt-4.1-mini")
.supportedCapabilities(RESPONSE_FORMAT_JSON_SCHEMA)
.strictJsonSchema(true)
.temperature(0.0)
.build();
JsonSchema schema = JsonSchema.builder()
.name("Ticket")
.rootElement(JsonObjectSchema.builder()
.addStringProperty("orderNo", "订单号，格式为 ORD- 加 8 位数字")
.addEnumProperty("category", List.of("REFUND", "LOGISTICS", "INVOICE", "OTHER"))
.addNumberProperty("amount", "涉及金额，单位元，没有就填 0")
.addIntegerProperty("priority", "优先级，1 到 5")
.addStringProperty("summary", "用户诉求摘要，不超过 30 个字")
.required("orderNo", "category", "amount", "priority", "summary")
.additionalProperties(false)
.build())
.build();
ChatRequest request = ChatRequest.builder()
.messages(
SystemMessage.from("你是客服工单抽取器，只输出 JSON。"),
UserMessage.from("订单 ORD-20261012 一直没发货，我要退款 399 元。"))
.responseFormat(ResponseFormat.builder()
.type(ResponseFormatType.JSON)
.jsonSchema(schema)
.build())
.build();
ChatResponse response = model.chat(request);
System.out.println(response.aiMessage().text());
}
}
JsonObjectSchema.builder() provides a series of convenience methods; addStringProperty, addEnumProperty, and addNumberProperty all take two parameters: "name + description." required specifies which fields must be present, and additionalProperties(false) prevents the model from adding extras.
The advantage of hand-writing the schema is control: field names, enum values, descriptions, and required fields are all in your hands. The downside is that it easily drifts out of sync with the DTO. Change one side and forget the other, and parsing will still break. My advice: for scenarios the interface proxy can cover, prefer Example 1; only hand-write when the schema needs to be generated dynamically.
Example 3: Business Validation + Limited Retry
The first two examples solve shape; this one solves semantics. Core idea: after parsing succeeds, don't return immediately—run the rules first. If there's a problem, feed "which rule failed" back to the model and have it rewrite. Retry at most twice.
package com.example.structured;
import com.fasterxml.jackson.databind.ObjectMapper;
import dev.langchain4j.data.message.SystemMessage;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.model.chat.ChatModel;
import dev.langchain4j.model.chat.request.ChatRequest;
import dev.langchain4j.model.chat.request.ResponseFormat;
import dev.langchain4j.model.chat.request.ResponseFormatType;
import dev.langchain4j.model.chat.response.ChatResponse;
import dev.langchain4j.model.openai.OpenAiChatModel;
import dev.langchain4j.model.output.structured.Description;
import dev.langchain4j.service.output.JsonSchemas;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Pattern;
import static dev.langchain4j.model.chat.Capability.RESPONSE_FORMAT_JSON_SCHEMA;
public class ValidatedExtractor {
private static final Pattern ORDER_NO = Pattern.compile("^ORD-\\d{8}$");
private static final ObjectMapper MAPPER = new ObjectMapper();
private static final int MAX_ATTEMPTS = 3;
private final ChatModel model;
public ValidatedExtractor(ChatModel model) {
this.model = model;
}
public static class Ticket {
@Description("订单号，格式为 ORD- 加 8 位数字")
public String orderNo;
@Description("问题分类：REFUND 退款、LOGISTICS 物流、INVOICE 发票、OTHER 其他")
public String category;
@Description("退款或涉及金额，单位元；没有就填 0")
public BigDecimal amount;
@Description("优先级，1 最低，5 最高")
public Integer priority;
@Description("用户诉求摘要，不超过 30 个字")
public String summary;
}
public Ticket extract(String text) {
ResponseFormat format = ResponseFormat.builder()
.type(ResponseFormatType.JSON)
.jsonSchema(JsonSchemas.jsonSchemaFrom(Ticket.class).orElseThrow())
.build();
List<String> problems = List.of();
for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
String userText = text;
if (!problems.isEmpty()) {
userText = text
+ "\n\n上一次输出有以下问题，请逐条修正后重新输出：\n- "
+ String.join("\n- ", problems);
}
ChatRequest request = ChatRequest.builder()
.messages(
SystemMessage.from("只做信息抽取。不确定的字段按默认值填，不要编造。"),
UserMessage.from(userText))
.responseFormat(format)
.temperature(0.0)
.build();
ChatResponse response = model.chat(request);
String raw = response.aiMessage().text();
try {
Ticket ticket = MAPPER.readValue(raw, Ticket.class);
problems = validate(ticket);
if (problems.isEmpty()) {
return ticket;
}
System.out.println("第 " + attempt + " 次校验失败: " + problems);
} catch (Exception e) {
problems = List.of("输出不是合法 JSON：" + e.getMessage());
System.out.println("第 " + attempt + " 次解析失败，原始输出：" + raw);
}
}
throw new IllegalStateException(
"连续 " + MAX_ATTEMPTS + " 次没有拿到合格结果，最后的问题：" + problems);
}
private List<String> validate(Ticket t) {
List<String> problems = new ArrayList<>();
if (t.orderNo == null || !ORDER_NO.matcher(t.orderNo).matches()) {
problems.add("orderNo 必须是 ORD- 加 8 位数字，当前是 " + t.orderNo);
}
if (t.category == null
|| !List.of("REFUND", "LOGISTICS", "INVOICE", "OTHER").contains(t.category)) {
problems.add("category 只能是 REFUND、LOGISTICS、INVOICE、OTHER");
}
if (t.amount == null
|| t.amount.signum() < 0
|| t.amount.compareTo(new BigDecimal("100000")) > 0) {
problems.add("amount 必须在 0 到 100000 之间");
}
if (t.priority == null || t.priority < 1 || t.priority > 5) {
problems.add("priority 必须在 1 到 5 之间");
}
return problems;
}
public static void main(String[] args) throws Exception {
ChatModel model = OpenAiChatModel.builder()
.apiKey(System.getenv("OPENAI_API_KEY"))
.modelName("gpt-4.1-mini")
.supportedCapabilities(RESPONSE_FORMAT_JSON_SCHEMA)
.strictJsonSchema(true)
.temperature(0.0)
.build();
ValidatedExtractor extractor = new ValidatedExtractor(model);
Ticket ticket = extractor.extract(
"订单 ORD-20261012 一直没发货，我要退款 399 元。");
System.out.println(MAPPER.writerWithDefaultPrettyPrinter().writeValueAsString(ticket));
}
}
There are several design points in this code worth explaining.
The schema only declares the shape; validation rules are written in Java. category in Ticket deliberately uses String instead of an enum. In native schema mode, it is an enum whitelist; when third-party compatible services ignore the schema, it is just a free-form string. Validation catches both cases.
Feed the failure reason back. Just saying "regenerate" is useless; the model doesn't know what went wrong. Tell it "orderNo must be ORD- followed by 8 digits, and the current value is 20261012," and the second attempt will basically get it right. This is what separates effective retries from ineffective ones.
Cap the retries. If it still fails after three attempts, the input itself is the problem, and continuing to burn tokens is pointless. Throw an exception directly and hand it off to a human or a dead-letter queue.
The raw output must be logged. When parsing fails, raw is the only clue. In production, wiring this log into alerting is far more useful than guessing afterward why the model went off the rails.
One retry is one full call. Every retry resends the system prompt, the user's original text, and the schema, so the cost is very real. Therefore, list all validation rules at once; don't drip-feed them one by one.
4. Pitfalls and Best Practices: The 8 Places Where Structured Output Most Easily Goes Wrong
Eighty percent of structured output pitfalls aren't in schema syntax; they're in "assuming it works as soon as you configure it."
-
Just writing "return JSON". That's not a constraint. Either use
ResponseFormat+JsonSchema, or accept that it'll add code blocks whenever it feels like it. -
Assuming
AiServiceswill always use the native schema. If the model doesn't declareRESPONSE_FORMAT_JSON_SCHEMA, the framework falls back to prompt mode. For OpenAI models, remember to add.supportedCapabilities(RESPONSE_FORMAT_JSON_SCHEMA), and printsupportedCapabilities()once before going live to confirm. -
You also want optional fields in strict mode. OpenAI’s strict mode puts every property into
required, while fields not marked as required allownull. The keys are always there; the values may be empty. On the Java side, use wrapper types (Integer,BigDecimal). If you useint, Jackson will silently convertnullto 0, so validation ends up with fake data—harder to track down than an error. If you really don’t want nulls, addUNKNOWNto enums, agree on 0 for amounts, and handle it in validation. -
For amounts, use
double. Floating-point errors in monetary amounts are a disaster. UseBigDecimalin the DTO andnumberin the schema, and Jackson can convert them correctly. -
Use date fields directly as
LocalDate. When generating the schema,LocalDateisn't treated asstring; instead, it gets translated into an object with no fields, leaving the model to fill it in blindly. The ObjectMapper on the parsing side also isn't guaranteed to have JavaTimeModule registered. AcceptStringin the DTO, then convert it toLocalDatein the business layer. Two extra lines save you an afternoon of debugging. - The prompt and the schema each say something different. The prompt pastes the field structure again, so the model may output according to the old version in the prompt. The prompt should only cover the business rules (what counts as a refund, how priority is determined), and leave the structure to the schema.
- The DTO is both wide and deep. Twenty fields and three levels of nesting — the schema alone takes up several hundred tokens, and the model also tends to miss fields. Split it into two extractions: first extract the order number and category, then extract the details as needed. Both accuracy and cost improve.
- Retries with no cap and no idempotency. Retrying endlessly on validation failure means that with input that never passes, the bill can go up tenfold. Cap it at two or three attempts, log the raw output and issue list for each attempt, and make writes idempotent.
A few more that are less common but potentially fatal:
-
Field names are too casually chosen. With names like
a,b, andtmp, the model can only guess. Use semantic English names such asorderNoandrefundAmount, and add@Descriptionto fill in the details. -
Forgot
additionalProperties(false). The model can add arbitrary fields to the object. These fields cannot enter the DTO, waste output tokens for nothing, and also show that it did not strictly follow your structure. Turn it off when writing the schema by hand. -
Streaming output paired with a POJO return value. The streaming method in
AiServicesreturnsTokenStreamand cannot directly return a DTO. If you want structured results, do not enable streaming, or parse the result yourself after streaming is complete. -
Third-party compatible services silently ignore
response_format. It does not report an error and still returns a passage of prose. So validation and parsing fallbacks are not optional; they are mandatory.
After launch, monitor three metrics: parse success rate, validation failure rate, and average retry count. If the parse success rate is low, check the model's capability flags and max_tokens first; if the validation failure rate is high, the business rules in the prompt aren't clearly defined; if retry counts rise, it's usually because the model version changed or the input distribution shifted. These three metrics are far more useful than "it feels less accurate lately."
5. Performance Comparison and Technology Selection: JSON Mode, JSON Schema, or Function Calling
For extraction tasks in production, prefer JSON Schema; use Function Calling only when you want the model to decide whether to call a tool.
| Approach | What it guarantees | Extra overhead | Suitable scenarios |
|---|---|---|---|
| Plain text + custom parsing | Guarantees nothing | None | One-off scripts, toy demos |
| JSON mode | The output as a whole is valid JSON | Very small | Older models; compatible services that only support json_object |
| JSON Schema | Field names, types, enums, required fields | The schema consumes input tokens; decoding is slightly slower | Production extraction; the preferred choice |
| Function Calling | Tool parameter structure, and the model decides whether to call a tool | Tool definitions consume tokens | Agents, where the model needs to choose actions autonomously |
The bulk of the latency is always output tokens and network round trips. The overhead from constrained decoding is barely perceptible on outputs of a few hundred tokens, like ticket extraction. What really sends costs out of control is retries: one failed retry means double the tokens and double the latency. So get the schema right the first time, and list every validation rule up front.
Schemas aren't free. They ride along with every request — the more fields and the deeper the nesting, the higher your input token count. Keeping it to ten fields per DTO and no more than two levels of nesting is a comfortable range. Beyond that, consider splitting the call rather than cramming everything in.
Choosing is simple:
- Internal systems doing information extraction, classification, or scoring: JSON Schema + Java validation, no hesitation.
- Hooking up to a third-party OpenAI-compatible service: first probe whether it supports
response_format; if not, fall back to JSON mode — and not a single line of validation logic can be skipped. - The model needs to decide which tool to call based on user intent: Function Calling. At heart it's also structured output, except what gets structured is "actions."
- You just want nicer-looking logs: don't bother — regex extraction of key fields is enough.
6. Summary
Structured output comes down to three sentences:
Shape goes to the schema. Use ResponseFormat with JsonSchema to push the constraints down to the protocol layer; returning a POJO from AiServices auto-generates the schema, but make sure the model actually declares the capability.
Facts go to validation. The schema guarantees amount is a number; it can't guarantee it's a refund amount. Order number format, amount ranges, enum whitelists, cross-field relationships — hard-code all of it in Java.
Failures go to bounded retries. Feed back to the model exactly which rule failed, at most two or three times, and log the raw output. Past that, give up — don't burn tokens out of spite.
The implementation checklist is six items: narrow the DTO, turn on the schema, think through strict mode, always write validation, cap the retries, keep the raw logs complete.
When should you not use structured output? Open-ended copywriting, long-form summarization, brainstorming. These tasks have no stable fields; forcing a schema on them just ties the model's hands and leaves the output flat. Structured output is good for "moving information around," not for "creating content."
One more note for anyone about to ship: build a regression set from twenty real tickets and run it every time you swap models, tweak prompts, or touch the schema. Models aren't compilers — an upgrade won't tell you what's incompatible.
The model writes the answers; the code grades the paper. If the grading rules aren't clear, no model — however large — will save you.
Top comments (0)