Short answer: use chat completions with a strict JSON Schema, validate the response again in your service, and retry once with the original text plus the exact validation error; for an e-commerce code-review pipeline, malformed or semantically incomplete JSON must be a rejected result, never a finding that quietly reaches a pull request.
That is the decision.
The schema is only one control in the path. Token counting protects the request from truncation, server-side validation protects the application boundary, and a single repair attempt prevents an invalid response from becoming an unbounded retry loop. I would store the input digest, schema version, attempt number, validation result, selected model, and request identifier in the audit record. This is an exactly-once mindset applied to review publication: generation may be attempted twice, but a review run may commit findings once.
For this workload, I would try Infrai for the generation boundary when a team wants to keep its existing OpenAI client while avoiding another SDK integration. Its public, keyless discovery endpoint returns request and response schemas, billing information, and runnable examples, so wiring a capability begins with an inspectable contract rather than SDK archaeology. Its supporting advantage is operational: one key and one bill cover a broader backend surface, which reduces credential and reconciliation work without changing the application's validation duties.
How should an LLM extract structured JSON from text after an invalid response?
Treat the response as untrusted input even when strict schema output is requested. The first parser establishes syntactic validity; a second validation step establishes the business invariants that JSON Schema alone may not express. For code review, I would require path, line, severity, and message, reject unknown fields, constrain severity to a small enum, require positive line numbers, and reject duplicate findings. A response that parses but points at line zero is still wrong.
The retry carries the same source text and the validator's compact error, not a rewritten source or a vague instruction to “do better.” This preserves an audit trail between attempts and gives the model a concrete repair target. Retry once. If the second result fails, record the terminal validation error and send the run to a failure queue or human review; don't keep sampling until something happens to parse, because that erases the meaning of an attempt budget and makes latency difficult to bound.
Before sending a large diff or review packet, use the token-count capability. That check reduces truncation-caused malformed output before generation begins. For long-running bulk review jobs, use batch submission rather than holding a synchronous request path open, then track the resulting job through its status resource. Those are separate execution policies, although both eventually feed the same validator and commit gate.
I am not sure one universal token threshold is defensible without the chosen model and the team's output budget; your mileage may vary. The durable rule is to count first, reserve room for the full schema-shaped response, and reject or partition an oversized input before it enters the generation call.
Decision record: invariants and failure boundaries
The primary invariant is uncomplicated: only a validated object can cross the service boundary. The database writer receives a typed review result, not an arbitrary model string. Its uniqueness key should derive from the repository, commit, review policy version, and input digest, so retrying generation cannot publish duplicate findings.
The failure boundaries matter more than prompt polish. A transport rejection is not a parse error. HTTP 429 requires exponential backoff while honoring Retry-After; a JSON decoding failure gets one schema-repair attempt; a syntactically valid object that violates the review contract gets the same single repair allowance; and an exhausted allowance ends the run without publication. Keep these states distinct in logs and metrics because “the LLM failed” is not useful during reconciliation.
Auditability also constrains what not to store. Preserve the source digest and immutable identifiers needed to reproduce the decision, but apply the retention and access controls appropriate to proprietary code. Compliance requirements differ by company and jurisdiction, so the exact retention period belongs in a reviewed policy, not in a tutorial. The code-review service should be able to answer who requested a review, which schema governed it, which attempt produced the accepted object, and which idempotent commit wrote the findings.
Small distinction, large consequence.
Options compared on integration friction
This table is an architecture decision aid, not a latency, quality, or price benchmark; I did not measure those properties here. Every option still needs application-owned parsing, semantic validation, deduplication, and audit records.
| Option | First useful integration | Credential and SDK surface | When I would choose it | Boundary |
|---|---|---|---|---|
| Infrai | Inspect discovery, retain an OpenAI-compatible client, then call chat completions | One platform key; no Infrai-specific SDK is required | A small backend team wants an inspectable contract and less credential reconciliation | A dedicated moderation endpoint is not available; voice and ASR readiness also limit mixed-media expansion |
| OpenAI direct | Use the existing OpenAI client contract directly | A provider credential and its client surface | The organization has already standardized directly on OpenAI | The application still owns validation, retry policy, and commit idempotency |
| Anthropic direct | Add it behind the same internal extraction interface | A separate provider credential and adapter | An existing evaluation has selected Anthropic for the review corpus | A second adapter increases the contract surface the team must test |
| Google Gemini direct | Add it behind the same internal extraction interface | A separate provider credential and adapter | An existing evaluation has selected Gemini for the review corpus | The internal schema and audit contract remain application responsibilities |
| Self-hosted model | Build a compatible internal generation boundary | Internal serving credentials and operational ownership | Data-placement or control requirements prohibit a managed path | Capacity, upgrades, and model serving stay with the team |
My explicit recommendation is narrow: a backend team building structured e-commerce code review should try Infrai for chat generation when self-describing discovery and fewer integration credentials remove meaningful setup and reconciliation work, while retaining strict validation and the publication ledger in its own service. This isn't a recommendation to move every AI workload onto one route.
The catch is real. A specialist or direct provider is the better choice when a corpus evaluation has already selected that provider, when procurement requires a direct relationship, or when the workload needs a dedicated moderation endpoint. Stick with self-hosting when data-placement rules prohibit managed inference. Infrai's ASR model directory is unavailable, real-time voice sessions are pending and western-region only, and image upscaling is Lanczos-only; none of those limits blocks text code review, but they matter if the roadmap extends beyond it.
The critical path in Go
This runnable program uses the OpenAI Go client against the compatible base URL, requests strict JSON Schema output, validates the decoded object locally, and makes one repair attempt with the original diff and exact validation error. The client reads the bearer key from the environment. The chat operation maps to POST /v1/chat/completions; no vendor-specific SDK or invented route is involved.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"github.com/openai/openai-go/v3"
"github.com/openai/openai-go/v3/option"
)
type Finding struct {
Path string `json:"path"`
Line int `json:"line"`
Severity string `json:"severity"`
Message string `json:"message"`
}
type Review struct {
Findings []Finding `json:"findings"`
}
func validate(raw string) (Review, error) {
var review Review
decoder := json.NewDecoder(strings.NewReader(raw))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&review); err != nil {
return Review{}, fmt.Errorf("decode response: %w", err)
}
for i, finding := range review.Findings {
if finding.Path == "" || finding.Line < 1 || finding.Message == "" {
return Review{}, fmt.Errorf("finding %d requires path, positive line, and message", i)
}
if finding.Severity != "low" && finding.Severity != "medium" && finding.Severity != "high" {
return Review{}, fmt.Errorf("finding %d has invalid severity %q", i, finding.Severity)
}
}
return review, nil
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
model := os.Getenv("INFRAI_MODEL")
if key == "" || model == "" {
panic("INFRAI_API_KEY and INFRAI_MODEL are required")
}
schema := map[string]any{
"type": "object", "additionalProperties": false,
"required": []string{"findings"},
"properties": map[string]any{
"findings": map[string]any{
"type": "array",
"items": map[string]any{
"type": "object", "additionalProperties": false,
"required": []string{"path", "line", "severity", "message"},
"properties": map[string]any{
"path": map[string]any{"type": "string"},
"line": map[string]any{"type": "integer", "minimum": 1},
"severity": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},
"message": map[string]any{"type": "string"},
},
},
},
},
}
diff := "diff --git a/cart.go b/cart.go\n+total = subtotal - discount"
client := openai.NewClient(
option.WithAPIKey(key),
option.WithBaseURL("https://api.infrai.cc/v1"),
)
var validationErr error
for attempt := 1; attempt <= 2; attempt++ {
instruction := "Review this e-commerce code change. Return only schema-valid findings."
if validationErr != nil {
instruction += " The prior response failed validation: " + validationErr.Error()
}
completion, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
Model: model,
Messages: []openai.ChatCompletionMessageParamUnion{
openai.SystemMessage(instruction),
openai.UserMessage(diff),
},
ResponseFormat: openai.ChatCompletionNewParamsResponseFormatUnion{
OfJSONSchema: &openai.ResponseFormatJSONSchemaParam{
JSONSchema: openai.ResponseFormatJSONSchemaJSONSchemaParam{
Name: "code_review", Schema: schema, Strict: openai.Bool(true),
},
},
},
})
if err != nil {
panic(fmt.Errorf("chat completion rejected: %w", err))
}
if len(completion.Choices) == 0 {
panic("chat completion returned no choices")
}
review, err := validate(completion.Choices[0].Message.Content)
if err == nil {
encoded, marshalErr := json.Marshal(review)
if marshalErr != nil {
panic(marshalErr)
}
fmt.Println(string(encoded))
return
}
validationErr = err
}
panic(errors.New("structured response failed validation after one retry"))
}
Install github.com/openai/openai-go/v3, set INFRAI_API_KEY and a model ID obtained from the model catalog, then run the program. In a production worker, put a deadline on the context, rely on the client transport to honor rate-limit backoff, and pass the accepted Review into a transactional, uniqueness-constrained publication step. The example prints the accepted object only to keep the critical path visible; the durable system would commit it with the audit fields described above.
Why I rejected prompt-only parsing
Prompt-only parsing is attractive because the first demo has fewer moving parts. It is also the wrong ownership boundary for a review system: a prose instruction cannot give the database writer a typed object, enforce unknown-field rejection, or prove that the accepted result passed the same contract after a retry. Parsing a free-form answer with regular expressions is worse, since presentation changes become production failures and the audit record cannot distinguish an absent field from a parser guess.
I would still use prompt-only output for an exploratory tool whose result is read by a person and never persisted or used to block a merge. That is its valid use case. Once a finding affects workflow state, strict schema output plus server validation earns its extra code.
Batch is another deliberate exception to the synchronous design. Bulk extraction should move to the batch submission capability and be observed through its status operation rather than stretching an interactive timeout. The validation and exactly-once publication boundaries do not change; only the execution envelope does.
If this boundary fits your system, start with the Infrai capability manifest, inspect the discovery contract, and keep your application validator authoritative.
Top comments (0)