Short answer: fix LLM JSON schema failures in a Node.js code-review extractor by separating missing fields from invalid values. Make the expected fields explicit, allow null values only where the diff can genuinely omit a fact, validate enum output locally, and give a semantic extraction repair one bounded attempt.
That approach is less exciting than adding another paragraph to an extraction prompt. It is also much easier to test. For a B2B SaaS review service, the output should be stable enough for a dashboard, a ticket, and an eval harness even when the changed code says nothing about a deadline or a severity.
The first red flag is a valid JSON object
Start with a failure matrix, not another prompt variant. A parseable object can still lose a reviewer-visible fact; a missing key, an explicit null, and an invalid enum each require a different response. Treating them as one generic \"bad JSON\" bucket makes the eval result useless.
For a B2B SaaS code-review service, classify each rejected finding as shape, evidence, taxonomy, or transport. Shape means the object violates its declared keys. Evidence means the diff does not establish a value. Taxonomy means the source phrase does not fit the enum. Transport covers timeouts and rate limits. Only the first three belong to extraction logic.
Consider a review comment saying that a new cache branch skips an authorization check. It may identify auth.py, but not a precise line after surrounding code has been quoted or reformatted. The honest output keeps the finding, records line: null, and preserves the summary. A separate comment that says \"this could become an outage during a deploy\" should not be promoted to high unless the team has defined that mapping. The number in a ticket ID is not an account number, a line number, or evidence of severity. Small distinctions. Big downstream effects.
The eval corpus should contain those cases before prompt tuning begins: one complete finding, one with no line reference, one with risk but no assigned severity, one distractor number, and one unfamiliar phrase. Record the expected JSON and the reason for each null. Track valid-object rate, field accuracy, placeholder rate, enum-confusion rate, repair rate, and input/output tokens separately. One aggregate score can hide a prompt that improves summaries while inventing line numbers.
Keep it boring.
How do LLM JSON schema rules handle missing fields and null values?
Keep the flow deliberately boring: send the diff and review context, request the declared JSON shape, parse it, validate it locally, and make one repair request only when the validation result tells you what is wrong. The retry should receive the original source and the concrete validation message. It should not receive a long transcript of earlier guesses.
The example below shows the contract boundary in Python. The same boundary can sit behind a Node.js adapter; keeping provider calls outside validate_finding lets the application swap transports without changing its business rules. The model call is represented by a function argument so the example does not invent a provider route or SDK behavior.
from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Callable
SEVERITIES = {"low", "medium", "high"}
REQUIRED_KEYS = {"file", "line", "severity", "summary"}
@dataclass(frozen=True)
class ValidationResult:
message: str | None
def validate_finding(value: Any) -> ValidationResult:
if not isinstance(value, dict):
return ValidationResult("finding must be a JSON object")
if set(value) != REQUIRED_KEYS:
return ValidationResult("finding must contain exactly file, line, severity, and summary")
if not isinstance(value["file"], str) or not value["file"].strip():
return ValidationResult("file must be a non-empty string")
if value["line"] is not None and (
not isinstance(value["line"], int) or value["line"] < 1
):
return ValidationResult("line must be a positive integer or null")
if value["severity"] is not None and value["severity"] not in SEVERITIES:
return ValidationResult("severity must be low, medium, high, or null")
if not isinstance(value["summary"], str) or not value["summary"].strip():
return ValidationResult("summary must be a non-empty string")
return ValidationResult(None)
def extract_finding(
source_text: str,
request_json: Callable[[str], dict[str, Any]],
) -> dict[str, Any]:
instruction = (
"Extract one code-review finding as JSON. Include every required key. "
"Use null for a line or severity that the source does not establish; do not guess.\n\n"
+ source_text
)
first = request_json(instruction)
result = validate_finding(first)
if result.message is None:
return first
repair = (
"Return corrected JSON only. Preserve facts from the original source, and use null "
"for an unavailable line or severity. Validation error: "
+ result.message
+ "\n\nOriginal source:\n"
+ source_text
)
second = request_json(repair)
result = validate_finding(second)
if result.message is not None:
raise ValueError("finding violates the contract: " + result.message)
return second
This code catches shape and semantic errors separately from transport errors. A timeout, authentication failure, or rate limit belongs in the provider adapter, with its own bounded retry policy. A response that parses as JSON but contains "severity": "critical-ish" is an extraction failure; silently converting it to high hides taxonomy drift.
Turn the diagnosis into a schema
Missing keys usually point to one of four causes: the schema did not mark the key as required, the prompt allowed omission, the source did not contain the fact, or the parser accepted a partial object. Check those in that order. Adding examples cannot repair a contract that permits the wrong shape. I've found that this classification is more useful than counting all malformed responses together, because each cause changes a different layer of the system.
The object can always contain file, line, severity, and summary; line or severity may still be null when the source does not establish them. A missing key and a present key with null are different signals, so choose one policy and enforce it consistently. Reject empty summaries, placeholder strings such as unknown, and unexpected keys locally.
Null values are different. A null rate that rises on short diffs may be correct because those diffs contain less evidence. A null rate that rises after a schema change can indicate that the application has made a field mandatory in name but impossible to infer in practice. Keep both cases in the eval corpus.
Enum mismatch is often a taxonomy problem wearing a prompt-shaped hat. Compare the rejected phrase with the allowed labels. If several phrases are being forced into one label, change the contract or insert a deterministic mapping table. Do not let a repair prompt become an undocumented policy engine.
A 422-style validation report that points to a missing key should carry the original text reference, schema version, and validator message; it should not trigger a long chain of prompt edits that nobody can replay. Enums deserve restraint: if the taxonomy cannot express the source, keep the phrase as text or return a nullable enum and classify it in a separately versioned step.
Portability at the application boundary
Provider portability is a reliability property, not a checkbox in a model catalog. Put provider-specific request construction in one adapter. Keep the domain schema, local validator, retry budget, redaction rules, and eval fixtures independent of that adapter. Then a provider change is an experiment against the same corpus rather than a rewrite of the code-review service.
The trade-off is real. A direct provider integration can expose native structured-output behavior or a model that performs better on your review corpus, but it also adds a separate contract and operational surface. A shared HTTP abstraction can reduce application changes across providers, but the common denominator may hide useful provider-specific controls. Your mileage may vary: portability is valuable only when the evaluation harness proves that the abstraction still preserves finding quality, latency, and token use.
| Approach | Interface | Best fit | Main limitation |
|---|---|---|---|
| Direct provider adapter | SDK or provider REST | A native structured-output feature is required | Provider-specific contract and migration work |
| Shared HTTP adapter | Common REST boundary | Several backends must share application plumbing | Lowest common denominator can hide useful controls |
| Self-hosted adapter | Internal REST or process boundary | Data and deployment controls dominate | The team owns serving, upgrades, and observability |
For a notebook-to-prod path, keep the first version small. Define one adapter interface, one schema version, and one recorded corpus. Add observability for validation reasons, model identifier, latency, and token counts; avoid storing source diffs by default when the log does not need them. When a schema or prompt changes, replay the corpus before deployment.\n\nThe catch is that this architecture is not suitable when a provider-specific feature is a hard requirement and the shared interface cannot represent it. Stick with the direct integration when its native behavior is essential, and isolate that choice behind the same application boundary so the rest of the system remains testable.
Put the operating rule beside the validator
Do not fix every rejected object with a larger prompt. First decide whether the source contains the fact, whether the schema can represent it, and whether the label vocabulary is stable. Then make the smallest change that answers that diagnosis.
Three words: validate the boundary.
Top comments (0)