Short answer: to extract structured JSON from text with an LLM, parse the complete response once, validate it against a narrow review contract, retry only correctable failures, and charge every attempt to the same property-management tenant.
That decision rule matters more than the prompt wording. A code-review service can return syntactically valid JSON that is still unsafe to publish: a finding may omit its file, invent a severity, attach a line outside the patch, or silently bill the wrong tenant after a retry. The boundary has to protect meaning and attribution, not merely make JSON.parse stop throwing.
For a property manager, one reviewed change may belong to a leasing portal, another to a maintenance workflow, and a third to an access-control integration. Shared infrastructure is reasonable. Shared, unattributed usage isn't. The request must carry an immutable tenant identifier through generation, validation, retries, storage, and cost reporting.
Why must every rejected response remain visible to its tenant?
Start with the smallest finding that a human reviewer can act on. A useful record needs a stable finding ID, a file path, a line number, a bounded severity, a concise explanation, and a suggested correction. It should also include the tenant ID supplied by the application, but the model must not be trusted to choose or rewrite that value. Attach tenancy outside the generated payload.
The same rule applies to cost dimensions. Record the model invocation, attempt number, input and output usage, latency, validation result, and terminal outcome beside the application-owned tenant ID. If a malformed response triggers a second call, that call belongs to the original tenant and review job. Losing that link creates a quiet accounting error even when the eventual finding is perfect. Consider one review job with two attempts: attempt 0 produces text that fails JSON parsing, while attempt 1 produces an accepted object. A ledger populated only after validation shows one successful call; the runtime actually performed two. A ledger populated from a model-supplied tenant field is worse, because the rejected candidate may not contain that field at all. The durable record therefore has to be opened from trusted request context before generation starts, updated for each attempt, and closed with an outcome even when no finding is stored. This isn't bookkeeping added after the feature. It is part of the extraction boundary.
Use a strict shape for the generated portion. The following Python dictionary shows the contract; a Node.js service can enforce the same object shape at its parsing boundary.
FINDINGS_SCHEMA = {
"type": "object",
"additionalProperties": False,
"required": ["findings"],
"properties": {
"findings": {
"type": "array",
"maxItems": 20,
"items": {
"type": "object",
"additionalProperties": False,
"required": [
"finding_id",
"path",
"line",
"severity",
"explanation",
"suggestion",
],
"properties": {
"finding_id": {"type": "string", "minLength": 1},
"path": {"type": "string", "minLength": 1},
"line": {"type": "integer", "minimum": 1},
"severity": {
"type": "string",
"enum": ["low", "medium", "high"],
},
"explanation": {"type": "string", "minLength": 1},
"suggestion": {"type": "string", "minLength": 1},
},
},
}
},
}
This is deliberately boring.
Good.
Schema validation still cannot prove that line: 412 exists in the submitted diff, that the path was part of the change, or that two findings are distinct. Those are application invariants. Validate them after structural validation and before persistence. Keep the patch manifest next to the request so those checks do not depend on another model call.
An edge-case hunter should test empty finding arrays, deleted files, renamed paths, very large line numbers, duplicate IDs, unexpected keys, Unicode in paths, and an explanation that contains text resembling JSON delimiters. Compliance also changes the contract: don't send resident names, phone numbers, email addresses, access codes, or raw maintenance notes when a minimized diff is enough for review. Redaction belongs before generation, while retention policy belongs around both raw responses and accepted findings.
How should Node.js extract structured JSON from LLM text after a parse error?
Separate four outcomes: transport failure, JSON parse failure, schema failure, and domain failure. They answer different questions. A parse failure means there is no JSON value to validate. A schema failure means a value exists but violates the declared shape. A domain failure means the shape is valid but the content cannot describe this patch. Collapsing all three into invalid_response makes retry policy, alerts, and tenant cost reports nearly useless.
If the response arrives as a stream, buffer the complete structured payload before parsing. Server-Sent Events are a one-way server-to-client transport with named events and a defined event-stream format; an event boundary is not automatically a complete JSON document. Parsing each fragment invites failures caused by framing rather than generation. Record transport metadata separately, then hand one completed candidate to the parser.
Don't repair generated JSON with regular expressions. Stripping fences, replacing quotes, or deleting trailing commas can turn a detectable contract violation into accepted but altered data. It also obscures which bytes the model returned. Preserve the raw candidate under the applicable retention rules, produce a small diagnostic such as parse_error, and let the retry layer decide whether another attempt is justified.
In Node.js terms, JSON.parse is only the first gate. The example below is Python because the boundary is language-independent: parse once, validate once, then run deterministic patch checks. The adapter that calls the model is intentionally outside this function.
import json
from dataclasses import dataclass
from typing import Any, Callable
@dataclass(frozen=True)
class ReviewContext:
tenant_id: str
review_id: str
changed_lines: dict[str, set[int]]
class CandidateError(ValueError):
def __init__(self, code: str, detail: str) -> None:
super().__init__(detail)
self.code = code
def accept_candidate(
raw: str,
context: ReviewContext,
validate_schema: Callable[[dict[str, Any]], None],
) -> dict[str, Any]:
try:
payload = json.loads(raw)
except json.JSONDecodeError as error:
raise CandidateError("parse_error", str(error)) from error
try:
validate_schema(payload)
except ValueError as error:
raise CandidateError("schema_error", str(error)) from error
seen_ids: set[str] = set()
for finding in payload["findings"]:
path = finding["path"]
line = finding["line"]
finding_id = finding["finding_id"]
if path not in context.changed_lines or line not in context.changed_lines[path]:
raise CandidateError("domain_error", "finding is outside the submitted patch")
if finding_id in seen_ids:
raise CandidateError("domain_error", "finding_id must be unique")
seen_ids.add(finding_id)
return {
"tenant_id": context.tenant_id,
"review_id": context.review_id,
"findings": payload["findings"],
}
Notice where tenant_id comes from. It is not in the model-owned schema and cannot drift during regeneration. This is the same defensive instinct used in OTP systems: the destination, rate-limit key, and billing owner must survive every asynchronous handoff. I've learned to distrust any retry path that reconstructs identity from a payload it did not author.
The review contract needs two validation layers
Retry one malformed or schema-invalid generation only when the second prompt includes a compact description of the failed constraint. Do not paste sensitive raw content into an error message, and do not keep retrying deterministic domain violations such as a line outside the patch. The model has no new evidence with which to correct that claim. Structural validation answers whether the candidate has the allowed keys and types; domain validation answers whether those values can be true for this particular change. Keeping those layers separate makes an alert actionable and prevents a prompt adjustment from being mistaken for a fix to missing repository context.
This is where many otherwise tidy implementations lose per-tenant visibility. Attempt zero is written to the usage ledger, attempt one succeeds, and only the successful attempt is joined to the review. The dashboard then undercounts the tenant whose input caused more work. Write an attempt record for every call, including rejected output, and aggregate by the application-owned tenant ID rather than by accepted findings.
from collections.abc import Callable
from typing import Any
def run_review(
context: ReviewContext,
generate: Callable[[str | None], tuple[str, dict[str, int]]],
validate_schema: Callable[[dict[str, Any]], None],
record_attempt: Callable[..., None],
) -> dict[str, Any]:
correction: str | None = None
for attempt in range(2):
raw, usage = generate(correction)
try:
accepted = accept_candidate(raw, context, validate_schema)
except CandidateError as error:
record_attempt(
tenant_id=context.tenant_id,
review_id=context.review_id,
attempt=attempt,
usage=usage,
outcome=error.code,
)
if attempt == 0 and error.code in {"parse_error", "schema_error"}:
correction = f"Return one object that satisfies the schema. Failure: {error.code}."
continue
raise
record_attempt(
tenant_id=context.tenant_id,
review_id=context.review_id,
attempt=attempt,
usage=usage,
outcome="accepted",
)
return accepted
raise RuntimeError("review ended without an accepted candidate")
The ledger writer should be idempotent on (tenant_id, review_id, attempt). That prevents a queue redelivery from recording the same model call twice. Whether usage is measured as tokens, characters, or provider units depends on the runtime contract; I'm not sure a single normalized unit is honest across every deployment. Preserve the provider's raw usage fields, then define an internal monetary allocation separately. That decision makes later reconciliation possible without rewriting history.
Retries need a second limit outside generation too. A transport retry caused by a transient client-side interruption and a content retry caused by invalid JSON must not share an unbounded counter. Keep both small, report them independently, and stop once the review can no longer meet its latency or cost budget. Fast failure is preferable to a finding that arrives after the pull request has changed underneath it.
Put retry budgets inside the usage ledger
Compare approaches after defining those constraints. A strict schema-capable generation path reduces the space of possible outputs, but it does not replace domain checks or tenant accounting. A plain-text path plus a parser offers broader runtime compatibility, but it leaves more malformed candidates for the application to reject. A self-hosted inference path may offer tighter data placement control, while shifting model operations and capacity planning onto the team. None removes the need for an application-owned ledger — the retry is billable work even when its output is discarded.
| Approach | Useful when | The catch |
|---|---|---|
| Schema-constrained generation | The runtime can enforce the exact review shape | Domain truth still requires patch-aware checks |
| Plain text plus strict parsing | Runtime portability matters more than rejection rate | More invalid candidates may consume the retry budget |
| Self-hosted inference | Data placement and operational control dominate | The team owns capacity, upgrades, and model behavior |
Do not add vector search merely to solve malformed output. A Postgres vector extension addresses vector similarity; structured response validation is a deterministic boundary. Retrieval can help supply repository context, but its results still enter the same minimized prompt and its output still faces the same schema and patch checks.
The unsuitable case is equally important. Automated findings should not be the final authority for access-control, rent calculation, resident notification, or compliance-sensitive changes. Route high-severity findings and sensitive modules to a human reviewer. Stick with deterministic static analysis when the rule can be expressed exactly, and skip generation entirely when code cannot leave the required trust boundary.
Roll out tenant visibility before automation
Begin in shadow mode: generate and validate findings, record every attempt, but do not publish comments. Compare accepted findings with human review, inspect rejection categories by tenant, and verify that ledger totals reconcile with invocation records. Then enable publication for low-risk repositories or modules while retaining human approval for sensitive paths.
Keep the rollout compact. Version the schema, store that version on every attempt, alert on changes in parse, schema, and domain failure rates, and sample accepted findings for semantic review. A rising parse-error rate calls for transport and generation inspection; a rising domain-error rate points toward context selection or an overly permissive contract. They are different incidents.
Per-tenant cost visibility is the release gate. If an operator cannot trace a published finding through its rejected attempts, usage records, schema version, and original review job without reading model prose, the system is not ready to comment on production code.
Top comments (0)