CommitBrief renders a code review as cards, JSON schema v1, or a CI exit code — which means the LLM has to hand back structured findings, not prose...
For further actions, you may consider blocking this person and/or reporting abuse
the 'syntax is a lint rule, schema is a type checker' framing is the right way to think about it. we built multi provider routing with a similar validation layer and the prompt only providers (deepseek, mistral, cohere in your case) are where the production incidents live.
what we added beyond the retry once path: detecting whether the failed attempt returned prose vs truncated JSON and branching the recovery prompt differently. uniform retry templates miss that split — confident prose needs a schema reset, partial JSON needs a continue nudge.
how are you handling providers that silently hallucinate the severity field rather than omit it entirely?
Good question — it splits into two cases that need different answers.
Out-of-vocab hallucination ("blocker", "P1", "severe"): caught client-side. severity is a closed five-value enum and the parser validates every finding against it — an unknown value fails the whole parse and rides the same retry-once-then-degrade path as malformed JSON. Deliberately no coercion: we never map "blocker" → "critical" or default to "medium", because severity is load-bearing — --fail-on=critical decides the exit code in pre-commit hooks and CI. A response that fails loudly beats a guessed severity that silently blocks (or fails to block) a commit. For the three native-schema providers the enum is also enforced server-side in the tool/response schema, so in practice this class only shows up on the prompt-only trio — which matches your incident distribution exactly.
In-vocab but miscalibrated ("critical" on a style nit): this is where the type-checker analogy stops — it type-checks fine. Schema can't see it, so we handle it outside the request path: (1) the severity rubric is a fixed prompt block that defines each level by observable consequence ("crash on common input", "should block release") rather than adjectives, and it's injected by the prompt builder — not the user-editable rules file — so it can't be accidentally dropped; (2) calibration is measured offline: our eval fixtures assert a severity floor per expected finding (actual ≥ min_severity), so a provider that inflates severities shows up as an eval regression instead of a production surprise; (3) the default gate is conservative (critical only), which bounds the blast radius of one inflated level.
The prose-vs-truncated-JSON branch on the recovery prompt is a genuinely good idea — our retry currently resends the identical request and lets sampling variance do the work, which is the bluntest possible instrument. Detecting which way the first attempt failed and adapting the prompt accordingly is going on the list. Thanks!
This is a useful distinction: native schema enforcement, JSON syntax mode, and prompt-only output are three different contracts.
I especially like the point that the parser is the real contract. In provider-neutral systems, capability truth needs to be part of the abstraction: record whether the provider gave a native guarantee, whether the harness validated the response, and whether a retry or repair path had to step in.
That makes fallback behavior visible instead of pretending every model supports the same shape equally.
Yeah, that's exactly the right way to frame it — and it's the part I had to learn the hard way.
CommitBrief does record some of this, but less cleanly than your three-signal version. The native-guarantee bit is static: it's known per-provider at config time (the spectrum table), not something I stamp on each response. What I do persist per-response is a format marker — FormatJSON or FormatMarkdownFallback — that gets cached alongside the result, so a degraded review replays as a degrade instead of silently looking like a clean one.
Where your framing is sharper than what I actually have: the retry path collapses into that one marker. A response that parsed on the first try and one that only parsed after the retry both end up as FormatJSON — so "the repair path had to step in" isn't visible after the fact, even though I act on it in the moment (token usage from both attempts is summed). Your version keeps those as three distinct facts, and that's the better contract. It's the difference between "did it end up valid" and "what did it cost us to get there," and the second one is what you actually want when you're comparing models.
The honest reason it's collapsed is that the marker started life as a cache concern, not an observability one. But you're right that in a provider-neutral system the capability truth deserves to be a first-class field, not a side effect of caching.
I’d probably keep
FormatJSON/FormatMarkdownFallbackas the replay-facing result, but add separate run facts for native capability, retry count, repair path used, parse failures, and total repair cost. Then the cached artifact stays simple, while model comparison and incident review still get the evidence they need.That's the split I landed on too. The cached artifact only has to replay deterministically, so the format is all it needs — folding retry count or repair cost into the key would make identical reviews miss the cache. Those belong in sidecar run facts, exactly as you describe: simple cache, rich evidence for model comparison and incident review.
One issue I've hit even after JSON passes syntax validation: numeric fields coming back as strings that Pydantic coerces silently.
severity: "0.8"becomes0.8in default strict=False mode with no error raised. For production monitoring I log the raw response against the validated output and diff the field types -- if coercion starts appearing on fields that were previously clean, that's an early signal the model is drifting from the schema contract before it escalates to malformed JSON. The degrade path masks this drift otherwise, because the output looks correct downstream.Strict unmarshaling saves me here — in Go, "0.8" hitting a float field errors loudly instead of coercing, so it surfaces as a parse failure rather than quiet drift. But your real point holds: type drift is a leading indicator, and the degrade path is exactly what swallows it. Diffing raw-vs-validated field types is going on my list.
The user wants me to write a casual, natural dev.to comment that reacts to one specific point from the article about getting structured JSON from LLM APIs.
Looking at the article and comments, I can react to several points:
I'll write a comment that reacts to one specific point naturally. The Tae Kim comment about Pydantic coercion is interesting - it's a subtle bug that could cause issues. I could also react to the Ollama distinction about format: "json" only constraining syntax not shape.
Let me write something natural and developer-friendly, max 30 words, no markdown, no links, no hashtags.
A good reaction could be to the Ollama point about syntax vs shape - that's a key distinction that could trip people up.
That Ollama distinction trips people up. "format: json" is really just a lint rule — the model can still give you structurally wrong output that parses fine. The contract is looser than it looks.
Exactly, and the native schema modes only buy you shape, not semantics. Valid JSON can still carry a drifted enum or an out-of-range severity. Syntax is a lint rule, schema is a type checker; the actual contract still needs validating downstream.
The degradation path is the part I spent the most time on in production. When the model ignores the schema entirely, the failure splits into two cases: plausible-looking prose versus malformed partial JSON. Routing those two cases to different recovery prompts rather than a single retry template improved successful parse rate by about 30 percent compared to a uniform fallback.
This is the part I under-built. Right now it's retry-once-then-degrade with a single template, which treats confident prose and truncated JSON as the same failure — they aren't. Prose needs a "you ignored the schema" reset; partial JSON needs a repair/continue nudge. A 30% lift easily justifies branching the recovery. Stealing this.
Great post! I'm dealing with a similar pain — LLMs ignore the requested structure when I ask for fertilizer dosage recommendations. In my RAG project (grounded horticulture assistant), I took a two-stage verification approach: