DEV Community: Muhammet ŞAFAK

Getting structured JSON out of five incompatible LLM APIs — and degrading when they ignore you

Muhammet ŞAFAK — Fri, 26 Jun 2026 00:00:00 +0000

CommitBrief renders a code review as cards, JSON schema v1, or a CI exit code — which means the LLM has to hand back structured findings, not prose. Every provider can do that. The catch is that no two of them do it the same way, and some don't really do it at all.

There's exactly one schema the whole system targets. Getting four native APIs to honor it takes four completely different mechanisms; getting three more is a matter of asking nicely and not trusting the answer. This is how that works, and what happens when a model ignores the contract anyway.

TL;DR

One schema, many dialects. Every provider targets the same Finding shape, expressed through whatever structured-output mechanism that vendor offers — tool_use, strict json_schema, responseSchema, or just format: "json".
Structured output is a spectrum, not a guarantee. It runs from "the API enforces the shape" down to "we asked in the prompt."
The real contract is your parser. One ParseFindings validates every provider's output the same way; failures retry once, then degrade to Markdown with a warning. The pipeline never crashes on a bad response.
The limit. A schema makes output parseable, not correct. It can't stop a model from inventing a plausible-but-wrong finding.

The one schema everyone targets

A finding is a flat struct. Five required fields, three optional, and a severity drawn from a closed vocabulary:

type Finding struct {
    Severity    Severity `json:"severity"`     // one of five, below
    File        string   `json:"file"`
    Line        int      `json:"line"`
    LineEnd     int      `json:"line_end,omitempty"`
    Title       string   `json:"title"`
    Description string   `json:"description"`
    Suggestion  string   `json:"suggestion"`
    Language    string   `json:"language,omitempty"`
    Snippet     string   `json:"snippet,omitempty"`
}

const (
    SeverityCritical Severity = "critical"
    SeverityHigh     Severity = "high"
    SeverityMedium   Severity = "medium"
    SeverityLow      Severity = "low"
    SeverityInfo     Severity = "info"
)

The envelope is {"findings": [ ... ]} and nothing else. That severity vocabulary is the wire contract with the model — deliberately English-only and fixed in code, so a user's custom COMMITBRIEF.md can change the rules of a review but never the shape of its output. Everything downstream — the cards renderer, --json, --fail-on=high — depends on those five strings meaning exactly five things.

Four native dialects for the same shape

The native API providers each enforce that schema through their own mechanism. Same target, four wire formats.

Anthropic — a forced tool call. The findings schema is registered as a tool, and tool_choice makes calling it non-optional:

params.Tools      = []sdk.ToolUnionParam{buildReportTool()}  // schema as "report_findings"
params.ToolChoice = sdk.ToolChoiceParamOfTool(toolName)      // must call it
// tool description: "Emit the review as structured findings. Always call this tool."

OpenAI — strict json_schema. With Strict set, the Chat Completions API holds the response to the schema server-side — and refuses the request outright rather than fall through to a model that would ignore it:

func buildResponseFormat() sdk.ChatCompletionNewParamsResponseFormatUnion {
    return sdk.ChatCompletionNewParamsResponseFormatUnion{
        OfJSONSchema: &shared.ResponseFormatJSONSchemaParam{
            JSONSchema: shared.ResponseFormatJSONSchemaJSONSchemaParam{
                Name:        schemaName,
                Description: sdk.String("Structured findings for a code review."),
                Strict:      sdk.Bool(true),
                Schema:      responseSchema,
            },
        },
    }
}

(Responses-API-only models express the same schema through a text.format json_schema config instead — one more dialect for the identical shape.)

Gemini — a response schema plus a MIME type. You hand the SDK a *Schema value and tell it to return JSON:

cfg.ResponseMIMEType = "application/json"
cfg.ResponseSchema   = responseSchema()  // the Findings envelope as a *genai.Schema

Ollama — format: "json", and that's all it promises. A local model can be told to emit JSON, but the flag constrains syntax, not shape:

Format: "json", // valid JSON guaranteed; the right keys are not

That distinction matters. Anthropic, OpenAI, and Gemini constrain the structure; Ollama only guarantees the output parses as some JSON. The schema conformance has to come from somewhere else.

Three providers that don't enforce at all

DeepSeek, Mistral, and Cohere reach CommitBrief through the OpenAI-compatible SDK (covered in part 2), but their strict-schema support is uneven, so they don't request response_format at all. Their JSON shape comes entirely from the prompt's contract block.

So structured output across the seven API providers is a spectrum:

Mechanism	Constrains	Providers
Forced tool / strict schema	The exact shape	`anthropic`, `openai`, `gemini`
`format: "json"`	Syntax only	`ollama`
Prompt instruction	Nothing, at the API level	`deepseek`, `mistral`, `cohere`

A pipeline that only trusted the strict-schema providers would work for three of seven. The other four need a backstop that doesn't care how the JSON was produced.

The real contract is your parser

That backstop is one function every provider's output funnels through. ParseFindings decodes the envelope and validates each finding — not just "is it JSON" but "is it a valid finding":

for i, f := range env.Findings {
    if !f.Severity.IsValid() {
        return nil, fmt.Errorf("parse findings: finding %d: unknown severity %q", i, f.Severity)
    }
    if f.File == "" {
        return nil, fmt.Errorf("parse findings: finding %d: missing file", i)
    }
    if f.Title == "" {
        return nil, fmt.Errorf("parse findings: finding %d: missing title", i)
    }
    if f.Description == "" { /* ... */ }
    if f.Suggestion == "" { /* ... */ }
}

An empty findings array is a clean review, returned as a non-nil empty slice — success, not an error. A made-up severity or a finding with no file is a parse failure, no matter which provider produced it. The strict-schema providers rarely trip it; the prompt-driven ones lean on it. Either way, the validation is identical, so a --fail-on=high gate means the same thing whether you ran Claude or a local qwen.

When the model ignores all of it

A strict schema reduces malformed output; it doesn't eliminate it, and three of the providers have no schema at all. So the call is wrapped in retry-once-then-degrade:

resp, err := prov.Review(ctx, req)
if err != nil {
    return "", provider.Usage{}, "", err
}
if _, parseErr := render.ParseFindings(resp.Content); parseErr == nil {
    return resp.Content, resp.Usage, cache.FormatJSON, nil
}
// First attempt unparseable — retry once (ADR-0014 §4).
onRetry()
resp2, err2 := prov.Review(ctx, req)
if err2 != nil {
    return resp.Content, resp.Usage, cache.FormatMarkdownFallback, nil
}
if _, parseErr := render.ParseFindings(resp2.Content); parseErr == nil {
    return resp2.Content, totalUsage, cache.FormatJSON, nil
}
// Both attempts failed — degrade: render the raw text as Markdown, warn once.
return resp.Content, totalUsage, cache.FormatMarkdownFallback, nil

Three things in that flow are deliberate. Token usage is summed across both attempts, so the cost footer reflects what you actually spent, even on a degrade. The outcome is recorded as a format marker (FormatJSON or FormatMarkdownFallback) and cached with the response, so a degraded review replays from cache silently instead of re-warning forever. And degrade means render the raw model text as Markdown and print one warning — never crash, never show the user a stack trace because an LLM got creative. A --fail-on gate is skipped on a degrade, with a note on stderr, because there are no structured findings to threshold.

What it is not

Structured output guarantees a response is parseable. It does not guarantee it's correct. A strict schema can't stop a model from inventing a line number, attaching a finding to the wrong file, or reporting a confident non-issue — which is why the prompt still carries an explicit "do not invent file paths or line numbers" directive, and why this is the zeroth reviewer, not the last one. The schema is what makes the output machine-readable; your judgment is what makes it trustworthy.

If you want the measured version of "how often is it right," the eval harness scores precision and false-positive rate per model against a known-answer corpus:

COMMITBRIEF_EVAL_PROVIDER=<name> make eval-live

Repo: github.com/CommitBrief/commitbrief.

Part 3 of **Building CommitBrief. Next: the pre-send secret scanner — eight patterns, added-lines-only, and a match record that never stores the secret it just caught.

One Go interface, ten LLMs, three transport classes

Muhammet ŞAFAK — Thu, 25 Jun 2026 00:00:00 +0000

CommitBrief reviews your git diff with whatever LLM you point it at — Claude, GPT, Gemini, a local Ollama model, or the claude CLI you already have installed. Ten providers, one Provider interface, zero special-casing in the review pipeline. This post is how that abstraction is built, because the providers are far less alike than "they're all LLMs" suggests.

The ten split into three transport classes that share almost nothing at the wire level: native HTTPS APIs, OpenAI-compatible endpoints, and local subprocesses. Making them satisfy one interface — without the pipeline knowing which is which — is the whole trick.

TL;DR

One interface, ten implementations. Every provider satisfies a 7-method Provider interface; the pipeline never type-switches on a vendor.
A database/sql-style registry. Each provider registers itself in init(); a blank import in main.go is all it takes to add one.
Three transport classes. Native APIs, OpenAI-compatible endpoints (reusing one SDK via a base URL), and subprocess-backed CLIs — the last opt out of the JSON contract through a marker interface.
The limit. Provider-agnostic is not provider-equal. A local model is a real review, not a frontier one — and the eval harness measures the gap instead of hiding it.

Key facts

Class	Providers	Transport
Native API	`anthropic`, `openai`, `gemini`, `ollama`	Each vendor's own HTTPS API (Ollama over `localhost`)
OpenAI-compatible	`deepseek`, `mistral`, `cohere`	`openai-go` SDK pointed at the vendor's base URL
CLI-backed	`claude-cli`, `gemini-cli`, `codex-cli`	A local subprocess; reuses the host CLI's auth

The interface every provider satisfies

This is the entire contract. Seven methods, no generics, no per-vendor escape hatch:

type Provider interface {
    Name() string
    DefaultModel() string
    ContextWindow(model string) int
    EstimateTokens(text string) int
    Pricing(model string) Pricing
    Review(ctx context.Context, req Request) (Response, error)
    TestConnection(ctx context.Context) error
}

Review does the work; the other six let the pipeline make decisions before the call — estimate tokens for the cost preflight, look up Pricing to warn over a threshold, check ContextWindow to catch an oversized prompt, and TestConnection so commitbrief providers test <name> can ping a key without running a review. The pipeline holds a Provider and never asks which concrete type it is.

A registry you've already used

If you've ever written _ "github.com/lib/pq" to register a database driver, you know this pattern. Providers register themselves; nothing imports them by name.

type Factory func(cfg config.ProviderConfig) (Provider, error)

func Register(name string, factory Factory) {
    if name == "" {
        panic("provider: Register called with empty name")
    }
    if factory == nil {
        panic("provider: Register called with nil factory for " + name)
    }
    registryMu.Lock()
    defer registryMu.Unlock()
    if _, exists := registry[name]; exists {
        panic("provider: duplicate registration for " + name)
    }
    registry[name] = factory
}

The panics are deliberate. A duplicate name or a nil factory is a programmer error, and it should crash at startup — when an init() runs — not silently shadow a provider that a user later selects. Each provider subpackage calls Register in its own init(), so wiring a new one into the binary is one line:

import (
    _ "github.com/CommitBrief/commitbrief/internal/provider/anthropic"
    _ "github.com/CommitBrief/commitbrief/internal/provider/deepseek"
    _ "github.com/CommitBrief/commitbrief/internal/provider/claude-cli"
    // ...one blank import per provider
)

New(name, cfg) looks the factory up under a read lock and returns a typed error listing the known names when you ask for one that isn't there. That's the seam every transport class plugs into.

Class 1 — native APIs

anthropic, openai, and gemini each talk to their vendor's own SDK and use that vendor's native structured-output mechanism. ollama is the same shape pointed at http://localhost:11434: same interface, but the diff never leaves the machine and the cost is zero. For a contractor under an NDA, that last property is the entire point — commitbrief --provider ollama is a real review with no third-party egress.

(How each vendor is coerced into returning structured findings — tool_use, strict json_schema, responseSchema — is its own story. That's the next post in the series.)

Class 2 — OpenAI-compatible, for the cost of a base URL

DeepSeek, Mistral, and Cohere all expose an OpenAI-shaped Chat Completions API. So instead of three new SDKs, they reuse the one already in the build — github.com/openai/openai-go — pointed at a different host:

func New(cfg config.ProviderConfig) (provider.Provider, error) {
    if cfg.APIKey == "" {
        return nil, fmt.Errorf("deepseek: %w", provider.ErrUnauthorized)
    }
    baseURL := cfg.BaseURL
    if baseURL == "" {
        baseURL = defaultBaseURL // https://api.deepseek.com
    }
    return &Client{
        sdk:   sdk.NewClient(option.WithAPIKey(cfg.APIKey), option.WithBaseURL(baseURL)),
        model: cfg.Model,
    }, nil
}

Three providers, one option.WithBaseURL, no new dependency to license-audit or keep current. Cohere even ships its compatibility surface at api.cohere.ai/compatibility/v1, so it slots in the same way. These three don't request a strict response format — support for it is uneven — so their JSON shape comes from the prompt contract plus a retry-once-then-degrade fallback, the same way Ollama works.

Class 3 — subprocess CLIs, and a marker interface

The third class is the unusual one. claude-cli, gemini-cli, and codex-cli don't make HTTP calls at all — they shell out to a CLI you already have on your PATH and reuse its auth. If you pay for a Claude or Gemini subscription, reviewing a diff through it costs nothing extra.

These three differ only in their binary name and a few flags, so they share one clireview.Backend. claude-cli, for instance, pipes the prompt on stdin:

claude -p - --output-format text

Two details are worth pulling out.

They opt out of the JSON contract via a marker interface. A CLI tool has already formatted its output; forcing it through JSON parsing and the cards renderer would mangle it. So the backend implements a marker:

// PlainTextEmitter is the marker interface for providers whose
// Review() returns formatted plain text instead of structured JSON.
type PlainTextEmitter interface {
    Provider
    EmitsPlainText()
}

func (b *Backend) EmitsPlainText() {} // the whole implementation

The pipeline asks once whether the provider has the capability — the same type-assertion idiom as http.Flusher or io.WriterTo — and reuses the answer:

// claude-cli / gemini-cli / codex-cli get the plain-text prompt
// contract instead of the JSON one — the host CLI's agentic system
// prompt makes structured-output guarantees unreliable.
_, plainText := prov.(provider.PlainTextEmitter)

var p prompt.Prompt
if plainText {
    p = prompt.BuildPlainText(loaded, app.Lang, numberedDiff, archContext, global.withContext)
} else {
    p = prompt.Build(loaded, app.Lang, numberedDiff, archContext)
}

That single plainText bool then drives the few places the two paths diverge: which prompt contract to build, whether to run the deterministic flaky-test pre-pass (skipped for CLI tools), and whether to stream the response verbatim or parse it as JSON. There's no switch over provider names anywhere in the pipeline — routing is on the capability, not the identity. A new plain-text provider implements EmitsPlainText() and inherits all of it.

DefaultModel() returns the binary's version, on purpose. The cache key includes the model string. A CLI provider has no model name in the API sense, so it reports binary + detected version — queried once with --version and memoized behind a sync.Once, because DefaultModel() is on the hot path of every cache-key computation:

func (b *Backend) DefaultModel() string {
    if v := b.versionOrEmpty(); v != "" {
        return b.spec.Binary + " " + v
    }
    return b.spec.Binary
}

func (b *Backend) ContextWindow(string) int { return 200_000 } // informational

When you upgrade the host CLI, the version string changes, so cached reviews from the old version cleanly invalidate. Correctness falls out of the cache key for free.

Choosing one

Selection is explicit — one provider per run, picked by flag or config:

commitbrief --provider openai          # override the configured provider
commitbrief --cli claude               # shorthand for --provider claude-cli
commitbrief providers use ollama --local   # set the default for this repo
commitbrief providers test deepseek    # ping the key, no review

A few combinations are rejected on purpose: --provider and --cli are mutually exclusive, and --cli can't pair with --json or --markdown because a plain-text provider doesn't produce the structured output those formats render.

What it is not

Provider-agnostic is not provider-equal. A local qwen2.5-coder review is a real second pass and beats no review, but it won't match a frontier model on subtle findings. CommitBrief doesn't paper over that — the eval harness scores each model against a known-answer corpus so you can see the gap yourself:

COMMITBRIEF_EVAL_PROVIDER=<name> make eval-live

And there's no automatic vendor-to-vendor failover. CommitBrief talks to exactly one provider per run; switching is a flag or a config write, never a silent retry against a different company's API on your diff. That's a deliberate choice about who decides where your code goes — you, every time.

Part 2 of **Building CommitBrief. Next: getting structured findings out of five incompatible LLM APIs — tool_use, strict json_schema, responseSchema, prompt-driven JSON — and degrading gracefully when the model ignores all of them.

I built a local-first LLM code reviewer in Go. Here's the entire pipeline.

Muhammet ŞAFAK — Wed, 24 Jun 2026 04:40:00 +0000

CommitBrief is a local-first CLI that runs an LLM review over your git diff before a teammate — or your future self — sees it. There's no server and no telemetry; the diff leaves your machine only for the provider you chose, and with a local model like Ollama it never leaves at all.

The interesting engineering isn't "call an LLM." It's everything that has to happen around that call so the review stays cheap, safe, and reproducible. Here's the whole path from commitbrief --staged to the findings on your screen.

TL;DR

What it is — a CLI that reviews your staged diff (or any git diff range) with the provider you pick: Claude, GPT, Gemini, or a fully local Ollama model.
The non-obvious part — the LLM call is one stage out of fourteen. Filtering, a pre-send secret scan, content-addressed caching, and a cost preflight do most of the work.
The limit — it's the zeroth reviewer, not a replacement for a human one.

Key facts

Go 1.25, GPL-3.0-or-later, no hosted service.
10 providers + a mock: Anthropic, OpenAI, Gemini, Ollama (native APIs); DeepSeek, Mistral, Cohere (OpenAI-compatible); claude-cli, gemini-cli, codex-cli (subprocess-backed).
The review path is read-only. The single git-write command is commitbrief commit, and even that only runs one git commit of already-staged changes — it never edits a file.
Install: brew install CommitBrief/tap/commitbrief, scoop install commitbrief, or go install github.com/CommitBrief/commitbrief/cmd/commitbrief@latest.

The shape of a review

Every review walks one linear pipeline. Here it is at altitude before we zoom in:

Stage	What happens	Why it's here
1. Resolve context	Walk up for `.git`, merge config (built-in < global < repo), apply env + flags	One deterministic config per run
2. Load rules	`./COMMITBRIEF.md` or the embedded default; validate the output template first	Fail on a broken template before spending a token
3. Acquire diff	Hybrid go-git + `exec git` fallback	Worktree state is git's, not a reimplementation's
4. Parse + filter	Three ignore layers, then an optional allowlist	Don't pay to review lock files
5. Pre-send guard	Refuse to leak `.commitbrief/**`; scan for secrets	The diff is about to leave the machine
6. Build prompt	Four XML blocks + an immutability guard	Structured and injection-resistant
7. Cache lookup	SHA-256 of the exact inputs	A re-run is a disk read, not a bill
8. Cost preflight	Estimate tokens, warn over a threshold	No surprise spend
9. Provider call	Structured JSON, or verbatim text for CLI providers	The actual review
10. Render + gate	Cards / JSON / Markdown, then `--fail-on`	Human output or a CI exit code

Five of these carry most of the weight. Let's take them in order.

Getting the diff: go-git, with git as the source of truth

You'd think reading a diff is trivial. It is — until you need staged-vs-unstaged, a worktree comparison, and git diff main...feature to all behave exactly like git, on Windows too.

CommitBrief runs a hybrid: a primary go-git implementation with a git CLI fallback (ADR-0002). Range operations that go-git models cleanly — commit-vs-first-parent, merge-base range diffs, branch diffs — stay in-process. Staged, unstaged, and arbitrary git diff <args> passthrough shell out to git with --no-color --no-ext-diff for stable parsing. The CLI stays the source of truth for index and worktree state; reimplementing that plumbing is exactly the kind of subtle drift you don't want under a review tool.

commitbrief --staged                 # the index
commitbrief --unstaged               # the working tree
commitbrief diff main...feature      # args forwarded verbatim to git diff

Filtering: three layers, so the model never reads noise

A diff is mostly signal and a pile of things no reviewer should read. Filtering is three composed layers (ADR-0006):

Built-ins — around 65 patterns: lock files, vendor/**, node_modules/**, generated code, build artifacts, binaries, IDE/OS noise, and .commitbrief/cache/**.
.commitbriefignore — gitignore syntax, repo-root, team-shared. It composes after the built-ins with last-match-wins, so a !pattern line can re-include something a built-in dropped.
Semantic prose — natural-language exclusions in COMMITBRIEF.md ("don't flag generated mocks"), applied by the model itself.

After the ignore layers, --file / --dir apply a narrower allowlist. If nothing survives, the run exits 0 having spent nothing — an empty diff is a success, not an error.

The guard nobody else runs: don't send secrets to the model

Stage 5 is the one I care about most, because it sits on the boundary where your code is about to leave the machine.

Two checks run before the provider call. First, a guard refuses to quietly ship your own config: if any path in the diff starts with .commitbrief/, it prompts, and auto-aborts when there's no TTY. Second, a secret scanner runs over added lines only against eight built-in patterns — AWS access keys, GitHub/GitLab tokens, Anthropic/OpenAI keys, JWTs, Stripe live keys, PEM private keys — and you can add your own through guard.secret_patterns.

One detail I'm proud of: a match records only {Line, Patterns} — never the matched substring. The scanner that exists to stop a leak can't itself become one through a log line, stderr, or a cache file.

The bypass policy is deliberate, too. --allow-secrets skips the scan; --yes does not. Auto-confirming prompts in a pipeline should never silently approve shipping a credential to a third party.

Caching is just content addressing

Reviewing the same diff twice should be free. The cache key is a SHA-256 over the exact inputs that could change the answer:

sha256( diff "::" systemPrompt "::" provider ":" model ":" lang ":" schemaVersion [":ctx"] [":mode:"+mode] )

Every input earns its place. The fully-assembled system prompt is in the key, so editing COMMITBRIEF.md invalidates stale reviews. The schema version is in the key, so bumping the output contract invalidates everything at once. --with-context and the commit-message mode each get a suffix so they never collide with a plain review. Entries are written atomically — temp file, 0600, then rename — to .commitbrief/cache/<key>.json: one file per response, no index.

The payoff lands at stage 8. Cost preflight estimates input tokens with a conservative (len+3)/4 heuristic, clamps expected output to [200, 1500] tokens, multiplies by a per-model price table, and prompts only when the estimate clears your cost.warn_threshold_usd (default $0.50). A cache hit skips preflight entirely — re-running a review on an unchanged diff costs one disk read.

The call, and what happens when the model misbehaves

For API providers, CommitBrief asks for structured findings and parses them into a fixed contract — severity, file, line, title, description, suggestion — emitted as JSON schema v1. If the model returns something unparseable, it retries once; if it's still bad, it degrades to rendering the raw text as Markdown and prints a warning instead of crashing. CLI-backed providers (claude-cli, gemini-cli, codex-cli) run as read-only subprocesses and stream their text verbatim.

Output is Cards by default, or:

commitbrief --json --fail-on=high       # CI gate: exit 1 on any high+ finding
commitbrief diff main...feature --markdown -o review.md

Exit codes stay simple: 0 for success — including a clean review or a --fail-on threshold that wasn't breached — and 1 for any error or a breached gate.

What it is not

It's the zeroth reviewer, not a replacement for a human one. It catches the obvious-but-easy-to-miss class: injection, missing nil checks, swallowed errors, a guard clause that's now unreachable. It does not catch intent-level design problems, and it won't tell you whether the feature should exist. That conversation stays with your reviewer.

I won't assert quality at you, either. There's a reproducible eval harness scoring real output against a known-answer corpus — run it yourself:

COMMITBRIEF_EVAL_PROVIDER=<name> make eval-live

If you want a second pair of eyes on your diff before anyone else gets one — locally, with the provider you already trust — that's the whole point:

commitbrief setup     # pick a provider, paste a key, ping it
commitbrief init      # optional: write project-specific rules
git add .
commitbrief --staged

Repo and install instructions: github.com/CommitBrief/commitbrief.

This is part 1 of **Building CommitBrief. Next: how one Go interface fans out to 10 LLMs across three transport classes — native APIs, OpenAI-compatible endpoints, and subprocess-backed CLIs.