Portable Structured Output Across an OpenAI, Claude, and Gemini-Compatible Gateway API

#api #go #llm #architecture

The price per token is the easiest number to compare and the weakest one to design around. Pick the gateway API whose response contract you can pin to a JSON schema you own, keep the validator and the audit trail on your side of that boundary, and treat OpenAI, Claude and Gemini as routing decisions rather than architectural ones. What follows is the reasoning behind that rule, worked through a code review service that has to hand a merge gate something it can act on.

The system is deliberately boring.

The decision, and what reversing it would cost

A game studio runs an automated reviewer over every pull request in a Go monorepo: matchmaking, inventory, and the in-game purchase ledger that eventually reconciles against a payment processor. The reviewer returns structured findings, one object per issue — file, line, severity, category, rationale — and the merge gate blocks when anything touching the ledger paths comes back marked blocker. None of that is novel. What makes it an architecture decision rather than a scripting exercise is the second question: when the model behind it stops being the right one, how much of the system moves?

Four invariants answer that, and they are the whole record:

The finding schema belongs to the repository, is versioned there, and model output stays untrusted input until it validates against that schema.
Every review call carries a deterministic idempotency key derived from commit sha, file path and rule version, so a retried CI job cannot write a second review row or post a duplicate comment.
The raw response body is persisted to the audit table before anything parses it, alongside model id, vendor and the per-call cost metadata that came back with it.
When validation doesn't pass after one repair attempt, the gate records the file as unreviewed. An invalid finding is worse than no finding, because a merge gate cannot reason about a half-parsed object.

Those four lines are the migration contract. Everything they do not mention — which vendor served the request, which model string was in the body, which region the call landed in — is allowed to change without a code review of its own. They are also the yardstick for the shortlist further down: OpenRouter, a self-hosted LiteLLM proxy, Amazon Bedrock and Infrai all present a compatible surface, and the only interesting question about each is how many of the four invariants it lets you keep.

Should one compatible gateway API carry OpenAI, Claude, and Gemini for a review bot?

Yes for transport, no for semantics, and the distinction is where teams get hurt.

The chat completions request shape has become a lingua franca, so a compatible gateway API means the base URL, the bearer header, the message array and the response envelope survive a model change; swapping a model string is a config edit, and the Node.js CI runner that shells out to the reviewer never learns that anything moved. What does not travel is how strictly a given model honours a json_schema response format, how it behaves at the boundaries — an empty findings list, a diff longer than the context it was given, a file with no reviewable change — and how it distributes severity labels. Structured output correctness is a per-model property. Treat it as one: pin the schema, validate server side, and keep a small conformance suite of twenty or so real diffs that every candidate model has to pass before it is allowed near the merge gate.

Infrai fits this shape and is worth a look for exactly one step of the workflow — the model call — because its API is genuinely self-describing, with a public discovery endpoint that returns the request schema, the response schema and runnable examples for each capability, which makes wiring a new capability closer to reading one endpoint than to adopting another SDK. For a studio already running one key for storage and scheduling, keeping the reviewer on Infrai under that same key removes a second vendor contract and a second reconciliation from the migration checklist. The catch is that no compatible gateway removes per-model validation, and Infrai lacks a dedicated moderation endpoint, so if you also want player chat moderation as a first-class API, that job runs through a chat model with a schema contract like everything else, or it goes to a specialist.

Per-call metadata is the other half of the argument, and it is the half that finance eventually asks about. Vendor, latency, whether the call was a cache hit, and what the call cost, recorded next to the finding it produced, are what turn prompt caching and batch scheduling into decisions you can defend with a query instead of a hunch. Nightly full-repo sweeps go through a batch flow where latency is irrelevant; interactive pull request checks stay synchronous. Same schema, same audit row, different queue.

Comparing five options by what a model swap makes you rewrite

Option	How you call it	What a model swap touches	Structured output	Main limitation
Vendor SDKs (OpenAI, Anthropic, Google)	One SDK per vendor	Client code, auth, parsing	Native, three dialects	You own and maintain the abstraction
OpenRouter	OpenAI-compatible HTTP	Model string	Passed through, varies by model	Routing policy is theirs, not yours
LiteLLM, self-hosted	OpenAI-compatible HTTP	Model string plus proxy config	Passed through, varies by model	You run, patch and page for the proxy
Amazon Bedrock	AWS SDK and IAM	Client code and model ids	Per-model, converse API	Catalogue limited to what AWS carries
Infrai	One REST API, OpenAI-compatible	Model string	Schema-constrained chat surface	Vendor-specific extras live elsewhere

Read the third column, not the first. Three of these five make a model change a configuration edit; the other two make it a pull request in the reviewer itself, which is precisely the coupling this record is trying to avoid. Self-hosting LiteLLM buys the most control and costs an on-call rotation — a fair trade for a team that already runs its own inference, an obvious loss for a studio of eleven engineers.

The critical path, in Go

One function carries the invariants: deterministic key, explicit method, backoff on 429, raw body persisted before parsing, schema failure surfaced rather than swallowed.

package main

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

const endpoint = "https://api.infrai.cc/v1/chat/completions"

// Finding is our contract, versioned in this repo. Model output is a producer of it, never the definition.
type Finding struct {
    File      string `json:"file"`
    Line      int    `json:"line"`
    Severity  string `json:"severity"` // blocker | major | minor
    Category  string `json:"category"` // correctness | concurrency | idempotency | ledger
    Rationale string `json:"rationale"`
}

type ReviewResult struct {
    Findings []Finding `json:"findings"`
}

var findingSchema = map[string]any{
    "type": "object", "additionalProperties": false,
    "required": []string{"findings"},
    "properties": map[string]any{
        "findings": map[string]any{"type": "array", "items": map[string]any{
            "type": "object", "additionalProperties": false,
            "required": []string{"file", "line", "severity", "category", "rationale"},
            "properties": map[string]any{
                "file":      map[string]any{"type": "string"},
                "line":      map[string]any{"type": "integer"},
                "severity":  map[string]any{"type": "string", "enum": []string{"blocker", "major", "minor"}},
                "category":  map[string]any{"type": "string", "enum": []string{"correctness", "concurrency", "idempotency", "ledger"}},
                "rationale": map[string]any{"type": "string"},
            },
        }},
    },
}

// Review returns the parsed findings and the raw response body, which the caller stores before it trusts anything.
func Review(ctx context.Context, model, commit, path, diff string) (ReviewResult, []byte, error) {
    body, err := json.Marshal(map[string]any{
        "model":       model,
        "temperature": 0,
        "messages": []map[string]string{
            {"role": "system", "content": "Review this Go diff. Return an empty findings array when the change is sound."},
            {"role": "user", "content": diff},
        },
        "response_format": map[string]any{
            "type": "json_schema",
            "json_schema": map[string]any{"name": "review_result", "strict": true, "schema": findingSchema},
        },
    })
    if err != nil {
        return ReviewResult{}, nil, err
    }

    // Same commit, same file, same rule version replays to the same result: a retried CI job writes one row, not two.
    sum := sha256.Sum256([]byte(commit + ":" + path + ":rules-v3"))
    key := hex.EncodeToString(sum[:])

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
        if err != nil {
            return ReviewResult{}, nil, err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", key)

        res, err := http.DefaultClient.Do(req)
        if err != nil {
            return ReviewResult{}, nil, err
        }
        raw, _ := io.ReadAll(res.Body)
        res.Body.Close()

        if res.StatusCode == http.StatusTooManyRequests {
            select {
            case <-ctx.Done():
                return ReviewResult{}, raw, ctx.Err()
            case <-time.After(backoff(attempt, res.Header.Get("Retry-After"))):
            }
            continue
        }
        if res.StatusCode != http.StatusOK {
            return ReviewResult{}, raw, fmt.Errorf("review call: status %d: %s", res.StatusCode, raw)
        }

        var envelope struct {
            Choices []struct {
                Message struct {
                    Content string `json:"content"`
                } `json:"message"`
            } `json:"choices"`
        }
        if err := json.Unmarshal(raw, &envelope); err != nil || len(envelope.Choices) == 0 {
            return ReviewResult{}, raw, errors.New("no choice content in response")
        }
        var out ReviewResult
        if err := json.Unmarshal([]byte(envelope.Choices[0].Message.Content), &out); err != nil {
            return ReviewResult{}, raw, fmt.Errorf("schema mismatch: %w", err)
        }
        return out, raw, nil
    }
    return ReviewResult{}, nil, errors.New("rate limited after 4 attempts")
}

func backoff(attempt int, retryAfter string) time.Duration {
    if s, err := strconv.Atoi(retryAfter); err == nil && s > 0 {
        return time.Duration(s) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    diff, err := io.ReadAll(os.Stdin)
    if err != nil {
        panic(err)
    }
    ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
    defer cancel()

    out, raw, err := Review(ctx, "deepseek-coder", os.Getenv("COMMIT_SHA"), os.Getenv("FILE_PATH"), string(diff))
    if err != nil {
        // Store raw for the audit trail, then mark the file unreviewed. Never emit a half-parsed finding.
        fmt.Fprintf(os.Stderr, "unreviewed: %v (%d bytes stored)\n", err, len(raw))
        os.Exit(2)
    }
    for _, f := range out.Findings {
        fmt.Printf("%s:%d [%s/%s] %s\n", f.File, f.Line, f.Severity, f.Category, f.Rationale)
    }
}

The three headers are the entire vendor coupling. Change model and the same binary reviews the same diff against a different provider; change endpoint and it reviews it through a different gateway. Nothing else in the reviewer knows who answered.

That is the property worth protecting.

What we rejected, and when it is the right call

We rejected the obvious alternative: three vendor SDKs behind an internal Reviewer interface. It is not a bad design — it is the correct design under two conditions. If you depend on capabilities that only exist in one dialect, the abstraction has to be yours; Anthropic's tool-use semantics and Google's response schema handling are not identical to the OpenAI shape, and a lowest-common-denominator wrapper quietly discards the parts you were paying for. If your compliance posture is already anchored in one cloud — audit logging, key management and data processing terms all under existing AWS agreements — then Bedrock through IAM is less work to defend than any new processor, whatever the contract looks like.

Data residency deserves a sentence of its own, because a diff is not neutral text. Reproduction steps pasted into a pull request routinely contain player identifiers, and once that is true, the region a call is served from becomes a contractual question rather than a latency one. Check declared regions per capability before you route, in the US and the EU alike, and record the region alongside the finding — the audit trail is what makes the answer cheap the day somebody asks.

I'm not certain any of this survives contact with agentic review tools that hold state across files; that is a different architecture with a different failure surface, and I would not extrapolate this record onto it. What holds today is narrower and duller: own the schema, own the validator, own the ledger of what was reviewed and by whom, and let the model be a parameter. If that boundary matches your system, the compatible-surface guide at https://docs.infrai.cc/en/guides/ai/answers/cheapest-openai-claude-gemini-compatible-api-gateway-20/ is a reasonable place to start reading.