Short answer: put multiple LLM providers behind one API key only after the code-review service has a provider-independent JSON contract, separate quality and latency SLOs, and a fallback policy that can stop with needs_review instead of turning every weak answer into another model call.
For a service that classifies code changes and returns structured findings, the gateway comparison is not really about who offers the longest model list or the cheapest route. It is about whether a route change preserves the meaning of a finding while staying inside the pull-request check's deadline. A response can be quick, valid JSON, and still be wrong. Another can be accurate but arrive after the developer has merged or abandoned the change. Those are different failures, so one success counter cannot govern both.
Start with two budgets. The quality budget covers schema-valid results that meet an evaluation threshold for each important label; the latency budget covers the end-to-end review deadline, including queue time, provider attempts, parsing, and publication. Then treat a shared key and gateway as replaceable plumbing. That ordering matters because automatic fallback otherwise hides the signal needed to decide whether a second call helped.
No magic here.
Classification output is a versioned contract
Define the result states before defining routes. For this workload, accepted means the output parses, matches the versioned schema, uses only allowed labels, and arrives before the deadline. needs_review means automation cannot make a defensible classification. Transport failure, deadline exhaustion, schema rejection, and low classification confidence belong in separate internal reason codes even if users see the same neutral result.
JSON syntax is only the first gate. If the taxonomy permits security, correctness, performance, and documentation, a syntactically valid response containing urgent is not an accepted classification. Nor is a finding without a file path when the contract requires one. Function calling or structured-output features can constrain how a model expresses an answer, but the application still owns validation of its business rules and should reject unknown enum values rather than quietly mapping them.
Fallback should then follow a small state machine. A rate-limit or unavailable transport path may permit a second provider if enough deadline remains. A schema-invalid response may also permit one second attempt, but count it against the quality budget rather than disguising it as a transport retry. A valid low-risk documentation result should not be sent through three providers merely to seek consensus. For a security-sensitive change, disagreement can terminate at needs_review, because picking whichever label arrived last creates confidence the evidence does not support.
The short version: fallback is a bounded decision, not a retry loop. Set an attempt ceiling, subtract elapsed queue and call time from the original deadline, and reserve enough time to validate and publish the result. Don't reset the clock when the route changes. That is how a nominal two-second timeout becomes a six-second pull-request check while every component claims it met its own target.
The ambiguous case is provider disagreement. I'm not sure a universal confidence threshold exists for arbitrary code-review taxonomies; a labeled evaluation set, reviewed by people who understand the repository, is what resolves that uncertainty. Store the route, schema version, prompt version, and outcome reason with each evaluation result, while applying an explicit retention and redaction policy to source diffs. Without those dimensions, a blended accuracy number cannot tell an operator whether the model changed, the traffic mix shifted, or the parser began rejecting a previously tolerated shape.
Implement bounded JSON fallback in Go
Keep the API key at the routing boundary, but keep provider identity out of the application's result contract. The caller should submit a request ID, repository policy, changed-file evidence, an allowed taxonomy, and an absolute deadline. It should receive a schema version, labels, findings, and a disposition. Provider and attempt metadata belong in restricted operational telemetry, not in logic that decides how the product renders a finding.
Here is the useful minimum in Go. It deliberately models providers as interfaces and contains no vendor endpoint, model ID, or transport assumption. The code makes one point that gateway demos often skip: a second attempt is allowed only while the original deadline still has enough room for the call and validation.
package review
import (
"context"
"encoding/json"
"errors"
"time"
)
type Request struct {
ID string `json:"id"`
Diff string `json:"diff"`
Labels []string `json:"allowed_labels"`
Deadline time.Time `json:"deadline"`
}
type Finding struct {
Path string `json:"path"`
Label string `json:"label"`
Summary string `json:"summary"`
}
type Result struct {
SchemaVersion string `json:"schema_version"`
Findings []Finding `json:"findings"`
Disposition string `json:"disposition"`
}
type Classifier interface {
Classify(context.Context, Request) ([]byte, error)
}
var ErrNeedsReview = errors.New("classification needs review")
func Classify(ctx context.Context, req Request, routes []Classifier) (Result, error) {
const validationReserve = 100 * time.Millisecond
allowed := make(map[string]struct{}, len(req.Labels))
for _, label := range req.Labels {
allowed[label] = struct{}{}
}
for _, route := range routes {
if time.Until(req.Deadline) <= validationReserve {
break
}
callCtx, cancel := context.WithDeadline(ctx, req.Deadline.Add(-validationReserve))
raw, err := route.Classify(callCtx, req)
cancel()
if err != nil {
continue
}
var result Result
if err := json.Unmarshal(raw, &result); err != nil {
continue
}
if result.SchemaVersion != "1" || result.Disposition != "accepted" {
continue
}
valid := len(result.Findings) > 0
for _, finding := range result.Findings {
_, known := allowed[finding.Label]
if !known || finding.Path == "" || finding.Summary == "" {
valid = false
break
}
}
if valid {
return result, nil
}
}
return Result{SchemaVersion: "1", Disposition: "needs_review"}, ErrNeedsReview
}
Production code needs bounded request and response sizes, cancellation-aware HTTP transport, secret rotation, and structured logs that do not leak repository content. It also needs idempotency at the publishing boundary: classification itself is read-only, but posting duplicate review comments is not. Generate findings first, persist the accepted result against the request ID, and let a separate idempotent publisher update the code-review system.
A gateway that exposes several providers through one credential can reduce credential distribution and adapter work. The catch is operational: it also creates a shared dependency and may limit control over scheduling, data location, or rollout timing. Direct integrations keep that control but multiply authentication, response normalization, retry behavior, and observability work. A self-hosted router moves policy and telemetry into the platform team's domain, along with upgrades, capacity planning, and on-call ownership.
| Control-plane choice | Useful when | Quality/latency risk | Team obligation |
|---|---|---|---|
| Managed gateway | Several routes must be evaluated quickly | An extra shared hop can consume deadline and concentrate failures | Test contract behavior and export route-level telemetry |
| Self-hosted router | Scheduling, residency, or custom policy needs direct control | Router saturation can distort every model comparison | Staff upgrades, capacity, security, and incidents |
| Direct integrations | One or two stable routes already meet the SLO | Provider differences can leak into application code | Maintain separate auth, adapters, and measurements |
The table is a buy-versus-build decision, not a ranking. Stick with direct calls when one provider meets the measured target and a gateway would add an unstaffed control plane. A managed gateway is not suitable when repository policy forbids another data processor or when its telemetry cannot support the required audit. Self-hosting is not suitable when nobody owns peak-capacity estimates, upgrades, and after-hours response. One API key is convenient; it is not an availability design.
Convenience isn't capacity.
How should one API key compare multiple LLM providers safely?
Build the evaluation set before enabling dynamic routing. Sample the actual classes of changes the service sees: small documentation edits, generated files, broad refactors, dependency changes, and security-sensitive modifications. Human reviewers should assign the expected labels and findings under a written rubric. Keep a holdout set so repeated prompt tuning does not turn the benchmark into a memory test.
Measure schema acceptance, precision and recall per label, false-negative rate for high-impact labels, and end-to-end latency percentiles. Report queue time separately from provider time. A route that wins on mean latency but has a long p99 can stall a merge queue; a route with good aggregate accuracy can still miss nearly every rare security label. Your mileage may vary with language mix and diff size, which is exactly why a result without payload-class dimensions is weak evidence.
Capacity planning comes before weights. Estimate peak incoming reviews per minute, concurrent calls at the chosen deadline, and retry amplification when the primary path slows. Then reserve fallback capacity independently. Consider the shape of a failure rather than inserting an invented percentage: requests already occupy primary-route concurrency while their deadlines drain, new reviews continue to arrive, and eligible calls begin entering the secondary queue. If the fallback pool was sized from ordinary single-attempt traffic, the router has multiplied demand at the precise moment that usable capacity shrank. Queue age rises before completed-call latency exposes the problem, so admission control should shed low-priority automatic rechecks, preserve room for fresh reviews, and cap attempts across the service as well as per request. The capacity worksheet should therefore include peak arrival rate, deadline, maximum attempts, expected payload classes, provider concurrency limits, and headroom for validation and publication; a provider comparison without those inputs is a demo, not an operating plan.
Run the candidate route in shadow mode, where it receives representative requests but cannot publish findings. Compare its output with the current path and the human-labeled set. Shadow traffic has privacy and cost implications, so sample deliberately and apply the same data-handling rules as the serving path. After it clears the offline quality gate, expose it to a small traffic slice, change one policy variable at a time, and attach the policy version to every decision.
Do not collapse the rollout dashboard into green or red. It should separate:
- accepted, schema-rejected, deadline-exhausted, and
needs_reviewoutcomes; - quality by label and payload class;
- queue, provider, validation, and publication latency;
- first-route success, fallback attempts, and route disagreement;
- policy, prompt, and schema versions.
This is where the cheapest-routing question gets put in its proper place. Cost belongs in the constraint set, measured per accepted and useful classification, after minimum quality and latency thresholds pass. A low-cost call that produces rejected JSON or forces a human to reconstruct missing evidence is not comparable to an accepted result. Avoid a universal weighted score unless the team can defend every weight during an SLO review; a quality floor followed by a latency objective is easier to operate and much harder to manipulate.
Verify quality and latency before rollback
Verification starts below production. Contract tests should feed every adapter missing fields, extra fields, unknown labels, truncated JSON, cancellation, and responses that arrive after the deadline. Evaluation tests should replay the fixed labeled set and fail the build when a protected label drops below its agreed floor. Load tests should include the fallback amplification calculated during capacity planning, not merely the normal single-attempt rate.
Then rehearse rollback.
Make it boring.
Keep routing policy in version control and make the last known-good version deployable without editing it during an incident. A rollback trigger should refer to the service's own indicators, such as schema acceptance, per-label quality on delayed human review, p99 end-to-end latency, queue age, or fallback rate. The exact thresholds cannot be inferred from a generic gateway comparison; set them from the code-review product's tolerance and available error budget. Require consecutive evaluation windows where traffic is sparse enough that one request would otherwise cause noise, and record who can override the automation.
Rollback should restore policy, not erase evidence. Preserve request IDs, policy versions, route outcomes, and redacted diagnostic samples under the retention policy so the team can distinguish a provider shift from router saturation or a schema rollout. Rotate a compromised credential independently of routing policy, and verify that revocation removes access for every route sharing that key. Also test the dull path: if all model routes are unavailable within the deadline, the service must return needs_review, keep the merge check state coherent, and avoid publishing partial findings.
The final decision rule is intentionally conservative. Adopt a multi-provider gateway only when shadow and canary results show that it preserves the classification contract, fits the full latency budget under fallback load, exposes enough telemetry to explain route decisions, and has an owner for rollback. Otherwise, keep the smaller control plane and invest in the evaluation set. Models and gateways will change; the labeled evidence, explicit failure states, and reversible policy are the parts worth keeping.
Top comments (0)