The decisive trade-off is control, not model count: a small team needs one stable code-review contract that can survive a provider change without flattening every model into the same feature set. Short answer: put authentication, routing, structured-output validation, usage accounting, and retry policy behind an application-owned multi-model API; then make a provider swap part of CI before calling the design portable.
This is narrower than a universal AI gateway, on purpose. A pull-request event supplies a diff and repository policy. The runtime selects an eligible model, asks for structured findings, validates the response, and returns a provider-neutral result to the B2B SaaS application. The app presents one internal key, while the gateway owns the upstream credentials. OpenAI, Claude, and Gemini are deployment choices at the edge, not types that leak into the product database.
That boundary buys leverage. It doesn't make the providers interchangeable.
How should a small team make practical multi-model API selection and avoid vendor lock-in?
Start with the artifact the product must trust: a finding. For code review, that means a file path, a line, a severity, a concise explanation, and a stable rule identifier. Model messages, tool-call envelopes, finish reasons, and token fields belong in adapters because they vary and can change independently of the product.
The selection test is blunt: can the same saved review cases run through a second adapter, produce valid findings, and preserve the product's audit fields without changing the caller? If switching requires a database migration, UI branching, or edits throughout business logic, the team has a model menu rather than portability.
Keep native capabilities available behind optional adapter methods. Lowest-common-denominator abstractions age badly. For example, a richer model-specific review mode can be exposed as an explicit capability, while the baseline review method stays portable. The caller can choose the richer path knowingly instead of discovering an accidental dependency months later.
One key is useful here — one application-facing credential reduces secret distribution across workers and preview environments. But a hosted multi-provider service holding all upstream access creates a new dependency. A self-owned gateway keeps the contract and routing policy movable; a managed gateway can reduce operational work. The catch is that its request format, logs, retention rules, export support, and failure semantics become part of the exit plan. Small teams should choose that trade deliberately.
What does a provider-independent code review look like in TypeScript?
This example runs without a network call. The two adapters stand in for upstream implementations, so the important part is visible: the product sends one request shape, the gateway validates every finding, and the stored record retains enough provenance to reproduce a routing decision. Real adapters can translate this contract into each provider's documented request and response format without changing reviewChange.
type Severity = "low" | "medium" | "high";
type Finding = {
ruleId: string;
path: string;
line: number;
severity: Severity;
message: string;
};
type ReviewRequest = {
repository: string;
commitSha: string;
diff: string;
policy: string[];
};
type ReviewResult = {
schemaVersion: 1;
route: string;
model: string;
findings: Finding[];
usage: {
inputUnits?: number;
outputUnits?: number;
};
};
type AdapterResult = {
model: string;
findings: unknown;
usage: ReviewResult["usage"];
};
interface ReviewAdapter {
readonly route: string;
review(input: ReviewRequest): Promise<AdapterResult>;
}
function parseFindings(value: unknown): Finding[] {
if (!Array.isArray(value)) throw new Error("findings must be an array");
return value.map((item, index) => {
if (typeof item !== "object" || item === null) {
throw new Error(`finding ${index} must be an object`);
}
const record = item as Record<string, unknown>;
const validSeverity = ["low", "medium", "high"].includes(
String(record.severity),
);
if (
typeof record.ruleId !== "string" ||
typeof record.path !== "string" ||
!Number.isInteger(record.line) ||
!validSeverity ||
typeof record.message !== "string"
) {
throw new Error(`finding ${index} does not match schema version 1`);
}
return record as Finding;
});
}
async function reviewChange(
apiKey: string,
input: ReviewRequest,
adapter: ReviewAdapter,
): Promise<ReviewResult> {
if (apiKey.length < 20) throw new Error("invalid gateway credential");
const raw = await adapter.review(input);
return {
schemaVersion: 1,
route: adapter.route,
model: raw.model,
findings: parseFindings(raw.findings),
usage: raw.usage,
};
}
function fixtureAdapter(route: string, model: string): ReviewAdapter {
return {
route,
async review(input) {
const changedConfig = input.diff.includes("timeoutMs: 0");
return {
model,
findings: changedConfig
? [{
ruleId: "CONFIG_TIMEOUT",
path: "src/config.ts",
line: 18,
severity: "high",
message: "A zero timeout disables the request deadline.",
}]
: [],
usage: {},
};
},
};
}
const request: ReviewRequest = {
repository: "acme/billing",
commitSha: "a84f2c1",
diff: "+ timeoutMs: 0",
policy: ["Every outbound request must have a positive deadline."],
};
const primary = fixtureAdapter("primary", "model-a");
const candidate = fixtureAdapter("candidate", "model-b");
const key = "local-development-key-0001";
const [a, b] = await Promise.all([
reviewChange(key, request, primary),
reviewChange(key, request, candidate),
]);
console.log(JSON.stringify({ primary: a, candidate: b }, null, 2));
This boundary refuses malformed output before it reaches the UI or database. It also records route and model separately. Those fields matter when an evaluation changes, a customer disputes a finding, or routing policy sends similar diffs to different destinations. The fixture is intentionally boring. Replace it with saved, redacted diffs and expected properties, not a provider-shaped mock that merely confirms the adapter copied fields correctly.
Don't normalize usage into a fictional universal token count. Tokenization can differ by encoding and model; tiktoken, for example, is an official BPE tokenizer library for OpenAI models and exposes model-specific encodings. Preserve upstream usage fields in adapter-owned telemetry, then map only the accounting units your product has explicitly defined. Otherwise, a clean dashboard can conceal an invalid cost comparison.
Fail closed before a finding reaches production
Free-form prose makes a quick demo and a poor review system. The difficult failures sit between valid JSON and useful findings: a line can be outside the changed hunk, two findings can describe the same issue, a severity can drift, or a message can quote source content that should not leave the review boundary. Schema validation catches syntax and types. Semantic validation has to inspect repository context.
A practical pipeline therefore has two gates. The first accepts only the declared schema version and rejects unknown enum values, missing paths, non-integer lines, and oversized fields. The second checks that the path exists in the submitted change, the line is eligible for review, the rule identifier is allowed, and duplicates collapse under a deterministic key. This logic belongs after every adapter, where one test suite can exercise it. Be conservative with fallback. Retrying another provider after a transport failure may be reasonable if the request is idempotent and the data policy permits that destination. Retrying because the first provider returned zero findings changes product semantics: “no issue found” is a valid answer, not proof of failure. Silent quality-based fallback also doubles work and makes latency and accounting difficult to explain. Privacy can narrow the routing pool further. A source diff may contain credentials, customer data, or regulated information even when the surrounding SaaS product is not marketed as a compliance product. Redaction should happen before routing, and destination eligibility should be policy data rather than an if statement buried in one adapter. If protected health information is in scope, the applicable administrative, physical, and technical safeguards under 45 CFR Part 164 need a real compliance review. An API abstraction does not answer that legal or operational question.
This part is easy to underestimate.
Turn provider changes into ordinary releases
Treat every provider or model change like a risky application release, with an artifact, a gate, a canary, and a rollback target. Run the candidate adapter against a versioned corpus built from the review job: small diffs, deleted files, renamed paths, generated code, policy conflicts, prompt-injection text inside comments, and intentionally clean changes. Score properties the product can defend. Useful measures include schema acceptance, finding precision after human adjudication, changed-line validity, duplicate rate, end-to-end latency, and billable usage as reported by the chosen route. Then shadow a bounded slice of live, policy-eligible traffic without publishing those findings. Compare validation failures and adjudicated usefulness before promotion; don't route by a vague belief that a newer model must be better.
I’m not sure a static “best model” table stays useful beyond the exact prompts, models, and corpus used to create it. Your mileage may vary. A repeatable replay harness resolves that uncertainty better than another ranking because it measures the workload the team will actually ship.
Use a release record like this, with values filled from your own tests rather than borrowed benchmarks:
| Release gate | Evidence to collect | Promotion threshold |
|---|---|---|
| Contract | Valid findings across saved cases | No caller or storage change |
| Quality | Human-adjudicated findings by rule | Meets the product's release bar |
| Latency | End-to-end distributions by diff size | Fits the review workflow budget |
| Accounting | Raw upstream usage plus internal units | Every request is attributable |
| Data policy | Eligible destinations by tenant and repository | No policy bypass during failover |
| Operations | Traces, route decisions, retries, and validation errors | A provider change remains explainable |
The recommended gateway pattern is not suitable when the application fundamentally depends on one provider's unique capability and that capability drives product value. In that case, use the native API directly, isolate it in one module, and accept the dependency in writing. It is also a questionable investment for a short-lived internal script with no stored outputs and no failover requirement. Portability has a carrying cost: adapters, replay fixtures, monitoring, and periodic tests all need ownership.
Can production routing stay boring during a provider switch?
Before deployment, pin the internal schema version, cap diff and output sizes, define time budgets, and decide which failures are retryable. Log a request identifier, tenant policy, chosen route, model identifier, schema version, validation result, latency, and provider-reported usage without logging raw source by default. Keep credentials server-side and scope the application key to the review operation.
Then rehearse the switch. Send the same shadow corpus through the candidate adapter, compare adjudicated results, review data eligibility, and promote the route through configuration rather than a release. Rollback should use that same control. The practical standard is simple: on-call can explain which route handled a review and why, while product code remains unaware of the provider envelope.
Finally, rerun the corpus whenever the prompt, schema, routing rule, adapter, or selected model changes. Version those inputs together. A small team doesn't need a grand platform; it needs a narrow boundary, evidence that the boundary holds, and an honest note about the cases where going native is the better engineering choice.
Top comments (0)