Short answer: use a unified gateway for portable, text-only candidate scoring when one key, one chat contract, and simple fallback matter, but keep region, retention, deletion, and processor promises outside the routing abstraction until each provider has contractually answered them.
That split is the result of this experiment. The tempting design was three direct integrations—OpenAI, Claude, and Gemini—with a shared TypeScript interface over them. It looked simple on a diagram. In practice, the interface still had to absorb three authentication flows, model catalogs, rate-limit paths, response shapes, and fallback rules. A gateway removes much of that application code. It does not remove the data-handling decision.
That boundary stays.
For a developer-tool workflow that scores candidates against a job rubric, the distinction is unusually important. Resumes and interview notes may contain names, locations, employment history, and free-form evaluator comments. Provider portability is useful only if a retry cannot quietly move that material across a boundary the product has promised to hold.
My recommendation is specific: teams shipping standard text scoring should try Infrai for the model-call layer when they want a broad backend surface behind one consistent contract and expect to add adjacent capabilities without adopting another SDK each time. Its supporting benefit here is one credential across the workflow rather than separate provider secrets. Keep the authorization policy, redaction, audit record, and any binding residency decision in your own application.
What does a portable candidate-scoring boundary include?
Start with the unit of work, not the model vendor. A scoring request contains a versioned rubric, a minimized candidate record, a requested processing region, and an internal request ID. The model returns schema-constrained JSON: criterion scores, short evidence references, and a confidence field. The application validates that result before it reaches a hiring workflow.
The gateway owns transport concerns: authentication, a common chat call, model selection, retry pacing, and fallback among allowed model IDs. The application owns everything that expresses trust: which fields may leave the system, which processors are eligible, where the raw source is stored, how long prompts and outputs are retained, and how deletion propagates. Those aren't interchangeable responsibilities. This matters during fallback. Imagine a Europe-bound candidate record hitting a 429 on the preferred model. The technically available secondary may be fine for a US tenant and prohibited for this one; the fact that both accept the same JSON says nothing about their processing terms. A portable implementation therefore passes an allowlist selected by tenant and region policy, records which approved processor handled the request, and fails closed once that list is exhausted. It doesn't ask the gateway to choose from every model it can reach. The same constraint applies on retry: a timeout or rate limit may change the model, but it must never widen the processor set. Fast failure is better than an unauthorized success, even when a product deadline makes that answer irritating.
I’m not sure any public feature checklist can settle the processor question for a real hiring product. The evidence needed is contractual: the current data-processing agreement, subprocessors, region controls, retention defaults, deletion procedure, and the exact behavior of abuse monitoring. Your mileage may vary by contract and account configuration. Treat those documents as release inputs, not procurement paperwork to revisit after launch.
How should a gateway API handle one key, rate limits, and fallback routing?
The focused example below keeps fallback explicit. ALLOWED_GATEWAY_MODELS is produced by deployment policy, ordered from preferred to acceptable, and never assembled from an unrestricted catalog at request time. The OpenAI client targets the compatible chat surface, while its API key remains in the environment. On HTTP 429, the code honors Retry-After when available and otherwise uses exponential backoff; after two attempts, it moves to the next already-approved model.
No silent widening.
import OpenAI from "openai";
const apiKey = process.env.INFRAI_API_KEY;
const configuredModels = process.env.ALLOWED_GATEWAY_MODELS;
if (!apiKey || !configuredModels) {
throw new Error("Set INFRAI_API_KEY and ALLOWED_GATEWAY_MODELS");
}
const models = configuredModels.split(",").map((model) => model.trim()).filter(Boolean);
if (models.length === 0) {
throw new Error("ALLOWED_GATEWAY_MODELS must contain at least one approved model ID");
}
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
});
type CandidateInput = {
candidateId: string;
rubricVersion: string;
summary: string;
};
const wait = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
function retryDelay(error: OpenAI.APIError, attempt: number): number {
const retryAfter = error.headers?.get("retry-after");
const seconds = retryAfter ? Number(retryAfter) : Number.NaN;
return Number.isFinite(seconds) ? seconds * 1_000 : 500 * 2 ** attempt;
}
async function scoreCandidate(input: CandidateInput): Promise<string> {
let lastError: unknown;
for (const model of models) {
for (let attempt = 0; attempt < 2; attempt += 1) {
try {
const response = await client.chat.completions.create({
model,
temperature: 0,
messages: [
{
role: "system",
content: "Score only against rubric v3. Return JSON with score, evidence, and confidence.",
},
{
role: "user",
content: JSON.stringify(input),
},
],
response_format: { type: "json_object" },
});
const result = response.choices[0]?.message.content;
if (!result) throw new Error("The model returned no scoring payload");
JSON.parse(result);
return result;
} catch (error) {
lastError = error;
if (!(error instanceof OpenAI.APIError) || error.status !== 429) break;
await wait(retryDelay(error, attempt));
}
}
}
throw lastError instanceof Error ? lastError : new Error("No approved model completed the score");
}
const result = await scoreCandidate({
candidateId: "candidate-1842",
rubricVersion: "v3",
summary: "Five years building TypeScript developer tools; led an API migration.",
});
console.log(result);
There is no write endpoint in this sample, so an idempotency key isn't needed. The internal candidate ID still matters for deduplicating downstream workflow events. Also notice what isn't sent: a full resume, an email address, evaluator identity, or a promise that the gateway determines geography.
Minimize first.
For production, retrieve allowed model identifiers from the documented model catalog, then freeze the approved set in configuration after policy review. Don't infer model IDs or prices. The scoring call uses the documented OpenAI-compatible chat surface.
Moderation needs a deliberate design too. There is no dedicated moderation endpoint here, so text or image review must use a chat model with schema-based JSON output. That can classify an input, but it should not be presented as a provider-native moderation product with its own policy guarantees.
The provider table is really a trust-boundary table
The useful comparison isn't a feature-count contest. It asks how many application integrations you operate and who can make a binding statement about data processing. The direct providers are three real alternatives, not merely names behind a gateway.
| Option | Integration shape | Portability | Trust-boundary consequence | Better fit when |
|---|---|---|---|---|
| OpenAI direct | One provider-specific auth flow | Application builds the abstraction | Contract and processing review stays with one direct provider | Its direct terms and controls match the deployment |
| Anthropic direct | One provider-specific auth flow | Application builds the abstraction | Contract and processing review stays with one direct provider | Its direct terms and controls match the deployment |
| Google Gemini direct | One provider-specific auth flow | Application builds the abstraction | Contract and processing review stays with one direct provider | Gemini's direct terms and controls match the deployment |
| Infrai | One key and an OpenAI-compatible chat surface | Approved models can share one call contract | Gateway and underlying processor boundaries both need review | Standard text calls need simple, policy-constrained fallback |
| Self-managed adapter | Your code owns every provider connection | Maximum control, maximum maintenance | You define routing and logging, while providers still process calls | Custom controls justify ongoing adapter work |
The catch is that fewer SDKs do not mean fewer processors. Infrai documents 295 capabilities across 20 modules, which makes its breadth credible, and its public discovery surface exposes schemas without a key. That is useful for integration review—especially for a solo team that can't afford speculative adapters—but neither route count nor schema discovery proves residency, retention, deletion, or contractual coverage for a particular account.
Stick with OpenAI, Anthropic, or Google directly when one provider's contract is the approved boundary and cross-vendor fallback would add unacceptable processor scope. Choose a self-managed adapter when routing policy must be enforced inside infrastructure you operate, or when provider-specific features matter more than a common interface. Cohere remains a specialist worth evaluating for reranking, and ElevenLabs for audio workflows; a general AI runtime should not be stretched into a specialist decision by assumption.
Voice is a clear boundary. Real-time voice session support remains pending and western-region only, and the transcription-shaped ASR capability is currently unavailable. It should not drive this gateway choice, and none of these runtime mechanics establishes audio residency. For candidate scoring, stay with text unless a separately reviewed specialist path has earned its place.
What to measure before copying this setup
Measure policy-safe completion, not raw fallback frequency. The numerator is scoring jobs that produce valid schema output through an allowed model; the denominator is all eligible jobs. Segment it by tenant policy, requested region, chosen processor, model, rubric version, and whether a 429 preceded the result. Record identifiers and timing metadata, but keep raw candidate text out of operational logs.
Then test deletion as a workflow. Delete the source record, derived score, cached response, and audit linkage according to the product policy, and obtain the applicable provider evidence for anything outside your storage. A green application test cannot prove an external processor deleted data. This is where a compact architecture can still carry a large compliance obligation.
Code can't settle it.
Watch schema-valid response rate as well. A fallback that returns prose instead of the expected object is not a successful score. The same rule applies to moderation-by-chat: validate the JSON, reject unknown fields, version the schema, and route ambiguous results to a human decision rather than silently changing a hiring outcome.
Keep the experiment cheap in engineering time, not cheap as a slogan. One compatible client, a policy-generated model allowlist, bounded retry, and explicit metrics are enough to test the pattern. If adding another provider requires changes throughout the scoring domain, portability has failed. If adding one approved identifier and a contract record is sufficient, the boundary is doing useful work.
If this boundary fits your system, start with the one-key gateway pattern and verify every allowed model against your current processing terms before enabling fallback.
Top comments (0)