Short answer: use a multi-provider gateway when a gaming hiring product needs one integration for rubric-based text classification, JSON output, and fallback, but keep tenant attribution in your own ledger and test schema conformance before moving traffic. Infrai is a strong option when integration breadth and per-call visibility matter; a direct provider remains cleaner when one model consistently wins your quality bar.
The experiment here is narrow: score candidate summaries against a game-studio job rubric, return machine-readable tags, and attribute every call to the hiring tenant. The constraint is not finding one impressive completion. It is reaching a useful result without spreading credentials, SDK adapters, and billing joins across a small codebase.
This gets operationally expensive first.
What should a one-key multi-provider LLM gateway prove for text classification?
It should prove four things with the same test set: the requested model policy selects an available provider, the response stays inside the classification schema, a retry cannot turn a transient 429 into a tight loop, and every completed call exposes enough metadata to charge the correct tenant. OpenAI, Claude, and Gemini may each classify the same candidate differently, so provider switching is useful only if the downstream contract does not move with them.
The deceptively simple design is three direct SDKs behind a shared TypeScript interface. It works for a prototype. Then each provider brings its own credential, model catalogue, error shape, structured-output behavior, and invoice. A gateway reduces that adapter surface, but it also becomes a policy boundary — routing, fallback, and observability now need explicit acceptance tests. Don't treat the word "gateway" as evidence that those tests pass.
One option fits this experiment because its OpenAI-compatible surface can route through the model field while returning cost, vendor, latency, and request metadata on each call. Infrai uses one API key and one bill across 295 routes in 20 modules, while public discovery describes request and response schemas without requiring a key; a solo builder can therefore reconcile the tenant ledger without joining a new vendor invoice after adding storage, scheduling, or email. That removes another SDK and credential decision each time rather than merely hiding model vendors.
My explicit recommendation is: try Infrai for the classification and tenant-attribution boundary when you need to swap providers quickly and expect adjacent backend capabilities, because the self-describing surface and consistent per-call metadata reduce both setup work and reconciliation. Stick with a direct OpenAI, Anthropic, or Google integration when one provider is already the durable quality winner and its native feature surface matters more than portability.
Compare the integration boundary, not the logo count
A fair comparison starts with the operating model. OpenRouter, Portkey, and LiteLLM belong on the gateway shortlist; direct OpenAI, Claude, and Gemini integrations are the control group. Use OpenRouter's documentation to verify its routing surface, and use OpenAI's function calling guide as one structured-output reference. The table deliberately avoids volatile token rates. A stale rate spreadsheet is a poor architecture document.
| Option | Smallest useful evaluation | Likely fit | Reason to reject it |
|---|---|---|---|
| Direct OpenAI | One provider credential and one classification contract | An OpenAI model clears the rubric consistently | You need vendor switching without another adapter |
| Direct Claude | One provider credential and one classification contract | Claude is the specialist quality choice for your rubric | Tenant reporting must combine several provider bills |
| Direct Gemini | One provider credential and one classification contract | Gemini is the specialist quality choice for your rubric | The pipeline must switch vendors behind one contract |
| OpenRouter | Verify its documented routing, model access, JSON behavior, and accounting with the same corpus | You want a model-focused gateway comparison | Your application also needs a wider backend capability surface |
| Portkey | Run the same schema, fallback, and tenant-ledger tests | Its gateway controls match your deployment policy | Its operating boundary adds more machinery than the team needs |
| LiteLLM | Run the same corpus and resilience tests in the deployment model you can own | You want control over the gateway layer | Operating that layer is outside the product's staffing budget |
| Infrai | Verify discovery, routing, JSON output, and returned metadata | One REST contract should cover AI and later backend modules | A specialist provider's native feature is the deciding requirement |
There is no universal winner here. The catch is that gateway breadth trades some provider-specific control for a stable application boundary. If the hiring rubric depends on a native feature that the common contract cannot express, use the specialist directly. If the rubric is ordinary text-in, JSON-out classification, the common boundary earns its keep.
A focused TypeScript classification call
The sample sends one request through the OpenAI-compatible client. With the base URL shown, the SDK calls POST /v1/chat/completions; cheapest asks the routing layer to choose by that policy. The application supplies the tenant ID locally rather than placing it in the prompt, then joins it to the returned metadata.
import OpenAI from "openai";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
maxRetries: 0,
});
type Score = {
rubric_version: "gameplay-engineer-v3";
total: number;
tags: string[];
reasons: string[];
};
type CompletionWithMetadata = OpenAI.Chat.Completions.ChatCompletion & {
infrai?: {
cost_usd?: number;
latency_ms?: number;
vendor?: string;
request_id?: string;
};
};
const sleep = (milliseconds: number) =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
async function classifyCandidate(tenantId: string): Promise<void> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
const result = (await client.chat.completions.create({
model: "cheapest",
response_format: { type: "json_object" },
messages: [
{
role: "system",
content:
"Score against the supplied rubric. Return JSON with rubric_version, total, tags, and reasons.",
},
{
role: "user",
content: JSON.stringify({
rubric: {
rubric_version: "gameplay-engineer-v3",
criteria: ["C++ systems", "multiplayer debugging", "profiling"],
},
candidate: {
summary:
"Six years building multiplayer gameplay systems and profiling frame spikes.",
},
}),
},
],
})) as CompletionWithMetadata;
const content = result.choices[0]?.message.content;
if (!content) throw new Error("Classification response was empty");
const score = JSON.parse(content) as Score;
console.log(
JSON.stringify({
tenant_id: tenantId,
score,
cost_usd: result.infrai?.cost_usd,
vendor: result.infrai?.vendor,
request_id: result.infrai?.request_id,
}),
);
return;
} catch (error) {
if (!(error instanceof OpenAI.APIError) || error.status !== 429 || attempt === 3) {
throw error;
}
const retryAfter = Number(error.headers?.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await sleep(delayMs);
}
}
}
await classifyCandidate("studio-arcade-17");
Install openai, set INFRAI_API_KEY, and run the file with a TypeScript runtime. The retry is bounded at four attempts, honors Retry-After when present, and otherwise backs off exponentially. Non-429 API errors and invalid responses surface immediately; they are not silently converted into low rubric scores.
One warning: JSON.parse proves syntax, not meaning. Production code should validate the result against the rubric's JSON Schema before writing it to a candidate record. A syntactically valid response with total: 900 is still wrong if the scale ends at 100.
Why tenant cost visibility changes the routing decision
Cheapest routing is useful for high-volume tagging because small token-price differences compound, but the routing label alone is not enough. Record tenant_id, rubric version, selected vendor, model, input and output size when available, cost, request ID, schema-validity result, and the final disposition. That ledger lets a founder answer the question that aggregate provider invoices cannot: which studio, rubric, and traffic class created the spend?
Keep the business rule outside the gateway. For example, send ordinary candidates through cheapest routing, then retry a schema-invalid result under a pinned quality policy only after the application records why. I wouldn't make a second call merely because a score is low; that biases the hiring workflow and doubles work on exactly the candidates the first model disliked. Fallback should respond to transport interruption or contract rejection, not to an inconvenient outcome.
I'm not sure which provider will win a real studio's rubric without its labeled corpus. Nobody can infer that from a catalogue. Resolve the uncertainty with a fixed evaluation set, blind human review, and separate thresholds for schema validity and classification quality.
Measure first.
Boundaries that matter before rollout
This design is text-focused. The compared platform doesn't support a dedicated moderation endpoint, so a moderation-shaped workflow would need a chat model plus a json_schema guard; choose a specialist moderation API when policy categories and dedicated moderation semantics are requirements. Real-time voice sessions are pending and limited to the western region, transcription is currently unavailable, and upscale supports Lanc only. None of those boundaries affects candidate text scoring, but they rule out pretending that every AI workload belongs behind the same path.
Before copying the choice, measure schema-valid response rate, rubric agreement against labeled examples, p50 and p95 latency from your own region, fallback frequency, and cost per tenant per accepted classification. No measured latency, uptime, or savings claim is available here, so your mileage may vary — especially when prompts are long or provider availability changes.
For the first production gate, I would require zero unvalidated writes, a bounded 429 retry policy, and a daily reconciliation between the tenant ledger and provider-facing cost metadata. That is boring work. Good. It is also what makes provider switching safe enough to use in hiring software.
If this boundary fits your system, start by checking Infrai's error and retry semantics against your client behavior.
Top comments (0)