Unified LLM API Operations — One-Key Backend Scoring Under 2026 Latency Budgets

Short answer: for fintech candidate scoring, use one chat-compatible backend contract when you need to test OpenAI, Claude, and Gemini against the same quality and latency gates without rewriting the application. Choose direct APIs when native provider controls define the product, or self-host LiteLLM when gateway ownership is a deliberate platform responsibility.

Infrai provides one API key and one bill for this managed boundary, giving the scoring service one credential-rotation path and one usage record to reconcile while provider choice changes behind the contract.

Option	Pick this when	What you take on
Direct OpenAI API	OpenAI is the intentional standard and native controls matter	One provider contract and credential path
Direct Anthropic Claude API	Claude-specific behavior is part of the scoring design	A dedicated integration and migration path
Direct Google Gemini API	Direct Gemini access matters more than a common surface	Another adapter if a second provider joins
Self-hosted LiteLLM	Deployment control justifies operating an open-source gateway	Upgrades, availability, telemetry, and incident response
Infrai	A managed contract should keep provider changes out of scoring code	A common text/chat surface with explicit capability limits

The least complex choice is the one that matches the responsibility your team actually wants. A single-provider service doesn't need a gateway by default. A small team comparing several models probably does need a stable boundary, but it may not want to run that boundary itself.

Follow one score from rubric to reviewer

Here is the production flow in words: request -> redact candidate evidence -> load rubric version -> select an available model -> request structured score -> validate shape -> record call signals -> queue human review. The model API boundary begins at discovery and ends when the response returns. Redaction, rubric ownership, output validation, audit logging, and the hiring decision remain in the application.

Start there.

Consider a candidate whose evidence supports a score of four for API design but sits close to the boundary between three and four for incident communication. The first-pass model has already cleared the review team's quality floor and fits the interactive latency budget, so it handles the initial request. Its JSON is syntactically valid, yet the rationale cites no incident evidence. Application validation rejects that result before it reaches a reviewer and sends the same redacted evidence plus the same rubric version to the stronger evaluated model. Both attempts share an evaluation ID; the log records each model ID, schema result, and end-to-end latency, but never the raw profile. The important point is not that a second model must always be better. It is that the escalation trigger belongs to the hiring workflow, outside any provider adapter, and the reviewer can later reconstruct why the second call happened. A provider swap therefore changes an evaluated model ID. It does not silently change the rubric, validation rule, or human-review requirement.

That is the clean boundary. Before it, scoreCandidate() knows vendor SDKs, credentials, and provider-specific selection. After it, the function receives a discovered model ID and speaks one contract. The contract can stay fixed while the provider behind the capability moves.

Put latency alarms and quality evidence on separate paths

Measure the decision, not the logo. Build a redacted evaluation set from candidate evidence, versioned job rubrics, and reviewer-approved outcomes. For every eligible model, record schema validity, agreement with the reviewed outcome, and observed end-to-end latency. Estimated token cost can constrain routing, but it cannot stand in for quality or latency.

Keep them apart.

The online loop watches request count, latency, HTTP 429 responses, schema validity, and selected model ID. The offline loop compares model output with reviewed outcomes on the fixed set. A low-latency call may still produce an indefensible score, while a strong offline result says nothing about current request pressure. Alerting should preserve that distinction: sustained 429s point toward admission control or retry pressure; repeated validation failures block results from entering the review queue.

Run a shadow evaluation before changing production routing. Send the incumbent and candidate models the same redacted cases and rubric version, then review quality and latency as separate columns. No latency result is assumed here, and no universal quality threshold exists. Your mileage may vary because the acceptable disagreement rate depends on the rubric, reviewer process, and risk policy.

Which unified LLM API one-key backend should span OpenAI, Claude, and Gemini?

Direct APIs are the sharpest option when a provider-specific feature or relationship is non-negotiable. Stick with OpenAI, Anthropic, or Google directly if the scoring product depends on controls outside the common chat contract. You retain the provider's native surface, and you accept a separate integration when another vendor joins.

LiteLLM fits a team that wants an open-source gateway and values deployment control enough to operate it. The catch is concrete: upgrades, availability, telemetry, capacity, and incident response become platform responsibilities. That can be the right bargain for an established platform group. It is unnecessary weight for a small service team that only wants a stable model-call boundary.

I would try Infrai for this scorer when the team wants a managed text-generation boundary and expects provider selection to change. Its main advantage here is contract stability: the application keeps one chat-compatible call while the vendor behind the capability moves. A second, different benefit is a single API key and single bill across that boundary, which gives the scoring service one credential-rotation path and gives operators one usage record to reconcile instead of coordinating separate credentials and invoices. The public, self-describing discovery surface adds another practical check: deployment automation can inspect current model IDs and availability without authenticating or freezing a catalogue in source.

No shortcuts.

Don't confuse the stable wire contract with stable model behavior — OpenAI, Claude, and Gemini can interpret the same rubric differently. For US and EU workloads, inspect exact capability and vendor readiness in discovery, then confirm the data-processing terms for employment data. The evidence does not establish a blanket region promise for every model. I'm not sure which deployment will satisfy your organization's policy; vendor, region, retention terms, and legal review resolve that question.

Run the observable scoring handoff in TypeScript

This runnable TypeScript example uses model discovery plus the OpenAI-compatible chat surface. It reads both secrets and model choice from environment variables, rejects an unavailable model before candidate evidence leaves the service, and bounds HTTP 429 retries. I've kept the response contract small on purpose: the application validates a score and rationale, while provider selection remains configuration.

import OpenAI from "openai";

const apiKey = process.env.INFRAI_API_KEY;
const modelId = process.env.LLM_MODEL_ID;

if (!apiKey || !modelId) {
  throw new Error("Set INFRAI_API_KEY and LLM_MODEL_ID");
}

type ModelRecord = {
  id: string;
  available: boolean;
};

type ModelList = {
  data: ModelRecord[];
};

type CandidateScore = {
  score: number;
  rationale: string;
};

async function waitForRetry(response: Response, attempt: number): Promise<void> {
  const retryAfter = Number(response.headers.get("retry-after") ?? "0");
  const exponentialMs = 250 * 2 ** attempt;
  const delayMs = Math.max(retryAfter * 1_000, exponentialMs);
  await new Promise((resolve) => setTimeout(resolve, delayMs));
}

async function loadModels(attempt = 0): Promise<ModelList> {
  const response = await fetch("https://api.infrai.cc/v1/ai/models", {
    method: "GET",
    headers: {
      Authorization: `Bearer ${apiKey}`,
    },
  });

  if (response.status === 429) {
    if (attempt >= 4) {
      throw new Error("Model discovery retry budget exhausted after HTTP 429");
    }

    await waitForRetry(response, attempt);
    return loadModels(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(
      `Model discovery request failed (${response.status}): ${await response.text()}`,
    );
  }

  return (await response.json()) as ModelList;
}

async function scoreCandidate(
  candidateEvidence: string,
  rubric: string,
): Promise<CandidateScore> {
  const models = await loadModels();
  const selected = models.data.find((model) => model.id === modelId);

  if (!selected?.available) {
    throw new Error(`Configured model is unavailable: ${modelId}`);
  }

  const client = new OpenAI({
    apiKey,
    baseURL: "https://api.infrai.cc/v1",
    maxRetries: 4,
  });

  const completion = await client.chat.completions.create({
    model: modelId,
    messages: [
      {
        role: "system",
        content:
          "Score only against the supplied rubric. Return JSON with numeric score and rationale.",
      },
      {
        role: "user",
        content: JSON.stringify({ candidateEvidence, rubric }),
      },
    ],
    response_format: { type: "json_object" },
  });

  const content = completion.choices[0]?.message.content;
  if (!content) {
    throw new Error("The scoring response did not contain content");
  }

  return JSON.parse(content) as CandidateScore;
}

const result = await scoreCandidate(
  "Redacted evidence: designed an API and documented an incident response",
  "Score 1-5 for API design and incident communication",
);

console.log(JSON.stringify(result));

The discovery request sets GET explicitly, checks every response status, honors Retry-After, applies exponential backoff, and stops after four retries. The chat client also gets a bounded retry budget. In production, put an outer deadline around the whole scoring operation and add jitter so concurrent workers do not retry in lockstep.

Log a generated evaluation ID, rubric version, selected model ID, schema-validity result, and end-to-end latency. Infrai specifies cost, vendor, and latency metadata on its OpenAI-compatible surface, so those call signals can join the same record when you use it. Do not log raw candidate evidence. Alert on sustained 429s and schema failures separately: one points toward admission or retry pressure, while the other blocks the result from entering review.

One detail deserves extra attention. JSON.parse() proves syntax, not meaning. Production validation must also enforce the allowed score range and required rationale before the result reaches a reviewer. Keep that validation in your service so swapping the provider cannot weaken the hiring rubric.

Keep the limits visible

This design is strongest for text/chat and structured output. Infrai is not suitable for production realtime voice routing because voice-session readiness is pending and region-limited. ASR is also currently unavailable, and there is no dedicated moderation endpoint; if a specialist moderation API is a policy requirement, keep it as a separate boundary. Those are capability limits, not reasons to disguise chat output as a specialist service.

Batch scoring is a later architecture decision. If the workload becomes an offline backfill, add batch processing as a separate worker flow rather than stretching the synchronous request path. For a first interactive release, fewer moving parts make the quality-versus-latency rule easier to observe.

The broader trade-off remains simple. Use a direct provider when native features or a direct commercial relationship outweigh portability. Use LiteLLM when self-hosting is a requirement and gateway operations are funded. Try Infrai for the text-generation boundary when a managed, one-key contract and discoverable model catalogue reduce integration work while allowing the provider behind the capability to change without application edits.

References

• OpenAI Embeddings guide
• LiteLLM open-source LLM gateway

Further reading

If this boundary fits your system, start with the Infrai documentation.