A Unified Node.js LLM Endpoint: API Keys, Model Mapping, Retries, and Evals

Short answer: put one authenticated gateway in front of the model providers, expose stable model aliases through one unified endpoint, and make retry behavior and response normalization part of that gateway's contract. Keep the upstream API keys in its environment, never in callers. Then require every alias change to pass the same evaluation set before it reaches production.

The tempting version is a thin proxy: accept JSON, replace a header, forward the body, and relay whatever comes back. It's quick to demonstrate. It also leaves every caller responsible for provider-shaped responses, streaming details, retry policy, and model renames. The more useful experiment asks a stricter question: can a notebook, a Python worker, and a Node.js backend all depend on the same small contract while the upstream mapping changes underneath them?

My constraint for that experiment is eval reproducibility, not minimum line count. A gateway earns its extra network hop only if it keeps credentials out of application services, makes model selection reviewable, and produces records that let a team compare quality, latency, token use, and attempts. Otherwise, direct provider calls are easier to understand.

What should one unified API endpoint promise?

Start with the narrowest contract that the applications actually share. A useful request contains an internal model alias, input messages, requested capabilities, a stream flag, and a caller-supplied request ID. A useful final record contains that alias, the resolved model identifier, normalized output, a normalized completion state, usage when reported, the attempt count, and timing. Provider-specific data can live in an explicitly optional field; ordinary application code shouldn't need it.

Contract layer	Stable field	Provider detail kept out of callers
Request	alias, messages, capabilities, stream, request ID	Authentication headers and upstream model ID
Response	output, completion state, usage, attempts, timing	Provider response shape and stop vocabulary
Operations	caller, config revision, resolved model	Secret values and prompt content

That boundary matters more than matching one provider's request shape. If the public request is copied from a single upstream, its optional fields quietly become your permanent interface. A later mapping may accept plain text but differ on tool schemas, image input, structured output, or streaming usage. Reject an unsupported capability before dispatch. Silently deleting it can turn an agent call into plausible prose, which is much harder to catch than a clear validation error.

I keep aliases declarative and boring. The following Python value stands in for config loaded and validated at process startup; the model IDs are placeholders because deployed IDs must be selected from the providers' current documentation and pinned by the operator.

MODEL_ALIASES = {
    "chat.default": {
        "provider": "provider_a",
        "model": "pinned-model-id-a",
        "capabilities": {"text", "tools", "structured_output"},
    },
    "chat.long_context": {
        "provider": "provider_b",
        "model": "pinned-model-id-b",
        "capabilities": {"text", "tools"},
    },
    "summarize.batch": {
        "provider": "provider_c",
        "model": "pinned-model-id-c",
        "capabilities": {"text"},
    },
}


def resolve_model(alias: str, required: set[str]) -> dict:
    route = MODEL_ALIASES.get(alias)
    if route is None:
        raise ValueError(f"unknown model alias: {alias}")

    missing = required - route["capabilities"]
    if missing:
        names = ", ".join(sorted(missing))
        raise ValueError(f"{alias} does not support: {names}")

    return route

Don't let clients submit a provider name and arbitrary model ID if centralized control is the point. Let them submit chat.default. The resolver records what that meant for this request, and the eval report groups results by both alias and resolved ID. That small distinction prevents a mapping change from smearing two model versions into one time series.

The response envelope needs the same discipline. Use explicit nulls for unavailable usage instead of converting unknown counts to zero. Preserve a stable request ID across gateway logs and application traces. Keep raw upstream fields behind an escape hatch, because a team that reads them is intentionally choosing provider coupling — sometimes that is the right choice, but it should be visible in code review.

How should a Node.js backend proxy map models behind one API key?

The Node.js service should authenticate to the gateway with one scoped key; the gateway alone holds the upstream credentials. “One key” should mean one credential surface for a caller, not one global secret copied into every workload. Issue separate gateway keys by service or environment so revoking a staging caller doesn't interrupt production, and keep those values in the deployment platform's secret store.

An environment layout can stay compact:

import os

GATEWAY_URL = os.environ["LLM_GATEWAY_URL"]
GATEWAY_API_KEY = os.environ["LLM_GATEWAY_API_KEY"]
MODEL_ALIAS = os.getenv("LLM_MODEL_ALIAS", "chat.default")

Those are the only model-access settings an application needs. The gateway process has separate upstream credentials and the alias configuration, but they don't cross the trust boundary into the Node.js backend, browser, notebook, or worker. In particular, a browser must never receive the gateway key just because the endpoint speaks HTTP; the backend remains the authenticated caller.

This is where the familiar provider names fit: OpenAI, Anthropic Claude, and Google Gemini belong in gateway adapter configuration, not in the application's public request contract. The env setup for an application points to the unified endpoint and supplies its scoped API key; the gateway's separate environment supplies whichever upstream secrets its current model mapping needs. Imagine a RAG worker sending chat.default during an evaluation run while the Node.js backend sends the same alias in production. If a candidate mapping moves that alias from one adapter to another, both callers should receive the same internal envelope, both records should retain the resolved provider and model, and the evaluation comparison should expose any quality or usage movement before the configuration is promoted. If instead each caller sends a literal upstream model name, owns a different response parser, and interprets retry errors independently, the proxy has centralized secrets but has not created a unified runtime. That half-abstraction is the simple approach I would reject: it adds a hop while leaving the difficult coupling in every application.

Configuration changes should be atomic and auditable. Parse a candidate mapping, reject duplicate aliases and unknown capabilities, run contract tests, then swap the whole validated snapshot. Half-applied configuration is a nasty failure mode: one process may report an alias while another resolves it differently. Include a configuration revision in logs so that a surprising output can be tied to the exact mapping without guessing.

The catch is capability drift. An alias called chat.default encourages callers to assume interchangeability, while model families can expose meaningfully different controls. Maintain a capability matrix beside the mapping and test the combinations that the product uses. I wouldn't build a universal schema for every possible provider feature on day one; I would normalize the stable core and allow an isolated provider-options object where a feature genuinely has no shared representation. That keeps notebook-to-prod promotion practical without pretending all models are identical.

Retry the request budget, not every failure

Retries need an explicit policy because generative requests can be slow, streamed, and expensive. A blanket “three retries” hides the decisions that matter: which failures are transient, whether any output has reached the caller, how much total time remains, and whether replaying the operation is acceptable. I make 429 a named test fixture, but I don't assume every non-success response is retryable. The adapter classifies the outcome; the gateway enforces a deadline shared by all attempts.

Keep it bounded.

For a non-streaming call, retry only classifications that the adapter marks as safe and temporary. Apply backoff within one total deadline, cap attempts, and log every attempt rather than only the winner. For a stream, stop automatic replay after the first output event has crossed the gateway boundary. Restarting then can duplicate text or tool actions, and hiding that choice inside middleware makes downstream state impossible to reason about.

from dataclasses import dataclass
import random
import time
from typing import Callable


@dataclass(frozen=True)
class AttemptResult:
    value: dict | None
    retryable: bool
    output_started: bool = False


def run_with_budget(
    call: Callable[[], AttemptResult],
    budget_seconds: float = 20.0,
    max_attempts: int = 3,
) -> dict:
    deadline = time.monotonic() + budget_seconds

    for attempt in range(1, max_attempts + 1):
        result = call()
        if result.value is not None:
            return result.value
        if result.output_started or not result.retryable:
            raise RuntimeError("request cannot be replayed safely")

        remaining = deadline - time.monotonic()
        if attempt == max_attempts or remaining <= 0:
            raise TimeoutError("request budget exhausted")

        delay = min(0.15 * (2 ** (attempt - 1)) + random.random() * 0.1, remaining)
        time.sleep(delay)

    raise AssertionError("unreachable")

This example deliberately leaves provider response classification in the adapter. That is where provider-specific transport knowledge belongs, while the outer loop owns the policy. It also avoids a misleading guarantee: an idempotency header is useful only where the receiving API defines its semantics, so a generic gateway shouldn't claim that adding a header makes every replay safe.

Streaming deserves its own contract test. Server-Sent Events use the text/event-stream media type, and event messages are separated by a blank line. Multiple data: lines are joined by the browser with newline characters; comment lines can be used to keep a connection alive. The MDN description also covers event IDs and reconnection behavior. A gateway that consumes one event stream and emits another must parse event framing rather than split arbitrary network chunks, because chunk boundaries are not message boundaries.

Short requests can still fail late. Feed the parser an event across several chunks, several events in one chunk, a multi-line data field, a comment, and a connection close before the terminating blank line. I am not sure one reconnection policy fits both token display and tool execution; the correct choice depends on whether downstream effects have begun. The contract should say which side reconnects and what a repeated event ID means before production traffic answers the question for you.

The eval harness is part of the gateway

Model mapping is a release, even when it is “only config.” Before changing an alias, replay a fixed evaluation set against the current and candidate mappings. Score the product behavior that matters: answer quality, groundedness for RAG, tool selection and arguments for agents, structured-output validity, latency distribution, reported token usage, error classification, and number of attempts. Store prompts carefully; an eval corpus can contain the same sensitive context as production input.

The focused test is small enough to run during development. Give every case a required capability and expected structural assertions, then compare semantic scores separately. A response can read well while violating a schema, and averaging those into one number conceals the reason for rejection.

from dataclasses import dataclass


@dataclass(frozen=True)
class EvalCase:
    name: str
    alias: str
    required_capabilities: set[str]
    messages: list[dict[str, str]]
    must_return_json: bool


CASES = [
    EvalCase(
        name="grounded_summary",
        alias="chat.default",
        required_capabilities={"text", "structured_output"},
        messages=[{"role": "user", "content": "Summarize only the supplied context."}],
        must_return_json=True,
    ),
    EvalCase(
        name="tool_selection",
        alias="chat.default",
        required_capabilities={"text", "tools"},
        messages=[{"role": "user", "content": "Find the order status."}],
        must_return_json=False,
    ),
]


def validate_cases() -> None:
    for case in CASES:
        resolve_model(case.alias, case.required_capabilities)

Prompt cost belongs in the evaluation record. Track input and output usage when the upstream reports it, mark missing usage as unknown, and include failed attempts in operational accounting. A retry that eventually succeeds still consumed latency and may have consumed tokens. The useful comparison is not “which model has the lowest listed unit price?” but “which mapping meets the quality threshold under the workload's latency and usage distribution?”

Observability follows the same envelope. Record request ID, caller identity, alias, resolved provider and model, config revision, requested capabilities, attempt classification, time to first output, total duration, usage availability, and completion state. Do not log prompts or outputs by default. If sampled content is necessary for quality analysis, define access, retention, and redaction separately from ordinary service logs.

A gateway is not suitable when one service calls one provider, needs provider features immediately, and has no credential-sharing problem. Direct integration has fewer moving parts and exposes the full upstream API. It is also the better choice when an extra hop or stream parser cannot fit the latency budget. Use a gateway when centralized keys, consistent policy, and controlled model substitution are worth operating another critical service. There is no free abstraction here.

Before copying this design, measure four things in a shadow or test environment: added gateway latency, stream framing correctness, retry amplification, and eval movement after an alias change. If the gateway cannot make those visible, its unified endpoint is hiding risk rather than managing it.

References

• MDN Web Docs, “Using server-sent events”: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events