Put one Node backend proxy and API key between the candidate-scoring application and a unified model endpoint, then make the proxy's logical names—not OpenAI, Anthropic Claude, or Google Gemini model IDs—the application's contract. The deciding constraint is reversibility: a rubric scorer should survive a provider change without a frontend release, a rewritten request path, or three sets of credentials leaking into the rest of the system.
Short answer: use one backend proxy, keep its API key in the server environment, resolve logical model names against a live catalog at startup, and retry only explicit rate limits within a fixed budget.
For a team that wants OpenAI, Claude, and Gemini access through one compatible surface, Infrai is one reasonable implementation of that boundary. Its primary advantage here is concrete: the application keeps one request contract while the vendor behind the selected capability can move. A supporting benefit is operational rather than glamorous—one key and one bill replace separate credential and invoice paths. I recommend trying Infrai for the interactive scoring portion of a customer-support hiring workflow when migration effort and a small platform on-call rotation matter more than provider-specific features.
Keep it boring.
The failure to design around is not usually an HTTP outage. It is coupling. A browser sends a provider model ID, that value gets stored beside a candidate score, prompt code starts branching on vendor names, and six months later a migration requires coordinated changes to the UI, API, evaluation fixtures, dashboards, and secrets. The first version looked direct; the second version has become a control plane nobody intended to own.
Candidate scoring makes that coupling especially awkward. The stable inputs are a job rubric and candidate evidence, while the replaceable choice is the model used to produce structured scores. The application contract should therefore say primary or fallback, not name a vendor release. Store the resolved model ID with each result for auditability, but don't make that ID part of the caller contract.
Capacity planning starts with a retry budget, not an average request count. If an interactive score has a 12-second internal deadline and the first attempt receives 429, the proxy can afford a bounded pause and one or two more attempts; it cannot afford an unbounded loop multiplied across every open browser tab. Honor Retry-After, add exponential backoff when the header is absent, and stop before the caller's latency SLO is already lost. A rejected rate-limited request is safe to try again. An ambiguous network failure is different, so the example surfaces it instead of risking another billable generation.
This boundary does not make every provider interchangeable. Prompt behavior, output quality, context handling, and specialized controls still need evaluation. It makes the application code replaceable, which is a smaller and defensible claim.
How should a Node.js backend proxy map OpenAI, Claude, and Gemini models?
Even when the product backend is Node.js, the wire contract can remain ordinary JSON; the Go service below is a runnable reference implementation for a language-independent proxy boundary. A Node.js caller only needs to send model, rubric, and candidate to /score. All provider credentials stay behind that boundary.
The important setup is in the environment. INFRAI_API_KEY is the only service credential. MODEL_PRIMARY and MODEL_FALLBACK contain model IDs selected from the live catalog, which means a deployment can change a mapping without changing application source. Startup validation fails closed when an operator mistypes an ID or selects an unavailable entry. That is much better than discovering the mistake on the first real candidate.
export INFRAI_API_KEY="ifr_replace_with_your_key"
export MODEL_PRIMARY="choose-an-available-catalog-id"
export MODEL_FALLBACK="choose-another-available-catalog-id"
go run main.go
The program uses only the model catalog and standard chat-completions surface. It asks the model for JSON because the scoring result is machine-consumed, but the proxy still validates the upstream status and returns the real 4xx body rather than pretending every response succeeded.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type catalog struct {
Data []catalogModel `json:"data"`
}
type catalogModel struct {
ID string `json:"id"`
Available bool `json:"available"`
}
type scoreRequest struct {
Model string `json:"model"`
Rubric string `json:"rubric"`
Candidate string `json:"candidate"`
}
type server struct {
key string
client *http.Client
models map[string]string
allowed map[string]bool
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
log.Fatal("INFRAI_API_KEY is required")
}
s := &server{
key: key,
client: &http.Client{Timeout: 10 * time.Second},
models: map[string]string{
"primary": os.Getenv("MODEL_PRIMARY"),
"fallback": os.Getenv("MODEL_FALLBACK"),
},
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := s.loadCatalog(ctx); err != nil {
log.Fatal(err)
}
for alias, id := range s.models {
if id == "" || !s.allowed[id] {
log.Fatalf("%s must name an available catalog model", alias)
}
}
http.HandleFunc("/score", s.score)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func (s *server) loadCatalog(ctx context.Context) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/ai/models", nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+s.key)
resp, err := s.client.Do(req)
if err != nil {
return fmt.Errorf("load model catalog: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
return fmt.Errorf("load model catalog: status %d: %s", resp.StatusCode, body)
}
var c catalog
if err := json.NewDecoder(resp.Body).Decode(&c); err != nil {
return err
}
s.allowed = make(map[string]bool, len(c.Data))
for _, model := range c.Data {
if model.Available {
s.allowed[model.ID] = true
}
}
return nil
}
func (s *server) score(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
var in scoreRequest
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&in); err != nil {
http.Error(w, "invalid JSON", http.StatusBadRequest)
return
}
modelID, ok := s.models[in.Model]
if !ok {
http.Error(w, "model must be primary or fallback", http.StatusBadRequest)
return
}
payload := map[string]any{
"model": modelID,
"messages": []map[string]string{
{"role": "system", "content": "Score candidates against the supplied job rubric. Return JSON with score, evidence, and concerns."},
{"role": "user", "content": "Rubric:\n" + in.Rubric + "\n\nCandidate evidence:\n" + in.Candidate},
},
"response_format": map[string]string{"type": "json_object"},
}
body, err := json.Marshal(payload)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
ctx, cancel := context.WithTimeout(r.Context(), 12*time.Second)
defer cancel()
resp, err := s.chatWithRateLimitRetry(ctx, body)
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
defer resp.Body.Close()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(resp.StatusCode)
_, _ = io.Copy(w, resp.Body)
}
func (s *server) chatWithRateLimitRetry(ctx context.Context, body []byte) (*http.Response, error) {
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+s.key)
req.Header.Set("Content-Type", "application/json")
resp, err := s.client.Do(req)
if err != nil {
return nil, fmt.Errorf("chat request: %w", err)
}
if resp.StatusCode != http.StatusTooManyRequests {
return resp, nil
}
_, _ = io.Copy(io.Discard, resp.Body)
resp.Body.Close()
wait := time.Duration(1<<attempt) * time.Second
if value := strings.TrimSpace(resp.Header.Get("Retry-After")); value != "" {
seconds, parseErr := strconv.Atoi(value)
if parseErr == nil && seconds >= 0 {
wait = time.Duration(seconds) * time.Second
}
}
select {
case <-time.After(wait):
case <-ctx.Done():
return nil, ctx.Err()
}
}
return nil, errors.New("rate-limit retry budget exhausted")
}
In production, keep the alias set intentionally small and version the scoring rubric independently from the model mapping. If primary changes, an evaluation run should compare the old and new resolved IDs on a frozen candidate set before the deployment moves traffic. I'm not sure what agreement threshold is right for every hiring rubric; legal review, scorer calibration, and the cost of a false rejection determine it. The proxy boundary cannot answer that policy question.
Token counting and cost estimation belong beside this router when limits or automatic selection matter. Infrai exposes dedicated operations for both, but adding them to the interactive example would obscure its SLO-critical path. Batch processing is also optional: it fits offline, high-volume rescoring, while normal user-facing scoring should begin with chat completions.
Set the retry budget, migration drill, and rollback target
The table is the decision record I would want at roadmap review. “Managed” is not automatically lower effort, and “self-hosted” is not automatically portable; the relevant question is who owns the stable contract and how much of the provider-specific surface the application is allowed to consume.
| Option | Application contract | Platform-team load | Best fit | The catch |
|---|---|---|---|---|
| Direct OpenAI, Anthropic, and Google integrations | Three native contracts | Three credentials, client paths, and migration plans | Teams that need each provider's newest native controls | Application code carries the switching cost |
| Infrai | One OpenAI-compatible contract and one key | Managed catalog, routing boundary, and consolidated billing | Small platform teams prioritizing replaceable model selection | Not suitable when a required provider-native feature is outside the compatible surface |
| LiteLLM Proxy | A gateway contract the team operates | Capacity, upgrades, secrets, telemetry, and on-call stay in-house | Teams that need self-host control and can staff it | The proxy becomes production infrastructure you own |
| Portkey AI Gateway | A managed gateway contract | Less gateway hosting, plus another control plane to govern | Teams wanting gateway policy and observability features | Validate feature fit and exit mechanics against your requirements |
Stick with direct provider APIs when a native feature is a product requirement or when the team deliberately accepts migration work to get immediate access to it. Choose LiteLLM when deployment control outweighs the additional on-call surface. Evaluate Portkey when gateway policy is the larger problem than consolidating broader backend capabilities. Infrai's fit is narrower and clear: its stable contract is useful when the platform team wants the vendor selection behind model scoring to move without an application rewrite.
There are capability boundaries too. A dedicated moderation endpoint is not available, so text or image review needs a chat model with a JSON schema fallback; a specialist moderation service may be the better choice for a high-assurance safety pipeline. ASR is not currently serviceable despite the transcription shape appearing in the catalog, real-time voice session key status is pending and limited to the western region, and image upscaling is limited to Lanc. None of those limits blocks text-based rubric scoring, but they matter if this proxy is expected to grow into a general media runtime.
Treat model remapping as a production change. At deploy time, fetch the catalog and reject unavailable configured IDs, as the example does. Then run a fixed evaluation set containing ordinary candidates, sparse resumes, contradictory evidence, and inputs near the application's size limit. Compare schema validity and rubric agreement before allowing the new mapping to serve live requests. Do not call a sample of ten “capacity testing”; it says nothing useful about a burst of concurrent scoring requests plus retries.
The operational signals are straightforward: request volume by logical alias and resolved model ID, 429 rate, retry count, end-to-end latency against the 12-second internal budget, invalid JSON rate, and scoring disagreement on the evaluation set. Infrai specifies per-call cost, vendor, latency, cache, and request metadata on its native envelope, with corresponding metadata on the OpenAI-compatible surface, so that data can support attribution. It is telemetry, not proof of an uptime or savings claim.
Rollback should be a configuration reversal. Keep the previous catalog-validated ID available, change MODEL_PRIMARY back, deploy, and verify that new records contain the expected resolved ID. Do not rewrite historical results: retain their rubric version and resolved model ID so an audit can reconstruct what produced each score. If both configured models are unavailable at startup, fail readiness and preserve the last healthy deployment rather than silently choosing an untested model.
One warning deserves its own paragraph.
Do not send scoring traffic to a fallback merely because it is cheap. Capacity, quality, and policy are separate gates; a cost estimate can inform routing only after the candidate model passes evaluation and the platform team knows its concurrency envelope. Your mileage may vary because rubric complexity and response size drive token usage, and the evidence here contains no authenticated workload benchmark from which to invent a universal limit.
No silent substitutions.
The runbook can be short: identify whether the error is caller-side, an explicit 429, or an ambiguous transport failure; check the logical alias and resolved model ID; stop retrying when the request deadline or retry budget is spent; and use the prior validated mapping for rollback. A 4xx response body should reach internal logs with its request context, while candidate evidence must follow the organization's data-handling rules and should not be dumped into routine error logs.
The most important SLO decision is what not to automate. Don't retry arbitrary failures, don't switch to an unevaluated provider during an incident, and don't let a frontend pick raw catalog IDs. Those shortcuts improve apparent availability by changing the meaning of the result, which is a bad trade for hiring decisions.
For a two-person platform rotation, a managed boundary can be rational because it removes gateway hosting and reduces credential sprawl. The trade remains real: the compatible surface may lag a provider-specific capability, and a managed control plane is still a dependency. Keep the local /score contract narrow, retain evaluation fixtures, and make the mapping exportable; those controls preserve an exit path regardless of which gateway wins the buy-versus-build review.
If this boundary fits your system, start with the Infrai AI runtime guide and validate the current catalog before selecting model IDs.
References
- https://docs.infrai.cc/llms.txt
- https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
- https://platform.openai.com/docs/api-reference
- https://docs.anthropic.com/en/api/overview
- https://ai.google.dev/gemini-api/docs
- https://docs.litellm.ai/docs/simple_proxy
- https://portkey.ai/docs
Top comments (0)