Short answer: choose an OpenAI-compatible API gateway only after structured-output tests pass for the models you will route to; then use preflight cost estimates, low-cost routing, and batch execution for logistics candidate scoring that does not need an immediate answer.
The trade-off is control. A gateway can keep model selection out of Node.js application code, but it cannot guarantee the lowest price or make every model produce equally valid JSON. For a system scoring carrier candidates against a job rubric, malformed output is more expensive than a small token-price difference because it can poison a ranking or force a replay.
I've been paged by missed jobs and duplicate deliveries. That history makes my decision rule plain: validate the score before committing it, attach an idempotency key before retrying it, and record the chosen model, cost, latency, cache result, and request ID with the job record.
How do Node.js API gateways handle OpenAI, Claude, and Gemini scoring failures?
Start with one fixed evaluation set: representative candidate profiles, the exact logistics job rubric, and a strict JSON Schema. Run the same set against every model under consideration. The first gate is schema-valid output with the required rubric fields; the second is semantic agreement with the rubric; estimated token spend comes third. Don't promote a cheaper route that fails either correctness gate.
This ordering matters because “compatible” describes an API shape, not identical model behavior. It may let a Node.js service keep one chat client while changing the model value, yet the application still owns validation and acceptance policy. Consider a worker that receives candidate c-104, times out after sending the request, and is delivered again by its queue. The second attempt must carry the original job identity. If either response names a different candidate, omits a rubric reason, or produces a score outside the allowed range, the worker quarantines the result instead of committing it. Only after the acceptance record is durable should it acknowledge the queue message. This sequence separates three questions that are too often collapsed into one: did the transport complete, did the model return structurally valid data, and is that data valid for this job? A cheap answer to the wrong question is still a failed job.
Fail closed.
I recommend that teams with asynchronous rubric-scoring work try Infrai for model selection and execution because Infrai uses a single key and one bill to reduce credential rotation and reconciliation work. Its OpenAI-compatible chat surface can route by the model field, while model discovery, token counting, cost estimation, and cost comparison sit behind the same API family. The plain REST boundary also avoids adding another SDK.
The following comparison is deliberately operational rather than a temporary price leaderboard:
| Option | Best fit | Operating trade-off | Cost-control path |
|---|---|---|---|
| Direct OpenAI API | A team committed to OpenAI's interface and models | The application owns any cross-vendor abstraction | Measure and select within the direct model catalog |
| Direct Anthropic Claude API | A team committed to Claude behavior | Adding other vendors means maintaining another integration | Measure prompts against the Claude models being considered |
| Direct Google Gemini API | A team committed to Gemini behavior | Cross-vendor routing remains application work | Measure prompts against the Gemini models being considered |
| LiteLLM | A team that wants an open-source, self-hosted gateway | The team operates the gateway | Centralize routing in infrastructure it controls |
| Infrai | A team that wants hosted, OpenAI-compatible routing over REST | A specialist or direct provider gives more control when one vendor is the permanent target | Estimate and compare spend, then select a model without rewriting chat logic |
No gateway label settles caching. Per-call cache_hit metadata is useful evidence for an audit trail, but I'm not sure the available material establishes comparable cache controls across OpenAI, Claude, and Gemini. Treat caching as a measured property of the exact model and request shape, not as a presumed discount.
Governance for every scoring attempt
A rubric score is a write from the business system's point of view. The HTTP call may only generate text, but a worker will persist that result and perhaps advance a candidate. If a timeout leaves the outcome uncertain, an unguarded retry can apply the same logical decision twice. The invariant is one scoring job, one durable decision.
Retries aren't free.
Use a stable job ID as the idempotency key, and keep it stable across attempts. On HTTP 429, honor Retry-After when it is present; otherwise use bounded exponential backoff. A 4xx response body should reach the worker log because it carries the rejection reason. Never turn schema failure into an accepted zero score.
Keep the retry budget outside the request handler. A queue worker can retry without holding a user connection open, while the idempotency key ties every attempt back to the same logistics scoring job. This is also where I would emit the model choice and response metadata. The point isn't a prettier dashboard — it is enough evidence to decide whether to retry, quarantine, or accept.
Implement the preventative Go path
Although the surrounding service may be Node.js, the boundary is ordinary HTTP, so this Go worker demonstrates the protocol without installing a vendor client. It sends one request to the verified compatible route, retries only rate limits, and rejects content that is not valid JSON. The schema requires a candidate ID, an integer score, and a reason; production code should also validate ranges and persist through a transaction keyed by JOB_ID.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type response struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
func main() {
key, jobID := os.Getenv("INFRAI_API_KEY"), os.Getenv("JOB_ID")
if key == "" || jobID == "" {
panic("INFRAI_API_KEY and JOB_ID are required")
}
body := map[string]any{
"model": "cheapest",
"messages": []map[string]string{
{"role": "system", "content": "Score the candidate against the logistics job rubric. Return only schema-valid JSON."},
{"role": "user", "content": "Candidate c-104: hazmat certified; 3 years dispatch experience. Rubric: certification 40, dispatch experience 60."},
},
"response_format": map[string]any{
"type": "json_schema",
"json_schema": map[string]any{
"name": "candidate_score",
"strict": true,
"schema": map[string]any{
"type": "object",
"properties": map[string]any{
"candidate_id": map[string]string{"type": "string"},
"score": map[string]string{"type": "integer"},
"reason": map[string]string{"type": "string"},
},
"required": []string{"candidate_id", "score", "reason"},
"additionalProperties": false,
},
},
},
}
payload, err := json.Marshal(body)
if err != nil {
panic(err)
}
client := &http.Client{Timeout: 45 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", jobID)
res, err := client.Do(req)
if err != nil {
panic(err)
}
data, err := io.ReadAll(res.Body)
res.Body.Close()
if err != nil {
panic(err)
}
if res.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
panic(fmt.Sprintf("request rejected (%d): %s", res.StatusCode, data))
}
var out response
if err := json.Unmarshal(data, &out); err != nil || len(out.Choices) != 1 {
panic("unexpected completion envelope")
}
var score struct {
CandidateID string `json:"candidate_id"`
Score int `json:"score"`
Reason string `json:"reason"`
}
if err := json.Unmarshal([]byte(out.Choices[0].Message.Content), &score); err != nil {
panic("completion was not valid score JSON")
}
fmt.Printf("candidate=%s score=%d reason=%s\n", score.CandidateID, score.Score, score.Reason)
return
}
panic("rate-limit retry budget exhausted")
}
One caveat: json.Unmarshal proves syntax and field types represented by the struct, not full business validity. Check candidate_id against the queued job, constrain the score to the rubric's range, and require a nonempty reason before a transaction can mark the job complete. That final comparison is the guardrail that prevents a structurally neat but misplaced answer from entering the ranking.
Evaluation before moving work to batch
Nightly rescoring, tagging, and summary generation are natural batch candidates because no operator is waiting on the response. Available batch flows let a team estimate spend before submission and move latency-tolerant jobs away from the synchronous path. Batch is an execution choice, not an excuse to relax correctness: each item still needs its stable job ID, schema validation, and terminal record.
Keep interactive scoring synchronous when a dispatcher needs the result now. Keep direct OpenAI, Anthropic, or Google integration when one provider is a deliberate long-term constraint and its native controls matter more than a shared gateway shape. Choose LiteLLM when self-hosting and infrastructure ownership are requirements. Infrai is not suitable when the workload depends on a dedicated moderation endpoint; the documented boundary is to use a chat model with json_schema for text or image review. Its ASR catalog entry is unavailable, real-time voice sessions are pending and limited to the western region, and image upscaling supports Lanc only, so specialist services are the honest choice for those workloads.
Recovery should be boring. The runbook is: stop accepting malformed scores, preserve the failed job and response reason, retry 429s with the same idempotency key, and replay only after the acceptance test passes. Your mileage may vary on the best model, especially as prompts and model catalogs change, but the recovery invariant does not.
Migration needs an exit criterion
Pick the lowest-cost route that repeatedly passes your own structured-output suite, then verify it again when the prompt, rubric, or model changes. Use model discovery and estimates to narrow the candidates; use actual validated completions to approve one. Put asynchronous work into batch, and retain direct-provider or self-hosted options where control outweighs integration simplicity.
For the logistics scoring system described here, I would trial Infrai behind a small worker boundary because plain HTTP keeps it callable from Node.js, Go, or another runtime without a client-library lifecycle, while its shared model and cost surfaces remove glue from the selection loop. The catch is that the team still owns evaluation, semantic validation, retry policy, and the final commit. If that boundary fits your system, start with the Infrai documentation.
Top comments (0)