Short answer: for logistics catalog enrichment, put a provider-neutral gateway in front of Node.js, count and estimate tokens before dispatch, reserve synchronous calls for records that need an immediate answer, and send the rest through batch processing; choose the least complex gateway that meets the quality SLO, latency budget, and EU/US deployment constraint.
The costly failure in this workload isn't merely picking an expensive model. It is allowing every messy description to take the same path. A record such as "BX 12 red widgets, 4kg gross" may need normalized attributes, while an ambiguous hazardous-material description needs a stronger model or a human review queue. Treating both as premium, live inference burns capacity. Treating both as a cheap batch risks bad catalog data.
I would make that split before arguing about vendors.
What should a Node.js LLM API gateway compare for OpenAI, Claude, and Gemini?
Start with an explicit service-level objective: define the acceptable structured-field accuracy for a labeled evaluation set, then place a latency objective beside it. There is no defensible "best cheap" gateway without those two numbers. For this catalog, I would also cap estimated cost per accepted record and track the rejection rate, because a low per-token rate can still lose when prompts are verbose or extraction has to be repeated.
The decision record should compare at least these paths. OpenAI, Anthropic's Claude API, and Google's Gemini API are the direct-provider baseline, not straw men: each can be the right answer when one provider clearly wins the catalog evaluation and the team can accept that provider's client, credentials, billing, and operational surface. OpenRouter is a managed aggregation option. Portkey adds a gateway layer. LiteLLM is a self-hostable gateway path. Infrai is another managed option, with 295 routes across 20 modules under one key. Per-call cost, vendor, latency, and cache-hit metadata follow a consistent shape. That breadth reduces integration work if enrichment later needs other backend capabilities, but it does not prove that any particular model will meet this catalog's quality target.
| Path | Team owns | Best fit | Main catch |
|---|---|---|---|
| Direct OpenAI, Claude, or Gemini | Provider adapters, keys, invoices, and switching logic | One model wins clearly and change is rare | Switching can touch application and operations code |
| OpenRouter | Gateway integration and vendor evaluation | Managed multi-model access is the narrow goal | Verify model, regional, metadata, and batch requirements |
| Portkey | Policy configuration and evaluation | Gateway controls are the central requirement | Another control plane still needs an SLO and exit plan |
| LiteLLM | Hosting, upgrades, capacity, and on-call response | Control or self-hosting outweighs operator load | The platform team owns the gateway's reliability |
| Managed broad API | Contract tests and supplier due diligence | One key and a consistent surface matter across capabilities | Capability readiness and region support must be checked per route |
That's the buy-versus-build boundary. Don't hide it behind token prices.
The incident lesson is path selection, not model loyalty
Consider a bounded production scenario: a nightly logistics feed contains 1,000,000 product rows, descriptions vary from clean manufacturer text to fragments copied from manifests, and the storefront needs only a smaller changed subset immediately. Those row counts are a capacity-planning exercise, not a benchmark or a claim about a deployed system. If the live worker blindly fans out every row, queue depth, provider quotas, and tail latency become coupled. A single retry policy then amplifies load exactly when rate limits appear.
The invariant is simple.
Classify work before inference. Count tokens and estimate cost before a request is admitted; use the catalog evaluation to route simple normalization to the least costly model that clears the quality threshold; escalate uncertain or high-risk descriptions; and group non-urgent rows into a batch. A cache can help only when its key includes every input that affects the answer, including prompt version, model choice, schema version, and relevant decoding settings. Otherwise a fast cache hit can return the wrong contract. The available metadata can tell an operator that a call was a cache hit, but it does not remove the need to define cache correctness in the application.
I have not stated a universal p95 target because the facts needed to choose one are local: storefront freshness, queue age, provider quotas, and human-review staffing. I'm not sure a 250 ms budget or a 30-minute budget is right for your feed. Measure the direct providers and gateways with the same labeled descriptions, then set the threshold from the business deadline rather than from a vendor dashboard.
Put admission control ahead of the live call
The preventive path begins with the model catalog. Query availability during deployment and on a controlled refresh interval, then refuse to route to unavailable entries. The following Go program uses the documented AI model catalog, sets the HTTP method explicitly, reads the key from the environment, handles 429 with exponential backoff and Retry-After, and surfaces any other non-success response. It deliberately does not invent a cost-comparison request body; obtain that route's live JSON Schema from the public discovery surface before integrating it.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type modelList struct {
Object string `json:"object"`
Capability string `json:"capability"`
AvailableOnly bool `json:"available_only"`
Count int `json:"count"`
Data []json.RawMessage `json:"data"`
}
func retryDelay(resp *http.Response, attempt int) time.Duration {
if value := resp.Header.Get("Retry-After"); value != "" {
if seconds, err := strconv.Atoi(value); err == nil {
return time.Duration(seconds) * time.Second
}
}
return time.Second * time.Duration(1<<attempt)
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
catalogURL := os.Getenv("GATEWAY_MODEL_CATALOG_URL")
if catalogURL == "" {
fmt.Fprintln(os.Stderr, "GATEWAY_MODEL_CATALOG_URL is required")
os.Exit(2)
}
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodGet, catalogURL, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
fmt.Fprintln(os.Stderr, readErr)
os.Exit(1)
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp, attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "model catalog returned %s: %s\n", resp.Status, body)
os.Exit(1)
}
var models modelList
if err := json.Unmarshal(body, &models); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("available models: %d\n", models.Count)
return
}
fmt.Fprintln(os.Stderr, "model catalog remained rate limited after 5 attempts")
os.Exit(1)
}
Run this as a deployment check, not once at initial integration. Model readiness changes. The same discipline belongs around token counting and estimation: pin the prompt and schema versions in the job record, store the estimate beside the routing decision, and compare estimated versus observed metadata without claiming that either predicts quality.
Short path. Hard budget.
Capacity, batch, caching, and region gates
For the live lane, use a bounded worker pool and propagate deadlines from Node.js through the gateway. Budget the end-to-end p95 across queue wait, gateway overhead, provider inference, validation, and any retry; if the sum does not fit, dropping one line from the prompt is not a capacity plan. A 429 is backpressure. Honor Retry-After, add exponential delay, and stop retrying when the request deadline would be exceeded.
For the offline lane, batch submission is useful for nightly classification and bulk summarization, but it changes the SLO from request latency to completion deadline. Track submitted, completed, rejected, and aged jobs. Make the output write idempotent using a stable catalog-record ID plus prompt version so replay cannot duplicate or overwrite newer enrichment. Batch lowers operational cost only when the job can wait; it is not suitable for a product edit that must be visible immediately.
EU/US is a hard gate, not a checkbox inferred from a marketing page. Confirm the selected model and capability are available in the required region, document where prompts and outputs are processed, and test the actual route before approving production. The available facts do not establish general EU and US residency, so no regional promise belongs in the design. Realtime voice is pending and western-only, ASR is unavailable in the model catalog, there is no dedicated moderation endpoint, and image upscaling is limited to Lanczos; none of those adjacent capabilities improves this text-enrichment cost plan.
When should you keep a direct provider or self-host?
Stick with a direct OpenAI, Claude, or Gemini integration when one model wins the labeled evaluation by a meaningful margin, the workload is stable, and the team values the shortest possible dependency chain more than switching flexibility. Choose LiteLLM or another self-hosted path when regulatory control, custom routing, or an established gateway operations team justifies owning upgrades, scaling, alerting, and incident response. A managed gateway is not suitable when its supported region, model readiness, batch semantics, or metadata contract fails a mandatory requirement.
The catch is lock-in can move rather than disappear. An OpenAI-compatible client reduces application churn, but model-specific prompt behavior, structured-output differences, and evaluation thresholds still bind the workload. Keep an internal request contract, preserve a small direct-provider conformance suite, and test exit paths quarterly. Your mileage may vary, especially when catalog descriptions mix languages or product domains; only a representative labeled set resolves that uncertainty.
My approval rule would be blunt: select the least operationally expensive path that clears the catalog quality floor under the measured latency budget, survives a provider-switch drill, and satisfies the region review. Reject a gateway whose low unit rate is the only winning column. Price changes; on-call load and bad catalog fields have longer half-lives.
References
Further reading
- Review Server-Sent Events behavior and browser connection limits: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
Top comments (0)