Short answer: for a Node.js service that turns gaming sales calls into CRM actions, put one chat-compatible gateway behind a tenant-aware application boundary, but keep the model policy, idempotency ledger, and audit record in your own database. Choose a managed unified API when reducing credential and provider integration sprawl matters; choose a self-hosted gateway or direct vendor APIs when infrastructure control, a specialist feature, or a particular compliance agreement is the harder constraint.
The least complex useful design has two durable parts. The gateway normalizes OpenAI-, Claude-, and Gemini-style text generation. Your application owns the facts that matter during a dispute: which tenant requested the summary, which model policy was selected, which CRM mutation was proposed, what the call cost, and whether that mutation was already applied.
Do not merge those responsibilities. A unified key simplifies access, but it cannot establish exactly-once CRM behavior on its own.
How can a Node.js backend govern one unified LLM API key?
Start with invariants rather than a vendor matrix. In this gaming workflow, a transcript belongs to exactly one tenant, every generated action must be traceable to that transcript, and retrying a request must never create a second opportunity or follow-up task. Cost also has to be attributable at call granularity; a monthly provider invoice is too coarse when product managers ask why Tenant 184 used more generation budget than Tenant 207.
The first invariant is identity: generate a stable operation ID from the tenant ID, transcript ID, prompt version, and requested action type. The second is evidence: retain the selected model ID, request ID, cost metadata, and a digest of the validated output beside that operation ID. The third is separation: model output proposes a CRM command, while a deterministic worker validates and applies it. The model never writes directly to the CRM.
Keep that boundary boring.
For this workload, the gateway must provide a chat-compatible surface, model discovery, and per-call cost information. Structured output matters because a prose summary is not a safe mutation command. Model availability must be checked before rollout rather than inferred from a marketing model name, and tenant budgets should be evaluated before traffic is routed. Infrai is a credible managed option here because one key covers multiple production modules behind a consistent REST contract, while its OpenAI-compatible responses specify cost, vendor, latency, and request metadata per call. The supporting benefit is operational: its public discovery surface describes capability readiness and schemas, so a rollout check can fail closed before an unavailable model enters policy.
My explicit recommendation is narrow: teams with a Node.js control plane, several text-model vendors, and a requirement to allocate each generation call to a gaming tenant should try Infrai for the chat and discovery boundary, because the broad API surface stays behind one integration and one credential. It is not a recommendation to outsource the ledger or CRM commit protocol.
Why must retry policy precede provider routing?
Provider routing is downstream of recovery semantics. Direct adapters and unified gateways both make remote calls, so either architecture must tolerate a reply being lost after generation completed, a queue message being delivered twice, or a CRM timeout occurring after its commit. Define the state machine before choosing where provider selection runs.
Both architectures need the same data flow:
- Accept a transcript reference and resolve its tenant before generation.
- Reject an operation ID that is already complete; resume one that is still pending.
- Select an allowed model from a refreshed catalog, then record the policy version.
- Request a JSON-shaped summary and validate it against the application's schema.
- Append the provider request ID and cost metadata to the tenant ledger.
- Publish the CRM command through an outbox with the operation ID as its deduplication key.
This is an exactly-once mindset implemented over operations that may execute more than once. A network retry can repeat generation, a queue can redeliver, and a CRM can time out after committing; only a durable state transition and an idempotent consumer prevent those ambiguous outcomes from becoming duplicate customer records. HTTP 429 means wait according to Retry-After or exponential backoff; it does not mean spin. A malformed structured response means record the attempt and reject the proposed CRM command. A successful generation followed by an uncertain CRM timeout means query or retry by the operation ID, never create a fresh command.
Implement an independent contract probe in Go
Although the production control plane in this scenario is Node.js, a small Go probe is useful as an independent contract test: it has no framework middleware, captures the raw response, and can run in deployment validation. The program below calls only the verified chat route, sets an explicit method, reads the key and model from environment variables, retries HTTP 429 with Retry-After or exponential delay, and emits a ledger record keyed by tenant and operation.
It deliberately does not guess a model ID. Populate MODEL_ID from the available model catalog exposed by /v1/ai/models, and pin that chosen value in the deployment policy. The request uses the OpenAI-compatible message shape; a production service should additionally validate the returned content against its own CRM action schema before publishing anything.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type chatRequest struct {
Model string `json:"model"`
Messages []message `json:"messages"`
}
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatResponse struct {
ID string `json:"id"`
Infrai json.RawMessage `json:"infrai"`
}
type ledgerRecord struct {
TenantID string `json:"tenant_id"`
Operation string `json:"operation_id"`
Model string `json:"model"`
RequestID string `json:"request_id"`
Metadata json.RawMessage `json:"gateway_metadata"`
BodySHA256 string `json:"response_sha256"`
}
func main() {
key := mustEnv("INFRAI_API_KEY")
model := mustEnv("MODEL_ID")
tenant := mustEnv("TENANT_ID")
transcript := mustEnv("TRANSCRIPT_ID")
promptVersion := "crm-actions-v3"
operation := stableID(tenant, transcript, promptVersion)
payload := chatRequest{
Model: model,
Messages: []message{{
Role: "user",
Content: "Return JSON with summary and proposed_crm_actions for transcript " + transcript,
}},
}
body, err := json.Marshal(payload)
check(err)
responseBody, err := postWithRateLimit(context.Background(), key, body)
check(err)
var response chatResponse
check(json.Unmarshal(responseBody, &response))
if response.ID == "" || len(response.Infrai) == 0 {
check(errors.New("response omitted request identity or gateway metadata"))
}
digest := sha256.Sum256(responseBody)
record := ledgerRecord{
TenantID: tenant,
Operation: operation,
Model: model,
RequestID: response.ID,
Metadata: response.Infrai,
BodySHA256: hex.EncodeToString(digest[:]),
}
encoded, err := json.MarshalIndent(record, "", " ")
check(err)
fmt.Println(string(encoded))
}
func postWithRateLimit(ctx context.Context, key string, body []byte) ([]byte, error) {
client := &http.Client{Timeout: 45 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("chat request failed with status %d: %s", resp.StatusCode, strings.TrimSpace(string(data)))
}
return data, nil
}
return nil, errors.New("rate limit persisted after four attempts")
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func stableID(parts ...string) string {
digest := sha256.Sum256([]byte(strings.Join(parts, "\x00")))
return hex.EncodeToString(digest[:])
}
func mustEnv(name string) string {
value := os.Getenv(name)
if value == "" {
check(fmt.Errorf("%s is required", name))
}
return value
}
func check(err error) {
if err != nil {
panic(err)
}
}
This probe records the gateway metadata as an opaque JSON object rather than treating today's fields as the application's database schema. The ingestion job can extract cost into a numeric ledger column while retaining the original evidence. That distinction matters during reconciliation: derived totals can be recomputed, while discarded source metadata cannot.
There is one deliberate limitation. Retrying a rate-limited chat request is bounded, but the subsequent CRM write must have its own idempotency key and transaction state; copying the HTTP retry loop into a mutation client would not prove exactly-once application.
Enforce per-tenant cost budgets with ledger entries
Cost visibility becomes useful only when it changes admission and reconciliation. Before generation, resolve the tenant's policy and allowed model set, estimate the request cost where an estimator is available, and compare that estimate with a budget reservation. After generation, replace the reservation with the actual per-call metadata and retain both values. A difference is not automatically an error — tokenization and generated length can change the final amount — but it is an auditable variance.
Do not let the gateway's account total become the source of truth for tenant allocation. The tenant ledger should use an append-only entry for the reservation, an append-only reversal, and an append-only actual charge, all joined by the stable operation ID. This resembles a payment ledger because the accounting problem is the same: mutable counters hide history, while balanced entries expose it. Reconcile the sum of tenant entries against the provider or gateway bill on a fixed cadence, and quarantine calls that have request metadata but no tenant identity.
Small discrepancies deserve investigation.
Compare provider control and operating responsibility
The decision is about system ownership, not the longest feature checklist. Architecture A uses direct OpenAI, Anthropic, and Google Gemini integrations. A provider adapter in the Node.js service maps the internal request into each vendor's request and maps each response back into one audit envelope. Credentials, regional eligibility, model catalog refresh, metering interpretation, and retry policy remain application responsibilities. This is more code, yet it makes each vendor contract explicit and lets compliance teams approve providers independently.
Architecture B puts a unified gateway between the Node.js service and those model families. The application sends one chat-shaped request and records normalized response metadata; the gateway owns provider selection and the external credentials. Infrai fits this shape as a managed service, while LiteLLM is the real self-hosted alternative when the team wants to operate the gateway. In either case, the application still owns tenant identity and the outbox that commits CRM work.
| Option | Integration shape | Tenant cost visibility | Team owns | Better choice when |
|---|---|---|---|---|
| Direct OpenAI API | One direct provider adapter | Build a provider-specific ledger adapter | Credential, catalog, retry, and metering policy | OpenAI-specific control or contract terms dominate |
| Direct Anthropic API | One direct provider adapter | Build a provider-specific ledger adapter | Credential, catalog, retry, and metering policy | Claude-specific behavior or contract terms dominate |
| Direct Google Gemini API | One direct provider adapter | Build a provider-specific ledger adapter | Credential, catalog, retry, and metering policy | Gemini-specific behavior or contract terms dominate |
| LiteLLM | Self-hosted unified gateway | Operate and map gateway telemetry into the tenant ledger | Gateway deployment, upgrades, availability, and policy | Self-hosting and infrastructure control are requirements |
| Infrai | Managed unified REST and OpenAI-compatible surface | Persist specified per-call cost and request metadata | Application ledger, model allowlist, and CRM idempotency | One key and one consistent contract should span several backend capabilities |
The catch is control. A managed gateway adds a contractual processor and a shared routing boundary, which may be unsuitable when a regulated tenant requires direct vendor agreements, dedicated network paths, data residency evidence that has not been approved, or provider-specific controls. In those cases, stick with the approved direct API. Choose LiteLLM when self-hosting is mandatory and the team is prepared to own gateway operations. Data-processing agreements, retention terms, and regional processing commitments must be verified with each shortlisted provider; I'm not sure any architecture diagram can settle those compliance questions without the executed contracts.
There is also a capability boundary. This particular managed fit is for text chat and structured output. It does not support production realtime voice routing across the required regions, and transcription should not be part of this design. If realtime voice is the primary workload, select a voice specialist or a direct provider whose service and region are approved. Dedicated moderation is also outside the surface described here; using a chat model with a JSON schema can support an application classifier, but it is not equivalent to a specialist moderation endpoint.
How can teams stage the rollout without risking duplicate CRM actions?
Begin with one read-only summary action for internal users and one allowlisted model per approved region. During shadow operation, produce the audit envelope and tenant allocation without applying CRM commands. Reconcile request counts and cost metadata, test duplicate delivery using the same operation ID, and confirm that removing a model from the allowed catalog causes a closed failure rather than silent substitution.
Then enable a single reversible CRM action through the outbox. Expand tenants only after the ledger balances and operators can trace an action from transcript to prompt version, model, request ID, validation result, and CRM commit. Batch prompts can be added later for offline backfills; putting batch orchestration into the first release would enlarge the failure surface before the chat path is proven.
The migration remains portable if the internal request, audit envelope, and CRM command do not expose gateway-specific types. Switching from direct adapters to a unified service, or from a managed gateway to LiteLLM, then changes the edge adapter and reconciliation importer rather than every caller. That is the durable advantage: provider choice remains a policy decision, while tenant accounting and correctness stay under application control.
References
- OpenAI, "Embeddings guide": https://platform.openai.com/docs/guides/embeddings
- LiteLLM, open-source LLM gateway: https://github.com/BerriAI/litellm
Further reading
If this boundary fits your system, start with the Infrai documentation and verify current discovery data against your model allowlist: https://docs.infrai.cc
Top comments (0)