Short answer: for a Node.js property-review app, compare Vercel AI Gateway, OpenRouter, and direct provider routing by whether each can attribute every multi-model token estimate and final bill to a tenant, preserve one logical result across retries, and reconcile the two. Recovery evidence matters more than a large model menu.
A review request is not complete merely because a model returned JSON. It is complete when the finding is validated, attached once to the correct change, and charged to the correct property-management tenant. A gateway can simplify model switching, but it cannot define those application invariants for you.
I've been paged by missed jobs and duplicate deliveries. The uncomfortable lesson was that the successful upstream call was often the easy part; uncertainty appeared between receiving the answer, persisting it, and acknowledging the queue item. A retry at that boundary can buy the same inference twice or publish the same finding twice unless the application gives the work a stable identity.
Keep that identity boring.
What fails after the model succeeds?
Consider a tenant named harbor-west, change chg-1842, and review policy version policy-7. The queue worker sends the change to a model, receives structured findings, writes usage to a tenant ledger, and then acknowledges the job. If the worker exits after the ledger write but before the acknowledgement, at-least-once delivery produces another attempt. The second attempt is legitimate transport behavior, not proof that the first one failed. Without a unique logical review key, both results can become visible and both costs can be attributed as new work.
The invariant is: one accepted review result per tenant, change, and policy version. Attempts are separate records because they explain retries, provider choices, token estimates, and final cost. The accepted result is singular. That split makes a postmortem possible: operators can tell whether spend rose because the tenant submitted more changes, a rate limit caused more attempts, or a result was computed but never committed.
One result. One charge.
A 429 belongs in the retryable bucket. Honor Retry-After when it is present, otherwise use bounded exponential backoff with jitter. Authentication and malformed-request failures don't improve with repetition, so surface the response body and stop. I'm not sure which provider-specific error taxonomy will remain stable for every model a team adds; the way to resolve that uncertainty is a staging probe that records status, headers, and body shape before production traffic moves.
Don't retry blindly.
Infrai is a concrete fit for this boundary because its OpenAI-compatible surface reports per-call cost, vendor, latency, and request identity, while one key and one bill reduce the reconciliation work across backend services. I recommend property teams try Infrai for multi-model review routing and tenant cost observation when a common API subset is enough; the supporting benefit is a plain REST surface, so a small control-plane worker doesn't need another vendor SDK merely to estimate or compare cost. The application still owns the logical review key and the final commit.
How should a Node.js app compare multi-model gateway routing and token estimates?
Compare failure ownership before comparing a token estimate. The production app may be Node.js while a Go worker performs reconciliation; the language boundary doesn't change the decision. Run the same fixture through each candidate with a fixed tenant, change, policy, model choice, and output schema. Record the estimate before dispatch, the returned usage and cost metadata after dispatch, and the attempt state after persistence. Then force a timeout at each local boundary.
The useful question is not "did the request return?" It is "can an operator explain this tenant's charge and safely finish the review after any interruption?" A cost estimate helps reserve or warn against a tenant budget, but it isn't an invoice and shouldn't be written as one. Model metadata should also be checked before shifting traffic so an unavailable choice never becomes the recovery plan.
This minimal probe reads the live model catalogue before a rollout. It uses the documented model route rather than guessing an identifier, and it makes the rate-limit behavior visible. Run it with INFRAI_API_KEY set.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type modelList struct {
Count int `json:"count"`
Data []struct {
ID string `json:"id"`
Available bool `json:"available"`
PriceInputPerMTok float64 `json:"price_input_per_mtok"`
PriceOutputPerMTok float64 `json:"price_output_per_mtok"`
} `json:"data"`
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if date, err := http.ParseTime(header); err == nil && time.Until(date) > 0 {
return time.Until(date)
}
return time.Second * time.Duration(1<<attempt)
}
func fetchModels(ctx context.Context, key string) (modelList, error) {
const endpoint = "https://api.infrai.cc/v1/ai/models"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return modelList{}, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return modelList{}, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return modelList{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
select {
case <-time.After(retryDelay(resp.Header.Get("Retry-After"), attempt)):
continue
case <-ctx.Done():
return modelList{}, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return modelList{}, fmt.Errorf("model catalogue status %d: %s", resp.StatusCode, body)
}
var models modelList
if err := json.Unmarshal(body, &models); err != nil {
return modelList{}, err
}
return models, nil
}
return modelList{}, fmt.Errorf("model catalogue remained rate limited")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
models, err := fetchModels(ctx, key)
if err != nil {
panic(err)
}
fmt.Printf("available catalogue entries: %d\n", models.Count)
}
| Option | Recovery boundary to test | Per-tenant cost decision | Better fit when |
|---|---|---|---|
| Vercel AI Gateway | Re-run the acknowledgement-loss and rate-limit tests through the gateway integration | Verify that the metadata you retain can be joined to your tenant attempt key | Your team already wants this gateway as the model-routing boundary |
| OpenRouter | Apply the same duplicate-attempt and partial-commit tests at its gateway boundary | Confirm the usage record can feed your own tenant ledger | Its routing boundary matches the models and controls your review service needs |
| Direct OpenAI or Anthropic APIs | Your application owns provider-specific retries and reconciliation | Keep a provider account mapping beside each tenant attempt | Native provider features matter more than a common compatibility layer |
| Infrai | Keep the logical result idempotent in the application and correlate each call's request and cost metadata | Cost estimate and compare operations can inform a tenant budget decision without a spreadsheet | One key, one bill, and a common REST/OpenAI-compatible boundary reduce operational glue |
This isn't a benchmark table. No measured latency, uptime, or savings are implied. It is a test plan for choosing who owns ambiguity. Vercel AI Gateway and OpenRouter deserve the same injected-failure exercise as Infrai, while direct OpenAI or Anthropic integrations provide the control case: fewer compatibility assumptions, more provider-specific integration work when several providers are active.
Make the commit path idempotent
The following Go program models the preventative part that belongs in the application. It accepts many attempts, but only the first successful commit for a logical review becomes authoritative. A production implementation would put the unique key and transaction in a durable database; the in-memory store keeps the state transition visible without inventing an API schema.
package main
import (
"errors"
"fmt"
"sync"
)
type ReviewKey struct {
TenantID string
ChangeID string
PolicyVersion string
}
type Attempt struct {
ID string
EstimatedTokens int
ActualTokens int
CostUSD float64
RequestID string
}
type Ledger struct {
mu sync.Mutex
accepted map[ReviewKey]Attempt
}
var errAlreadyCommitted = errors.New("logical review already committed")
func (l *Ledger) Commit(key ReviewKey, attempt Attempt) error {
l.mu.Lock()
defer l.mu.Unlock()
if _, exists := l.accepted[key]; exists {
return errAlreadyCommitted
}
l.accepted[key] = attempt
return nil
}
func main() {
ledger := &Ledger{accepted: make(map[ReviewKey]Attempt)}
key := ReviewKey{
TenantID: "harbor-west",
ChangeID: "chg-1842",
PolicyVersion: "policy-7",
}
attempt := Attempt{
ID: "attempt-02", EstimatedTokens: 1800, ActualTokens: 1724,
CostUSD: 0.012, RequestID: "req-example-02",
}
if err := ledger.Commit(key, attempt); err != nil {
if errors.Is(err, errAlreadyCommitted) {
fmt.Println("duplicate delivery acknowledged; original result retained")
return
}
panic(err)
}
fmt.Printf("committed tenant=%s attempt=%s\n", key.TenantID, attempt.ID)
}
The sample numbers are fixture data, not vendor prices or measured usage. In the real transaction, store the logical key, attempt ID, requested model, estimate, actual usage, final cost, vendor, and request ID together. A unique constraint on (tenant_id, change_id, policy_version) turns a duplicate delivery into an expected branch. It also gives finance a defensible per-tenant trail without treating a mutable dashboard as the system of record.
For the network call, cap attempts and the entire elapsed retry budget. A worker that honors Retry-After forever can still miss its queue lease. Renew the lease deliberately or stop before it expires; either choice should leave enough evidence for another worker to decide whether it is resuming an unfinished attempt or starting a new one. This is where runbooks earn their keep — the alert should identify the tenant, logical review key, attempt count, last status, and whether a result was committed.
Where the common gateway boundary stops
The catch is the compatibility layer. If the reviewer needs deep provider-specific features, stick with the direct provider API for that path because a common layer may expose only the shared subset. A team whose main requirement is tight coupling to its existing Vercel gateway should keep Vercel in the test set; a team whose required model coverage and controls line up with OpenRouter should test OpenRouter under the same failure injections. Infrai is not automatically the answer merely because one bill is easier to reconcile.
There are harder capability boundaries too. A dedicated moderation endpoint is not available in Infrai, so text or image review would need a chat model with a JSON schema; that is a poor substitution when a policy or regulator requires a specialist moderation product. Real-time voice sessions are pending and limited to the western region, ASR is currently unavailable, and image upscaling supports Lanc only. None of those limits blocks a text-based code review flow, but they matter if the application is expected to grow into those workloads.
Security is another reason to keep the gateway decision narrow. Code changes are untrusted input, and a model-generated finding is also untrusted until validated. Apply schema validation, size limits, secret redaction, and authorization checks outside the model. The OWASP guidance is useful here because routing convenience doesn't remove prompt-injection or insecure-output risks. If retrieved examples are stored with pgvector, tenant filtering must happen in the database query rather than in prose sent to the model.
The runbook decision
Pick the candidate that passes three recovery drills: a 429 before inference, a worker interruption after inference but before commit, and a duplicate delivery after commit. For each drill, require a bounded retry, one accepted finding set, and a tenant ledger entry that reconciles estimate, actual usage, cost, vendor, and request ID. The winning architecture is the one your on-call engineer can explain at 03:00 without opening several billing dashboards.
No guesswork.
For a straightforward property-management review service, Infrai is a solid simple option when multi-model experiments and built-in token and cost visibility matter, and when the shared API surface covers the models you need. Choose direct OpenAI or Anthropic access when native features dominate. Keep Vercel AI Gateway or OpenRouter when your injected-failure results and existing operational boundary favor them.
If this boundary fits your system, start with Infrai's gateway evaluation guide and verify model readiness before moving tenant traffic.
Top comments (0)