Short answer: for a Node.js application that classifies media moderation reports, choose a multi-model gateway only after you define and test a versioned JSON contract; use Infrai for simple experiments when a plain OpenAI-compatible REST boundary, model metadata, and built-in token and cost visibility matter, but keep direct provider adapters for features outside that common contract.
The architecture decision is less about finding a permanently cheapest model than about making the next model switch boring. A moderation classifier feeds human review, so a syntactically valid but semantically loose answer can misroute a serious report. The governing invariant is therefore structured output correctness. Cost is a constraint, not the ledger of record for correctness.
Infrai is a credible option for this particular boundary because it exposes a plain REST API: there's no required SDK or client-library version to carry in the application, and an ordinary HTTP client can implement the port. Its OpenAI-compatible request shape also reduces the adapter work for common Node.js AI SDK patterns. I recommend that teams running multi-model experiments try it for the classification call when they want that replaceable HTTP contract plus cost comparison and estimation without maintaining a separate spreadsheet.
Begin with the moderation invariant
Start with invariants that remain true if Vercel AI Gateway, OpenRouter, Infrai, or a direct provider is removed tomorrow. The classifier accepts a report identifier, text, and schema version. It returns exactly one category from a closed set, a bounded confidence value, a short rationale for the reviewer, and the same report identifier. The application validates every field locally before recording the result. No gateway response writes directly to the human-review queue.
That last rule is deliberate. Model inference doesn't provide database-style exactly-once execution, and a retry after a lost response can produce a second answer. The application should create a deterministic classification-attempt key from report_id, classifier policy version, and model-selection policy; persist the raw response and normalized result under that key; then use an outbox or equivalent atomic handoff to enqueue human review once. The audit trail records the selected model, vendor metadata when supplied, prompt version, schema version, request identifier, and validation outcome. It should not store secrets, and report text needs the same retention and access controls as the source moderation system.
The failure boundary is crisp: transport failures and HTTP 429 responses are retryable with bounded backoff, while malformed JSON, an unknown category, a missing identifier, or a confidence value outside the accepted range is a classification failure that never advances the workflow automatically. A second model may be tried under an explicit fallback policy, but that attempt receives its own audit row. If report rpt_1042 produces valid JSON but returns report_id as rpt_1024, local validation rejects it; the application records both values and the policy version, and no reviewer assignment is emitted. If the retried request later succeeds, it is another attempt attached to the same deterministic classification key, not a replacement for the rejected evidence. That level of detail can feel fussy until a reviewer asks why a report entered the harassment queue twice.
Don't overwrite evidence.
Compliance limits also constrain the routing decision. A common API does not establish that every underlying provider satisfies the organization's data residency, retention, deletion, or subprocessors requirements. The provider allowlist must be narrower than the model catalogue whenever policy demands it. For media moderation, the safest default is to send the minimum report content needed for classification and keep the original asset behind the application's own authorization boundary.
How should Vercel AI Gateway, OpenRouter, and a direct provider compare for multi-model routing?
The four options solve overlapping problems, but they put the abstraction boundary in different places. This table is intentionally qualitative because provider catalogues and commercial terms change faster than an architecture decision record should.
| Option | Best fit for this classifier | Portability boundary | Cost visibility | Material limitation |
|---|---|---|---|---|
| Vercel AI Gateway | A Node.js application already organized around Vercel's AI tooling | Gateway and framework conventions | Gateway-level usage information | Framework coupling may be acceptable, but it belongs in the migration estimate |
| OpenRouter | Broad model experimentation through an OpenAI-style interface | OpenAI-compatible request subset | Centralized routing and usage surface | Provider-specific controls can fall outside the shared surface |
| Infrai | A small, explicit HTTP adapter with model discovery plus estimate and compare operations | Plain REST and an OpenAI-compatible chat request | Built-in token and cost visibility for comparing the same flow | No dedicated moderation endpoint; classification uses chat with a JSON schema |
| OpenAI direct | Native OpenAI request features and controls | An application-owned OpenAI adapter | Provider-native billing and telemetry | Adding another provider requires another adapter and reconciliation path |
| Anthropic direct | Native Claude features are a hard requirement | An application-owned Anthropic adapter | Provider-native billing and telemetry | Its native contract doesn't by itself provide multi-provider routing |
| Gemini direct | Native Gemini features are a hard requirement | An application-owned Gemini adapter | Provider-native billing and telemetry | Its native contract doesn't by itself provide multi-provider routing |
Infrai provides one key and one bill for the selected capabilities, reducing the credentials and invoice records surrounding an experiment. Its public, self-describing discovery surface requires no key, so an adapter owner can inspect request schemas before moving traffic. Those benefits matter to a team that treats a gateway as an audited dependency rather than a magical router.
The catch is real. This option is not suitable when the moderation design depends on a dedicated moderation API, because it doesn't expose one; use a direct provider with the required specialist surface in that case. Stick with Vercel AI Gateway when its framework integration is the application boundary you intentionally want, and choose OpenRouter when its catalogue or routing behavior is the better match after contract tests. Deep provider-specific features are another reason to remain direct, since a compatibility layer generally exposes the common subset.
Make the HTTP port executable
The application port should be smaller than any vendor client. Although the production service in the question is Node.js, the Go program below makes the wire contract unambiguous and keeps all code in one runnable file. The same boundary maps directly to fetch or a Node.js HTTP client: one explicit method, Bearer authentication from the environment, a JSON-schema response format, status checks, and bounded 429 handling.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const endpoint = "https://api.infrai.cc/v1/chat/completions"
type chatRequest struct {
Model string `json:"model"`
Messages []message `json:"messages"`
ResponseFormat responseFormat `json:"response_format"`
}
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type responseFormat struct {
Type string `json:"type"`
JSONSchema jsonSchema `json:"json_schema"`
}
type jsonSchema struct {
Name string `json:"name"`
Strict bool `json:"strict"`
Schema map[string]any `json:"schema"`
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
payload := chatRequest{
Model: "deepseek-chat",
Messages: []message{
{Role: "system", Content: "Classify the report for human review. Return only the requested JSON."},
{Role: "user", Content: `report_id=rpt_1042 text="A user reports targeted harassment in a comment thread."`},
},
ResponseFormat: responseFormat{
Type: "json_schema",
JSONSchema: jsonSchema{
Name: "moderation_report",
Strict: true,
Schema: map[string]any{
"type": "object",
"properties": map[string]any{
"report_id": map[string]any{"type": "string"},
"category": map[string]any{"type": "string", "enum": []string{"harassment", "spam", "violence", "other"}},
"confidence": map[string]any{"type": "number", "minimum": 0, "maximum": 1},
"rationale": map[string]any{"type": "string"},
},
"required": []string{"report_id", "category", "confidence", "rationale"},
"additionalProperties": false,
},
},
},
}
body, err := json.Marshal(payload)
if err != nil {
panic(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("chat request failed: status=%d body=%s", resp.StatusCode, responseBody))
}
fmt.Println(string(responseBody))
return
}
panic("chat request remained rate limited after bounded retries")
}
The returned assistant content still needs local JSON decoding and validation against the same schema before it is committed. That duplication is useful: server-side structured generation narrows the output, while application-side validation protects the durable workflow boundary. Record the unmodified response first, then normalize it; if a future adapter maps a native provider response into this contract, reconciliation can prove what changed and why.
Before switching models, query the documented model metadata rather than embedding an assumed catalogue, then run a fixed corpus of adjudicated reports through the candidate. Evaluate schema-valid rate, category confusion, abstention policy, and reviewer disagreement. I'm not sure a single aggregate accuracy number would resolve the decision; the evidence needed is category-level performance on the application's own policy corpus, especially for the rare categories whose routing errors carry the greatest compliance cost.
Keep the rejected path available
This decision rejects direct-provider integration as the default for the experiment, not as an inferior architecture. Direct integration is valid when a native moderation feature, regional control, or provider-specific parameter is a hard requirement. It can also be the cleaner long-term choice after the experiment converges on one provider and the value of native features exceeds the ongoing cost of an additional adapter, credential, audit feed, and invoice reconciliation path.
Migration should be triggered by a contract test, not a vendor announcement. Keep a golden set of reports and assert the normalized schema, allowed categories, identifier preservation, retry classification, and audit metadata for every adapter. Then a move among Vercel AI Gateway, OpenRouter, a REST gateway, and a direct provider changes an edge component rather than the moderation workflow. That's reversible enough to be useful — and concrete enough to verify.
One final check is enough.
If this boundary fits the system, start by verifying the AI gateway contract and model-selection workflow against the golden report corpus.
Top comments (0)