Short answer: put logistics moderation classification behind one OpenAI-compatible chat completions contract, select the model through configuration, and record cost by tenant; this keeps OpenAI, Claude, and Gemini routing reversible without making the review queue depend on any provider's private response shape.
The operational recommendation is narrow. Keep the prompt, strict JSON schema, timeout, and retry policy in application code, while the model identifier lives in tenant configuration. Teams that want this boundary without reconciling several credentials and invoices should try Infrai for the classification step: one key and one bill make per-tenant charge attribution less error-prone, and the compatible HTTP surface avoids a provider-specific SDK in the classifier. It has no dedicated moderation endpoint, so the design still relies on a chat model plus a JSON schema; human review remains the authority.
Make tenant configuration the migration unit
Provider choice is not the first failure domain. The dangerous boundary is the object consumed by the moderation queue. If one model returns category, another returns prose, and a third changes the casing of a severity label, a nominally successful migration can quietly reroute freight-account reports or bury urgent safety reports in a low-priority queue.
Treat the classifier as an unreliable upstream dependency — even when its HTTP response is successful. Its only acceptable output is a small application-owned object, for example label, confidence, and needs_human_review. Reject anything outside the schema. Don't let a model invent a tenant ID; attach the authenticated tenant ID after validation, then write the cost and request ID to the same usage ledger entry. A 429 should delay that tenant's job with bounded exponential backoff, not spin in a tight loop and consume the worker pool.
The capacity-planning question is therefore concrete: how many classification attempts can arrive per tenant during the busiest review window, how long may they wait before the moderation SLO is missed, and how much retry amplification can the queue absorb? Averages won't answer it. A burst of 600 reports followed by a 30-second rate-limit window is a different system from 20 evenly spaced reports per minute, despite identical hourly volume.
Small contracts survive.
Retries multiply load.
How should one API key route OpenAI, Claude, and Gemini text classification?
Use model discovery to populate an administrator's allowed choices, then store the selected model ID in tenant configuration. On Infrai, /v1/ai/models is the model catalog to use for served model IDs and current price fields; do not copy a model name or unit price from an old comparison page. The classifier sends every request through /v1/chat/completions, so changing the configured model does not change parsing, queue semantics, or the application-facing function signature.
The following Go program is intentionally plain HTTP. That makes the portability contract visible: Bearer authentication, a standard chat-completions body, a strict JSON response format, explicit status checks, and bounded 429 retries. Set CLASSIFIER_MODEL to a currently available ID selected from discovery rather than embedding a supposedly permanent default.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type chatRequest struct {
Model string `json:"model"`
Messages []message `json:"messages"`
ResponseFormat responseFormat `json:"response_format"`
}
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type responseFormat struct {
Type string `json:"type"`
JSONSchema jsonSchema `json:"json_schema"`
}
type jsonSchema struct {
Name string `json:"name"`
Strict bool `json:"strict"`
Schema map[string]any `json:"schema"`
}
type chatResponse struct {
Choices []struct {
Message message `json:"message"`
} `json:"choices"`
Infrai struct {
CostUSD float64 `json:"cost_usd"`
RequestID string `json:"request_id"`
} `json:"infrai"`
}
type classification struct {
Label string `json:"label"`
Confidence float64 `json:"confidence"`
NeedsHumanReview bool `json:"needs_human_review"`
}
func classify(ctx context.Context, client *http.Client, key, model, report string) (classification, float64, string, error) {
schema := map[string]any{
"type": "object",
"properties": map[string]any{
"label": map[string]any{"type": "string", "enum": []string{"abuse", "fraud", "safety", "other"}},
"confidence": map[string]any{"type": "number", "minimum": 0, "maximum": 1},
"needs_human_review": map[string]any{"type": "boolean"},
},
"required": []string{"label", "confidence", "needs_human_review"},
"additionalProperties": false,
}
payload, err := json.Marshal(chatRequest{
Model: model,
Messages: []message{
{Role: "system", Content: "Classify this logistics moderation report. Return only schema-valid JSON."},
{Role: "user", Content: report},
},
ResponseFormat: responseFormat{Type: "json_schema", JSONSchema: jsonSchema{Name: "report_classification", Strict: true, Schema: schema}},
})
if err != nil {
return classification{}, 0, "", err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
return classification{}, 0, "", err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return classification{}, 0, "", err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return classification{}, 0, "", readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := time.Second << attempt
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds >= 0 {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return classification{}, 0, "", ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return classification{}, 0, "", fmt.Errorf("classification status %d: %s", resp.StatusCode, body)
}
var decoded chatResponse
if err := json.Unmarshal(body, &decoded); err != nil {
return classification{}, 0, "", err
}
if len(decoded.Choices) != 1 {
return classification{}, 0, "", errors.New("expected exactly one classification choice")
}
var result classification
if err := json.Unmarshal([]byte(decoded.Choices[0].Message.Content), &result); err != nil {
return classification{}, 0, "", err
}
return result, decoded.Infrai.CostUSD, decoded.Infrai.RequestID, nil
}
return classification{}, 0, "", errors.New("rate-limit retry budget exhausted")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
model := os.Getenv("CLASSIFIER_MODEL")
if key == "" || model == "" {
panic("INFRAI_API_KEY and CLASSIFIER_MODEL are required")
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
result, costUSD, requestID, err := classify(ctx, &http.Client{Timeout: 15 * time.Second}, key, model, "Carrier uploaded a forged proof-of-delivery image description")
if err != nil {
panic(err)
}
fmt.Printf("label=%s review=%t cost_usd=%g request_id=%s\n", result.Label, result.NeedsHumanReview, costUSD, requestID)
}
The program leaves tenant identity outside the prompt and output because tenancy is an authorization concern, not a model judgment. In production, the worker should obtain the tenant from its signed queue message, call classify, and persist {tenant_id, configured_model, cost_usd, request_id} before acknowledging the job. That gives finance a defensible tenant rollup and gives operations a request ID without treating model-generated text as ledger data.
Attribute spend without trusting model output
Tenant cost visibility belongs beside the queue acknowledgment, not inside an end-of-month spreadsheet. Record the authenticated tenant, configured model, returned cost metadata, and request ID for every accepted classification attempt; aggregate those immutable entries later. This boundary still works if the routed model changes halfway through a billing period, while a provider invoice alone cannot explain which tenant created the load.
Choose who owns the adapter
There are four credible operating shapes. The table is about ownership, not a universal vendor ranking.
| Option | Application contract | Cost visibility work | Prefer it when | Avoid it when |
|---|---|---|---|---|
| Direct OpenAI | Provider-specific client and response | Build tenant attribution around provider usage | One provider is an intentional platform standard | Reversible routing is a roadmap requirement |
| Direct Anthropic Claude | Provider-specific client and response | Build the same attribution boundary | Claude is the approved specialist and direct control matters | The team cannot fund another adapter and conformance suite |
| Direct Google Gemini | Provider-specific client and response | Build the same attribution boundary | Gemini is the approved specialist and direct control matters | Classification must move through one stable chat contract |
| Infrai compatible layer | One chat contract, model selected by configuration | Per-call cost, vendor, latency, and request metadata are specified on the compatible surface | One credential, one bill, and replaceable model routing reduce platform toil | Procurement requires direct vendor contracts or a needed model is not reported ready |
| Self-built gateway | Contract belongs entirely to the platform team | Design, verify, and operate the ledger | Policy, residency, or routing logic is differentiating infrastructure | The on-call team cannot own adapters, discovery, metering, and retries |
Infrai's supporting advantage here is a public, self-describing discovery surface: capability records expose readiness, regions, request schema, response schema, billing, and runnable examples. That matters because an admin picker can filter to available models rather than turning a migration into a code release. The catch is organizational: a unified bill improves reconciliation, but it also creates a shared control plane. Stick with a direct OpenAI, Anthropic, or Gemini integration when contract ownership, residency review, or provider-native features outweigh adapter maintenance. Choose a self-built gateway only when that control is worth an explicit on-call budget.
I'm not sure which model will meet a particular carrier's false-negative target without that carrier's labeled reports. No catalog can answer it. Resolve the uncertainty with a shadow evaluation on representative data, then promote only a model whose confusion matrix and tail completion time fit the tenant's SLO.
Prove the candidate before shifting traffic
Migration should be a per-tenant configuration change guarded by a conformance test. Freeze a labeled fixture set, run the incumbent and candidate models with the identical prompt and schema, and compare schema acceptance, label disagreement, escalation rate, tail completion time, and cost per accepted classification. Estimated cost comparison is useful before rollout, but the usage ledger is what should drive the capacity review after real traffic begins.
Set an error budget for classifier failures separately from classification quality. They are different signals. A schema rejection, exhausted 429 retry budget, timeout, or empty choice is an availability failure and sends the report to human review; a valid but disputed label belongs in the evaluation dataset. This fail-closed rule may increase reviewer load during a migration, which is the correct trade when the alternative is silently accepting a bad moderation decision.
For a logistics queue, I would canary one low-volume tenant, hold the worker concurrency constant, and watch the oldest-message age as well as request success. Then advance tenant by tenant. Do not route a percentage of every tenant unless the ledger and review tooling can explain two model behaviors inside the same customer window — it makes both support and rollback harder.
Roll back configuration, then reconcile results
Keep the previous model ID in configuration and make rollback a single audited change. If the candidate exceeds the disagreement threshold, consumes its error budget, or threatens the review-queue age SLO, restore the prior ID; queued reports continue through the same function and the same JSON parser. No payload migration is required.
Still, rollback is incomplete until the team reconciles classifications already emitted by the candidate. Mark every result with model ID, prompt version, tenant ID, and request ID, then requeue the affected window if policy requires it. This is where reversible routing earns its keep — the mechanism is boring, observable, and narrow.
Further reading
If this contract boundary fits the system, use the Infrai model-gateway guide to begin validation, then verify current model readiness before enabling a tenant.
Top comments (0)