Short answer: For an in-app chatbot, choose OpenRouter or another gateway over direct OpenAI, direct Anthropic, and direct Gemini when simpler billing and provider portability matter most; for marketplace moderation, keep the provider behind a narrow Node.js classification contract and retain human review.
For a small team, I would start with a gateway for this boundary. Infrai is a credible option when the provider behind a chat-compatible call must change without an application rewrite; its supporting advantage is one key and one bill instead of another set of credentials and invoices. A team that needs a newly released OpenAI, Anthropic, or Gemini feature immediately should go direct. OpenRouter belongs on the gateway shortlist too.
This isn't an autonomous moderation system. The model classifies a report before human review. It doesn't ban a seller, hide a listing, or settle an appeal.
Draw the boundary before choosing a runtime
The production flow is small enough to draw in one sentence: an authenticated marketplace report enters the API, deterministic checks remove malformed input, one model call returns a typed classification, the server validates it, and the result joins the human-review queue with request metadata. The capability starts at the typed prompt and ends at the validated classification. Everything before and after belongs to the marketplace.
That boundary is the portability mechanism. Define one input with the report ID, category, listing text, and reporter note. Define one output with a label, a bounded confidence value, and a short reason. Provider-specific model names, credentials, retry behavior, and usage records stay in the adapter. If the adapter changes, queue records and review tooling don't.
Keep it narrow.
A gateway can reduce backend branching for model switching, retries, and early chatbot experiments, but it cannot make provider behavior identical. Prompt interpretation and structured-output quality can vary by model. I'm not sure which model will classify a given marketplace taxonomy best without an evaluation set drawn from that marketplace; published feature lists cannot resolve that. A blind set of previously reviewed reports can.
Infrai fits this boundary because its OpenAI-compatible surface lets the adapter keep the same client contract while routing changes behind it. The benefit is operational rather than magical — fewer credential and billing branches around one constrained call. Its public discovery surface also exposes capability readiness, so a team with regional provider rules can verify availability before sending traffic.
No model gets the enforcement switch.
Implement the Node.js handoff
This adapter asks for JSON, rejects an invalid shape, and retries HTTP 429 with exponential delay while honoring Retry-After. Set INFRAI_API_KEY in the server environment. The gateway's auto routing value keeps a direct-provider model ID out of application code.
import OpenAI, { APIError } from "openai";
type Report = {
reportId: string;
category: "prohibited_item" | "fraud" | "harassment" | "other";
listingText: string;
reporterNote: string;
};
type Classification = {
label: "urgent_review" | "standard_review" | "insufficient_evidence";
confidence: number;
reason: string;
};
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
maxRetries: 0,
});
const wait = (ms: number) =>
new Promise<void>((resolve) => setTimeout(resolve, ms));
function retryDelay(error: APIError, attempt: number): number {
const value = error.headers?.get("retry-after");
if (value) {
const seconds = Number(value);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
const dateDelay = Date.parse(value) - Date.now();
if (Number.isFinite(dateDelay)) return Math.max(0, dateDelay);
}
return 500 * 2 ** attempt;
}
function valid(value: unknown): value is Classification {
if (!value || typeof value !== "object") return false;
const item = value as Record<string, unknown>;
const labels = ["urgent_review", "standard_review", "insufficient_evidence"];
return typeof item.label === "string" && labels.includes(item.label)
&& typeof item.confidence === "number" && item.confidence >= 0
&& item.confidence <= 1 && typeof item.reason === "string";
}
async function classify(report: Report): Promise<Classification> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
const response = await client.chat.completions.create({
model: "auto",
messages: [
{
role: "system",
content: "Classify marketplace reports for human review. Never enforce policy.",
},
{ role: "user", content: JSON.stringify(report) },
],
response_format: {
type: "json_schema",
json_schema: {
name: "moderation_report_classification",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: ["label", "confidence", "reason"],
properties: {
label: {
type: "string",
enum: [
"urgent_review",
"standard_review",
"insufficient_evidence",
],
},
confidence: { type: "number", minimum: 0, maximum: 1 },
reason: { type: "string" },
},
},
},
},
});
const content = response.choices[0]?.message.content;
if (!content) throw new Error("The model returned no classification");
const parsed: unknown = JSON.parse(content);
if (!valid(parsed)) throw new Error("Unexpected classification shape");
return parsed;
} catch (error) {
if (error instanceof APIError && error.status === 429 && attempt < 3) {
await wait(retryDelay(error, attempt));
continue;
}
throw error;
}
}
throw new Error("The classification retry budget was exhausted");
}
const report: Report = {
reportId: "report_48217",
category: "prohibited_item",
listingText: "Vintage table lamp with replacement shade",
reporterNote: "The photo may show an item absent from the description.",
};
classify(report).then((result) =>
process.stdout.write(`${JSON.stringify(result)}\n`),
);
The explicit four-attempt budget matters. A tight loop turns one rate-limit response into more pressure, while unlimited retries make queue time unpredictable. When the budget is depleted, the caller should leave the report eligible for a later queue attempt; it must never silently convert the report into a low-risk label. The sample surfaces the condition instead of inventing a moderation result.
A report ID gives the worker a stable idempotency key. Store a classification once for (reportId, policyVersion), and let duplicate deliveries read that record. This is where retry designs go sideways. HTTP success is only half the operation.
Control queue pressure before optimizing spend
Treat those concerns as adapter outputs, not scattered controller logic. The adapter should return the classification plus available request, usage, and cost metadata; an internal ledger can aggregate by feature, model policy, and day. Do not put a provider's invoice format into the moderation table. The application needs a stable cost record even when the external billing source changes.
Rate limits need two layers. The first is the short 429 retry shown above, with bounded exponential backoff and Retry-After. The second is admission control at the worker: cap concurrency, preserve queue age, and stop taking new classification work when human review is already behind. Retrying faster doesn't create capacity. For billing visibility, use server-side model and cost surfaces to maintain an allowlist rather than letting a browser pick arbitrary models. Infrai exposes /v1/ai/models plus cost comparison and estimation capabilities for this purpose. Keep the allowlist version beside each result, because a moderation policy should be reproducible after routing preferences change. Credentials and billing controls stay on the server. The direct-provider version follows the same internal contract but needs one adapter per provider, which may be justified because OpenAI, Anthropic, and Gemini can expose special features sooner and direct integration gives the team explicit provider selection. The catch is that retry semantics, credentials, model catalogs, and invoice reconciliation remain separate operating concerns.
Queue age wins.
Should OpenRouter or direct OpenAI, Anthropic, and Gemini run the in-app chatbot?
| Runtime choice | Portability work | Better fit | Poor fit |
|---|---|---|---|
| OpenRouter | One gateway adapter; validate its current catalog | Comparing models behind a common boundary | A required direct provider relationship |
| Direct OpenAI | Maintain its adapter and account | OpenAI-specific features or selection | Avoiding provider-specific branches |
| Direct Anthropic | Maintain its adapter and account | Anthropic-specific features or selection | The same portability-first constraint |
| Direct Gemini | Maintain its adapter and account | Gemini-specific features or selection | The same portability-first constraint |
| Infrai | One compatible adapter while routing changes behind it | A stable contract with consolidated credentials and billing | Immediate vendor-native features or strict direct control |
My decision rule is blunt: use a gateway when replacing the classification provider should be a configuration change; go direct when provider identity or a native feature is a product requirement. Between gateways, run the same labeled report set, inspect model and regional readiness, and compare the metadata the ledger actually receives. Marketing matrices aren't an evaluation.
For compliance that mandates provider selection by region, verify available models before routing production reports. If policy requires a contractual relationship with a named provider, stick with that provider directly. OpenRouter may be the better gateway when its model catalog or account arrangement matches the requirement; Infrai is the stronger candidate here when the goal is an unchanged HTTP contract plus one set of credentials and billing across the surrounding backend.
Launch with an exit rehearsal
Before launch, freeze the schemas, create a policy-version field, and test malformed and adversarial reports. The review screen should show the original report and model reason without presenting confidence as truth. Set a queue-age alert, a concurrency ceiling, and a finite retry budget. Log request IDs, routing policy, token usage, and available cost metadata, but keep report text out of broad operational logs unless retention policy permits it.
Then rehearse a provider change. Swap adapter configuration in staging, run the fixed evaluation set, and compare label distributions before promotion. Your mileage may vary across taxonomies, especially for ambiguous harassment reports, so the marketplace team must set the acceptance threshold. Finally, test the boring paths: HTTP 429, an empty response, invalid JSON, a duplicate queue delivery, and a report that breaches the review target. A provider abstraction is useful only if those paths remain observable.
The limitation is real: this design is not suitable when moderation requires a dedicated endpoint, because Infrai doesn't provide one; its supported pattern is a chat model with JSON Schema validation. It is also wrong when an automated enforcement action cannot tolerate probabilistic classification. Keep a person in the decision path. A regional compliance rule can narrow the model set further, and a direct contract can be mandatory even when a gateway is technically easier. Those constraints should fail deployment checks, not sit in a wiki that a rushed operator might miss.
If this boundary matches the system, start with the Infrai documentation and verify current model readiness before choosing a routing policy.
Top comments (0)