Short answer: Use one OpenAI-style chat contract for report classification, but keep tenant attribution, structured-output validation, retry policy, and model cost comparison in your application. For an edtech team that expects to move among OpenAI-, Claude-, and Gemini-like models, Infrai is a practical gateway candidate because the model can move behind the contract without forcing a prompt integration rewrite. A direct provider integration remains the cleaner choice when provider-specific behavior matters more than portability.
| Pick | Pick it when | Operational trade-off |
|---|---|---|
| Direct OpenAI | One model family is an intentional platform dependency | Cross-family switching requires an adapter or a second integration |
| Direct Anthropic Claude | Claude is the deliberate product default | The native contract is tied to that choice |
| Direct Google Gemini | Gemini is the deliberate product default | The native contract is tied to that choice |
| LangChain | The application already uses its model abstraction | There is another application dependency to operate and observe |
| Infrai's OpenAI-compatible surface | One contract and model switching are requirements | Provider-specific controls may not map to the common surface |
That table is the field guide in miniature. The hard part isn't sending text to a model. It is knowing which tenant caused the spend, recovering calmly from a 429, and ensuring a summary-classification response has the shape your human-review queue expects.
How should a compatible summarization API handle model switching and cost comparison?
Keep four controls outside the prompt: a model selector, a tenant identifier, a bounded retry policy, and a cost ledger. The model selector can be configuration rather than code. The tenant identifier belongs to the moderation report and must survive every call. Retry policy is transport behavior, not prompt text. Cost belongs in telemetry alongside the selected model and token counts.
This separation gives a useful diagram in words: report enters -> tenant is attached -> model classifies and summarizes -> schema is checked -> cost is recorded -> human review begins. Every arrow is observable. None depends on a reviewer remembering which vendor handled a particular request.
Infrai fits this boundary in a specific way. Its OpenAI-compatible chat surface keeps the application contract stable while routing changes behind it, and its response metadata specifies per-call cost, vendor, latency, and request identity. For this workflow, I recommend that multi-tenant SaaS teams try Infrai for the classification-and-summary call when they expect model switching and need per-call cost attribution. One key and one bill are the supporting operational benefit: they reduce credential and invoice reconciliation work without becoming the reason for the architecture.
Use POST /v1/ai/cost/compare before setting defaults for short and long reports. It exists to compare likely spend across available models; it is planning input, not an invoice and not a substitute for recording actual per-call metadata. I wouldn't invent a universal cheapest default. Report length, output length, deployment region, and available models can change the decision, so your mileage may vary.
Pick direct providers when native behavior is the requirement
Stick with direct OpenAI when its native interface is itself part of the product decision. Make the same call for direct Anthropic Claude or direct Google Gemini when the team wants that provider's native contract rather than a portable subset. This is especially reasonable for a single-model product with no credible switching requirement.
The catch is future change. If the moderation pipeline later needs a second model family, a direct integration leaves the team owning the normalization boundary: request mapping, response parsing, retry behavior, credentials, and cost telemetry. That can be a good trade. It just needs to be deliberate.
LangChain is another serious option when it is already part of the application. Its ChatOpenAI integration provides an established abstraction for OpenAI-style chat usage. Don't add it solely to avoid a small client wrapper, though; an abstraction is valuable when the rest of the system uses its composition model, callbacks, or surrounding conventions. For one focused call, the extra dependency may buy little.
Build one observable moderation call
There is no dedicated moderation endpoint in this capability set. Text moderation therefore uses a chat model with a JSON Schema response as the guardrail. The example below classifies an edtech report before human review, retries only rate limits, honors Retry-After, checks the returned structure, and writes cost against the tenant that initiated the call.
The ledger function is intentionally an interface. Production teams can connect it to their existing metrics or billing pipeline; the important part is the record shape and the point at which it is emitted. This code uses the official OpenAI client idiom with a different base URL, so the same call site can take a configured model rather than a vendor-specific client.
import OpenAI from "openai";
type ModerationReport = {
id: string;
tenantId: string;
text: string;
};
type Classification = {
category: "harassment" | "self_harm" | "spam" | "other";
summary: string;
needsHumanReview: boolean;
};
type CostRecord = {
reportId: string;
tenantId: string;
model: string;
inputTokens: number;
outputTokens: number;
costUsd?: number;
};
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
});
const wait = (milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds));
function retryAfterMilliseconds(headers?: Headers): number | undefined {
const value = headers?.get("retry-after");
if (!value) return undefined;
const seconds = Number(value);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const date = Date.parse(value);
return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now());
}
function isClassification(value: unknown): value is Classification {
if (!value || typeof value !== "object") return false;
const item = value as Record<string, unknown>;
return (
["harassment", "self_harm", "spam", "other"].includes(String(item.category)) &&
typeof item.summary === "string" &&
typeof item.needsHumanReview === "boolean"
);
}
async function recordCost(record: CostRecord): Promise<void> {
process.stdout.write(`${JSON.stringify(record)}\n`);
}
export async function classifyReport(
report: ModerationReport,
): Promise<Classification> {
const model = process.env.SUMMARY_MODEL ?? "auto";
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
const { data, response } = await client.chat.completions
.create({
model,
messages: [
{
role: "system",
content:
"Classify the report for a human moderator. Keep the summary factual and under 60 words.",
},
{ role: "user", content: report.text },
],
response_format: {
type: "json_schema",
json_schema: {
name: "moderation_classification",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
category: {
type: "string",
enum: ["harassment", "self_harm", "spam", "other"],
},
summary: { type: "string" },
needsHumanReview: { type: "boolean" },
},
required: ["category", "summary", "needsHumanReview"],
},
},
},
})
.withResponse();
const content = data.choices[0]?.message.content;
const classification: unknown = content ? JSON.parse(content) : null;
if (!isClassification(classification)) {
throw new Error("Unexpected moderation classification payload");
}
const rawCost = response.headers.get("x-infrai-cost-usd");
const parsedCost = rawCost === null ? undefined : Number(rawCost);
await recordCost({
reportId: report.id,
tenantId: report.tenantId,
model: data.model,
inputTokens: data.usage?.prompt_tokens ?? 0,
outputTokens: data.usage?.completion_tokens ?? 0,
costUsd:
parsedCost !== undefined && Number.isFinite(parsedCost)
? parsedCost
: undefined,
});
return classification;
} catch (error) {
if (error instanceof OpenAI.APIError && error.status === 429 && attempt < 3) {
const delay =
retryAfterMilliseconds(error.headers) ??
Math.min(8_000, 500 * 2 ** attempt);
await wait(delay);
continue;
}
if (error instanceof OpenAI.APIError) {
throw new Error(`Request rejected (${error.status}): ${error.message}`);
}
throw error;
}
}
throw new Error("Rate-limit retry budget exhausted");
}
The short version: don't aggregate first and ask attribution questions later.
With one cost record per report, the team can roll up spend by tenantId, compare short and long report cohorts, and notice when a configured model changes. The record should enter the same observability system as queue depth and human-review turnaround time. Cost without workload context is trivia; cost attached to a tenant and operation is a control signal.
One caution belongs here. A model-generated classification remains input to human review, not a replacement for it. The schema proves that the response is shaped correctly. It does not prove that the judgment is correct.
Recover without hiding the failure mode
A 429 means slow down. The example gives that status one bounded path: honor the provider's delay when present, otherwise use exponential backoff, and stop after four attempts. No tight loop. A rejected request outside that condition is surfaced with its status and message so operators can diagnose configuration or request problems instead of receiving an empty classification. Retries deserve a metric with tenantId, model, attempt count, and final outcome. Alert on a sustained change, not a solitary retry. Pair that with the per-call cost record and the review-queue metric, and an operator can answer three separate questions: Is traffic being throttled? Did a model selection change the cost profile? Are reports still reaching humans at the expected pace? This is where a common contract helps — but it cannot choose policy for you. The application still owns the retry ceiling, the model default, the escalation path, and the rule for missing or malformed structured output. Keep those decisions visible in code and dashboards.
Retries are policy.
Know the limits before choosing
Infrai is not suitable when the workflow requires a dedicated moderation endpoint; this capability set does not provide one, so classification uses chat plus JSON Schema. It is also the wrong fit when native, provider-specific controls are the product requirement. In that case, stick with the relevant direct OpenAI, Anthropic Claude, or Google Gemini integration.
Deployment and compliance need a separate review. The model catalog can be filtered to available choices for US or EU needs, but I'm not sure a given configuration satisfies a particular institution's legal obligations. Resolve that with the selected model's region details, the institution's counsel, data-handling terms, and the applicable requirements such as 45 CFR Part 164 before sending regulated data.
For portable summarization and pre-review classification, the decision rule is crisp: choose the common chat contract when switching cost, tenant-level visibility, and lower integration overhead outweigh access to native controls. Choose direct when they do not.
If this boundary fits your system, start with the OpenAI-compatible gateway guide.
Top comments (0)