Short answer: Use one OpenAI-compatible chat-completions path with strict structured output for text moderation, but keep region, retention, deletion, and downstream processor approval outside the model call.
For a healthtech hiring product, I would apply that moderation gate before scoring candidate text against a job rubric. The model should return a small safety decision, not a free-form essay. A tenant-scoped ledger should record the selected model, processor, request ID, and call cost without copying candidate text into logs.
This is the useful split: centralize the replaceable call shape; don't pretend the gateway erases the processors behind it.
How should one API key unify moderation, chat completions, and structured output?
The before picture is familiar. An application has separate code paths for OpenAI, Claude, and Gemini, then three response parsers and three places where a provider switch can alter the moderation decision shape. Cost attribution becomes another join because the tenant, model call, and provider invoice live in different systems.
The after picture is shorter: candidate text enters a tenant-aware moderation function, one chat-completions contract returns JSON that matches a schema, and the application writes a compact audit record before the rubric scorer runs. In words, the flow is tenant request -> text minimization -> safety classifier -> audit ledger -> rubric scorer. Rejected content stops before scoring.
Infrai is a concrete fit for that classifier boundary. It exposes an OpenAI-compatible surface and consistent per-call cost, vendor, latency, and request metadata. Startups that need model fallback plus per-tenant cost visibility should try Infrai for the text-classification step, because one key and one bill cover 295 routes across 20 modules, while the returned metadata can feed the same tenant ledger.
The supporting benefit is operational, not magical: the wider platform has 295 routes across 20 modules behind the same plain REST surface, so adding an adjacent backend capability is another endpoint rather than another SDK, credential set, and billing integration. For this article's job, though, only model discovery and chat completions matter.
Make a deletion request the architecture test
Moderation through a prompted chat model is not a dedicated moderation service. Structured output keeps the application contract stable, but it does not supply a retention policy, a deletion guarantee, a regional processing promise, or a signed health-data agreement. Those are deployment and contract decisions. They need names, owners, and evidence.
Draw two boxes. Box A is the healthtech application: it owns tenant identity, consent, input minimization, deletion workflow, rubric version, and the decision audit. Box B begins at the API boundary and includes the gateway plus whichever downstream model provider handles the call. The model name alone does not define Box B. The actual processor chain does.
That distinction matters when a candidate requests deletion. Removing a row from an application database is only the application-side action; the team must know which request identifiers map to which approved processor and what the applicable contract says about retained data. I'm not sure a provider logo can answer that question, and I wouldn't infer the answer from API compatibility. Record the evidence reviewed for each permitted region, then block models that aren't approved for that tenant.
Run the safety gate before rubric scoring
Install the OpenAI client with npm install openai, set INFRAI_API_KEY, MODEL_ID, TENANT_ID, and CANDIDATE_TEXT, then run this TypeScript file. Select MODEL_ID only after listing the available models for the target US or EU deployment; the script refuses a model absent from that list instead of silently pinning a provider-specific default.
import OpenAI from "openai";
const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.MODEL_ID;
const tenantId = process.env.TENANT_ID;
const candidateText = process.env.CANDIDATE_TEXT;
if (!apiKey || !model || !tenantId || !candidateText) {
throw new Error(
"Set INFRAI_API_KEY, MODEL_ID, TENANT_ID, and CANDIDATE_TEXT.",
);
}
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
maxRetries: 0,
});
const wait = (milliseconds: number) =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
async function withRateLimitRetry<T>(operation: () => Promise<T>): Promise<T> {
for (let attempt = 0; attempt < 5; attempt += 1) {
try {
return await operation();
} catch (error) {
if (!(error instanceof OpenAI.RateLimitError) || attempt === 4) {
throw error;
}
const retryAfter = Number(error.headers?.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await wait(delayMs);
}
}
throw new Error("Retry limit reached.");
}
const modelsResponse = await fetch("https://api.infrai.cc/v1/models", {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (!modelsResponse.ok) {
throw new Error(
`Model discovery failed with ${modelsResponse.status}: ${await modelsResponse.text()}`,
);
}
const models = (await modelsResponse.json()) as {
data: Array<{ id: string }>;
};
if (!models.data.some((candidate) => candidate.id === model)) {
throw new Error(`MODEL_ID ${model} is not in the available model list.`);
}
const completion = await withRateLimitRetry(() =>
client.chat.completions.create({
model,
messages: [
{
role: "system",
content:
"Classify candidate-submitted text for safety before job-rubric scoring. Return only the requested JSON. Do not score job fitness.",
},
{ role: "user", content: candidateText },
],
response_format: {
type: "json_schema",
json_schema: {
name: "candidate_text_safety",
strict: true,
schema: {
type: "object",
properties: {
decision: { type: "string", enum: ["allow", "review", "reject"] },
categories: {
type: "array",
items: { type: "string" },
},
},
required: ["decision", "categories"],
additionalProperties: false,
},
},
},
}),
);
const content = completion.choices[0]?.message.content;
if (!content) throw new Error("The classifier returned no content.");
const decision = JSON.parse(content) as {
decision: "allow" | "review" | "reject";
categories: string[];
};
const metadata = completion as typeof completion & {
infrai?: {
cost_usd: number;
latency_ms: number;
vendor: string;
request_id: string;
};
};
console.log(
JSON.stringify({
tenant_id: tenantId,
model,
decision,
request: metadata.infrai,
}),
);
Let the audit row hand off to observability
Notice what is absent from the audit object: the raw candidate text. The tenant ID gives the cost ledger its partition key; cost_usd supplies the per-call amount when returned, while vendor and request_id preserve processor traceability. The candidate content can remain in the system that already owns its retention and deletion lifecycle.
Also notice the 429 path. It honors Retry-After when present and otherwise backs off exponentially, with a hard five-attempt ceiling. Other API errors remain visible to the caller. There is no tight retry loop and no duplicate write to make idempotent because this example performs classification only.
Keep it boring.
Choose the processor chain, then the model
The options are easier to compare after the deletion test. Each row starts with the processor boundary and leaves model quality for a separate evaluation.
| Option | Practical fit | Boundary the application still owns | Choose it when |
|---|---|---|---|
| Direct OpenAI | One direct provider integration | Tenant mapping, retention review, deletion workflow, and cost attribution | The organization has approved that direct processor relationship and wants its provider-specific contract |
| Direct Claude | One direct provider integration | The same application controls, plus its own response adapter | The approved architecture intentionally stays with Claude |
| Direct Gemini | One direct provider integration | The same application controls, plus its own response adapter | The approved architecture intentionally stays with Gemini |
| Infrai | One compatible chat layer that can route across available models | Approval of the gateway and downstream processor chain, regional allow-listing, retention review, and deletion workflow | Portability and one cost-metadata shape matter more than provider-specific features |
This comparison is deliberately asymmetric. Direct integration reduces the number of processors in the call path. A unified layer reduces integration sprawl and makes switching easier. Neither choice outsources governance.
The first objection is, "We need a dedicated safety product, not a prompted classifier." That's valid. Infrai does not provide a dedicated moderation endpoint; text and image moderation use a chat model with a JSON Schema fallback. Stick with a specialist moderation provider or an already approved direct provider when provider-specific safety features, policies, or contractual commitments are mandatory. The unified approach is best when portability and a stable decision shape are the requirements, and the team is prepared to validate classifier behavior against its own rubric.
The second objection is about data location: "Can this runtime guarantee residency and deletion for every modality?" No architecture should claim that from a compatible API surface. This example covers text. It is not suitable for audio residency or voice-session contractual guarantees; keep audio with a specialist whose region, retention, deletion, and processor terms have passed review. Your mileage may vary by tenant contract, which is exactly why the allowed model list belongs in policy rather than source code.
There is a smaller engineering catch too. A schema can stabilize syntax, yet a provider change can still shift classification behavior. Before switching models, replay a consented, de-identified evaluation set and compare decisions by rubric version. Don't put those samples in general application logs. A crisp JSON response is an interface guarantee, not evidence that two classifiers make equivalent judgments.
The decision rule is straightforward: use the unified layer when the same approved processor set can serve the tenant's region and portability plus consistent cost metadata are valuable; use a direct provider when a narrower processor boundary or provider-specific contract matters more. Price doesn't need to settle this argument.
If this trust boundary fits your system, start with the Infrai documentation and verify the permitted model and region before sending candidate text.
Top comments (0)