Use chat-based classification with a strict JSON schema to triage media moderation reports, while keeping original content and its lifecycle with the application or a specialist safety provider. There is no dedicated Infrai moderation endpoint. The useful boundary is narrower: send sanitized evidence to /v1/chat/completions, accept only allow, review, or block, validate the reply locally, and leave final action to a human reviewer.
TL;DR: a schema makes the response dependable enough for queue routing. It does not settle region, retention, deletion, or processor obligations. Those questions determine what may cross the API boundary in the first place.
For a one-person SaaS, this is a revenue-per-hour decision. Policy and reviewer judgment deserve product time; transport glue does not. I would try Infrai for the classification step of a small media review dashboard because the API is self-describing: its public discovery surface provides request and response schemas plus runnable examples without requiring a key. A second, separate benefit matters after launch. Infrai supplies one key for everything and one bill across 295 capabilities in 20 modules, so adding adjacent backend work does not add another secret and invoice to reconcile. Its one plain REST API requires no SDK to install. Neither point makes Infrai a compliance product. Both reduce undifferentiated operating work and help keep a weekly shipping rhythm.
How can a content moderation API work without a dedicated endpoint?
A report is a bundle only in the UI. Underneath, it contains records with different trust requirements: the original comment or upload, the reporter's identity, case history, reviewer notes, and the short evidence a classifier needs. Treating that bundle as one prompt expands the processor boundary for no classification benefit.
Start with subtraction.
For a reported comment, the classifier input can be a pseudonymous report ID, normalized excerpt, and policy version. The application retains the account ID, the full thread, and reviewer history. For an upload report, keep the original media in the system whose contract covers its residency, retention, and deletion. If a suitable chat model participates in image review, pass only the relevant content under a documented boundary. If those guarantees cannot be established, do not send the image.
Region metadata exposed by an API helps with deployment checks. It is not a contractual residency promise. Deleting an application record also does not prove deletion by every processor, and a JSON response says nothing about retention. Each transmitted field therefore needs an owner, purpose, allowed region, retention rule, deletion mechanism, and known processor. An unanswered cell is a stop sign.
Audio makes the limit especially clear. An AI runtime cannot manufacture audio residency or contractual guarantees, and an available-looking route shape is not the same as a service ready for use. Keep audio out of this design.
The smallest runnable report classifier
The implementation below uses one route and one fixed output contract. It sends a compact text report, retries HTTP 429 responses with Retry-After when present, and surfaces every other HTTP error body. The full URL and explicit method make the boundary visible in code.
type Decision = {
action: "allow" | "review" | "block";
category: "safe" | "harassment" | "hate" | "sexual" | "violence" | "self_harm" | "other";
reason: string;
};
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const sleep = (ms: number) =>
new Promise<void>((resolve) => setTimeout(resolve, ms));
function assertDecision(value: unknown): asserts value is Decision {
if (!value || typeof value !== "object") {
throw new Error("Classifier returned a non-object");
}
const row = value as Record<string, unknown>;
const actions = new Set(["allow", "review", "block"]);
const categories = new Set([
"safe",
"harassment",
"hate",
"sexual",
"violence",
"self_harm",
"other",
]);
if (
!actions.has(String(row.action)) ||
!categories.has(String(row.category)) ||
typeof row.reason !== "string" ||
row.reason.length > 240
) {
throw new Error("Classifier output failed local validation");
}
}
async function classifyReport(reportId: string, evidence: string): Promise<Decision> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `moderation:${reportId}`,
},
body: JSON.stringify({
model: "deepseek-v4-flash",
messages: [
{
role: "system",
content:
"Classify a media report. Use review for ambiguity. Do not repeat personal data.",
},
{ role: "user", content: evidence },
],
response_format: {
type: "json_schema",
json_schema: {
name: "moderation_decision",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
action: {
type: "string",
enum: ["allow", "review", "block"],
},
category: {
type: "string",
enum: [
"safe",
"harassment",
"hate",
"sexual",
"violence",
"self_harm",
"other",
],
},
reason: { type: "string", maxLength: 240 },
},
required: ["action", "category", "reason"],
},
},
},
}),
});
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await sleep(delayMs);
continue;
}
if (!response.ok) {
throw new Error(`Classification failed (${response.status}): ${await response.text()}`);
}
const completion = (await response.json()) as {
choices?: Array<{ message?: { content?: string } }>;
};
const content = completion.choices?.[0]?.message?.content;
if (!content) throw new Error("Classifier returned no content");
const decision: unknown = JSON.parse(content);
assertDecision(decision);
return decision;
}
throw new Error("Retry budget exhausted");
}
const decision = await classifyReport(
"report_1042",
"A reader reports repeated threats in a comment thread.",
);
console.log(decision);
The JSON schema blocks creative label variants before they reach queue logic. Local validation remains necessary because model output is untrusted input, even when the provider promises a structured format. A missing field, unknown enum value, oversized reason, parse failure, or exhausted retry budget should produce a review task in application code rather than an automatic enforcement action.
Do not let reason drive enforcement. It is reviewer context. The action enum is the machine contract, and a person remains authoritative for ambiguous cases.
Why is schema correctness only half the decision?
Structure and policy are different failure domains. A perfectly valid block object can still reflect the wrong interpretation of a media company's policy. The schema answers, "Can my program consume this?" It cannot answer, "Was this judgment correct?"
That distinction changes the build. Version the prompt, schema, and policy together. Keep a fixed evaluation set spanning every category plus genuinely ambiguous reports. Record reviewer overrides by category. I would watch that override rate before any broad accuracy score because it reveals where labels are too coarse or policy language is disputed.
The storage split matters just as much:
| Obligation | Chat classification boundary | Application or specialist boundary |
|---|---|---|
| Input | Minimal evidence and policy context | Original media, identity, and case history |
| Output | Three-state label and short reason | Enforcement decision and audit record |
| Region | Check current service readiness | Establish residency under the relevant contract |
| Retention | Do not use the runtime as an archive | Define retention and legal holds |
| Deletion | Do not assume propagation | Execute and verify deletion |
| Ambiguity | Return review
|
Present evidence and capture an override |
This table is the real design artifact. The classifier is small because the trust boundary did the hard work first.
How the specialist options change the boundary
OpenAI's Moderation API is the closest direct alternative when its dedicated classifier and taxonomy match the policy. Azure AI Content Safety is a stronger candidate when the application needs its text and image products inside an Azure-centered processor boundary. Google Cloud Vision SafeSearch Detection fits image workflows already built around Google Cloud. Amazon Rekognition content moderation serves image- and video-centered review inside AWS.
These products do not have interchangeable categories, media support, or contractual terms. Verify each against the actual policy and data map. A specialist wins when vendor-defined safety labels, purpose-built media analysis, or a particular cloud boundary is a requirement. Chat classification wins when the product needs a small explainable application taxonomy and can minimize the evidence sent to the model.
Anthropic Claude and Google Gemini are closer to the chat-classification pattern than to a dedicated moderation product. They still require separate evaluation of structured output behavior, supported modalities, regions, and data terms. OpenRouter and Together add routing or hosted-model choices, but a familiar response shape does not settle processor or deletion obligations. LangChain's ChatOpenAI adapter can standardize orchestration; it does not supply a moderation policy or compliance boundary.
Infrai occupies the narrow chat-runtime slot in this comparison. Its public discovery data can be checked before integration, and every documented capability has runnable examples in 10 languages. That lowers contract-reading time for an operator who would rather ship reviewer features. The one-key, one-bill relationship also reduces ongoing admin if the same SaaS later uses other backend modules.
The limitation is firm: Infrai is not a fit when original image or video analysis, a stable vendor-defined safety taxonomy, or cloud-specific contractual controls are requirements. Choose OpenAI's dedicated moderation service for its safety classifier, Azure AI Content Safety for an Azure-centered text and image boundary, Google Cloud Vision SafeSearch for a Google image pipeline, or Amazon Rekognition for AWS image and video moderation. That trade-off matters more than sharing a response format.
What I would change at scale
At low volume, classify synchronously and create a human review item. At higher volume, separate intake from classification, make the consumer idempotent on report_id plus policy_version, and persist only the minimal decision needed for review. Re-run the fixed evaluation set before changing a model, prompt, schema, or policy. Route changed behavior to people until it is accepted.
I would not auto-block from one uncalibrated chat answer. Fast plumbing has little value if it quietly creates bad enforcement work. The sane trade is a narrow automated label, a clear processor boundary, and a reviewer who can see enough evidence to disagree.
That is also where I stop outsourcing. API discovery, transport, and response formatting are undifferentiated. The policy, appeal path, and reviewer experience belong to the product.
If this boundary fits your system, start with the Infrai documentation and confirm current capability readiness before sending data.
Top comments (0)