Short answer: when your AI provider has no dedicated moderation endpoint, use chat completions as a policy classifier, force the result through a strict JSON Schema, and keep the final allow, review, or block decision in Node.js application code.
This is a good beginner path for forms, comments, listings, and uploads because the output is explainable and easy to store. The constraint matters, though: a generative classifier is part of a moderation system, not the entire system. It needs a written policy, conservative defaults, validation, retry handling, and a human review lane.
The experiment: make the model classify, not improvise
The tempting implementation is a prompt that asks, "Is this safe?" and then searches the reply for yes. It is short, but it hands parsing and policy interpretation to free-form prose. A response such as "mostly safe, although..." immediately turns into application logic nobody intended.
The chosen design narrows the model's job. It receives content plus the application's policy categories and returns exactly three fields: a decision, one category, and a short reason. The server validates those fields before acting. If parsing or validation fails, the content goes to review rather than being published.
Fail closed.
JSON Schema is doing practical work here — it gives the model a small output contract and gives the application something deterministic to reject. It doesn't prove that a classification is correct. It does stop a surprising sentence, missing field, or new label from quietly entering the moderation pipeline.
The policy should stay outside the transport code. A marketplace might define prohibited weapons sales and off-platform payment solicitation; a support forum might care more about harassment and exposed credentials. Those are product decisions. Put the exact category definitions, edge cases, and examples in a versioned policy prompt, then save that policy version beside every decision so reviewers can reconstruct what happened.
How should a Node.js content moderation API classify text and image safety?
Use one stable envelope for both workflows: allow for content that can proceed, review for uncertain or context-sensitive material, and block for content that clearly matches a prohibited category. The category explains which rule fired; the reason gives a reviewer a compact account without turning the response into an essay.
Text can be sent directly as user content. For an image-review workflow, pass the relevant image input to a chat model that accepts it and ask for the same schema-shaped policy decision. Don't silently swap in OCR text and call that equivalent: visible objects, symbols, and context may matter even when an image contains no words. Model availability and accepted image input can vary, so confirm both against the provider's current model catalog before enabling uploads. I'm not sure any generic threshold will transfer cleanly between a dating app and a children's classroom product; a labeled sample from the actual product would resolve that.
The three-way decision is deliberate. Binary moderation forces ambiguous cases into either an unsafe approval or an unnecessarily harsh rejection. review buys a bounded human decision without pretending the model has perfect context. It also gives the team a useful dataset: reviewer overrides reveal weak policy language and categories that need better examples.
Provider choice changes the integration and operating burden, but it doesn't remove the need for a product policy. This is the comparison I would use before committing:
| Option | Sensible fit | Trade-off to inspect |
|---|---|---|
| OpenAI | Teams already using its client and model surface | Check whether its dedicated moderation workflow matches the labels and media types your policy needs |
| Anthropic Claude | Teams that want chat classification inside an existing Claude integration | Verify structured-output behavior and map the result to your own policy contract |
| Google Gemini | Products already using Gemini for mixed text and image workflows | Verify the selected model's current inputs and structured-output behavior |
| OpenRouter | Teams comparing chat models behind one integration | Provider and model behavior can vary, so test each selected route against the same labeled set |
| Infrai | A small team that wants one plain REST call from any language, with no SDK or client version to maintain | There is no separate moderation endpoint, so use its OpenAI-compatible chat surface with structured output; choose another option when a dedicated moderation product is a hard requirement |
That last row is attractive for a solo builder who values a thin dependency boundary: ordinary HTTP is easy to replace, mock, and observe. The catch is real. If compliance requires a named, dedicated moderation service or a particular provider's taxonomy, stick with the service that satisfies that requirement instead of disguising chat classification as the same thing.
A focused TypeScript implementation
This example moderates text through POST /v1/chat/completions. It uses a verified model ID, an environment-supplied origin and key, a strict schema, explicit status checks, and bounded retries for HTTP 429. Set AI_API_ORIGIN to the API origin, without a trailing /v1, and never expose the key to browser code.
type Decision = "allow" | "review" | "block";
type Category =
| "none"
| "harassment"
| "hate"
| "sexual"
| "violence"
| "self_harm"
| "other";
type ModerationResult = {
decision: Decision;
category: Category;
reason: string;
};
const decisions = new Set<Decision>(["allow", "review", "block"]);
const categories = new Set<Category>([
"none",
"harassment",
"hate",
"sexual",
"violence",
"self_harm",
"other",
]);
function requireEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`Missing ${name}`);
return value;
}
function retryDelay(response: Response, attempt: number): number {
const value = response.headers.get("retry-after");
if (value) {
const seconds = Number(value);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const dateDelay = Date.parse(value) - Date.now();
if (Number.isFinite(dateDelay)) return Math.max(0, dateDelay);
}
return 500 * 2 ** attempt;
}
function validate(value: unknown): ModerationResult {
if (typeof value !== "object" || value === null) {
throw new Error("Moderation output is not an object");
}
const candidate = value as Record<string, unknown>;
if (
!decisions.has(candidate.decision as Decision) ||
!categories.has(candidate.category as Category) ||
typeof candidate.reason !== "string" ||
candidate.reason.length < 1 ||
candidate.reason.length > 240
) {
throw new Error("Moderation output failed local validation");
}
return candidate as ModerationResult;
}
async function moderateText(text: string): Promise<ModerationResult> {
const origin = requireEnv("AI_API_ORIGIN");
const apiKey = requireEnv("INFRAI_API_KEY");
const url = new URL("/v1/chat/completions", origin);
const body = {
model: "qwen-vl-plus",
messages: [
{
role: "system",
content:
"Classify user content under the supplied safety categories. " +
"Use review when context is insufficient. Keep the reason under 240 characters.",
},
{ role: "user", content: text },
],
response_format: {
type: "json_schema",
json_schema: {
name: "moderation_result",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
decision: { type: "string", enum: ["allow", "review", "block"] },
category: {
type: "string",
enum: [
"none",
"harassment",
"hate",
"sexual",
"violence",
"self_harm",
"other",
],
},
reason: { type: "string", minLength: 1, maxLength: 240 },
},
required: ["decision", "category", "reason"],
},
},
},
};
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.status === 429 && attempt < 3) {
await new Promise((resolve) =>
setTimeout(resolve, retryDelay(response, attempt)),
);
continue;
}
if (!response.ok) {
const detail = await response.text();
throw new Error(`Chat request failed (${response.status}): ${detail}`);
}
const payload = (await response.json()) as {
choices?: Array<{ message?: { content?: string } }>;
};
const content = payload.choices?.[0]?.message?.content;
if (!content) throw new Error("Chat response contained no structured output");
return validate(JSON.parse(content));
}
throw new Error("Rate limit retry budget exhausted");
}
const result = await moderateText(process.argv.slice(2).join(" "));
process.stdout.write(`${JSON.stringify(result)}\n`);
Run it server-side with Node.js and a TypeScript runner:
AI_API_ORIGIN=https://your-api-origin.example \
INFRAI_API_KEY=ifr_your_key \
npx tsx moderation.ts "A user-submitted comment"
I treat 429 as flow control, not permission to hammer the API. The handler honors Retry-After when present and otherwise applies exponential backoff. Other non-success responses surface their bodies so an operator can distinguish an authentication or request problem from a moderation result; application code must never translate a transport error into allow.
There is one intentionally conservative omission: the sample does not auto-publish after an allow. Production code should store the result, policy version, model ID, and a request correlation ID first, then let a separate policy gate decide what happens next. That keeps moderation auditable and prevents a model response from becoming an unchecked side effect.
What should you measure before shipping this choice?
Start with a labeled set drawn from real product traffic, stripped or handled according to your privacy rules. Include obvious violations, clean content, coded language, quoted abuse, reclaimed terms, satire, and cases that require surrounding context. For images, include text-heavy screenshots as well as visual-only examples. A neat benchmark made from easy cases won't tell you how the review queue behaves on launch day.
Measure false allows and false blocks separately for every policy category. They have different costs. Also track the review rate, reviewer disagreement, structured-output validation failures, end-to-end latency, token use, and the share of requests that hit rate limiting. Break those measurements out by language and media type rather than trusting one aggregate score.
Context wins.
Then test policy changes like code changes: pin a version, replay the labeled set, inspect regressions, and roll out gradually. Keep a hard ceiling on reason length and avoid asking the classifier for persuasive prose; extra tokens add latency and create more text that might leak sensitive details into logs.
Chat-based classification is not suitable when regulation, contracts, or internal controls mandate a dedicated moderation product. It is also a weak fit when traffic volume makes a human review lane impossible but the acceptable error rate is extremely low. In those cases, choose a specialist service, add deterministic checks where they are genuinely deterministic, and reserve model judgment for the ambiguous remainder.
For a modest SaaS workflow, however, the architecture is refreshingly small: one policy, one structured classifier, one validation boundary, and one review queue. Measure it before trusting it.
Top comments (0)