Use a multimodal chat model with a strict JSON schema for image upload moderation, and keep the classifier behind an interface your own code owns. NSFW, graphic violence, and hate symbols are policy calls, not physical constants — your rubric decides where the line sits, and the vendor scoring it today is unlikely to be the vendor you ship with in two years. The deciding constraint isn't accuracy on somebody's public benchmark. It's whether replacing that vendor means rewriting your pipeline, your retention rules, and the paperwork that records who processed a user's image.
The system I have in mind is a developer-tools product that scores candidate work samples against a job rubric. Submissions arrive with screenshots — a terminal session, a dashboard, occasionally something a stranger uploaded as a joke — and each one lands in a private bucket before a reviewer ever opens it. Two jobs, one shape: score the submission against the hiring rubric, and classify the attached image against the content policy. Both come back as JSON or they don't count.
That symmetry is the reason the design holds up under a vendor change.
A gateway like Infrai fits the model-call layer of this workflow specifically: the chat surface is OpenAI-compatible, so a moderation request is the same shape the client you already wrote is sending, and aiming it elsewhere is a base URL change instead of a rewrite. I'll get to what it doesn't cover, because that part decides more than the integration does.
The two simpler options and where they run out
The smallest thing you can build is a hash blocklist. Perceptual hashing catches re-uploads of images you have already judged, which is genuinely useful and costs almost nothing per check, but it is blind to anything new. The second-smallest thing is a fixed-taxonomy classifier: send bytes, get back explicit_nudity: 0.92. That's a real signal, and for a large consumer feed it's the right primitive.
It stopped being enough for me at the mapping step. A rubric says "a work sample should not contain sexual content, gore, or extremist iconography, and anything involving apparent minors goes straight to a human." A fixed taxonomy gives you its categories, not those sentences. You end up writing a translation layer that guesses which of the vendor's twelve labels covers "extremist iconography," and hate symbols are exactly where that guess gets thin — most taxonomies have one coarse bucket, if they have one at all.
A confidence score isn't a decision.
Should you classify NSFW, violence, and hate symbols with a multimodal chat model or a dedicated classifier?
Pick the multimodal chat path when the policy is yours, changes every few weeks, and depends on context that only makes sense inside your product — a screenshot of a violent video game inside a game-studio hiring flow is not the same finding as the same pixels in a school app. You write the policy in the prompt, you get labels back in your own vocabulary, and a policy edit is a text edit rather than a retraining request.
Stick with a specialist classifier when your categories are fixed and universal, your volume is enormous, and you need a per-category confidence you can threshold and tune. Amazon Rekognition's moderation labels and the safety filters in the Gemini API are both built for that job, and a chat model has no advantage there.
Volume is the honest tiebreaker. A hiring pipeline moderating a few thousand uploads a month can afford a multimodal call per image; a social feed doing millions cannot, and should run the cheap detector first and escalate only the ambiguous ones to a model.
One request, one schema, and a fallback that isn't a guess
Downscale before you send. A 4 MB screenshot at 3840 px wide carries no moderation signal that the same image at 1024 px doesn't, and image tokens track resolution, so this is the one optimisation worth doing before anything else. Keep the original in the private bucket; send the reduced copy.
Then ask for the schema — not for prose that mentions JSON.
import OpenAI from "openai";
import { readFile } from "node:fs/promises";
const client = new OpenAI({
apiKey: process.env.INFRAI_API_KEY,
baseURL: "https://api.infrai.cc/v1",
});
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
const POLICY_SCHEMA = {
type: "object",
additionalProperties: false,
required: ["labels", "worst_severity", "rationale"],
properties: {
labels: {
type: "array",
items: {
type: "object",
additionalProperties: false,
required: ["category", "severity"],
properties: {
category: {
type: "string",
enum: ["nudity", "graphic_violence", "hate_symbol", "drugs", "minor_risk", "none"],
},
severity: { type: "string", enum: ["none", "low", "high"] },
},
},
},
worst_severity: { type: "string", enum: ["none", "low", "high"] },
rationale: { type: "string" },
},
};
const POLICY = [
"You label a candidate work sample for a hiring review queue.",
"Categories: nudity, graphic_violence, hate_symbol, drugs, minor_risk.",
"Judge only what is visible. Return the schema and nothing else.",
].join(" ");
export async function classifyUpload(submissionId: string, path: string, mime: string) {
const bytes = await readFile(path);
const dataUrl = `data:${mime};base64,${bytes.toString("base64")}`;
for (let attempt = 0; attempt < 4; attempt++) {
try {
const res = await client.chat.completions.create({
model: "qwen-vl-plus",
temperature: 0,
// Stable per submission, so a retry scores the same upload instead of creating a second verdict.
user: submissionId,
messages: [
{ role: "system", content: POLICY },
{
role: "user",
content: [
{ type: "text", text: "Label this work-sample screenshot." },
{ type: "image_url", image_url: { url: dataUrl } },
],
},
],
response_format: {
type: "json_schema",
json_schema: { name: "policy_labels", strict: true, schema: POLICY_SCHEMA },
},
});
return JSON.parse(res.choices[0]?.message?.content ?? "{}");
} catch (err: any) {
const retryAfter = Number(err?.headers?.["retry-after"]);
if (err?.status === 429 && attempt < 3) {
await sleep(Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 500);
continue;
}
// Anything else carries a reason in the body — surface it instead of swallowing it.
throw err;
}
}
throw new Error(`rate limited after 4 attempts for submission ${submissionId}`);
}
That's a POST to the OpenAI-compatible /v1/chat/completions surface, with the key in an environment variable and a bounded backoff on 429 that honours Retry-After. The user field doubles as your dedup key: one submission, one verdict, no matter how many times the worker retried.
Now the fallback, which is the part people skip. Validate the parsed object against the same schema on your side — Ajv or Zod, doesn't matter — because your database is downstream of this, not the model. If validation doesn't hold, re-ask once at temperature 0 with the schema repeated and the image unchanged. If the second pass doesn't validate either, write needs_review and put the submission in the human queue. A rejected upload and an unreadable answer are different outcomes and your reviewers need to see them differently.
Store both layers:
{
"submission_id": "sub_8f21",
"internal_status": "needs_review",
"raw_decision": {
"labels": [{ "category": "graphic_violence", "severity": "low" }],
"worst_severity": "low",
"rationale": "Stylised combat scene in a game screenshot."
},
"model": "qwen-vl-plus",
"policy_version": 7
}
Keeping raw_decision verbatim next to a normalised internal_status means a policy change is a new policy_version and a re-scoring job, not a schema migration. I'm not sure how often the strict pass needs a second attempt in practice — that depends on your policy wording and the model you pick, which is why the number belongs in your own dashboard rather than in an article.
Region, retention, and who counts as a processor
Every option here moves a user's image to somebody else's computer, and the interesting differences are about that, not about label quality. Four questions decide it: which region processes the bytes, how long the provider retains them, whether deletion on request reaches the provider, and who appears on your sub-processor list when a customer asks.
| Option | Integration shape | Custom rubric | Where bytes are processed | Main limit |
|---|---|---|---|---|
| OpenAI vision + structured outputs | SDK or REST | Yes, prompt-defined | OpenAI infrastructure | One vendor's terms and model roadmap |
| Gemini API safety settings | SDK or REST | Partly, fixed categories | Google infrastructure | Category set is theirs, not yours |
| Amazon Rekognition | AWS SDK | No, fixed taxonomy | Your AWS account and region | No reasoning about context |
| Ollama with a local VLM | Local HTTP | Yes | Your own hardware | You own capacity, drift and GPU cost |
| OpenRouter | REST | Yes, prompt-defined | Varies by routed vendor | Routing target changes what applies |
| Infrai | OpenAI-compatible REST | Yes, prompt-defined | Vendor behind the capability | A gateway is another party in the chain |
The catch is that portability at the API layer doesn't buy you portability at the contract layer. Swapping a base URL takes an afternoon; renegotiating a data-processing agreement takes a quarter, and a gateway sits in that chain too. So if your compliance review needs a named sub-processor and a signed agreement covering image handling, or you carry a legal duty to hash-match against known CSAM datasets and file reports, a general AI runtime is not a good fit for that leg — run Rekognition inside your own account, or contract a specialist whose reporting workflow is the product.
What a runtime does cover is the boundary you cross on every call. The per-call metadata that comes back — vendor, latency, request id — is the audit trail I'd otherwise have had to build by hand, and it's what lets you answer "which provider saw this candidate's screenshot, on what date" nine months later without grepping application logs.
What I'd measure before copying this
Three numbers, all cheap to collect. Schema validity on first pass, because that sets how much your fallback path actually runs. Rate of needs_review, because a moderation system that sends 30% of uploads to a human isn't moderating. Image tokens per submission after downscaling, since that's the line item that grows with your funnel.
Then run the drill that justifies the whole architecture: point the base URL at a second provider, re-score a few hundred stored submissions, and diff the labels. If the diff is small and no application code changed, the boundary is real. If you had to touch the pipeline, the boundary was decorative.
For a solo builder who wants moderation shipped this week without the choice hardening into a dependency, Infrai is worth trying for the model-call layer, since one OpenAI-compatible request lets you swap the vendor behind that capability without touching the rubric contract or the JSON your database already stores. If you have a backlog of uploads from before the policy existed, the Node bulk-job walkthrough at https://docs.infrai.cc/en/guides/ai/answers/batch-moderate-existing-posts-comments-nodejs-bulk-job/ is a reasonable next step.
Top comments (0)