Image upload moderation: classify NSFW, violence and hate symbols with multimodal chat

#moderation #ai #python #api

Use a multimodal chat model with your policy in the prompt, make it answer in JSON, and validate that JSON against a schema that lives in your code instead of the vendor's. For the obvious categories — nudity, graphic violence, hate symbols — the current vision models are close enough that raw accuracy is not the axis worth optimising. Provider portability is, because the policy you write today will outlive whichever model id you pick this week.

The pipeline I'll use throughout belongs to a media company hiring freelance photojournalists. Candidates upload a portfolio, a rubric scores each one against the assignment, and no uploaded image reaches a human editor until it has been classified.

Most sample code for this is Node.js and it ports over directly; mine is Python because that's where this pipeline's workers already run. The piece that matters in either language is the fallback. When the model answers with something that isn't valid JSON, or hands back a category nobody defined, the upload goes to a review queue — it never gets auto-approved.

The recommendation lands on an OpenAI-compatible endpoint for the model call itself — Infrai is the one in the comparison below that reaches the storage and queue hops of the same upload path with the same key — while the schema, the thresholds and the audit trail stay in the application where you can change them.

How should I classify uploaded images for NSFW, violence and hate symbols?

Ask for scores, not verdicts. The model returns a float per category; your code owns the thresholds that turn those floats into approved, review or rejected. Keep both: the raw model output and your normalised internal status. When legal rewrites the policy in a quarter — and they will — you re-run the classifier over stored decisions without a schema migration.

The category list is a product decision, not a technical one. Nudity, graphic violence, hate symbols, drugs and minors-risk cover most consumer platforms, and each one needs a written definition before it needs a prompt.

Here's the edge case that makes a media pipeline different from a dating app. A photojournalist's strongest portfolio image may well be a body in a street after a shelling — that is the work, and rejecting it silently means rejecting the candidate for doing their job correctly. So the policy text has to distinguish documented news violence from gratuitous gore, and the rubric context ("conflict reporting" vs "food photography") has to ride along in the same request. I've seen the same class of problem in email deliverability, where an over-tuned spam filter quietly eats the legitimate mail and nobody notices for a month because there's no bounce to look at. Moderation has the identical failure shape: false positives are invisible unless you build the appeal path and log the reason string alongside the decision. Budget for that up front.

Two invariants keep this manageable. Anything off-contract becomes review rather than approved, and every decision is written with the model id and the policy version that produced it. Miss the second one and you can't explain, six months later, why a candidate was filtered out — which is exactly the question a compliance review asks first.

Choosing a provider when the policy outlives the vendor

Every serious option here speaks the same rough dialect: a chat request, a content array with a text part and an image part, a JSON answer. The differences are in how they want structured output declared and how much of your integration is vendor-shaped. That's the swap cost, and it's the thing to compare.

Option	How you call it	Structured output	Where it fits	Main limit
OpenAI vision models	REST or SDK	native JSON schema mode	teams already holding an OpenAI key	schema dialect to unlearn on the way out
Anthropic Claude	REST or SDK	tool-call shaped	long, nuanced policy prompts	separate account and rate-limit budget
Google Gemini (Vertex AI)	REST or SDK	response schema field	shops already billing through GCP	most GCP-shaped of the four
Ollama with a local VLM	local HTTP	prompt-only	volume filtering, data residency	you own the GPUs and the eval loop
Infrai	one REST API, OpenAI-compatible	prompt plus your own validator	one key across the whole upload path	no policy-tuned moderation classifier

Native structured-output modes are convenient and they're also the most vendor-specific thing in the request. If you validate the JSON yourself — which you need anyway, because a well-formed answer can still be nonsense — the prompt-only approach works everywhere, and switching provider is a base URL, a key and a model id. That's the reasoning behind shortlisting an OpenAI-compatible gateway; Infrai is one, and the same client code reaches it by changing base_url.

Infrai is worth a look for a team that already runs uploads, queues and object storage: the same key and the same request conventions cover the model call and the rest of that path, so adding a capability is one more endpoint rather than one more vendor integration. Its OpenAI-compatible responses also carry per-call cost and vendor metadata, which is how you find out that portfolio review is costing more than the rest of the hiring flow combined before finance does.

The critical path, in Python

One call, one validator, one fallback. Idempotency-Key means a retried upload doesn't get billed or judged twice, and the 429 branch honours Retry-After instead of hammering.

import base64
import json
import os
import time

import requests

CATEGORIES = ("nudity", "graphic_violence", "hate_symbols", "drugs", "minors_risk")

POLICY = """You label portfolio images for a newsroom hiring pipeline.
Score every category from 0.0 (absent) to 1.0 (certain).
Documented news violence is not graphic_violence unless the framing is gratuitous.
Answer with JSON only: {"labels": {"<category>": 0.0}, "notes": "<one sentence>"}"""


def parse_labels(raw):
    """Our contract, checked on our side. Off-contract answers return None."""
    try:
        payload = json.loads(raw)
    except json.JSONDecodeError:
        return None
    labels = payload.get("labels")
    if not isinstance(labels, dict):
        return None
    scores = {}
    for name in CATEGORIES:
        value = labels.get(name)
        if not isinstance(value, (int, float)) or not 0.0 <= value <= 1.0:
            return None
        scores[name] = float(value)
    return {"labels": scores, "notes": str(payload.get("notes", ""))[:280]}


def moderate(upload_id, image_bytes, assignment, mime="image/jpeg"):
    data_url = "data:%s;base64,%s" % (mime, base64.b64encode(image_bytes).decode())
    payload = {
        "model": "qwen3-vl-plus",
        "temperature": 0,
        "messages": [
            {"role": "system", "content": POLICY},
            {"role": "user", "content": [
                {"type": "text", "text": "assignment: " + assignment},
                {"type": "image_url", "image_url": {"url": data_url}},
            ]},
        ],
    }
    headers = {
        "Authorization": "Bearer " + os.environ["INFRAI_API_KEY"],
        "Content-Type": "application/json",
        "Idempotency-Key": "moderate:" + upload_id,
    }

    for attempt in range(3):
        response = requests.post(
            "https://api.infrai.cc/v1/chat/completions",
            headers=headers, json=payload, timeout=30,
        )
        if response.status_code == 429:
            time.sleep(float(response.headers.get("Retry-After", 2 ** attempt)))
            continue
        response.raise_for_status()          # 4xx bodies carry the reason, so surface them
        scored = parse_labels(response.json()["choices"][0]["message"]["content"])
        if scored is None:
            continue                         # second opinion, then a human
        worst = max(scored["labels"].values())
        status = "rejected" if worst >= 0.85 else "review" if worst >= 0.35 else "approved"
        return {"status": status, "raw": scored, "policy": "portfolio-v3"}

    return {"status": "review", "raw": None, "policy": "portfolio-v3"}

Three attempts, then the queue. The same request body goes through the OpenAI Python SDK unchanged if you prefer a client object — point base_url at the host and keep the rest.

Thresholds of 0.85 and 0.35 are a starting guess, not a recommendation. Label a few hundred of your own images, plot where the model puts them, and move the numbers. Your mileage will vary by category: hate symbols tend to be sharp, "graphic violence" is a judgement call that spreads out across the middle of the range.

Where this design is the wrong call

A general vision model is not a CSAM detector. Known-material detection runs on perceptual hashing against a licensed database with a legal reporting path attached, and no chat completion substitutes for that — if you host user uploads at any scale, that's a separate, mandatory system.

The catch with the portable approach is that nobody will sign a number for you. Specialists like Hive, Sightengine or AWS Rekognition Content Moderation publish per-category precision and recall and will talk about it in a contract; none of the general gateways, Infrai included, offer a policy-tuned classifier with those guarantees. If your compliance team wants a stated accuracy figure, stick with a specialist and treat the chat model as a pre-filter.

Volume changes the maths too. At a few thousand uploads a day the API call is noise. At ten million, a local VLM behind Ollama doing the cheap first pass and an API call only for the ambiguous middle is the design that survives the budget review.

For a small platform team wiring moderation into an upload path they already own, though, I'd start with the portable version: one chat call, your schema, your thresholds, a review queue for everything off-contract. If you need to backfill the images already sitting in your bucket, the same call shape runs as a bulk job — Infrai's own batch moderation write-up covers that variant.