Upload Moderation: Node.js NSFW, Violence, Hate-Symbol Classification + JSON Fallback

#node #moderation #multimodal

Short answer: for media support tickets that include an image, keep classification, policy enforcement, and tenant cost accounting as three separate steps. Send the image to a multimodal chat model with a strict JSON Schema, validate the returned object locally, and send invalid or uncertain cases to review. The fallback is a queue, not a guess.

That design matters because a support agent is usually triaging a complaint, not publishing a photo. The same upload might be evidence of a violent broadcast, a screenshot containing a hate symbol, or an ordinary account avatar. A boolean called safe throws away the context that the agent needs.

Keep it boring.

How can a Node.js image moderation flow classify risky uploads without trusting JSON?

Start with a versioned taxonomy. For this media workflow, I would keep nsfw, violence, and hate_symbols as separate observations, add uncertain, and retain a short evidence string. The model describes what it can see; application code decides whether a ticket is visible, blocked, or waiting for a human. This boundary also makes an eval harness useful: a prompt change can be tested independently from the enforcement policy.

The tempting shortcut is to ask for a sentence and search it for words. It feels flexible in a notebook, then becomes difficult to replay: punctuation changes the parser, a missing category looks like a negative result, and a tenant's policy cannot be reconstructed from a free-form answer. Typed output is not a safety decision, but it gives the rest of the pipeline a stable input.

Here is a deliberately small adapter. The surrounding Node.js upload service can call the same contract from any language; the example keeps the model call behind an OpenAI-compatible chat client and uses environment variables for the concrete base URL and model. It does not publish an upload merely because the response parses.

import json
import os

from openai import OpenAI


MODERATION_SCHEMA = {
    "name": "media_upload_labels",
    "strict": True,
    "schema": {
        "type": "object",
        "additionalProperties": False,
        "required": ["nsfw", "violence", "hate_symbols", "uncertain", "evidence"],
        "properties": {
            "nsfw": {"type": "boolean"},
            "violence": {"type": "boolean"},
            "hate_symbols": {"type": "boolean"},
            "uncertain": {"type": "boolean"},
            "evidence": {"type": "string", "maxLength": 240},
        },
    },
}


def classify_upload(image_data_url: str) -> dict:
    client = OpenAI(
        api_key=os.environ["MODERATION_API_KEY"],
        base_url=os.environ["MODERATION_BASE_URL"],
        timeout=30.0,
        max_retries=0,
    )
    response = client.chat.completions.create(
        model=os.environ["MODERATION_MODEL"],
        messages=[
            {
                "role": "system",
                "content": (
                    "Classify only visible evidence in the supplied image. "
                    "Set uncertain when the image or its context is insufficient."
                ),
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Return labels for NSFW, violence, and hate symbols.",
                    },
                    {"type": "image_url", "image_url": {"url": image_data_url}},
                ],
            },
        ],
        response_format={
            "type": "json_schema",
            "json_schema": MODERATION_SCHEMA,
        },
    )
    content = response.choices[0].message.content
    if not content:
        return {"status": "review", "reason": "empty_model_response"}

    try:
        labels = json.loads(content)
    except json.JSONDecodeError:
        return {"status": "review", "reason": "invalid_json"}

    required = {"nsfw", "violence", "hate_symbols", "uncertain", "evidence"}
    if set(labels) != required or labels["uncertain"]:
        return {"status": "review", "labels": labels}
    if labels["nsfw"] or labels["violence"] or labels["hate_symbols"]:
        return {"status": "hold", "labels": labels}
    return {"status": "triage", "labels": labels}

The production adapter still needs upload-size and MIME validation before the model call, access controls around stored images, and a timeout policy at the worker boundary. A rate limit, empty response, or schema mismatch belongs on the same review path as visual uncertainty. Avoid logging the image or sensitive ticket text into ordinary application logs.

Where do tenant cost visibility and moderation policy meet?

Per-tenant cost is a decision axis, not a reason to weaken the classifier. Make one moderation record for each image attempt, keyed by tenant, ticket, policy version, model version, and request ID. Record input and output token counts when the provider exposes them, plus latency, retry count, result status, and review outcome. The resulting ledger answers questions that a monthly invoice cannot: which tenant sends the largest images, which policy creates the most reviews, and whether a prompt edit increased tokens without improving recall.

I keep the accounting record beside the decision record, but I don't let it choose the label. A large customer may need a stricter review SLA; that is an operational rule, separate from whether an image contains violence. Token counting can be estimated before launch with tiktoken, then compared with the runtime usage fields. Estimates are useful for experiments, not a substitute for the provider's measured usage.

Choice	Helps with	Cost or limitation to verify
Multimodal chat plus a schema	One adaptable interface for image evidence and a small typed result	The application owns taxonomy, validation, and enforcement
Dedicated moderation classifier	A maintained safety taxonomy and a focused review workflow	Its labels may not map cleanly to a media team's policy
Self-hosted vision model	Keeping images inside an organization's boundary	The team owns serving, model updates, calibration, and evaluation
Human-first triage	Ambiguous context, appeals, and high-impact decisions	Queue volume and response time need an explicit service target

The catch is that a single cost metric can reward the wrong optimization. Shrinking prompts may reduce token usage while increasing uncertain results and reviewer time. Conversely, sending every image through a more expensive path may improve recall but make a small tenant's support workflow impractical. Compare total operational cost per resolved ticket, not just model tokens, and keep that comparison segmented by tenant.

Measure it.

What should the fallback, schema, and review record contain?

Treat the JSON Schema as a contract and the fallback as a state transition. The minimum states here are triage, hold, and review; they are not synonyms for model labels. review must carry a reason such as invalid_json, empty_model_response, or uncertain, while hold carries the observed category and the policy version that caused the hold. A reviewer should be able to see the original image, relevant ticket context, model evidence, and the exact prompt and schema versions without editing the original event.

Do not silently coerce a missing field to false. That turns an integration problem into a false clean result. Do not retry forever, either: use a bounded worker budget, make the request idempotent, and expose queue age to the support team. If the same ticket is replayed after a model update, preserve both decisions and identify which version produced each one.

Context is the hard part. A hate symbol in a news report, a violent frame submitted as evidence, and an endorsement may share pixels but call for different treatment. Image-only automation is not suitable when caption, conversation history, jurisdiction, or an appeal determines the action. Stick with human review when those inputs are material, and let the policy say what evidence is required before release.

Which measurements justify copying this design?

Freeze a labeled set of real-looking media support cases before changing the prompt. Include ordinary avatars, screenshots, documentary material, partial occlusion, low resolution, and legitimate discussions of harmful imagery where those cases occur in the product. Have reviewers label each category independently from the final action. Then compare configurations on false negatives, false positives, per-category recall, uncertain rate, schema-valid response rate, review volume, latency, and token usage. Keep the examples that caused disagreement instead of averaging them away: a single screenshot with a partially obscured symbol may expose a taxonomy ambiguity, while a low-resolution frame may expose a quality boundary, and both can change the queue experience for one tenant even when the aggregate score looks better. I would also rerun the exact same cases through the fallback path, because a malformed response should be measured as a review event and not disappear from the quality report.

The failure modes should be visible in the report. A parser failure means the contract or integration needs attention. A disagreement on a clear image points to the classifier or prompt. A disagreement caused by missing ticket context belongs in the review design. A policy change may alter the correct action without changing the original visual evidence, which is why both records need separate versions.

I've kept prompt-cost awareness beside quality metrics because the cheap-looking experiment is often the expensive one after review hours are counted. Your mileage may vary for culturally specific symbols and borderline images; the way to resolve that uncertainty is a local, adjudicated test set, not a stronger adjective in the prompt.

Three words: fail closed.

Before launch, decide what the customer sees while a result is pending, who can release a held image, how long evidence is retained, and how tenants can appeal. Run the same evaluation after every model, prompt, schema, or policy change. The useful artifact is not a clever chat response; it is a replayable moderation record whose classification, enforcement, review, and cost can be inspected separately.