AI Engine

Posted on Oct 20 Originally published at ai-engine.net

NSFW Filter for AI-Generated Images: Check Before You Show

#ai #security #api #python

If your product generates images (an avatar app, an AI art tool, an ad creative generator, a chatbot that draws), every output is content you publish. An NSFW filter for AI-generated images checks each output before a user sees it, so a bad generation becomes a retry instead of a screenshot on social media.

Every number below is a real response from a live moderation API, on images produced by a text-to-image model.

👉 Try the NSFW Detect API on your own generations

Why prompt filters and built-in checkers are not enough

Harmless prompts can produce unsafe images. Models fill in details nobody asked for.
Users route around word lists with misspellings, other languages and image-to-image uploads.
Built-in checkers are coarse. The Stable Diffusion safety checker in diffusers returns one boolean per image (or None when it is not loaded), and the pipeline accepts safety_checker=None. A yes/no flag cannot tell alcohol from nudity.
Hosted providers moderate on their terms, and that changes when you switch provider.

Where the check goes

Prompt filter, then generate, then moderate the bytes, and only then storage, CDN or client. Never upload to a public bucket and moderate from the URL: if the check fails, the image is already public.

Step 1: moderate the generated bytes

import requests

NSFW_URL = "https://nsfw-detect3.p.rapidapi.com/nsfw-detect"
HEADERS = {
    "x-rapidapi-key": "YOUR_API_KEY",
    "x-rapidapi-host": "nsfw-detect3.p.rapidapi.com",
}

def moderate(image_bytes, filename="output.png"):
    """Send generated image bytes to the moderation API, return its labels."""
    r = requests.post(
        NSFW_URL,
        headers=HEADERS,
        files={"image": (filename, image_bytes, "image/png")},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["body"]["ModerationLabels"]

On a generated photo of a man smoking, the API returned Smoking under Drugs & Tobacco Paraphernalia & Use under Drugs & Tobacco, all at 97.1%. Key your policy on the label with an empty ParentName (the top-level category): a generated joint came back under the sub-label Pills, but its top-level category was still Drugs & Tobacco at 99.7%.

Step 2: a policy for generated content

# Top-level category -> (action, minimum confidence). Absent = allowed.
POLICY = {
    "Explicit": ("block", 50),
    "Non-Explicit Nudity of Intimate parts and Kissing": ("block", 50),
    "Violence": ("block", 60),
    "Visually Disturbing": ("block", 60),
    "Hate Symbols": ("block", 40),
    "Drugs & Tobacco": ("block", 60),
    "Swimwear or Underwear": ("review", 60),
    "Gambling": ("review", 50),
    "Rude Gestures": ("review", 60),
}

def decide(labels):
    verdict, reasons = "allow", []
    for label in labels:
        if label["ParentName"]:  # keep top-level categories only
            continue
        rule = POLICY.get(label["Name"])
        if not rule or label["Confidence"] < rule[1]:
            continue
        reasons.append(f"{label['Name']} ({label['Confidence']:.1f}%)")
        if rule[0] == "block":
            verdict = "block"
        elif verdict == "allow":
            verdict = "review"
    return verdict, reasons

The thresholds are set per category, so one table covers every style your generator produces: the illustrated casino was caught as Gambling and sent to review. An adult (18+) art platform would flip several rows (nudity and swimwear allowed), but keep violence and hate symbols blocked.

👉 Full guide with the policy table and latency measurements

Step 3: generate_safely() with retries and an audit log

import hashlib
import json
import random
import time

def audit(record):
    with open("moderation_audit.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

def generate_safely(prompt, generate, user_id, max_attempts=3):
    """generate(prompt, seed) is your image model or provider, returning PNG bytes."""
    for attempt in range(1, max_attempts + 1):
        seed = random.randint(0, 2**31 - 1)
        image = generate(prompt, seed)
        started = time.time()
        try:
            verdict, reasons = decide(moderate(image))
        except requests.RequestException as e:
            verdict, reasons = "review", [f"moderation unavailable: {e}"]  # fail closed
        audit({
            "ts": time.time(), "user": user_id, "prompt": prompt, "seed": seed,
            "attempt": attempt, "verdict": verdict, "reasons": reasons,
            "sha256": hashlib.sha256(image).hexdigest(),
            "check_ms": round((time.time() - started) * 1000),
        })
        if verdict == "allow":
            return {"status": "ok", "image": image, "seed": seed}
        if verdict == "review":
            return {"status": "held_for_review", "seed": seed, "reasons": reasons}
        # blocked: discard this image and try a new seed
    return {"status": "refused", "reasons": reasons}

Block means regenerate with a new seed (capped), review means hold, errors fail closed.
Log the decision and a hash, not the image.
Repeated refused results for one user are a probing pattern: rate-limit or review that account.

Latency

About 0.8 s for a 140 KB JPEG and 1.3 s for a 1.2 MB PNG, end to end. Moderate the exact bytes you publish.

The rules are tightening

Google Play's AI-Generated Content policy: prevent restricted content, offer in-app reporting or flagging.
EU AI Act, Article 50: machine-readable marking of synthetic images from 2 August 2026.

Production tips

Calibrate on your own presets: run a few hundred outputs per style through the API and tune the per-category thresholds.
Start strict, then relax: send borderline categories to review first.
Extend to video: generated clips use the same policy on sampled frames (NSFW video and GIF moderation in Python).

Sources

👉 Get a key on the NSFW Detect API page and point the wrapper at your generator.

DEV Community