Build a UGC video moderation pipeline with FFmpeg + NudeNet

TL;DR

If your product lets strangers upload video, you need moderation before launch, not after the first bad upload. We will build a small-team pipeline: extract frames with FFmpeg, score them with NudeNet (ONNX Runtime, CPU-friendly), route uploads into approve / human-review / block by confidence, and log every decision. No trust-and-safety department required.

📦 Code: github.com/USER/ugc-moderation, replace before publishing

⚠️ Note: this is a sensitive area. The goal here is the engineering shape (sampling, scoring, routing, auditing), not detection of any specific content. Keep test fixtures clean and lawful.

A model does not decide what is allowed. It produces a score. You decide where the lines go. The whole design is about routing scores sensibly and sending the uncertain middle to a human.

The architecture 🧠

upload ──> extract sample frames (ffmpeg)
        ──> score frames (NudeNet / ONNX)
        ──> aggregate to one confidence
        ──> route:
              high-confidence clean   -> auto-approve
              uncertain middle band   -> human review queue
              high-confidence violation -> auto-block
        ──> write an audit record for every decision

The economics only work if the middle band is small. A decent model makes most uploads obviously fine or obviously not, so a human only ever sees the genuinely ambiguous slice.

1. Sample frames, do not score every frame

You cannot afford every frame and you do not need it. Pull one frame per second (or scene-change keyframes) with FFmpeg.

# extract 1 frame per second into ./frames
mkdir -p frames
ffmpeg -i upload.mp4 -vf "fps=1" -q:v 3 frames/frame_%05d.jpg

Prefer scene changes to catch more variety with fewer frames:

# keyframes where the scene actually changes
ffmpeg -i upload.mp4 -vf "select='gt(scene,0.3)',showinfo" -vsync vfr frames/scene_%05d.jpg

💡 Tip: a 30-minute upload at 1 fps is ~1,800 frames. Scene-change sampling often cuts that by an order of magnitude with little loss for moderation purposes.

2. Score frames with NudeNet

NudeNet runs on ONNX Runtime on plain CPU, the default model is about 7 MB, and it scores an image in roughly 100 to 300 ms on CPU (well under 50 ms on GPU). Install:

python3 -m venv venv && source venv/bin/activate
pip install nudenet onnxruntime

Score every sampled frame and keep the highest-risk signal:

# score.py
from pathlib import Path
from nudenet import NudeDetector

detector = NudeDetector()  # loads the default ~7MB ONNX model

# labels you treat as risk signals (tune to your policy)
RISK_LABELS = {
    "FEMALE_BREAST_EXPOSED",
    "FEMALE_GENITALIA_EXPOSED",
    "MALE_GENITALIA_EXPOSED",
    "BUTTOCKS_EXPOSED",
    "ANUS_EXPOSED",
}

def score_frame(path: str) -> float:
    """Return the max risk-label confidence found in one frame (0.0 - 1.0)."""
    detections = detector.detect(path)
    risks = [d["score"] for d in detections if d["class"] in RISK_LABELS]
    return max(risks, default=0.0)

def score_video(frames_dir: str) -> float:
    frames = sorted(Path(frames_dir).glob("*.jpg"))
    if not frames:
        return 0.0
    return max(score_frame(str(f)) for f in frames)

We aggregate with max because one clearly bad frame is enough to flag a video. Depending on policy you might also track how many frames cross a threshold.

3. Route by confidence into three buckets

This is the core decision logic. Two thresholds, three outcomes:

# route.py
from enum import Enum

class Decision(str, Enum):
    APPROVE = "approve"
    REVIEW = "review"
    BLOCK = "block"

# tune per surface; public feeds should be stricter than private channels
APPROVE_BELOW = 0.30   # below this: confidently clean
BLOCK_ABOVE = 0.85     # above this: confidently a violation

def route(confidence: float) -> Decision:
    if confidence < APPROVE_BELOW:
        return Decision.APPROVE
    if confidence > BLOCK_ABOVE:
        return Decision.BLOCK
    return Decision.REVIEW  # the human band

⚠️ Note: do not auto-block on a single model with thin margins. The middle band exists so borderline calls reach a person. Make it wide enough that you are not silently auto-deciding hard cases.

4. The boring parts that actually save you 🗂️

The model gets the glory; the audit log keeps you out of trouble. Log every decision.

# audit.py
import json, time, uuid
from pathlib import Path

LOG = Path("moderation_audit.jsonl")

def record(upload_id, confidence, decision, reviewer=None):
    entry = {
        "id": str(uuid.uuid4()),
        "upload_id": upload_id,
        "confidence": round(confidence, 4),
        "decision": decision,
        "reviewer": reviewer,          # set when a human acts on a REVIEW item
        "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }
    with LOG.open("a") as f:
        f.write(json.dumps(entry) + "\n")
    return entry

Wire it together:

# pipeline.py
from score import score_video
from route import route, Decision
from audit import record

def moderate(upload_id: str, frames_dir: str):
    confidence = score_video(frames_dir)
    decision = route(confidence)
    record(upload_id, confidence, decision.value)
    if decision == Decision.REVIEW:
        enqueue_for_human(upload_id, confidence)  # your review-queue impl
    return decision

def enqueue_for_human(upload_id, confidence):
    print(f"[review] {upload_id} needs eyes (confidence={confidence:.2f})")

5. Know your blind spots

Frame sampling has real limits, and pretending otherwise is how you ship a false sense of safety:

• Audio and on-screen text are invisible to an image model. A clean-looking video can still violate policy through its soundtrack or captions. Multimodal scoring is the direction; until then, your reporting flow and human reviewers cover this gap.
• A single bad frame between samples can slip through. Tighten sampling for high-risk surfaces.
• Models drift and adversaries adapt. Re-tune thresholds with real review-queue outcomes.

For the cases your local model is unsure about, a cloud video-intelligence API (which scores categories like adult, violent, and suggestive content and reasons across the timeline) makes a good second-pass escalation. Run the cheap local model first, pay for the API only on the middle band.

6. Tune thresholds from your own review queue

You will not get APPROVE_BELOW and BLOCK_ABOVE right on day one, and you should not guess them forever. Once humans have judged a few hundred review-queue items, you have labeled data: the model's confidence plus the human verdict. Use it to move the lines.

# tune.py: suggest thresholds from human-reviewed outcomes
# rows: (confidence, human_verdict)  where verdict in {"clean","violation"}
def suggest_thresholds(rows):
    cleans = sorted(c for c, v in rows if v == "clean")
    violations = sorted(c for c, v in rows if v == "violation")
    if not cleans or not violations:
        return None
    # approve below the 95th percentile of confirmed-clean scores
    approve_below = cleans[int(0.95 * (len(cleans) - 1))]
    # block above the 5th percentile of confirmed-violation scores
    block_above = violations[int(0.05 * (len(violations) - 1))]
    return round(approve_below, 3), round(block_above, 3)

If approve_below creeps above block_above, your model cannot cleanly separate the classes at the current sampling rate. That is a signal to sample more frames, add a second-pass model, or widen the human band, not to force the thresholds together.

⚠️ Note: review every threshold change like a deploy. Loosening BLOCK_ABOVE to shrink the queue is exactly how bad content starts slipping through.

What's next

• Build the human review queue as a real internal tool, not a SQL query. The people reviewing this content deserve decent tooling.
• Make thresholds configurable per surface without a deploy.
• Add audio transcription and on-screen text extraction for a multimodal second pass.
• Add reviewer wellbeing support; this work is hard on people.

If your app has an upload button and not this pipeline, build it before launch, while the only thing on the line is your time and not your reputation.