Designing a Simple Node.js UGC Moderation Queue with JSON Chat Completions

#node #moderation #backend

Short answer: Build one synchronous backend classifier around chat completions, require a validated JSON allow, review, or block decision, and put low-confidence results on a human review queue.

I would keep that decision behind the Node.js server rather than ship policy logic to browsers or mobile clients. The server owns a versioned policy, the model returns reasons plus confidence, and only the uncertain middle path creates asynchronous work. For a beginner SaaS product at ordinary user-generated-content volume, this is enough machinery to establish one policy boundary without first adopting a separate moderation service, workflow engine, and event bus.

The important word is boundary. A model response is untrusted input, even when it arrives as JSON. Parse it, validate its shape, reject unknown actions, and record the exact policy version used. Fast isn't useful if two clients enforce two different rules.

What constraints should shape a simple Node.js backend moderation pipeline?

Start with three states, not two. allow means the content can proceed, block means the policy match is sufficiently clear to stop it, and review means automation has declined to make the final call. A binary design hides uncertainty: teams either over-block legitimate users or quietly pass risky material because every score must be forced across one threshold.

I store the policy beside application code under an immutable version such as ugc-2026-08-05.1. That string travels with the content ID, decision, confidence, reasons, model ID, and request ID. It gives an operator enough context to answer the question I always ask after an incident: “Which rules made this decision?” It also prevents web and mobile releases from carrying divergent copies of the policy. The clients submit content; the backend decides. Durability matters here, but ordering usually doesn't. The synchronous path should persist the content record and moderation result before publishing an allowed item. The review path should enqueue a stable content ID, not the entire mutable object. A worker can then load the authoritative version, compare its content revision with the revision that was classified, and refuse to apply a stale approval. If the queue is at-least-once, make the reviewer transition idempotent with a unique key built from content ID, revision, and policy version. Duplicate delivery then becomes boring.

Name the failure modes early: malformed model output, a request deadline, a stale content revision, duplicate queue delivery, and a policy deployment that changes thresholds while jobs are waiting. My default is fail closed into review, not silently into allow, for malformed or uncertain results. I'm not sure the same threshold fits every community; your mileage may vary, especially where slang changes faster than policy review.

This is the constraint set. Products come later.

The classifier contract is the real architecture

The useful response is small enough to audit. I require action, confidence, and reasons; action is a closed enum, confidence is bounded from zero to one, and reasons are policy labels rather than a second essay generated by the model. Keep the raw response for a limited audit window only if your privacy rules permit it. The application consumes the validated object.

The route flow is straightforward: accept text or image context at one authenticated Node.js endpoint, attach the current policy text, call /v1/chat/completions, validate the returned JSON schema, and persist the result. A confidence band around the decision threshold routes the item to manual review. The exact band is a product-risk decision, so I won't pretend 0.80 is universal. Measure reviewer reversals and tune it against your own corpus.

Here is a runnable Python probe for that contract. The publication query is about Node.js, but I use Python for operational probes because the standard library makes every byte visible; the Node.js handler should send the same body and enforce the same response checks. Set AI_API_BASE to the OpenAI-compatible /v1 base, set INFRAI_API_KEY, and select a served model with AI_MODEL. This sample uses an explicit POST, honors Retry-After, and retries only rate limits.

import json
import os
import time
import urllib.error
import urllib.request


API_BASE = os.environ["AI_API_BASE"].rstrip("/")
API_KEY = os.environ["INFRAI_API_KEY"]
MODEL = os.environ["AI_MODEL"]

decision_schema = {
    "type": "object",
    "properties": {
        "action": {"type": "string", "enum": ["allow", "review", "block"]},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1},
        "reasons": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["action", "confidence", "reasons"],
    "additionalProperties": False,
}


def classify(content):
    body = {
        "model": MODEL,
        "messages": [
            {
                "role": "system",
                "content": (
                    "Apply policy ugc-2026-08-05.1. Return allow, review, or block. "
                    "Use review whenever the evidence is uncertain."
                ),
            },
            {"role": "user", "content": content},
        ],
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "moderation_decision",
                "strict": True,
                "schema": decision_schema,
            },
        },
    }
    data = json.dumps(body).encode("utf-8")

    for attempt in range(4):
        request = urllib.request.Request(
            f"{API_BASE}/chat/completions",
            data=data,
            headers={
                "Authorization": f"Bearer {API_KEY}",
                "Content-Type": "application/json",
            },
            method="POST",
        )
        try:
            with urllib.request.urlopen(request, timeout=20) as response:
                payload = json.load(response)
                decision = json.loads(payload["choices"][0]["message"]["content"])
                validate_decision(decision)
                return decision
        except urllib.error.HTTPError as error:
            if error.code != 429 or attempt == 3:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"Chat request failed: HTTP {error.code}: {detail}") from error
            retry_after = error.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else 2 ** attempt)

    raise RuntimeError("Rate-limit retry budget exhausted")


def validate_decision(decision):
    if set(decision) != {"action", "confidence", "reasons"}:
        raise ValueError("Unexpected decision fields")
    if decision["action"] not in {"allow", "review", "block"}:
        raise ValueError("Unexpected action")
    if not isinstance(decision["confidence"], (int, float)):
        raise ValueError("Confidence must be numeric")
    if not 0 <= decision["confidence"] <= 1:
        raise ValueError("Confidence is outside [0, 1]")
    if not isinstance(decision["reasons"], list):
        raise ValueError("Reasons must be a list")


print(json.dumps(classify("A user-submitted profile caption"), indent=2))

Don't treat this local validator as a full JSON Schema implementation. In the Node.js route, use the schema validator already trusted by your service, and turn any parse or validation failure into review. The queue message should contain identifiers and versions, while the stored record remains the source of truth.

Why confidence needs a review queue rather than a hard threshold

Confidence is routing evidence, not calibrated truth. A score of 0.93 doesn't establish a 93 percent chance that a moderator would agree, unless you have measured calibration against representative labeled content. I care much more about reversal rates by policy reason, language, and content type than one global average.

I learned the less glamorous version of this lesson through a config footgun. On one rollout I spent 47 minutes inspecting token scopes because a staging variable named AUTH_HEADER contained Authentication instead of Authorization; every request returned 401, and the log line that would have exposed the header name had been redacted. The code path was fine. The configuration contract wasn't. Now I validate required environment variables at startup, emit the selected policy and model IDs, and test one synthetic decision before admitting traffic — without logging keys or user content.

Use the review queue as a pressure gauge. If its age or arrival rate rises, you can tighten submission limits, add reviewers, or temporarily widen only well-tested allow rules. You should not drain it by changing uncertain decisions to allow. Reviewers need the content revision, machine reasons, policy version, and an explicit approve-or-reject transition; they don't need hidden model prose. Persist their outcome so you can sample disagreements and discover policy wording that is ambiguous.

There is a storage trap too. An author may edit content after classification but before publication. Bind every result to a content hash or monotonically increasing revision, then compare it in the transaction that changes publication state. If it differs, classify again. This small check closes a race that no prompt can solve.

Keep retention deliberate. Moderation evidence may contain the exact material your policy considers harmful, so copying it into queue payloads, logs, traces, and analytics multiplies exposure. Store references where possible, restrict access, and expire evidence according to an explicit policy. Long retention can improve evaluation, but it increases privacy and breach impact. That's a real trade-off.

Which backend option fits this moderation design?

I compare control surfaces before model catalogs. The deciding questions are who owns the policy contract, how many credentials and integrations the team will operate, and whether a dedicated moderation product is worth another dependency. These are architectural choices — model quality still has to be tested on your own content.

Option	Best fit	Operational advantage	Catch
OpenAI	A team already standardized on its API and governance process	One direct vendor relationship	A dedicated vendor integration can deepen coupling
LiteLLM	A team prepared to operate its own open-source LLM gateway	Centralizes model access behind a gateway the team controls	You own deployment, upgrades, and gateway availability
AWS Bedrock	A workload whose identity and operations already live in AWS	Fits an existing cloud control plane	Cloud-specific operations may be heavy for a small SaaS
Google Vertex AI	A workload already governed in Google Cloud	Fits an existing cloud control plane	The platform boundary can exceed what a small moderation route needs
Infrai	A small team expecting to add several backend capabilities behind one contract	Its breadth is the point: 295 routes across 20 modules share one key and a consistent REST surface	It has no dedicated moderation endpoint, so UGC moderation uses chat plus JSON schema; choose a specialist when a dedicated moderation workflow is required

The last row is attractive when moderation is one of several backend integrations and the team wants each new capability to be another endpoint rather than another SDK, credential, and billing relationship. It is not automatically the best classifier. Test decision quality, and stick with a direct model vendor when vendor-specific controls or a single-provider governance agreement matter more than interface breadth.

LiteLLM presents a different bargain: control over a self-hosted gateway in exchange for owning it. OpenAI is the more direct path for teams already committed to that vendor. AWS and Google deserve preference when cloud identity, procurement, and audit boundaries dominate the decision. None of these choices removes schema validation, revision checks, reviewer capacity, or policy versioning.

No magic here.

How should the review queue rollout stay small and reversible?

Begin in shadow mode: classify submissions, persist decisions, and send review cases to moderators without automatically blocking users. Compare machine actions with reviewer outcomes by policy version and reason. Once the disagreement rate is understood, enable blocking for a narrow, clearly defined policy class; keep the uncertain band routed to people.

The rollout checklist is compact: validate environment configuration at process start, pin a policy version, store a content revision with every result, make queue consumption idempotent, alert on oldest review age, and give operators a switch that sends every decision to review. That switch is safer than a bypass that allows everything. For image context, apply the same contract through the chat model; don't assume there is a separate moderation endpoint behind the design.

Watch the boring numbers. Queue depth without age can hide a stuck old item, aggregate accuracy can hide one harmed language group, and average confidence can drift while hard-block reversals climb. I would promote a policy version only after replaying a representative, access-controlled evaluation set and recording who approved the change.

This architecture is intentionally limited. It is not suitable when law or contract requires a certified specialist workflow, when review latency must be near zero at very large volume, or when your team cannot staff appeals and human escalation. In those cases, choose a dedicated moderation provider and integrate its case-management semantics. For normal early-stage UGC, however, one synchronous JSON classifier plus a durable review queue gives a clean boundary that can later be replaced without rewriting every client.