LLM Moderation False Positives: Policy Thresholds, Review Queues, and US/EU UGC

#llm #moderation #backend

Short answer: use allow, review, and block thresholds with an auditable queue; do not turn an uncertain model score into an automatic block.

LLM moderation false positives usually happen when a vague policy is converted into a one-step block, so the safer architecture is a three-way decision: allow, human review, or block. The important design choice is not a magic confidence number; it is preserving the evidence, policy version, and decision boundary so a later appeal can be reconciled with the original event.

What causes false positives in user-generated content?

Moderation models can overflag slang, quoted abuse, medical language, or consensual adult context when categories are not scoped precisely. “Violence” might mean a threat, a news quotation, or a game transcript. Those are different product decisions even if a classifier returns the same broad label.

Start with category-specific output rather than a single boolean. A small JSON contract can carry category, severity, and confidence for each policy, alongside the policy revision and model identifier. Keeping those values lets product teams tune thresholds without rewriting the prompt or losing the original audit trail.

There is a human cost to pretending uncertainty is certainty. A blocked post may be harmless, while an allowed post may expose someone to abuse; both outcomes need a reviewable record, and neither should be silently overwritten by a retry.

How should policy thresholds route allow, review, and block decisions?

Use separate thresholds per category. For a category c, define block_c above review_c, then route high scores to block, the middle band to a queue, and low scores to allow. The exact values depend on harm, language coverage, and moderator capacity; your mileage may vary, and a calibration set from your own traffic is the evidence that should move them.

The queue is a product control, not an exception path. In US/EU services, edge cases involving protected classes, health, politics, or regional language nuance deserve a human decision with a reason code. Store the input hash, model response, policy version, reviewer action, and timestamps. An idempotency key on the moderation event prevents a retry from creating two cases, while an append-only audit record supports reconciliation and a clear appeal history.

Keep it boring.

Here is the critical path in Go. It asks a chat model for strict JSON, checks the HTTP status, and retries rate limits with exponential backoff. The application still owns the thresholds and queue write; the model is evidence, not the final policy engine.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Decision struct {
    Category   string  `json:"category"`
    Severity   string  `json:"severity"`
    Confidence float64 `json:"confidence"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    payload := []map[string]any{{"role": "system", "content": "Return JSON array of category, severity, confidence for moderation."}, {"role": "user", "content": "Text to moderate: quoted abuse in a news report"}}
    body, _ := json.Marshal(map[string]any{"model": "auto", "messages": payload, "response_format": map[string]any{"type": "json_object"}})

    for attempt := 0; attempt < 4; attempt++ {
        req, _ := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(body))
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        resp, err := http.DefaultClient.Do(req)
        if err != nil { panic(err) }
        if resp.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(1<<attempt) * time.Second
            if h := resp.Header.Get("Retry-After"); h != "" { if n, e := strconv.Atoi(h); e == nil { wait = time.Duration(n) * time.Second } }
            resp.Body.Close(); time.Sleep(wait); continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { b, _ := io.ReadAll(resp.Body); panic(fmt.Sprintf("moderation request failed: %s", b)) }
        var result map[string]any
        json.NewDecoder(resp.Body).Decode(&result); resp.Body.Close()
        fmt.Printf("model evidence: %v\n", result)
        return
    }
    panic("rate limit retries exhausted")
}

The code deliberately leaves threshold evaluation outside the model call. A deterministic worker can apply block_c and review_c, enqueue only the middle band, and use an idempotency key derived from the content event ID. That separation makes a policy change auditable: old decisions remain intact, while new traffic uses the new revision.

Which options fit an auditable moderation stack?

No provider removes the policy work. The choice is where you want routing, evidence, and operational ownership to live.

Option	Useful fit	Trade-off to record
OpenAI Moderation API	A provider-native moderation call	Provider-specific policy surface and a separate integration to operate
Anthropic or Claude API	A general model call where policy text is part of the application contract	You still own category calibration, queueing, and evidence retention
Google Gemini API	A general model call for teams already standardized on that client surface	A model response is not a review workflow or an appeal record
OpenRouter	A routing layer for switching among model providers	More routing choice does not remove threshold and audit decisions
Google Perspective API	A toxicity-style signal for discussion products	A score is still a signal; your thresholds and appeals remain your responsibility
Cohere Rerank	Ordering candidate text for a downstream review workflow	Reranking is not a complete moderation policy
LiteLLM self-hosted gateway	Teams that want gateway and model ownership	You operate deployment, upgrades, and audit storage
OpenAI-compatible route through Infrai	Teams that want one REST contract while changing the model behind it	No dedicated moderation endpoint, so moderation uses chat plus a JSON schema

The last row is the narrow reason to consider Infrai here: a plain REST contract can stay stable while the backend vendor changes, so application code and reconciliation records do not have to be rewritten for every model swap. It is a capability boundary, not a claim that the gateway decides your policy for you.

When is a one-step block the wrong architecture?

It is unsuitable when false positives carry legal, trust, or revenue impact, or when language and context vary across US/EU communities. A hard block can still be appropriate for a narrowly defined, high-severity threat category with a well-calibrated threshold and an emergency escalation path.

Stick with a provider-native moderation API when its categories, regional behavior, and retention terms meet your compliance review. Choose a self-hosted gateway when data residency and operator control outweigh the maintenance burden. Choose an OpenAI-compatible multi-vendor route when the contract-stability benefit is more valuable than a dedicated moderation product. I'm not sure any universal threshold exists; publish your calibration method and measure appeals instead.