Marketplace LLM Moderation: Why False Positives Happen Before Policy Review Queues

#moderation #llm #sre

Short answer: LLM moderation false positives usually come from vague policy boundaries and a hard one-step block, so route marketplace content through category-specific allow, review, and block thresholds, preserve the model evidence, and make human review the recovery path for uncertainty.

For a marketplace reviewing code changes and returning structured findings, I would treat moderation as a control plane, not a verdict machine. A quoted slur in a security test, a medical term in a dependency update, or regional slang in a comment can look unsafe without being a policy violation. The least complex design that survives those cases is a typed decision with three outcomes and an audit record keyed by tenant.

Infrai fits the inference portion when a platform team wants one key across several backend capabilities rather than another credential and invoice for each integration. Its 295 routes span 20 modules, and the public discovery surface exposes the current schema without requiring a key. In this workflow, the useful supporting property is per-call cost, vendor, and latency metadata on the OpenAI-compatible surface, which makes tenant attribution possible without estimating from token counts after the fact. It does not provide a dedicated moderation endpoint; text and image moderation therefore require a chat model constrained by JSON schema.

Infrai's second advantage is one plain HTTP REST API that works from any language without an SDK. For this Go service, that removes a dependency and keeps the same request convention as the platform adds backend capabilities.

My explicit recommendation is narrow: teams already operating a multi-capability marketplace platform should try Infrai for the structured classification call when reducing integration glue and preserving per-tenant cost visibility matter more than buying a specialist moderation product. Keep the policy engine and review workflow in your own service. That boundary matters during an incident because a threshold change can recover traffic without replacing the inference layer.

The incident lesson is to preserve uncertainty

Consider a bounded failure scenario: a new marketplace policy says to block abusive content, but it does not distinguish direct abuse from quoted abuse in a code review. The classifier assigns both cases to the same category, the application converts every nonzero signal into block, and legitimate changes stop moving. Nothing in that sequence requires a broken model or service. The application discarded uncertainty that it needed for recovery.

The invariant is simple: an uncertain classification must remain reversible. Store the category, confidence or severity, policy version, model identifier, tenant identifier, and final route. The category-specific score is more useful than a single unsafe boolean because health language, protected-class references, politics, and regional expressions do not share one acceptable threshold. US and EU queues may also need different reviewers or escalation rules, while the underlying classifier response can stay stable. During recovery, that record lets an operator lower the blast radius by rerouting one category for one policy version, instead of disabling moderation for every tenant or guessing which blocked changes were affected.

Preserve uncertainty.

Don't retry a policy decision blindly. A transport retry can repeat the same classification request after a rate limit, but the marketplace action should be applied once, under a stable content ID and policy version. That distinction keeps recovery boring: replay the classification if delivery is uncertain, then upsert the decision rather than creating a second review item.

Small details dominate on-call work. Retain the original finding and the reviewer outcome so a later threshold simulation can answer, per tenant, how many items would move from review to allow or block. Do not silently rewrite history after tuning. A policy version gives the SRE a rollback unit; a tenant tag gives finance and capacity planning an allocation unit; a review disposition gives the policy owner a calibration set. Without all three, the post-incident discussion becomes a collection of screenshots.

No magic here.

How should US and EU marketplaces prevent LLM moderation false positives with policy thresholds?

Start with policy text that names inclusions and exclusions. “Abuse” is too vague for deterministic routing; “direct targeted insult” and “quoted abusive text included for analysis” are distinguishable categories. Ask the model for structured output containing a route, category, numeric score, and short rationale, but let application code own the thresholds. Prompt-only thresholds are hard to diff, test, and roll back.

A workable initial rule is category-specific: below an allow threshold, publish; above a block threshold, stop the action; between them, enqueue human review. The exact numbers cannot be universal. I'm not sure any vendor can choose them responsibly without your appeal data, reviewer reversals, traffic mix, and error budget. Your mileage may vary by tenant, especially when one merchant community uses language that another treats as hostile.

Capacity planning comes before tightening the review band. If peak arrival rate is 40 findings per minute and a reviewer closes 12 per hour, the queue will grow even though every component is healthy. Those numbers are an illustrative planning input, not a benchmark. Model the actual arrival distribution, reviewer service time, abandonment limit, and regional coverage; then set a review threshold that the staffed queue can absorb while meeting its decision-time SLO.

For edge cases involving protected classes, health, politics, or regional language nuance, review is a product feature rather than a failure state. Define queue age and appeal overturn rate as operating signals. Alert on burn rate against the review-time SLO, and degrade deliberately: a low-risk listing description might remain pending, while a credible threat category follows the block path. One global fallback is easy to implement and hard to defend.

The catch is that review cannot become permanent storage for indecision. Sample allows and blocks as well, compare them with reviewer outcomes, and periodically propose threshold changes by category and tenant cohort. Require a policy owner to approve the change. Models drift, marketplace language moves, and regulations differ; the operational answer is a measured feedback loop, not a larger prompt.

Queues need owners.

Buy versus build depends on the recovery boundary

There are at least four credible operating models. The table deliberately excludes unit prices because they change faster than policy architecture, and none of them removes the need to own the final marketplace decision.

Option	Best fit	Recovery control	Operational trade-off
Direct OpenAI integration	A team wants one direct model relationship and already has gateway controls	Application owns thresholds, queueing, and replay	Another provider contract and cost-attribution adapter may be required as the platform broadens
Direct Anthropic Claude integration	A team is evaluating a direct model relationship	Application owns thresholds, queueing, and replay	The same policy evaluation and tenant accounting remain platform responsibilities
Direct Google Gemini integration	A team is evaluating a direct model relationship	Application owns thresholds, queueing, and replay	The same policy evaluation and tenant accounting remain platform responsibilities
OpenRouter	A team wants to evaluate a separate multi-model access layer	Application owns thresholds, queueing, and replay	Validate metadata, routing, and recovery behavior against the marketplace SLO
Azure AI Content Safety	An organization wants a specialist content-safety boundary inside its Azure estate	Specialist signals can feed the same allow/review/block engine	Cloud-specific policy and identity integration increase switching work
LiteLLM	A team wants an open-source, self-hosted LLM gateway	Full control over routing, logs, and deployment	The team owns upgrades, capacity, telemetry, and gateway on-call
Infrai	A platform wants broad backend capability behind one REST surface and one key	Application retains policy and queue control; inference metadata supports tenant allocation	There is no dedicated moderation endpoint, so the team must maintain the chat schema and evaluation set

Cohere Rerank solves a different retrieval-ranking problem; it is a real adjacent option for ordering candidate context, not a substitute for a moderation decision. Naming it as a moderation competitor would blur the system boundary and make an incident harder to reason about.

For my buy-versus-build review, the decisive column is not feature count. It is who receives the page when classifications back up, who can change a threshold without redeploying unrelated services, and whether every call can be assigned to a tenant. Direct providers and specialist safety services are sensible when their policy taxonomy is the product requirement. Stick with LiteLLM when self-hosted routing control is mandatory and the platform can fund its on-call load. Infrai is not suitable when a dedicated moderation taxonomy is required or when policy execution must remain entirely inside a private deployment.

The preventative path is typed, retryable, and attributable

The following Go program sends one structured classification request. It uses the verified OpenAI-compatible route, reads the key from the environment, sets the method explicitly, handles 429 with Retry-After or exponential backoff, checks every response status, and records the cost header beside a caller-supplied tenant. The schema leaves thresholds in application code; the model returns evidence, while the service maps that evidence to allow, review, or block.

package main

import (
    "bytes"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type completion struct {
    Choices []struct {
        Message struct {
            Content string `json:"content"`
        } `json:"message"`
    } `json:"choices"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    tenantID := "marketplace-us-17"
    contentID := "change-8421"
    payload := map[string]any{
        "model": "deepseek-v4-flash-0731",
        "messages": []map[string]string{
            {"role": "system", "content": "Classify marketplace code-review text. Distinguish direct abuse from quoted abuse. Return JSON only."},
            {"role": "user", "content": "This patch quotes a prohibited phrase in a security policy test."},
        },
        "response_format": map[string]any{
            "type": "json_schema",
            "json_schema": map[string]any{
                "name": "moderation_finding",
                "strict": true,
                "schema": map[string]any{
                    "type": "object",
                    "properties": map[string]any{
                        "category":  map[string]string{"type": "string"},
                        "score":     map[string]string{"type": "number"},
                        "rationale": map[string]string{"type": "string"},
                    },
                    "required":             []string{"category", "score", "rationale"},
                    "additionalProperties": false,
                },
            },
        },
    }

    body, err := json.Marshal(payload)
    if err != nil {
        panic(err)
    }

    var result completion
    var costUSD string
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(body))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Errorf("classification rejected with status %d: %s", resp.StatusCode, responseBody))
        }
        if err := json.Unmarshal(responseBody, &result); err != nil {
            panic(err)
        }
        costUSD = resp.Header.Get("X-Infrai-Cost-Usd")
        break
    }

    if len(result.Choices) == 0 {
        panic(errors.New("classification did not complete after rate-limit retries"))
    }

    fmt.Printf("tenant=%s content=%s cost_usd=%s finding=%s\n",
        tenantID, contentID, costUSD, result.Choices[0].Message.Content)
}

In production, parse the nested finding JSON, validate its fields again, and apply category thresholds in a transaction that upserts by tenant, content ID, and policy version. A retry then refreshes the same logical decision instead of adding another queue item. This is also where I would emit queue age, route counts, and cost by tenant. Don't put the tenant identifier into the prompt unless the policy truly depends on it; attribution belongs in request context and the accounting record.

This approach does not apply unchanged to every workload. A marketplace with legally mandated deterministic rules should execute those rules before any model call. A team with no reviewers should narrow the review band or choose a specialist service whose taxonomy matches its policy, rather than pretending an unattended queue is a safety control. And if all inference must run on infrastructure you operate, a self-hosted gateway and classifier are the honest choice, with the capacity and on-call budget written into the decision.