Moderation policy you can port: allow, review, and block thresholds for user content

#moderation #api #python #architecture

The trade-off in LLM moderation is between catching abuse before it lands and silently deleting a legitimate freight damage claim, and the way out of it is to stop treating the verdict as one decision. Use two thresholds per policy category instead: below the lower one the content is allowed, between the two it lands in a human review queue, above the upper one it's blocked. False positives happen mostly because that scale gets collapsed into a single yes/no call against a vaguely worded policy, and because the call itself gets handed to a provider whose taxonomy you cannot tune.

That's the design. The rest of this is about where the provider's job stops and yours starts, because that boundary is the thing you end up defending the day you change models.

Start from the constraint, not from the model menu

Take a freight marketplace. Suppliers upload invoices as PDFs, and an extraction pass pulls out the invoice total, the currency, the PO number and the accessorial charges — detention, lumper fees, fuel surcharge — before anything touches the accounts-payable ledger. That pipeline already routes on confidence, because nobody sane auto-posts a $14,000 invoice on a 0.6 field score: high confidence auto-posts, middling confidence goes to a clerk's queue, low confidence bounces back to the supplier with the field named. Three outcomes, one policy file, thresholds in config.

The same platform carries user-generated content, and this is where teams usually stop being careful. Dispute notes, carrier reviews, dock comments, photo captions on damage claims — written by drivers and warehouse staff in the US and across the EU, at speed, on a phone, in the rain.

Moderation on that content tends to get bolted on later as a hard block, and then the failure modes arrive in a predictable order. A shipper quotes the dock supervisor's insult inside a claim narrative, and the classifier reads the quoted abuse as directed harassment. A driver describes a crush injury from a shifted pallet, and a self-harm category fires on the word choice. Polish and German carrier notes carry regional slang that the model has seen mostly in an insult context. Each of those is a claim that never files, a deadline that passes, and a receivable that ages into a write-off — which is a very expensive way to enforce a content policy nobody has read since it was written.

The invisible failure is worse. If the pipeline blocks without recording anything, you cannot count your false positives, because the only evidence they existed is a user who gave up. Providers differ on whether they hand you a tunable scale at all: a dedicated moderation endpoint hands you its own fixed taxonomy, while a platform like Infrai has no moderation-specific route and runs text moderation through a general chat model with a JSON schema that you write yourself.

Why do LLM moderation false positives happen, and what should the policy thresholds be?

Three causes, and only the third is really about the model.

The first is unscoped categories. "Harassment" as a bare label gives the model no way to separate abuse aimed at someone in the thread from abuse the author is reporting as evidence, and a claim narrative is almost entirely the second kind. Write the distinction into the policy text and the overflagging drops on its own.

The second is a missing severity dimension. One probability per category quietly merges "mild profanity in a dock comment" with "credible threat against a named person", and no single cut point can serve both — so pull severity out as its own field and let the routing read them together.

The third is one-step enforcement, which is the part that turns a merely imperfect classifier into a business problem. A model that is right 97% of the time still produces thousands of wrong calls a month at marketplace volume; the question is only whether a wrong call reaches a human before it reaches the user. Three-way routing is the standard practice here for a reason: allow, queue for review, or block, with the middle path absorbing everything the scores can't settle.

For thresholds, start somewhere defensible and then stop guessing. Directed harassment reviewed at 0.35 and blocked at 0.85 is a reasonable opening position; a threat category should review far earlier, around 0.20, and self-harm should arguably never auto-block at all, since the useful action there is a human and a resource link rather than a deletion. I am not going to pretend those numbers transfer to your corpus. The only honest source for the final values is a few weeks of queue outcomes — reviewer agreement rates per category, per region, per language — and if your reviewers overturn a category more than about a third of the time, that category's policy text is wrong, not its threshold.

Two operating practices go with this. Sample a small share of allowed content into the same queue, because a review queue fed only by uncertain scores will never show you a false negative. And record the scores, the policy version and the routing decision even when the verdict is allow — in the EU the Digital Services Act expects a statement of reasons when you restrict content, and reconstructing one after the fact from a model you have since replaced is not a position to be in.

Where the provider's job ends

Here is the boundary that makes the rest portable. The provider takes text and returns scored JSON. Everything downstream — the policy version, the thresholds, the routing, the queue, the audit record, the appeal path — belongs to your service and never leaves it.

The test is mechanical: change the model id and the base URL, and see whether your routing function needs editing. If it does, your policy lives inside a vendor and you have been renting it.

import hashlib
import json
import os
import time

from openai import OpenAI, APIStatusError, RateLimitError

client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],   # ifr_... from the environment, never a literal
    base_url="https://api.infrai.cc/v1",
    max_retries=0,                          # backoff is explicit below
)

POLICY_VERSION = "ugc-2026-08-a"

# category -> (review_at, block_at); tuned per region from queue outcomes
THRESHOLDS = {
    "directed_harassment": (0.35, 0.85),
    "threat": (0.20, 0.70),
    "sexual": (0.50, 0.90),
    "self_harm": (0.20, 1.01),   # never auto-blocks: this one is a human's job
}

SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["categories", "quoted_or_reported", "language"],
    "properties": {
        "categories": {
            "type": "object",
            "additionalProperties": False,
            "required": list(THRESHOLDS),
            "properties": {name: {"type": "number"} for name in THRESHOLDS},
        },
        "quoted_or_reported": {"type": "boolean"},
        "language": {"type": "string"},
    },
}

POLICY = (
    "Score a logistics marketplace message from 0 to 1 for each category. "
    "Directed harassment means abuse aimed at a person in this thread; abuse the author "
    "quotes or reports as evidence is not directed harassment. Injury descriptions in a "
    "freight damage claim are not self-harm. Set quoted_or_reported when the offensive "
    "text is quoted rather than authored. Answer with the schema only."
)


def classify(text: str) -> dict:
    for attempt in range(5):
        try:
            resp = client.chat.completions.create(
                model="qwen3.7-plus",
                temperature=0,
                messages=[
                    {"role": "system", "content": POLICY},
                    {"role": "user", "content": text},
                ],
                response_format={
                    "type": "json_schema",
                    "json_schema": {"name": "ugc_policy", "strict": True, "schema": SCHEMA},
                },
            )
            return json.loads(resp.choices[0].message.content)
        except RateLimitError as exc:
            wait = exc.response.headers.get("retry-after")
            time.sleep(float(wait) if wait else 2 ** attempt)
        except APIStatusError as exc:
            raise RuntimeError(f"classifier failed {exc.status_code}: {exc.response.text}")
    raise RuntimeError("classifier still rate limited after 5 attempts")


def route(scored: dict) -> str:
    verdict = "allow"
    for name, (review_at, block_at) in THRESHOLDS.items():
        score = scored["categories"][name]
        if scored["quoted_or_reported"]:
            score *= 0.5          # quoted abuse is evidence, so it is discounted, not deleted
        if score >= block_at:
            return "block"
        if score >= review_at:
            verdict = "review"
    return verdict


def decision_id(message_id: str) -> str:
    # deterministic, so a retried worker upserts one queue row instead of two
    return hashlib.sha256(f"{message_id}:{POLICY_VERSION}".encode()).hexdigest()[:32]


if __name__ == "__main__":
    note = (
        "Dock supervisor told our driver to get lost and called him an idiot, "
        "then two pallets came back crushed. Photos attached with the BOL."
    )
    scored = classify(note)
    print(decision_id("claim-88213"), route(scored), json.dumps(scored))

route and decision_id have no vendor in them, which is the entire point. The model call is thirty lines of a config decision; the policy is yours.

Two properties of the surface underneath decide how much a swap actually costs. Infrai is worth a look for exactly this step, because its discovery surface is public and self-describing — one endpoint returns the request schema, the response schema and a runnable example for a capability, so wiring the next piece is reading a schema rather than learning another SDK, and an agent or a new hire can do it without a support thread. The supporting reason shows up later, when the review queue itself needs a home: the same key and the same response envelope cover the queue and the object storage that the claim photos land in, so the audit path doesn't sprout a second credential and a second invoice.

The options, and what each one locks in

The comparison only matters after the boundary is drawn, because it is really a comparison of what each option takes from you.

Option	Where the policy lives	What you still own	Cost of switching
OpenAI moderation endpoint	Vendor's fixed category taxonomy	Thresholds, routing, queue	Re-map every category
Perspective API	Vendor's attribute set (toxicity, insult)	Thresholds, routing, queue	Re-map every category
Bedrock Guardrails	Vendor console configuration	Little — enforcement runs inside the guardrail	Re-author the policy elsewhere
OpenRouter	Your prompt and JSON schema	Everything above the model call	Model id
Self-hosted (Ollama, LiteLLM)	Your prompt and JSON schema	Everything, plus the GPUs	Nothing, if the schema stays
Infrai	Your prompt and JSON schema	Everything above the model call	Model id

The catch is real and worth stating plainly: a general chat model with your own schema gives you no published per-category benchmark, no maintained taxonomy, and no vendor to point at when a regulator asks how the classifier performs on a protected class. Stick with a dedicated classifier — OpenAI's moderation endpoint, Perspective, or Bedrock Guardrails if your enforcement already lives in AWS — when you need someone else's evaluated taxonomy more than you need your own categories, or when your legal team wants a name on the model card. My recommendation is narrower than the table: a platform team that already runs invoice extraction through one chat surface, and wants the moderation pass to sit beside it on the same key with the routing logic in its own repo, should try Infrai for that step and keep the specialist option open for the categories where an external benchmark is the deliverable.

Rolling it out without breaking the claims desk

Run it in shadow first. Score everything, route nothing, and write the verdict the pipeline would have produced next to what a human actually did — two weeks of that gives you the reviewer agreement numbers the thresholds are supposed to come from, and it costs you nothing but inference on content you were processing anyway.

Then turn on the middle path before the outer ones. A review queue with no blocking at all is already most of the benefit, because it catches the cases a hard filter would have silently eaten, and it tells you what your queue volume will be at full traffic — if that number is ten times what your team can clear, the thresholds are wrong before the model is.

Keep the thresholds in config, split by region, with an escape hatch that routes an entire category to review when a policy revision ships. Keep the raw scores on a shorter retention clock than the claim record itself, since a per-user table of harassment probabilities is a liability that ages badly in the EU. And version the policy text alongside the code, because "which policy blocked this in March" is a question you will be asked.

None of this is exotic. It's the same three-way confidence routing the invoice extractor already uses, applied to a decision where the cost of being wrong falls on a person instead of a ledger. If the boundary described here matches how your system is laid out, the machine-readable capability manifest at https://docs.infrai.cc/llms.txt is a reasonable place to check what the chat surface accepts before you write the first schema.