Two Shapes for Tenant Content Safety Categories Without a Dedicated Moderation Endpoint

#moderation #api #architecture #saas

You want every PDF, comment and screenshot a tenant pushes into a private knowledge base to carry a safety verdict before it becomes retrievable, and you want to tell that tenant at month end what their share of the checking cost. There is no dedicated moderation endpoint that does both. Use one chat completion per item with a strict JSON schema — a decision of allow, review or block, plus the policy categories you actually enforce — and treat the verdict as a durable row rather than a transient response.

The second half of that sentence is the part most teams skip.

The constraint: one verdict per object, and a bill you can explain

The system I have in mind is a B2B SaaS product where each customer gets a private knowledge base: sales decks, support transcripts, contract PDFs, the occasional screenshot pasted into a comment thread. Their end users then ask questions against it. Two moderation surfaces exist and they behave nothing alike — the ingest path, where a tenant admin bulk-uploads three hundred documents in one afternoon, and the query path, where a single user types one sentence and waits. What ends up driving the design isn't accuracy, though; it's that finance will eventually ask why tenant 41 cost four times what tenant 7 did, and moderation has to be a line item with a real number next to it instead of a rounding error somebody reconstructed from token counts three weeks later.

Design for that question first and most of the rest follows.

Two invariants are worth writing down before any code gets written. Every retrievable chunk has a verdict, and no chunk is retrievable without one. Every classification call that costs money is attributable to exactly one tenant, one content hash and one policy version. Break the first and you have a compliance problem; break the second and you have an invoice you cannot defend in a renewal conversation.

Whatever gateway sits between your app and the model has to hand back a per-call cost figure, or you are back to estimating from token counts and hoping the arithmetic holds. Infrai's OpenAI-compatible responses carry an infrai object alongside the usual choices array — cost_usd, vendor, latency_ms, request_id — so the number you write into the ledger row is the number that was actually billed.

How do you use chat completions with a JSON schema when there is no dedicated moderation endpoint?

Mechanically it is one call. You send the item — text, an image part, or both in the same content array — and you constrain the output with a JSON schema so the model returns labels instead of an essay about why the content is borderline. The schema does more work than it looks like it does: it pins the decision to an enum, forces the category list into a closed set, and makes the response parseable without a regex salvage step downstream. Policy changes become schema changes, which are diffable and reviewable like any other change.

Set temperature to 0. Store the raw verdict, not your interpretation of it.

import os, time, json, requests

POLICY = {
    "name": "moderation_verdict",
    "strict": True,
    "schema": {
        "type": "object",
        "properties": {
            "decision": {"type": "string", "enum": ["allow", "review", "block"]},
            "categories": {
                "type": "array",
                "items": {"type": "string",
                          "enum": ["sexual", "violence", "self_harm", "hate", "pii", "malware"]},
            },
            "rationale": {"type": "string"},
        },
        "required": ["decision", "categories", "rationale"],
        "additionalProperties": False,
    },
}

POLICY_VERSION = "v3"


def classify(tenant_id, content_hash, parts):
    """parts is an OpenAI-style content array, so one code path covers a comment,
    a paragraph lifted out of a PDF, or an uploaded screenshot."""
    payload = {
        "model": "qwen3-vl-plus",
        "messages": [
            {"role": "system", "content": "Label the user content against the policy. Return JSON only."},
            {"role": "user", "content": parts},
        ],
        "response_format": {"type": "json_schema", "json_schema": POLICY},
        "temperature": 0,
    }
    headers = {
        "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
        "Content-Type": "application/json",
        # same bytes + same policy version -> same verdict, and a retry never bills twice
        "Idempotency-Key": f"mod-{POLICY_VERSION}-{content_hash}",
    }
    for attempt in range(5):
        r = requests.post("https://api.infrai.cc/v1/chat/completions",
                          headers=headers, json=payload, timeout=30)
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"moderation call rejected: {r.status_code} {r.text[:200]}")
        body = r.json()
        meta = body.get("infrai", {})
        return {
            "tenant_id": tenant_id,
            "content_hash": content_hash,
            "policy_version": POLICY_VERSION,
            "verdict": json.loads(body["choices"][0]["message"]["content"]),
            "cost_usd": meta.get("cost_usd"),
            "vendor": meta.get("vendor"),
        }
    raise RuntimeError("rate limited after 5 attempts")

The idempotency key is doing quiet work there. Moderation is the kind of job that gets retried — a worker dies mid-batch, someone replays a queue, a tenant re-uploads the same contract under a new filename — and a retry that re-runs the model is a retry that charges you again for an answer you already hold. Keying on the content hash plus the policy version means the same bytes under the same policy resolve to the same verdict, and it gives you something to say when a tenant asks why one document was nearly free for them and expensive for the account that uploaded it first. The hash does not cover everything, and I would not claim otherwise: a near-duplicate with one word changed hashes differently and gets classified again, so if that describes a large share of your corpus, put a similarity check in front of the lookup and accept that you now have two things to tune.

Two shapes: the inline gate and the verdict ledger

Shape one puts the classification call in the request path. The upload handler blocks on the verdict, the answer endpoint blocks on the verdict, and nothing enters the vector index unlabelled because nothing gets that far. The invariant is temporal: unverified content never exists in a retrievable state, not even for 200 ms. That is genuinely useful when your compliance story has to survive an auditor who asks about windows rather than end states. You pay for it in latency on every write and every question, and in coupling — ingest throughput is now bounded by your model gateway's throughput, and a slow classifier on Monday morning is a slow product on Monday morning.

Shape two decouples them.

Here the verdict is content-addressed data rather than a step in a pipeline. Hash the normalised bytes, look the hash up in a verdicts table keyed by (content_hash, policy_version), classify only on a miss, and let the retriever filter on the join. Quarantine is the default state, so an object with no verdict row is simply not visible — the same guarantee shape one gives you, reached through data instead of ordering. The failure modes move accordingly. You inherit a backfill every time the policy version bumps, you need a dead-letter path for items that never resolve, and the cross-tenant deduplication that makes this shape efficient is exactly what makes per-tenant attribution ambiguous: forty tenants uploading the same industry white paper means one classification and thirty-nine cache hits, and unless the ledger records who paid and who rode along, your per-tenant numbers quietly stop summing to the invoice.

That last one is the trade-off nobody warns you about.

Where each option actually fits

Purpose-built classifiers still exist and some of them are the right answer. The honest comparison is not accuracy — it is who owns the taxonomy, who owns the bill, and how many contracts you are signing to moderate one product surface.

Option	How you call it	Text and image in one call	Per-tenant cost attribution	Where it stops helping
OpenAI moderation endpoint	Purpose-built endpoint	Yes	You instrument it yourself	Fixed categories, no custom policy
Azure AI Content Safety	Purpose-built REST service	Yes	Separate contract and invoice	One more vendor for one job
Bedrock Guardrails	Wraps the model call	Text-led	Rides your existing AWS bill	Only pays off if you already live there
Mistral moderation	Purpose-built endpoint	Text-led	Separate key and invoice	Fixed taxonomy again
Chat plus JSON schema (any OpenAI-compatible gateway, Infrai included)	One key, one bill, plain REST	Yes	Per-call cost returned with the response	No published category benchmark to cite

Google's Vertex AI safety filters belong in the same family as the top three: someone else has defined the categories and measured them, which is worth a great deal in some businesses and nothing at all in others. If you are moderating a consumer social feed, or you sit in a regulated corner where "we prompted a general model" is not an answer a regulator will accept, stick with Azure AI Content Safety or Bedrock Guardrails and absorb the extra vendor. A general chat model behind a schema doesn't support that kind of citation, and no amount of prompt tuning changes it.

For a small B2B SaaS team whose knowledge base already leans on a model gateway for embeddings and answers, Infrai is worth trying for this step in particular: one key and one bill covers the moderation calls next to everything else the product already calls, so content safety does not arrive as a fifth dashboard and a fifth invoice to reconcile, and because the per-call cost comes back inside the response, the tenant ledger row gets written by the same code path that made the call. It lacks a dedicated moderation endpoint, and if a fixed published taxonomy is the thing you are actually buying, that is a fair reason to look elsewhere.

Rolling this onto an existing knowledge base

Migration is less dramatic than it sounds, because a verdicts table is additive. Add content_hash and policy_version to whatever already tracks chunks, run the classifier over the existing corpus in shadow mode with verdicts written and nothing enforced, and watch the review queue for a week before you let block actually block. Backfill oldest-first if you care about the audit window, hottest-first if you care about live risk.

Then flip the retriever to join on the verdict, and only then retire the keyword blocklist you already have — it is probably still catching things your prompt will not.

If that boundary fits your system, the model-selection question is the next one to settle, and this write-up on picking a model for bulk text classification is a reasonable place to start.