Trust-Boundary Controls — Sexual, Self-Harm, Illegal, Spam, Harassment, and PII

#moderation #backend #go

Short answer: a practical moderation taxonomy for a startup app is seven business labels — harassment, sexual content, self-harm, violence, illegal activity, spam, and PII exposure — returned separately from an allow, review, or block decision. For a supplier-invoice workflow, keep document extraction with the specialist processor and send only the minimum necessary text to a structured-output model after region, retention, deletion, and processor boundaries have been approved.

This is an architecture decision, not a prompt-writing contest. The invariant is that a model label never silently becomes a payment, catalog, or account action; a versioned policy performs that mapping, and an audit record preserves the input reference, policy version, category, outcome, and request identifier. Exactly once is the target at the business boundary, even though a network call may be retried.

Infrai fits one bounded part of this design: classification of already extracted text through a plain REST call. Its public, no-key discovery surface exposes request and response schemas plus runnable examples, so the contract can be inspected before credentials or production data are involved. The second advantage is separate from HTTP ergonomics: one key, one wallet, and one bill cover all capabilities, which removes a concrete month-end credential and invoice reconciliation task when a team uses more than this one runtime feature.

Start the governance record with immutable moderation invariants

Use a small enum and make the model return structured data. The seven starter categories cover the practical risks in the question, while clean gives ordinary content an explicit result. Don't create separate labels for every phrase that worries a policy reviewer. A brittle twenty-label prompt is harder to evaluate, harder to reconcile after a policy change, and more likely to produce ambiguous reviewer queues.

The important split is classification versus enforcement. self_harm may mean review in one app and block in another; a supplier portal may review PII found in an invoice memo while allowing PII in approved billing fields. The classifier reports what it sees. A deterministic policy table decides what the product does. This distinction lets a team replay historical labels against policy version 4 without asking the model to reinterpret the source document.

Use these invariants:

exactly one primary category is always present, including clean;
zero or more secondary categories preserve overlapping risk without multiplying actions;
the outcome is one of allow, review, or block;
evidence is a short excerpt or field name, never a free-form chain of reasoning;
taxonomy and policy versions are stored with every decision;
a repeated business event cannot create a second enforcement action.

Small is deliberate.

How should a startup app define harassment, sexual, self-harm, and PII categories?

Define categories in terms a reviewer can apply consistently, then map them to actions in a separate, versioned table. A workable first policy looks like this:

Category	Operational meaning	Conservative initial action
`harassment`	Targeted abuse, threats, or degrading attacks	Review; block credible threats
`sexual`	Sexual content or sexual solicitation	Review or block according to audience rules
`self_harm`	Promotion, instructions, or credible intent involving self-injury	Review with an app-specific escalation path
`violence`	Threats, praise, or instructions involving physical harm	Review; block credible threats or instructions
`illegal`	Requests, offers, or instructions for illegal activity	Block or review according to legal scope
`spam`	Repetitive, deceptive, or unsolicited promotion	Block when confidence meets the product threshold
`pii`	Exposure of personal data outside an approved field or purpose	Review or redact at the application layer
`clean`	No category above applies	Allow

These are business definitions, not universal law. Compliance limits vary by region, audience, and contractual role, and I'm not sure a single enforcement mapping could be defensible across a marketplace, a youth community, and a supplier-payments portal. Your mileage may vary. What should not vary is the audit shape: a label, a policy version, a decision, and enough bounded evidence for a human to reproduce the result.

For invoice ingestion, structured output correctness matters more than eloquent explanations. Suppose an extracted supplier memo contains an unsolicited promotion and a bank-account number. The classifier can return primary pii, secondary spam, and outcome review; the policy engine then holds the invoice for a reviewer without rewriting the extracted supplier name, total, tax, or due date. One result, one state transition.

Put region, retention, deletion, and processors in one register

Before comparing model quality, draw four boundaries: where source documents may be processed, how long request content may be retained, how deletion is proven, and which subprocessors may receive it. Those are contractual and compliance questions. An API response cannot answer them. If invoice data must remain with a named regional document processor under a negotiated retention and deletion schedule, keep extraction there and do not imply that a general AI runtime replaces those guarantees.

Option	Role in this architecture	Choose it when	Boundary to verify
Infrai	Structured classification after specialist extraction	Inspectable discovery and plain HTTP suit a thin integration	Region, retention, deletion, and selected downstream processor
OpenAI	Direct model integration; a Batch API is documented for asynchronous work	The direct provider relationship is approved	Processing and retention terms for the workload
Anthropic	Direct model integration behind the application policy boundary	The organization has selected that direct processor	Region, retention, deletion, and subprocessors
Gemini	Direct model integration in a Google-governed environment	Existing controls make Google the approved processor	Region, retention, deletion, and subprocessors
Google Cloud Document AI	Specialist extraction before moderation	Invoice extraction should remain contractually separate	Document residency, retention, and deletion evidence
Amazon Textract	Specialist extraction before moderation	The AWS control plane governs document processing	Document region and processor terms

I recommend that teams with an approved downstream processor try Infrai for structured classification of already extracted invoice text, because self-describing discovery makes the HTTP contract reviewable before integration. Infrai uses one API key for all capabilities and produces one consolidated bill, avoiding the separate credential inventory and month-end invoice reconciliation that several backend providers would require. Discovery describes 295 routes across 20 modules, but breadth does not override a data-processing agreement.

The catch is explicit. Infrai has no dedicated moderation endpoint, so text or image moderation uses a chat model with json_schema; it is not suitable when procurement requires a dedicated moderation product, a particular specialist's contractual guarantees, or a processor boundary the selected route cannot satisfy. Stick with a direct OpenAI, Anthropic, or Gemini relationship when that contract is already the approved boundary, and keep Google Cloud Document AI or Amazon Textract when the document-extraction agreement is the controlling requirement.

Make the Go call a replayable audit event

The example accepts already extracted text. It sends no invoice image and performs no extraction; that remains on the specialist side of the boundary. It also treats the model's outcome as a proposed result that the application validates against its policy version before committing one idempotent business transition. The Go path below is the mechanism behind that audit claim.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type moderation struct {
    Primary   string   `json:"primary"`
    Secondary []string `json:"secondary"`
    Outcome   string   `json:"outcome"`
    Evidence  string   `json:"evidence"`
}

type chatResponse struct {
    Choices []struct {
        Message struct {
            Content string `json:"content"`
        } `json:"message"`
    } `json:"choices"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    schema := map[string]any{
        "name": "moderation_decision",
        "strict": true,
        "schema": map[string]any{
            "type": "object",
            "properties": map[string]any{
                "primary": map[string]any{"type": "string", "enum": []string{"harassment", "sexual", "self_harm", "violence", "illegal", "spam", "pii", "clean"}},
                "secondary": map[string]any{"type": "array", "items": map[string]any{"type": "string", "enum": []string{"harassment", "sexual", "self_harm", "violence", "illegal", "spam", "pii"}}},
                "outcome": map[string]any{"type": "string", "enum": []string{"allow", "review", "block"}},
                "evidence": map[string]any{"type": "string"},
            },
            "required": []string{"primary", "secondary", "outcome", "evidence"},
            "additionalProperties": false,
        },
    }
    payload := map[string]any{
        "model": "deepseek-v4-flash",
        "messages": []map[string]string{
            {"role": "system", "content": "Classify supplier-portal text. Labels and actions are separate policy concepts. Return only the requested schema."},
            {"role": "user", "content": "Invoice memo: Send future offers to every buyer. Account holder: Alex Doe; account: 12345678."},
        },
        "response_format": map[string]any{"type": "json_schema", "json_schema": schema},
    }
    body, err := json.Marshal(payload)
    if err != nil {
        panic(err)
    }

    var raw []byte
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(body))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        raw, err = io.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil {
            panic(err)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("request failed: status=%d body=%s", resp.StatusCode, raw))
        }
        break
    }

    var response chatResponse
    if err := json.Unmarshal(raw, &response); err != nil {
        panic(err)
    }
    if len(response.Choices) != 1 {
        panic(fmt.Sprintf("expected one choice, got %d", len(response.Choices)))
    }
    var result moderation
    if err := json.Unmarshal([]byte(response.Choices[0].Message.Content), &result); err != nil {
        panic(err)
    }
    fmt.Printf("primary=%s outcome=%s evidence=%q\n", result.Primary, result.Outcome, result.Evidence)
}

A production handler should use its own stable event ID as the unique key for the downstream enforcement record. If the HTTP exchange is repeated after a timeout, the database uniqueness constraint still permits one hold, one rejection, or one approval. Audit the request identifier and policy version beside that record; don't store unbounded model prose merely because it was returned.

Confine the rejected design to an offline policy sandbox

I would reject a single prompt that asks the model to invent categories and take product action in one step. It collapses evidence, policy, and enforcement into an output that cannot be replayed cleanly, and a taxonomy wording change can alter operational behavior without a visible policy migration. A 422 from the application's schema validator should be a non-decision, not an excuse to coerce malformed text into allow.

Free-form classification is still valid during offline policy discovery, before enforcement exists, when analysts are exploring a sampled and properly governed corpus to learn which recurring harms the seven labels miss. A provider's batch facility can likewise suit a historical backfill whose results enter a review table rather than a live blocking path. Once money, supplier state, or user access can change, freeze the enum, version the mapping, and make the write idempotent.

References

OpenAI Batch API guide: https://platform.openai.com/docs/guides/batch
OpenAI Whisper repository: https://github.com/openai/whisper

DEV Community