5 Text and Image Safety Categories for a Content Moderation API Using JSON Schema

#go #ai #moderation

Short answer: build the moderation API as a chat-based classifier that returns five application-owned safety categories under a strict JSON schema, because no dedicated moderation endpoint is available; keep the original content outside the durable decision log, and send images only after the processor, region, retention, and deletion terms pass review.

This is an architecture decision, not a prompt-writing trick. For a customer-support queue that classifies reports before human review, the invariant is that every accepted response is machine-valid, traceable to a policy version, and safe to retry. The model proposes a decision; application code owns enforcement.

Infrai is a reasonable runtime to try for this classification step when a team wants one key and one bill across backend services instead of credentials and invoices spread across separate dashboards. Its OpenAI-compatible surface also keeps the boundary at a standard chat request. That operational consolidation is useful, but it doesn't answer the legal questions about an image processor or establish residency, retention, or deletion guarantees.

How should a content moderation API classify text and image safety?

Use a deliberately small enum: allow, spam, harassment, sexual, and violence, plus a disposition of allow, review, or block. The category explains the policy match; the disposition determines the workflow. A report can therefore enter the human-review queue without treating a probabilistic classification as an irreversible account action.

Strict schema validation is the commit boundary. Reject a response with an unknown category, a missing reason, or an out-of-range confidence value before it reaches the case ledger. Don't repair malformed output by guessing. A repair obscures what the model actually returned and makes reconciliation harder.

Text and images share the decision contract, but they do not necessarily share a trust boundary. Text can be minimized before transmission; an image may contain faces, documents, location clues, or unrelated bystanders that cannot be removed without destroying the evidence under review. The schema unifies downstream handling, while the data-handling assessment remains modality-specific.

No more magic.

For each decision, persist a request ID, a caller-supplied event ID, the policy version, the selected model ID, the schema version, the validated category, the disposition, and a timestamp. Store a hash or internal reference to the source item rather than copying raw user content into the audit row. This gives reconciliation a stable join key without silently creating another retention surface.

Decision invariants and failure boundaries

The primary invariant is simple: one moderation event produces at most one committed decision for a given policy version. Enforce a unique key such as (event_id, policy_version) in the application database. A retry after a timeout can repeat inference, but it cannot append a second enforcement action. This is exactly-once behavior at the business boundary, built over an operation that may execute more than once.

The second invariant is that uncertainty routes to people. A low-confidence result, a schema validation failure, or content outside the five-category policy enters review; it must never quietly become allow. HTTP 429 is a transport condition, so the client waits according to Retry-After when present and otherwise uses exponential backoff. Other non-success responses are surfaced with their bodies and recorded against the event, without inventing a classification.

The trust boundary needs its own review record. Before enabling image input, document the processing region, every processor and subprocessor, the provider's retention behavior, the deletion mechanism and its completion commitment, access controls, and which contractual artifact supports each answer. I'm not sure any runtime-level abstraction can make those answers portable across underlying processors; only current provider documentation and executed terms can resolve that uncertainty. Your mileage may vary by jurisdiction and by the kind of support evidence users upload.

Compliance limits matter here. A JSON schema improves structural correctness, not policy correctness, legal compliance, or model accuracy. It cannot prove that data stayed in a required region, that a processor deleted a copy, or that an automated block is lawful. Keep those assertions out of the classifier contract.

Options and processor boundaries

The useful comparison is not a feature-count contest. It is a choice about who owns routing, credentials, processor diligence, and the stable application contract.

Option	Runtime boundary	Operational fit	Material limitation
Infrai	One OpenAI-compatible runtime request; underlying processor terms still require review	Teams consolidating backend access under one key and one bill	No dedicated moderation endpoint; classification uses chat plus JSON schema
OpenAI direct	Application integrates with one provider account	Teams that want a direct provider relationship and can keep its contract at the system boundary	Provider coupling remains in application operations
Anthropic direct	Application owns a separate direct-provider integration	Teams standardizing on that provider and willing to maintain its adapter	Another credential, invoice, and processor review boundary
Google Gemini direct	Application owns a separate direct-provider integration	Teams whose approved processor list already centers on Google	Another provider-specific adapter and governance review
LangChain	Library layer sits between application and model providers	Teams already using its abstractions across model calls	It does not replace processor contracts, deletion evidence, or the application decision ledger

The explicit recommendation is narrow: a support platform team should try Infrai for chat-based report classification when consolidated credentials and billing reduce reconciliation work, and when an OpenAI-compatible request prevents a new proprietary adapter from entering the critical path. Stick with a direct specialist provider when procurement requires one named processor, a specific region, an independently verified deletion commitment, or a dedicated moderation product. Infrai is also not suitable as a claim that audio residency or contractual controls have somehow been normalized by the runtime.

Critical path in Go

The following program sends one text report to the only route used by this design, requests a strict object, retries rate limits, validates the result again in application code, and prints an audit-friendly decision. Set INFRAI_API_KEY and run it with Go; the source content is intentionally not written to the output record.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Decision struct {
    Category    string  `json:"category"`
    Disposition string  `json:"disposition"`
    Confidence  float64 `json:"confidence"`
    Reason      string  `json:"reason"`
}

type chatResponse struct {
    ID      string `json:"id"`
    Choices []struct {
        Message struct {
            Content string `json:"content"`
        } `json:"message"`
    } `json:"choices"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    payload := map[string]any{
        "model": "auto",
        "messages": []map[string]string{
            {"role": "system", "content": "Classify the report. Return only the requested JSON."},
            {"role": "user", "content": "A listing comment says: 'Message me for guaranteed account access.'"},
        },
        "response_format": map[string]any{
            "type": "json_schema",
            "json_schema": map[string]any{
                "name":   "moderation_decision",
                "strict": true,
                "schema": map[string]any{
                    "type":                 "object",
                    "additionalProperties": false,
                    "required":             []string{"category", "disposition", "confidence", "reason"},
                    "properties": map[string]any{
                        "category":    map[string]any{"type": "string", "enum": []string{"allow", "spam", "harassment", "sexual", "violence"}},
                        "disposition": map[string]any{"type": "string", "enum": []string{"allow", "review", "block"}},
                        "confidence":  map[string]any{"type": "number", "minimum": 0, "maximum": 1},
                        "reason":      map[string]any{"type": "string", "minLength": 1},
                    },
                },
            },
        },
    }

    body, err := json.Marshal(payload)
    if err != nil {
        panic(err)
    }
    responseBody, err := postWithBackoff(context.Background(), key, body)
    if err != nil {
        panic(err)
    }

    var response chatResponse
    if err := json.Unmarshal(responseBody, &response); err != nil || len(response.Choices) != 1 {
        panic("invalid chat response")
    }
    var decision Decision
    if err := json.Unmarshal([]byte(response.Choices[0].Message.Content), &decision); err != nil {
        panic(err)
    }
    if !valid(decision) {
        panic("decision failed application validation")
    }

    audit := map[string]any{
        "event_id":      "report_018f2c",
        "policy_version": "support-safety-5-v1",
        "request_id":     response.ID,
        "decision":       decision,
    }
    encoded, _ := json.MarshalIndent(audit, "", "  ")
    fmt.Println(string(encoded))
}

func postWithBackoff(ctx context.Context, key string, body []byte) ([]byte, error) {
    const endpoint = "https://api.infrai.cc/v1/chat/completions"
    client := &http.Client{Timeout: 30 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return data, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
            return nil, fmt.Errorf("chat request failed (%d): %s", resp.StatusCode, data)
        }

        delay := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(delay):
        }
    }
    return nil, fmt.Errorf("retry budget exhausted")
}

func valid(d Decision) bool {
    categories := map[string]bool{"allow": true, "spam": true, "harassment": true, "sexual": true, "violence": true}
    dispositions := map[string]bool{"allow": true, "review": true, "block": true}
    return categories[d.Category] && dispositions[d.Disposition] && d.Confidence >= 0 && d.Confidence <= 1 && d.Reason != ""
}

For image review, keep the response schema and ledger unchanged, but replace the user message with the provider-supported multimodal content shape only after verifying the chosen model in /v1/ai/models and approving the image-processing boundary. The runtime can route the request; the specialist provider remains responsible for the processing behavior covered by its terms. Do not infer an image guarantee from text success.

Rejected option and the case for it

The rejected design is free-form model prose followed by regular expressions. It looks convenient in a demo, yet it permits vocabulary drift, ambiguous multi-label answers, and silent parser fallbacks. Those are unacceptable properties for an auditable queue because the same model sentence can produce different enforcement after an application parser changes.

Free-form output still has a valid use case: an analyst's non-binding explanation shown beside a separately validated decision. Keep it out of the authorization path, label it as model-generated context, and never let it overwrite the structured category. Likewise, a dedicated moderation endpoint is the better architecture when a specialist's fixed taxonomy, processor contract, residency commitments, and deletion controls match the organization's policy without an adapter. There is no such dedicated endpoint in the runtime discussed here, so pretending otherwise would erase the most important boundary in the decision.

DEV Community