AI Image Prompt Moderation in Go: Chat JSON Schema Without a Moderation Endpoint

#go #ai #moderation

Short answer: moderate each text prompt with a structured chat classification before image generation, then enforce the returned labels with deterministic application policy; don't let the model make the final allow-or-block decision.

For an edtech catalog, the least complex workable design is a two-stage gate. The first stage turns a messy course description into typed evidence. The second applies rules owned by the product team. This keeps quality and latency visible instead of hiding both inside one clever prompt.

The operational invariant is blunt: no image job enters the generation queue until moderation has produced a valid decision tied to the exact input. Retries must reuse that decision. Otherwise a timeout can become two generations, and a policy edit can make the same description pass once and fail once.

How can a team moderate text prompts for AI image generation without a dedicated endpoint?

Use an ordinary chat completion that can return a constrained JSON object. Ask it to classify policy-relevant categories, supply confidence, and quote a short fragment of the input as evidence. Then validate the object locally and map it to allow, review, or block with normal code.

That split matters. A classifier can say that a prompt contains a real-person likeness request or ambiguous sexual content; it shouldn't quietly decide what an education catalog permits. Policy changes more often than the wire contract, and reviewers need to see why an item was held. Keep labels descriptive and the policy explicit.

The schema should be small enough to inspect during an incident. Avoid an open-ended explanation field and avoid asking for rewritten prompts in the same call. Rewriting couples safety to content transformation, makes evaluation harder, and adds another place where the original meaning can drift.

Here is the response contract used by the example below:

{
  "decision": "allow",
  "categories": [],
  "confidence": 0.97,
  "evidence": "photosynthesis for grade 6"
}

Treat this as classifier output, not authorization.

Who governs the moderation policy and its data?

The application should reject malformed JSON, unknown fields, invalid enum values, confidence outside zero to one, and evidence that isn't found in the submitted text. That last check does not prove the classification is right. It does stop the response from citing text that the policy engine never received.

The table is intentionally a product decision, not a universal safety standard:

Classifier result	Catalog action	Reason
No flagged category	Allow generation	The prompt passed the current policy
Ambiguous category or low confidence	Send to review	Uncertainty needs a human decision
Category prohibited by catalog policy	Block generation	Code, rather than model prose, enforces policy
Invalid or missing response	Fail closed and retry moderation	Generation has no valid authorization

Fail closed here means the catalog item remains pending. It does not mean retry forever. Put a deadline and an attempt budget around the moderation task, expose exhaustion to an operator, and leave image generation untouched until the decision is resolved.

I've been paged by duplicate deliveries and missed jobs. The common lesson is that “call service, then enqueue” is not an atomic workflow — a process can stop after the remote call and before the local state update. A catalog pipeline needs a stable key derived from the normalized description and policy version, plus a durable record of the moderation result. When a worker sees the key again, it should return the recorded outcome rather than classify or generate again.

Small detail, big outage.

Make retries preserve the original decision

The sample uses a generic chat endpoint supplied through configuration. It makes no assumption about a proprietary moderation route, and it keeps the transport ordinary HTTP so the same contract can sit behind different gateways. The schema is sent as data, the response body is decoded strictly, and policy remains a local function.

package moderation

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "strings"
    "time"
)

type Result struct {
    Decision   string   `json:"decision"`
    Categories []string `json:"categories"`
    Confidence float64  `json:"confidence"`
    Evidence   string   `json:"evidence"`
}

type chatResponse struct {
    Choices []struct {
        Message struct {
            Content string `json:"content"`
        } `json:"message"`
    } `json:"choices"`
}

type Client struct {
    Endpoint   string
    APIKey     string
    Model      string
    HTTPClient *http.Client
}

func JobKey(prompt, policyVersion string) string {
    normalized := strings.Join(strings.Fields(prompt), " ")
    sum := sha256.Sum256([]byte(policyVersion + "\n" + normalized))
    return hex.EncodeToString(sum[:])
}

func (c Client) Classify(ctx context.Context, prompt string) (Result, error) {
    if strings.TrimSpace(prompt) == "" {
        return Result{}, errors.New("empty prompt")
    }

    schema := map[string]any{
        "name":   "prompt_moderation",
        "strict": true,
        "schema": map[string]any{
            "type":                 "object",
            "additionalProperties": false,
            "required":             []string{"decision", "categories", "confidence", "evidence"},
            "properties": map[string]any{
                "decision": map[string]any{"type": "string", "enum": []string{"allow", "review", "block"}},
                "categories": map[string]any{
                    "type":  "array",
                    "items": map[string]any{"type": "string"},
                },
                "confidence": map[string]any{"type": "number", "minimum": 0, "maximum": 1},
                "evidence":   map[string]any{"type": "string"},
            },
        },
    }

    payload := map[string]any{
        "model": c.Model,
        "messages": []map[string]string{
            {"role": "system", "content": "Classify the supplied image prompt. Return evidence copied from the prompt. Do not rewrite it."},
            {"role": "user", "content": prompt},
        },
        "response_format": map[string]any{"type": "json_schema", "json_schema": schema},
    }

    body, err := json.Marshal(payload)
    if err != nil {
        return Result{}, fmt.Errorf("encode request: %w", err)
    }

    callCtx, cancel := context.WithTimeout(ctx, 4*time.Second)
    defer cancel()
    req, err := http.NewRequestWithContext(callCtx, http.MethodPost, c.Endpoint, bytes.NewReader(body))
    if err != nil {
        return Result{}, fmt.Errorf("build request: %w", err)
    }
    req.Header.Set("Authorization", "Bearer "+c.APIKey)
    req.Header.Set("Content-Type", "application/json")

    resp, err := c.HTTPClient.Do(req)
    if err != nil {
        return Result{}, fmt.Errorf("classify prompt: %w", err)
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        io.Copy(io.Discard, resp.Body)
        return Result{}, fmt.Errorf("classification status: %d", resp.StatusCode)
    }

    var wire chatResponse
    if err := json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&wire); err != nil {
        return Result{}, fmt.Errorf("decode response: %w", err)
    }
    if len(wire.Choices) != 1 {
        return Result{}, fmt.Errorf("expected one choice, got %d", len(wire.Choices))
    }

    decoder := json.NewDecoder(strings.NewReader(wire.Choices[0].Message.Content))
    decoder.DisallowUnknownFields()
    var result Result
    if err := decoder.Decode(&result); err != nil {
        return Result{}, fmt.Errorf("decode moderation result: %w", err)
    }
    if err := validate(prompt, result); err != nil {
        return Result{}, err
    }
    return result, nil
}

func validate(prompt string, result Result) error {
    switch result.Decision {
    case "allow", "review", "block":
    default:
        return errors.New("invalid decision")
    }
    if result.Confidence < 0 || result.Confidence > 1 {
        return errors.New("confidence outside zero to one")
    }
    if result.Evidence != "" && !strings.Contains(prompt, result.Evidence) {
        return errors.New("evidence is not present in prompt")
    }
    return nil
}

The endpoint, model identifier, category vocabulary, and timeout belong in deployment configuration. I'm not sure a four-second deadline will fit your provider and region; only latency histograms from your path can settle that. What should not vary per request is the policy version included in JobKey, because that version explains why two otherwise identical descriptions received different outcomes after a deliberate rule change.

The example checks the outer chat envelope and the inner structured value separately. In production, also cap the input length before building the request, redact secrets from logs, and store a digest when retaining raw catalog text would violate your data policy. Don't log full rejected prompts by default. The material most useful for debugging may also be the material you least want copied across observability systems.

Measure quality and latency as separate failure budgets

A parser test proves syntax. It says nothing about moderation quality. Build an evaluation set from catalog descriptions your team is permitted to retain, label it under a written policy, and include ordinary prompts, prohibited prompts, ambiguous prompts, misspellings, mixed languages, and attempts to instruct the classifier. Keep the test set out of the classifier prompt.

Track false allows and false blocks separately. A false allow is a safety failure; a false block delays a legitimate catalog update. The acceptable balance depends on the product policy, which is why a single “accuracy” number is a weak release gate. Review disagreements should update the written labeling guide before they update production thresholds.

Latency needs its own budget: moderation queue wait, provider time, validation time, review wait, and generation time. The median can look healthy while catalog publishing stalls in the tail. Report the age of the oldest pending item and the number of items awaiting human review alongside request percentiles. Those two gauges answer the on-call question faster than an average ever will.

Roll out policy or model changes against recorded, authorized test inputs first, then shadow them on live traffic without changing the production decision. Compare category drift and review volume. Promote one version at a time, retain the prior configuration for rollback, and attach the version to every decision. No mystery state.

Cases that need another control

This chat-plus-schema pattern is not suitable when policy requires a certified or contractually specified moderation control, when images themselves must be assessed, or when the provider cannot guarantee structured output. Use the required control in the first case, add post-generation image moderation in the second, and place a strictly validated adapter or human review queue in the third. Text moderation cannot see harmful details introduced by an image model, so high-risk workflows need controls on both sides of generation.

The catch is latency. Synchronous classification adds a network call before generation. For an interactive classroom tool, a locally hosted classifier may fit the latency budget better, provided the team can evaluate, patch, and operate it. For a back-office catalog, asynchronous moderation with a visible pending state is often easier to run because correctness wins over immediate response. Your mileage may vary; measure the full queue-to-publish path, not a single request on a laptop.

Do not use this design to pretend that JSON Schema creates factual certainty. It constrains shape. Your evaluation set, review process, policy ownership, and deployment records create the evidence needed to trust a release.