Short answer: treat every text prompt as a tenant-scoped authorization before it reaches image generation, using a small schema-validated decision and an idempotency key. A moderation endpoint is optional; malformed, ambiguous, or unpriced work must not become an image job.
How should teams moderate text prompts for image generation without an endpoint?
In a gaming CRM, a sales call can produce a proposed image brief for a campaign banner. Imagine a worker receives a timeout after the classifier accepted the request, retries with a new id, and sends two images while billing one tenant once. That is a reconciliation defect, not merely a content-safety defect.
The safe path is a pipeline: normalize the text, apply deterministic deny rules, ask a constrained classifier for a decision, and write that decision to a tenant ledger before dispatching image work. The classifier can be an ordinary chat-style JSON request; it does not need a special moderation route.
The contract should be deliberately small: allowed, reason, policy_version, and a bounded confidence score. Unknown fields are rejected. Empty reasons are rejected. A timeout is a recorded denial or a review state, never an implicit allow. This makes the safety boundary auditable and keeps the CRM action separate from the image side effect.
I use exactly-once thinking for the ledger even though distributed systems deliver at-least-once messages. RFC 9110 describes when an HTTP request is idempotent and therefore safe to retry; application-level uniqueness still belongs in a database constraint keyed by tenant and request id. That distinction matters when a worker retries after a network response was lost.
The decision is less about a brand and more about where responsibility sits. Each option has a different failure boundary and audit burden, so I document the invariant before comparing implementations.
| Option | Useful property | Failure boundary | Tenant cost record |
|---|---|---|---|
| Dedicated moderation service | Narrow policy contract and independent scaling | Extra quota, data path, and residency review | Meter each request before image dispatch |
| Schema-constrained chat classification | One HTTP interface can express labels and reasons | Prompt injection or invalid JSON must fail closed | Store token estimate and final usage by tenant |
| Local rules plus human review | Predictable behavior for known terms | Ambiguous language consumes reviewer capacity | Charge rule checks and review work separately |
For this workflow, local rules handle obvious disallowed strings, schema-constrained classification handles context, and a review queue handles uncertainty. The ledger reserves a moderation unit before classification, then settles actual usage. A duplicate key returns the original decision instead of reserving twice. Keep the policy version, tenant id, redacted excerpt, and timestamps in the audit trail; retention limits and data residency are compliance controls, not documentation chores. A rejected prompt still gets a visible ledger event, while no image job is created.
That ordering is the important part.
A fail-closed critical path in Go
The following generic interfaces keep policy tests independent of a provider SDK. The Classifier may send a chat request with a JSON schema, while the Ledger enforces per-tenant accounting and durable outcomes.
package gate
import (
"context"
"encoding/json"
"fmt"
"strings"
)
type Decision struct {
Allowed bool `json:"allowed"`
Reason string `json:"reason"`
PolicyVersion string `json:"policy_version"`
Score int `json:"score"`
}
type Classifier interface {
Classify(context.Context, string, []byte) ([]byte, error)
}
type Ledger interface {
Reserve(context.Context, string, string) (bool, error)
Record(context.Context, string, string, Decision) error
}
func Admit(ctx context.Context, tenant, requestID, prompt string, c Classifier, l Ledger) (Decision, error) {
reserved, err := l.Reserve(ctx, tenant, requestID)
if err != nil {
return Decision{}, err
}
if !reserved {
return Decision{}, fmt.Errorf("duplicate tenant request")
}
clean := strings.TrimSpace(prompt)
if clean == "" {
d := Decision{Allowed: false, Reason: "empty prompt", PolicyVersion: "2026-01"}
return d, l.Record(ctx, tenant, requestID, d)
}
schema := []byte(`{"type":"object","required":["allowed","reason","policy_version","score"],"additionalProperties":false}`)
raw, err := c.Classify(ctx, clean, schema)
if err != nil {
d := Decision{Allowed: false, Reason: "classifier unavailable", PolicyVersion: "2026-01"}
return d, l.Record(ctx, tenant, requestID, d)
}
var d Decision
if err := json.Unmarshal(raw, &d); err != nil || d.Reason == "" || d.PolicyVersion == "" || d.Score < 0 || d.Score > 100 {
d = Decision{Allowed: false, Reason: "invalid classifier result", PolicyVersion: "2026-01"}
}
if err := l.Record(ctx, tenant, requestID, d); err != nil {
return Decision{}, err
}
return d, nil
}
The image worker checks the recorded decision and the same request id immediately before dispatch. That second check protects against a split-brain worker that saw an old cache entry. In a deployment review, I would trace one request through reservation, classifier timeout, retry, durable denial, and worker replay, checking that every transition carries the tenant and request id; this is where teams usually discover that a queue payload dropped the policy version or that a metrics label used an account id from the wrong region. Tests should assert that a duplicate reservation creates no second image job, invalid JSON is denied, and a retry preserves the original decision. Export attempted, denied, reviewed, retried, and admitted counts by tenant so finance can reconcile work without making price the safety argument.
Limits and the valid alternative
Reject free-form classifier prose, because a parser cannot reliably tell a refusal from a label. Reject an allow result without a policy version, tenant id, or request id. Reject retries that mint a new idempotency key. These are contract rules, so property tests can exercise them without calling a model.
The catch is that a remote classifier remains probabilistic, and a keyword list misses euphemisms. This design is not suitable when policy requires deterministic on-premise enforcement or a human decision for every borderline image; use a local classifier and an explicit review workflow then. Stick with a dedicated moderation service when its residency guarantees and audit controls fit better than a shared chat classifier. Your mileage may vary, and I'm not sure one score threshold transfers across game genres; calibrate it against a labeled, tenant-scoped sample and have compliance approve retention.
Top comments (0)