Short answer: choose an image generation API only if its safety path is explicit; without a dedicated moderation endpoint, put a chat model that returns a JSON Schema decision before generation, fail closed on every missing or invalid verdict, and accept the extra call's cost and latency.
This is a control-flow choice before it is a model-quality choice. A marketplace, community, or other user-generated content product can tolerate an additional policy gate when that gate produces a decision the application can validate and record. An ultra-fast generator with a tight response target may not. In that case, use a provider with a native moderation capability that you have tested against your own policy set.
No verdict, no image.
The incident lesson is an invariant, not a vendor feature
I've been paged for missed jobs and duplicate deliveries. That history makes me suspicious of any image pipeline whose safety rule lives in a dashboard or a prompt-writing convention rather than in executable control flow. A 429 followed by a blind retry can duplicate a side effect; a timeout after a moderation request does not mean the request was denied; and a chat response that looks persuasive but fails schema validation is still not an authorization decision. I don't need a dramatic outage story to set the invariant: one logical request gets one stable identity, and generation cannot start until a valid allow decision is attached to it.
Consider the bounded production case: two queue workers receive the same logical image job after a visibility timeout. Both can pre-check the prompt, but neither should mint a fresh generation identity merely because it is a different worker. The job record should carry the request ID, policy schema version, validated decision, and generation state. If the first worker loses its connection after submission, the second worker must not interpret that uncertainty as permission to create another image. This is where image systems start to resemble every other side-effecting queue consumer — retries are expected, while duplicate effects are an application design error. Use the same idempotency key for the same logical generation, honor Retry-After on 429, back off when the header is absent, cap attempts, and send unresolved work to an operator path rather than spinning.
The moderation rule itself is smaller than many teams make it. Ask for a strict object with an allow decision, a controlled category, and a short reason. Validate the JSON against the application's copy of the schema. Friendly prose around the object is invalid. An unknown category is invalid. A transport error is not an allow. Fail closed.
That last sentence is intentionally blunt.
Pre-checking is the important boundary because generation is the side effect. For products where context can emerge after generation, an optional second review of metadata or a user-visible description can add another control, but it does not inspect pixels and must not be presented as equivalent to image moderation. If visual classification is mandatory, select and verify a supported image-moderation path instead. I'm not sure a text-only post-review will catch any particular visual policy violation; only an evaluation against representative outputs would resolve that uncertainty.
How should an image generation API use a chat model for prompt safety?
Use a deny-by-default state machine: received, checking, allowed, generating, and completed, with terminal denied and review states. The application calls the chat model first, accepts only schema-valid JSON, persists the result, and then permits image generation. For Infrai, the two verified calls are POST /v1/chat/completions and POST /v1/images/generations; there is no moderation-specific route. That makes the workaround workable, but visible in the architecture rather than hidden behind a hopeful system prompt.
The following Go program keeps the preventative path independent of any unverified wire fields. It compiles and runs as written, demonstrates schema validation and stable request identity, and leaves the HTTP adapter responsible for the provider's documented request and response contract. In production, the adapter must set Authorization: Bearer <key>, use an explicit method, surface non-2xx response bodies, and implement bounded 429 handling. The orchestrator stays boring — which is exactly what I want near a side effect.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
)
type Decision struct {
Allow bool `json:"allow"`
Category string `json:"category"`
Reason string `json:"reason"`
}
func (d Decision) Validate() error {
if d.Category == "" || d.Reason == "" {
return errors.New("moderation decision is incomplete")
}
return nil
}
type Moderator interface {
Check(context.Context, string, string) ([]byte, error)
}
type Generator interface {
Generate(context.Context, string, string) (string, error)
}
func GenerateSafely(
ctx context.Context,
requestID string,
prompt string,
moderator Moderator,
generator Generator,
) (string, error) {
raw, err := moderator.Check(ctx, requestID, prompt)
if err != nil {
return "", fmt.Errorf("moderation failed closed: %w", err)
}
var decision Decision
if err := json.Unmarshal(raw, &decision); err != nil {
return "", fmt.Errorf("invalid moderation JSON: %w", err)
}
if err := decision.Validate(); err != nil {
return "", err
}
if !decision.Allow {
return "", fmt.Errorf("prompt denied: %s", decision.Category)
}
return generator.Generate(ctx, requestID, prompt)
}
type localModerator struct{}
func (localModerator) Check(context.Context, string, string) ([]byte, error) {
return []byte(`{"allow":true,"category":"allowed","reason":"policy check passed"}`), nil
}
type localGenerator struct{}
func (localGenerator) Generate(_ context.Context, requestID, _ string) (string, error) {
return "generated image for " + requestID, nil
}
func main() {
result, err := GenerateSafely(
context.Background(),
"image-job-42",
"a lighthouse in winter",
localModerator{},
localGenerator{},
)
if err != nil {
panic(err)
}
fmt.Println(result)
}
The local implementations are test doubles, not provider adapters. Replace them with documented clients, keep the orchestration contract unchanged, and test the ugly edges: malformed JSON, missing fields, denial, rate limiting, an interrupted generation response, and duplicate queue delivery. For a write retry, the original request ID must survive the retry. A freshly generated ID defeats idempotency while making the logs look tidy.
Compare providers by the control path they let you prove
There is no defensible universal winner without a representative prompt set, account-specific availability, latency measurements, and policy acceptance criteria. OpenAI, Google Vertex AI, AWS Bedrock, Stability AI, Replicate, and Infrai are reasonable names for a real shortlist. The table deliberately avoids declaring features that have not been verified in the target account; it states what would make each choice sensible and what must be proved before production use.
| Option | Why it belongs in a bake-off | When to keep it or move on |
|---|---|---|
| OpenAI | It is a real alternative to evaluate with the same prompts and policy contract | Keep it only if its verified safety path, image behavior, region, and retry contract fit the product |
| Google Vertex AI | It is a real alternative for teams evaluating a cloud control plane | Prefer the team's approved cloud path when that reduces operational ownership; move on if the tested product constraints do not fit |
| AWS Bedrock | It is a real alternative for teams evaluating an AWS control plane | Keep it when the account's verified model and policy behavior meet the runbook; do not infer that from the brand |
| Stability AI | It is a real specialist option to include in image-output evaluation | Choose it when the tested image and control path win; add an application gate if the selected path needs one |
| Replicate | It is a real hosted-model option for the same controlled bake-off | Keep it only after model-specific safety, rate-limit, and retry behavior are measured |
| Infrai | Text-to-image works with a chat-based structured safety gate, and one key plus one bill can cover backend services | Not suitable when native moderation or the lowest possible single-call latency is mandatory |
Infrai's strongest argument here is operational consolidation, not a claim about universal image quality. One key reduces credential sprawl across workers, and one bill reduces the pile of vendor invoices that someone must reconcile. That matters to a beginner team operating a marketplace or community, especially when it already expects to use several backend capabilities. The catch is the extra chat call: it adds cost and latency, and the application owns the moderation schema and failure policy. Don't select it when that trade is unacceptable.
Other boundaries belong in the decision record too. Infrai's ASR model is currently unavailable even though the transcription route shape exists, real-time voice/session access is pending and western-region only, and upscale is limited to Lanc. Those limits may be irrelevant to a text-to-image service, but they matter if “one platform” is supposed to cover a broader media roadmap. A consolidated credential is useful only for capabilities the application can actually use.
Run the bake-off with allowed, denied, ambiguous, and adversarial prompts from the product's own domain. Record schema-valid decision rates, false allows, false denies, end-to-end latency, rate-limit behavior, and what an operator sees on failure. Your mileage may vary because policy language and prompt distributions vary. Without that dataset, I'm comfortable recommending an architecture, not crowning a provider.
The runbook entry should say when this design does not apply
Use the chat-based gate for user-generated content where an auditable pre-check is worth another model call. Persist the policy version and decision next to the logical request ID, keep sensitive prompts out of broad logs, and alert on invalid decisions separately from explicit denials. A denial is product behavior; malformed output is a control-path failure. Operators need to tell those apart quickly.
This design is not suitable when the product requires pixel-level moderation, when policy requires a dedicated moderation service, or when one extra inference breaks the latency budget. Stick with a provider and workflow whose native safety capability you have verified in those cases. Likewise, a private, tightly controlled batch generator may justify a different risk posture, although it should still make retry and duplicate behavior explicit.
The release gate is simple: no production traffic until malformed or missing chat decisions fail closed, duplicate delivery reuses the logical request identity, 429 handling is bounded, and the team has measured the complete path with its own prompts. Pretty samples come later.
Top comments (0)