Failure Budgets for Image Generation APIs Using Chat-Model JSON Schema Safety

#imagegeneration #promptsafety #sre

Short answer: choose an image generation API by its delivery contract and operating envelope, then place a fail-closed prompt safety gate in front of it that asks a chat model for a narrow JSON decision and validates that decision locally; a native moderation endpoint is useful, but it is not a substitute for an end-to-end safety and reliability design.

The decision is not really "which model makes the nicest sample?" It is whether the whole path can reject unsafe input predictably, absorb the offered load, avoid duplicate rendering, and prove that an accepted request became a usable artifact. I would make the buying decision from those four properties before comparing aesthetic output on a representative evaluation set.

One warning up front: a prompt classifier cannot determine everything that may appear in generated pixels. It reduces one risk at one boundary. It doesn't eliminate the need for output review where the harm model calls for it.

The incident lesson is queue shape, not classifier cleverness

Consider a bounded production exercise: 30 interactive requests arrive each second, the safety gate sustains 40 decisions per second, and the renderer sustains 12 jobs per second. I would expect the first dashboard to look comforting. Classification latency stays flat, decisions keep flowing, and the front door reports that work was accepted. Meanwhile, the render queue grows by roughly 18 jobs every second until a timeout, deploy, or retry wave turns ordinary overload into duplicated expensive work. Those figures are an illustrative capacity model, not a benchmark for any service, but the invariant is real arithmetic: a fast gate cannot raise the capacity of the slowest required stage.

This is where image API selection goes wrong in production reviews. Teams compare generation quality and single-request latency, then treat moderation as a small synchronous call to bolt on later. The safety decision, render admission, generation work, and artifact verification are different failure domains. They need separate concurrency limits, latency distributions, and outcome counters. If they share one undifferentiated "request success" metric, an operator cannot tell whether policy evaluation is slow, render capacity is exhausted, or delivery failed after generation.

I use an end-to-end SLI with a deliberately strict numerator: accepted jobs that expose a valid artifact within the promised window. The denominator is all jobs admitted after a valid safety decision. Policy denials belong in a separate decision metric, because counting an intentional denial as service failure makes the availability number meaningless; malformed or timed-out safety responses do count against the gate's own SLO, because the system could not make a defensible decision.

Short queues win.

Walk the overload forward for one minute and the operational consequence becomes easier to see. At the illustrative rates above, 1,080 more jobs enter the render queue than leave it; if a caller has a 30-second patience budget, many of those jobs are already doomed before a worker starts them, yet they still occupy memory, retain request context, and may consume render capacity after the caller has gone away. A naive client then retries because it never received a useful result. The retry presents a new request ID, the service cannot connect it to the original work, and both copies compete with fresh traffic. Meanwhile the policy gate remains healthy — its p95 may even improve after callers time out — so a team watching only gate latency and HTTP status can misdiagnose the event as a client problem. The control I want is earlier and less dramatic: a bounded admission queue rejects or defers work once its age predicts an SLO miss, the original job ID survives every permitted retry, cancellation propagates to work that can still be stopped, and a reconciliation worker resolves ambiguous jobs without opening the front-door throttle. That's the pager-saving distinction between accepting bytes and accepting responsibility for completion.

Capacity planning follows the minimum sustainable throughput, not the sum or the fastest component. Admission should stop or shed load before the render queue violates the latency budget. A retry budget must be smaller than the spare capacity budget, otherwise retries consume the capacity required for recovery. RFC 9110 supplies the method semantics behind retry decisions, including the distinction between idempotent and non-idempotent methods, but an application still needs its own stable job identity and idempotency contract. A client cannot infer that replaying a generation request is harmless merely because the first response was lost.

The preventative move is mundane: reserve capacity, persist a job identity before rendering, bound every stage with a deadline, and make overload visible as its own outcome. This advice does not apply unchanged to an offline batch that can wait hours and is allowed to drain at a fixed rate. There, a durable queue and explicit completion deadline matter more than interactive shedding. The invariant still holds; only the latency objective changes.

What should a chat model return before an image generation API call?

Ask for a small policy decision, not a rewritten essay. The useful fields are a boolean decision, one or more stable reason codes, the exact policy version, and a normalized prompt only if normalization is part of the documented policy. Express that contract as JSON Schema at the model boundary, then validate the decoded object again in the application. The model response is untrusted input even when the model was asked to produce structured output.

The local validator should reject missing required fields, unknown fields, an unexpected policy version, invalid reason codes, trailing JSON values, and an allowed decision with an empty rendering prompt. Fail closed on timeouts and parse failures. Don't silently send the original prompt to the renderer after the gate fails, because that turns the safety layer into an optional latency tax precisely when it is least healthy.

The schema provides syntax and a limited set of invariants; it does not define the policy's meaning. Policy text, category definitions, model configuration, schema version, and evaluation corpus must move through change control together. Before a rollout, replay a fixed, versioned corpus and inspect false accepts and false rejects by category. A single blended accuracy value can hide the category that carries the actual harm. I'm not sure what threshold is right for an arbitrary application, because that depends on its content policy, users, review process, and consequence of each error; the release record should state the chosen thresholds and the evidence used to set them.

Store enough evidence to reproduce the decision without turning logs into a prompt archive. A job ID, policy version, model identifier, decision, reason codes, timing, and a keyed prompt digest are a workable operational record. Retaining raw prompts is a separate data-governance choice, not a logging default. Access, deletion, and retention requirements should settle that choice before launch.

There is a hard limit. Input-only moderation misses risks introduced or revealed in the output, and it can miss context encoded outside the submitted text. Add output analysis or human approval when the threat model requires examination of the pixels. If review is mandatory, "generated" and "released" must be separate job states with separate timestamps.

Put admission and completion in the code path

The following Go sketch shows the boundary I want during a design review. Network adapters are intentionally abstract because API routes and payloads vary. The orchestration accepts only a validated typed decision, uses one stable idempotency key, and refuses to call a job complete until it has non-empty image bytes of an allowed media type. Strict JSON decoding belongs inside the classifier adapter, before it returns Decision.

package imagejob

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "fmt"
    "strings"
)

type Decision struct {
    Allow         bool
    ReasonCodes   []string
    RenderPrompt  string
    PolicyVersion string
}

type Artifact struct {
    MediaType string
    Data      []byte
}

type SafetyGate interface {
    Evaluate(ctx context.Context, prompt string) (Decision, error)
}

type Renderer interface {
    Generate(ctx context.Context, prompt, idempotencyKey string) (Artifact, error)
}

type Permit interface {
    Acquire(ctx context.Context) (release func(), err error)
}

type Result struct {
    JobID  string
    Digest string
}

func Run(
    ctx context.Context,
    jobID string,
    prompt string,
    requiredPolicy string,
    gate SafetyGate,
    renderer Renderer,
    capacity Permit,
) (Result, error) {
    decision, err := gate.Evaluate(ctx, prompt)
    if err != nil {
        return Result{}, fmt.Errorf("evaluate prompt: %w", err)
    }
    if decision.PolicyVersion != requiredPolicy {
        return Result{}, errors.New("unexpected policy version")
    }
    if !decision.Allow {
        return Result{}, fmt.Errorf("prompt denied: %s", strings.Join(decision.ReasonCodes, ","))
    }
    if strings.TrimSpace(decision.RenderPrompt) == "" {
        return Result{}, errors.New("allowed decision has no render prompt")
    }

    release, err := capacity.Acquire(ctx)
    if err != nil {
        return Result{}, fmt.Errorf("acquire render capacity: %w", err)
    }
    defer release()

    artifact, err := renderer.Generate(ctx, decision.RenderPrompt, jobID)
    if err != nil {
        return Result{}, fmt.Errorf("generate artifact: %w", err)
    }
    if len(artifact.Data) == 0 || !strings.HasPrefix(artifact.MediaType, "image/") {
        return Result{}, errors.New("invalid image artifact")
    }

    sum := sha256.Sum256(artifact.Data)
    return Result{JobID: jobID, Digest: hex.EncodeToString(sum[:])}, nil
}

This function does not show persistence, and production code needs it. Write the job record and idempotency identity before the side effect, then advance the record through explicit states. If the renderer returns final bytes synchronously, verification can occur in the same worker. If generation is asynchronous, the callback or polling worker must authenticate the job identity, validate the artifact, and perform a conditional state transition so two completion messages cannot release the same job twice.

The test matrix should be boring and adversarial: invalid structured output never renders; a denied decision never renders; a policy mismatch never renders; admission cancellation reaches the caller; repeated delivery notifications cannot complete twice; empty bytes cannot complete; an unexpected media type cannot complete; and a retry reuses the original identity. Add load tests that increase offered traffic past sustainable render capacity and verify that queue age, not CPU utilization alone, triggers protection.

Observe transitions rather than one request span. I want decision latency and outcome by policy version, admission wait, queue age, render duration, artifact validation failures, end-to-end completion latency, and retry count. Avoid prompt text in metric labels. Alert on user-visible budget burn and sustained queue age; a page for every individual denied prompt would confuse policy working as designed with system failure.

Compare the operating contract before the sample gallery

Image quality still matters, but evaluate it with prompts and scoring rules that represent the intended workload. The operational screen comes first because a visually strong renderer can still be the wrong dependency if its concurrency behavior, completion evidence, or request identity cannot support the SLO. Require a written answer for rate limits, timeout behavior, synchronous versus asynchronous delivery, idempotency, artifact format, retention, data handling, and change notification. An unanswered item is risk to price into the decision, not permission to invent a favorable contract.

Use failure injection during evaluation. Cancel calls at different points, lose a response after work may have started, repeat a callback, submit malformed safety output to the application validator, and saturate the admission limit. The goal is not to make a provider look bad; it is to learn whether your client can reach one unambiguous terminal state under ordinary distributed-systems failures. Your mileage may vary across workloads, especially when image size and generation settings change service time, so measure the actual request mix rather than extrapolating from one prompt.

I would record the selection as a compact scorecard:

Decision axis	Evidence required	Rejection signal
Safety control	Versioned policy decision, strict local validation, evaluation results by category	Rendering can proceed after an invalid or missing decision
Delivery contract	Stable job identity and verifiable terminal artifact	Transport acceptance is the only completion evidence
Capacity behavior	Load test through saturation, documented limits, observable queue age	Retries or hidden queues grow without a bound
Data handling	Written retention, access, deletion, and training terms	The team cannot state where prompts and images persist
Change control	Version visibility and a regression process	Behavior can change without a detectable release boundary
Portability	Adapter boundary and exportable artifacts and metadata	Policy state and job state exist only inside one opaque workflow

Do not collapse those rows into a single weighted number too early. A hard failure on data handling or safety cannot be compensated by better image preference scores. For the remaining candidates, run a canary with explicit abort criteria and consume only a small, agreed fraction of the error budget. The rollout plan is part of the selection evidence.

Should the team buy, build, or split the runtime?

My default is a split architecture: own the policy contract and orchestration, while treating classification and rendering engines as replaceable implementations behind narrow adapters. This keeps the safety state machine, audit record, and SLO semantics under the platform team's control. It also adds a network hop, increases the combined failure surface, and leaves the team responsible for policy evaluation. The catch is real.

Path	Suitable when	Operational burden	Walk away when
Buy an integrated runtime	Its policy, evidence, delivery contract, and limits match the workload	Vendor evaluation, contract monitoring, exit testing	Required policy categories or audit evidence cannot be represented
Split managed services	The team needs an independent policy lifecycle or renderer portability	Two dependencies, deadline allocation, adapter and corpus ownership	The extra hop consumes the interactive latency budget
Self-host both stages	Data boundaries or model control justify accelerator operations	Capacity headroom, patching, rollout safety, model evaluation, on-call	The organization has no funded owner for the queue, corpus, and pager

An integrated service with an adequate safety interface is the simpler choice when its policy maps cleanly to the application and its evidence satisfies review requirements. Stick with that path when independence would add machinery without changing a meaningful control. A split gate is not suitable when a few milliseconds determine the product experience and the added call cannot fit inside the deadline; test an integrated design or move appropriate review out of the synchronous path. Self-hosting is not a moral victory either. Choose it only when control is worth the capacity reserve and operational ownership.

The final capacity sheet should name peak admitted rate, service-time distribution, concurrency ceiling, queue limit, retry budget, and required headroom for every synchronous stage. It should also name the human owner of policy drift and the person carrying the renderer pager. If those cells are blank, the architecture is not ready to buy or build.

The best choice is the one whose unsafe, overloaded, duplicated, and incomplete states are explicit and testable. Model preference matters after that bar is cleared. No moderation workaround can compensate for a runtime whose failure states the team cannot observe or control.