Choosing an Image Generation API: Prompt Safety Without a Moderation Endpoint

#ai #imagegeneration #moderation

Choose an image generation API for measured output quality and tail latency, then own prompt safety as a separate, typed policy gate when the API has no moderation endpoint.

For my text-to-image feature, the simple approach was to let the generation request make every safety decision. It was easy to ship, but it mixed two contracts: my product policy and the image service's acceptance policy. I now put a small gate before generation and require a JSON-schema-shaped verdict from a chat model. That isn't a magic classifier. It's an enforceable boundary I can test, version, observe, and replace without changing the image API.

The evaluation constraint matters: I won't pick a provider from a feature checklist. I run the prompts my users actually submit, score the resulting images with a fixed rubric, and compare p50 and p99 latency, refusal behavior, operational control, and the cost of a completed user task. The winning setup is the one that survives that workload, not the one with the longest model list.

How should I choose an image generation API when there is no prompt moderation endpoint?

Start by separating three questions that are often collapsed into one. Can the image API produce acceptable results for your prompt distribution? Can your application decide whether a prompt fits its own policy before spending a generation call? Can you inspect the returned image when output risk matters? A provider-side filter may contribute to the first or third question, but it doesn't define what is acceptable for your product.

My shortlist test uses a frozen prompt set and a blind review. I include ordinary prompts, ambiguous prompts, clear policy violations, multilingual input, misspellings, and attempts to hide instructions inside long text. For output quality, I score instruction following, unwanted text, composition, and consistency across repeated runs. I don't pretend one aggregate score settles the choice; a product that needs clean diagrams weights different failures from a product that makes loose concept art.

Then I record the whole request path. Time to first accepted result is more useful to me than a provider's isolated generation latency because rejected prompts, retries, and unusable images still consume a user's patience. Cost per accepted result is the matching economic measure. As a solo founder, I care about unit price, but a cheap call that needs three attempts isn't cheap in practice.

No single option wins every column:

Approach	Policy ownership	Portability	Main limitation
Generation call alone	Mostly external	Low	The application learns little before generation
Deterministic local rules	Application	High	Rules miss context and require careful maintenance
Chat model with a typed verdict	Application	Medium to high	Model decisions need evaluation and versioning
Dedicated policy service	Shared with a specialist	Medium	Adds another contract and operational dependency

The catch is clear. A schema-guided chat gate is not suitable when a regulator, customer contract, or internal control requires a particular published taxonomy or independently managed policy service. In that case, use the required control and treat application rules as an extra layer, not a substitute.

Make the safety verdict a narrow contract

I want the gate to answer a boring question: allow, rewrite, or block? Free-form prose is the wrong return type. It makes downstream behavior depend on wording, and wording drifts. A narrow object gives the application a closed set of branches while keeping the policy text separate from the transport code.

The schema should be smaller than your policy document. I use stable category identifiers, a policy version, and an optional replacement prompt. I also reject unknown properties. That last detail catches accidental interface growth — useful when I'm moving fast and don't have another engineer reviewing every policy edit.

type SafetyDecision = "allow" | "rewrite" | "block";

type SafetyVerdict = {
  decision: SafetyDecision;
  categories: Array<"sexual" | "violence" | "real_person" | "other">;
  replacementPrompt: string | null;
  policyVersion: "2026-08-03";
};

const verdictSchema = {
  type: "object",
  additionalProperties: false,
  required: ["decision", "categories", "replacementPrompt", "policyVersion"],
  properties: {
    decision: { type: "string", enum: ["allow", "rewrite", "block"] },
    categories: {
      type: "array",
      items: { type: "string", enum: ["sexual", "violence", "real_person", "other"] },
    },
    replacementPrompt: { type: ["string", "null"] },
    policyVersion: { type: "string", enum: ["2026-08-03"] },
  },
} as const;

I validate the response at runtime even when the model API claims to enforce a schema. Types disappear after compilation; network input doesn't become trustworthy because TypeScript says it has a friendly shape. I also apply invariants after schema validation: rewrite requires a nonempty replacement, while allow and block require null. The original prompt never gets silently replaced on a malformed verdict.

Keep the prompt policy concrete. Define prohibited content, permitted edge cases, and how to handle uncertainty. Don't ask the gate to be “safe.” That word doesn't produce a testable contract. I store the policy text and schema version beside the evaluation set, then promote them together. A model change is a policy-system change, too, even if the TypeScript interface stays identical.

The focused request path I actually ship

The hot path has four stages: normalize enough for stable hashing, obtain a verdict, select the original or rewritten prompt, and call an abstract image generator. I don't lowercase or rewrite the user's prompt during normalization because case and punctuation can carry meaning. The hash is for correlation and exact-result caching, not fuzzy policy matching.

Here is the provider-neutral shape. The two injected functions are deliberate. They keep chat-model and image-model details outside the orchestration layer, so I can benchmark or replace either side without turning the product code into a migration project.

type GateResult = {
  decision: "allow" | "rewrite" | "block";
  categories: string[];
  replacementPrompt: string | null;
  policyVersion: string;
};

type GeneratedImage = {
  bytes: Uint8Array;
  mediaType: "image/png" | "image/jpeg" | "image/webp";
};

type ClassifyPrompt = (prompt: string) => Promise<GateResult>;
type GenerateImage = (input: {
  prompt: string;
  requestKey: string;
}) => Promise<GeneratedImage>;

export async function createCheckedImage(
  userPrompt: string,
  requestKey: string,
  classifyPrompt: ClassifyPrompt,
  generateImage: GenerateImage,
): Promise<GeneratedImage> {
  const prompt = userPrompt.trim();
  if (prompt.length === 0) throw new Error("Prompt must not be empty");

  const verdict = await classifyPrompt(prompt);
  if (verdict.decision === "block") {
    throw new Error(`Prompt rejected by policy ${verdict.policyVersion}`);
  }

  const selectedPrompt = verdict.decision === "rewrite"
    ? verdict.replacementPrompt
    : prompt;

  if (!selectedPrompt) throw new Error("Rewrite verdict requires replacementPrompt");
  return generateImage({ prompt: selectedPrompt, requestKey });
}

Notice what this example doesn't do: it doesn't invent a universal retry loop. RFC 9110 says a client should not automatically retry a non-idempotent request unless it knows the request is idempotent under the relevant semantics or can detect that the original request was never applied. Image generation commonly sits behind a POST-shaped operation, so I require an explicit request key in my own interface and verify a provider's documented behavior before retrying. A locally invented key has no effect unless the receiving system honors it.

Small detail. Big bill.

I log the request key, prompt hash, policy version, decision, model configuration identifier, duration of each stage, and final outcome. I don't put raw user prompts into general logs. Access-controlled samples can support review, but retention and redaction need to be product decisions rather than debugging accidents.

Tail latency changed my architecture

I learned this under real traffic, not in a notebook. My gate looked negligible during warm manual tests, then a launch burst reached 43 requests per minute and p99 end-to-end latency jumped from 12 seconds to 31 seconds. The spike only appeared after scale-out: a cold application instance established upstream connections for the classifier, waited for that serial result, and only then began the slower image request. p50 moved much less, so the dashboard I checked first made the release look healthy. I'm not sure how much of the tail belonged to connection setup versus upstream queueing; as far as I can tell, both contributed, and I couldn't observe the provider's internal queue.

I missed it.

That episode changed the experiment. I now test from a cold deployment, with burst arrival patterns, and report stage-level percentiles instead of one average. I measure gate latency, generation latency, total latency, block rate, rewrite rate, image acceptance rate, and retries. I also compare a warm path against a scale-from-zero path. Your mileage may vary, especially if your runtime keeps connections warm or your image request dominates every other stage, but hiding the gate inside a single end-to-end timer makes diagnosis needlessly hard.

There are a few safe ways to shorten the path. Cache an exact verdict only when the cache key includes the normalized prompt, policy version, classifier configuration, and schema version. Use deterministic rules for truly unambiguous cases, but test those rules against adversarial spelling and Unicode. Keep a small amount of compute warm when the measured tail justifies it. Don't run the classifier and generator in parallel; generating before the verdict defeats the purpose of avoiding an unsafe or wasted call.

Retries deserve their own budget. I separate connection failure, timeout before any response, explicit rejection, and malformed data because they imply different actions. The gate may be repeatable from my application's perspective, while the generation operation may create another billable result. RFC 9110 provides the semantic warning; the provider contract must supply the remaining facts. If those facts are unclear, I fail the request and preserve its key for reconciliation rather than guessing.

Measure the policy before copying the pattern

A typed response makes integration safer, but it doesn't prove the decisions are good. Before rollout, I label a representative evaluation set and calculate false allows, false blocks, rewrite acceptance, and disagreement by category. I keep ambiguous examples instead of deleting them from the set. Those are usually where policy language needs work, and a single top-line accuracy number can hide them.

I run the same set whenever I change the classifier configuration, system policy, categories, or schema. Then I shadow the new version on a sample of live requests without letting it control generation, compare verdicts, and manually review disagreements under an access policy. Only after that do I shift traffic. Ship-first doesn't mean blind-first.

Output review is a separate decision. Prompt screening can't guarantee what an image contains, so products with meaningful output risk need an image-side control or a human review queue appropriate to that risk. Products with fixed, application-authored prompts may not need a chat gate at all; enumerated inputs and deterministic validation are simpler, faster, and easier to reason about. Likewise, a low-volume internal prototype can begin with manual review, while a public product needs an appeal path and a way to update policy without redeploying unrelated code.

My go/no-go sheet has five lines: accepted-image quality, false-allow rate, false-block rate, p99 completion time, and cost per accepted image. I add operational questions beside them: Can I export my policy tests? Can I change either model behind a stable interface? Do retries have documented semantics? Can I explain a rejection without exposing sensitive internal instructions? Can I remove stored prompt data on schedule?

Copy the architecture only if the measurements show that its extra control is worth its latency and operational surface. If a required dedicated moderation system already supplies the policy contract, stick with it. If user input is closed and deterministic, skip the model gate. For the messy middle — open text prompts and a generation API without a separate moderation endpoint — a versioned JSON verdict is the least coupled approach I've found, but the evaluation set, not the schema, is what keeps it honest.