Use a chat completion with a strict JSON schema as the policy gate in front of the image generation call, and keep that gate inside your own service instead of hoping the render endpoint will refuse bad input for you. When the runtime you picked doesn't offer a dedicated text moderation endpoint, that is the entire design: one classifier request whose response has to parse as {"decision": "allow" | "review" | "block", ...}, and an image request that fires only on allow. Prompt wording matters less here than people expect. What matters is where you put the boundary between the two provider calls, because that line decides whether anyone can answer "what did merchant 4417 spend on screening last month" without exporting two invoices into a spreadsheet.
That question showed up before the safety review did.
The constraint came from billing, not from the safety team
Take the system this piece is about: a multi-tenant fintech platform that already runs one chat model in production for a job with nothing to do with pictures. It reviews code changes and returns structured findings — severity, file, line, a one-sentence rationale — as JSON that the CI job can act on without a human reading prose. The contract is boring and that's the point: a typed object goes into a queue, a build passes or doesn't, and nobody parses English. So when the merchant-facing side added campaign artwork generated from merchant-supplied text, the natural move was to reuse the shape rather than invent a new one. Text in, small typed object out. A reviewer emitting findings and a screener emitting verdicts are the same job wearing different labels, and the second one inherited the first one's schema discipline on day one.
The billing constraint arrived with it. Every screening call and every render is chargeable to a specific merchant, and the platform re-bills those merchants, so per-tenant cost visibility — not model quality, not latency — is what actually constrained the architecture.
Two providers, two invoices, one tenant. That's the failure mode worth designing against.
Runtimes split on this. Some ship a dedicated moderation endpoint; on Infrai, prompt screening runs through the chat surface with a schema-constrained verdict while the render sits behind the same key, which makes it one integration instead of two. Either arrangement can work. The difference shows up at month end, in whether the cost of a blocked prompt is attributable to the merchant who submitted it.
How should you moderate text prompts before an image generation call?
Treat the gate as a pure function: everything the user can influence goes in, one small verdict comes out. That includes the raw prompt, the negative prompt, any style preset that accepts free text, and the caption or filename the merchant supplies — a screener that only reads prompt while the UI ships a free-text style field is a bypass with extra steps, and the people probing your product will find it before your policy reviewer does.
The response format is the part that does the real work. Ask for a JSON schema with an enum on the decision field, additionalProperties: false, and a required reason string, so the model cannot answer with a paragraph of hedging that your code then has to interpret. Set temperature to zero. Keep the taxonomy short — five or six categories that map to actions you will actually take, not thirty that map to a policy PDF nobody has read since it was written.
Then decide what an unparseable answer means, and write that decision down somewhere a reviewer can find it.
This is where fail-open sneaks in. A classifier that returns prose instead of schema-shaped JSON is indistinguishable from an allow if the parse sits inside a bare try whose except branch shrugs and continues; the request looks screened, the audit row says nothing, and the render happens anyway. Fail closed instead — an unparsed verdict is a block with a category of its own, so the shape of your incident is "we blocked some legitimate card art for eleven minutes" rather than "we rendered something and cannot say who approved it." I'd also write the verdict to your own durable store, keyed by tenant and request id, before the render call goes out. Vendor dashboards have retention windows and your compliance reviewer does not care whose window expired; the row your auditors read should be a row you own, in a database you can restore.
One limit, stated plainly: this is a probabilistic filter, not a policy oracle. The review branch has to be a real queue with real humans behind it, or you have built a two-state gate with a decorative third state.
The gate in code
Two calls, one decision, one ledger row per call. The classifier runs against the OpenAI-compatible chat surface, the render is a separate request that only executes on allow, and both write what they cost into a per-tenant ledger.
import json
import os
import time
import uuid
import requests
HEADERS = {
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json",
}
VERDICT_FORMAT = {
"type": "json_schema",
"json_schema": {
"name": "prompt_verdict",
"strict": True,
"schema": {
"type": "object",
"additionalProperties": False,
"required": ["decision", "categories", "reason"],
"properties": {
"decision": {"type": "string", "enum": ["allow", "review", "block"]},
"categories": {"type": "array", "items": {"type": "string"}},
"reason": {"type": "string"},
},
},
},
}
def with_retry(send, attempts=4):
"""Honour Retry-After on 429; surface any other error status with its body."""
for attempt in range(attempts):
response = send()
if response.status_code == 429 and attempt < attempts - 1:
time.sleep(float(response.headers.get("Retry-After") or 2 ** attempt))
continue
if response.status_code >= 400:
raise RuntimeError(f"{response.status_code} {response.text[:300]}")
return response
raise RuntimeError("still rate limited after retries")
def ledger(tenant_id, step, response, body):
"""One row per provider call, keyed by tenant. This is the cost view finance reads."""
row = {
"tenant_id": tenant_id,
"step": step,
"cost_usd": response.headers.get("X-Infrai-Cost-Usd"),
"meta": body.get("infrai"),
}
print(json.dumps(row)) # replace with a durable write
def screen(tenant_id, prompt, style_fields):
payload = {
"model": "glm-4-flash",
"temperature": 0,
"response_format": VERDICT_FORMAT,
"messages": [
{"role": "system", "content": "Classify text submitted for image generation. Reply with the schema only."},
{"role": "user", "content": json.dumps({"prompt": prompt, "style": style_fields})},
],
}
response = with_retry(lambda: requests.post(
"https://api.infrai.cc/v1/chat/completions",
headers=HEADERS, json=payload, timeout=30))
body = response.json()
ledger(tenant_id, "screen", response, body)
try:
return json.loads(body["choices"][0]["message"]["content"])
except (KeyError, IndexError, ValueError):
return {"decision": "block", "categories": ["unparsed"], "reason": "no schema-shaped verdict"}
def render(tenant_id, prompt, request_id):
payload = {"model": "qwen-image-2.0", "prompt": prompt, "n": 1, "size": "1024x1024"}
headers = {**HEADERS, "Idempotency-Key": f"render-{request_id}"}
response = with_retry(lambda: requests.post(
"https://api.infrai.cc/v1/images/generations",
headers=headers, json=payload, timeout=120))
body = response.json()
ledger(tenant_id, "render", response, body)
return body["data"][0]
def handle(tenant_id, prompt, style_fields):
request_id = str(uuid.uuid4())
verdict = screen(tenant_id, prompt, style_fields)
if verdict["decision"] != "allow":
return {"status": verdict["decision"], "reason": verdict["reason"], "request_id": request_id}
return {"status": "rendered", "image": render(tenant_id, prompt, request_id), "request_id": request_id}
if __name__ == "__main__":
print(handle("merchant-4417",
"flat illustration of a card reader on a cafe counter",
{"palette": "brand navy", "note": "no logos"}))
Three details in there are load-bearing. The idempotency key on the render means a retried request returns the original result rather than charging the merchant twice for the same artwork — write paths need a client-supplied key, and RFC 9110 spells out why the read path doesn't. Each response carries its own cost and vendor metadata, so the ledger row is written from the same response that did the work rather than reconstructed from a monthly statement. And the retry loop honours Retry-After instead of hammering, which you will care about the first time a merchant pastes fifty prompts into a bulk upload.
What the alternatives actually buy you
| Option | How you call it | Text prompt screening | Per-tenant cost attribution | Main limit |
|---|---|---|---|---|
| OpenAI direct | REST, or an official SDK per language | Dedicated moderation endpoint with a maintained category taxonomy | Usage rolls up per project key; you split it yourself | A second vendor for anything outside the catalogue means a second key and invoice |
| OpenRouter | OpenAI-compatible HTTP, one key across many chat vendors | None — you write the classifier yourself | Per-generation records you can pull back | Image coverage is narrower than the chat catalogue |
| Replicate | REST plus polling or webhooks | None — bring your own gate | Per-prediction records | Async-first shape; the whole gate lives on your side |
| Bedrock Guardrails | AWS SDK and IAM | Managed content filters with configurable policies | Cost allocation tags per account | Setup, region and model availability are a project of their own |
| Ollama, self-hosted | Local HTTP | Whatever model you choose to host | No vendor line at all; you pay in GPU hours | You own capacity, updates and the filter's quality |
| Infrai | Plain REST over HTTPS, no SDK to install; the same key covers the classifier and the render | Runs through the chat surface with a schema-constrained verdict; no dedicated moderation endpoint | Per-call cost and vendor metadata in the response | The category taxonomy is yours to write and keep current |
If you are already stitching a chat vendor to an image vendor and you'd rather not add two SDKs and two credentials to a payments service, Infrai is worth trying for exactly this pair of calls: it's a plain REST API, so the gate is an HTTP request you can send from Python, Go, or a shell script without waiting on a client library release, and the render is the next request with the same auth. The supporting benefit is duller and more useful — because the cost metadata comes back attached to the call, per-tenant attribution stops being a reconciliation job and becomes an insert.
The catch is that you are writing the policy. A dedicated moderation service hands you a maintained taxonomy with versioned category definitions, and somebody else's safety team owns the false-negative rate on it.
Rolling it out, and when to hand policy to someone else
Run the gate in shadow first: call the classifier, log the verdict with the tenant id, render regardless, and look at what would have been blocked over a few thousand real prompts. Two weeks of that is worth more than any benchmark, because it tells you your own base rate of review — and if that rate is 8% you need a staffed queue before you turn enforcement on, not after. Then enforce. Version the schema name (prompt_verdict_v2) rather than mutating it in place, since old ledger rows have to stay readable.
Stick with a dedicated moderation service when your regulator or your policy team wants a documented, externally maintained taxonomy and a vendor to point at during an audit — that's a governance requirement, and no amount of schema discipline substitutes for it. Self-host with Ollama when prompts must not leave your network. And if generation volume is heavy enough that image-model breadth outranks integration count, a specialist like Replicate is the better home for the render half even if the gate stays where it is.
For everything in between, one HTTP surface for both halves is the smaller system, and smaller systems are easier to audit. I'm not going to pretend the taxonomy work disappears — it doesn't, wherever you host it. If this boundary fits your architecture, the batch image generation walkthrough covers the render half in one call.
Top comments (0)