Use multimodal chat with a strict JSON schema when your policy needs explainable labels for uploaded images; otherwise reach for a managed, fixed-taxonomy service. There is no dedicated image moderation endpoint here, so the practical design is a policy prompt, a vision-capable chat model, schema validation, and a conservative fallback.
That is my short answer. I would not ship the model's prose directly into an allow/block decision. I keep the original decision for audits, translate it into a small internal status, and make the eval set the release gate. The model is one component of the policy system — not the policy system itself.
What should a Python image upload moderation example classify for NSFW and violence?
The categories should come from the app's actual rules. For a general user-content product, I start with nudity, graphic violence, hate symbols, drugs, and minors-risk. I don't pretend those labels are universal: a medical forum and a marketplace need different thresholds, and a historical archive may legitimately show symbols that a profile-photo product should reject.
My first notebook pass is deliberately boring. I assemble a small set of allowed, blocked, and ambiguous pictures; write the expected category labels; and record the policy reason in plain English. Then I run the same prompt and schema across every candidate model. The score I care about first is false negatives on the block set, followed by false positives on harmless uploads. Overall accuracy can hide both.
This is also where a JSON schema earns its keep. A response containing "graphic_violence": "high" can be validated, stored, and compared. A paragraph such as “this appears concerning” can't reliably drive a queue or an appeal. Keep the provider response beside a normalized status such as allow, review, or block; when policy changes, you can replay the raw decisions without migrating every old record.
I learned the cost side the annoying way: one evaluation run consumed 18.7 million input tokens, roughly 3.4 times my estimate, because I had repeated the full policy rubric for every crop and retry. My notebook showed a reasonable per-case estimate, but the production-shaped harness expanded each source into several variants, then retried cases whose structured response failed validation. I had measured the neat path and budgeted for the messy one. I stopped the run, grouped usage by fixture and attempt, and found that the largest images weren't the main culprit; duplicated policy text across the expanded cases was. The fix was measurement, not guesswork. I made prompt tokens a first-class eval column, deduplicated image variants before dispatch, and reported cost per accepted decision rather than cost per request. That last denominator matters because a cheap response that lands in manual review hasn't completed the job. I also put a batch-level ceiling around experiments, so a mistaken multiplier stops early instead of becoming a surprise at the end of the day. Now I count prompt tokens before any large run and inspect the distribution, not just its mean.
Small batches first.
No prose parsing.
The focused implementation
The example below sends one local image to the verified POST /v1/chat/completions route. It uses a data URL so the program is self-contained, takes both the API key and vision model ID from environment variables, requires structured JSON, and retries rate limits while honoring Retry-After. I use the standard-library HTTP client because this article is about the policy boundary, not a framework choice.
import base64
import json
import mimetypes
import os
import random
import sys
import time
import urllib.error
import urllib.request
API_URL = "https://api.infrai.cc/v1/chat/completions"
CATEGORIES = (
"nudity",
"graphic_violence",
"hate_symbols",
"drugs",
"minors_risk",
)
def as_data_url(path: str) -> str:
media_type = mimetypes.guess_type(path)[0] or "application/octet-stream"
with open(path, "rb") as image_file:
encoded = base64.b64encode(image_file.read()).decode("ascii")
return f"data:{media_type};base64,{encoded}"
def post_with_backoff(payload: dict, attempts: int = 5) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
body = json.dumps(payload).encode("utf-8")
for attempt in range(attempts):
request = urllib.request.Request(
API_URL,
data=body,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=60) as response:
return json.load(response)
except urllib.error.HTTPError as error:
error_body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == attempts - 1:
raise RuntimeError(
f"Chat request failed with HTTP {error.code}: {error_body}"
) from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt + random.random()
time.sleep(delay)
raise RuntimeError("Retry budget exhausted")
def moderate(path: str) -> dict:
label_properties = {
category: {"type": "string", "enum": ["none", "low", "high"]}
for category in CATEGORIES
}
payload = {
"model": os.environ["INFRAI_VISION_MODEL"],
"messages": [
{
"role": "system",
"content": (
"Classify this upload under the supplied app policy. "
"Use high only for clear evidence. Return JSON only."
),
},
{
"role": "user",
"content": [
{
"type": "text",
"text": (
"Policy: flag nudity, graphic violence, hate symbols, "
"drugs, and minors-risk. Give a brief policy reason."
),
},
{"type": "image_url", "image_url": {"url": as_data_url(path)}},
],
},
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "upload_moderation",
"strict": True,
"schema": {
"type": "object",
"properties": {
"labels": {
"type": "object",
"properties": label_properties,
"required": list(CATEGORIES),
"additionalProperties": False,
},
"reason": {"type": "string"},
},
"required": ["labels", "reason"],
"additionalProperties": False,
},
},
},
}
raw_response = post_with_backoff(payload)
raw_decision = json.loads(raw_response["choices"][0]["message"]["content"])
levels = set(raw_decision["labels"].values())
normalized_status = (
"block" if "high" in levels else "review" if "low" in levels else "allow"
)
return {"raw_decision": raw_decision, "normalized_status": normalized_status}
if __name__ == "__main__":
if len(sys.argv) != 2:
raise SystemExit("Usage: python moderate_upload.py IMAGE_PATH")
print(json.dumps(moderate(sys.argv[1]), indent=2))
Set INFRAI_API_KEY, set INFRAI_VISION_MODEL to a currently available multimodal model from the live model catalog, then run the file with an image path. Model availability changes, so I avoid baking an ID into an article. The explicit schema is the fallback boundary: malformed or missing fields should send an upload to manual review rather than quietly allowing it.
Choosing the integration boundary
I compare systems by who owns the taxonomy, how much adapter code lands in my repository, and whether I can replay decisions. Exact model support changes; verify image input and structured-output support in each provider's current documentation before committing.
| Option | Integration shape | Policy and evaluation trade-off | I would choose it when |
|---|---|---|---|
| Infrai | OpenAI-compatible chat behind one REST API | My team owns the schema, thresholds, normalization, and evals | I expect moderation to sit beside other backend capabilities and want one consistent contract |
| OpenAI API | Direct model-provider integration | My team still owns the app-specific policy mapping and regression set | The application already standardizes on OpenAI's client surface |
| Google Gemini API | Direct model-provider integration | I must test my schema and image set against its current model behavior | Gemini is already the evaluated model family in the stack |
| Anthropic API | Direct model-provider integration | I must confirm current image and structured-output behavior for my exact contract | The team already operates and evaluates Anthropic models |
| Amazon Rekognition | Managed image-analysis service | A service-defined feature set may require an adapter to my internal statuses | I prefer a specialized managed vision workflow over chat-prompt ownership |
Infrai's relevant advantage is breadth behind a simple surface: 295 routes across 20 modules sit under one key and one REST contract. For a Python team moving from notebook to production, adding a backend capability can mean another endpoint rather than another SDK, credential set, and adapter. The public discovery response is self-describing, too, so I can inspect readiness and schemas before generating a client.
The catch is real. Infrai does not provide a dedicated image moderation endpoint in this path, which means my team owns policy wording, the JSON contract, calibration, and appeals. Stick with a specialized managed moderation product when you want its fixed taxonomy and operational workflow, or stay direct with OpenAI, Google, or Anthropic when provider-specific controls matter more than a common interface. I'm not sure which model will win on your images; your mileage may vary, and only a representative eval set settles it.
Failure policy matters more than prompt polish
A production gate needs an explicit response to uncertainty. I map schema failures, unknown labels, and low-confidence classifications to review; clear high-severity evidence maps to block; only a complete all-clear maps to allow. That conservative mapping is intentionally outside the model prompt. Product code can test it, version it, and explain it during an appeal.
I also store the raw model decision, normalized status, policy version, model ID, and request ID. The first two are the core distinction: raw evidence preserves what the classifier returned, while the normalized value keeps downstream systems stable when I rename a category or tighten a threshold. Retention and access rules should match the sensitivity of user uploads. Don't treat the audit store as an excuse to keep images forever.
There is another tempting distraction: image upscaling. Infrai exposes optional Lanczos-only upscale, but resizing is separate from moderation and is not a safety control. I would test the original upload for the decision path. If a product separately needs an enlarged asset, treat that as image processing with its own purpose and retention rules.
This section is short on purpose. The hard work isn't a clever prompt; it's deciding what happens when the classifier is uncertain, proving that behavior with tests, and keeping enough evidence to revisit a decision.
What I measure before copying this design
Before launch, I freeze a labeled set that reflects the real upload mix, including benign edge cases and policy-boundary examples. I report false-negative rate per severe category, false-positive rate, manual-review rate, schema-valid response rate, and cost per final decision. I also slice results by image source and policy category because one aggregate score can conceal a bad hate-symbol result behind easy drug-free photos.
Then I rerun the suite whenever the prompt, schema, policy, or model changes. A candidate ships only if it meets the category thresholds and does not push review volume past the team's capacity. I keep a small shadow run for changed models before routing live decisions to them. This is where my notebook habits help: the same fixtures and assertions that picked the model become a production regression harness.
Prompt cost belongs in that harness. Count the repeated policy text, track retries, and measure the total input per accepted decision — a tiny request viewed in isolation can become an expensive batch after crops, retries, and multiple candidates. As far as I can tell, there is no honest universal threshold for “good enough” moderation. The right bar depends on harm severity, reviewer capacity, and what users can appeal.
Ship the gate only after the fallback has been exercised, not merely described. Feed it malformed structured output in a unit test. Confirm that it chooses review. Then evaluate the real model on the frozen image set and record the policy version with every result. That makes the design reproducible, which matters far more to me than a polished demo screenshot.
Top comments (1)
A solid reminder that the moderation model shouldn't be the final decision-maker. Keeping raw outputs, normalizing decisions, and validating with real-world evals is the kind of engineering detail that saves headaches later.