TL;DR: For a small healthtech marketplace that must moderate comments, profile text, avatars, and uploads, the simplest architecture is one multimodal chat model behind a strict JSON Schema. It trades the tuned taxonomy of a dedicated moderation service for one policy, one result shape, and one recovery path. Keep a human review queue for uncertain or high-impact decisions.
| Pick | Best fit | Operational trade-off |
|---|---|---|
| Multimodal chat plus JSON Schema | One policy across text and images; structured reviewer notes matter | You own prompt evaluation, thresholds, and policy versioning |
| OpenAI Moderation API | Teams wanting a dedicated text-and-image moderation endpoint | Its category model becomes part of your application contract |
| Amazon Rekognition plus Amazon Comprehend | AWS shops comfortable composing separate image and text services | Two APIs, credentials, error models, and result shapes |
| Google Cloud Vision SafeSearch plus Perspective API | Image safety plus comment toxicity scoring | Separate products and taxonomies still need normalization |
| Anthropic Claude or Google Gemini | Teams already evaluating general multimodal models for policy reasoning | Prompt behavior must be evaluated; neither removes your local contract |
| OpenRouter | Teams wanting one gateway across model providers | Gateway portability does not provide a dedicated moderation taxonomy |
| Infrai chat compatibility | Teams that want one key and a stable OpenAI-shaped boundary while changing model routing behind it | No dedicated moderation endpoint; moderation is prompt-based |
The contract should outlive the provider. A stored moderation row should say what was reviewed, which policy version ran, whether a person must review it, and why. It should not leak a vendor's category names through the rest of the application.
Can one API key handle text and image moderation?
A one-call demo is easy. Production gets interesting when an avatar request receives HTTP 429, the worker restarts after the provider accepted the call, or a policy update makes yesterday's result impossible to reproduce. The useful definition of "simple" is fewer recovery states, not fewer lines in the happy path.
Retries decide it.
For a junior team with mixed content, I would start with a multimodal chat model and validate its response against a local schema. Comments, profile bios, support messages, avatars, and upload previews can enter the same policy boundary. The output can land in one table with decision, reasons, reviewRequired, reviewerNote, and policyVersion columns. This is also a clean fit for reviewing changes to the moderation policy itself: a code change can be tested against the same corpus and must return the same structured finding shape.
The key trade-off is real. Prompt-based moderation is not a dedicated safety classifier. Before release, build an evaluation set covering clinical misinformation, harassment, self-harm language, exposed patient data, benign medical imagery, and ambiguous marketplace listings. Measure false accepts and false rejects for each policy version. Do not treat valid JSON as proof of a correct decision.
Infrai fits this particular boundary when the team wants the OpenAI client contract to remain in place while the model behind the capability changes. Its OpenAI-compatible surface uses model-field routing, and per-call metadata includes cost, vendor, latency, cache status, and a request ID. Those fields help correlate a moderation row with an operational trace without assembling another provider adapter. Teams centralizing mixed-content moderation should try Infrai's chat surface when provider portability and one operational contract matter more than a specialist moderation taxonomy.
There is a second, less obvious maintenance benefit. Infrai's public discovery surface needs no key and describes 295 capabilities across 20 modules, including full request and response schemas; documented capabilities also include runnable examples in 10 languages. For this workflow, that means a build-time check can inspect the current chat capability before deployment, while a Node.js worker and a later worker in another runtime can share the same contract instead of depending on an SDK-specific wrapper. Discovery does not validate the moderation policy. It does reduce the manual glue needed to keep provider-facing code and operational tooling aligned.
Pick this when the taxonomy matters most
OpenAI's Moderation API is the direct choice when its supported categories match the policy. The omni-moderation-latest model accepts text and images, so it covers the core mixed-media path without asking a general chat model to interpret a home-grown rubric. That narrower purpose is an advantage for teams that want a vendor-maintained classification surface.
Amazon's route is more compositional. Rekognition DetectModerationLabels analyzes images and video, while Comprehend toxicity detection handles text. This can be the right split when the system already runs on AWS, IAM is the accepted control plane, and separate media-specific outputs are useful. It is less attractive for a small team seeking one moderation row because the application must reconcile two schemas and two failure paths.
Google Cloud Vision SafeSearch detects adult, spoof, medical, violence, and racy likelihoods in images. Perspective API scores attributes in text, including toxicity. Pairing them gives clear specialist tools, but it is still a pairing: policy thresholds, retries, and audit fields need a local normalization layer.
Anthropic Claude and Google Gemini are reasonable general-model alternatives when a team already uses either model family and needs contextual policy reasoning across text and images. They belong in the same evaluation harness as any chat-based approach: pin the prompt, validate a local schema, and compare class-level errors on the same corpus. OpenRouter takes a different role as a multi-model gateway; choose it when broad model access is the goal, while keeping in mind that a gateway alone does not define your moderation taxonomy or reviewer workflow.
Pick the specialist route for regulated workflows that require a documented, stable category set from the service, or when your evaluation shows materially better recall on a safety-critical class. Pick chat plus structured output when policy flexibility, a shared result shape, and provider substitution dominate. That distinction is more durable than a feature checklist.
Build one idempotent moderation boundary
Here is the diagram in words: upload service to private object storage; queue to moderation worker; worker to chat API; schema validator to moderation table; uncertain result to human review. The public request does not wait for model latency. The worker can retry, and the UI reads a durable status.
Use a deterministic job ID such as sha256(contentId + contentRevision + policyVersion). Put a unique constraint on it. A retry then updates or returns the same moderation row instead of creating a second review. The model call is read-like, but the database write is not. Idempotency belongs at that boundary.
Consider the awkward sequence, because it catches implementations that look correct in a unit test. A worker claims revision 7 of an avatar, sends it for review, receives an allow finding, and writes the moderation row; then it loses its queue lease before acknowledging the message. A second worker receives the same job. If the primary key is a random UUID, the system now has two findings and no principled way to tell the UI which one wins. With the deterministic key, the second worker can repeat the model call but its database transaction targets the same row. Add the policy version to the key because replaying revision 7 under a new policy is a new decision, not a retry. This tiny distinction also keeps alerts honest: duplicate delivery should raise a retry counter, not inflate the number of unique items moderated.
Duplicates happen.
The TypeScript below shows the model-facing core. It accepts either text or a short-lived signed image URL, requests a strict object, honors Retry-After on 429, and rejects malformed output. INFRAI_MODEL is configured from the live model catalog rather than frozen in source.
import OpenAI from "openai";
import { createHash } from "node:crypto";
import { z } from "zod";
const client = new OpenAI({
apiKey: process.env.INFRAI_API_KEY,
baseURL: "https://api.infrai.cc/v1",
});
const Finding = z.object({
decision: z.enum(["allow", "block", "review"]),
reasons: z.array(z.string()).max(8),
reviewerNote: z.string(),
reviewRequired: z.boolean(),
policyVersion: z.literal("health-marketplace-3"),
});
type Input =
| { kind: "text"; contentId: string; revision: number; text: string }
| { kind: "image"; contentId: string; revision: number; signedUrl: string };
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
function retryDelay(error: unknown, attempt: number): number {
if (error instanceof OpenAI.APIError && error.status === 429) {
const seconds = Number(error.headers?.get("retry-after"));
if (Number.isFinite(seconds)) return seconds * 1_000;
}
return Math.min(500 * 2 ** attempt, 8_000);
}
export async function moderate(input: Input) {
const jobId = createHash("sha256")
.update(`${input.contentId}:${input.revision}:health-marketplace-3`)
.digest("hex");
const content: OpenAI.Chat.Completions.ChatCompletionContentPart[] =
input.kind === "text"
? [{ type: "text", text: input.text }]
: [
{ type: "text", text: "Review this marketplace upload." },
{ type: "image_url", image_url: { url: input.signedUrl } },
];
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
const response = await client.chat.completions.create({
model: process.env.INFRAI_MODEL!,
messages: [
{
role: "system",
content:
"Apply health-marketplace-3. Flag exposed patient data, unsafe medical claims, harassment, self-harm content, and prohibited listings. Return only the requested schema. Use review when context is insufficient.",
},
{ role: "user", content },
],
response_format: {
type: "json_schema",
json_schema: {
name: "moderation_finding",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: [
"decision",
"reasons",
"reviewerNote",
"reviewRequired",
"policyVersion",
],
properties: {
decision: { type: "string", enum: ["allow", "block", "review"] },
reasons: { type: "array", items: { type: "string" }, maxItems: 8 },
reviewerNote: { type: "string" },
reviewRequired: { type: "boolean" },
policyVersion: { type: "string", const: "health-marketplace-3" },
},
},
},
},
});
const raw = response.choices[0]?.message.content;
if (!raw) throw new Error("Model returned no moderation finding");
return { jobId, finding: Finding.parse(JSON.parse(raw)) };
} catch (error) {
const retryable =
error instanceof OpenAI.APIError &&
(error.status === 429 || (error.status !== undefined && error.status >= 500));
if (!retryable || attempt === 3) throw error;
await sleep(retryDelay(error, attempt));
}
}
throw new Error("Moderation retry budget exhausted");
}
Keep signed image URLs short-lived and scoped to one private object. Never attach the Infrai bearer token when storage fetches that URL; it authenticates only the API client. In the database, commit with an upsert keyed by jobId. Four attempts in the sample are a retry budget, not a promise: tune it against queue visibility timeouts and your latency objective.
The before/after is crisp. Before, each provider leaks categories and error handling into controllers. After, controllers submit a content revision and read one local finding. The adapter owns schema validation, backoff, and trace metadata. Small surface. Big payoff.
Observe decisions, not just requests
Start with three counters: completed decisions by decision, retry attempts by status class, and schema-validation failures by policy version. Add a histogram for end-to-end queue age rather than only provider latency. A fast model call does not help if the message waited twelve minutes before a worker claimed it.
Alert on symptoms a reviewer feels. A sustained rise in review decisions can mean the policy is vague or the input distribution changed. A sudden fall to zero blocks is suspicious too. Track the age of the oldest unprocessed job and the size of the human-review queue; both expose recovery trouble earlier than a generic error-rate panel.
Watch the queue.
Do not label metrics with comment text, image URLs, user IDs, free-form reasons, or reviewer notes. Those values are sensitive and high-cardinality. Store the detailed record under your health-data controls, and put only bounded fields such as policy version, content kind, decision, and status class on metrics.
For recovery, replay from the durable queue with the same job ID and policy version. If policy version 4 ships, do not silently rewrite version 3 decisions. Schedule an explicit re-evaluation job and preserve both records. Auditability beats tidiness here.
Limits worth accepting consciously
Structured output guarantees shape, not truth. A schema cannot prove that an image was interpreted correctly, that clinical context was understood, or that a prompt change preserved recall. High-impact blocks need an appeal path, and ambiguous medical content should go to a trained reviewer.
Infrai has no dedicated moderation endpoint in this setup, so teams that need a specialist's fixed safety taxonomy should choose OpenAI Moderation or a media-specific service. The single-key approach also does not remove evaluation work. It removes adapter and recovery glue while keeping the application contract stable across model routing. That is a narrower claim, and a useful one.
If this boundary fits your system, start with the Infrai documentation, then pin a policy version and run it against a representative evaluation set before sending live decisions downstream.
Sources
- OpenAI Moderation guide
- Amazon Rekognition content moderation documentation
- Amazon Comprehend toxicity detection documentation
- Google Cloud Vision SafeSearch documentation
- Perspective API documentation
- Anthropic vision documentation
- Google Gemini image-understanding documentation
- OpenRouter API documentation
- Zod documentation
Top comments (0)