Short answer: put a schema-constrained chat classifier in front of image generation, fail closed on malformed output, and charge every classification attempt to the same tenant ledger as the resulting invoice job.
For a fintech workflow that extracts fields from supplier invoices, the least complex useful design has three steps. Accept the tenant ID and proposed image prompt, classify the text into a small typed decision, then enqueue image generation only when the decision is allow. Keep the raw classifier response out of business logic. This works without a dedicated moderation endpoint because the contract belongs to your application, while the chat model supplies a structured assessment.
The important boundary is accounting, not model cleverness. A supplier can retry an upload, a worker can redeliver a message, and a classifier can return invalid JSON. If those paths aren't attached to one tenant-scoped operation ID, the safety gate may work while the invoice margin report quietly lies.
How should Node.js moderate text prompts for AI image generation without a moderation endpoint?
Treat moderation as a deterministic application gate around a probabilistic classifier. The HTTP handler validates the request. A chat adapter requests JSON matching a narrow schema. A policy function makes the final allow-or-block decision. An append-only usage record captures the tenant, operation, attempt, token counts, and outcome before the image job can proceed.
Don't let the model return a prose essay. For this invoice use case, the useful output is deliberately boring: a decision, a bounded set of labels, and a short reason suitable for an internal audit trail. The model may identify categories, but code owns the policy. That separation lets a team tighten a rule without changing the prompt or replay historical records through a new policy version.
The gate should also receive the workflow context. A prompt such as “remove the account number from this invoice scan” means something different from an unrestricted request to synthesize a photorealistic identity document. Context isn't permission, though. Pass a fixed task identifier such as supplier_invoice_field_extraction, never free-form claims like “trusted tenant.”
Build the typed gate before the image worker
Here is a compact TypeScript example. The chat URL and model are configuration because the application depends on a contract, not a particular provider. The example uses native fetch, validates the response rather than trusting a type assertion, and records usage for blocked, allowed, and invalid responses.
type SafetyLabel =
| "sexual_content"
| "graphic_violence"
| "identity_fraud"
| "personal_data"
| "none";
type SafetyDecision = {
decision: "allow" | "block";
labels: SafetyLabel[];
reason: string;
};
type ChatUsage = {
inputTokens: number;
outputTokens: number;
};
type GateResult = {
safety: SafetyDecision;
usage: ChatUsage;
};
const safetySchema = {
name: "invoice_image_prompt_safety",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: ["decision", "labels", "reason"],
properties: {
decision: { type: "string", enum: ["allow", "block"] },
labels: {
type: "array",
uniqueItems: true,
items: {
type: "string",
enum: [
"sexual_content",
"graphic_violence",
"identity_fraud",
"personal_data",
"none",
],
},
},
reason: { type: "string", minLength: 1, maxLength: 160 },
},
},
} as const;
function isSafetyDecision(value: unknown): value is SafetyDecision {
if (!value || typeof value !== "object") return false;
const item = value as Record<string, unknown>;
const labels = new Set([
"sexual_content",
"graphic_violence",
"identity_fraud",
"personal_data",
"none",
]);
return (
(item.decision === "allow" || item.decision === "block") &&
Array.isArray(item.labels) &&
item.labels.every((label) => typeof label === "string" && labels.has(label)) &&
typeof item.reason === "string" &&
item.reason.length > 0 &&
item.reason.length <= 160
);
}
async function classifyPrompt(prompt: string): Promise<GateResult> {
const response = await fetch(process.env.CHAT_API_URL!, {
method: "POST",
headers: {
authorization: `Bearer ${process.env.CHAT_API_KEY}`,
"content-type": "application/json",
},
body: JSON.stringify({
model: process.env.CHAT_MODEL,
messages: [
{
role: "system",
content:
"Classify an image-generation prompt for a supplier-invoice extraction workflow. Return only the requested structured decision. Block identity fraud, exposed personal data, sexual content, and graphic violence.",
},
{ role: "user", content: prompt },
],
response_format: {
type: "json_schema",
json_schema: safetySchema,
},
}),
});
if (!response.ok) {
throw new Error(`Classifier request failed with status ${response.status}`);
}
const payload = (await response.json()) as {
choices?: Array<{ message?: { content?: string } }>;
usage?: { prompt_tokens?: number; completion_tokens?: number };
};
const content = payload.choices?.[0]?.message?.content;
if (!content) throw new Error("Classifier returned no structured content");
const parsed: unknown = JSON.parse(content);
if (!isSafetyDecision(parsed)) {
throw new Error("Classifier response did not match the safety schema");
}
return {
safety: parsed,
usage: {
inputTokens: payload.usage?.prompt_tokens ?? 0,
outputTokens: payload.usage?.completion_tokens ?? 0,
},
};
}
The endpoint shape above follows the common chat-completions request and response convention documented by OpenRouter, but the adapter is intentionally generic. A different compatible service can sit behind CHAT_API_URL. If a provider expresses structured output differently, change this adapter; don't leak that variation into the queue worker or tenant ledger.
Now connect classification to the business operation. The idempotency key belongs to the invoice attempt, while stage distinguishes the classification charge from later image work. A unique constraint on (tenantId, operationId, stage, attempt) prevents one delivered message from being counted twice.
type UsageRecord = {
tenantId: string;
operationId: string;
stage: "prompt_classification" | "image_generation";
attempt: number;
inputTokens: number;
outputTokens: number;
outcome: "allowed" | "blocked" | "invalid";
policyVersion: string;
};
interface UsageLedger {
append(record: UsageRecord): Promise<void>;
}
async function screenInvoiceImageRequest(
ledger: UsageLedger,
input: {
tenantId: string;
operationId: string;
attempt: number;
prompt: string;
},
): Promise<SafetyDecision> {
try {
const result = await classifyPrompt(input.prompt);
await ledger.append({
tenantId: input.tenantId,
operationId: input.operationId,
stage: "prompt_classification",
attempt: input.attempt,
inputTokens: result.usage.inputTokens,
outputTokens: result.usage.outputTokens,
outcome: result.safety.decision === "allow" ? "allowed" : "blocked",
policyVersion: "invoice-image-v1",
});
return result.safety;
} catch (error) {
await ledger.append({
tenantId: input.tenantId,
operationId: input.operationId,
stage: "prompt_classification",
attempt: input.attempt,
inputTokens: 0,
outputTokens: 0,
outcome: "invalid",
policyVersion: "invoice-image-v1",
});
throw error;
}
}
Zero token counts on an invalid attempt mean “usage unavailable,” not “free.” Preserve that distinction in reporting. If exact provider usage is required for reconciliation, store the provider request ID beside the record and reconcile it later rather than guessing token counts.
Cost attribution changes the retry design
HTTP semantics matter once money is attached to attempts. RFC 9110 defines idempotent methods and explains that a client can automatically retry an idempotent request after a communication failure. A chat classification call is normally a POST, so don't assume transport-level retries are free of duplicate work. Give each application attempt a stable operation ID, record the attempt number, and make the ledger write idempotent even if the upstream inference request is not.
Consider one supplier invoice request for tenant tenant_42. Attempt 1 reaches the classifier, but the worker loses its connection before it receives the result. Attempt 2 may cause another classification. The cost report should show two inference attempts attached to one operation, while the job table should still show one logical invoice request. Collapsing both views into a single counter hides retry overhead; treating them as two invoices overstates customer activity. This is the small data-model choice that decides whether per-tenant gross margin is actionable or fictional.
Keep these three units separate:
- A logical operation is the tenant's requested invoice task.
- An attempt is one execution of a stage, including a retry.
- A billable usage event is the provider-reported consumption for that attempt.
Short version: deduplicate work, not evidence.
The catch is that schema-constrained chat is not suitable when policy requires a provider-certified moderation taxonomy, calibrated category scores, or a formal appeal artifact. Use a dedicated moderation service in that case. Stick with a local rules engine when policy is mostly exact terms, tenant allowlists, or data-loss-prevention patterns; calling a generative model for deterministic matching adds latency, variable cost, and uncertainty. A chat classifier fits the middle ground, where context matters and your application can own the policy decision.
Test policy behavior, not persuasive wording
A safety prompt is code with fuzzy execution. Version it, test it, and deploy it behind a shadow comparison before changing enforcement. The fixture set should include obvious blocks, obvious allows, obfuscated terms, mixed-language prompts, quoted unsafe text, and invoice-specific ambiguity such as account numbers that appear in extraction instructions. Expected labels must come from a written policy, not from whatever the current model happened to answer.
I'm not sure one universal threshold can be defensible across fintech tenants; contractual rules and jurisdictions vary. What resolves that uncertainty is a tenant-level policy record reviewed by the people responsible for compliance, plus an evaluation set that reports false allows and false blocks for each policy version. Don't quietly put tenant policy prose into the user message, because a malicious prompt can then compete with it. The system instruction, schema, and executable policy remain under server control.
Test parser failures too. Feed the validator an empty body, extra properties, an unknown label, a 161-character reason, and syntactically invalid JSON. Each case should fail closed before image generation. Then test the ledger under duplicate delivery: replay the same (tenantId, operationId, stage, attempt) and verify that reporting has one usage event, while a new attempt number creates another.
Be strict here.
Observability should answer four questions without inspecting raw prompts: which policy version ran, how often each outcome occurred, how much classifier usage each tenant generated, and how many operations required another attempt. Raw invoice prompts may contain personal or financial data, so store a one-way digest or a tightly controlled reference when the audit need doesn't justify retaining the text itself. Redaction and retention are policy decisions; the classifier shouldn't decide them.
Operate the gate as part of the invoice pipeline
Before release, walk one request from ingress to ledger to queue. Confirm that tenant identity comes from authenticated server context, the operation ID survives redelivery, the classifier output passes runtime validation, and only allow reaches image generation. Confirm separately that block produces a stable application response and a recorded policy version. Finally, compare ledger totals with provider usage over the same reporting window, and alert on unexplained drift rather than on raw spend alone.
Keep the classifier adapter replaceable, but don't build a grand provider abstraction on day one. A narrow interface for classifyPrompt and a stable internal SafetyDecision are enough. Provider-specific request fields live in one file; policy, metering, and queue behavior stay independent. That gives a solo team a practical exit path without paying an abstraction tax before there is a second implementation.
Ship the first policy with a small labeled fixture set and explicit fail-closed behavior. Expand the evaluation corpus from reviewed production categories, not from raw customer prompts copied into a test repository. Watch tenant cost per logical operation alongside attempts per operation: the first reveals workload economics, while the second exposes retry amplification that a blended token total would miss.
No magic required.
Further reading
- RFC 9110: HTTP Semantics — https://www.rfc-editor.org/rfc/rfc9110
- OpenRouter documentation — https://openrouter.ai/docs
Top comments (0)