Why Prompt-Only Moderation Failed in My AI Generation App

Bob — Fri, 10 Apr 2026 06:50:29 +0000

When I first added moderation to my AI generation app, I treated it as a text problem.

That seemed reasonable at the time. A user sends a prompt, I check the prompt, and if it looks unsafe, I block the request before it reaches the model.

That approach worked for a very short time.

It stopped working the moment I supported image inputs, reference images, and multiple generation flows. At that point, I realized something important: prompt-only moderation is not really moderation. It is just one partial check inside a much larger pipeline.

This post is about what changed in my backend once I accepted that.

The mistake: treating moderation as a wrapper

A lot of AI products start with moderation as a thin wrapper around generation:

receive a prompt
run a text safety check
call the model provider
return the result

The problem is that real generation workflows are rarely that simple.

Once users can upload source images, provide reference images, or switch between text-to-image and image-to-image generation flows, the prompt becomes just one component of the overall request. A completely harmless prompt can still be paired with problematic input images. If the backend only inspects the text, the system will inevitably have a blind spot.

That was the first issue I had to fix.

Moderation belongs inside the generation pipeline

I ended up moving moderation into the backend generation workflow itself instead of treating it as a separate utility.

Conceptually, the flow became:

validate the request
load the selected provider and model
inspect both prompt text and image inputs
block flagged requests before spending credits
create the generation task only if moderation passes

That decision helped for two reasons.

First, it kept moderation close to the actual business rules. I did not want unsafe requests to consume credits, create external jobs, or leave behind half-failed task records.

Second, it forced me to normalize the input shape. Instead of only thinking in terms of prompt, I had to define a moderation input that could include prompt text, image URLs, model context, and generation scene.

That made the system much easier to reason about.

Prompt checks are useful, but incomplete

Text moderation is still valuable. It catches a lot of obvious cases early, and it is usually cheaper and faster than processing images.

But text-only checks have two major limitations.

The first is obvious: users can submit problematic visual input even if the prompt itself looks harmless.

The second is less obvious: language coverage is uneven. Depending on the moderation provider, some languages are better supported than others. That means your confidence level should not be the same across all prompts.

In my case, that pushed me toward a more defensive design: if text checks are incomplete, the rest of the safety system has to acknowledge that limitation instead of pretending the problem is solved.

Images changed the design

The biggest improvement came from treating image inputs as first-class moderation targets.

That sounds straightforward, but it changed several implementation details:

the moderation step now had to collect image URLs from different request fields
the backend needed one normalized moderation interface, even if the underlying provider had different APIs for text and image checks
moderation results had to return structured categories and scores, not just a single boolean
failure behavior had to be explicit

That last point matters more than it seems.

If a moderation provider fails, what should happen?

You have to choose between two imperfect options:

fail-open: allow the request and accept some risk
fail-closed: block the request and accept some false positives or degraded UX

There is no universal correct answer. It depends on the kind of product you are building, your abuse tolerance, and how costly a bad generation is for you. But the important part is to make the decision deliberately. Silent fallback logic is where safety systems get weak.

Provider-specific APIs should not leak everywhere

Another lesson was that moderation providers should be isolated behind a small internal interface.

Not because provider abstraction is fashionable, but because safety logic tends to spread if you let it.

If one route handler knows how text moderation works, another knows how image moderation works, and a third knows how to interpret provider-specific category names, you do not have a moderation layer anymore. You have moderation fragments.

I found it much cleaner to keep a moderation manager in the backend and let the generation route ask one question: “Is this request safe enough to proceed?”

That does not remove complexity. It contains it.

The practical takeaway

The most useful shift in my thinking was this:

Moderation is not a feature attached to generation. It is part of generation.

Once I started treating it that way, the backend became easier to evolve. I could add checks for both prompt text and image inputs, make blocking decisions before credits were consumed, and keep provider-specific moderation details out of the rest of the app.

I am using this approach while building videoflux.video, where one workflow needs to support AI image and video generation without assuming that a prompt alone tells the full safety story.

Disclosure: I’m the builder of videoflux.video.