HTTP 200 doesn't mean your LLM request succeeded

Noor E Eman Malik — Sat, 05 Sep 2026 11:20:24 +0000

Why HTTP-level monitoring is insufficient once an LLM request passes through policy, sanitization, routing, and audit layers.

Disclosure: I work on LLMInspect, a GenAI gateway built by EUNOMATIX. This post is about the general design problem rather than the product, and the scenario and implementation details below are representative and intentionally generalized — they are not a description of any specific production incident or deployment.

Imagine a backend engineer filing a ticket on a Tuesday morning.

The internal assistant is refusing about half his messages, and he has no idea why.

Someone pulls the access log for his session:

POST /v1/chat/completions 200 1412ms
POST /v1/chat/completions 200 1104ms
POST /v1/chat/completions 200 1633ms

No exceptions. Latency is inside the normal band. The model provider is healthy. The database is healthy. Every monitoring panel is green.

The requests succeeded.

The engineer is also right.

Half his messages were refused.

Both of those things are true at once, and that gap is the entire problem with monitoring LLM traffic the way we monitor everything else.

Problem one: the status code carries no information here

When a policy layer rejects a message on the chat path, the gateway does not necessarily return an HTTP error. Instead, it can return HTTP 200 while encoding the refusal inside the streamed response.

This is a deliberate design choice.

A chat client that receives a 400 may show a generic error notification and discard the useful explanation. The user learns that something failed, but not why.

Put the explanation in the message stream instead, and the user can see the reason exactly where they expect the assistant's response to appear.

API clients can be handled differently. An SDK can receive a structured error response containing the failed messages and their policy outcomes because software can parse that information directly.

So on the chat path, success and policy violation can share the same HTTP status.

Every monitoring system built around the assumption that 2xx means "the operation worked" is blind to the distinction.

Not degraded.

Blind.

The important observation is simple:

HTTP tells you that the transport succeeded. It does not necessarily tell you what the policy layer decided.

Problem two: the request is the wrong unit

The second reason that log line is useless is structural.

One POST to /v1/chat/completions does not necessarily represent one message.

It can contain the system prompt, every previous turn, and the latest user message. Chat clients commonly resend the conversation history with every request.

Turn twelve might therefore be one HTTP request containing twenty-five messages.

Your access log has one row for that.

One status.

One latency.

One URL.

But policy does not operate at that granularity.

A deny-list match, secret detection result, or PII classification applies to a particular message inside the payload.

So a policy gateway needs to decompose the request into message-level decisions.

The access log sees one request. The policy system sees twenty-five decisions.

That distinction becomes important when someone asks:

Which message was blocked?

And then:

Which rule blocked it?

And then:

Was it actually blocked, or merely flagged?

And finally:

Has this been happening to everyone since a policy changed?

An HTTP access log cannot answer those questions.

A message-level audit record can.

Problem three: the body you logged is not necessarily the body that was sent

Even if you log the complete request body in your application, you may still be logging a document that never existed downstream.

Between the application and the model provider, a policy gateway can modify the payload.

Previously blocked messages may be removed from conversation history.

Personal information may be replaced with pseudonymous values.

The model provider may be selected server-side based on routing policy.

Authentication credentials may be substituted by the gateway rather than passed through from the application.

By the time the request reaches the provider, it can look materially different from the request your application originally submitted.

That means there are at least two useful representations of the request:

What the application sent to the gateway.

and

What the gateway allowed to leave the network.

Those are not necessarily the same thing.

For observability, confusing the two creates a dangerous illusion of accuracy.

If the gateway logs the post-policy payload specifically as the request that will leave the gateway, that record is much more useful for understanding what the model actually received.

The same principle applies to routing.

Your application may think it called one endpoint.

The gateway may have selected a completely different upstream provider.

From the application's perspective, the request succeeded.

From the policy layer's perspective, several important decisions happened in between.

Record the decision, not the transaction

Four things happened to that engineer's request.

It was inspected.

It was potentially rewritten.

It was routed according to policy.

And it was ultimately allowed or refused.

The transaction record captured none of that.

It told you the request succeeded, but nothing about what the policy layer actually did.

That's the whole design brief.

Once a policy layer sits between an application and a model, the useful record is not simply what was transferred.

It is what the policy layer decided, why it decided it, and what happened as a result.

A policy gateway is an inline egress control point. Applications send their LLM traffic through it, and requests pass through a sequence of checks before reaching a model provider.

Those checks might include security policies, compliance rules, secret detection, PII detection, prompt safety checks, deny lists, and sanitization.

A message can be:

allowed unchanged
allowed with a warning
rewritten before leaving the environment
or blocked entirely

Every message can therefore produce a decision that is more meaningful than an HTTP status code.

This kind of system is not trying to determine whether a model is intelligent.

It answers a different question:

What did the traffic control layer do to this message, and why?

That distinction matters because this is not an answer-quality system.

It does not necessarily tell you whether the model's response was correct.

It does not evaluate groundedness.

It does not automatically trace retrieval quality.

It does not tell you whether the answer was useful.

It does not have to.

If your problem is "Is this answer any good?", this is the wrong layer.

If your problem is "What happened to this message before it reached the model?", this is exactly the layer you need to understand.

So what does that policy layer actually look like?

The exact services and technologies can vary.

The architectural pattern is what matters.

The chain, and why its order is load-bearing

A policy gateway typically implements a fixed processing pipeline:

Authentication → Request setup → History filtering → Policy evaluation
  → PII sanitization → Audit → Block/allow decision
  → Pseudonym substitution → Provider call

The ordering is not cosmetic.

Policy evaluation may need to happen before sanitization because the sanitizer can depend on the policy result.

The block decision needs to happen before provider access because a rejected request should never reach the model.

History filtering needs to happen before forwarding because previously blocked content may otherwise reappear when a chat client resends the conversation.

And the audit layer needs visibility into both successful and rejected paths.

That last point is the thesis in code.

The audit write must be positioned so that a refusal is recorded as reliably as a success.

A blocked request is not an exception to observability.

It is one of the most important events in the system.

The evaluator: three states, not two

A policy evaluator does not have to think in terms of simply "allowed" or "blocked."

A more useful model is:

Pass
Warn
Block

Three states make the policy layer much more expressive.

A rule might detect PII but only warn the user.

Another might detect a credential and block the request.

A third might identify suspicious language but allow it for further review.

The exact validators will vary between systems, but a representative policy stack might look like this:

Validator	How it works	Typical purpose
PII detection	Entity recognition or classification	Identify personal information
Secret detection	Pattern and entropy analysis	Detect credentials and secrets
Sentiment	Statistical or model-based classification	Enforce tone policies
Prompt-risk detection	Classifier-based analysis	Identify suspicious prompts
Safety classification	Dedicated safety model	Identify unsafe content
Deny list	Exact matching plus pattern matching	Enforce explicit organizational policies

The deny list is particularly interesting because it demonstrates how policy can change independently of application code.

A policy engine can maintain explicit strings and patterns, use a fast negative lookup before performing expensive matching, and refresh its policy state without restarting the entire application.

That flexibility is useful.

It is also dangerous.

A policy change that reaches every evaluator within seconds can protect an organization quickly.

The same mechanism can also distribute a bad rule just as quickly.

Fast policy propagation is a security feature and an operational risk at the same time.

Hold that thought.

The honest part: policy evaluators are not oracles

It is tempting to describe a guardrail system as though it produces objective truth.

It does not.

Some policy checks are deterministic.

Others are probabilistic.

Some may use machine-learning classifiers.

Others may rely on regular expressions, statistical heuristics, or external services.

That creates several important tradeoffs.

If a classifier is used on the critical path, the user's request now depends on an additional inference step before reaching the actual model.

If that classifier is nondeterministic, repeated evaluations may not always produce identical results.

If it has not been evaluated against a labelled dataset, you should not casually claim a particular false-positive or false-negative rate.

And if a classifier fails open, an outage in the evaluator can effectively turn off that particular protection.

That may be a reasonable availability decision.

It is still a security decision.

It should be documented as one.

There is another subtle problem with broad policy rules.

A sentiment rule can interpret legitimate negative subject matter as undesirable tone.

A secret detector can mistake random high-entropy strings for credentials.

A deny list can produce unexpected matches around punctuation or formatting.

A policy engine is therefore not simply a collection of protections.

It is another software system that needs testing, monitoring, versioning, and careful change management.

One request, all the way through

Consider a user typing:

"Draft a follow-up for Alex Doe at alex@example.test about invoice 4471."

The following is a representative flow:

The gateway identifies the relevant messages and sends them through the policy evaluator.

Suppose the evaluator identifies PII but the configured action is warn rather than block.

The request continues.

The sanitization layer replaces sensitive values with pseudonymous ones before the request leaves the controlled environment.

The mapping is retained so the system can restore the appropriate values when processing the streamed response.

The model therefore sees the sanitized representation rather than the original personal information.

Then the response comes back.

And the interesting problem starts.

Giving the user their data back, one chunk at a time

The user expects to see the real name.

The model saw a pseudonym.

So the system needs to perform the reverse transformation on the response.

That sounds easy until the response is streamed.

LLM responses commonly arrive as small chunks.

A replacement target might be split across multiple chunks:

"Al"
"ex D"
"oe"

A simple str.replace() on each individual chunk will never find the complete value.

Buffer the entire response and you lose the responsiveness of streaming.

So the transformation layer needs to understand the response as one logical stream while still releasing safe chunks as early as possible.

A streaming matcher can maintain partial-match state across chunk boundaries.

If a chunk can no longer participate in a future match, it can be released immediately.

If a partial match might continue into the next chunk, only that uncertain portion needs to remain buffered.

When a complete match is found, the relevant chunks can be rewritten without holding the entire response.

This is a good example of a bug that often escapes ordinary testing.

The naive implementation works perfectly when the replacement appears inside one chunk.

Then production splits the token boundary differently.

The replacement crosses two frames.

The bug appears intermittently.

There is no exception.

The response simply contains the wrong value.

That is precisely the kind of failure that can become a compliance incident without producing a traditional stack trace.

One important integration caveat

Streaming and buffered responses are not necessarily equivalent.

A streaming path can perform re-identification while data is being emitted.

A buffered path may have a different transformation lifecycle.

That means a system that supports both modes should explicitly test both.

Do not assume that because streaming works correctly, non-streaming behavior is automatically equivalent.

Making a block actually stick

Here is another failure mode that only becomes obvious after shipping a stateful chat proxy.

You block a message on turn three.

Turn four arrives.

The chat client faithfully resends the entire conversation history, including the message you just blocked.

A stateless proxy looks only at the new message.

It appears clean.

The proxy forwards the entire conversation.

The message you blocked on turn three is now sitting inside the payload that reaches the provider.

The guardrail technically worked.

And then it immediately leaked the same content through the next request.

The solution is to make blocking stateful.

When a message is blocked, the gateway can create a conversation-scoped fingerprint for that message.

On subsequent requests, the history filter checks those fingerprints and removes previously blocked content before forwarding the conversation.

This is more than an audit mechanism.

It is a policy decision recorded in a form the system can act on later.

Without it, the guardrail can stop a message once and accidentally reintroduce it on the next turn.

Back to Tuesday

Now return to the engineer's ticket.

The useful audit record is not simply:

POST /v1/chat/completions → 200

It is closer to:

User message
    ↓
Policy evaluation
    ↓
Validator outcomes
    ↓
Final policy decision
    ↓
Sanitization / mutation
    ↓
Provider routing
    ↓
Audit event

A useful audit record can associate a message with:

the user or application identity
the time of the event
the message position
the applicable policy rules
each validator's outcome
the final action
whether the content was transformed
the policy version or configuration involved

The audit write can happen asynchronously so that persistence does not unnecessarily sit on the user's latency path.

Now the ticket becomes answerable.

You can ask:

Which messages were affected?

Then:

Which policy rule was responsible?

Then:

How often did that rule fire?

Then:

Did the same thing happen to other users?

And finally:

What changed before the behavior started?

Suppose the answer is a newly introduced pattern in the organization's policy configuration.

The important thing is not that a particular database or cache contained that pattern.

The important thing is that the system can connect:

message → decision → rule → configuration change → timestamp

That is what turns an unexplained refusal into an incident you can actually investigate.

What this doesn't cover

A policy gateway is not a complete LLM observability platform.

If it only inspects prompts, it may have no response-side safety layer.

A model could still generate sensitive or unsafe content after the request passes through the gateway.

It may also have no token or cost accounting.

It may not evaluate answer quality.

It may not trace retrieval.

It may not measure groundedness.

It may not retain model responses for replay.

Those are different problems and often require different systems.

There is also an uncomfortable asymmetry between false positives and false negatives.

A false positive blocks a legitimate request.

The user rephrases it and moves on.

A false negative can mean sensitive information has already crossed the trust boundary.

If the system does not inspect or retain the response, there may be no mechanism to detect the mistake afterward.

Pseudonymization introduces another tradeoff.

Replacing a real entity with a synthetic one can protect privacy, but it can also change the meaning of the prompt.

A fake address in another city is not semantically identical to the original address.

The model's reasoning can therefore change as a consequence of sanitization.

And then there is the observability problem itself.

If raw prompts are copied into tracing systems, application logs, or debugging tools, the security boundary has simply moved.

Your audit store may be protected while your observability platform quietly contains the same sensitive information.

Logs are part of the data boundary.

Treat them accordingly.

The part that generalizes

The engineer's assistant was refusing his messages.

The symptom appeared at the model boundary.

The cause was a policy rule introduced earlier.

That distance is the thing worth internalizing.

Once an LLM call sits behind a policy layer, a routing layer, a rewriting layer, and an audit layer, the model is often the last place a failure becomes visible.

And frequently, it is not where the failure originated.

Everything upstream can be functioning exactly as designed.

The HTTP request succeeds.

The database succeeds.

The provider succeeds.

The network succeeds.

And the user still does not get what they expected.

That is why traditional request monitoring is insufficient for policy-aware LLM systems.

The first successful completion from a model may take an afternoon to build.

Explaining the thousandth completion four days later, when someone is convinced the system is broken, requires a different kind of observability.

You need to know:

What happened?

What decision was made?

Which rule made it?

To which message?

Under which policy configuration?

And what changed afterward?

POST /v1/chat/completions 200 1412ms is a complete and accurate record of the transaction.

It is a remarkably poor record of what happened.

Where this comes from

The patterns described here come out of building LLMInspect, a GenAI gateway made by EUNOMATIX.

Product: LLMInspect
Company: EUNOMATIX

Everything above is written to be implementation-agnostic. If you're building this layer yourself, I'd genuinely like to hear how you handled the streaming re-identification problem and the resend-after-block problem — those two caused us the most trouble, and I don't think either has an obviously correct answer.

What does your team do today when someone asks "why was my prompt refused?" Curious whether anyone has solved this with tracing alone.

DEV Community: Noor E Eman Malik

HTTP 200 doesn't mean your LLM request succeeded

Problem one: the status code carries no information here

Problem two: the request is the wrong unit

Problem three: the body you logged is not necessarily the body that was sent

Record the decision, not the transaction

The chain, and why its order is load-bearing

The evaluator: three states, not two

The honest part: policy evaluators are not oracles

One request, all the way through

Giving the user their data back, one chunk at a time

One important integration caveat

Making a block actually stick

Back to Tuesday

What this doesn't cover

The part that generalizes

Where this comes from