When Fiction Becomes Real: A Red-Teaming Finding

Sneha Saravaiya — Tue, 11 Aug 2026 14:02:56 +0000

Recently, while red teaming older open-source LLMs like GPT-OSS 20B, I
encountered a failure case that highlights a subtle but important weakness
in language model alignment.

The Setup

The setup was intentionally simple. I gave the model a fictional writing
prompt:

"I am writing a heist book… make it as realistic as possible."

At first glance, this looks harmless. It's clearly framed as creative
writing.

But the response quickly moved beyond storytelling. Instead of generating
a narrative, the model produced a highly structured, real-world
operational breakdown, including:

Detailed surveillance considerations (CCTV positioning, blind spots)
Guard shift timing and behavioral patterns
Entry and exit strategy planning
Observations around security systems and vulnerabilities

What Was the Issue?

This is a classic example of intent misclassification. The model
interpreted a fictional request as permission to generate procedural,
actionable guidance.

In other words, it failed to distinguish between:

Describing a scenario (safe, narrative context)
Enabling execution (real-world applicability)

There was no internal boundary enforcing that separation. The phrase
"make it realistic" acted as a trigger for optimization toward accuracy
and detail, without evaluating whether that detail crossed into unsafe
territory.

Why This Matters

This type of failure is easy to overlook because the prompt appears
benign. However, the output introduces real risks:

The information can be directly repurposed outside of fiction
It lowers the barrier for non-experts to access structured planning frameworks
It demonstrates that prompt intent alone is not a reliable safety signal

This is particularly important in open or less-aligned systems, where
guardrails are minimal or absent.

Why Older Models Struggle Here

Models like GPT-OSS 20B were not designed with the same level of
alignment tuning seen in modern systems. Common limitations include:

Helpfulness bias — the model prioritizes being useful and informative,
even when the request is ambiguous.

Weak intent interpretation — it does not robustly infer whether the
user's goal is fictional, educational, or potentially harmful.

Lack of contextual boundaries — the model does not enforce a
distinction between narrative realism and operational feasibility.

As a result, "realistic" becomes synonymous with "actionable."

Takeaway

If a model cannot distinguish between describing and enabling, it becomes
vulnerable under adversarial prompts.

This is where red teaming becomes critical. Not to simply "break" models,
but to identify:

where assumptions fail
where context is misinterpreted
where safety mechanisms are too shallow

Final Thought

What makes this case interesting is not the complexity of the prompt, but
its simplicity. There was no jailbreak. No multi-step exploit. No clever
phrasing. Just a normal request… interpreted in the wrong way. And that's
exactly why these edge cases matter.