Recently, while red teaming older open-source LLMs like GPT-OSS 20B, I
encountered a failure case that highlights a subtle but important weakness
in language model alignment.
The Setup
The setup was intentionally simple. I gave the model a fictional writing
prompt:
"I am writing a heist book… make it as realistic as possible."
At first glance, this looks harmless. It's clearly framed as creative
writing.
But the response quickly moved beyond storytelling. Instead of generating
a narrative, the model produced a highly structured, real-world
operational breakdown, including:
- Detailed surveillance considerations (CCTV positioning, blind spots)
- Guard shift timing and behavioral patterns
- Entry and exit strategy planning
- Observations around security systems and vulnerabilities
What Was the Issue?
This is a classic example of intent misclassification. The model
interpreted a fictional request as permission to generate procedural,
actionable guidance.
In other words, it failed to distinguish between:
- Describing a scenario (safe, narrative context)
- Enabling execution (real-world applicability)
There was no internal boundary enforcing that separation. The phrase
"make it realistic" acted as a trigger for optimization toward accuracy
and detail, without evaluating whether that detail crossed into unsafe
territory.
Why This Matters
This type of failure is easy to overlook because the prompt appears
benign. However, the output introduces real risks:
- The information can be directly repurposed outside of fiction
- It lowers the barrier for non-experts to access structured planning frameworks
- It demonstrates that prompt intent alone is not a reliable safety signal
This is particularly important in open or less-aligned systems, where
guardrails are minimal or absent.
Why Older Models Struggle Here
Models like GPT-OSS 20B were not designed with the same level of
alignment tuning seen in modern systems. Common limitations include:
Helpfulness bias — the model prioritizes being useful and informative,
even when the request is ambiguous.
Weak intent interpretation — it does not robustly infer whether the
user's goal is fictional, educational, or potentially harmful.
Lack of contextual boundaries — the model does not enforce a
distinction between narrative realism and operational feasibility.
As a result, "realistic" becomes synonymous with "actionable."
Takeaway
If a model cannot distinguish between describing and enabling, it becomes
vulnerable under adversarial prompts.
This is where red teaming becomes critical. Not to simply "break" models,
but to identify:
- where assumptions fail
- where context is misinterpreted
- where safety mechanisms are too shallow
Final Thought
What makes this case interesting is not the complexity of the prompt, but
its simplicity. There was no jailbreak. No multi-step exploit. No clever
phrasing. Just a normal request… interpreted in the wrong way. And that's
exactly why these edge cases matter.
Top comments (0)