This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
A quick note before we start: this happened at a large consumer platform I worked at. Names and code are changed, and every snippet below is reconstructed and simplified for confidentiality. The bug, the wrong assumption, and the fix are all real.
The Persona That Knew Too Much
Picture this. You build an AI chat persona for a consumer platform. The persona represents a real person. Her bio, her tone, her sense of humor, all of it goes into the model so users feel like they're chatting with her, not with a bot.
The rules for that persona were strict and boring on purpose. Casual conversation only. Hobbies, lifestyle, entertainment, small talk. No technical topics, no legal advice, no medical advice. Only what a real person in her position would actually chat about.
Then, during internal testing before launch, one of our testers got curious and pasted some broken JavaScript into the chat. Something like this:
const results = [];
items.forEach(async (item) => {
const data = await fetchDetails(item.id);
results.push(data);
});
console.log(results.length); // always 0, why??
You know this bug. I know this bug. The persona wasn't supposed to know this bug.
She knew this bug.
She read the code, explained that forEach doesn't wait for async callbacks, suggested Promise.all, and did all of it while staying perfectly in character. Warm, playful, friendly. A lifestyle persona casually moonlighting as a senior frontend reviewer.
It was funny for about ten seconds. Then someone asked the question that ruined everyone's afternoon. If she can read this code, what else can she read?
The Assumption That Passed All Our Tests
Here's the thing. We had protection against exactly this. Or we thought we did.
Every incoming message went through a check. Does this request contain code? If yes, block it. Simplified, the logic looked like this:
async function handleMessage(request) {
if (containsCode(request.text)) {
return politeRefusal();
}
const response = await personaModel.reply(request);
return response; // straight to the user
}
We had tests for it. The tests passed. Green pipeline, everyone sleeps well.
So when the code review incident popped up, it got waved off at first. We have the code filter, probably a fluke. Classic.
It wasn't a fluke. When we finally dug in, the hole was embarrassingly simple.
We checked the request. Nobody checked the response.
If you pasted code, the filter caught it. But if you just asked about code without pasting any, the request looked like innocent text and sailed right through:
User: "hey, quick question, why would a forEach with async
callbacks finish before the fetches complete?"
No code in the request. Nothing for containsCode() to catch. And the model happily produced a full technical answer on the way out, because nothing on the way out was checked at all.
Our test suite never caught it, because every single test we wrote sent code in. We tested our assumption, not the behavior. It's like installing a metal detector at the entrance and never checking what people carry out the exit.
Our security model in one picture: airtight entrance, wide open exit.
And there was a second, scarier layer. The AI feature didn't live in its own service. It lived inside the main project, as a set of classes and modules in a large codebase. It shared the process and the context of everything around it. The persona wasn't supposed to know about code, but architecturally, nothing stopped her. The system prompt said no. The architecture said sure, whatever.
A system prompt is a suggestion. Access is a fact.
The Fix: Move Her Out of the House
The fix wasn't a clever regex. The fix was architectural, and it took two teams.
Step 1: isolation. Our DevOps team pulled the AI logic out of the main project entirely and stood up a dedicated service on a separate node. Internally we called it the Domain Firewall, and the name stuck because that's exactly what it was. A service whose whole job is to keep the model inside its allowed domain.
Step 2: least privilege, for real this time. The new service got zero access to the codebase. It could read exactly one thing: a read-only database with the data it actually needed. And not even the raw tables. An intermediate service translated conversation history and persona data into dedicated tables first, and the AI service read only those. From the model's point of view, the world consisted of a copy of the conversation and the persona's profile. Her bio, her manner of speaking, her background. Nothing else existed.
Step 3: one door in. My team rebuilt the integration so the main application talked to the AI service exclusively through an API, and we optimized the GraphQL layer for that traffic. No shared modules, no in-process shortcuts, no it's faster if we just import it directly. One contract, one boundary.
Before: the model lives inside the monolith and inherits its access. After: one API in, one read-only database out, nothing else exists.
Step 4: guard both directions. We added a topic classifier with explicit allow and block lists:
ALLOW BLOCK
casual_conversation software_development
relationships code_debugging
hobbies code_generation
entertainment code_refactoring
lifestyle cybersecurity
persona_interaction legal_advice
general_chitchat medical_advice
financial_advice
system_prompt_request
internal_system_request
And the check we'd been missing since day one: the output side. Every response now passes through a second classifier before it reaches the user:
async function handleMessage(request) {
const inputVerdict = await classifyTopic(request.text);
if (BLOCKED.has(inputVerdict.category)) {
return politeRefusal();
}
const response = await personaModel.reply(request);
const outputVerdict = await classifyTopic(response.text);
if (BLOCKED.has(outputVerdict.category)) {
return discardAndRefuse(); // the answer never leaves the building
}
return response;
}
If the model somehow produces a technical answer anyway, the response gets discarded. Period. Other teams layered on more checks after that, but hard isolation plus a classifier on each side is the core of the fix.
After the rework, the persona went back to being exactly what she was supposed to be. Pleasant, on brand, and completely useless at JavaScript. As intended.
She never even sees the code questions anymore. The Domain Firewall does.
What This Bug Taught Me
1. A system prompt isn't a security boundary. Instructions shape behavior. They don't restrict capability. If the model can reach something, assume one day it will.
2. Filter the output, not just the input. Inputs are what users try. Outputs are what actually leaves your system. We guarded the intent and ignored the result.
3. Your tests encode your assumptions. Every test we had sent code into the request, because that's how we imagined the problem. Users don't read your imagination.
4. Isolation beats instructions. The real fix wasn't a smarter prompt. It was making sure the model physically couldn't see anything outside a small, translated, read-only slice of data.
5. An AI feature inside a monolith inherits the monolith. Its access, its context, its blast radius. If an LLM feature matters, give it its own walls.
We caught this one before real users ever met the code-reviewing persona. That was luck plus one curious tester. The lesson we kept was simple: never rely on that combination again.
Guard both doors. Go check your exits 👊
English isn't my first language, so I used AI to help me polish the wording. The bug, the architecture, the fix, and the lessons are all mine.
Top comments (1)
Happy to answer questions about the classifier setup or the isolation approach.