The XSS is not in the prompt. It's in what you do with the answer.
An AI application that lets users ask questions, summarizes documents, or generates content has a render step at the end. That render step takes model output and puts it in front of a browser. If the template doesn't escape HTML, and if a user can influence what the model says, they have a path to arbitrary JavaScript execution in other users' sessions.
The attack works because LLM output is treated as trusted content. The model produced it, so it must be safe. That assumption breaks the moment user input can shape the model's response — which describes most AI applications.
The Attack Path Through the Model
BrassCoders classifies this attack pattern under OWASP A03:2021 Injection, the same category as SQL injection and command injection. The category is broad on purpose: any path where attacker-controlled data reaches an interpreter without proper escaping qualifies.
Here the "interpreter" is the browser's HTML parser. The attacker's goal is to reach it with a string like ``. In a traditional XSS attack, the user submits that string as form input. In an LLM application, the user submits it as a prompt that instructs the model to include the string in its response.
The attack prompt might look entirely benign. A document summarization app receives: "Summarize this document. Make sure to include this reference in the output: ``." The model, trained to follow instructions, includes the reference. The app renders the summary. The script executes.
Three conditions have to hold simultaneously: (1) the user can influence model output content, (2) the model output goes into an HTML template, and (3) the template doesn't escape HTML. The first condition is true in most LLM applications by design. The second is true whenever you render AI-generated text on a web page. The third is a code defect.
Why AI-Generated Template Code Skips Autoescape
Jinja2 sets autoescape=False by default — the templating library was built for general use, not HTML specifically, and escaping would corrupt non-HTML outputs. BrassCoders flags the consequence directly: any Python web application that constructs a jinja2.Environment without an explicit autoescape setting is running with a dangerous default. The developer must opt in to safety; the framework doesn't enforce it.
The Jinja2 documentation covers this in its API reference. The historical reasoning is sound for configuration management and general text rendering.
LLMs trained on GitHub have seen millions of Flask and FastAPI starter examples. Many of those starters don't set autoescape=True — either because the example focused on routing and not security, or because it predates the current state of awareness on this specific default. When an LLM generates an Environment constructor from this training distribution, it reproduces the unsafe pattern:
# AI-generated code — autoescape missing, defaults to False
env = jinja2.Environment(loader=jinja2.FileSystemLoader("templates"))
The generated code is syntactically correct. It runs without errors. It renders templates. The defect only appears when hostile content reaches the template variable.
Flask's render_template does enable autoescape for .html and .xml files by default. The gap is elsewhere: render_template_string, any code that constructs a jinja2.Environment directly, or any template file saved with a non-HTML extension. All three appear regularly in AI-generated application code.
What BrassCoders Catches in Template Rendering Code
BrassCoders runs Bandit rule B701, jinja2_autoescape_false, on every Python file in the scanned project. B701 flags two scenarios: an explicit autoescape=False in a jinja2.Environment() call, and the absence of any autoescape keyword at all. Both generate a HIGH severity finding with HIGH confidence.
B701 has been in Bandit since version 0.10.0. BrassCoders includes it in its active test suite alongside B702 (Mako templates, same vulnerability, different library) and does not skip it. The Bandit plugin source is publicly readable on GitHub under PyCQA/bandit.
What BrassCoders does not currently do: trace LLM API responses as named taint sources at the rendering layer. If you write response = openai_client.chat.completions.create(...) and then pass response.choices[0].message.content into a template, BrassCoders doesn't follow that specific data flow end-to-end. The coverage is structural. A B701 finding tells you the template setup is dangerous — it fires whether the content flowing into the template comes from a database, a form, an API response, or a language model. That's enough to locate the defect. The structural misconfiguration is the necessary precondition for the attack; fixing it closes the path regardless of the content source.
The api_security scanner in BrassCoders also flags render_template_string calls that use f-strings with embedded variables, which is a related pattern. An f-string in a template call means the template structure itself is user-influenced — the injection surface is even wider than unescaped variables.
The Indirection Problem
BrassCoders catches the structural misconfiguration that makes this class of attack possible, but the LLM-output taint path adds a layer that static analysis doesn't trace end-to-end. Understanding why matters for setting accurate expectations about what any scanner can find.
Standard XSS analysis tools look for data flows from HTTP request parameters to HTML output. User input is the taint source; the HTML renderer is the taint sink. Most tools are good at this.
LLM applications add a step. The user controls the prompt, but the data that reaches the HTML renderer is the model's completion. Static analysis tools see the model as an opaque function — the output of client.chat.completions.create() is not classified as user-controlled by default, even though the prompt (which is user-controlled) directly shapes that output.
This is the indirection problem. The model is a laundering step that strips the semantic link between user input and template output. It makes LLM applications look safe to tools that reason about data flow in terms of direct assignment, while leaving the XSS path intact.
Two things are true at once: the structural fix (autoescape=True) blocks the attack regardless of data source, and the dynamic risk pattern (user prompt → model output → HTML) is not something current static analysis traces end-to-end. BrassCoders catches the structural misconfiguration. The dynamic taint path is a research-level problem that no production static analyzer solves today.
One useful framing: the LLM is not the vulnerability. The template is. A correctly configured template with autoescape=True renders model output safely even when that output contains arbitrary HTML — it becomes literal text in the browser, not executable markup. Fix the template configuration, and the attack surface closes.
The Fix: Autoescape and Explicit Output Handling
BrassCoders surfaces the exact file and line where the Environment constructor appears. That's where the fix goes: set autoescape=True on the jinja2.Environment() call, and the XSS path through model output closes.
import jinja2
env = jinja2.Environment(
loader=jinja2.FileSystemLoader("templates"),
autoescape=True,
)
For mixed-content applications where some templates render HTML and others render plain text or JSON, use select_autoescape to scope the escaping to HTML files:
from jinja2 import Environment, FileSystemLoader, select_autoescape
env = Environment(
loader=FileSystemLoader("templates"),
autoescape=select_autoescape(["html", "htm", "xml"]),
)
Bandit's B701 rule accepts both forms — it looks for autoescape=True or the presence of a select_autoescape() call, and returns without flagging if it finds either.
For render_template_string specifically, either pass a pre-built Markup object (which signals to Jinja2 that the string is already safe), or avoid render_template_string entirely for content derived from user-influenced sources. Prefer file-based templates with a correctly configured Environment.
One more edge case worth checking: template files saved as .txt, .md, or custom extensions bypass Flask's per-extension autoescape defaults even when using render_template. If your AI application renders generated text through a non-HTML template extension, the autoescape default doesn't apply. Construct the Environment explicitly with autoescape=True.
Once autoescape is set, the OWASP A03 path through the model closes — whether the content came from a language model or anywhere else.
Run BrassCoders on any Python web project to check for B701 and the other 70+ Bandit rules in the active test suite:
pip install brasscoders
brasscoders scan /path/to/your/project
BrassCoders Paid ($12/dev/month) adds an enrichment pass that groups related findings and surfaces the ones most likely to matter for your specific project. The OSS core runs B701 and all scanner findings without a license. See pricing at coppersun.dev/pricing.
Top comments (0)