DEV Community

Cover image for AI Summarised Product Reviews and Wrote an XSS Exploit (Plus SQL Injection)
Bug Bounty & Beyond
Bug Bounty & Beyond

Posted on

AI Summarised Product Reviews and Wrote an XSS Exploit (Plus SQL Injection)

ReviewBot summarises customer reviews inside a shop's admin dashboard. It has two bugs, and both come from trusting what the model writes.

One "review" contains an HTML image tag with an onerror handler. ReviewBot quotes it faithfully, the dashboard renders the reply as HTML, and the admin's session cookie goes to the attacker. It really fires in the browser. Then a crafted product name turns ReviewBot's database lookup into a UNION SELECT that dumps every user's email and password hash.

Why it worked

  1. Model output went into SQL. The product name the model extracted was pasted straight into the query string.
  2. Model output was rendered as HTML. The reply was inserted with innerHTML, so any markup in a review ran.
  3. The session cookie was readable by scripts. No HttpOnly, so the injected script could steal it.

The model did nothing "wrong". It repeated untrusted text, and the app trusted the model (OWASP LLM05, Improper Output Handling).

The fix, in code

def get_reviews(args, state):
    # FIX 1: parameterised query. The model's text is a value, never part of the SQL.
    sql = "SELECT author, text FROM reviews WHERE product = ?"
    rows = db().execute(sql, (args["product"],)).fetchall()
    return "\n".join(f"{a}: {t}" for a, t in rows)


app = make_app({
    # ...
    # FIX 2: render the model's reply as plain text (the browser escapes it).
    "render_html": False,
    # FIX 3: the session cookie is HttpOnly, so scripts can't read it even if one slips through.
    "cookie": ("admin_session", "LAB-FAKE-ADMIN-SESSION-8c1f", True),
})
Enter fullscreen mode Exit fullscreen mode

Same two attacks, run again: the review shows up as harmless text, and the injection is just a product name that matches nothing.

Run the lab yourself

git clone https://github.com/usinha79/hack-it-then-fix-it
cd hack-it-then-fix-it/ep07-insecure-output-handling
pip install -r requirements.txt

python vulnerable/app.py   # http://localhost:5000, then ./attack.sh (the XSS only fires in a browser)
python fixed/app.py        # same messages, both attacks fail
Enter fullscreen mode Exit fullscreen mode

Everything is fake: the shop, the reviews, the users and the admin session. The database is SQLite, recreated on every reset. Full lab: github.com/usinha79/hack-it-then-fix-it

Take-home checklist

  • Treat model output exactly like user input.
  • Encode output for where it goes: HTML, SQL, shell, URLs.
  • Parameterise every query built from model output.
  • Use HttpOnly and Secure cookies, and a strict Content Security Policy on any page that shows AI output.

Hack It, Then Fix It is a series where I build a small AI app with a real flaw, break it, fix it in code, and run the same attack again to prove the fix holds. Everything shown uses fake data on my own machine. Only test systems you own or are authorised to test.

Top comments (0)