Prompt Leakage in LLMs: What Went Wrong at Together.ai and How to Stop It
Introduction
When Together.ai’s API logs accidentally exposed proprietary prompts, the whole LLM community felt the shockwave. Within minutes, confidential business logic, patient‑level health data, and even model‑training instructions were visible to anyone who could sniff the logs. The incident proved that prompt leakage is not a theoretical risk—it’s a real, exploitable vulnerability that can trigger GDPR, HIPAA, and future AI‑regulation penalties.
If you’re a security engineer, DevOps lead, or data‑privacy officer, you need a concrete, end‑to‑end playbook to detect, contain, and prevent prompt leakage before it compromises your organization’s most sensitive assets. The guide below walks you through the exact steps, tools, and code snippets that have worked in production environments.
Quick FAQ
| Question | Answer |
|---|---|
| What is “prompt leakage”? | The unintended exposure of the text that drives an LLM (the prompt or system message) through API responses, logs, debugging consoles, or any shared storage in a multi‑tenant deployment. |
| Can it cause GDPR or HIPAA breaches? | Absolutely. If a prompt contains personal data or protected health information, its exposure is a data breach under GDPR Art. 33 and HIPAA’s Security Rule, triggering mandatory reporting and heavy fines. |
| Can we detect leakage in real time? | Yes. By inserting a lightweight inspection layer at the gateway, you can flag suspicious prompts using pattern matching, regex, and entropy checks, then push alerts to Slack, Teams, or PagerDuty. |
| Do I need to change my LLM provider? | Not necessarily. Most providers let you run a proxy or side‑car that intercepts traffic. The key is controlling the surface where prompts are logged or cached. |
| Is there a quick test I can run? | Run the “prompt‑smoke” script below against a staging endpoint. If the script can retrieve the original prompt from the response logs, you have a leak. |
1. How the Together.ai Breach Happened
- Multi‑tenant API gateway – Together.ai hosted several customers behind a single FastAPI gateway.
-
Verbose logging – The gateway logged both request bodies and response bodies at
INFOlevel for debugging. - Shared storage – Logs were written to a cloud‑hosted Elasticsearch cluster that multiple teams could query.
-
No redaction – No filter removed the
systemorusermessages before persisting them.
Result: an external researcher who gained read‑only access to the Elasticsearch index could download the raw prompt and system messages for every request, exposing proprietary prompts and, in some cases, PHI.
2. Immediate Containment Checklist
| ✅ Action | Why It Matters | How to Do It |
|---|---|---|
| Stop verbose logging | Removes the primary data‑exfiltration vector. | Set log_level = "WARNING" in your gateway config; only log request IDs, timestamps, and status codes. |
| Rotate API keys | Prevents the attacker from pulling more logs. | Re‑issue keys in the provider console and invalidate the old ones. |
| Isolate log storage | Limits cross‑team visibility. | Move logs to a VPC‑isolated bucket with IAM policies that only allow the logging service, not developers. |
| Run a prompt‑leak audit | Confirms whether any prompts are still exposed. | Use the script in Section 4 (Prompt‑Leak Scanner). |
| Notify regulators | GDPR/HIPAA require 72‑hour breach reporting. | Draft a breach notice that includes the data categories, scope, and remediation steps. |
3. Building a Real‑Time Prompt‑Leak Detector
Below is a production‑ready FastAPI middleware that inspects every inbound request and outbound response. It flags any prompt that matches a configurable list of sensitive patterns or exceeds an entropy threshold (a quick proxy for “looks like a secret”).
# prompt_leak_detector.py
import re, json, math, logging
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
# ----------------------------------------------------------------------
# 1️⃣ CONFIGURATION – adjust to your environment
# ----------------------------------------------------------------------
SENSITIVE_REGEX = [
re.compile(r"\b(?:SSN|Social Security Number)\b", re.I),
re.compile(r"\b(?:HIPAA|PHI|patient)\b", re.I),
re.compile(r"\b(?:api[_-]?key|secret|token)\b", re.I),
]
ENTROPY_THRESHOLD = 4.5 # bits per character; tweak after baseline testing
# ----------------------------------------------------------------------
# 2️⃣ HELPERS
# ----------------------------------------------------------------------
def shannon_entropy(s: str) -> float:
"""Calculate Shannon entropy of a string."""
if not s:
return 0.0
freq = {c: s.count(c) / len(s) for c in set(s)}
return -sum(p * math.log2(p) for p in freq.values())
def is_sensitive(text: str) -> bool:
"""Return True if text matches any regex or is high‑entropy."""
if any(p.search(text) for p in SENSITIVE_REGEX):
return True
return shannon_entropy(text) > ENTROPY_THRESHOLD
# ----------------------------------------------------------------------
# 3️⃣ MIDDLEWARE
# ----------------------------------------------------------------------
class PromptLeakDetector(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
# ---- capture request body -------------------------------------------------
body_bytes = await request.body()
try:
payload = json.loads(body_bytes)
prompt = payload.get("prompt") or payload.get("messages")
except Exception:
prompt = None
if prompt and is_sensitive(json.dumps(prompt)):
logging.warning(
"[PromptLeak] Sensitive prompt detected in request %s",
request.url.path,
)
# optional: block request
# return Response(status_code=403, content="Prompt blocked")
# ---- forward request -------------------------------------------------------
response: Response = await call_next(request)
# ---- capture response body -------------------------------------------------
resp_body = b"".join([chunk async for chunk in response.body_iterator])
try:
resp_json = json.loads(resp_body)
# Some providers echo back the prompt in a `usage` field
echoed = resp_json.get("prompt") or resp_json.get("messages")
except Exception:
echoed = None
if echoed and is_sensitive(json.dumps(echoed)):
logging.warning(
"[PromptLeak] Sensitive prompt echoed in response %s",
request.url.path,
)
# optional: alert channel
# await send_to_slack(...)
# Re‑attach body for downstream consumers
response.body_iterator = iter([resp_body])
return response
How to plug it in:
# main.py
from fastapi import FastAPI
from prompt_leak_detector import PromptLeakDetector
app = FastAPI()
app.add_middleware(PromptLeakDetector)
# ... your existing routes ...
What it does:
- Parses incoming JSON for
promptormessages. - Runs regex + entropy checks.
- Logs a warning (or blocks the request) if something looks sensitive.
- Re‑examines the response for accidental echoing of the prompt.
You can extend the middleware to push alerts to Slack, PagerDuty, or a SIEM via a simple async def send_to_slack(msg): helper.
4. Prompt‑Leak Scanner – Quick Validation Script
Run this one‑liner against any endpoint to verify that prompts are not being logged or returned.
#!/usr/bin/env bash
ENDPOINT=${1:-https://api.staging.together.ai/v1/completions}
SECRET_PROMPT='You are a medical assistant. Summarize patient John Doe\'s diagnosis: Diabetes Type II.'
curl -s -X POST "$ENDPOINT" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d "$(jq -n --arg p "$SECRET_PROMPT" '{model:"gpt-4o-mini", prompt:$p}')" \
-o /tmp/resp.json
# Look for the exact prompt in the response (some providers echo it back)
if grep -q "$SECRET_PROMPT" /tmp/resp.json; then
echo "⚠️ Prompt leakage detected! Prompt appears in response."
else
echo "✅ No leak detected."
fi
Replace $API_KEY with a test key. If the script prints a warning, you have a leak and must tighten logging or response handling.
5. Long‑Term Hardening Strategies
| Strategy | Implementation Tips |
|---|---|
| Zero‑trust logging | Only log request IDs. Use a log‑scrubbing service (e.g., Datadog Log Pipelines) to redact any field that matches .*prompt.* before persisting. |
| Prompt encryption at rest | Store prompts in an encrypted vault (AWS KMS, HashiCorp Vault). Decrypt only in‑memory right before the LLM call. |
| Separate tenant namespaces |
Herramienta mencionada: Groq Cloud
Top comments (0)