I reconstructed a 500 I keep seeing in incident channels. A client called GET /orders/not-an-id with a session token. FastAPI raised ValueError. The exception handler logged dict(request.headers) “for context.”
I copied the traceback into a model chat. Why? Because int("not-an-id") looked like a one-line fix. The paste also contained Authorization: Bearer test-token-do-not-use. The model did not need that header. My log pipeline never stopped it.
That is the trust-boundary violation. The failing request already happened. The privileged action is the next paste.
What actually crossed the boundary
Logs are not a scratch pad. They are an egress path.
Once a header lands in stdout, it can move to a file, a log shipper, a Slack thread, an agent tool result, or a model context window. Do you redact in the worker, or do you hope someone notices a token in frame 7?
I treat this as three boundaries, not one:
-
Request vs process. Headers belong to the HTTP connection. The app may read
Authorizationto authenticate. It must not serialize that value into aLogRecord. - Process vs log sink. stdout, journald, and your aggregator are a different trust domain. Whoever can read the sink can replay the session.
- Log sink vs model. A paste or a tool call is a fourth party. Self-hosted or not, the model runtime is not your authorization layer.
The invariant I want in CI is boring: captured exception text must not contain Authorization, Cookie, or Bearer.
This is a constructed lab, not a CVE against FastAPI. I did not find a vendor bug. I reproduced a handler anti-pattern that turns a 400 into a credential disclosure the moment a human—or an agent tool—forwards stderr.
[Client] -- Authorization --> [FastAPI worker]
| logging.exception(..., dict(request.headers))
v
[stdout / journal]
| human paste / agent tool result
v
[Model context]
Redaction belongs in the worker. The chat window is not a control.
Lab pin (unexecuted until you run it)
Pinned for the fixture, not a production endorsement:
- Python 3.12
fastapi==0.115.6pytest==8.3.4-
httpx==0.28.1
python -m venv .venv
source .venv/bin/activate
pip install 'fastapi==0.115.6' 'pytest==8.3.4' 'httpx==0.28.1'
If your resolver floats Starlette, freeze the lockfile before you trust the assertion. Label the next blocks as a lab recipe until pytest is green on your machine.
Negative fixture: the leak
Save as app_leaky.py. This handler is the failure I want the test to catch.
# app_leaky.py — lab fixture, constructed leak
import logging
from fastapi import FastAPI, Request
logging.basicConfig(level=logging.INFO)
log = logging.getLogger("orders")
app = FastAPI()
@app.middleware("http")
async def log_headers_on_error(request: Request, call_next):
try:
return await call_next(request)
except Exception:
log.exception("handler failed headers=%s", dict(request.headers))
raise
@app.get("/orders/{order_id}")
async def get_order(order_id: str):
return {"order_id": int(order_id)}
The request that should fail. The token is fake on purpose.
# Unexecuted until you run it.
curl -sS -D - \
-H 'Authorization: Bearer test-token-do-not-use' \
http://127.0.0.1:8000/orders/not-an-id
Expected failure evidence: process logs contain Bearer test-token-do-not-use next to the traceback. If you paste that stderr into a model, the token went with it. That is the whole bug.
Uvicorn access logs are a side path, not the main one. Default access format prints method, path, and status. The dangerous line is the one you added: dict(request.headers) inside logging.exception.
Look at the extra dict. %s plus a header map is how secrets survive a “harmless” stack trace. log.exception already includes the traceback. You do not need the header dump. You wanted a request id. You shipped a replay handle instead.
Positive fixture: redact before getMessage()
A logging.Filter runs in-process, before handlers write. That is the right layer.
# redact.py
import logging
import re
HEADER_RE = re.compile(
r"(?i)\b(authorization|cookie|set-cookie|x-api-key|x-auth-token)\b\s*[:=]\s*\S+"
)
BEARER_RE = re.compile(r"(?i)\bBearer\s+\S+")
DENY_HEADERS = {
"authorization",
"cookie",
"set-cookie",
"x-api-key",
"x-auth-token",
}
def scrub(text: str) -> str:
text = HEADER_RE.sub(lambda m: m.group(1) + "=[REDACTED]", text)
return BEARER_RE.sub("Bearer [REDACTED]", text)
class HeaderRedactFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
record.msg = scrub(str(record.msg))
if record.args:
if isinstance(record.args, dict):
record.args = {k: _scrub_obj(v) for k, v in record.args.items()}
else:
record.args = tuple(_scrub_obj(a) for a in record.args)
headers = getattr(record, "headers", None)
if isinstance(headers, dict):
record.headers = {
k: "[REDACTED]" if k.lower() in DENY_HEADERS else v
for k, v in headers.items()
}
return True
def _scrub_obj(value):
if isinstance(value, str):
return scrub(value)
if isinstance(value, dict):
return {
k: "[REDACTED]" if str(k).lower() in DENY_HEADERS else _scrub_obj(v)
for k, v in value.items()
}
return value
Wire it once, at process start, on the root logger and on uvicorn / uvicorn.access if you customize those. Installing it only on orders will miss a library that logs the request object.
# app.py
import logging
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from redact import HeaderRedactFilter
logging.basicConfig(level=logging.INFO)
for name in ("", "orders", "uvicorn", "uvicorn.access"):
logging.getLogger(name).addFilter(HeaderRedactFilter())
log = logging.getLogger("orders")
app = FastAPI()
@app.exception_handler(ValueError)
async def value_error_handler(request: Request, exc: ValueError):
# Path and request id only. Not the header map.
log.warning("invalid order_id path=%s", request.url.path)
return JSONResponse({"detail": "invalid order_id"}, status_code=400)
@app.get("/orders/{order_id}")
async def get_order(order_id: str):
return {"order_id": int(order_id)}
Notice the handler logs the path, not the header map. The filter is defense in depth. It is not permission to dump request.headers again. If a reviewer asks “why not log everything and redact later?”, the answer is encodings. Regex will miss the second copy.
Regression tests that fail closed
test_redact.py is the artifact I actually want in CI. Positive and negative. If the negative test starts passing without the leaky logger, the harness is lying.
# test_redact.py
import logging
from redact import HeaderRedactFilter, scrub
def test_scrub_removes_bearer():
raw = "handler failed headers={'authorization': 'Bearer test-token-do-not-use'}"
assert "test-token-do-not-use" not in scrub(raw)
assert "Bearer [REDACTED]" in scrub(raw)
def test_filter_redacts_logrecord_args(caplog):
logger = logging.getLogger("orders_test")
logger.setLevel(logging.INFO)
logger.addFilter(HeaderRedactFilter())
with caplog.at_level(logging.INFO, logger="orders_test"):
logger.info("headers=%s", {"Authorization": "Bearer test-token-do-not-use"})
text = caplog.text
assert "test-token-do-not-use" not in text
assert "REDACTED" in text
def test_negative_raw_logger_still_leaks(caplog):
"""Guard the guard. If this fails, caplog is not capturing what we think."""
logger = logging.getLogger("orders_leaky")
logger.setLevel(logging.INFO)
with caplog.at_level(logging.INFO, logger="orders_leaky"):
logger.info("headers=%s", {"Authorization": "Bearer test-token-do-not-use"})
assert "test-token-do-not-use" in caplog.text
Run:
pytest -q test_redact.py
Expected: three passed. The third test documents the leak. Do not “fix” it by adding the filter to orders_leaky. You want that assertion to stay red-team honest.
Optional grep gate on fixture output, after you capture a real 500:
# Unexecuted template. Fail the job on a leftover credential shape.
rg -n -i -e 'Bearer ' -e 'authorization=' -e 'cookie=' captured-logs/ && exit 1 || true
Threat model
| Asset | Trust boundary | Failure mode | Impact if a model sees it |
|---|---|---|---|
Authorization bearer |
App → log → paste/tool | log.exception(..., dict(request.headers)) |
Session replay against the API |
Cookie / Set-Cookie
|
Same | Same | Session replay, CSRF token theft |
X-Api-Key |
Same | Gateway key in middleware logs | Service impersonation |
Query ?token=
|
Access log path | Token in URL | Replay; also Referer leaks |
| User email in path | Path in traceback | /users/ada@example.com |
PII in model logs |
Internal Host / kube DNS |
Error page / traceback | Cluster layout | Recon, not credentials |
I do not treat “the model promised not to train on it” as a control. Promises are not invariants. Tool calling makes this worse, not better: the same dump can re-enter context as a tool result without anyone clicking paste. Who owns that hop in your harness?
Prevent / detect / recover
| Stage | Control | Evidence it works |
|---|---|---|
| Prevent | Never log request.headers or request.cookies. Log path, status, X-Request-ID only. |
Code search + review |
| Prevent |
HeaderRedactFilter on root and uvicorn loggers |
test_filter_redacts_logrecord_args |
| Detect | CI pytest plus a grep gate on captured fixture logs |
Job fails on Bearer
|
| Detect | Gateway already authenticated the client; the app must not re-emit the secret | Architecture review |
| Recover | Rotate the token, invalidate the session, treat the paste as disclosure | Incident ticket |
| Recover | If the dump already left the network, rotate first, argue about the model vendor second | Time-to-revoke < time-to-debate |
Regex will miss encodings. Base64 of the header, split tokens, Authorization%3A%20Bearer, Unicode lookalikes. The filter is a seatbelt. The real prevent control is: do not put the header map in the log call.
What I still will not send a model
Even after redaction, I keep a deny list for pastes:
- Canonical request dumps with bodies (
password=,otp=) -
Set-Cookieand session IDs in any form - Connection strings, even “expired”
- Customer email, phone, government IDs in paths or payloads
- Internal signed URLs
- Full
repr(request)/ StarletteHeadersobjects - Other users’ data that happened to be in the same traceback
If the remaining text is only ValueError: invalid literal for int() with base 10: 'not-an-id' plus a path /orders/{id}, that is enough for a model. Anything else is curiosity, not debugging. Ask yourself: would I put this string in a public GitHub issue? If no, it does not belong in a prompt.
Where a self-hosted workspace actually fits
Redaction is the control. Location is residual risk.
If you still want a model to read the sanitized stack, do not confuse “our GPU” with “safe to dump secrets.” A self-hosted runtime still has operators, swap, crash dumps, and prompt logs. I use it only after the filter and the deny list.
MonkeyCode is an open-source AI development platform. The two availability options that matter for this workflow are free model access and a free server, so a trial does not have to begin by shipping exception text to a third-party inference API. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
That is optional. The pytest file is not. If you already gate logs in CI, you do not need a new product to enforce the invariant.
Who should not use this approach
- Teams that will treat regex redaction as a license to log
request.headersagain. - Anyone pasting live production dumps—including into a laptop LLM—without rotating the credential that appeared.
- Pipelines that forward raw stdout to an agent tool. The tool result is another model ingress. Filter before the tool, not after the chat.
- Orgs without session revocation. Detection without recover is a diary.
Limitations
I have not claimed a production breach. The leaky middleware is a fixture. FastAPI and Uvicorn versions drift; pin them.
logging.Filter will not see writes that bypass logging: print(request.headers), traceback.print_exc() to a custom file, OpenTelemetry span attributes, or an APM agent that captures request headers by default. If you enable header capture in a tracing SDK, this pytest will stay green and you will still paste a token. Extend the same deny list to span processors, or disable header capture.
Access logs that include query strings need a separate test. This article does not cover HAR files, container env dumps, Terraform state, or WAF 403 bodies. Those are different sinks. Same invariant, different fixture.
Boundary question
Which invariant belongs in CI, and which layer should enforce it?
I want no Bearer in captured exception text as a pytest gate on every service that handles Authorization. The worker enforces it with a filter and by not logging headers. The model is not a control plane. If a sanitized stack still leaves the machine, that is a separate decision about residual recon data—not a reason to skip the fixture.
Top comments (0)