The export looked harmless. A 48 KB trace.json from a local Jaeger query, one HTTP client span, p99 stuck at 1.8s. I was one paste away from asking a coding assistant why checkout stalled on POST /pay.
Then I grepped the file. http.request.header.authorization sat in span.attributes as Bearer eyJhbGciOi.... That is the whole incident. No production dump. No vendor bug report. A throwaway fixture I generated so the paste would fail before a model context window became a log sink.
Would you ship that JSON to a remote tokenizer? I would not. The failure is not "AI is risky." The failure is a missing trust boundary between an observability export and a prompt.
The trust boundary, drawn once
OpenTelemetry is a debug surface. A model is a third party, even when it runs on a box you own. Those are not the same room.
flowchart LR
App[App SDK] -->|OTLP| Collector
Collector --> Backend[Jaeger / Tempo]
Backend -->|export JSON| Laptop
Laptop -->|redact gate| Prompt[Prompt builder]
Prompt -->|allowlisted attrs only| Model
Laptop -.->|raw paste| ModelThe dotted line is the bug. Everything left of redact gate may legally contain a Bearer token, a Cookie, a db.connection_string, or a session.id that maps to a person. Everything right of it should not.
Ask the ugly question early: which process is allowed to see Authorization? If the answer is "the collector, maybe the backend, never the model," you already have an invariant. The rest of this article is that invariant as a fixture.
Minimal failing export
I did not scrape a live cluster. I wrote a span that matches what HTTP instrumentations emit when header capture is on. Treat this as a lab fixture, not evidence of a CVE.
Pinned for the lab: Python 3.12, pytest==8.3.3. No OTel SDK required to fail the test.
{
"resourceSpans": [{
"scopeSpans": [{
"spans": [{
"name": "POST /pay",
"attributes": [
{"key": "http.method", "value": {"stringValue": "POST"}},
{"key": "http.route", "value": {"stringValue": "/pay"}},
{"key": "http.status_code", "value": {"intValue": 504}},
{"key": "http.request.header.authorization",
"value": {"stringValue": "Bearer eyJhbGciOiJIUzI1NiJ9.lab-fixture"}},
{"key": "http.request.header.cookie",
"value": {"stringValue": "sid=s%3Alab.fixture"}},
{"key": "db.connection_string",
"value": {"stringValue": "postgres://app:lab-pass@db.internal:5432/pay"}}
]
}]
}]
}]
}
Save it as fixtures/trace.negative.json. That file must never become prompt context. If your assistant workflow cannot reject it, you do not have a gate. You have a habit.
Positive twin: fixtures/trace.positive.json. Same shape, no secrets. Keep http.method, http.route, http.status_code, net.peer.name, duration. That is the debug signal. The Bearer is not.
What actually leaks, and what does not
I keep a short allow/deny list next to the fixture. It is boring on purpose.
| Attribute / field | Send to a model? | Why |
|---|---|---|
http.method, http.route, http.status_code
|
Yes | Needed to reason about the stall |
http.request.header.authorization |
Never | Credential |
http.request.header.cookie / set-cookie
|
Never | Session |
db.connection_string, db.user, db.statement with literals |
Never | Credential + possible PII |
enduser.id, user.email
|
Never by default | Identifier |
span events with exception stacktrace
|
Maybe | Can embed DSNs from config loaders |
resource service.name, service.version
|
Yes | Identity of the binary, not the caller |
trace_id / span_id
|
Team policy | Correlates to production traffic |
Is trace_id a secret? Usually no. Is it a join key into a system the model should not see? Sometimes yes. I treat join keys as confidential when the backend is production.
Reproduce the leak with a 40-line gate
The gate is a pure function. No network. No model. If this function is green, you still have not proved the assistant is safe. You have proved the export is.
# redact_otel.py — lab fixture, not a production redactor
from __future__ import annotations
import json
import re
from typing import Any, Iterable
DENY_KEY_FRAGMENTS = (
"authorization",
"cookie",
"set-cookie",
"x-api-key",
"api_key",
"api-key",
"secret",
"password",
"connection_string",
"enduser.id",
"user.email",
)
DENY_VALUE = re.compile(
r"(?i)(bearer\s+[a-z0-9._\-]+|postgres://\S+|mysql://\S+|sid=\S+)"
)
REDACTED = "[REDACTED]"
def _key_denied(key: str) -> bool:
k = key.lower()
return any(frag in k for frag in DENY_KEY_FRAGMENTS)
def walk(node: Any) -> Any:
if isinstance(node, dict):
key = node.get("key")
if isinstance(key, str) and "value" in node:
if _key_denied(key):
return {"key": key, "value": {"stringValue": REDACTED}}
value = node["value"]
if isinstance(value, dict) and "stringValue" in value:
if DENY_VALUE.search(value["stringValue"] or ""):
return {"key": key, "value": {"stringValue": REDACTED}}
return {k: walk(v) for k, v in node.items()}
if isinstance(node, list):
return [walk(x) for x in node]
return node
def forbidden_hits(node: Any, acc: list[str] | None = None) -> list[str]:
acc = acc if acc is not None else []
if isinstance(node, dict):
key = node.get("key")
if isinstance(key, str) and _key_denied(key):
val = node.get("value", {})
text = val.get("stringValue", "") if isinstance(val, dict) else ""
if text and text != REDACTED:
acc.append(key)
for v in node.values():
forbidden_hits(v, acc)
elif isinstance(node, list):
for x in node:
forbidden_hits(x, acc)
return acc
def load(path: str) -> Any:
with open(path, encoding="utf-8") as f:
return json.load(f)
Pytest, two directions. Negative fixture must be caught. Positive fixture must survive.
# test_redact_otel.py
import json
from redact_otel import forbidden_hits, load, walk, REDACTED
def test_negative_export_is_rejected():
raw = load("fixtures/trace.negative.json")
hits = forbidden_hits(raw)
assert "http.request.header.authorization" in hits
assert "db.connection_string" in hits
def test_negative_export_redacts_bearer_and_dsn():
raw = load("fixtures/trace.negative.json")
cleaned = walk(raw)
blob = json.dumps(cleaned)
assert "eyJhbGciOi" not in blob
assert "lab-pass" not in blob
assert REDACTED in blob
assert forbidden_hits(cleaned) == []
def test_positive_export_keeps_route_and_status():
raw = load("fixtures/trace.positive.json")
assert forbidden_hits(raw) == []
cleaned = walk(raw)
blob = json.dumps(cleaned)
assert "/pay" in blob
assert "504" in blob
Run it:
python -m pytest -q test_redact_otel.py
Expected on the negative file before walk: assertion finds http.request.header.authorization. Expected after walk: the JWT substring is gone. If your first run is all green without a negative fixture, the test is lying. Add the failing export first. Then make it fail closed.
Prevent / detect / recover
A redactor in a gist is not a control. Name the layer.
| Layer | Prevent | Detect | Recover |
|---|---|---|---|
| SDK | Do not capture Authorization / Cookie. Turn header capture off unless a named allowlist exists. |
CI grep for OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS and for http.request.header.authorization in golden traces |
Rotate the token the span held; treat the export as a credential dump |
| Collector | Attribute processor drops deny-listed keys before storage | Collector debug log sampled without values; metric on dropped keys | Flush or re-process the bad batch; mark the backend index tainted |
| Export CLI |
jq / this fixture as a required pipe before clipboard |
Pre-commit or Make target: pytest test_redact_otel.py
|
Delete trace.json from disk and chat history |
| Prompt builder | Refuse to attach any .json unless forbidden_hits is empty |
Unit test on the builder, not on the model | Invalidate the thread; do not "ask the model to forget" |
| Model host | Network egress allowlist; no default path to a public tokenizer | Prompt-size + attachment hash logged locally, not the body | Rotate anyway; the host is not your source of truth |
Notice recover always includes rotate. Redaction after a paste is not recovery. The bytes already left.
Where a self-hosted assistant still sits behind this gate
I evaluate coding assistants the same way I evaluate log shippers: what crosses the boundary, and who can read the buffer.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source AI development platform with free model access and a free server option, which is enough to stand up a local prompt path and point this fixture at it. That is the only product fact this workflow needs. I am not pinning model names, quotas, or hardware here; those change, and a stale number is worse than silence.
Self-hosting moves the tokenizer. It does not delete Bearer eyJ... from trace.json. A teammate with disk access, a crash dump, a session recorder, or a "helpful" debug log of prompts will still see it. If you use a self-hosted path, run forbidden_hits in the prompt builder before the bytes hit localhost. Localhost is not a policy.
If you want one check against that path: export the negative fixture, attach it the way you would attach a Jaeger dump, and confirm the builder rejects it. If the builder cannot reject, do not debug production traces there. Debug the builder.
Commands I actually keep next to the fixture
Unexecuted until you wire them to your backend. Labels matter.
# 1) Confirm the raw export still contains a deny-listed key (lab file).
grep -n "authorization" fixtures/trace.negative.json
# 2) Fail closed in CI. No model invocation.
python -m pytest -q test_redact_otel.py
# 3) Optional clipboard pipe for humans who insist on interactive debug.
# Template — unexecuted. Replace jaeger-export.json with a file you own.
python - <<'PY'
from pathlib import Path
from redact_otel import walk, forbidden_hits, load
import json, sys
p = Path("jaeger-export.json")
data = load(p)
hits = forbidden_hits(data)
if hits:
print("refusing export; keys:", hits, file=sys.stderr)
sys.exit(2)
Path("jaeger-export.redacted.json").write_text(json.dumps(walk(data), indent=2))
print("wrote jaeger-export.redacted.json")
PY
Do not curl a production Jaeger into this. Copy a synthetic span first. If you cannot explain every attribute in the file, you are not ready to paste it.
Limitations — read these before you copy the regex
This approach is a regression fixture for a known shape: OTLP JSON with key / stringValue pairs. It is not a DLP product.
- Nested protobuf, zipkin v1, Tempo parquet, or a screenshot of Grafana will bypass it. Different codec, different test.
- Base64 blobs, JWTs split across events, and
db.statementwith inline emails need parsers you have not seen above. I did not pretend to write those. - Allowlisting
http.routecan still leak identifiers if your routes are/users/{email}. -
DENY_VALUEis a coarse regex. It will false-positive on the word "bearer" in a comment span. Good. Fail closed, then add an allow. - A green pytest does not prove your assistant UI skipped the file input. Test the builder, or you tested a script nobody runs.
Who should not use this as their only control: teams under a legal hold that must retain raw traces; teams whose model already runs inside the same confidentiality domain and whose prompt store is production-grade; anyone hoping a redactor will make it acceptable to paste live customer traffic. It will not. Use a scrubbed staging trace.
Who should use it: platform and security engineers who already dump spans when latency spikes, and who have started feeding those dumps to a coding assistant. If that is you, the invariant belongs in CI, not in a wiki.
The boundary question
I want one line in the pipeline, not a culture speech. Does forbidden_hits run in CI on every golden trace and on the prompt builder, or only in a gist you remember after the leak?
The SDK should stop recording Authorization. The collector should drop it anyway. The prompt builder should refuse the file if either layer missed. Three layers, one invariant. Which of those three can you prove with a fixture tonight?
Top comments (0)