A backend engineer adds OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS=.* to staging to debug a 403. The config reaches production. Six months later, 400k Authorization headers per day are queryable in Jaeger by anyone on the VPN.
The official OTel narrative is that sensitive data capture requires explicit opt-in. CVE-2024-32028 contradicts that narrative: url.query was captured by default in the .NET SDK, sending API keys as query params directly to the trace backend. When engineers enable header capture for debugging, the Authorization header goes with it. Jaeger, which ships with no authentication by design, stores all of it. The result is privilege inversion: observability access grants visibility into tokens that production access controls were designed to protect.
url.query Is Captured by Default, and It Had a CVE
CVE-2024-32028 (GHSA-vh2m-22xx-q94f) confirms that OTel query string exposure is not theoretical. The .NET SDK prior to version 1.8.1 captured url.full on HTTP client spans and url.query on server spans without filtering values. The severity is CVSS 4.1, CWE-201 and CWE-212, and API keys passed as ?api_key=, ?token=, or ?access_token= went directly to the trace backend.
OTel semantic conventions define url.full as a required attribute on HTTP client spans. This behavior is specified in the convention, not an isolated SDK bug. The fix in v1.8.1 changed the default to redact all query string values, confirming that the previous default was exposure by omission.
CI/CD pipelines that skip regular telemetry dependency updates carry this risk silently. The leak happens in the observability plane, not the application plane.
The Debug Config That Ships to Production
Authorization header capture requires opt-in, but the opt-in is trivial and rarely reverted after debugging. The OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS env var accepts regex: when an engineer sets .*, all headers are captured, including Authorization: Bearer eyJhbGci.... The resulting span attribute is http.request.header.authorization, containing the full token value.
Java, Python, and Go SDKs support the same capture pattern with equivalent variables. The env var lives in the infra file (docker-compose, Kubernetes ConfigMap, ECS task definition) and does not go through security code review. Staging configs promote to production in monorepo deployments with no structural difference between environments.
OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS exists to sanitize specific headers after capture. That sanitization is ineffective when the engineer who enabled the capture has left the team. Configs written months ago for a different debug session rarely include sanitization.
Jaeger Ships With No Authentication, by Design
Jaeger v1.48 ships with no auth of any kind. The security documentation is direct:
"Jaeger UI does not have any built-in authentication or authorization."
Anyone with network access to the UI port can view and query all stored traces. Port 16686 exposes the web interface and the REST query API; ports 4317 and 4318 accept spans over OTLP with no authentication.
Grafana Tempo OSS follows the same model: authentication is the operator's responsibility. The documentation does not require auth configuration to bring the service up.
CVE-2020-10750 (CVSS 5.5, CWE-532) documents Jaeger leaking Kafka credentials to container logs. The project treats credential exposure as an operational concern. That position explains why the recommended solution to the auth gap is an external proxy, not a change in Jaeger.
The Privilege Inversion
The standard access model stores prod API keys in secrets managers like Vault or AWS SSM, with role restrictions and an audit trail. Jaeger access is granted to all engineers as part of observability onboarding, without the same level of control. This inverts the access hierarchy: an engineer who cannot read a secret directly can read it from a trace.
Captured Bearer tokens remain queryable until they expire. Long-lived tokens, common in B2B integrations, are available for the full retention window. The default is 7 to 30 days in Jaeger and Grafana Tempo.
A compromised CI/CD runner with VPN access reaches Jaeger without touching secrets management. The insider threat vector is direct: an engineer queries Jaeger by service and time range and extracts tokens from services they should no longer access. Traces sent to third-party APMs like Datadog or Honeycomb constitute data transfer without adequate legal basis under GDPR and CCPA.
Defense: Scrub at the Collector, Not at the Instrumentation
Relying on opt-in to prevent secret capture does not work in practice. Configs change, engineers do not track what is being captured, and the span is already in the backend when the problem surfaces. The reliable control is allowlist-based redaction at the OTel Collector before data reaches the trace backend.
The redaction processor operates on an allowlist model: any attribute not explicitly permitted is blocked or replaced with REDACTED. The attributes processor with a delete action removes http.request.header.authorization and http.request.header.cookie from the collection pipeline. The transform processor applies regex-based masking on attribute values before export to any backend.
For Jaeger UI, the documented path is an OAuth sidecar proxy. The Jaeger Tracing blog documents the pattern with Keycloak or NGINX in front of port 16686. Treat the trace backend as sensitive data storage and apply service-level RBAC, not just network perimeter security.
Detection: Query Your Own Traces First
Auditing the trace backend takes under 10 minutes using the Jaeger REST API. GET /api/traces?service=<svc>&tags={"http.request.header.authorization":"Bearer"} returns all spans carrying Authorization headers. A search across url.full for ?token=, ?api_key=, or ?access_token= maps query string exposure by service.
In Grafana Tempo, the TraceQL query {span.http.request.header.authorization != nil} returns all spans with the header present. Most teams have never run this query against their own traces, and that visibility gap is where the risk persists.
The MAGO Intel tool (intel.mago.team) detects exposed Jaeger interfaces via service fingerprinting. Jaeger presents no authentication prompt and responds with identifiable patterns on port 16686. The Shodan query port:16686 product:Jaeger returns self-hosted instances exposed to the internet. The same reconnaissance pattern applies to internal networks during security assessments.
CVE-2024-32028 proves that a major OTel SDK shipped query string exposure for multiple release cycles. The fix was a default behavior change to redact-by-default, confirming that the previous default was exposure by omission. Apply the same standard to your trace backend: assume it exposes data until you verify it does not.
Top comments (0)