The Hidden Security Risk Inside OpenTelemetry OTLP Trace Exports
When debugging microservice tail latency, database deadlocks, or LLM agent chains, platform engineers regularly export OpenTelemetry (OTLP/JSON) trace files containing resourceSpans and scopeSpans. However, a single exported trace file routinely embeds production secrets and regulated customer PII across span attributes:
-
http.request.header.authorization: Bearer JWTs, session cookies, and internal API keys attached to outbound RPC spans. -
db.statement&db.query.text: Unparameterized SQL queries containing literal customer email addresses, billing tokens, or tax identifiers. -
gen_ai.prompt&gen_ai.completion: Raw user prompts and proprietary RAG context captured by OpenTelemetry GenAI semantic conventions. -
net.peer.ip&service.instance.id: Internal Kubernetes pod CIDR addresses and private cluster endpoints.
Pasting an OTLP JSON trace dump into a cloud formatter or attaching it unredacted to a support ticket exposes production authentication tokens and customer PII outside your security boundary.
Architectural Solution: 100% Client-Side RAM OTLP Span Scrubbing
To eliminate third-party telemetry exposure during incident response, we built OpenTelemetry Trace Redactor — a zero-upload browser studio and standalone offline HTML5 workstation that sanitizes OpenTelemetry trace attributes entirely in local RAM.
1. Multi-Tier Span Attribute Redaction Without Breaking Trace Waterfalls
Instead of corrupting the JSON structure or breaking parent-child spanId links, the local redaction engine targets sensitive attribute values while preserving trace hierarchy:
// Client-side OTLP span attribute scrubbing pattern
const otlpSecretRegex = /otel_bearer_[A-Za-z0-9._-]+|sql_bind_[A-Za-z0-9._-]+|genai_prompt_[A-Za-z0-9._-]+|eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}(?:\.[A-Za-z0-9_-]+)?|[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}|sk_live_[a-zA-Z0-9_]+|AKIA[0-9A-Z]{16}|\b\d{1,3}(?:\.\d{1,3}){3}\b/g;
Engineers can toggle between three redaction strictness levels in real time:
- 35% Partial Trace ID Mask: Retains attribute prefixes for correlation while masking token suffixes and final IPv4 octets.
-
70% Standard Span Attribute Mask: Replaces bearer JWTs, SQL bind literals, GenAI prompt PII, and internal IPs with deterministic type-tagged placeholders (
[OTLP_BEARER_MASKED],[SQL_BIND_MASKED]). - 95% SOC2 & GDPR Safe Harbor Scrub: Performs full irreversible token stripping across all sensitive span attributes and events.
2. Interactive 12-Column Free Workbench & 3-in-1 Offline PRO Appliance
- Instant Browser Workbench: Load the built-in OTLP trace sample or paste your own trace table, adjust the redaction slider, and immediately copy or download the first 70% of sanitized span records for free.
-
Standalone Offline PRO Workstation: For air-gapped incident response teams, the PRO workstation compiles into a self-contained single-file HTML5 application featuring 4 dedicated OpenTelemetry batch modules (
OTLP Span Attribute & HTTP Header Scrubber,SQL Bind Parameter Redactor,GenAI Trace Prompt PII Stripper, andTraceID & Internal Endpoint Anonymizer).
Try the Live Zero-Upload Sandbox
- Launch Interactive Studio: OpenTelemetry Trace Redactor
- Direct Free Workbench: OpenTelemetry OTLP Span Sandbox
Top comments (0)