Tracing pipelines flatten nested OpenTelemetry data (resource attributes + span attributes) into a single flat record before storage. Simple in theory. In OpenObserve, it was quietly losing data.
The bug: When a span's attributes and its resource's attributes shared the same key, for example service_name appearing in both places, the flattening step let the span-level value silently overwrite the resource-level one. Resource attributes are supposed to be the authoritative source (they describe the entity emitting the trace); span attributes are per-operation and shouldn't be able to clobber them. But the flattening code had no collision handling at all: last write wins, whoever happened to serialize second.
That's a correctness bug that's invisible until it isn't. Your dashboards keep working. Your queries keep returning results. They're just returning the wrong service_name for a subset of spans, and nothing errors to tell you.
Root cause: No collision-avoidance strategy existed between BLOCK_FIELDS and RESERVED_SPAN_FIELDS. Two separate reserved-field lists, no coordination between them, and no prefixing scheme to keep colliding keys distinguishable during flattening.
The fix:
- Resource's canonical service.name stays authoritative as service_name, full stop.
- Any other resource attribute that collides gets a service_attr_ prefix instead of silently competing.
- Colliding span attributes get an attr_ prefix before flattening, so they land next to the resource value instead of over it.
- Merged the two separate reserved-field lists into one unified list, closing the gap that let the collision happen in the first place.
- Added regression tests specifically covering the three-way collision: resource service.name, resource name, and span service_name all present at once.
Impact: Trace data integrity restored for any pipeline where span and resource attributes overlap, which in practice is common (service metadata gets attached at both levels by different instrumentation layers). The unified reserved-field list also closes the door on the same class of bug reappearing with a different field name.
Lesson: Attribute collisions are a "works on the happy path" bug class. Nothing throws, nothing fails a test that only checks the non-colliding case. The fix wasn't just patching the one field, it was building a naming convention (canonical vs. prefixed) that makes the next collision safe by construction instead of requiring another one-off fix.
PR: openobserve/openobserve#12456 (merged)
This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
Top comments (0)