Your Service Map Is Lying

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete. "The agent traces everything" is the sentence repeated in every onboarding doc.

This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see.

The flow everyone trusts

The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent.

I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic, so it would always reflect what the system actually does.

Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans.

The edge that wasn't there

I generated the graph and the synchronous edges lit up immediately:

• api-gateway → user-service
• user-service → MySQL
• notification-service → PostgreSQL

Then I looked for the one edge I actually cared about — user-service → notification-service, the asynchronous hop over Kafka.

It wasn't there.

The naive conclusion (and why it's wrong)

The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer.

So I checked. And the consumer was completely fine. notification-service had consumed every event and written every corresponding row to PostgreSQL. Its database edge was lit. Liveness was perfect; the feature worked end to end.

That is the trap. A missing edge looks exactly like a broken feature, and the instinct is to go "fix" something that was never broken. The defect wasn't in the message path at all — it was in the observability of the message path. Those are two different failure domains that happen to render identically on a dashboard.

So I stopped trusting the picture and went to the source of truth: the trace store.

Evidence over assertion

Two TraceQL queries settled it.

{ span.messaging.system = "kafka" }
→ 0 results

{ resource.service.name = "user-service" } && { resource.service.name = "notification-service" }
→ 0 traces

Zero Kafka messaging spans anywhere in the system. Zero traces spanning both services. The agent — this build, under this Spring Boot version — simply was not instrumenting the Kafka client. Nothing errored. No warning was logged. The map wasn't wrong about the data; the data was never produced.

This is the part worth sitting with: a green dashboard would have let me believe the chain was fully traced. The absence of red is not the presence of coverage.

How the graph is actually built

The pipeline that produces the map is worth seeing, because one of its links is also a silent path you have to consciously turn on:

OTel Java Agent (spans)
        │
        ▼
Tempo  (service-graphs processor)
        │  remote_write
        ▼
Prometheus  (--web.enable-remote-write-receiver)
        │  query
        ▼
Grafana  (Tempo data source → serviceMap → nodeGraph)

Three deliberate decisions shaped it:

Generated edges, not declared ones. Tempo reads client/server span pairs and emits the edge metric. The graph is a test, not a drawing — it fails when reality diverges from expectation, which is exactly what makes it valuable.

One processor, scoped on purpose. I enabled service-graphs only — deliberately not span-metrics. The latter generates RED/latency series and would have inflated cardinality for a deliverable that was strictly about topology. Minimal blast radius, single responsibility.

A push boundary you must open explicitly. Tempo remote-writes its generated metrics into Prometheus — the inverse of every other component, which Prometheus scrapes. That requires flipping Prometheus into a receiver with --web.enable-remote-write-receiver. Forget it, and the metrics silently never land. Same lesson as the Kafka gap, one layer down: the data paths that fail quietly are the ones nobody turned on.

One missing edge, two real causes

There's a subtlety the graph forced me to articulate. Even once the agent emits Kafka spans, the user-service → notification-service edge would still not look like a normal synchronous call — because my publish doesn't happen on the request thread.

I use the transactional outbox pattern: the request persists the user and an outbox row in one local transaction, and a scheduled poller publishes to Kafka afterward. So the Kafka branch roots its own trace, off the poll cycle, structurally separate from the originating HTTP request. That's the outbox decoupling working as designed — not a propagation bug.

So a single missing edge had two distinct, both-legitimate explanations: a real instrumentation gap (no Kafka spans) and a real design boundary (async decoupling). Conflating them would have been the naive read. The generated graph didn't just find a bug — it made an architecture decision legible.

Making the map legible, not just generated

There was one more way the graph was technically correct but practically dishonest: both databases first rendered as a single node named localhost. The agent labels an uninstrumented peer by its host, and both databases live on the loopback interface — so MySQL and PostgreSQL collapsed into one vertex. A heterogeneous-persistence design rendered as if it shared one database.

The fix is a peer-service mapping on the agent:

-Dotel.instrumentation.common.peer-service-mapping=\
  localhost:3306=MySQL,localhost:5432=PostgreSQL,\
  localhost:8085=SchemaRegistry,localhost:8180=Keycloak

Now each dependency resolves to a real, named vertex, and the Schema Registry shows up as its own node. I confirmed it the same way I confirmed everything else — in the backend, not the UI:

traces_service_graph_request_total{server="MySQL"}        → present
traces_service_graph_request_total{server="PostgreSQL"}   → present
traces_service_graph_request_total{server="localhost"}    → 0 (gone)

A graph that says localhost is generated; a graph that says MySQL and PostgreSQL is legible. The labels are what turn telemetry into topology.

Key takeaways

• Auto-instrumentation is a claim, not a guarantee. "The agent traces everything" holds until a version boundary quietly says otherwise. Verify spans exist in the backend; never infer coverage from the absence of errors.
• A missing edge is not a missing feature. Liveness and observability are independent failure domains. A system can be correct and unobservable at the same time — the most dangerous state, because it looks fine.
• A generated graph is a test, not a decoration. Its entire value is that it can fail when reality diverges. A hand-drawn diagram cannot fail; that's precisely why it's worthless as a safety net.
• Silent data paths must be consciously enabled. Remote-write receivers, peer mappings, opt-in instrumentation — the telemetry that "just never lands" is the telemetry nobody turned on. Make the gaps loud.
• Name your peers. Topology you can't read is only half-built; labels are the difference between a generated graph and a useful one.
• Verify with queries, not screenshots. Green is the absence of evidence, not evidence of absence.

The platform is open source — gateway, two services, Kafka, the full observability stack, and this service-graph setup: https://github.com/Rummy43/ai-microservices-platform

What's the most expensive silent gap you've found by verifying instead of trusting the dashboard?

Originally published on Medium.