DEV Community

Cover image for How to Land Distributed Tracing in Production: Bytecode Agents, OTel, and Demo Screenshots
AIdevops2088
AIdevops2088

Posted on • Originally published at github.com

How to Land Distributed Tracing in Production: Bytecode Agents, OTel, and Demo Screenshots

#opentelemetry #skywalking #apm #devops

Distributed tracing adoption starts with understanding how a non-intrusive Agent weaves Spans at the bytecode layer and how TraceId propagates through gateways and message queues. This guide explains bytecode enhancement in SkyWalking and OpenTelemetry Java Agents, gives a five-step landing checklist, and walks through service RED, global topology, Trace lists, and Span waterfalls in the Databuff Demo — connecting theory to an on-call-ready UI.

The frequent confusion around "how to land distributed tracing" is usually not missing tools, but missing an executable sequence: first understand what the Agent does in the JVM, then define propagation and sampling, then choose SkyWalking OAP or OTel plus a lightweight APM backend.

Bytecode enhancement: where distributed traces come from

SkyWalking Java Agent and OTel Java Agent both use attach at startup + bytecode weaving — zero business code changes.

Agent attach at JVM startup

Common production startup:

-javaagent:skywalking-agent.jar
# or
-javaagent:opentelemetry-javaagent.jar
Enter fullscreen mode Exit fullscreen mode

Before the main class loads, the JVM runs the Agent's premain and registers a ClassFileTransformer — from then on, every class load is a chance for the Agent to rewrite bytecode (SkyWalking Java Agent).

Weaving Spans: one HTTP call 

[ Business thread enters Controller ] → get/create TraceId, SpanId, record startTime
[ Call downstream HTTP/RPC ]          → create Child Span, write traceparent / sw8 headers
[ Downstream Agent intercepts ]       → parse parent Span, continue same Trace
[ Method returns ]                    → span.end(), report to OAP / OTLP
Enter fullscreen mode Exit fullscreen mode
  • TraceId identifies the whole call chain; SpanId identifies one hop.
  • SkyWalking uses sw8 propagation headers; OTel defaults to W3C Trace Context (traceparent) (OTel Java auto-instrumentation).

First lesson: No matter how polished the UI, if propagation headers are not unified, the Trace tree breaks in the middle — standardize this before picking a UI.

Plugins and auto-instrumentation

  • SkyWalking — Plugin directory for Tomcat / Spring / MySQL / Redis / Kafka, etc. · Export: gRPC → OAP (11800/12800) (Agent plugins)
  • OpenTelemetry — Auto Instrumentation + semantic conventions · Export: OTLP 4317/4318

The gap is not "whether Spans exist," but whether the data model and export protocol are standard OTel — that determines if you can swap backends without swapping Agents.

Five steps to land distributed tracing

  • ① Pilot path — Pick login/checkout, wire 2–3 services only · Pitfall: full rollout day one with broken chains and unclear ownership
  • ② Propagation — Gateway generates TraceId; unified HTTP/gRPC/MQ headers · Pitfall: async threads lose Context
  • ③ Sampling & limits — Prod 1–10% head sampling + always sample errors · Pitfall: 100% sampling overwhelms storage
  • ④ Log correlation — JSON logs with trace_id · Pitfall: Trace UI only, no log context
  • ⑤ On-call drill — Given TraceId, find slow Span in 10 minutes · Pitfall: Agent installed but never used in postmortems

SkyWalking path: Agent → OAP → ES / BanyanDB (Quick Start).

OTel path: Agent → OTLP → Collector (optional) → APM backend.

Parallel transition: Legacy SkyWalking keeps reporting to OAP; new services export OTLP; Collector handles sampling and redaction — many teams switch over 1–2 quarters, not overnight.

Databuff Application Performance: demo walkthrough

Screenshots below are from the Application Performance module on demo.databuff.ai (live session 2026-06-30). Open-source APM Databuff uses OTLP ingestion; UI path: Services → Topology → Tracing → Trace detail.

Service RED: the on-call landing screen

After rollout, on-call usually opens service list + RED metrics (request rate, error rate, response time). In the Demo, service-a averages ~240ms, service-b ~70ms, 0% errors — filter abnormal services first, then drill into Traces.

Service RED metrics — request rate, errors, duration

Figure 1 · Service page shows response time, request volume, error rate trends and table summary; minute-level RED from OTel, same ingest pipeline as Traces.

Global topology: are dependencies fully captured?

Aggregated Traces produce a service dependency topology. In the Demo, service-a links to MySQL, Redis, Kafka, Elasticsearch, and service-b — used to validate propagation rules and whether middleware Spans are missing.

Global service dependency topology

Figure 2 · Global topology shows cross-service and middleware edges; compare against your architecture diagram for "expected but missing" edges (often MQ headers without TraceId).

Tracing: from chart point to Trace list

The Tracing page shows Trace count and P50–P99 latency over time; click a point on the chart to open the Trace list (TraceId, API, duration) — the UI for step ⑤ above.

Trace count and latency percentiles

Figure 3 · Trace count and latency percentiles; hint at bottom "click any point for request details" — macro to micro drill-down.

Span waterfall: the end state of bytecode weaving

Opening a TraceId shows the call order waterfall: root Span GET /demo/checkout (~240ms), child Spans for Redis, HTTP outbound, downstream service-b, MySQL, etc. — exactly what bytecode collection looks like in the UI.

Trace Span waterfall — checkout call chain

Figure 4 · Waterfall shows Span duration and type (Web/RPC/DB/Cache) on a timeline; right panel has TraceId, SpanId, entry Span — answers "which hop is slow."

Service flow: downstream contribution from entry service

Service flow expands downstream edges from an entry service (e.g. service-a) with response contribution — answers "is slowness local logic or a dependency," similar to SkyWalking topology latency share.

Service flow with response contribution

Figure 5 · Service flow: entry service-a to MySQL, Elasticsearch, etc., with response contribution percentages; useful for SLO reviews and dependency governance.

Comparison with SkyWalking UI: SkyWalking console also offers topology, Traces, and service metrics; Databuff Application Performance uses OTLP + unified storage, path "Services → Topology → Tracing → Trace detail." Compare query latency and ops component count for the same Trace based on your OTel progress.

Summary

Distributed tracing rollout = understand how bytecode Agents collect Spans + five-step engineering discipline + choose SkyWalking OAP or OTel APM backend.

Run one pilot checkout Trace in Demo or test env first, then expand to the cluster — better than comparing ten tools on day one. With existing SkyWalking, OTel-ize new services and ingest in parallel to Databuff, compare Traces for the same request, and evaluate dual-stack ops cost.

References

Top comments (0)