Migrating from SkyWalking to OpenTelemetry: Dual-Stack Coexistence and Distributed Tracing in Practice

#java #opentelemetry #devops #opensource

Bottom line: You do not need to decommission SkyWalking in one shot—first move app-side collection to OTel SDK/Collector, use OTLP dual export to validate distributed tracing in parallel, then gradually route new traffic to an OTLP-native open-source APM. This article covers collection replacement, checklists, and Remote MCP transition coexistence.

When to Start SkyWalking → OTel Migration

Migration is not a rejection of SkyWalking—it aligns the distributed tracing ingestion layer when organizational strategy shifts

Apache SkyWalking is an ASF top-level project; Probe + OAP + Storage + UI covers Trace / Metrics / Logs with a mature ecosystem. Many teams are not "abandoning" it—they evaluate dual-stack coexistence at these inflection points: legacy on SkyWalking, new services on OpenTelemetry.

- **Trigger Scenario** — Typical Pain · Migration Direction - **OTLP unified strategy** — Maintaining both SkyWalking Agent and OTel SDK; changing backend means changing probes · Unify collection on OpenTelemetry; backends accept only OTLP gRPC 4317 / HTTP 4318 - **AI-native ops** — No conversational Trace queries, no MCP-to-IDE workflow · Evaluate AI Native open-source APM: LLM queries Traces and metrics; Remote MCP reads legacy during transition - **Component & ops relief** — OAP + ES/BanyanDB and more; storage upgrades consume SRE time · Backend collapses to Ingest + Doris + Web; one-line script for minute-level POC

Recommended path: Parallel dual-write validation → gradually switch alerts and dashboards → Remote MCP reads legacy platform during transition—not a one-shot OAP shutdown. If you rely heavily on Mesh/eBPF zero-code coverage, staying on SkyWalking as primary stack remains reasonable.

- **Layer** — Typical SkyWalking Shape · After Migration (DataBuff example) - **App collection** — SkyWalking Java/Node Agent · OTel SDK / Auto-Instrumentation, standard OTLP export - **Optional aggregation** — Agent → OAP (11800/12800, etc.) · OTel Collector (optional) → Ingest 4317/4318 - **Backend storage** — ES / BanyanDB / H2, etc. · Apache Doris unified Trace + metrics - **Console** — SkyWalking UI · Web UI (27403) + AI platform + MCP

One-line install: curl -fsSL https://databuff.ai/databuff/ai-apm-install.sh | bash. Ingest exposes OTLP ports 4317 (gRPC) and 4318 (HTTP) by default.

Data Collection: SkyWalking Agent → OpenTelemetry

Understand the mapping first, then change startup args or Collector routing by language in batches

- **SkyWalking Side** — OTel Equivalent · Migration Notes - **agent.service_name** — OTEL_SERVICE_NAME · Keep service name consistent for topology and alert comparison - **collector.backend_service** — OTEL_EXPORTER_OTLP_ENDPOINT · Point to Ingest: http://:4318 - **-javaagent:skywalking-agent.jar** — -javaagent:opentelemetry-javaagent.jar · Remove SW Agent to avoid dual-probe CPU overhead - **SW sampling** — OTEL_TRACES_SAMPLER · 100% sampling during parallel phase; tune to parentbased_traceidratio after cutover

Before (SkyWalking Agent):

java -javaagent:/path/skywalking-agent.jar \
  -Dskywalking.agent.service_name=order-service \
  -Dskywalking.collector.backend_service=oap:11800 \
  -jar order-service.jar

After (OTel Java Agent → DataBuff Ingest):

export OTEL_SERVICE_NAME=order-service
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<ingest-host>:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

java -javaagent:opentelemetry-javaagent.jar -jar order-service.jar

SkyWalking has supported OTLP Trace ingestion since v9—if you cannot change Agents short term, OTel Collector can take one OTLP in and export two ways. Long term, apps should emit OTLP directly.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp/skywalking:
    endpoint: oap:11800
    tls:
      insecure: true
  otlp/databuff:
    endpoint: <ingest-host>:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/skywalking, otlp/databuff]

OAP OTLP ports depend on your deployment. Collector dual export uses less CPU than dual app probes.

Parallel Operation and Data Comparison

Migration success depends on quantifiable equivalence—not merely "the new UI opens"

Step 1 — Install DataBuff three components; confirm 4317/4318 and 27403 reachable
        ↓
Step 2 — Collector dual export or canary Pods reporting only to DataBuff
        ↓
Step 3 — Fixed load test / traffic replay; compare both platforms in the same window
        ↓
Step 4 — Sign off checklist before switching alerts and on-call views

- **#** — Validation Item · Pass Criteria (example) - **1** — Service registration completeness · Service list diff between platforms < 5% - **2** — Trace volume · Span count in same hour deviates < 10% - **3** — Error rate / P95 · Key API trends align; deviation explainable - **4** — Topology edges · No missing upstream/downstream edges or ghost nodes - **5** — Slow Trace drill-down · Top N slow requests locatable as Spans on both platforms

Fig. 1 · Step 3 comparison — verify complete service registration (acceptance ①: inter-platform service diff < 5%)

Fig. 1 · Step 3 comparison — verify complete service registration (acceptance ①: inter-platform service diff < 5%)

Fig. 2 · Step 3 comparison — verify upstream/downstream edges and middleware nodes (acceptance ④: topology edges match)

Fig. 2 · Step 3 comparison — verify upstream/downstream edges and middleware nodes (acceptance ④: topology edges match)

Fig. 3 · Anchor on entry service service-a; verify call chain breakdown and latency contribution explain delay sources

Fig. 3 · Anchor on entry service service-a; verify call chain breakdown and latency contribution explain delay sources

Fig. 4 · Trace overview — compare trace count and P50–Max response time distribution (acceptance ②③: Span count and percentile trends)

Fig. 4 · Trace overview — compare trace count and P50–Max response time distribution (acceptance ②③: Span count and percentile trends)

Fig. 5 · Slow trace drill-down — waterfall locates Redis / MySQL / downstream Spans (acceptance ⑤)

Fig. 5 · Slow trace drill-down — waterfall locates Redis / MySQL / downstream Spans (acceptance ⑤)

Suggested triage path: confirm registration in service list → find missing edges in global topology → confirm entry chain in service flow → drill slow Spans in trace list.

Dual-Stack Wrap-Up: Historical Data · Alerts · Remote MCP

Traces are hard to "move losslessly"—focus on read-only retention access and aligned alert semantics

- **Strategy** — Approach · Best For - **Dual-stack read-only** — Scale SkyWalking to read-only for 30–90 days; new data only into DataBuff · Compliance retention, audit replay - **Hard cut, no rollback** — Decommission OAP after parallel validation; history expires with TTL · POC / non-production

SkyWalking storage backends vary (ES, BanyanDB, etc.)—there is no universal one-click import. Production often uses dual-stack read-only: new platform handles live application performance monitoring; old platform read-only fallback.

Alert cutover order: create silenced rules in DataBuff for 3 days → compare noise rate → then enable notifications and disable matching SW rules to avoid double paging.

During parallel ops, on-call may switch between two UIs. DataBuff supports Remote MCP: register SkyWalking Open API as an external Tool so AI conversation can query OTLP data in Doris and SkyWalking legacy—"one chat window, two backends."

- **Phase** — Duration (reference) · Collection · On-Call View - **P0 POC** — 1–3 days · Single service OTel reporting · DataBuff validation only - **P1 Parallel** — 2–4 weeks · Collector dual export · SW primary + DataBuff comparison - **P2 Primary cutover** — 1–2 weeks · Full OTel; SW Agent retired · DataBuff primary alerts - **P3 Wrap-up** — Per retention policy · OTLP only · SW read-only or decommissioned

FAQ

- **Question** — Short Answer - **Must migration stop SkyWalking?** — No. Use Collector dual export so SkyWalking and OTLP backend receive in parallel; cut over after comparison. - **OTLP port 4317 or 4318?** — HTTP 4318 is friendliest for Demo and most SDKs; gRPC 4317 suits high-throughput production. - **Which alerts are authoritative during parallel?** — P1: SkyWalking primary alerts, DataBuff for comparison; P2: migrate alert rules after primary cutover.

From an industry perspective, Gartner expects observability platforms to analyze multiple telemetry types to detect behavior changes affecting end-user experience. Service list, topology, service flow, and trace waterfall comparisons in this migration guide turn that "user experience anchor" into a sign-off checklist.