Ktor + OpenTelemetry: Distributed Tracing for Microservices Without the Observability Tax

---
title: "Ktor + OpenTelemetry: Distributed Tracing for Microservices Without the Observability Tax"
published: true
description: "Wire OpenTelemetry into Ktor microservices with coroutine-safe span propagation and Grafana Tempo — end-to-end distributed tracing for under $50/month, no Datadog required."
tags: kotlin, architecture, devops, mobile
canonical_url: https://mvpfactory.co/blog/ktor-opentelemetry-distributed-tracing
---

## What We Are Building

By the end of this workshop, you will have OpenTelemetry wired into a Ktor microservice with coroutine-safe span propagation, exporting traces to a self-hosted Grafana Tempo backend. End-to-end distributed tracing across services — for under $50/month, no Datadog contract required.

## Prerequisites

- Ktor 2.x project with Gradle
- Docker (for running Grafana Tempo locally)
- Familiarity with Kotlin coroutines

---

## The Observability Tax Is Optional

Let me show you a pattern I use in every production microservices project.

Most teams conflate "enterprise-grade observability" with "expensive SaaS vendor." That conflation is costing them money. The OpenTelemetry project — now a CNCF graduated project — has commoditized the instrumentation layer. The only remaining cost is the backend, and open-source options have closed that gap.

The math is not close. Datadog APM starts at roughly $31 per host per month, and that scales fast in a microservices environment. A self-hosted Grafana Tempo instance on a modest VM with object storage sits well under $50/month for most teams — same W3C TraceContext propagation, same OTLP wire format, query speeds that hold up at scale.

---

## Step 1: Add Dependencies

kotlin
implementation("io.opentelemetry:opentelemetry-api:1.38.0")
implementation("io.opentelemetry:opentelemetry-sdk:1.38.0")
implementation("io.opentelemetry:opentelemetry-exporter-otlp:1.38.0")
implementation("io.opentelemetry.instrumentation:opentelemetry-ktor-2.0:2.4.0-alpha")

## Step 2: Initialize the SDK

Here is the minimal setup to get this working. Wire the OTLP exporter to your Tempo endpoint and register the global instance:

kotlin
val openTelemetry: OpenTelemetry = OpenTelemetrySdk.builder()
.setTracerProvider(
SdkTracerProvider.builder()
.addSpanProcessor(
BatchSpanProcessor.builder(
OtlpGrpcSpanExporter.builder()
.setEndpoint("http://tempo:4317")
.build()
).build()
)
.setResource(Resource.create(
Attributes.of(ResourceAttributes.SERVICE_NAME, "order-service")
))
.build()
)
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal()

## Step 3: Install the Ktor Plugin

kotlin
install(KtorServerTelemetry) {
setOpenTelemetry(openTelemetry)
}

This intercepts every incoming request, extracts the `traceparent` header, and creates a root span. Child spans created inside route handlers inherit the trace automatically.

## Step 4: Fix Coroutine Context Propagation

Here is the gotcha that will save you hours.

Kotlin coroutines do not use thread locals — which is the right design decision for performance, but it means naive OpenTelemetry instrumentation breaks span propagation the moment you `launch` or `async` across coroutine boundaries.

The fix is a `CoroutineContext.Element` that carries the OTel `Context` explicitly:

kotlin
class OtelContextElement(val otelContext: io.opentelemetry.context.Context) : CoroutineContext.Element {
companion object Key : CoroutineContext.Key
override val key = Key
}

suspend fun withSpan(name: String, block: suspend () -> T): T {
val tracer = GlobalOpenTelemetry.getTracer("app")
val currentOtelCtx = coroutineContext[OtelContextElement]?.otelContext
?: io.opentelemetry.context.Context.current()
val span = tracer.spanBuilder(name).setParent(currentOtelCtx).startSpan()
val newOtelCtx = currentOtelCtx.with(span)
return try {
withContext(OtelContextElement(newOtelCtx)) { block() }
} finally {
span.end()
}
}

Build this once and it works everywhere — every `withSpan` call correctly parents to the active trace regardless of which thread the coroutine resumes on.

---

## Choosing a Backend

| Backend | Ingestion cost | Retention control | Query language | Self-hosted |
|---|---|---|---|---|
| Datadog APM | $31+/host/month | Vendor-controlled | Proprietary | No |
| Grafana Tempo + Cloud | ~$8/GB ingested | Configurable | TraceQL | Yes |
| Jaeger (self-hosted) | Infrastructure only | Full control | Jaeger UI | Yes |
| Honeycomb | $130+/month base | Vendor-controlled | BubbleUp | No |

Grafana Tempo with S3-compatible object storage (Backblaze B2 or Cloudflare R2) is the sweet spot. TraceQL gives you powerful filtering without learning a proprietary query language:

cypher
{ .service.name = "order-service" && duration > 500ms }

---

## Gotchas

**Thread locals do not survive coroutine suspension.** This is the most common failure mode. The `OtelContextElement` pattern above is not optional — without it, spans created inside `launch` or `async` blocks will either attach to the wrong parent or create orphaned root spans.

**Use the OTLP exporter, not a vendor SDK.** The OpenTelemetry SDK is vendor-neutral. Instrument once and swap backends without touching application code. If you wire in a Datadog-specific SDK today, migrating later means rewriting instrumentation.

**Run Tempo with object storage from day one.** Starting with local disk storage and migrating later is painful. The operational overhead of connecting to Backblaze B2 or Cloudflare R2 upfront is low, and you get a predictable cost ceiling immediately.

**Add business attributes to spans.** The docs do not emphasize this enough, but `span.setAttribute("order.id", orderId)` is what transforms traces from infrastructure noise into actionable debugging data. Do it at every service boundary.

---

## What the Trace Actually Shows You

Each trace carries a full coroutine-safe span hierarchy, HTTP attributes (method, status, route), custom business attributes, and cross-service `traceparent` propagation via outgoing `HttpClient` interceptors.

The first time engineers see a flame graph spanning three services — HTTP ingress, async Kafka consumer, downstream gRPC call — the conversation about observability investment changes. You stop guessing where latency lives and start measuring it.

---

## Conclusion

You do not need a five-figure observability contract. Wire the OpenTelemetry SDK into Ktor, carry context explicitly through your coroutine graph, and point the OTLP exporter at a self-hosted Grafana Tempo instance. That is production-grade distributed tracing for under $50/month.

**Further reading:**
- [OpenTelemetry Kotlin SDK](https://opentelemetry.io/docs/languages/java/)
- [Grafana Tempo docs](https://grafana.com/docs/tempo/latest/)
- [TraceQL reference](https://grafana.com/docs/tempo/latest/traceql/)