Tail Sampling with OpenTelemetry, Measured: It Keeps What Matters Until You Scale It or Shrink It

Logs, Metrics and Traces Are Not Three Pillars. They're One Request. ended on an uncomfortable number. A 10% head sampler decides whether to keep a trace when the request arrives, before anyone knows it will be slow. So of 20 slow checkouts, it kept about 2, and the histogram that told you about the slow ones pointed at traces that no longer existed. I promised the fix in that post, and again in the structured logging one: decide at the end of the trace instead. That is tail sampling, and this is the post.

Tail sampling does what it promises. On my run it kept every slow trace and every failed one, while dropping 90% of the rest. It also has two failure modes that a single-Collector demo never shows, and both lose exactly the traces you added it for: a buffer that is too small, and a second Collector.

Everything here runs locally: three Node services, the OpenTelemetry Collector (contrib 0.164.0) and Jaeger 2.22, in Docker.

The system, and how to know what was kept

Aurora Coffee Co.'s checkout calls pricing, then inventory. Two things go wrong on purpose, at known rates, so every result can be checked against the truth:

• 1 order in 50 is slow. Seasonal SKUs make pricing wait 600 ms on its promotion rules.
• 1 order in 100 fails. One inventory warehouse shard answers 503, and checkout answers 502.

The services are plain node:http. Node 24 and later run these TypeScript files directly, with no build step:

{
  "name": "aurora-tail-sampling",
  "private": true,
  "type": "module",
  "engines": { "node": ">=24" },
  "dependencies": {
    "@opentelemetry/api": "1.10.0",
    "@opentelemetry/auto-instrumentations-node": "0.82.0",
    "@opentelemetry/exporter-trace-otlp-http": "0.300.0",
    "@opentelemetry/sdk-node": "0.300.0",
    "import-in-the-middle": "3.5.3"
  }
}

// pricing.ts — quotes a price. Seasonal SKUs wait on a slow promotion-rules lookup.
import { createServer } from 'node:http'
import { setTimeout as sleep } from 'node:timers/promises'

const port = Number(process.env.PORT ?? 3001)

createServer(async (req, res) => {
  const sku = new URL(req.url ?? '/', 'http://pricing').searchParams.get('sku') ?? ''
  // Stands in for the real cost: 2,400 promotion rules against 12 for a house blend.
  if (sku.startsWith('seasonal-')) await sleep(600)
  res.writeHead(200, { 'content-type': 'application/json' })
  res.end(JSON.stringify({ sku, cents: sku.startsWith('seasonal-') ? 2100 : 1600 }))
}).listen(port)

// inventory.ts — reserves stock. One warehouse shard is unavailable for 1 order in 100.
import { createServer } from 'node:http'

const port = Number(process.env.PORT ?? 3002)

createServer((req, res) => {
  const order = Number(new URL(req.url ?? '/', 'http://inventory').searchParams.get('order'))
  if (order % 100 === 37) {
    res.writeHead(503, { 'content-type': 'application/json' })
    res.end(JSON.stringify({ error: 'warehouse shard unavailable' }))
    return
  }
  res.writeHead(200, { 'content-type': 'application/json' })
  res.end(JSON.stringify({ order, reserved: true }))
}).listen(port)

// checkout.ts — the front door: prices the order, then reserves stock.
import { createServer } from 'node:http'
import { trace } from '@opentelemetry/api'

const port = Number(process.env.PORT ?? 3000)
const pricingUrl = process.env.PRICING_URL ?? 'http://localhost:3001'
const inventoryUrl = process.env.INVENTORY_URL ?? 'http://localhost:3002'

async function call(url: string): Promise<unknown> {
  const response = await fetch(url, { signal: AbortSignal.timeout(5_000) })
  if (!response.ok) throw new Error(`${url} answered ${response.status}`)
  return response.json()
}

createServer(async (req, res) => {
  const order = Number(new URL(req.url ?? '/', 'http://checkout').searchParams.get('order'))
  // One order in 50 is a seasonal SKU, the slow path in pricing.
  const sku = order % 50 === 0 ? `seasonal-${order}` : `house-${order % 7}`
  // Returned so the load generator can look the trace up in Jaeger afterwards.
  res.setHeader('x-trace-id', trace.getActiveSpan()?.spanContext().traceId ?? '')

  try {
    const price = await call(`${pricingUrl}/quote?sku=${sku}`)
    await call(`${inventoryUrl}/reserve?order=${order}`)
    res.writeHead(200, { 'content-type': 'application/json' })
    res.end(JSON.stringify({ order, price }))
  } catch (error) {
    res.writeHead(502, { 'content-type': 'application/json' })
    res.end(JSON.stringify({ order, error: error instanceof Error ? error.message : 'checkout failed' }))
  }
}).listen(port)

The tracing setup is two small files, loaded before the service with two --import flags. The first installs the loader hook that lets auto-instrumentation patch ES modules:

// register.ts — loaded first: `node --import ./register.ts --import ./telemetry.ts service.ts`.
import { register } from 'node:module'
import { register as registerSyncHooks, supportsSyncHooks } from 'import-in-the-middle/register-hooks.mjs'

// Node 24.11.1+ and 26 run the hook in-thread with module.registerHooks(). module.register(),
// the older off-thread loader, is deprecated in Node 26 and kept only as the fallback.
if (supportsSyncHooks()) registerSyncHooks()
else register('@opentelemetry/instrumentation/hook.mjs', import.meta.url)

The second starts the SDK:

// telemetry.ts — loaded second, after register.ts, and before any service code.
import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'

const sdk = new NodeSDK({
  serviceName: process.env.OTEL_SERVICE_NAME ?? 'unnamed-service',
  // Keep host names, user names and PIDs out of every span.
  autoDetectResources: false,
  traceExporter: new OTLPTraceExporter({
    url: `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318'}/v1/traces`,
  }),
  instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false } })],
})

sdk.start()

for (const signal of ['SIGINT', 'SIGTERM'] as const) {
  process.once(signal, () => {
    void sdk.shutdown().finally(() => process.exit(0))
  })
}

The hook is the part that changed since September, and it is now a file of its own for a reason. Without a hook, auto-instrumentation never sees node:http under ESM and every service starts its own trace. The September post used module.register(), which Node 26 deprecates with a warning on every start. import-in-the-middle, the library behind OpenTelemetry's hook, ships an in-thread version built on module.registerHooks(), and supportsSyncHooks() says whether the running Node can use it. It has to run in its own --import: static import statements are evaluated before the code of the file that contains them, so a hook registered at the top of telemetry.ts would run after that file's imports had already loaded.

The ground truth

To score a sampler you need to know which requests were slow or failed, independently of the traces. The load generator records that from the client side, along with each request's trace ID, which checkout returns in a header:

// load.ts — sends N checkouts and records what really happened to each one.
// Usage: node load.ts <requests> <concurrency> <out.json>
import { writeFileSync } from 'node:fs'

type Outcome = { order: number; traceId: string; status: number; ms: number }

const [requests = '1000', concurrency = '10', out = 'outcomes.json'] = process.argv.slice(2)
const total = Number(requests)
const outcomes: Outcome[] = []
let next = 1

async function worker(): Promise<void> {
  while (next <= total) {
    const order = next++
    const started = performance.now()
    const response = await fetch(`http://localhost:3000/checkout?order=${order}`)
    await response.arrayBuffer()
    outcomes.push({ order, traceId: response.headers.get('x-trace-id') ?? '', status: response.status, ms: Math.round(performance.now() - started) })
  }
}

const started = performance.now()
await Promise.all(Array.from({ length: Number(concurrency) }, worker))
writeFileSync(out, JSON.stringify(outcomes))
const slow = outcomes.filter((o) => o.ms >= 500).length
const failed = outcomes.filter((o) => o.status >= 500).length
console.log(`${outcomes.length} checkouts in ${((performance.now() - started) / 1000).toFixed(1)} s: ${slow} over 500 ms, ${failed} failed`)

Then a second script asks Jaeger, trace ID by trace ID, whether it has each trace, and whether the trace is whole. A checkout in this system is exactly five spans: checkout's server span, its two outgoing calls, and the pricing and inventory server spans. That second question matters later.

// measure.ts — asks Jaeger which of the recorded traces it actually has, and whether they are whole.
// Usage: node measure.ts <outcomes.json>
import { readFileSync } from 'node:fs'

type Outcome = { order: number; traceId: string; status: number; ms: number }
type JaegerTrace = { spans: unknown[] }

const outcomes: Outcome[] = JSON.parse(readFileSync(process.argv[2] ?? 'outcomes.json', 'utf8'))
const jaeger = process.env.JAEGER_URL ?? 'http://localhost:16686'
// Checkout's server span, its two fetch calls, and the pricing and inventory server spans.
const spansPerCheckout = 5

async function lookup(traceId: string): Promise<JaegerTrace | null> {
  const response = await fetch(`${jaeger}/api/traces/${traceId}`)
  if (response.status === 404) return null
  const body = (await response.json()) as { data?: JaegerTrace[] }
  return body.data?.[0] ?? null
}

const rows = new Map<string, { requests: number; kept: number; whole: number }>()
for (const outcome of outcomes) {
  const kind = outcome.status >= 500 ? 'failed' : outcome.ms >= 500 ? 'slow' : 'normal'
  const row = rows.get(kind) ?? { requests: 0, kept: 0, whole: 0 }
  row.requests++
  const found = await lookup(outcome.traceId)
  if (found) {
    row.kept++
    if (found.spans.length === spansPerCheckout) row.whole++
  }
  rows.set(kind, row)
}

console.log('kind      requests  in Jaeger  whole')
for (const kind of ['slow', 'failed', 'normal']) {
  const row = rows.get(kind) ?? { requests: 0, kept: 0, whole: 0 }
  console.log(`${kind.padEnd(9)} ${String(row.requests).padStart(8)}  ${String(row.kept).padStart(9)}  ${String(row.whole).padStart(5)}`)
}

The stack

One Compose file covers every scenario in this post. Each one is a profile, so you bring up exactly the Collectors it needs:

# One Collector image, five roles. `docker compose --profile <name> up -d` picks the scenario.
x-collector: &collector
  image: otel/opentelemetry-collector-contrib:0.164.0
  command: ["--config=/etc/otelcol/config.yaml"]
  # memory_limiter counts Go heap; this limit counts the whole process, which ran at about
  # 1.35x the heap in these runs. 2 GiB leaves room above limit_mib for that difference.
  mem_limit: 2048m

services:
  jaeger:
    image: jaegertracing/jaeger:2.22.0
    ports:
      - "16686:16686"

  # head: the SDK decides, this Collector only forwards.
  forward:
    <<: *collector
    profiles: [head]
    volumes: ["./collector-forward.yaml:/etc/otelcol/config.yaml:ro"]
    ports: ["4318:4318"]

  # tail: one Collector decides after seeing the whole trace.
  tail:
    <<: *collector
    profiles: [tail]
    volumes: ["${TAIL_CONFIG:-./collector-tail.yaml}:/etc/otelcol/config.yaml:ro"]
    ports: ["4318:4318", "8888:8888"]

  # split: two tail samplers, each fed directly by the services that point at it.
  tail-a:
    <<: *collector
    profiles: [split]
    volumes: ["./collector-tail.yaml:/etc/otelcol/config.yaml:ro"]
    ports: ["4318:4318"]
  tail-b:
    <<: *collector
    profiles: [split]
    volumes: ["./collector-tail.yaml:/etc/otelcol/config.yaml:ro"]
    ports: ["4319:4318"]

  # routed: two agents route by trace ID to two samplers.
  agent-a:
    <<: *collector
    profiles: [routed]
    volumes: ["./collector-agent.yaml:/etc/otelcol/config.yaml:ro"]
    ports: ["4318:4318"]
  agent-b:
    <<: *collector
    profiles: [routed]
    volumes: ["./collector-agent.yaml:/etc/otelcol/config.yaml:ro"]
    ports: ["4319:4318"]
  sampler-1:
    <<: *collector
    profiles: [routed]
    volumes: ["./collector-tail.yaml:/etc/otelcol/config.yaml:ro"]
  sampler-2:
    <<: *collector
    profiles: [routed]
    volumes: ["./collector-tail.yaml:/etc/otelcol/config.yaml:ro"]

Each scenario below runs the same way: bring up its profile, start the three services, send the load, wait for the Collectors to decide, measure.

docker compose --profile tail up -d

OTEL_SERVICE_NAME=pricing   PORT=3001 node --import ./register.ts --import ./telemetry.ts pricing.ts &
OTEL_SERVICE_NAME=inventory PORT=3002 node --import ./register.ts --import ./telemetry.ts inventory.ts &
OTEL_SERVICE_NAME=checkout  PORT=3000 node --import ./register.ts --import ./telemetry.ts checkout.ts &

node load.ts 1000 10 outcomes.json
sleep 25   # the SDK batches for 5 s, the sampler waits 10 s, plus margin
node measure.ts outcomes.json

Head sampling, the baseline

First the September setup: the SDK keeps 10% of traces, chosen when the request arrives, and the Collector just forwards what it gets:

# Forwards everything it receives. Used with head sampling, where the SDK already decided.
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  otlp_grpc:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_grpc]

With docker compose --profile head up -d, and every service started with OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1:

1000 checkouts in 2.8 s: 20 over 500 ms, 10 failed
kind      requests  in Jaeger  whole
slow            20          2      2
failed          10          1      1
normal         970         95     95

That is the September number again, with the other half of it: of 20 slow checkouts, 2 have a trace, and of the 10 failed checkouts, 1 does. With a 10% sampler, nine customer complaints out of ten point at a trace that was never kept. Head sampling keeps a random 10% of everything, and the requests you care about are a small, non-random slice of everything.

Tail sampling: decide after the trace is complete

The tail_sampling processor holds each trace's spans in memory, waits, and then runs its policies against the complete trace. A trace is kept if any policy says yes:

# Keeps every slow trace, every failed trace, and 10% of the rest.
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # First in the pipeline: refuses data before the Collector runs out of memory and loses
  # every trace it is holding.
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
    spike_limit_mib: 256
  tail_sampling:
    # How long to wait for a trace's spans before deciding. Longer than the slowest request
    # plus the SDKs' export delay (5 s by default), or late spans miss the decision.
    decision_wait: 10s
    # How many traces to track. Past this, the oldest are evicted, decided or not.
    num_traces: 50000
    policies:
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: failed
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch: {}

exporters:
  otlp_grpc:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp_grpc]

The services now send every span (the SDK's default sampler keeps everything), and docker compose --profile tail up -d:

1000 checkouts in 2.2 s: 20 over 500 ms, 10 failed
kind      requests  in Jaeger  whole
slow            20         20     20
failed          10         10     10
normal         970         99     99

All 20 slow traces and all 10 failed ones, each with all five spans, while the baseline policy still drops roughly 90% of the healthy traffic. Jaeger got about as many traces as with head sampling (129 against 98, a third more), and now they are the right ones.

Two things in that config are worth a second look. The failed policy works without touching the service code because the instrumentation marks spans as errors on its own: an HTTP server span when it answers 5xx, and an outgoing fetch span when the answer is 400 or above. That second rule matters in production: a downstream 404 that your code handles on purpose still makes the whole trace "failed", and status_code will keep it. And exporter: otlp_grpc is not a typo: the Collector renamed the otlp exporter in 0.144, and the old name still works but logs a deprecation warning on every start.

What it costs: every span now leaves every service, so the network and the Collector carry 100% of the traffic that head sampling used to cut at the source. In September I measured about 889 bytes per span on the wire. Tail sampling moves the savings from the network to the storage.

What the buffer costs, and what a small one loses

decision_wait and num_traces look like two sizes for the same buffer. They are not, and only one of them is what costs memory.

To see what it holds, I sampled the Collector's own metrics (port 8888 in the config above) every half second with this script:

// monitor.ts — samples the tail-sampling Collector's own metrics every 500 ms until stopped,
// then prints the peaks. Usage: node monitor.ts <seconds>
const seconds = Number(process.argv[2] ?? 60)
const peaks = { tracesInMemory: 0, rssMiB: 0, heapMiB: 0 }
let droppedTooEarly = 0
let refusedSpans = 0

// Sums every series of a metric, whatever its labels.
function read(text: string, name: string): number {
  return text
    .split('\n')
    .filter((l) => l.startsWith(name + ' ') || l.startsWith(name + '{'))
    .reduce((sum, l) => sum + Number(l.slice(l.lastIndexOf(' ') + 1)), 0)
}

const deadline = Date.now() + seconds * 1000
while (Date.now() < deadline) {
  const text = await (await fetch('http://localhost:8888/metrics')).text()
  peaks.tracesInMemory = Math.max(peaks.tracesInMemory, read(text, 'otelcol_processor_tail_sampling_sampling_traces_on_memory'))
  peaks.rssMiB = Math.max(peaks.rssMiB, read(text, 'otelcol_process_memory_rss') / 2 ** 20)
  peaks.heapMiB = Math.max(peaks.heapMiB, read(text, 'otelcol_process_runtime_heap_alloc_bytes') / 2 ** 20)
  droppedTooEarly = read(text, 'otelcol_processor_tail_sampling_sampling_trace_dropped_too_early')
  // Spans the memory_limiter turned away at the receiver.
  refusedSpans = read(text, 'otelcol_receiver_refused_spans')
  await new Promise((resolve) => setTimeout(resolve, 500))
}
console.log(`peak traces held: ${peaks.tracesInMemory}, peak RSS: ${peaks.rssMiB.toFixed(0)} MiB, peak heap: ${peaks.heapMiB.toFixed(0)} MiB, dropped too early: ${droppedTooEarly}, spans refused: ${refusedSpans}`)

Two runs with the same traffic, 60,000 checkouts at a little over 1,000 a second, first with decision_wait: 10s:

60000 checkouts in 52.0 s: 1200 over 500 ms, 600 failed
peak traces held: 50000, peak RSS: 564 MiB, peak heap: 333 MiB, dropped too early: 0, spans refused: 0

Then with decision_wait: 30s (TAIL_CONFIG=./collector-tail-30s.yaml, the same file with one line changed):

60000 checkouts in 59.1 s: 1200 over 500 ms, 600 failed
peak traces held: 50000, peak RSS: 913 MiB, peak heap: 687 MiB, dropped too early: 0, spans refused: 0

(The load line counts slowness as the client saw it. With the services, the Collector and the load generator on one laptop, a few dozen extra requests can cross 500 ms in a run; the design produces exactly 1,200.)

Two numbers move and one does not. The Collector tracked 50,000 traces in both runs: that is num_traces, and the buffer is a circular buffer, so it fills to that size and stays full. But tracking is cheaper than holding. Once a trace is decided, the processor releases its spans and keeps only a record of the decision, so that spans arriving late can inherit it until a newer trace pushes the record out. The expensive part is the traces still waiting, held with every span. The two runs went at about 1,150 and 1,020 checkouts a second, so roughly 11,500 complete traces were waiting at any moment with a 10-second wait, and 30,500 with a 30-second one. The heap went from 333 MiB to 687 MiB.

So the memory model is: traces per second × decision_wait × the size of a trace, plus a record per num_traces slot. Rough arithmetic from these runs, not a measurement of its own: about 19 KB of heap per waiting five-span trace, and a few KB per slot. If the slots matter to you, the processor's decision_cache option keeps decisions in a separate cache sized on its own and releases the slot.

The wait is what you pay for, and that is why the config puts memory_limiter first and gives the container room above it. spans refused: 0 in both runs means the limiter never stepped in, but the 30-second run peaked at 687 MiB of heap against a soft limit of 768: about 12% more traffic and it would have. When it does, it cannot shrink the traces already waiting. It refuses new spans, the SDKs retry and then drop them once their queues fill, and you get incomplete traces instead of an out-of-memory kill that loses everything. Alert on otelcol_receiver_refused_spans above zero; it means you are already losing data.

The other knob has its own trap. The tempting way to save memory is a smaller buffer. Here is num_traces: 1000 (TAIL_CONFIG=./collector-tail-small.yaml) with 20,000 checkouts:

20000 checkouts in 20.8 s: 400 over 500 ms, 200 failed
peak traces held: 1000, peak RSS: 253 MiB, peak heap: 66 MiB, dropped too early: 19000, spans refused: 0
kind      requests  in Jaeger  whole
slow           400         25     25
failed         200          9      9
normal       19400         98     98

19,000 traces were dropped before any policy saw them, which is exactly 20,000 minus the buffer. A buffer of 1,000 holds about a second of this traffic, so every trace was pushed out before its 10 seconds were up, except the last thousand to arrive, which survived only because the load stopped and nothing came after them. That is not quite the last thousand orders. A slow checkout's trace starts arriving 600 ms later than a fast one sent at the same moment, so the 25 slow traces that survived reach back to order 18,750, and they pushed out the first failure among the last thousand orders. Under steady traffic, the same mechanism would leave close to nothing, and the only sign of it is a counter.

The rules that fall out:

• num_traces has to cover at least peak new traces per second × decision_wait, with margin. Alert on otelcol_processor_tail_sampling_sampling_trace_dropped_too_early above zero, and watch otelcol_processor_tail_sampling_sampling_trace_removal_age: when it gets close to decision_wait, you are about to start dropping.
• decision_wait has to cover the slowest request plus the time the slowest span takes to reach the Collector. The JS SDK batches spans for up to 5 seconds before sending them, so a 600 ms request can have spans arriving more than 5 seconds after its first one. That is why this config waits 10.
• If the memory for that is too much, shorten the wait only as far as those two delays allow. Past that, the latency policy starts deciding on incomplete traces.

The second Collector

One Collector runs out of capacity eventually, so the next step is two. The common way to get two is an agent per node: each service sends to the Collector next to it. Here, checkout and inventory send to tail-a, and pricing to tail-b (the same tail-sampling config, started with --profile split, and pricing started with OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4319):

1000 checkouts in 2.2 s: 20 over 500 ms, 10 failed
kind      requests  in Jaeger  whole
slow            20         20     20
failed          10         10      0
normal         970        103    103

Every failed trace reached Jaeger, and none of the 10 was whole. All ten arrived without their pricing span.

Each Collector applies its policies to the part of the trace it received. tail-a saw checkout and inventory: the 503 and the 502, so failed kept them. tail-b saw only pricing, where nothing failed, so the error policy said no and only the 10% baseline could save that span. In this run it saved none; in a run before it, with the same config, it saved three.

The slow traces survived whole, but not because this setup works. The slow span is in pricing, and checkout waited for it, so both halves were over 500 ms and both Collectors kept their half. The healthy traces are whole too, for a subtler reason: the probabilistic policy decides by hashing the trace ID, so both Collectors make the same choice for the same trace. Any policy that looks at the spans themselves (latency, status, attributes) can disagree between the two halves of a trace, and the trace you get in Jaeger is the half that matched.

The fix: route by trace ID

Every span of a trace has to reach the same sampler. The Collector does that with a second tier. The first tier samples nothing and routes, with the load_balancing exporter keyed on trace ID:

# First tier: receives spans and routes every span of a trace to the same sampler.
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      static:
        hostnames:
          - sampler-1:4317
          - sampler-2:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [load_balancing]

The services keep exactly the same split as before, pricing on one agent and the rest on the other (--profile routed):

1000 checkouts in 3.6 s: 20 over 500 ms, 10 failed
kind      requests  in Jaeger  whole
slow            20         20     20
failed          10         10     10
normal         970         99     99

Ten failed traces, ten whole. The agents are stateless, so you can run as many as you like. The samplers are where the state lives, and the routing is what makes each one see whole traces.

This is a scaling fix, not a high-availability one. Each sampler owns a slice of the trace IDs and holds that slice in memory, so a sampler that restarts loses its slice. And with the static resolver and this config, the agents have nowhere else to send that slice while the sampler is down: the exporter's sending queue and retries are off unless you turn them on.

In production the static resolver becomes the dns or k8s resolver, so samplers can be added without editing the agents. Routing uses consistent hashing, so adding or removing a sampler moves only a share of the trace IDs, not all of them. But the exporter's README warns of a window in which the agents' view of the samplers does not match reality, and a trace in flight while its IDs move can still be split. Scale samplers deliberately, not on a CPU autoscaler that changes the list every few minutes.

When head sampling is still the right call

Tail sampling is not free, and it is not always the answer:

• At very high volume, the cost moves rather than disappears. Every span leaves every service, and every span waits in a sampler's memory for the whole decision_wait. If you run hundreds of thousands of requests per second, a head sampler in front (say, keep 25% at the SDK) with tail sampling behind it can be the only affordable shape. You lose some slow traces, knowingly.
• If nobody looks at individual traces, and your questions are answered by metrics and logs, head sampling at a low rate is cheaper and simpler.
• If you cannot run a stateful tier, a single tail-sampling Collector with no redundancy is a real risk: when it restarts, every trace it was holding is gone.

Several paid observability platforms offer tail-based sampling as a feature, run on their side. If they bill by ingested spans, you still pay for every span you send, which is the same trade-off on someone else's invoice.

Key Takeaways

---

• Head sampling loses the requests you care about. A 10% head sampler kept 2 of 20 slow traces and 1 of 10 failed ones. Tail sampling, deciding after the trace is complete, kept all 30 while still dropping about 90% of healthy traffic.
• Memory is traffic × decision_wait. The Collector holds every span of every undecided trace and only a stub of the decided ones: at the same traffic, a 30-second wait took the heap from 333 MiB to 687 MiB. Put memory_limiter first, give the container room above it, and alert on refused spans.
• A buffer that is too small drops everything. With num_traces: 1000, 19,000 of 20,000 traces were evicted before any decision; only the last thousand to arrive survived, because the load stopped. Cover peak traces per second × decision_wait, and alert on sampling_trace_dropped_too_early.
• Two tail samplers without routing break your traces. Each one judges the part it sees: all 10 failed traces lost their pricing span. Only policies that decide by trace ID agree, and only because they ignore the spans.
• Route by trace ID before you sample. A stateless tier with the load_balancing exporter in front of the samplers made all ten failed traces whole again.

---