How to Configure Tracer Components in Redpanda Connect for Distributed Tracing

Distributed data pipelines have become a core building block of modern event-driven architectures. As these pipelines grow in complexity with multiple processors filtering, transforming, and routing messages, debugging them becomes a real challenge. When a message goes missing or a processor behaves unexpectedly, you need visibility into exactly where things broke down and how long each step took.

Distributed tracing solves this problem. Instead of relying solely on logs that tell you something happened, traces show you the full journey of every message through your pipeline. I'm going to show you how to implement that kind of tracing using Redpanda Connect's tracer component, which writes OpenTelemetry-compatible trace data directly to a Redpanda topic, so you can consume trace data the same way you consume any other event stream.

Here's what I'll walk you through:

• Configure the Redpanda Connect tracer component with a Jaeger backend
• Set up the OpenTelemetry Collector tracer for a backend-agnostic approach
• Build a multi-step pipeline that ingests, filters, transforms, and routes application events
• Explore sampling strategies for controlling trace volume in production
• Visualize trace spans in the Jaeger UI to identify bottlenecks in your pipeline

Configuring Tracer Components in Redpanda Connect for Distributed Tracing

Suppose that you work for an online payment processing company that handles thousands of transaction events per second. These events come in different types such as purchase, refund, and chargeback. You've got a Redpanda Connect pipeline that ingests these events from a Redpanda input topic, filters out invalid transactions, enriches each event with additional metadata, and routes them to different output topics based on event type.

The pipeline works well, but you keep running into the same problem: when a transaction takes longer than expected or silently fails to reach its output topic, there's no easy way to pinpoint which processor caused the delay or failure. Multi-step pipelines with conditional logic are exactly where bottlenecks and silent failures hide.

The following diagram shows the high-level architecture of the pipeline with tracing enabled:

By enabling the Redpanda tracer component, every transaction event that flows through the pipeline produces a trace. Each processor in the chain generates a child span, a nested timing record showing how long that step took and whether it succeeded. That lets you see exactly how long filtering, transformation, and routing take for each event. If a refund event takes three times longer than a purchase event, the trace data will show exactly which processor is responsible.

Prerequisites

You'll need the following:

• Docker and Docker Compose (Docker Engine 29 or higher)
• Redpanda Connect 4.88.0 or higher installed via rpk connect install or brew install redpanda-data/tap/redpanda
• rpk CLI for managing Redpanda topics and consuming trace data
• Basic familiarity with Redpanda Connect's YAML pipeline configuration
• Jaeger 2.17.0 or higher (runs via Docker Compose in this tutorial)

How the Redpanda Connect Tracer Component Works

Every Redpanda Connect pipeline config supports an optional top-level tracer block. When that block is present and pointing at a live backend, the runtime allocates a root span for each message the moment it enters the input. As the message flows through your processor chain, each processor appends a child span to that root. The child span captures the processor type, its execution duration, and any error status if the processor fails.

A single message trace produces a nested span hierarchy flowing from input through each processor to output:

Because trace context propagates through Kafka message headers, a Redpanda Connect pipeline sitting in the middle of a larger distributed system can both receive an upstream trace context and pass it downstream. You get end-to-end traces across multiple services, extending well beyond the pipeline boundary.

The tracer block is a sibling to input, pipeline, and output in your config file. Adding it activates trace emission without changing any pipeline behavior.

Setting Up the Demo Pipeline

I'm using a synthetic data pipeline for these examples: a generate input emitting JSON events, a bloblang processor enriching each event, a log processor confirming the transformation, and a stdout output. This setup makes it easy to observe trace structure without any external Kafka dependency.

Run the following command to create a workspace directory and navigate into it:

mkdir -p ~/redpanda-tracer-tutorial && cd ~/redpanda-tracer-tutorial

Create a file named 01-basic-pipeline.yaml with the following content:

input:
  generate:
    mapping: |
      root.user_id = uuid_v4()
      root.event = "purchase"
      root.amount = random_int(min: 1, max: 500)
    interval: 1s
    count: 20

pipeline:
  processors:
    - bloblang: |
        root = this
        root.currency = "USD"
        root.processed_at = now()
    - log:
        message: "Processed event for user ${!json(\"user_id\")}"

output:
  stdout: {}

The generate input creates 20 synthetic events at one-second intervals, each with a user_id, an event type, and a random amount. The bloblang processor enriches each event by adding currency and processed_at fields, and the log processor prints a confirmation message for each processed event.

Run the following command to confirm the pipeline works before adding tracing:

rpk connect run 01-basic-pipeline.yaml

You should see 20 JSON objects emitted to stdout. Once the base pipeline is confirmed, you're ready to add the tracer block.

Configuring the Jaeger Tracer in Redpanda Connect

Jaeger remains one of the most common local tracing backends for development. Redpanda Connect ships a native jaeger tracer type that communicates with a Jaeger agent over UDP (port 6831 by default) or with a Jaeger collector over HTTP.

Download the following docker-compose.yml file to start a local Jaeger all-in-one instance:

curl -O https://raw.githubusercontent.com/SystemCraftsman/redpanda-connect-configuring-tracer-components-demo/main/docker/docker-compose.yml

Run the following command to start Jaeger:

docker compose up jaeger

The all-in-one image exposes the Jaeger UI at http://localhost:16686, accepts HTTP collector traffic on port 14268, and accepts UDP agent traffic on port 6831.

Copy 01-basic-pipeline.yaml to a new file named 02-jaeger-tracing.yaml and add the following tracer block at the end:

tracer:
  jaeger:
    collector_url: http://localhost:14268/api/traces
    sampler_type: const
    sampler_param: 1
    tags:
      pipeline: purchase-events
      env: local

The collector_url field sends spans via HTTP to the Jaeger collector endpoint. This is the recommended approach for local Docker setups because the UDP-based agent_address option can be unreliable on the macOS Docker Desktop. sampler_type: const with sampler_param: 1 traces every message, which is appropriate for local development where you want complete visibility. The tags map attaches arbitrary key-value pairs to every span from this pipeline, which helps when you're running multiple pipelines and filtering traces in the Jaeger UI by service or environment.

Run the following command to start the traced pipeline:

rpk connect run 02-jaeger-tracing.yaml

After the 20 messages process, open http://localhost:16686, select the service named benthos (the default service name for Redpanda Connect pipelines), and click Find Traces. Each trace represents a single generated message. Click into any trace to see the root span covering the full pipeline execution time, with child spans for the bloblang processor and the log processor nested underneath.

To verify that traces are being collected, you should see 20 traces listed in the Jaeger UI. Each trace must contain a root span and two child spans corresponding to the bloblang and log processors.

The agent_address field is an alternative that sends spans via UDP to a Jaeger agent (requires port 6831):

tracer:
  jaeger:
    agent_address: localhost:6831
    sampler_type: const
    sampler_param: 1

Use agent_address in environments where a local Jaeger agent sidecar is already running and UDP connectivity is reliable. In Kubernetes environments, most teams prefer collector_url pointing at a centralized collector.

Configuring the OpenTelemetry Collector Tracer

The open_telemetry_collector tracer type sends spans via the OpenTelemetry Protocol (OTLP) over gRPC or HTTP. This approach is backend-agnostic, pointing Redpanda Connect at an OTel Collector that routes spans to Jaeger, Grafana Tempo, Honeycomb, Datadog, or any other OTLP-compatible destination.

Download the following otel-collector-config.yaml file. It accepts OTLP traffic on port 4317 (gRPC) and port 4318 (HTTP), applies a batch processor, and exports spans to the local Jaeger instance:

curl -O https://raw.githubusercontent.com/SystemCraftsman/redpanda-connect-configuring-tracer-components-demo/main/docker/otel-collector-config.yaml

The downloaded file should have the following content:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  otlp_grpc/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_grpc/jaeger]

Run the following command to start both Jaeger and the OTel Collector together:

docker compose up

Copy 01-basic-pipeline.yaml to a new file named 03-otel-tracing.yaml and add the following tracer block at the end:

tracer:
  open_telemetry_collector:
    service: purchase-pipeline
    grpc:
      - address: localhost:4317
    tags:
      env: local
    sampling:
      enabled: false

The service field sets the service name that appears in your tracing UI. Changing it from the default benthos to something descriptive (like purchase-pipeline) makes it much easier to locate spans when you have multiple pipelines sending to the same collector.

Note that the grpc list accepts multiple endpoints, which is useful when your collector runs behind a load balancer or when you need redundancy. Each entry takes an address and an optional secure field for TLS.

Run the following command to start the pipeline:

rpk connect run 03-otel-tracing.yaml

To verify, open the Jaeger UI at http://localhost:16686 and search for the purchase-pipeline service. The span structure is identical to what you saw with the native Jaeger tracer: a root span per message with processor child spans nested underneath. You should see the traces listed under the purchase-pipeline service name instead of the default benthos.

The HTTP transport option works the same way, using port 4318 by OTLP convention. Make sure port 4318 is exposed in your docker-compose.yml (the compose file used in this tutorial already includes it):

tracer:
  open_telemetry_collector:
    service: purchase-pipeline
    http:
      - address: localhost:4318

Note that the address value does not include a protocol prefix. Redpanda Connect handles the HTTP transport internally based on the http block.

Use HTTP when your environment restricts gRPC traffic or when your OTel Collector only exposes an HTTP endpoint.

Choosing a Sampling Strategy for Redpanda Connect Traces

Sampling controls what fraction of traces gets exported. At high message volumes, exporting every span is expensive in both network overhead and backend storage. The OpenTelemetry sampling documentation covers the tradeoffs in depth.

The jaeger tracer supports two sampler types you'll use most often. With sampler_type: const and sampler_param: 1, every message produces a trace. Setting sampler_param: 0 disables tracing completely without removing the tracer block, which is useful when you want to keep the config ready but temporarily stop generating spans.

For probabilistic sampling, set sampler_type: probabilistic and give it a ratio:

tracer:
  jaeger:
    collector_url: http://localhost:14268/api/traces
    sampler_type: probabilistic
    sampler_param: 0.1

sampler_param: 0.1 samples approximately 10% of messages. That's roughly where I'd start for production pipelines doing more than 10,000 messages per second, but you can always tighten it once you see real trace volume.

The open_telemetry_collector tracer has its own sampling block:

tracer:
  open_telemetry_collector:
    service: purchase-pipeline
    grpc:
      - address: otel-collector:4317
    sampling:
      enabled: true
      ratio: 0.1

The ratio field takes a float value between 0 and 1. Wrapping it in quotes causes an unmarshal error because the parser expects a numeric type. Setting enabled: false (the default) passes all spans through to the collector and lets the collector make sampling decisions, which is the preferred pattern when you want centralized sampling policy control across multiple services via the OTel Collector's tail sampling processor.

Configuring Schema Registry for Trace Data

In the previous examples, I used generate and stdout with no external dependencies. The Redpanda tracer type, however, writes trace data to a Redpanda topic, so it requires a running Redpanda cluster with broker and schema registry endpoints accessible. You can start a local Redpanda cluster using rpk container or add a Redpanda service to your docker-compose.yml.

By default, the Redpanda tracer emits trace data in JSON format. While this works well for development, serializing trace data with Redpanda's schema registry matters for downstream consumers that need a consistent and validated schema to parse traces reliably.

To enable schema registry serialization, set format to schema-registry-json and configure the schema_registry.url in your tracer block:

tracer:
  redpanda:
    seed_brokers:
      - localhost:9092
    topic: otel-traces
    format: schema-registry-json
    schema_registry:
      url: http://localhost:8081
    service: purchase-pipeline
    tags:
      env: local

Notice that the format field supports several options: json, protobuf, schema-registry-json, and schema-registry-protobuf. The schema-registry-json option registers the trace schema with the schema registry and serializes each trace message accordingly.

You can verify the deserialized trace output by consuming the otel-traces topic with rpk:

rpk topic consume otel-traces --format json

You should see the following output with trace spans in JSON format:

{
  "trace_id": "abc123...",
  "span_id": "def456...",
  "operation_name": "bloblang",
  "service_name": "purchase-pipeline",
  "duration_ms": 12,
  "tags": {
    "env": "local"
  },
  "child_spans": [...]
}

Note: While the tracer config API is stable, the format of spans, tags, and logs is subject to change. Avoid building brittle downstream parsers that depend on specific span field names.

Adding Compression and Shutdown Delay

For production environments where trace volume can be significant, you should enable compression to reduce bandwidth and storage costs. Add compression: lz4 to your tracer configuration:

tracer:
  redpanda:
    seed_brokers:
      - localhost:9092
    topic: otel-traces
    format: schema-registry-json
    schema_registry:
      url: http://localhost:8081
    compression: lz4
    service: purchase-pipeline
    sampling:
      enabled: true
      ratio: 0.05
    tags:
      env: production

shutdown_delay: "5s"

Notice that compression supports several codecs: lz4, snappy, gzip, zstd, and none. The lz4 codec provides a good balance between compression ratio and CPU overhead.

Keep in mind that shutdown_delay is a root-level field in the pipeline configuration, not inside the tracer block. It gives the tracer time to flush remaining spans before the process exits. Without it, you may lose traces that are still buffered when the pipeline shuts down. While the shutdown delay is in effect, the HTTP metrics endpoint continues to be available for scraping and any active tracers are free to flush remaining traces.

IMPORTANT: The Redpanda Connect documentation recommends a sampling.ratio between 0.01 and 0.1 for high-throughput production environments. I've settled on starting at 0.05 (5%), as a reasonable default that balances observability with overhead.

Conclusion

In this tutorial, I showed you how to configure tracer components in Redpanda Connect for distributed tracing. You set up a multi-step pipeline, configured the native Jaeger tracer for local development, and then switched to the OpenTelemetry Collector tracer for a backend-agnostic approach. You also explored sampling strategies to control trace volume in production, configured schema registry serialization for downstream consumers, and added compression and shutdown delay for production readiness.

Because trace data lands in the otel-traces topic just like any other event, you can consume it the same way, correlating traces across pipeline instances or routing it downstream for further processing.

You can find the demo resources for this tutorial in this GitHub repository.