DEV Community: taman9333

Traces at Scale: Head or Tail? Sampling Strategies & Scaling the Collector

taman9333 — Mon, 11 Aug 2025 12:19:16 +0000

🚨 Tracing at Scale Isn’t Free

In the previous article, we got everything working - traces flowed between services, we visualized them in Jaeger, and we saw end-to-end visibility in action.

But here’s the catch:

In production, things look very different.

Your system might generate millions of traces every day, and I’ve seen companies that run with 100% sampling, storing every single trace, but only keeping them for a very short period, like 1 to 2 days.

Even if the company can afford the storage cost for a longer period than 1 or 2 days, this setup is inefficient:

Developers can't investigate incidents that happened more than a couple of days ago
A lot of traces are just noise, health checks, fast 200 OKs, and routine traffic
The traces that actually matter (slow requests, failures, edge cases) get lost in the crowd
High cost for exporting and storing all spans - especially when using hosted platforms

Let me quote this visual from the OpenTelemetry documentation:

This image perfectly illustrates the problem. Sampling would solve the above issues 🚀

🎯 This is where sampling comes in

Sampling helps you reduce volume while still capturing the traces that matter, the ones that help you debug real problems and improve performance.

In this article, we’ll cover:

The difference between head-based and tail-based sampling
How to configure tail-based sampling using OpenTelemetry
And how to scale your collector setup to handle production traffic

Let’s get into it.

Head based Sampling

Head based sampling means deciding whether to keep or drop a trace right at the start, as soon as the first span is created. The decision is made without knowing how the full trace will look.

A common example of this is probability sampling. It uses the trace ID and a set percentage to decide which traces to keep. For example. you might keep 30 percent of all traces. If a trace is selected, all its spans are kept together, so you do not end up with missing spans.

In OpenTelemetry, we usually combine a parent-based sampler with a probability-based sampler. This means if the parent span was sampled, all child spans will be sampled as well. If not, the entire trace will be dropped.

How to Configure Head based Sampling

Head based sampling is simple to set up. You do not need to change your system architecture or add extra components.

You can configure it directly in your application's code using the OpenTelemetry SDK.

In our case, we are going to use a 30 percent sampling rate in all of the services.

For example, in our Go service, here is how you can enable head based sampling by combining a parent-based sampler with a trace ID ratio-based sampler:

+ sampler := sdktrace.ParentBased(
+              sdktrace.TraceIDRatioBased(0.3)
+            )

tp := sdktrace.NewTracerProvider(
      sdktrace.WithBatcher(exp),
+     sdktrace.WithSampler(sampler),
      sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("service-x"),

In a Ruby service you will need to add a single line

OpenTelemetry::SDK.configure do |c|
  c.service_name = 'service-y'
  c.use 'OpenTelemetry::Instrumentation::Sinatra'
  c.use 'OpenTelemetry::Instrumentation::Faraday'
end

+ OpenTelemetry.tracer_provider.sampler = OpenTelemetry::SDK::Trace::Samplers::TraceIdRatioBased.new(0.3)

In the Node service, you will need to add the following:

+ const { ParentBasedSampler, TraceIdRatioBasedSampler } = require('@opentelemetry/sdk-trace-base');

const provider = new NodeTracerProvider({
  resource: new resourceFromAttributes({
    [ATTR_SERVICE_NAME]: "service-z",
  }),
  spanProcessors: [new SimpleSpanProcessor(exporter)],
+ sampler: new ParentBasedSampler({
+   root: new TraceIdRatioBasedSampler(0.3),
+ })
});

With these changes in place, all three services now use head based sampling by combining a parent-based strategy with a probability sampler set to 30 percent.

🧪 Time to test:

I will run the hit_x_service.sh script three times which will generate 30 requests, so we will probably see around 9 traces (It’s a statistical estimate, the actual count may vary but trends align with the percentage as requests grow) in the Jaeger UI.

Yaaay 🙌 it is working, as you can see in the screenshot we have 9 traces sampled out of 30 requests.

🚨 The Catch with Head Based Sampling

If you look again at the screenshot, you will notice the problem with head based sampling. The nine sampled traces do not include any of the error requests. You can also confirm this by observing the scatter plot in the top. The 9 dots represent the 9 sampled traces and the time they were captured. All of them are blue, which means they are successful requests. If an error trace was captured, it would appear as a red dot.

This shows a major limitation. Although head based sampling is simple to understand and configure, it makes the sampling decision before the request is fully processed. That means it can miss important spans such as failures or high latency cases.

In our case, all three errors were dropped. This makes head based sampling unreliable when your goal is to capture anomalies or debug edge cases.

Tail based sampling

Tail based sampling works differently from head-based sampling, instead of making a decision right when a trace starts, the decision to sample a trace takes place by considering all or most of the spans within the trace.

This means you can make smarter choices by looking at the full picture, like checking if any span had an error or latency.

Let me show you a visual from the OpenTelemetry docs that explains it well:

Tail-based Sampling Rules

With tail based sampling, you can define smart rules to decide which traces to keep. For example:

Always keep traces that include an error
Keep traces with high latency
Sample based on specific span attributes - like keeping more traces from a newly deployed service
Drop traces that match certain paths - like health check endpoints

And many other policies you can define based on your needs or business logic.

How to Implement Tail-based Sampling

To use tail based sampling, you’ll need to introduce a new component into your infrastructure, the OpenTelemetry Collector.

But wait… what the heck is that?

Let’s break it down.

What is the OpenTelemetry Collector?

The OpenTelemetry Collector is a vendor-agnostic implementation of how to receive, process and export telemetry data(logs, metrics & traces) to an observability backend.

It simplifies your setup by removing the need to run different agents for each type of telemetry. Instead, it acts as a single, unified point to collect and forward all your data.

Looking at the photo makes it much clearer.

On the left side, we have a typical cluster or host with different services running. These services continuously produce logs, metrics, and traces.

All of this data is sent to the OpenTelemetry Collector. The collector performs three steps for each telemetry type:

Receive the data
Process it based on your pipeline configuration before it’s exported. These processors can:
- Filter unnecessary data
- Transform or enrich spans with additional metadata
- Batch data to improve performance and reduce backend load
Export it to your observability backend

On the right, you can see popular observability platforms where the data can be exported, such as Prometheus, Grafana, Datadog, Loki and others.

The real power of using the OpenTelemetry Collector is that it acts as a central hub for all your telemetry data. Instead of asking every single service in your system to know where to send logs, metrics, or traces, or how to talk to different backends like Prometheus, Grafana, or Datadog, you let the collector handle it all in one place.

This means:

Your services stay lightweight and simple
You can change or add observability backends without touching the code in your services
You gain more control over processing and filtering data before it gets stored

In short, the collector decouples your application code from your observability tooling, which makes your system more flexible, maintainable, and scalable.

Installing OpenTelemetry Collector

To keep things simple, we won’t include the changes we made earlier for head-based sampling. That code lives in the head-based-sampling branch.

Instead, we’ll use the same instrumentation setup from the first article, which is available in the main branch.

All the changes required for tail-based sampling will be done in a new branch called tail-based-sampling. These changes include:

Updating the docker-compose.yml file to add the OpenTelemetry Collector

version: '3'

services:
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:0.130.0
+    command: ["--config=/etc/otel-collector.yaml"]
+    volumes:
+      - ./otel-collector.yaml:/etc/otel-collector.yaml
+    ports:
+      - 4317:4317
+      - 4318:4318
+    depends_on:
+      - jaeger

  jaeger:
    image: jaegertracing/all-in-one:1.71.0
-    command:
-      - "--collector.otlp.grpc.tls.enabled=false"
    ports:
      - "16686:16686"   # Jaeger UI
-      - "4317:4317"     # OTLP gRPC
-      - "4318:4318"     # OTLP HTTP

Notice that we removed ports 4317 and 4318 from the Jaeger service. That’s because we won’t send traces from our services directly to Jaeger anymore.

Instead, we’ll route all trace data to the otel-collector service first. To enable that, we exposed ports 4317 and 4318 in the otel-collector service, where our x, y, and z services send traces via HTTP to port 4318.

Then, the collector will export traces to Jaeger internally via OTLP gRPC on port 4317 inside the Docker network.

Creating a new file named otel-collector.yaml in the root directory for the collector configuration

This file defines how the OpenTelemetry Collector will receive, process, and export trace data using tail-based sampling.

Let’s break down what each section in the otel-collector.yaml file is doing.

📥 Receivers

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

Receivers are the entry point to the OpenTelemetry Collector. They collect telemetry data from your services and pass it into the processing pipeline.

In our case, we are only dealing with traces. All three services, Go, Ruby, and Node.js, are configured to send traces via the OTLP HTTP protocol.

The OTLP receiver starts both HTTP and gRPC servers, listening on ports 4318 and 4317 respectively. But since all our services send traces using HTTP, we do not need the gRPC server but we will not remove the code of grpc endpoint since we will use it later when scaling the collector.

The HTTP server on port 4318 will continue receiving traces, we configured all of our 3 services to send data through http to this port.

Receivers are mandatory in every collector configuration, without at least one, the collector will not function.

🔧 Processors

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 10
    policies:
      [
        {
          name: errors,
          type: status_code,
          status_code: { status_codes: [ERROR] }
        },
        {
            name: drop-health-checks,
            type: drop,
            drop: {
              drop_sub_policy:
              [
                {
                    name: drop-health-paths,
                    type: string_attribute,
                    string_attribute: {key: url.path, values: [\/health], enabled_regex_matching: true}
                }
              ]
            }
         },
        {
          name: probabilistic_30_percent,
          type: probabilistic,
          probabilistic: { sampling_percentage: 30 }
        }
      ]

Processors sit in the middle of the pipeline between collecting data and exporting it. They handle tasks like filtering out noise, enriching spans with more context, transforming data formats, or batching data together to improve performance. This step ensures your telemetry data is optimized before being sent to your backend.

🧠 Understanding the Processor Configuration

Let’s go through the processors section of the otel-collector.yaml file.

We’re using the tail_sampling processor, one of many available in the OpenTelemetry Collector.

⏳ `decision_wait`

The decision_wait option sets how long the collector should wait (starting from the first span of a trace) before making a sampling decision.

By default, it's set to 30s. In our config, we’ve reduced it to 10s to speed things up for local development.

When to increase `decision_wait`:

Long-running traces – If your system involves operations that take time to complete (e.g., async workflows, retries, or background jobs), a longer wait ensures all spans are included.
Retry and backoff logic – If some spans are delayed due to retries, a short wait might cause them to be missed in the sampling decision.

Potential downsides of increasing `decision_wait`:

Increased memory usage – The collector needs to buffer spans in memory for a longer period.
More latency – Traces will be processed and exported later, as the collector waits before deciding.

🚣 `num_traces`

Default: 50000
Purpose: Controls how many traces are kept in memory at a time.
If your services generate a high number of traces, you might need to increase this to avoid dropping spans before a sampling decision is made.

📈 `expected_new_traces_per_sec`

Default: 0
Purpose: An estimate of how many new traces the collector expects per second.
This helps the collector allocate memory and data structures more efficiently.

Why it matters:

Too low → Frequent reallocations, hurting performance.
Too high → Wasted memory.

🧠 `decision_cache`

Even though we haven’t included decision_cache in our config, it’s a useful tuning parameter for systems with delayed or long-lived traces (async, retry & backoff logic).

Purpose: Controls the number of traces for which a sampling decision is cached.
Even after a sampling decision is made for a trace, spans might continue to arrive. This cache ensures those late spans are handled properly.
Use case: In systems where spans can arrive out of order or with delays (e.g., async processing or retries), setting this appropriately helps avoid dropping important spans that arrive after the decision.

It keeps a short-term memory of sampling decisions:

sampled_cache_size: remembers trace IDs that were sampled → accepts late spans
non_sampled_cache_size: remembers trace IDs that were dropped → drops late spans

🧪 Sampling Policies

The policies section defines how we want to sample traces based on specific criteria. Here's what each policy does:

📛 errors Samples any trace that contains a span with status.code = ERROR. This ensures we always keep traces that highlight problems.

{
  name: errors,
  type: status_code,
  status_code: { status_codes: [ERROR] }
},

🚫 drop-health-checks Drops traces where the span path matches /health. Health checks are frequent and usually not helpful for debugging, so we exclude them to reduce noise.

{
  name: drop-health-checks,
  type: drop,
  drop: {
    drop_sub_policy:
    [
      {
          name: drop-health-paths,
          type: string_attribute,
          string_attribute: {key: url.path, values: [\/health], enabled_regex_matching: true}
      }
    ]
  }
}

🎯 probabilistic_30_percent Samples 30% of the remaining traces. This helps retain a representative view of normal, successful traffic. Even though these traces aren’t errors, they’re useful for:
- Monitoring overall system behavior
- Identifying performance trends
- Analyzing latency patterns

{
  name: probabilistic_30_percent,
  type: probabilistic,
  probabilistic: { sampling_percentage: 30 }
}

You can fine-tune these policies based on your system’s needs and business logic. You can find more examples of available policies here.

📤 Exporters

Exporters are the final step in the Collector pipeline. They send the processed telemetry data like traces, metrics, and logs to a backend system where the data can be stored, visualized, and analyzed.

At least one exporter must be defined for the Collector to function.

In our setup, we use the OTLP exporter to send trace data to Jaeger using gRPC over port 4317 since Jaeger supports OTLP(OpenTelemetry Protocol):

exporters:
  otlp:
    endpoint: jaeger:4317
    tls:
      insecure: true

This exporter sends data to the jaeger service over the Docker network.
We use insecure: true because this is a local dev setup. Avoid this in production.

⚙️ Service

This section defines how everything in the collector connects together.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp]

The service block brings all the configured pieces - receivers, processors, and exporters into action.
If you define a component but forget to include it here, it won’t be used.

📦 Pipelines

Under pipelines, you configure how the data flows through the system. In our case, we only define a traces pipeline. Other types like metrics and logs are possible too.

Each pipeline must include:

At least one receiver (to accept data)
Zero or more processors (to modify or filter it)
At least one exporter (to send it somewhere)

Make sure each part used in the pipeline is properly defined in its corresponding top-level section.

🧪 Time to Test

To try everything out:

Run the collector and Jaeger with: docker-compose up --build
Start all three services:
- Go: go run main.go
- Ruby: bundle exec ruby server.rb
- Node.js: node index.js

Then let's generate some traffic:

Run the hit_x_service.sh script three times to send 30 requests, just like we did with head-based sampling.
In addition, call the health check endpoint several times:
curl localhost:3000/health

This will help us verify if health checks are correctly excluded from sampling.

We’ll probably see around 9 traces (again it's a statistical estimate) in the Jaeger UI.

Yaaay 🙌 We can see 10 traces. If you look at the top in the scatter plot, you'll notice 3 red dots, which indicate the error traces. This means:

✅ Our probabilistic sampling of 30% is working
✅ The 3 errors are all present, so error sampling is working
✅ No health check traces appear, they’ve been dropped as expected

🚀 Scaling the Collector – Why and How

As our system grows and the number of instrumented services increases, the volume of telemetry data can quickly overwhelm a single collector instance.

Whether the collector is processing traces only or handling traces, metrics, and logs, running everything through one instance can lead to:

Bottlenecks in processing and exporting data
Increased latency in trace availability
Risk of dropped data during traffic spikes
Limited fault tolerance if the single collector fails

To handle larger volumes reliably, we need to scale the collector horizontally by running multiple instances and distributing the load across them — while ensuring spans from the same trace go to the same collector.

🛠️ Deployment patterns

🐢 No Collector

When we used head based sampling, services sent traces straight to Jaeger. That is a direct integration - no collector in the path.

Visual from the OpenTelemetry docs

🕵🏻 Agent

With tail based sampling, we introduced the collector next to each service. This is the agent deployment pattern.

Applications instrumented with OTLP send telemetry to a collector running with the app or on the same host.

Visual from the OpenTelemetry docs

Pros

Simple to get started
Clear 1-1 mapping between application and collector

Cons

Limited scalability, especially if the collector must handle traces, logs, and metrics
Harder to manage at scale across many hosts

⛩️ Gateway

The solution is the third pattern - the gateway.

In the gateway deployment, apps or sidecar collectors send telemetry to one OTLP endpoint that fronts a pool of collectors running as a standalone service - per cluster, per data center, or per region.

Visual from the OpenTelemetry docs

🚨⚠️ Important note - collectors are stateful 🚨⚠️

Collectors hold data in memory. Tail sampling buffers spans until a decision is made.

If you scale collectors horizontally without coordination, different replicas may receive spans from the same trace. Each replica will decide on sampling independently. Results can diverge. You may end up with traces missing spans, which misrepresents what happened.

How to scale correctly

Place a load balancing layer of collectors in front of the tail sampling collectors.

Use the load-balancing exporter to route all spans of the same trace to the same backend collector.

It does this by hashing the trace id, or the service name, and consistently sending related spans to the same target.

OpenTelemetry provides this load-balancing exporter out of the box. Next, we will see how to configure it in code.

Docker changes for scaling with a gateway

We renamed the otel-collector service to otel-collector-1, and duplicated it as otel-collector-2 and otel-collector-3
These three collectors run tail based sampling, and do not expose host ports, since services will not talk to them directly
We added an otel-gateway service that runs the load balancing exporter, and exposes ports 4317 and 4318
All app services send OTLP traffic to the gateway, and the gateway consistently routes each trace to one of the collectors

services:
  otel-collector-1:
    image: otel/opentelemetry-collector-contrib:0.130.0
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml
    ports:
      - "4317"        # OTLP gRPC receiver

  otel-collector-2:
    image: otel/opentelemetry-collector-contrib:0.130.0
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml
    ports:
      - "4317"        # OTLP gRPC receiver

  otel-collector-3:
    image: otel/opentelemetry-collector-contrib:0.130.0
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml]
    ports:
      - "4317"        # OTLP gRPC receiver

  # Otel gateway running load balancing exporter
  otel-gateway:
    image: otel/opentelemetry-collector-contrib:0.130.0
    command: ["--config=/etc/otel-gateway.yaml"]
    volumes:
      - ./otel-gateway.yaml:/etc/otel-gateway.yaml
    ports:
      - 4317:4317     # OTLP gRPC
      - 4318:4318     # OTLP HTTP
    depends_on:
      - otel-collector-1
      - otel-collector-2
      - otel-collector-3

otel-gateway.yaml - what it does

This file defines a gateway collector that accepts OTLP traffic from services and forwards spans to a pool of tail sampling collectors using the load balancing exporter.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      static:
        hostnames:
          - otel-collector-1:4317
          - otel-collector-2:4317
          - otel-collector-3:4317

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [loadbalancing]

Receivers

otlp listens on 4318 for HTTP since our services send traces via HTTP to 0.0.0.0:4318

Exporters - loadbalancing

resolver.static.hostnames lists the downstream collectors to send to
We use Docker Compose service names since within the Docker Compose network, service names act as DNS hostnames. so otel-collector-1:4317, otel-collector-2:4317, otel-collector-3:4317
routing_key: traceID means all spans that share the same trace id are routed to the same downstream collector, avoiding cases where different collectors sample parts of the same trace and cause incomplete or misleading results.

Service

telemetry.logs.level: debug helps with debugging the gateway behavior. We’ve also added the same telemetry configuration to otel-collector.yaml so that all three collectors produce debug-level logs, making it easier to verify that everything is working correctly.
pipelines.traces wires the otlp receiver to the loadbalancing exporter

🧪 Time to Test

If you still have Docker running, run:

docker-compose down
docker-compose up --build

Then execute the ./hit_x_service.sh script three times, just like we did when testing tail-based sampling without scaling the collector. This will generate 30 requests. We’d expect to see around 9 traces in Jaeger.

After checking Jaeger UI, we’ve received 10 traces (by coincidence, same as last test), 3 error traces and 7 normal traces. All traces show complete spans, meaning nothing was dropped. This confirms that spans from the same trace were routed to the same collector.

Now we need to confirm that traces were actually distributed across collectors, and not all sent to a single one. Since we enabled telemetry debug logs in every collector, we can run the following command for each collector service to filter useful logs:

docker-compose logs {{SERVICE_NAME}} 2>&1 | grep '"batch.len": [1-9]'

This filters out noisy logs, showing only batches where spans were sent.

📊 Log results:

otel-collector-1 → 2 logs - total traces: 10 (sampled: 5, notSampled: 5). Here, total traces = sum of "batch.len" values (2 from first log + 8 from second log).
otel-collector-2 → 3 logs - total traces: 11 (sampled: 2, notSampled: 9)
otel-collector-3 → 2 logs - total traces: 9 (sampled: 3, notSampled: 6)

✅ This means all collectors together received exactly 30 traces, matching the requests sent.

✅ Total sampled traces = 10, which matches what we see in Jaeger UI.

✅ Total not-sampled traces = 20, as expected.

📌 Final Architecture

And here is our final architecture:

This setup allows us to build a robust distributed tracing system that can absorb millions of traces efficiently, while keeping costs lower and reducing noise as much as possible.

By combining tail-based sampling, load balancing across multiple collectors, and selective sampling policies, we ensure that we capture the most valuable traces without overloading our backend.

Distributed Tracing Instrumentation with OpenTelemetry and Jaeger

taman9333 — Thu, 31 Jul 2025 20:46:25 +0000

Distributed tracing is a way to track a request as it moves through a system, especially in setups where multiple services talk to each other, like in microservices.

Imagine a user clicking "buy" on an e-commerce site. That action might hit a front-end service, a payment processor, an inventory checker, a database and a Redis cache.

If something goes wrong, figuring out where it failed can be a nightmare without a clear map. That’s where distributed tracing comes in. It’s like a GPS for your application, showing the path of a request across services, how long each step takes, and where things might break.

Unlike logs, which are like diary entries of what happened, or metrics, which give you numbers like CPU usage, tracing gives you the full story of a request’s journey. It’s critical for spotting bottlenecks, debugging errors, and understanding how your system behaves under real world conditions.

Logs are great for capturing detailed information about what your application is doing. But they are often scattered and not tied together. Traces, on the other hand, bring structure. You can think of them as stitched logs that belong to the same request. When you attach key log details as attributes or events inside spans, you end up with the same information, but now it is grouped by request and connected across services.

This article explains distributed tracing using my GitHub repository distributed_tracing.

The repo includes:

📦 Instrumentation with OpenTelemetry & Jaeger
🎯 Head-based sampling
🧠 Tail-based sampling
⚖️ Scaling collectors with a load balancer

We will walk through each step and explain what the code is doing.

⚠️Disclaimer:
There are other important topics that we will not cover here, such as:

Custom instrumentation for capturing application-specific spans

How to define a good trace and what makes a trace useful

Correlating logs with traces so that logs are grouped around a single request

Aggregating data from your traces to derive metrics without exporting them separately

Before we dive into the repository, we will rely on automatic instrumentation to keep things simple. Specifically, this includes instrumenting HTTP servers to capture incoming requests and HTTP clients to trace outgoing calls. But in real systems, it is rarely enough. You often need to go beyond that.

Good traces require good data🤖 That means making sure you are instrumenting all the key parts of your system. HTTP clients and servers, relational databases, cache layers, Elasticsearch, and any other critical services should be automatically instrumented where possible. Then you layer on custom instrumentation to fill the gaps and highlight the things that matter most in your business logic.

Repository Overview

This repository demonstrates distributed tracing using OpenTelemetry with Jaeger as the backend to collect and visualize traces.

It features three services:

X: written in Go
Y: written in Ruby
Z: written in Node.js

The services are connected in a chain: Service X calls Y, and Y calls Z. This creates a traceable path for a request as it moves across the system.

The goal is to trace the full lifecycle of a request as it flows from the entry point (service X) to the final service (Z). In the real world, this pattern is common in microservice-based applications where distributed tracing can help identify where time is spent or where failures occur.

To simulate real behavior and failures:

Service Z is configured to return a 500 Internal Server Error on every 10th request. This is done deliberately to help us observe how different sampling strategies (head-based vs tail-based) behave when errors are present in the trace.
The script hit_x_service.sh repeatedly sends 10 HTTP GET requests to the /x endpoint in service X. This creates a consistent flow of traces that travel through all three services.

Architecture Overview

Here’s a high level diagram that shows how everything fits together under the hood. Each of the three services (X, Y, and Z) is instrumented using OpenTelemetry and exports trace data via OTLP over HTTP to the Jaeger Collector. The Jaeger Collector receives and processes the traces, forwarding them to the backend for storage and visualization in the Jaeger UI.

Instrumentation and Basic Tracing

To start, we have a docker-compose.yml file that sets up the environment. It installs Jaeger along with the necessary ports to receive and visualize trace data.

version: '3'

services:
  jaeger:
    image: jaegertracing/all-in-one:1.71.0
    command:
      - "--collector.otlp.grpc.tls.enabled=false"
    ports:
      - "16686:16686"   # Jaeger UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP

Let’s break down what each port does:

Port 16686 is used to access the Jaeger UI.
Port 4317 allows the Jaeger Collector to receive trace data using the OpenTelemetry Protocol (OTLP) over gRPC.
Port 4318 does the same, but over HTTP instead of gRPC.

Ports 4317 and 4318 are both handled by the Jaeger Collector, which ingests trace data from services instrumented with OpenTelemetry. These services generate spans, and the collector receives, processes, and forwards them to the Jaeger backend for storage and visualization.

With this setup in place, you can start sending traces from your services to Jaeger using either OTLP over gRPC or HTTP. This flexibility makes it easier to integrate tracing into different environments and across various tech stacks.

Sampling: Default Behavior

By default, OpenTelemetry samples 100% of traces. That means every span created in your service will be recorded and exported.

Unless you have a specific need to manage trace volume such as in high-throughput production environments, you don’t need to configure a custom sampler.

The default sampler is a combination of ParentBased and ALWAYS_ON. Here's what that means:

The root span of a trace is always sampled.
All child spans inherit the sampling decision of their parent.

This guarantees that once a trace is started, every span within it will be sampled and exported.

In the first step, the tracing logic added to all three services (X, Y, and Z) will use the default sampler, meaning no sampling limits are applied.

Here’s how 100% sampling is configured in each language used in this repository:

Go (Service X)

We start our Go app by invoking the initTracer function. This function is responsible for tracing the HTTP server (receiving requests).

There are two key things to consider in this function:

It sends traces through HTTP to the Jaeger Collector on port 4318, and disables TLS since we are running locally:

exp, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint("localhost:4318"),
        otlptracehttp.WithInsecure(), // disables TLS
    )

Context propagation: the process of passing trace context (like trace and span IDs) across service boundaries, enabling the tracking of requests as they move through a distributed system. It ensures that the trace remains intact and connected, providing full observability. Propagation is usually handled by instrumentation libraries as we will see in the next snippet, however In the event that you need to manually propagate context, you can use the Propagators API.

otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.TraceContext{}, propagation.Baggage{},
        ),
    )

You can observe context propagation in action by inspecting the request headers passed between services.

For example, since the Go service (X) calls the Ruby service (Y), if you log the incoming request headers in the Ruby app, you will see something like:

As you can see, the HTTP_TRACEPARENT header is present. This header carries trace context across service boundaries and allows the spans created by each service to be linked to the same trace.

Finally, we trace outgoing HTTP requests made by the Go service using an instrumented http.Client. This is essential for tracing HTTP client calls from Go to downstream services:

var client = http.Client{
    Transport: otelhttp.NewTransport(http.DefaultTransport),
}

Ruby (Service Y)

In Ruby, we use Sinatra to serve web requests and Faraday as the HTTP client. Instrumenting Ruby with OpenTelemetry is much simpler and requires less code compared to Go.

Here’s what you need in the server.rb file:

OpenTelemetry::SDK.configure do |c|
  c.service_name = 'service-y'
  c.use 'OpenTelemetry::Instrumentation::Sinatra'
  c.use 'OpenTelemetry::Instrumentation::Faraday'
end

Unlike Go, we don’t need to manually configure context propagation.

The OpenTelemetry Ruby SDK handles this automatically as long as you are using auto-instrumented libraries like Sinatra and Faraday. It will extract incoming context from request headers and inject it into outgoing HTTP requests without additional setup.

Node.js (Service Z)

In Node.js, we use Express as our web server. The instrumentation setup is located in a separate file, tracing.js, which is imported at the top of index.js.

tracing.js file configures the OpenTelemetry setup for the service.

We send traces to the Jaeger Collector using HTTP:

const exporter = new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' })

We enable automatic instrumentation for both the HTTP layer and Express. It’s important to instrument the HTTP layer first, since Express relies on it:

const provider = new NodeTracerProvider({
  resource: new resourceFromAttributes({
    [ATTR_SERVICE_NAME]: "service-z",
  }),
  spanProcessors: [new SimpleSpanProcessor(exporter)],
});

registerInstrumentations({
  tracerProvider: provider,
  instrumentations: [
    // Express instrumentation expects HTTP layer to be instrumented
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

🎉 Installation Complete, Time to Trace 🚀

Start all three services (X, Y, and Z)
Run the Jaeger backend using: docker-compose up
Now run the following script to send 10 requests to Service X: ./hit_x_service.sh
Open your browser and go to: http://localhost:16686

You should see 10 traces listed in the Jaeger UI and if you click into the last one, you'll notice it contains an error. That’s because service Z is configured to fail on every 10th request, just like we planned.

When you click on the link for the last trace with error you will be able to trace the full request lifecycle across all three services.

🧵 Below is a trace that shows:

The request starts in service-x

It propagates to service-y

Then it hits service-z and fails with a 500

Back in service-y, we log the error and correlate it with the trace

This makes it super easy to debug distributed systems and pinpoint which service is failing and why.

🚨 But Wait There’s a Problem

Cool - at this point everything is working. You’ve got traces flowing, spans being recorded, and the Jaeger UI showing the request paths across your services.

But here’s the catch, In production things look very different. Your system might generate millions of traces every day. And with that comes a few serious challenges:

High cost for exporting and storing all spans - especially when using hosted platforms
Too much noise - making it hard to focus on what's important (for example health checks)
Hard to catch the interesting traces - the ones with high latency, errors, or performance bottlenecks

This is where sampling comes in. It helps you reduce the volume of trace data while keeping the insights that matter most.

We’ll talk about sampling strategies - including head-based and tail-based sampling in the next article.

Stay tuned 🔥

Scalable Url Shortener Part2

taman9333 — Mon, 26 Aug 2024 23:59:40 +0000

Encoding a Long URL to a Short URL

In this part, we'll explore how to encode a long URL into a short URL. There are different techniques we can use to achieve this.

Option 1: Hash Functions

According to Wikipedia, a hash function is any function that maps data of arbitrary size to fixed-size values.

Let’s take MD5 as an example. The output length of the MD5 hash function is 128 bits, or 16 bytes. When represented as a hexadecimal string, it’s 32 characters long.

Here’s a simple example of using MD5 in Ruby:

require 'digest/md5'
Digest::SHA256.hexdigest("www.test.com")
# => "84cc0e5c525dc728e1769ad6663341c8"

As you can see, the output is 32 characters, which is too long for our use case. To address this, we can use a simple trick: taking only the first 7 characters.

Digest::MD5.hexdigest("www.test.com")[0..6]
# => "84cc0e5"

Cool; it's pretty easy, right?

Unfortunately, this is not a perfect solution as the MD5 algorithm might lead to collisions. Here are the scenarios where collisions could happen:

The MD5 algorithm can possibly generate the same hash code for different strings (this is very rare).
Even if the entire hash code is not the same, you could encounter a collision where two different hash codes share the same first 7 characters that we plan to use.

We could introduce a unique index to solve this problem. This would allow us to catch any duplicate hash codes when writing them to the database and retry generating a new hash code. However, using a unique index has its downsides. It would place a lock on our database, which wouldn't scale well if we receive a lot of writes. Additionally, if we plan to shard our database to scale writes across different regions, the unique index approach would no longer work effectively.

Option 2: Counter (Optimal Solution)

Instead of using a hash function, we can use a counter-based approach to generate short URLs. This method is much simpler and avoids the collision issues that can occur with hash functions.

How It Works

The idea is straightforward: we maintain a global counter that increments with every new URL request. Each time a new URL is submitted, we increment the counter and convert the value of the counter into a short string using Base62 encoding.

Base62 encoding uses a character set of 62 characters (0-9, a-z, A-Z), which is perfect for generating short, readable URLs. By encoding the incremented counter value into Base62, we generate a unique and compact string for each URL.

Why It’s Optimal

Uniqueness: Since the counter increases sequentially, every value is guaranteed to be unique, which eliminates the risk of collisions.
Short Length: With Base62 encoding, we can generate short strings that are much smaller than the original counter value, making the URLs compact and easy to share.
Scalability: The counter-based approach is scalable and performs well, even with large numbers of URLs, since it's a simple increment and encoding operation.
No Need for a Unique Index: Unlike the hash-based approach, we don’t need to rely on a unique index in the database, as the counter ensures uniqueness on its own.

The Problem with Counters in Scalable Applications

When scaling the application with multiple instances (e.g., 3 or more), managing the counter across instances can become problematic. If the counter logic is handled by one instance, that instance becomes a single point of failure. If it goes down, the entire counter mechanism fails, which disrupts the generation of short URLs.

Even if each instance manages its own counter, there’s still a challenge. Once an instance exhausts its local counter range, it would need a mechanism to obtain the next range of counters. This leads to more complexity in coordination across instances.

To avoid these issues, we need a global counter service responsible for managing the counter in a distributed and scalable manner. This ensures that all instances can safely and consistently generate unique short URLs, without risking collisions or relying on a single instance to manage the counter.

Ensuring Scalability with Distributed Systems

In a distributed system where multiple instances of the URL shortener are running, it's critical to ensure that each instance generates unique short URLs without collisions. To achieve this, we rely on etcd as a distributed coordination service.

What is etcd?

etcd is a distributed, reliable key-value store used for coordinating configuration data across multiple servers or machines. It ensures strong consistency and provides a way to manage shared data between multiple instances of our application. In our case, etcd will manage the global counter used to generate unique short URLs.

etcd plays a key role in ensuring that all instances of the URL shortener service are synchronized, so that each instance retrieves the correct counter range without collisions, even in a distributed, multi-instance environment.

Why Use a 3-Node etcd Cluster?

To avoid creating a single point of failure, etcd is not run in standalone mode. Instead, we deploy a 3-node etcd cluster to ensure high availability and fault tolerance. With this setup, even if one node goes down, the other nodes will continue to manage the distributed counter, ensuring that the service remains functional.

Running etcd in a 3-node cluster guarantees that our coordination service is highly available and resilient to failures. Each etcd node shares the responsibility of managing the counters, and they work together using the Raft consensus algorithm to ensure consistency across all nodes.

How etcd Coordinates Between Machines

In our URL shortener service, etcd acts as a coordination service between the different machines or servers. Here’s how it works:

Tracking Active Machines: etcd maintains a list of machines (or instances) that are currently active. Each instance communicates with etcd to register itself and retrieve the range of counters it is responsible for.
Assigning Counter Ranges: etcd keeps track of the last counter that was used across all instances. When a new instance is added to the system (e.g., for scalability), it talks to etcd and receives a new, unallocated range of counters that it can use to generate short URLs.
Handling Counter Exhaustion: If an instance exhausts its current counter range, it communicates with etcd again to request the next available counter range. This ensures that every instance is always generating unique short URLs, even when it runs out of its initial counter range.

By coordinating with etcd, all instances of the service can generate unique short URLs without needing to worry about collisions or stale counters. etcd ensures that only one instance uses a specific range of counters at a time, making the system scalable and resilient.

Here is a link of Docker Compose configuration, we have set up a 3-node etcd cluster to ensure high availability.

Example: Using a Counter with Base62

Let’s say the counter starts at 1 and increments by 1 with every new URL. The counter values will be encoded into Base62 like this:

Counter: 1 → Base62: 1
Counter: 62 → Base62: 10
Counter: 3844 → Base62: 100

Each counter value generates a unique, encoded string that we can use as the short URL.

Here’s a simplified version of how the encoding would work in Ruby:

class Base62
  CHARSET = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ".chars
  BASE = CHARSET.size

  def self.encode(num)
    return CHARSET[num] if num.zero?
    raise ArgumentError, "Number must be non-negative" if num.negative?

    str = ""
    while num > 0
      str = CHARSET[num % BASE] + str
      num /= BASE
    end
    str
  end
end

Getting the Next Counter via `CounterService`

The CounterService is responsible for managing the global counter, ensuring that each request for a new counter is handled in a thread-safe manner, even in a multi-threaded environment. When a new URL is shortened, CounterService.get_next_counter is invoked to retrieve the next available counter.

Here’s how CounterService.get_next_counter works:

Counter Initialization at Boot-Up: The counter range is initialized once during the server boot-up in config.ru. This ensures that the counter range is prepared and ready to handle requests as soon as the server is up and running.
Thread Safety with Mutex: To handle concurrent requests, get_next_counter uses a mutex to ensure that only one thread can modify the counter at a time. This prevents race conditions and ensures that the counter is incremented consistently, even in a multi-threaded environment.

def get_next_counter
  counter_mutex.synchronize do
    current_counter = counter
    if current_counter >= counter_range.last
      self.counter_range = get_counter_range
      self.counter = counter_range.first
      current_counter = counter
    end
    self.counter += 1
    current_counter
  end
end

Counter Range Exhaustion: Once the current counter reaches the end of the allocated range, CounterService will request a new counter range by calling get_counter_range. This ensures that a fresh range of counters is always available for the next requests.

Here’s the implementation of the get_counter_range method:

def get_counter_range
  loop do
    current_value = ETCD_CLIENT.get(COUNTER_KEY).kvs.first&.value.to_i
    new_value = current_value + RANGE_SIZE
    txn = ETCD_CLIENT.transaction do |txn|
      txn.compare = [
        txn.value(COUNTER_KEY, :equal, current_value.to_s),
      ]
      txn.success = [
        txn.put(COUNTER_KEY, new_value.to_s)
      ]
    end

    if txn.succeeded
      puts "Instance #{ENV['SERVICE_NAME']} obtained counter range #{current_value} to #{new_value - 1}"
      return (current_value...new_value)
    end
  end
end

Here’s how the `get_counter_range` method works:

Fetching the Current Counter Value: The method begins by retrieving the current counter value from etcd, using ETCD_CLIENT.get(COUNTER_KEY).kvs.first&.value.to_i. This fetches the most up-to-date value of the global counter from the etcd cluster.
Calculating the New Counter Range: The new range is calculated by adding a fixed RANGE_SIZE to the current_value. This ensures that the instance requesting the range will handle a specific block of counters.
Distributed Counter Allocation: The method employs etcd’s transactional operations to ensure safe and consistent counter updates across multiple instances:
1. Atomic Update: The txn.compare block checks if the current value in etcd is still the same as the current_value retrieved earlier. This is to make sure that no other instance has updated the counter in the meantime.
2. Success Block: If the comparison succeeds (i.e., no other instance has modified the counter), the transaction proceeds with updating the counter to new_value using txn.put.
Consistency in a Distributed Environment: This method ensures consistency in distributed environments where multiple server instances are running. By atomically checking and updating the counter using etcd's transaction mechanism, it guarantees that each instance gets its own unique range of counters without overlap or collisions.
Retry Mechanism: If the transaction fails (which means another instance updated the counter), the loop retries by fetching the new current value from etcd. This ensures that the service will always get a valid counter range.
Logging: A log message (puts) is used to track which instance obtained a counter range, which can be helpful for debugging and monitoring.

By using this method, CounterService can handle distributed counter allocation in a consistent, safe manner, ensuring that each instance of the URL shortener service always has a unique counter range to generate short URLs, even in a multi-instance, distributed environment.

Use Cases Supported by `CounterService`

Unique Counter Generation: CounterService guarantees the generation of a unique counter value for every request. By using a distributed counter system (e.g., with etcd or another source), it ensures that no two instances of the service generate the same counter value.
Thread-Safe Operations: The use of a mutex ensures that multiple threads in a multi-threaded web server (e.g., Puma) can safely access and modify the counter without causing race conditions. This is critical for ensuring the integrity and uniqueness of counter values in a concurrent environment.
Range-Based Counter Allocation: To improve performance, CounterService works with ranges of counters. It keeps track of a counter range, incrementing the current counter within that range. When the range is exhausted, it requests a new range, minimizing the need to repeatedly fetch individual counter values from an external source.
Distributed Counter Coordination: In distributed environments, where multiple instances of the URL shortener are running, CounterService can work with a coordination service like etcd. This allows each instance to request a unique range of counters, ensuring there are no overlaps or collisions across instances.

Conclusion

The CounterService is a crucial component of our URL shortener system, handling the management and allocation of counters in a way that is both thread-safe and scalable. By utilizing a distributed counter system and supporting range-based allocation, it ensures the generation of unique, incremented counters for every URL shortening request, even in a multi-threaded or distributed environment.

Explore the Source Code

If you want to explore the code or run two instances of the URL shortener service locally, you can find the full source code in my GitHub repository:

scalable_url_shortener

The repository includes everything you need, including Docker Compose configurations, to set up and run multiple instances of the service, a 3-Node etcd Cluster, and MongoDB.

Feel free to clone the repository and experiment with running the service locally!

Scalable Url Shortener Part1

taman9333 — Mon, 26 Aug 2024 23:59:02 +0000

Building a Scalable URL Shortener Service That Can Handle Billions of Requests

In this article, we’ll explore how to implement a scalable URL shortener service step-by-step. To follow along with the code, you can check out the full implementation in the scalable_url_shortener repository. This repository contains all the necessary code, including setup instructions and configurations using Docker Compose. By the end of this article, you’ll have a clear understanding of how to build a robust and scalable URL shortener service that can handle billions of requests.

URL shortening is a technique that creates shorter versions of URLs, serving as aliases for longer ones. This is particularly useful for sharing links on platforms with character limits or for improving the aesthetics of links.

Functional Requirements

URL Shortening: Our service should be able to shorten the URL provided by the user. The shortened URL code should consist of a mix of uppercase and lowercase alphabet characters (A-Z, a-z) and digits (0-9).
Redirection: When a user accesses the shortened URL, they should be redirected to the original, full-length URL.

Non-Functional Requirements

Low Latency: The service must respond quickly to both URL shortening and redirection requests.
High Availability: The service should be reliably accessible at all times, ensuring users can shorten URLs and be redirected without interruption.
Strong Consistency: Each unique long URL should generate a unique short URL, and there should never be a case where two different long URLs map to the same short URL.

Assumptions

The ratio of read (redirection) operations to write (URL shortening) operations is 100:1.
The service is expected to generate approximately 100 million unique shortened URLs per month.

Topics We Won’t Cover(Don't ignore in production)

Caching: While caching is essential for any scalable service to reduce database load and improve response times, it’s not unique to URL shorteners. Therefore, we will omit caching from this discussion.
Load Balancing: Like caching, load balancing is a general technique used to distribute traffic across multiple servers. Although critical for scalability, it is not specific to URL shorteners, so we will not dig deep into load balancing here.
Rate Limiting: Rate limiting is crucial to protect your service from abuse and to ensure fair usage across all users. By controlling the number of requests a user can make within a certain time period, you can prevent excessive load on your system and mitigate the risk of Denial of Service (DoS) attacks.

Stack We Are Going to Use
Ruby: Although the codebase introduced in this article is written in Ruby, you can use whatever language you prefer, as the key focus is on the algorithm we’re going to implement. We will use the Sinatra web framework since it’s a minimal framework, and our use case doesn’t require anything more complex.
MongoDB: We will use MongoDB as our NoSQL database, and we’ll explain why it’s a good fit for this service.
etcd: We’ll use etcd, a distributed key-value store, to reliably manage data across a cluster of machines. It will help us generate unique short URLs in a distributed environment.

API Design

To shorten a url you will need to send the following request

POST /shorten

Request Body

{
  "url": "http://test.com"
}

Response

{"short_url":"av32cd"}

Database Choice

Given that our servers will receive millions of requests, it's crucial to consider database scalability from the very beginning.

For a service like URL shortening, the amount of data we need to store is relatively small. However, due to the high volume of read-heavy traffic, our storage solution must be horizontally scalable to handle the load efficiently and maintain low-latency responses as the service grows.

Our data model is straightforward, with minimal need for complex joins beyond associating each shortened URL with a specific user. This simplicity makes NoSQL databases a better fit for our requirements. While it's possible to use an SQL database, doing so would necessitate careful planning and the implementation of multiple read replicas to achieve the desired scalability and performance.

We have chosen MongoDB as our database solution because it scales more easily compared to traditional SQL databases. MongoDB's flexible schema and built-in support for horizontal scaling make it well-suited for handling large-scale, distributed applications like our URL shortener service.

However, employing multiple read replicas, which is essential for scaling, introduces potential concurrency issues. Specifically, we must ensure that once a short URL code is generated, it is not duplicated by another request before the write operation has been propagated to all replicas. Addressing this challenge is critical to maintaining the uniqueness and integrity of our shortened URLs across a distributed system.

Implementation of Endpoints in Ruby

Here is the implementation of the two primary endpoints for our application (you can find the actual code in this GitHub repository):

post '/shorten' do
  url = params[:url]

  halt 400, 'URL is required' unless url

  counter = CounterService.get_next_counter
  short_url = Base62.encode(counter)

  Url.create(short_url, url)

  content_type :json
  { short_url: short_url }.to_json
end

get '/:short_url' do
  short_url = params[:short_url]
  url_doc = Url.find_by_short_url(short_url)

  halt 404, 'URL not found' unless url_doc
  redirect url_doc[:original_url]
end

As you can see, to generate a unique short URL, we are using CounterService and Base62. I will discuss these components in detail in the next article to keep this one concise.

I hope you enjoyed this first article, and I look forward to seeing you in the next part of the series.

Introducing Do Notation in the Mo Package for Golang

taman9333 — Thu, 27 Jun 2024 19:47:50 +0000

What is Do Notation?

Do notation is a syntactic sugar primarily used in functional programming languages like Haskell and Scala. It simplifies the chaining of monadic operations, making the code more readable and maintainable. By bringing this feature to Go, we can now write cleaner, more expressive code when working with monads.

Why Do Notation?

When dealing with monads, especially in complex business logic, chaining operations can become cumbersome. Error handling and managing different states often lead to deeply nested structures that are hard to follow. Do notation addresses this by allowing us to write monadic operations in a sequential style, akin to imperative programming, but with all the benefits of functional programming.

How Does It Work in the Mo Package?

In Go, implementing do notation wasn't straightforward, but I managed to achieve it using the Do function. Here's a quick look at how you can use it with an example:

package main

import (
    "errors"
    "fmt"
    "github.com/samber/mo"
)

func validateBooking(params map[string]string) mo.Result[map[string]string] {
    if params["guest"] != "" && params["roomType"] != "" {
        return mo.Ok(params)
    }
    return mo.Err[map[string]string](errors.New("validation failed"))
}

func createBooking(guest string) mo.Result[string] {
    if guest != "" {
        return mo.Ok("Booking Created for: " + guest)
    }
    return mo.Err[string](errors.New("booking creation failed"))
}

func assignRoom(booking string, roomType string) mo.Result[string] {
    if roomType != "" {
        return mo.Ok("Room Assigned: " + roomType + " for " + booking)
    }
    return mo.Err[string](errors.New("room assignment failed"))
}

// This could be a service package that performs the entire process
func bookRoom(params map[string]string) mo.Result[[]string] {
    return mo.Do(func() []string {
        // Validate booking parameters
        values := validateBooking(params).MustGet()

        // Create booking
        booking := createBooking(values["guest"]).MustGet()

        // Assign room
        room := assignRoom(booking, values["roomType"]).MustGet()

        // Return success with booking and room details
        return []string{booking, room}
    })
}

func main() {
    params := map[string]string{
        "guest":   "Foo",
        "roomType": "Suite",
    }

    result := bookRoom(params)
    if result.IsError() {
        fmt.Println("Error:", result.Error())
    } else {
        fmt.Println("Success:", result.MustGet())
    }
}

In this example, bookRoom uses the Do function to sequentially perform several operations: validating booking parameters, creating a booking, and assigning a room. Each step returns a Result which can be seamlessly chained using the Do function, ensuring clean and readable error handling.

Comparison of bookRoom Function

Without Do-Notation

You can have two options:

1. Using bind (if implemented):
The "bind" operation in monads can resemble callback hell when there are many monadic operations because of the nested and sequential nature of these operations. When many such operations are chained together, the code can become deeply nested and harder to read, similar to how deeply nested callbacks can be in asynchronous programming. If bind were implemented in the Mo package, using it in this example would look something like this:

func bookRoom(params map[string]string) mo.Result[[]string] {
    return bind(validateBooking(params), func(values map[string]string) mo.Result[[]string] {
        return bind(createBooking(values["guest"]), func(booking string) mo.Result[[]string] {
            return bind(assignRoom(booking, values["roomType"]), func(room string) mo.Result[[]string] {
                return mo.Ok([]string{booking, room})
            })
        })
    })
}

This approach quickly becomes hard to read and maintain.

2. Using .Get():
Another option is to use .Get() on the monad to unwrap the monad and get the underlying value and error. This looks like typical Go code, but error handling can be verbose:

func bookRoom(params map[string]string) mo.Result[[]string] {
    values, err := validateBooking(params).Get()
    if err != nil {
        return mo.Err[[]string](err)
    }

    booking, err := createBooking(values["guest"]).Get()
    if err != nil {
        return mo.Err[[]string](err)
    }

    room, err := assignRoom(booking, values["roomType"]).Get()
    if err != nil {
        return mo.Err[[]string](err)
    }

    return mo.Ok([]string{booking, room})
}

This approach is more readable than using bind, but still involves a lot of boilerplate error handling.

With Do-Notation

With do notation, you can call .MustGet() on the monad to get the underlying value directly without error. This function (MustGet()) will panic if the monad has an error; however, do notation will handle that and short circuit the execution if there is an error or return the unwrapped value back:

func bookRoom(params map[string]string) mo.Result[[]string] {
    return mo.Do(func() []string {
        values := validateBooking(params).MustGet()
        booking := createBooking(values["guest"]).MustGet()
        room := assignRoom(booking, values["roomType"]).MustGet()
        return []string{booking, room}
    })
}

This approach is clean, concise, and easy to read, significantly reducing boilerplate error handling code.

Final Thoughts

One of the great advantages of using do notation is that you don't have to check for errors after every monadic operation. Even though a monad can have an error type, do notation will automatically handle error propagation and short-circuit the execution if an error occurs. This leads to cleaner and more maintainable code, which is particularly valuable in complex workflows.

DEV Community: taman9333

Traces at Scale: Head or Tail? Sampling Strategies & Scaling the Collector

🚨 Tracing at Scale Isn’t Free

🎯 This is where sampling comes in

Head based Sampling

How to Configure Head based Sampling

🧪 Time to test:

🚨 The Catch with Head Based Sampling

Tail based sampling

Tail-based Sampling Rules

How to Implement Tail-based Sampling

What is the OpenTelemetry Collector?

Installing OpenTelemetry Collector

📥 Receivers

🔧 Processors

🧠 Understanding the Processor Configuration

⏳ decision_wait

When to increase decision_wait:

Potential downsides of increasing decision_wait:

🚣 num_traces

📈 expected_new_traces_per_sec

Why it matters:

🧠 decision_cache

🧪 Sampling Policies

📤 Exporters

⚙️ Service

🧪 Time to Test

🚀 Scaling the Collector – Why and How

🛠️ Deployment patterns

🐢 No Collector

🕵🏻 Agent

⛩️ Gateway

🚨⚠️ Important note - collectors are stateful 🚨⚠️

How to scale correctly

Docker changes for scaling with a gateway

otel-gateway.yaml - what it does

🧪 Time to Test

📌 Final Architecture

Distributed Tracing Instrumentation with OpenTelemetry and Jaeger

Repository Overview

Architecture Overview

Instrumentation and Basic Tracing

Sampling: Default Behavior

Go (Service X)

Ruby (Service Y)

Node.js (Service Z)

🎉 Installation Complete, Time to Trace 🚀

🚨 But Wait There’s a Problem

Scalable Url Shortener Part2

Encoding a Long URL to a Short URL

Option 1: Hash Functions

Option 2: Counter (Optimal Solution)

How It Works

Why It’s Optimal

The Problem with Counters in Scalable Applications

Ensuring Scalability with Distributed Systems

What is etcd?

Why Use a 3-Node etcd Cluster?

How etcd Coordinates Between Machines

Example: Using a Counter with Base62

Getting the Next Counter via CounterService

Here’s how the get_counter_range method works:

Use Cases Supported by CounterService

Conclusion

Explore the Source Code

Scalable Url Shortener Part1

Building a Scalable URL Shortener Service That Can Handle Billions of Requests

Functional Requirements

Non-Functional Requirements

Assumptions

Topics We Won’t Cover(Don't ignore in production)

Stack We Are Going to Use

API Design

Database Choice

Implementation of Endpoints in Ruby

Introducing Do Notation in the Mo Package for Golang

What is Do Notation?

Why Do Notation?

How Does It Work in the Mo Package?

Comparison of bookRoom Function

⏳ `decision_wait`

When to increase `decision_wait`:

Potential downsides of increasing `decision_wait`:

🚣 `num_traces`

📈 `expected_new_traces_per_sec`

🧠 `decision_cache`

Getting the Next Counter via `CounterService`

Here’s how the `get_counter_range` method works:

Use Cases Supported by `CounterService`