DEV Community: Aspecto

Guide to Distributed Tracing with OpenTelemetry Dotnet

Motti Bechhofer — Wed, 25 Jan 2023 12:51:56 +0000

In this OpenTelemetry dotnet guide, you will learn exactly how to set up a .Net project that leverages OpenTelemetry, from scratch, and without any prior knowledge of OpenTelemetry.

We will build a simple to-do app that uses ASP.NET CORE and MongoDB. We will then use OpenTelemetry to generate spans, and send them to Aspecto for visualization.

You can view the complete code on GitHub here.

OpenTelemetry Overview

If you're somewhat new to OpenTelemetry, here are the basics you need to know (you can also visit our OpenTelemetry Bootcamp if you want to become an OTel ninja).

OpenTelemetry is a suite of APIs and SDKs that facilitate the collection, export, and generation of trace data, log data, and metrics (also known as the three pillars of observability). Led by the Cloud Native Computing Foundation (CNCF, the organization responsible for Kubernetes), we utilize OpenTelemetry to gather data from events and operations occurring within our system, enabling us to instrument our distributed services.

This data is ultimately helpful in understanding and analyzing the behavior of our software, as well as troubleshooting any performance issues or errors.

OpenTelemetry serves as a single library that captures all data under a single specification and transports it to a designated location (e.g., backend, collector, or open-source supporting tools). In this OpenTelemetry dotnet guide, there are a few key terms to understand:

Span: A span represents an action or operation that occurs within our system. This can include an HTTP request or a database operation that takes place over a specific duration of time. Spans often have a parent-child relationship with other spans.
Trace: Traces are essentially "call stacks" for distributed services, representing a tree of spans connected in a parent-child relationship. Traces outline the progression of requests across various services and components in our application (such as databases, data sources, queues, etc.). For example, sending an API call to the user service may result in a database query to the users database.

Exporter: Once we create a span, we need to send it to a designated backend. This may be in memory, Jaeger Tracing, or even output to the console. The exporter handles the process of sending the data to the backend.
Instrumentation: Instrumentation libraries allow us to gather data and generate spans based on the libraries used in our applications, such as Kafka, Mongo, AspNetCore, etc. There are two approaches to instrumenting our application:
- Auto instrumentation: Automatically creating spans from the libraries used in our application with the aid of ready-to-use OpenTelemetry libraries.
- Manual instrumentation: Manually adding code to our application to define the beginning and end of each span and the payload.

For a more comprehensive understanding of OpenTelemetry terminology, visit the official documentation.

OpenTelemetry Dotnet: Building Our Project

Setting up .Net app

As mentioned, we are going to build a to-do app that uses ASP.NET CORE and MongoDB.

Here's how it's done:

1. Start by setting up a new web API project by running the following command:

dotnet new webapi -o OpenTelemetryExample

2. We will create a simple to-do list application using MongoDB in this example. So let's install the Mongodb driver for dotnet.

dotnet add package MongoDB.Driver

3. Create our to-do model. Create a file named Todo.cs in a directory called Models. Our model will have an id and a description. So our code should look like this:

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;

namespace OpenTelemetryExample.Models;

public class Todo
{
  [BsonElement("_id")]
  [BsonId]
  [BsonRepresentation(BsonType.ObjectId)]
  public string Id { get; set; } = null!;

  [BsonElement("title")]
  public string Title { get; set; } = null!;
}

4. Add our controller to the Controllers folder. Our controller will expose three endpoints:

POST /todo -- To create a new todo item
GET /todo -- To get all todos
GET/todo/:Id -- To get a specific todo item

Our controller should look like this:

using Microsoft.AspNetCore.Mvc;
using MongoDB.Driver;
using OpenTelemetryExample.Models;

namespace OpenTelemetryExample.Controllers;

[ApiController]
[Route("[controller]")]
public class TodoController : ControllerBase
{
  private readonly IMongoCollection<Todo> _todos;
  private readonly ILogger<TodoController> _logger;

  public TodoController(ILogger<TodoController> logger, IMongoDatabase database)
  {
      _logger = logger;
      _todos = database.GetCollection<Todo>("todos");
  }

  [HttpPost]
  public async Task<ActionResult<Todo>> PostTodo([FromBody] Todo todo)
  {
      await _todos.InsertOneAsync(todo);
      return CreatedAtAction(nameof(GetTodos), todo);
  }

  [HttpGet]
  public async Task<ActionResult<IEnumerable<Todo>>> GetTodos()
  {
      var todos = await _todos.Find(m => true).ToListAsync();
      return Ok(todos);
  }

  [HttpGet("{id}")]
  public async Task<ActionResult<Todo>> GetTodoById(string id)
  {
      var todo = await _todos.Find(todo => todo.Id == id).FirstAsync();
      return Ok(todo);
  }
}

5. Register the Mongodb client in the dependency injection. Go to the program.cs file in the root directory:

In the main method, we will add the following code to init Mongodb (we will run mongo on running on your local machine on port 27017)

var mongoUrl = MongoUrl.Create("mongodb://localhost:27017");
var clientSettings = MongoClientSettings.FromUrl(mongoUrl);
var mongoClient = new MongoClient(clientSettings);

6. Then, register it as a singleton.

builder.Services.AddSingleton(mongoClient.GetDatabase("todo"));

7. Now we can run our dotnet project with the command

dotnet run

8. And create a todo with a post request to /todo and view it by making a get request to /todo.

Adding OpenTelemetry and visualizing our Dotnet traces

You are now familiar with the fundamental concepts of spans, traces, instrumentations, and generally how OpenTelemetry can be utilized to generate traces for Dotnet applications. Let's add OpenTelemetry to our project and send spans for visualization with Aspecto.

Using Aspecto to visualize data is incredibly straightforward. If you wish to explore Aspecto for yourself, you can try the free-forever plan which offers unlimited features including unlimited workspaces, user invites, sampling rules, and more.

Consider playing with the Live Playground to gain a deeper understanding.

Here's how you do it:

1. Install the following dependencies:

dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Extensions.Hosting --prerelease
dotnet add package OpenTelemetry.Instrumentation.AspNetCore --prerelease
dotnet add package OpenTelemetry.Instrumentation.Http --prerelease
Dotnet add package MongoDB.Driver.Core.Extensions.DiagnosticSources
dotnet add package MongoDB.Driver.Core.Extensions.OpenTelemetry

We will be exporting spans to Aspecto using this package we just installed.

dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

2. Configure OpenTelemetry by going to the program.cs file and add the following code to the main method. Make sure to add your aspecto token to your ASPECTO_API_KEY environment variable (get it here)

        var serviceName = "your-service-name";
        var serviceVersion = "your-service-version";
builder.Services.AddOpenTelemetryTracing(tracerProviderBuilder =>
            tracerProviderBuilder
                .AddOtlpExporter(opt =>
                {
                    opt.Endpoint = new Uri("https://collector.aspecto.io:4317/v1/traces");
                    opt.Headers = $"Authorization={Environment.GetEnvironmentVariable("ASPECTO_API_KEY")}";
                })
                .AddSource(serviceName)
                .SetResourceBuilder(
                    ResourceBuilder.CreateDefault()
                        .AddService(serviceName: serviceName, serviceVersion: serviceVersion))
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation()
                .AddMongoDBInstrumentation()
        );

💡Good to know -- If you wish to export traces to Jaeger, you should use the AddJaegerExporter instead of the AddOtlpExporter. Visit the opentelemetry-dotnet repository to see how it's done.

3. Next, add this code to register the mongo db instrumentation:

clientSettings.ClusterConfigurator = cb => cb.Subscribe(new DiagnosticsActivityEventSubscriber());

4. The complete program.cs file should look like this:

using MongoDB.Driver;
using MongoDB.Driver.Core.Extensions.DiagnosticSources;
using OpenTelemetry.Trace;
using OpenTelemetry.Resources;

internal class Program
{
    private static void Main(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);

        //Mongodb instrumentation config
        var mongoUrl = MongoUrl.Create("mongodb://localhost:27017");
        var clientSettings = MongoClientSettings.FromUrl(mongoUrl);
        clientSettings.ClusterConfigurator = cb => cb.Subscribe(new DiagnosticsActivityEventSubscriber());
        var mongoClient = new MongoClient(clientSettings);

        //OTel config
        var serviceName = "todo-dotnet";
        var serviceVersion = "1.0.0";
        builder.Services.AddOpenTelemetryTracing(tracerProviderBuilder =>
            tracerProviderBuilder
                .AddOtlpExporter(opt =>
                {
                    opt.Endpoint = new Uri("https://collector.aspecto.io:4317/v1/traces");
                    opt.Headers = $"Authorization={Environment.GetEnvironmentVariable("ASPECTO_API_KEY")}";
                })
                .AddSource(serviceName)
                .SetResourceBuilder(
                    ResourceBuilder.CreateDefault()
                        .AddService(serviceName: serviceName, serviceVersion: serviceVersion))
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation()
                .AddMongoDBInstrumentation()
        );

        //dependency injection config
        builder.Services.AddControllers();
        builder.Services.AddSingleton(mongoClient.GetDatabase("todo"));
        var app = builder.Build();
        app.MapControllers();
        app.Run();
    }
}

Now you should be able to run your application and view the traces in Aspecto.

Run Mongo locally. You can do it with docker.

docker run -p 27017:27017 --name mongo -d mongo:latest

The end result should look something like this:

Advanced OpenTelemetry Learning

If you want to learn more about OpenTelemetry and become an OTel ninja, check out this free, 6-episode, OpenTelemetry Bootcamp (vendor-neutral).

This is your OpenTelemetry playbook where you will learn everything, from the very basics all the way to running OTel in production. Including how to manage cost, sampling, messaging systems, and much more.

Episode 1: OpenTelemetry Fundamentals
Episode 2: Integrate Your Code (logs, metrics, and traces)
Episode 3: Deploy to Production + Collector
Episode 4: Sampling and Dealing with High Volumes
Episode 5: Custom Instrumentation
Episode 6: Testing with OpenTelemetry

Aspecto OpenTelemetry Sampler for NodeJS

Amir Blum — Mon, 19 Dec 2022 15:17:00 +0000

At Aspecto, we enable users to configure advanced sampling rules (both head and tail sampling) in an easy-to-use UI with reach features and centralized management. It's a control plane for your trace sampling.

The Aspecto Sampler for NodeJS is an implementation of the OpenTelemetry head sampling that allows you to remotely configure all head sampling needs from one UI with zero code changes.

Introduction

Until today, head sampling capabilities were part of our OpenTelemetry distribution, which included a full OpenTelemetry SDK, instrumentations, resource detectors, payload collection, and more. Due to ongoing demand, we are introducing AspectoSampler. Now you can craft your own custom vendor-agnostic OpenTelemetry NodeJS SDK setup and plug the Aspecto sampler with just a few lines of code.

Here is an example of setting up instrumentation for your typescript service from the official OpenTelemetry docs:

/*tracing.ts*/
import * as opentelemetry from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
const sdk = new opentelemetry.NodeSDK({
 traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
 instrumentations: [getNodeAutoInstrumentations()]
});
sdk.start()

Your real-life setup will probably include more components, like resource detectors and an otlp exporter to ship the telemetry data to an OpenTelemetry Collector or your chosen vendor (Aspecto, ahem ahem 👀)

This simple installation of OpenTelemetry will record all traces -- each operation going on inside your service. While collecting all spans is fine for POCs, production workloads usually create a firehose of costly and noisy data. Sampling is a powerful tool in your OpenTelemetry toolbox to reduce your costs and focus on telemetry that is useful for you. Here is the same code with an Aspecto sampler integrated (just two lines of code):

/*tracing.ts*/
import * as opentelemetry from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { AspectoSampler } from "@aspecto/opentelemetry-sampler";
const sdk = new opentelemetry.NodeSDK({
  traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
  instrumentations: [getNodeAutoInstrumentations()],
  sampler: new AspectoSampler(),
});
sdk.start()

That is it! Just two lines of code that you will never need to touch again, and you unlocked the FULL power of sampling 🕺

Now let's see how it works.

Plain OpenTelemetry Sampling

Once you realize you collect millions of boring and useless health-check traces or that active endpoint in your system that floods your trace viewer and blows up your motley bill, you ask yourself -- how do I get rid of them!? And the immediate answer is sampling. It is a widely used practice and the ideal go-to solution for this problem.

OpenTelemetry provides out-of-the-box samplers such as TraceIdRatioBased. These are great if you want to blindly record a percentage of the traces (sample 1% of spans).

It is a simple strategy to implement, but despite reducing the overall trace amount, you randomly sample traces while dropping other critical ones, those you always want to collect.

After you realize you need more fine-grained control of your sampling, you can write your own custom sampler by implementing the Sampler interface. You might start with something like this:

class MySampler implements Sampler {
  shouldSample(context: Context, traceId: string, spanName: string, spanKind: SpanKind, attributes: Attributes) {
      if (attributes[SemanticAttributes.HTTP_TARGET] === '/health-check') {
          return { decision: SamplingDecision.NOT_RECORD };
      }
      // fallback
      return { decision: SamplingDecision.RECORD_AND_SAMPLED };
  }
  toString() { return 'MySampler'; }
}

You then wrap with the following, to apply the sampling decision on the entire trace based on its root span.

const sampler = new ParentBasedSampler({ root: new MySampler() });

Then you test this code, create a PR, get it approved, and deploy it to your dozens of microservices.

A few days later, your vendor's Billing dashboard already looks happier with fewer spans and less cost.

Then you browse your Trace Search screen and realize that 95% of your remaining traces come from one heavily used endpoint that gets executed millions of times daily. You still want observability into it but ideally, collect fewer spans than what you do today. You wish you could sample just 1% of this endpoint.

You might go back to your custom sampler and add something like this:

import { Sampler, ParentBasedSampler, TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-base';
import { SemanticAttributes } from '@opentelemetry/semantic-conventions';
import { Context, SpanKind, Attributes, SamplingDecision } from '@opentelemetry/api';
class MySampler implements Sampler {
  private probabilitySampler = new TraceIdRatioBasedSampler(0.01);
  shouldSample(context: Context, traceId: string, spanName: string, spanKind: SpanKind, attributes: Attributes): SamplingResult {
      if (attributes[SemanticAttributes.HTTP_TARGET] === '/health-check') {
          return { decision: SamplingDecision.NOT_RECORD };
      }
      if ( attributes[SemanticAttributes.HTTP_TARGET] === '/my/busy/endpoint') {
          return this.probabilitySampler.shouldSample(context, traceId);
      }
      // fallback
      return { decision: SamplingDecision.RECORD_AND_SAMPLED };
  }
  toString() { return 'MySampler'; }
}
const sampler = new ParentBasedSampler({ root: new MySampler() });

And again, create a PR, review, merge, deploy, test, and watch your billings drop and your trace viewer becomes less busy.

At this point, you, your manager, and every other trace consumer in your company are happy, but that is not where the story ends.

Now, maintaining this sampler becomes one of your tasks. You might need to implement it in multiple languages, add dozens of complex statements, synchronize it across dozens of microservices and serverless functions and debug it when it fails to do what you meant. Requests keep flowing. "I want to see more of this", "can you hide that?", and "why can't I find my trace??"

One day, an incident in production hits your /my/busy/endpoint. You say to yourself -- I wish I could see all traces for just a few hours until we understand what is going on. But modifying and deploying new versions is not your first priority, while the incident keeps everyone stressed.

If you deploy changes in sampling, you need to remember to go back and revert these changes after the incident is resolved.

This problem does not scale well. Managing sampling for a real-life production environment in code can be a massive headache.

Introducing Aspecto Sampler 👏🎉

Aspecto Sampler

Aspecto has years of experience implementing OpenTelemetry in high-workload production environments. We have collected needs and use cases from our customers and iterated them to crystallize the best-managed sampling mechanism, so you can focus on your task and leave the hard work and technical implementation to us.

Aspecto sampler abstracts away all the nitty details and implementation code and exposes a UI where you configure the logic in a simple interface. It is basically the sampler from above, but you do not need to write it yourself. Plus, it provides advanced features you can explore to customize your sampling to your specific needs.

Let's review how it works and its features.

Remote Configuration

All sampling configurations are centralized and managed on the Aspecto platform. You only need to install the sampler into your OpenTelemetry SDK once or use our OpenTelemetry distribution and never touch it again.

Up-to-date Configuration

Aspecto Sampler always fetches the latest configuration. Sampling configuration is always up-to-date and consistent across all your services. You do not need to deploy any code or worry that some services still run with an old configuration that is not getting updated.

Automatic Updates

Your changes are immediately and automatically pushed to all your samplers in seconds. With one click, all your services are updated live. No need to restart, deploy or monitor anything.

No Code

Adding, updating, or removing sampling rules does not require any code changes. It is all done in an easy-to-use UI and involves zero code. Any function in your organization can manage it, not only developers who master a specific programming language.

Instant

Any change takes effect immediately. You can experiment and fine-tune your configurations and get fast feedback.

UI

The entire sampling workflow is managed and configured in a dedicated UI in the Aspecto app, built to give you focused and easy yet powerful tools to achieve your sampling goals.

Multiple Languages

One unified interface to customize your sampling for multiple programming languages. No need to master or jump between languages to implement a sampling policy in your organization.

Timer Rules

If you are debugging something or working on a specific service or endpoint, it is convenient to "sample everything" while you need it, but you also do not want to forget about it and see your tracing costs inflate. You can add rules with a timer -- so you get instant visibility into the desired area and sleep well knowing that your traces bill will not blow up.

Turn Rules On and Off

With a single click, you can turn a rule on and off, dynamically adapting sampling as you go and integrating sampling tools into your everyday workflow to find the right balance between costs and telemetry verbosity.

Search Capabilities

OpenTelemetry implementation can easily include many sampling rules. Search your rules with free text to quickly find what you need. You can sort them and check who added each rule and when.

Rich Sampling Language

Define your sampling strategy with rules, which are evaluated in order to derive a sampling decision for each new trace.

Conditions

Use span attributes like http path, http method, messaging queue or topic name, or any custom span attribute to add conditions. Describe each one with a rich set of operators, such as HTTP path starts with "/user" or even write regular expressions for parameterized routes like */account/:id/users*

Services and Environments

Narrow a rule to affect only a specific service/s or environment. For example, you can easily add a rule that only affects the users-service in the production environment.

Sampling Rates and Priority

Arrange your rules according to priorities, place-specific important rules on top, and general fallback rules at the bottom. Each rule defines a sampling rate that can be 0% (record nothing).

Installation Instructions

Visit our docs for complete instructions on installation and configuration.

SAMPLER DOCS

Tail Sampling

The sampler above is a Head Sampler that applies sampling decisions in the SDK as spans are created.

A popular alternative is to use the Aspecto Tail Sampler -- an OpenTelemetry Collector distribution that aggregates spans into traces and applies a sampling decision on an entire trace. Each option has pros and cons, and we encourage you to research and choose the option that best fits your needs.

Not sure what to choose? Schedule a call with our OpenTelemetry experts to assist in the process.

Feedback

We are in a constant process of learning and improving and would love to hear from you. Do not hesitate to contact us (also via our website chat) to share feedback / ask questions / or get support for our managed sampling product.

Our OpenTelemetry experts are here for you.

Supported Languages

Our managed sampler has the following support:

NodeJS -- integrated with our OpenTelemetry distribution, or as a standalone sampler which you can integrate into your custom opentelemetry setup
Ruby -- integrated into our Ruby OpenTelemetry distribution

Support for more languages is coming soon.

Our Tail Sampler can be used with any OpenTelemetry SDK and components that produce trace data.

OpenTelemetry Operator for Kubernetes: Practical Guide | Part 4

Yoav Danieli — Tue, 15 Nov 2022 15:39:00 +0000

In this article, you will learn how to use the OpenTelemetry Operator. We will explore what a Kubernetes Operator is, how to use it, and tackle common issues you might encounter when setting up the Opentelemetry Operator in your cluster.

This is the 4th part of our OpenTelemetry on Kubernetes series.

In the previous articles, I explained what the Opentelemetry Collector is, how to configure it, and how to set up a local Kubernetes cluster using Minikube and deploy the collector to that cluster. In addition, I described the deployment methods for a collector and how to use them (Collector as a Gateway and a Sidecar agent). Lastly, I showed for any deployment how to examine traces generated by our programs and visualize them using Aspecto.

Good to know before reading this article:

OpenTelemetry Collector on Kubernetes Basics | Part 1
OpenTelemetry Collector as an Agent on Kubernetes | Part 2
Opentelemetry Collector on Kubernetes Using Helm | Part 3

What is a Kubernetes Operator?

An operator extends Kubernetes to manage applications and their components using custom resources. Custom resources allow you to define new resources that group API objects with specific settings.

For example, we can create an application custom resource that accepts an image of a web server application and a port. This resource is composed of a deployment object and a service object that exposes this port.

To manage resources, Kubernetes uses the controller pattern. Notably the control loop. It means that Kubernetes runtime is always watching the current state of the resources in a cluster and comparing them to the desired state. Then it tries to reconcile to bridge the gap between the states if one exists.

Operators are custom controllers and custom resources put together. It is software running in your cluster, the same as the Kubernetes runtime, and watches the custom resources and their state. Operators allow complex clusters and systems to run automatically.

In short, Operators extend the cluster's behavior without modifying Kubernetes code.

What is OpenTelemetry Operator?

The OpenTelemetry Operator is an operator written by the OpenTelemetry community and aims to manage and simplify the way we incorporate observability into our system. It aims to solve the challenges of any developer that wants to add observability to their cluster encounters.

The first challenge is the necessity to configure and manage OpenTelemetry Collectors that answer the specific requirements of your system. The second challenge is instrumenting your core business logic and generating telemetry data to observe.

The OpenTelemetry Operator introduces two CRDs as a solution to these challenges:

OpentelemetryCollector -- This resource simplifies the configuration and maintenance of a Collector.
Instrumentation -- This resource can instrument your application for you. No change to your code is required.

In the remainder of this article, I will demonstrate how to use the Operator to build a working and instrumented web server.

OpenTelemetry Operator: The Practical Part

Setting up our application

First, we will create our application and deploy it to a local development cluster.

The application code is short. We are creating a simple express server that listens on port 3000 and returns a hello world message. The code is written in typescript.

Let's create tsconfig.ts, package.json, and a Dockerfile to build the image.

Here you will find the source code.

// application/index.ts

import express, { Request, Response } from 'express';

const PORT = process.env.PORT || 3000;

const app = express();
app.get('/', (req: Request, res: Response) => {
res.send('Hello World!');
});

app.listen(PORT, () => {
console.log(`Running on ${PORT}`);
});

Use the following Docker file to build the application:

FROM node:16
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
EXPOSE 3000
CMD [ "node", "index.js" ]

Now run:

docker build -t example-express-server ./application

Now it's time to set up our cluster. To set up a development environment please follow part 1 of this series guide.

Let's create a new cluster. We will clean up everything we did so far and start fresh:

minikube delete --purgeminikube start

Create a new file to define the resources for our application:

touch application.yml

And a Kubernetes namespace to isolate our application logic:

kubectl create namespace application

And a deployment resource and a service:

apiVersion: v1
kind: Service
metadata:
name: express-server
namespace: application
labels:
  app: application
  component: express-server
spec:
ports:
  - name: express # Default endpoint for OpenTelemetry gRPC receiver.
    port: 3000
    protocol: TCP
    targetPort: 3000
selector:
  component: express-server
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: express-server
namespace: application
labels:
  app: application
  component: express-server
spec:
selector:
  matchLabels:
    app: application
    component: express-server
replicas: 1
template:
  metadata:
    labels:
      app: application
      component: express-serve
  spec:
    containers:
      - name: express-server
        ports:
          - containerPort: 3000
        image: example-express-server
        imagePullPolicy: Never
        resources:
          limits:
            memory: '128 Mi'
            cpu: '500m'

We created a service of type Load balancer to expose the application to the outside of the cluster.

Note that we are using a locally built image, so we need to configure minikube not to look for this image elsewhere. Read about this issue here.

eval $(minikube docker-env)

Create a minikube tunnel to expose our cluster to our machine:

# run in a different shell and keep it open
> minikube tunnel

Now let's apply the changes and call the server to see that it is working:

> kubectl apply -f application.yml
> curl localhost:3000
Hello World!%

Collector Gateway with OpenTelemetry Operator

The server is up and running. Now it's time to design and extend our system by adding observability.

You can read more about deployment strategies in previous articles. For now, our setup will be as follows:

OpenTelemetry Collector gateway deployed as a Kubernetes deployment. This Collector will receive telemetry data on port 4317 and exports the data to a local Jaeger instance.
A Jaeger 'all-in-one' instance to observe telemetry data (traces)
The application we defined above will run with a collector agent as a sidecar. In addition, we will use the OpenTelemetry Operator Instrumentation resource to instrument the application without changing its code.

Create a namespace for our OpenTelemetry resources:

kubectl create namespace opentelemetry

Let's use the following manifest to deploy a Jaeger instance and network services:

# jaeger.yml

apiVersion: v1
kind: Service
metadata:
name: jaeger-all-in-one
namespace: opentelemetry
labels:
  app: opentelemetry
  component: otel-collector
spec:
ports:
  - name: collector
    port: 14250
    protocol: TCP
    targetPort: 14250
selector:
  component: otel-collector
---
apiVersion: v1
kind: Service
metadata:
name: jaeger-all-in-one-ui
namespace: opentelemetry
labels:
  app: opentelemetry
  component: otel-collector
spec:
ports:
  - name: jaeger
    port: 16686
    protocol: TCP
    targetPort: 16686
selector:
  component: otel-collector
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jaeger-all-in-one
namespace: opentelemetry
labels:
  app: opentelemetry
  component: otel-collector
spec:
replicas: 1
selector:
  matchLabels:
    app: opentelemetry
    component: otel-collector
template:
  metadata:
    labels:
      app: opentelemetry
      component: otel-collector
  spec:
    containers:
      - image: jaegertracing/all-in-one:1.35
        name: jaeger
        ports:
          - containerPort: 16686
          - containerPort: 14268
          - containerPort: 14250

Apply the resources in the cluster:

> kubectl apply -f jaeger.yml

Visit http://localhost:16686 and check that the Jaeger UI is running and available.

Now we will use the OpenTelemetryCollector CRD to deploy a collector as a gateway.

First, we need to install the OpenTelemetry Operator. Follow the official documentation for more information.

> kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

You will probably receive an error about needing a cert-manager installed. So Let's install that also:

> kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml
> kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

This installs the OpenTelemetry Operator control plane in your cluster under the namespace 'opentelemetry-operator-system'. You can check out all the extra software your cluster is now running with:

> kubectl get all -n opentelemetry-operator-system

Now that we have the Operator installed, we can use the custom resources it offers. Let's create a gateway.yml file with the following config:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-collector
namespace: opentelemetry
labels:
  app: opentelemetry
  component: otel-collector
spec:
mode: deployment
config: |
  receivers:
    otlp:
      protocols:
        grpc:
        http:
  exporters:
    jaeger:
      endpoint: jaeger-all-in-one:14250
      tls:
        insecure: true
    logging:
  processors:
    batch:
    resource:
      attributes:
        - key: test.key
          value: "test-value"
          action: insert
  extensions:
    health_check:
    zpages:
      endpoint: :55679
  service:
    telemetry:
      logs:
        level: "debug"
    extensions: [zpages, health_check]
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch, resource]
        exporters: [logging, jaeger]

Apply the changes:

> kubectl apply -f gateway.yml

This resource deploys the OpenTelemetry Collector with the defined config as a deployment. I will not get into the configuration file since we explained it in depth in previous articles. What is important to note here is the mode specification. The available values are:

Deployment
Sidecar
DeamonSet

In addition to the Collector deployment, this resource also handles running the necessary service to communicate with the collector. How simple is that? If you read my previous articles, you can see much larger manifest files. Here this is not the case. The Operator simplified all of this process for us.

Collector Agent with OpenTelemetry Operator

The next part will add the opentelemetry collector agent to run as a sidecar for our application. Create a new file for the sidecar:

# sidecar.yml

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: sidecar
namespace: application
spec:
mode: sidecar
config: |
  receivers:
    otlp:
      protocols:
        grpc:
        http:
  processors:
    batch:
  exporters:
    logging:
    otlp:
      endpoint: "http://otel-collector-collector.opentelemetry.svc.cluster.local:4317"
      tls:
        insecure: true
  service:
    telemetry:
      logs:
        level: "debug"
    pipelines:
      traces:
        receivers: [otlp]
        processors: []
        exporters: [logging, otlp]

As you can see, the mode is set to sidecar while the config is set to the agent config we defined in the previous article.

An important note that can cause issues is that we need to pass the telemetry data forward to the gateway collector. Now, Because they are running in different namespaces, we need to specify the full cluster DNS for the desired service.

Get the name of the Gateway service by typing:

> kubectl get svc -n opentelemetry

Choose the service that listens on port 4317.

The full DNS is composed as follows:

[service_name].[namespace].svc.cluster.[local | remote_url]:[port]

Let's wait with applying this (I will explain why after the next section). For now, continue and create the instrumentation for our application.

Auto Instrumentation with OpenTelemetry Operator

Create a new manifest file with the following resource:

# instrumentation.yml

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: nodejs-instrumentation
namespace: application
spec:
propagators:
  - tracecontext
  - baggage
  - b3
sampler:
  type: always_on
nodejs:

We are using the Instrumentation CRD. It lets us define all kinds of settings regarding how we want to instrument our application. The most important ones are the propagator, sampler, and programming language (or runtime) specification.

We will not get into each of the configurations available by the instrumentation since this requires a separate article. If you are interested in reading more about it, visit the official docs.

The Propagators let our injected instrumentation SDK know how to propagate telemetry data. Let's use the propagators defined in the OpenTelemetry repo example.

The sampler specifies the sampling strategy to be executed by this instrumentation. Go here to read more about sampling.

Lastly, we define the Nodejs part of the instrumentation to use the default values. The other programming languages supported are Python, .Net, and Java.

OpenTelemetry Operator: Putting it all together

The last thing we need to do is update our application resource and add annotations. It is how the Operator knows to inject the sidecar and the instrumentation into the deployment resource.

Update application.yml and add the following annotation:

spec:
template:
  metadata:
    annotations:
      instrumentation.opentelemetry.io/inject-nodejs: 'nodejs-instrumentation'
      sidecar.opentelemetry.io/inject: 'sidecar'

Notice that the instrumentation only works if it runs before the application. It needs to be configured and applied in the cluster before the application.

Shut down the application:

> kubectl delete -f application.yml

And apply all of the above configurations:

> kubectl apply -f instrumentation.yml
> kubectl apply -f sidecar.yml
> kubectl apply -f application.yml

Let's see what's going on by examining the application:

> kubectl logs deployments.apps/express-server -n application
Defaulted container "express-server" out of: express-server, otc-container, opentelemetry-auto-instrumentation (init)

We can see two more containers run inside the deployment. The otc-container which is the sidecar, and the instrumentation runs as an init container. It means it runs just before the other containers and is used for running commands and prefixing them to other image commands.

Let's invoke the application endpoint one more time:

> curl localhost:3000
Hello World!%

We can see that we got back the hello world message. Check out http://localhost:16686 and verify that the telemetry data reached its final destination.

That is it! Using the OpenTelemetry Operator was simple, intuitive, and with much fewer issues than configuring each Kubernetes object by ourselves.

The most important advantage is the ability to instrument applications without changing their code.

It was a fascinating subject to write about, and I hope you enjoyed it as much as I did.

If you have any questions or comments, feel free to CONTACT me on LinkedIn. As my wife says, I'm wrong quite a lot, so if you see a mistake, please let me know.

See you next time.

OpenTelemetry Collector on Kubernetes with Helm Chart – Part 3

Yoav Danieli — Sun, 13 Nov 2022 14:16:00 +0000

Written by Yoav Danieli @ Team Aspecto

In this article, you will learn how to use the OpenTelemetry Helm chart to deploy a collector as a gateway in a Kubernetes cluster.

In previous tutorials, I demonstrated how to deploy the OpenTelemetry Collector as both Gateway and Agent on a Kubernetes cluster. It took a lot of work and hours of research. Luckily there is an easier way. Enter Helm Charts.

Related Guides

What is Helm?

Before we start using Helm, let's give it a quick introduction.

Helm is an open-source package manager for Kubernetes. It allows software built for Kubernetes to be provided, used, and shared. It's also a graduate project from the Cloud Native Computing Foundation (CNCF), similar to Kubernetes, Jaeger, and OpenTelemetry (although OTel is still incubating).

Helm packages software uses a format called Charts. A chart is a collection of files that form a set of Kubernetes resources. We can use a single chart to deploy simple things like a single-pod hello-world program or enormously complex stuff like a full-stack web app with HTTP servers, databases, caches, and more.

For more information about Helm, visit Helm Docs.

Now let's dive right in and figure out how to use the Helm chart provided by OpenTelemetry.

Using OpenTelemetry Helm Chart

Installing Opentelemetry-Collector Helm Chart

1. Install Helm using homebrew.

brew install helm

Add the OpenTelemetry helm repository. A helm repository is an HTTP server from which we can find and install helm charts. The following command will establish a connection with the repository server and allow us to find and install its resources.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

3. Search the repo and find out which charts it offers:

helm search repo open-telemetry

This will produce the following output:

NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
open-telemetry/opentelemetry-collector  0.34.0          0.61.0          OpenTelemetry Collector Helm chart for Kubernetes
open-telemetry/opentelemetry-demo       0.3.1           0.4.0-alpha     opentelemetry demo helm chart
open-telemetry/opentelemetry-operator   0.13.0          0.60.0          OpenTelemetry Operator Helm chart for Kubernetes

The repository offers three charts:

OpenTelemetry-collector -- The chart that we will use to create the collector.
OpenTelemetry-demo -- This is a demo chart that creates an incredibly complex microservice system. It's written in many programming languages, and you can find good examples for almost any use case you need.
OpenTelemetry-operator -- This chart helps us set up an OpenTelemetry operator in our project. The operator provides the implementation of the OpenTelemetry collector and injects auto instrumentation to the services running inside our cluster. I will demonstrate the usage of the OpenTelemetry collector in future articles.

When using a helm chart, we are provided with a pre-defined set of parameters. These parameters, as well as their default values, can be found in a file called values.yml. To show the file, we can use the command:

helm show values open-telemetry/opentelemetry-collector

But it is easier to jump to the chart source code and look at the files in the browser. I will not elaborate on each of the values, instead, I will demonstrate how to create the same Gateway Collector deployment we created above. Hopefully, it would be much simpler.

First, clean up your cluster and remove all resources within it. Getting started with a clean slate

Let's install the chart as is. With its default values.

helm install opentelemetry-collector open-telemetry/opentelemetry-collector

We get an error.

[ERROR] 'mode' must be set.

As stated in the source code from the link above. The default value for mode is set to an empty string while valid values are 'deployment'. 'daemonset' or 'statefulset'.

Let's add a mode value.

Create a collector-values.yml file and set the mode to deployment, the same as our Gateway.

helm install opentelemetry-collector open-telemetry/opentelemetry-collector -f ./collector-values.yml

So now we have an up-and-running collector.

💡Pro Tip -- We can replace the command 'install' with 'template' to see the exact Kubernetes resources rendered and deployed to the cluster. Looking at the resources generated by the helm chart, we see similarities to the resources we used ourselves. And all we did was install the chart with a specific mode!

OpenTelemetry-Collector Helm Chart Configuration

To configure it the same way as the Gateway collector we deployed in the previous chapters, there are more values we need to set. Let's start with the configuration taken from the chart's examples under 'deployment-otlp-traces'.

mode: deployment

ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

config:
  receivers:
    jaeger: null
    prometheus: null
    zipkin: null
  service:
    pipelines:
      traces:
        receivers:
          - otlp
      metrics: null
      logs: null

Here we disable the ports we are not using and enable only the traces pipeline, with only the otlp receiver. This configuration will install a collector that receives telemetry data via grpc and http protocols in otlp format.

There are no exporters defined, so the chart will render the default exporter which is a simple logging. Let's add the OTLP exporter for Aspecto, our distributed tracing platform. And let's also add Jaeger exporter:

config:
 receivers:
   jaeger: null
   prometheus: null
   zipkin: null
 exporters:
   otlp/aspecto:
     endpoint: otelcol.aspecto.io:4317
     headers:
       Authorization: ${ASPECTO_API_KEY}
   jaeger:
     endpoint: jaeger-all-in-one:14250
     tls:
       insecure: true
   logging:
     loglevel: debug
 service:
   telemetry:
     logs:
       level: debug
   pipelines:
     traces:
       receivers:
         - otlp
       exporters:
         - otlp/aspecto
         - jaeger
         - logging
     metrics: null
     logs: null

Populate the ASPECTO_API_KEY with a secret, same as above. The OpenTelemetry collector chart supports it by using the extraEnv property:

extraEnvs:
 - name: ASPECTO_API_KEY
   valueFrom:
     secretKeyRef:
       name: aspecto
       key: api-key
       optional: false

We also need to expose our collector to the internet using the load balancer service:

service:
 type: LoadBalancer

The last thing we should do is tell the helm chart which OpenTelemetry collector image it should use and how. By default, it uses the latest image from the otel/opentelemetry-collector-contrib docker repo.

We want to change it to the stable collector-core image of our choosing tag.

image:
 repository: otel/opentelemetry-collector
 tag: "latest"

command:
 name: otelcol

Upgrade the deployment:

helm upgrade opentelemetry-collector open-telemetry/opentelemetry-collector -f ./collector-values.yml

And also deploy Jaeger UI, so we can visualize our traces:

kubectl apply -f jaeger.yml

After checking the logs and status of our Collector and Jaeger deployments and verifying everything is ready, we can start our test service to generate traces.

Once the service runs, you should see traces reaching the destinations configured in the exporter's section.

Final Thoughts and Aspecto Helm Chart

Using the OpenTelemetry-Collector Helm chart was simple and quick compared to manually defining each resource. I highly recommend using it in your clusters.

If you reach a point where you need to extend those resources or add properties that are not enabled via the helm chart, you can output the template resource with the helm template command and use it. Easy peasy.

In addition, some distributed tracing vendors offer their own helm chart to add a vendor-specific functionality to the Collector configuration.

Check Aspecto Helm charts here.

ASPECTO HELM CHART

Thanks for reading, and if you have any questions, I'm happy to connect and help.

P.S. You can find a lot of the concepts covered here in our latest live workshop on deploying the OpenTelemetry Collector on Kubernetes.

OpenTelemetry Collector as an Agent on Kubernetes – Part 2

Yoav Danieli — Fri, 11 Nov 2022 13:11:00 +0000

Written by Yoav Danieli @ Team Aspecto

In this article, you will learn how to deploy and configure the OpenTelemetry Collector as an agent on Kubernetes.

In part 1, we described how to set up a local Kubernetes environment with Minikube. We configured an OpenTelemetry collector and deployed it to a local Kubernetes cluster. We outlined and explained each of the Kubernetes resources we need to deploy the Opentelemetry Collector as a Gateway.

You can read this article as a stand-alone guide or as part 2 of our OpenTelemetry Collector on Kubernetes series.

Related Guides

Why Deploy the OpenTelemetry Collector as an Agent

We use the collector gateway as a central point for all our telemetry data from our distributed architecture. It means the collector usually receives a lot of traffic and therefore is responsible for handling this traffic.

Deploying specific collector instances for certain services can help us offload some of those responsibilities. For example, the agent can handle actions like batching, compression, retries, and more.

In addition, it improves the overall performance of the system. Since the agent is running in the same host as our services, there is no DNS resolving (it's just localhost). Thus it receives the data much faster.

Lastly, having a specific collector instance per service means we can add specific features and configurations to our telemetry data. We can add specific attributes and use any collector process configuration specific to this service.

Here is an illustration of running a collector gateway + agent as sidecar architecture:

There are a couple of strategies to deploy the collector as an agent.

We can deploy it as a DeamonSet, Sidecar, or StatefullSet. Each strategy has its use cases. To read more about deployment strategies, I recommend reading this article.

In this tutorial, I will deploy the OpenTelemetry collector agent as a sidecar.

Kubernetes Resource Configuration for Collector as an Agent

Let's create a new manifest file named business-application.yml. We will start by creating the agent's configuration.

One exporter and one receiver on OTLP protocols:

apiVersion: v1
kind: ConfigMap
metadata:
 name: otel-agent-conf
data:
 otel-agent-config: |
   receivers:
     otlp:
       protocols:
         grpc:
         http:
   processors:
     batch:
   exporters:
     otlp:
       endpoint: otel-collector:4317
       tls:
         insecure: true
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: []
         exporters: [otlp]
---

Secondly, we will create a deployment with two containers. The application and the sidecar.

For the application service, I will use a test service that sends an example span every 5 seconds to the agent's endpoint. For all code examples, please check out the source code.

I built it using:

docker build -t test-service ./test-service

Here is the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: test-service
spec:
 selector:
   matchLabels:
     app: test-service
 template:
   metadata:
     labels:
       app: test-service
   spec:
     containers:
       - name: test-service
         image: test-service
         env:
           - name: SERVICE_NAME
             value: test-service
         resources:
       - name: otel-agent
         image: otel/opentelemetry-collector:latest
         command:
           - '/otelcol'
           - '--config=/conf/otel-agent-config.yaml'
         resources:
         volumeMounts:
           - name: otel-agent-config-vol
             mountPath: /conf
     volumes:
       - configMap:
           name: otel-agent-conf
           items:
             - key: otel-agent-config
               path: otel-agent-config.yaml
         name: otel-agent-config-vol

Deploying the OpenTelemetry Collector Gateway + Agent locally

Let's apply the configuration we have so far to check the new service is working and deploy the agent and gateway to verify the new service is working.

You can find all gateway configurations in this gateway.yml file.

Deploy it using

kubectl apply -f gateway.yml

Next deploy the manifests we just wrote using:

kubectl apply -f business-application.yml

The application crashed.

Issue #1 -- Use locally built docker inside a Kubernetes cluster. Printing the log from the test-service pod we get the following message:

Defaulted container "test-service" out of: test-service, otel-agent
Error from server (BadRequest): container "test-service" in pod "test-service-5646dd8d77-fp27c" is waiting to start: trying and failing to pull image

The cluster can't build the image because it can't pull it. It searches the cluster's docker daemon for the image and can't find it. The solution, as described here, is to push the image to the local docker environment of minikube.

Following the documentation, we enter the following, change the imagePullPolicy to Never, and run it again:

eval $(minikube docker-env)

Let's verify that everything runs smoothly. The logs state that the exporter could not be found. It makes sense because we didn't create a service to enable this communication.

Add another service resource in the gateway manifest. It needs to be of type ClusterIP and have the gateways labels in its selector block:

apiVersion: v1
kind: Service
metadata:
 name: otel-collector-in
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 type: ClusterIP
 selector:
   component: otel-collector
 ports:
   - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
     port: 4317
     protocol: TCP
     targetPort: 4317
   - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
     port: 4318
     protocol: TCP
     targetPort: 4318

Let's run the application once more. Still no connection with the exporter.

Issue #2 -- Cross Namespace Communication in Kubernetes. When running pods in different namespaces the pod's DNS is suffixed with the name of the namespace. In our config map, we need to specify the full DNS in the exporter's endpoint:

endpoint: otel-collector.opentelemetry.svc.cluster.local:4317

Read the Kubernetes documentation regarding DNS to learn more.

After fixing the last issue and running the cluster once more, we can see the traces reaching the exports we configured in the Gateway. Success!

Final Thoughts

In this article, we configured a simple Collector agent that runs as a sidecar for an example span-generating application.

We outlined how to deploy and configure the OpenTelemetry Collector as an agent on Kubernetes.

Please feel free to check out the source code for this article. Thanks for reading and please let me know if you have any questions or comments.

The next part of this series will explain how to save us a lot of time and effort by using Helm.

OpenTelemetry Collector on Kubernetes – Part 1

Yoav Danieli — Thu, 10 Nov 2022 15:02:00 +0000

Written by Yoav Danieli @ Team Aspecto

In this guide, you will learn how to set up a working OpenTelemetry Collector on Kubernetes that is configured to receive data in OTLP format via HTTP or gRPC and send traces for visualization in both Jaeger and Aspecto.

The purpose of this tutorial is to simplify the process by going through the main steps and solving common issues.

Before we start I recommend reading the following blog posts that explain:

To demonstrate how the OpenTelemetry collector can be deployed on a Kubernetes cluster, I will use a similar configuration to that in the second article.

Also, a basic knowledge of Kubernetes could help. I will explain the Kubernetes commands and resources I'm using, but the inner workings of Kubernetes are out of this article's scope.

Next in this series

OpenTelemetry Collector on Kubernetes: Creating a Development Environment

We will use Minikube Kubernetes runtime in this article. There are many other choices for local development such as Kind, k3s, MicroK8s, and more. Each one has its unique setup, but the configuration files for the cluster are (almost) the same so feel free to play with your preferred runtime.

To set up a Kubernetes cluster we will first need to install:

Kubectl:

brew install kubectl

Then, install minikube:

brew install minikube

To start a cluster run:

minikube start

Minikube automatically configures kubectl to work with the newly created cluster.

A cool feature of minikube is the dashboard feature. By running minikube dashboard in a different terminal, you can open a convenient UI that you can use to examine your local cluster. It is helpful for beginners, but I would recommend relying on this sparingly.

Once you migrate to a cloud environment, you'll still need to rely on the command line.

Run OpenTelemerty Collector as a Gateway in Kubernetes cluster

Kubernetes Resource Configuration

Namespace

Now that we have a cluster running, we can start creating Kubernetes resources. Start by creating a namespace for all of our OpenTelemetry-related work. Although it is not a must, it's best practice to separate logical groups into namespaces.

Create a file named otel-collector-config.yml and add the namespace object:

# otel-collector-config.yml
apiVersion: v1
kind: Namespace
metadata:
 name: opentelemetry

To deploy the configuration to the cluster and create or update resources we use the command:

kubectl apply -f ./otel-collector-config.yml

By running kubectl get namespaces, we can verify that our 'opentelemetry namespace' was indeed created. We can also see namespaces used for the Kubernetes internal workings as well as a default namespace.

ConfigMap

Next, we want to create a ConfigMap resource. To pass the collector a configuration file, we need to store it inside the cluster.

We use a ConfigMap. In the config file, enter three dashes after the namespace --- This lets kubectl know where a new resource ends and another begins.

After that, enter the following ConfigMap resource:

---
apiVersion: v1
kind: ConfigMap
metadata:
 name: otel-collector-conf
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector-conf
data:
 otel-collector-config: |
   receivers:
     otlp:
       protocols:
         grpc:
         http:
   exporters:
     otlp/aspecto:
       endpoint: otelcol.aspecto.io:4317
       headers:
         Authorization: ${ASPECTO_API_KEY}
     jaeger:
       endpoint: localhost:14250
       tls:
         insecure: true
     logging:
   processors:
     Batch:
     resource:
       attributes:
         - key: test.key
           value: "test-value"
           action: insert
   extensions:
     health_check:
     zpages:
       endpoint: :55679
   service:
     telemetry:
       logs:
         level: "debug"
     extensions: [zpages, health_check]
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [logging, jaeger, otlp/aspecto]

Note this is a similar configuration to the one in the article I mentioned at the beginning. But let's go over it quickly to be sure:

Receivers -- We are defining the receivers to receive data in otlp format in both grpc and http protocols on their default endpoint. Which is the local host and the port 4317 for grpc and 4318 for http.
Exporters -- We define two exporters. Jaeger and Aspecto. Jaeger is an open-source visualization tool that we will need to set up locally from an image. And Aspecto is a distributed tracing platform, to use Aspecto you will need to create a free account and generate an Aspecto api key (Go to Setting -> Integrations -> Tokens)
Processors -- We are using a batch processor to send all telemetry data to our exporters in batches. We also use a custom resource to insert a test attribute.
Extensions -- Defining the health check and zPages extensions. We will use them once we deploy the collector.
Service -- Enabling the traces pipelines for the receivers, processors, exporters, and extensions we defined above. We also set the collector telemetry log level to be 'debug' so we can see all sorts of fascinating things when we run.

Service

To pass data to the collector, we need a network service. Let's create and open all the relevant ports.

Enter the following Service resource:

---
apiVersion: v1
kind: Service
metadata:
 name: otel-collector
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 ports:
   - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
     port: 4317
     protocol: TCP
     targetPort: 4317
   - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
     port: 4318
     protocol: TCP
     targetPort: 4318
   - name: zpages
     port: 55679
     protocol: TCP
     targetPort: 55679
   - name: health-check
     port: 13133
     protocol: TCP
     targetPort: 13133
 selector:
   component: otel-collector

Make sure you open all the ports the collector is using. Otherwise, we will not be able to receive data or communicate with the collector.

Deployment

Because we run the collector as a gateway, we will use a deployment. It will allow us to scale once we encounter an increase in traffic.

Enter the following configuration:

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: otel-collector
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 selector:
   matchLabels:
     app: opentelemetry
     component: otel-collector
 minReadySeconds: 5
 replicas: 1
 template:
   metadata:
     namespace: opentelemetry
     labels:
       app: opentelemetry
       component: otel-collector
   spec:
     containers:
       - command:
           - '/otelcol'
           - '--config=/conf/otel-collector-config.yaml'
         image: otel/opentelemetry-collector:latest
         name: otel-collector
         resources:
           limits:
             cpu: '1'
             memory: 2Gi
           requests:
             cpu: 200m
             memory: 400Mi
         ports:
           - containerPort: 4317   #otlp grpc
           - containerPort: 4318   # otlp http
           - containerPort: 55679  # zpages
           - containerPort: 13133  # health check
         volumeMounts:
           - name: otel-collector-config-vol
             mountPath: /conf
     volumes:
       - configMap:
           name: otel-collector-conf
           items:
             - key: otel-collector-config
               path: otel-collector-config.yaml
         name: otel-collector-config-vol

In this configuration, we define one container to run the opentelemetry-collector image. It receives a command line argument to run with a configuration file created from our ConfigMap resource.

The configuration file is created in the volumes section of the deployment and in the volumeMounts section inside the container.

We are setting the deployment to have exactly 1 replica and setting the container CPU and memory limits according to the minimum that was checked for performance in their docs.

Deploying the OpenTelemerty Collector Gateway locally

Now that we created our cluster configuration, it's time to apply it to the cluster.

Run:

Kubectl apply -f ./otel-collector-config.yml

Let's check the collector is alive by making a health check call:

curl localhost:13133

We got back the following response:

curl: (7) Failed to connect to localhost port 13133 after 5 ms: Connection refused

Issue #1 -- Specifying the right type of service: This is not a problem with the collector. This is because we forgot to specify to our service that we need these ports exposed outside of the cluster. This means we need a service of the type LoadBalancer. Let's add ' type: LoadBalancer' inside the service spec block.

Calling the health check again still brings the same result.

Issue #2 -- Opening minikube tunnel: In minikube, in order to communicate with our load balancer, we need to run the command 'minikube tunnel' in a different terminal. This tunnel needs to stay open as long as our cluster runs.

Calling the health check once more gives us the following response:

{
"status":"Server available", "upSince":"2022-10-08T04:55:17.253942835Z", "uptime":"2h19m34.104112936s"
}

Now let's start generating some traces for testing.

I created a simple Typescript program that sends a span every 5 seconds to a collector's endpoint using an otlp-http exporter.

import { trace } from '@opentelemetry/api';
import * as opentelemetry from '@opentelemetry/sdk-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const otelColEndpoint =
 process.env.OTEL_COL_OTLP_ENDPOINT || 'http://localhost:4318';

console.log(`Exporting to traces to ${otelColEndpoint}`);

const exporter = new OTLPTraceExporter({
 url: otelColEndpoint,
});

const sdk = new opentelemetry.NodeSDK({
 spanProcessor: new SimpleSpanProcessor(exporter),
 serviceName: process.env.SERVICE_NAME,
});

sdk.start();
console.log(`${process.env.SERVICE_NAME}: Start Tracing...`);

let spanCounter = 0;
setInterval(() => {
 trace.getTracer('My Tracer').startSpan('Span number: #' + spanCounter++).end();
}, 5000);

Here you might start having issues. In order to solve these issues it's useful to look at the collector's logs.

First set the current context so you look at the OpenTelemery namespace by default:

kubectl config set-context --current --namespace=opentelemetry

Secondly, print all the pods in the namespace:

kubectl get pods

Copy the collector pod id and print its logs:

kubectl logs otel-collector-647fc9c4fd-g958h

Do it all in a one-liner:

kubectl logs `kubectl get pods | awk '/otel-collector/ {print $1}'`

You can also watch new logs by adding the -f flag to the command.

Now we are running the trace generator service. The minikube cluster is also up and running. But both Jaeger nor Aspecto received any traces. Where is the problem now? Looking at the collector's logs, we see no trace for traces (pun intended). Our collector never received any telemetry data.

Issue #3 -- Traces being sent to the wrong collector's endpoint: Our trace generator sent traces to http://localhost:4318 but the collector expects to receive traces at /v1/traces. Changing the Collector endpoint inside the trace generator service worked.

We start seeing logs indicating that the trace was received by one of the receivers.

2022-10-08T07:14:16.204Z    info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 1}

Here is a log printed by our logging exporter.

Looking at our visualization platforms, we still do not see any traces. Solving one issue at a time:

Issue #4 -- Run Jaeger inside the cluster with its own service and deployment: Because localhost is a local interface IP address. Hosts, nodes, and pods have their own localhost interfaces and they are not connected to each other. This is also true for our local machine where the Jaeger image runs.

Let's deploy Jaeger inside the cluster with its own service and deployment.

Create a directory for all our Kubernetes manifests. Move the previous configuration and create a new jaeger.yml file:

mkdir otel-k8s
mv otel-collector-config.yml otel-k8s/otel-collector-k8s.yml
cd otel-k8s
touch jaeger.yml

In the Jaeger manifest, create a deployment with the same properties as the container we ran before with docker compose.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: jaeger-all-in-one
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 replicas: 1
 selector:
   matchLabels:
     app: opentelemetry
     component: otel-collector
 template:
   metadata:
     labels:
       app: opentelemetry
       component: otel-collector
   spec:
     containers:
       - image: jaegertracing/all-in-one:latest
         name: jaeger
         ports:
           - containerPort: 16686
           - containerPort: 14250
---

Make sure the labels and namespace are the same as the rest of our OpenTelemetry setup.

Add a headless service with a selector matching our deployments so they can communicate with each other.

apiVersion: v1
kind: Service
metadata:
 name: jaeger-all-in-one
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 ports:
   - name: collector
     port: 14250
     protocol: TCP
     targetPort: 14250
 selector:
   app: opentelemetry
   component: otel-collector

Finally, add the Jaeger UI port 16686 to the load balancer and change the Jaeger exporter endpoint to the newly created deployment.

Add Jaeger to the load balancer list of ports:

 ports:
   - name: otlp-grpc
   ...
   - name: otlp-http
   ...
   - name: zpages
   ...
   - name: health-check
   ...
   - name: jaeger
     port: 16686
     protocol: TCP
     targetPort: 16686

Modify the Jaeger exporter:

exporters:
     otlp/aspecto: ...
     jaeger:
       endpoint: jaeger-all-in-one:14250
       tls:
         insecure: true
     logging:

Deploy by applying the entire directory:

kubectl apply -f ./

Now, let's look at Jaeger UI at localhost:16686, and voila:

We can see traces from our trace service.

Time to handle the issue with our second exporter

Issue #5 -- Set a Kubernetes Secret resource for your secret environment variables: In previous tutorials, we used environment variables to pass the remote tracing platform's credentials. But here it doesn't work. The correct way to handle sensitive information is with Secrets.

Let's create a secret using kubectl:

kubectl create secret generic aspecto --from-literal=api-key=$ASPECTO_API_KEY

Verify the secret was created and encrypted by running:

kubectl get secrets -o json

We get back a JSON with the api-key encrypted. Let's add the ASPECTO_API_KEY environment variable to the container:

...
containers:
 - command:
     - '/otelcol'
     - '--config=/conf/otel-collector-config.yaml'
   env:
     - name: ASPECTO_API_KEY
       valueFrom:
         secretKeyRef:
           name: aspecto
           key: api-key
           optional: false
   image: otel/opentelemetry-collector:latest
...

Now, after re-running our cluster we can see traces in both our visualization platforms:

We have a working local gateway for telemetry data.

Final Thoughts and Next Steps

Setting up your Kubernetes cluster to start collecting telemetry data could be a challenging task. In this guide, we set up a working OpenTelemetry Collector that is configured to receive data in OTLP format via Http or gRPC.

We then ran the collector using Kubernetes. Lastly, we attempted to solve some of the most common issues you might encounter as a beginner with OpenTelemetry and Kubernetes.

This guide is suitable for practice and learning. For real-world production systems, I recommend reading the following articles:

Thanks for reading, and please let me know if you have any questions about this article.

How to Instrument AWS Services with OpenTelemetry

Yoav Danieli — Mon, 19 Sep 2022 07:18:23 +0000

In this AWS OpenTelemetry guide, you will learn how to instrument your AWS services with OpenTelemetry. I will demonstrate how to instrument a simple microservice system running on AWS. We will use AWS SQS, DynamoDB, and Lambda.

First, we will create all the resources we require using Terraform and AWS. Secondly, we will instrument our code automatically using Opentelemetry node sdk. Lastly, we will deploy the system and visualize the traces it creates.

Let’s begin!

What is OpenTelemetry

OpenTelemetry is a community-driven open-source project and a unified specification for how we collect, generate, and export telemetry data to analyze our software’s behavior.

Sponsored by the CNCF (Cloud Native Computing Foundation), the OpenTelemetry project provides APIs and SDKs per programming language for generating telemetry, a centralized Collector that receives, processes, and exports data, and the OTLP protocol for shipping telemetry data.

In a cloud-native environment, we use OpenTelemetry (OTel for short) to gather data from our system operations and events. In other words, to instrument our distributed services. This data enables us to understand and investigate our software’s behavior and troubleshoot performance issues and errors.

To get a deeper understanding of this technology, watch our free OpenTelemetry Bootcamp video series that, in 6 episodes, cover OTel from end to end (it’s binge-worthy, really).

AWS + OpenTelemetry: Creating our application

We will create a simple Order taking application. It will be composed of the following components:

An order-api written with Node and deployed as an AWS Lambda.
AWS DynamoDB database
AWS SQS messaging queue
External service written with Node that listens on the SQS queue.

The application will receive orders and insert those orders into the database with status processing. It will publish an order received message to SQS then the external service in charge of processing these orders will receive this message and change the order status in the database to completed.

That is it! Let’s start by setting up AWS lambda.

Setup Disclaimer

There are many ways of interacting with the AWS platform. Each with its pros and cons. I like using Terraform and other PaaS code so that I can easily change, configure, replicate or migrate my work. So in this tutorial, I will create all the AWS services and resources I require using Terraform. Explanations regarding Terraform and AWS permissions and configurations are out of this article’s scope.

For the exact implementation details, you’re welcome to read the code example for this guide.

AWS Lambda Setup

The first component of our system is the order-api. Let’s create a new Typescript project and add the lambda code to the index.ts file:

import { SQS, DynamoDB } from "aws-sdk";
import { Context, APIGatewayEvent, APIGatewayProxyResult } from 'aws-lambda';
import { SendMessageRequest } from "aws-sdk/clients/sqs";
const sqs = new SQS();
const dynamo = new DynamoDB.DocumentClient();
const handler = async (event: APIGatewayEvent, context: Context): Promise<APIGatewayProxyResult> => {
   let body; ``
   let statusCode = 200;
   const routeKey = `${event.httpMethod} ${event.resource}`
   const headers = {
       "Content-Type": "application/json"
   };
   try {
       const tableName = process.env.DDB_TABLE_NAME as string
       const sqsUrl = process.env.SQS_URL as string
       if (!tableName) {
           return { statusCode: 500, body: 'Missing environment variable DDB_TABLE_NAME', headers }
       }
       if (!sqsUrl) {
           return { statusCode: 500, body: 'Missing environment variable SQS_URL', headers }
       }
       const id = event.pathParameters?.id
       switch (routeKey) {
           case "DELETE /items/{id}": // Delete Item...
           case "GET /items/{id}": // get item...
           case "GET /items": // get all items...
           case "PUT /items":
               let requestJSON = JSON.parse(event.body as string);
               await dynamo.put({
                   TableName: tableName,
                   Item: {
                       id: requestJSON.id,
                       price: requestJSON.price,
                       name: requestJSON.name,
                       status: "processing"
                   }
               }).promise();
               const params: SendMessageRequest = {
                   MessageBody: JSON.stringify({ id: requestJSON.id }),
                   QueueUrl: sqsUrl,
               }
               await sqs.sendMessage(params).promise();
               body = `Put item ${requestJSON.id}`;
               break;
           default:
               throw new Error(`Unsupported route: "${routeKey}"`);
       }
   } catch (err: any) {
       statusCode = 400;
       body = err.message;
   } finally {
       body = JSON.stringify(body);
   }
   return {
       statusCode,
       body,
       headers
   };
};
module.exports = { handler }

As you can see, the Put clause creates a new item with status “processing” and inserts it into DynamoDB. It then sends the newly created order’s id as a string to SQS.

Let’s compile the code using tsc and ensure a new dist folder was created.

Now add this to your terraform configuration:

resource "aws_lambda_function" "order_api" {
 function_name = "order_api"
 filename         = data.archive_file.lambdas_code.output_path
 source_code_hash = data.archive_file.lambdas_code.output_base64sha256
 role    = aws_iam_role.iam_for_lambda.arn
 handler = "index.handler"
 runtime = "nodejs12.x"
 timeout = 10
  environment {
   variables = {
     SQS_URL = # TBD
     DDB_TABLE_NAME = # TBD

   }
 }
}
data "archive_file" "lambdas_code" {
 type        = "zip"
 output_path = "${path.module}/dist.zip"
 source_dir  = "${PATH_TO_DIST_FOLDER}/dist"
}

Important Notes

Before deploying the Lambda, verify that the compiler target language is compatible with the Node runtime you specified. For compatibility check, visit here.
Check that the source dir specified in the archive_file resource contains index.js and node_modules.
In the example code for this blog, I also added an AWS api gateway so we can trigger the Lambda using a public URL. That is out of scope for this blog, but you can visit the source code and check it out.

AWS SQS Setup

Create the SQS queue and add the created queue URL as an environment variable to our lambda.

resource "aws_sqs_queue" "order_queue" {
 name = "orderQueue"
}

Change SQS_URL:

SQS_URL = aws_sqs_queue.order_queue.id

AWS DynamoDB

Create a DynamoDB table and add the table name as an environment variable to our lambda.

resource "aws_dynamodb_table" "order_table" {
 name           = "Orders"
 billing_mode   = "PROVISIONED"
 read_capacity  = 20
 write_capacity = 20
 hash_key       = "id"
 attribute {
   name = "id"
   type = "S"
 }
}

Change DDB_TABLE_NAME:

DDB_TABLE_NAME = aws_dynamodb_table.order_table.name

OpenTelemetry + AWS: External Service

The external service is a simple Node service. It can run anywhere as long as it has the permissions necessary to read messages from the queue and update items in the DynamoDB table. This example runs the service locally.

To receive the messages from the queue, we will use the sqs-consumer library. The service will receive messages describing newly created orders. After some processing, it will change the order status in the table to ‘completed’.

Create sqs-listner.ts:

import { DynamoDB } from "aws-sdk";
import { UpdateItemInput } from "aws-sdk/clients/dynamodb";
import { Consumer } from "sqs-consumer";
const dynamo = new DynamoDB.DocumentClient({ region: process.env.AWS_REGION as string });
async function dynamoUpdateCompleted(id: string) {
   const params: UpdateItemInput = {
       TableName: process.env.DDB_TABLE_NAME as string,
       Key: {
           // @ts-ignore
           id
       },
       ExpressionAttributeNames: {
           "#S": "status"
       },
       // This expression is what updates the item attribute
       ExpressionAttributeValues: {
           ":s": { S: "complete" }
       },
       UpdateExpression: "SET #S = :s",
       ReturnValues: "ALL_NEW",
   };
   await dynamo.update(params).promise()
}

const app = Consumer.create({
   queueUrl: process.env.SQS_URL as string,
   handleMessage: async (message) => {
       if (message.Body) {
           const parsedBody = JSON.parse(message.Body);
           // do some processing and then change status to complete
           await dynamoUpdateCompleted(parsedBody.id);
       }
   }
});
app.on('error', (err) => {
   console.error(err.message);
});
app.on('processing_error', (err) => {
   console.error(err.message);
});
app.start();

Important Notes

Make sure you have AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID configured on your machine.
The message is deleted from the queue once the handleMessage function is completed. That is important because it lets whoever uses this queue know that the message was received and acknowledged.

Instrumenting AWS services with OpenTelemetry and exporting telemetry data

To instrument Node services, we create a file containing the code and the configuration for our tracing.

We then require this file before any other code in our system runs so it can wrap the relevant function and create spans for any of the operations our system is performing.

On AWS, it is no different. Let’s create a tracing.ts file and add the OpenTelemetry configuration to it.

The OpenTelemetry is implemented and accessed by the application through the NodeSDK instance. So we will initiate it with our configuration.

First, let’s add the instrumentations. Our Lambda services use the aws-sdk library and the aws-lambda library. So any library that provides auto-instrumentations for these operations should be enough. Luckily when using Node, we can use the auto-instrumentation-node library.

It includes all Node auto instrumentations available to the public, including:

Important Configuration Notes

The lambda instrumentation is trying to use the X-Ray context headers by default (even when we’re not using X-Ray), causing us to have a non-sampled context and a NonRecordingSpan. To fix this, we use the disableAwsContextPropagation flag. More information about this can be found here and in the instrumentation docs.
The SQS queue we have created is configured by default to show message attributes in its payload. That is called ‘Raw Message Delivery.’ You can read more about it in AWS docs. When this is the case, we need to explicitly tell our program to receive its context from the payload. We do this by enabling sqsExtractContextPropagationFromPayload and setting it to true. Know that there will be performance implications because now the instrumentation will run JSON.parse to get the data.

Let’s process our spans using a batchSpanProcessor and a custom OTLPTtraceExporter. We can export our traces to the console, but to watch them, we will have to visit AWS CloudWatch (which will l be a bit messy).

Visualizing OpenTelemetry Data in Aspecto

AWS has good tools for tracing, but in this example, I will use another remote and distributed tracing platform – Aspecto.

To follow along, you can open a new free-forever Aspecto account or log in to your existing one.

Below, make sure to replace the {ASPECTO_API_KEY} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

Finally, let’s give our service a name that can be taken from either the SERVICE_NAME or the AWS_LAMBDA_FUNCTION_NAME environment variables.

Putting it all together, it looks something like that:

import { NodeSDK } from "@opentelemetry/sdk-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
const exporter = new OTLPTraceExporter({
   url: 'https://otelcol.aspecto.io/v1/traces',
   headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
   }
})
const sdk = new NodeSDK({
   spanProcessor: new BatchSpanProcessor(exporter),
   serviceName: process.env.SERVICE_NAME || process.env.AWS_LAMBDA_FUNCTION_NAME,
   instrumentations: [
       getNodeAutoInstrumentations({
           "@opentelemetry/instrumentation-aws-sdk": {
               sqsExtractContextPropagationFromPayload: true
           },
           "@opentelemetry/instrumentation-aws-lambda": {
               disableAwsContextPropagation: true
           }
       })
   ]
});
sdk.start()

To use OpenTelemetry Node SDK, we must load and run it before our application is loaded. So let’s compile it using tsc so it ends up inside the dist folder and be packaged together with the Lambda code.

After that, we need to add a NODE_OPTIONS environment variable to our Lambda, notifying the node runtime to run this process before the lambda code. Here’s the final Lambda configuration:

resource "aws_lambda_function" "order_api" {
 function_name = "order_api"
 filename         = data.archive_file.lambdas_code.output_path
 source_code_hash = data.archive_file.lambdas_code.output_base64sha256
 role    = aws_iam_role.iam_for_lambda.arn
 handler = "index.handler"
 runtime = "nodejs12.x"
 timeout = 10
  environment {
   variables = {
     ASPECTO_API_KEY = var.ASPECTO_API_KEY
     SQS_URL = aws_sqs_queue.order_queue.id
     DDB_TABLE_NAME = aws_dynamodb_table.order_table.name
     NODE_OPTIONS =  "--require tracing.js"
   }
 }
}

Important Notes

You don’t have to create an opentelemetry configuration file such as this for each of your lambdas. In fact, you shouldn’t. In AWS, you can use Lambda Layers. You can define the OpenTelemetry tracing piece of code as a Lambda layer and use it in any Lambda you want. Furthermore, OpenTelemetry went ahead and implemented this opentelemetry-lambda layer for us. All we need to do is use it with our config.

Running AWS Services and Visualizing OpenTelemetry Data in Aspecto

When running the SQS-listener service, remember to require the tracing file configuration.

Note that for this service, you can remove the disableAwsContextPropagation flag.

Let’s compile and run the service:

SERVICE_NAME=sqs-listener node --require path/to/tracing.js path/to/sqs-listener.js

Now the service is waiting for messages to be sent to the queue. Let’s deploy our Lambda so we can invoke it and send messages to the queue:

terraform -chdir=path/to/terraform/config apply --var ASPECTO_API_KEY=${ASPECTO_API_KEY}

You can invoke the Lambda using AWS CLI, or if you set up an api gateway, you can also make http requests.

curl --header "Content-Type: application/json" \
--request POST \
--data '{"id":"1","price":100, "name": "Awesome Item"}' \
http://your-api-gateway-lambda-invoke-url/items

You should get back a 200 response with the message “Put item 1”. Let’s check out our traces in Aspecto:

I received the following traces:

As you can see, our flow is divided into two traces. This happens because SQS sends messages in batches. I will explain more about it below. For now, let’s examine the traces.

Click the “order_api” lambda trace:

We can see 5 spans:

The Lambda trigger
AWS-SDK operation for dynamo DB + http request.
AWS-SDK operation for SQS + http request

Clicking the SQS span:

We can see that the message is linked and was received by another trace. A click on this link redirects us to the second trace:

This trace contains 7 spans. By clicking the Node service, I can see a more convenient view of the spans (on the left)

It seems that 4 spans were created from the AWS-SDK instrumentation, while the other 3 from the HTTP-instrumentation. We can disable the http instrumentations by passing the following flag in the configuration: suppressInternalInstrumentation.

By putting these two traces together we can get a full and clear view of our system.

OpenTelemetry AWS SQS: Batch Receiving

According to the opentelemetry specification for messaging systems, When a process receives messages in a batch it is impossible for this process to determine the parent span for the span that it is currently creating.

Since a span can only have one parent if it is propagated and the propagated trace and span IDs are unknown when the receiving span is started, the receiving span will not have a parent, and the processing spans are correlated with the producing spans via links.

Conclusion

In this guide, you hopefully learned how to create a simple application using AWS lambdas, SQS, DynamoDB, and instrument the application using OpenTelemetry. I also pointed out some tips and gotchas I encountered while working on this project.

Please feel free to contact me with any questions, comments, or suggestions.

Top 5 Jaeger Alternatives for Distributed Tracing

Yoav Danieli — Tue, 13 Sep 2022 11:16:00 +0000

In this article, we will explore 5 Jaeger alternatives you can test out immediately. We chose to cover these tools specifically because you can start using them today, and they provide a free version with enough room to play around with.

The tools we are covering:

Aspecto
Honeycomb
SigNoz
Lightstep
Logz.io

As you probably know, Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.

These Jaeger alternatives can replace the entire Jaeger suite, though we will focus mainly on trace visualization.

Another noteworthy aspect of these Jaeger alternatives is OpenTelemetry support. Most of these tools support OpenTelemetry whether you already have it running in your system or looking for an OpenTelemetry distro (an OpenTelmetry SDK with some vendor-specific customizations).

P.S. Shoutout to all of them for providing excellent OpenTelemetry education and resources.

Top Jaeger Alternatives in 2022

Before jumping in, make sure to check these checkboxes when looking for a Jaeger alternative:

A free version and you can test it out quickly
Supports OpenTelemetry
Provides the same or better trace visualization capabilities
Allows you to search and investigate trace data efficiently

Aspecto

Aspecto is an end-to-end distributed tracing platform that is fully compatible with OpenTelemetry. Aspecto allows developers to troubleshoot performance bottlenecks and errors within their microservices and correlate root causes across traces and logs.

Dev and observability teams use Aspecto as their OpenTelemetry infrastructure. From fully supporting the implementation to the consumption, management, scalability, and visualization of OpenTelemetry data.

Notable features

1) Advanced OpenTelemetry data search and filtering capabilities such as free text search, key value search, and filters by parameters.
2) Advanced service map visualization: Aspecto offers a service map visualization to improve your troubleshooting abilities and create a clear image of the relations between services.
3) Logs and traces integration: Integrate Aspecto and your logging solution to correlate logs with their matched traces, side-by-side, under a single view so you short-cut your debugging workflow.
4) Remote sampling configuration: Remotely configure new OpenTelemetry head and tail sampling rules from the Aspecto UI without changing your code.
5) End-to-end message broker visibility: Visualize and understand the entire journey messages go through (including Kafka, RabbitMQ, SQS, and more)

Pricing

Aspecto has a Free Forever Plan with unlimited features and users. Its paid plans start as low as $4 per 5M traces per month. Aspecto also offers a plan for R&D teams needing tailored OpenTelemetry implementation.

You can also give the free playground a try.

Product Overview

Honeycomb

Honeycomb is an observability tool designed for engineering teams providing visibility to troubleshoot distributed systems. Honeycomb provides an automatic instrumentation agent called Honeycomb Beelines and also supports OpenTelemetry.

Notable features

BubbleUp — Automatically detecting anomalies, highlighted and visualized on a heatmap
SLOs — Set up triggered alerts based on incident response time and other important service level objectives.
Metrics — View system metrics alongside application observability data

Pricing

Honeycomb offers a free plan, and its paid plan starts at $110. The pricing is based on event volume captured per month.

Product Overview

Lightstep

Lightstep offers two solutions — incident response and observability tool. Incident Response helps IT Operations and DevOps find their path to fast incident resolution. The Lightstep Observability platform combines metric and tracing data to gain observability to your system.

Notable features

Change Intelligence — This allows you to surface important changes in your system. Change Intelligence correlates changes found in traces with deviations in your data.
Notebook — Query and save both your infrastructure (metrics) and app performance (traces) data in one place
Create alerts for metric data — You can create alerts for your metric data by configuring thresholds against metric queries that when crossed, trigger the alert

Pricing

For its observability solution, Lightstep offers a Free version to help you get started. Its paid plan starts at $100 per month and is based on the number of monthly active services.

Product Overview

SigNoz

SigNoz is a full-stack open-source APM and observability tool. It captures both metrics and traces. Since SigNoz is an open-source project It can be self-hosted so you won’t need to send your data to any third party.

Notable features

Metrics support — visualize your Prometheus metrics.
Live view — test your app by debugging traces in real time
Service map visualization — visualize all services and the relations between them

Pricing

Signoz has a Free Plan for self-hosted apps and limited features. Its paid plans start from $200 which acquire you additional premium features such as hosting & managing your app by Signoz

Product Overview

Logz.io

Similar to Lightstep, Logz.io offers various features among them you can find Distributed tracing. Logz.io offers managed Jaeger Tracing as their distributed tracing solution and, on top of Jaeger, they offer additional features. This is a good option if you want to stick to Jaeger but looking for a managed solution.

Notable features

Managed Elasticsearch with Jaeger
Real-time alerting — identify high-priority production incidents and receive alerts via Slack, email, and other tools.
Service Performance Monitoring dashboard — View request rate, error rate, and latency under a single dashboard. The dashboard includes a breakdown of R.E.D data based on the operations running inside the chosen service.

Pricing

Logz.io offers a free plan for collecting small log volumes with advanced analytics and Up to 1 GB of log data. Its paid plan starts at 5$ Per million spans.

Product Overview

https://www.aspecto.io/wp-content/uploads/2022/07/Yalla-Zehooo.mp4

Jaeger and why you may want alternatives

If you’re reading this article, you probably have some experience with Jaeger and concluded that Jaeger is powerful, but it might not be the best solution for you.

We hope that the option we provided above, each unique in its way, will serve you well in finding your observability match.

Since Jaeger is so tied together with OpenTelemetry, and if you want to learn more, we recommend you check out our free OpenTelemtry Bootcamp which you can find here or on Youtube.

Getting Started with Aspecto

Aspecto has a free-forever tier and provides everything included in Jaeger and more. Like Jaeger on steroids. Feel free to try the playground and sign up for the free forever plan.

Distributed Tracing for Kafka with OpenTelemetry in Python

Tom Weiss — Mon, 05 Sep 2022 07:35:03 +0000

In this tutorial, I will cover Apache Kafka, OpenTelemetry, and how they play out together with practical examples in Python from 0 to 1.

You will learn how to enable OpenTelemetry tracing in Python to generate spans and visualize traces for various Kafka operations.

What to Expect

What is Apache Kafka?
What is OpenTelemetry?
OpenTelemetry Traces
What are OpenTelemetry Instrumentations?
How to Use Kafka with OpenTelemetry in Python
Visualizing OpenTelemetry Tracing Data

This guide has two parts:

A “hello world” kind of intro to OpenTelemetry and Kafka in Python in which you will end up seeing the outputs in a console.
In the second part, you will learn to visualize traces to troubleshoot your microservices.

To generate spans for the different Kafka operations (e.g., consume and produce), we will use OpenTelemetry libraries.

If you’re already familiar with Kafka and OpenTelemetry, feel free to jump right to the practical part of this guide.

What is Apache Kafka?

Kafka is an open-source distributed event stream platform, where we can send events to topics and consume them. You can learn more about Kafka here.

What is OpenTelemetry?

In a world where microservices are becoming a standard architecture and distributed messaging systems like Kafka play a core part in enabling the independent microservices to communicate, the world needed a way to visualize and troubleshoot the complexity of a distributed architecture.

Led by the CNCF (Cloud Native Computing Foundation), OpenTelemetry is an open-source project, a set of APIs and SDKs, that allows us to collect, export, and generate traces, logs, and metrics.

With OpenTelemetry, we gather data from the events happening within the different components in our systems (including all sorts of message brokers like Kafka), which ultimately help us understand our software’s performance and behavior and visualize our microservices architecture.

For this specific guide, we will focus on traces, you can learn more about OpenTelemetry in this friendly guide for developers.

OpenTelemetry Traces

A trace tracks the progression of a single request, as it is handled by the different microservices/ moving parts.

The atomic unit of work in a trace is called a span, for example, a call to a database, external service, send message to Kafka topic, etc.

Each trace contains a collection of 1 or more spans and more information like how much time the entire trace took to complete.

What are OpenTelemetry Instrumentations?

So now you know what traces and spans mean, but how do you create them?

That can be done using the SDKs provided to us by OpenTelemetry, which bundles useful instrumentations.

What’s an instrumentation? A piece of code that is responsible for creating spans and traces and sending them to some backend, for later visualizing them.

OpenTelemetry contains instrumentations for a lot of different libraries, including ones for Kafka.

How to Use Kafka with OpenTelemetry in Python

By now you should have all the theory you need, so let’s start writing some code.

This guide assumes you are running Kafka in your local (if you don’t, visit this quick guide).

The Practical Part

First, let’s install the relevant libraries:

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install opentelemetry-instrumentation-kafka-python
pip install kafka-python

Then, let’s add our tracing.py file:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    SimpleSpanProcessor,
    ConsoleSpanExporter,
)
from opentelemetry.instrumentation.kafka import KafkaInstrumentor
def instrument(*args, **kwargs):
    provider = TracerProvider()
    simple_processor = SimpleSpanProcessor(ConsoleSpanExporter())
    provider.add_span_processor(simple_processor)
    trace.set_tracer_provider(provider)
    KafkaInstrumentor().instrument()

This file is responsible for using OpenTelemetry SDKs to enable the creation of spans and traces.

Notice we have a console span exporter which means we will be sending the spans to the console output.

As you can see we use the KafkaInstrumentor to create spans for Kafka-related operations, like consume and produce.

This is all the OpenTelemetry code we need to create Kafka spans.

Let’s create the producer file:

from tracing import instrument
instrument()
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
for _ in range(1):
   producer.send('foobar', b'some_message_bytes')
   producer.flush()

And create the consumer file:

from tracing import instrument
instrument()
from kafka import KafkaConsumer
consumer = KafkaConsumer('foobar', bootstrap_servers='localhost:9092')
for msg in consumer:
   print(msg)

Notice that both files have these 2 lines:

from tracing import instrument
instrument()

This is the calling of the instrument function and enables the creation of the spans when we run the consumer and producer files using the following commands:

python producer.py
python consumer.py

Since we use console exporter – we can see the spans of the consumer and producer on the console output:

{
    "name": "foobar receive",
    "context": {
        "trace_id": "0x436c0e9807b2f61b4a72bc63e5205954",
        "span_id": "0xdda310172f7297cd",
        "trace_state": "[]"
    },
    "kind": "SpanKind.CONSUMER",
    "parent_id": "0x711a6a5d5bb977d8",
    "start_time": "2022-03-02T10:40:36.695519Z",
    "end_time": "2022-03-02T10:40:36.720864Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "messaging.system": "kafka",
        "messaging.destination": "foobar",
        "messaging.kafka.partition": 0,
        "messaging.url": "\"localhost:9092\""
    },
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.9.1",
        "service.name": "unknown_service"
    }
}

{
    "name": "foobar send",
    "context": {
        "trace_id": "0x436c0e9807b2f61b4a72bc63e5205954",
        "span_id": "0x711a6a5d5bb977d8",
        "trace_state": "[]"
    },
    "kind": "SpanKind.PRODUCER",
    "parent_id": null,
    "start_time": "2022-03-02T10:40:36.609135Z",
    "end_time": "2022-03-02T10:40:36.610410Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "messaging.system": "kafka",
        "messaging.destination": "foobar",
        "messaging.url": "\"localhost:9092\""
    },
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.9.1",
        "service.name": "unknown_service"
    }
}

But I have a feeling that you want to get a better visualization than just a console log. If you do – read on 🙂

Visualizing OpenTelemetry Tracing Data

For this article, I will be using Aspecto to visualize my traces. You can follow along by quickly creating a free account.

Aspecto has built-in support for visualizing async messaging platforms like Kafka so it fits perfectly with this guide’s purpose.

Once you’re signed in, you get an API key and all you need to do is change a few lines of code, environment variables and install some packages. I’ll be showing you how to do that below.

The end result should look like this:

First, let’s install:

pip install opentelemetry-exporter-otlp-proto-grpc

Then let’s modify our tracing.py file so that it uses the otlp exporter instead of the console exporter, to send data to Aspecto:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
   BatchSpanProcessor,
   SimpleSpanProcessor,
   ConsoleSpanExporter,
)
from opentelemetry.instrumentation.kafka import KafkaInstrumentor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
def instrument(*args, **kwargs):
   provider = TracerProvider()
   simple_processor = SimpleSpanProcessor(OTLPSpanExporter())
   provider.add_span_processor(simple_processor)
   trace.set_tracer_provider(provider)
   KafkaInstrumentor().instrument()

To run the consumer

OTEL_SERVICE_NAME=kafka-consumer OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 OTEL_EXPORTER_OTLP_HEADERS=Authorization=YOUR_ASPECTO_API_KEY_HERE python consumer.py

To run the producer

OTEL_SERVICE_NAME=kafka-producer OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 OTEL_EXPORTER_OTLP_HEADERS=Authorization=YOUR_ASPECTO_API_KEY_HERE python producer.py

And that’s it! When you enter the Aspecto platform and click on Trace Search, after a minute or two you should be able to see your traces.

Feel free to click on the chat button and contact us, we’d be glad to help you figure everything out and answer any questions you may have.

P.S. If you prefer to use an open-source to visualize your traces, check out Jaeger Tracing. Follow this guide to learn how to use OpenTelemetry to send traces to Jaeger. Keep in mind that it takes a bit more time to set up the required environment.

Hope this has been a useful guide for you, feel free to reach out if you have any questions!

How to Get Started with OpenTelemetry Go

Michael Haberman — Wed, 31 Aug 2022 07:28:19 +0000

In this guide, you will learn hands-on how to create and visualize traces with OpenTelemetry Go without prior knowledge.

We will start with creating a simple to-do app that uses Mongo and the Gin framework. Then, we will send tracing data to Jaeger Tracing and to Aspecto for visualization. You can find all the relevant files in this Github repository.

What to Expect

Intro to OpenTelemetry
Hello world: OpenTelemetry GO example
1. Create main.go file with Gin and Mongo
2. Install OpenTelemetry GO
3. Gin instrumentation: gin.Context
Visualization with Jaeger and Aspecto
1. OpenTelemetry Go and Jaeger Tracing
2. OpenTelemetry Go and Aspecto

Intro to OpenTelemetry

OpenTelemetry is a collection of APIs and SDKs that allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

It is a CNCF community-driven open-source project (Cloud Native Computing Foundation, the folks in charge of Kubernetes).

OpenTelemetry serves as a standard observability framework that captures all data under a single specification. It provides several components, including:

APIs and SDKs per programming language for generating telemetry
The OpenTelemetry Collector; receives, processes, and exports telemetry data to different destinations.
OTLP protocol for shipping telemetry data

To get a deeper understanding of this technology, including its structure and the primary motivation, visit this guide.

For this OpenTelemetry Golang guide, here are the terms you need to know:

Span: The most basic unit. A span represents an event in our system (e.g., an HTTP request or a database operation that spans over time). A span would usually be the parent of another span, its child, or both.
Trace: ‘Call-stacks’ for distributed services. Traces represent a tree of spans connected in a child/parent relationship. Traces specify the progression of requests across different services and components in our app (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.
Exporter: Once we create a span, the exporter handles sending the data to our backend (e.g., in memory, Jaeger Tracing, or console output)
Context propagation – The mechanism that allows us to correlate events across distributed services. Context is referred to as the metadata we collect and transfer*. Propagation* is how the context is packaged and transferred across services, often via HTTP headers. Context propagation is one of the areas where OpenTelemetry shines.
Instrumentation – instrumentation libraries gather the data and generate spans based on different libraries in our applications (Kafka, Mongo, Gin, etc.).

There are two ways to instrument our app – manually and automatically:

Auto instrumentation: Automatically create spans from the application libraries we use with ready-to-use OpenTelemetry libraries.
Manual instrumentation: Manually add code to your application to define the beginning and end of each span and the payload.

To understand more of the OpenTelemetry jargon, visit the official documentation.

Hello world: OpenTelemetry Go example

We will start by creating our to-do service and installing two libraries (Gin and Mongo) to understand how instrumentations work.

Step 1: Create main.go file for our to-do app

1) Install Gin and Mongo-driver

go get -u github.com/gin-gonic/gin
go get go.mongodb.org/mongo-driver/mongo

2) Set up gin and mongo to listen on “/todo”

3) Create some to-do’s to seed Mongo

package main
import (
  "context"
  "net/http"
  "github.com/gin-gonic/gin"
  "go.mongodb.org/mongo-driver/bson"
  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
)
var client *mongo.Client
func main() {
  connectMongo()
  setupWebServer()
}
func connectMongo() {
  opts := options.Client()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
func setupWebServer() {
  r := gin.Default()
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //Important: Make sure to pass c.Request.Context() as the context and not c itself - TBD
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}

Now that our small todo app is ready, let’s introduce OpenTelemetry.

Step 2: Install OpenTelemetry Go

We will be configuring OpenTelemetry to instrument our Go app.

1) To install the OTel SDK, run:

go get go.opentelemetry.io/otel /
go.opentelemetry.io/otel/sdk /

2) Instrument our Gin and Mongo libraries to generate traces.

3) Gin instrumentation: Install otelgin

go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

4) Mongo instrumentation: Install otelmongo

go get go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo

Gin instrumentation: gin.Context

We previously discussed the idea of context propagation – the way to transfer metadata between distributed services to correlate events in our system.

The Gin framework has its own type gin.Context which gets passed as a parameter to an HTTP handler. However, the context that should be passed down to the mongo operations is the standard Go library Context object, available in gin.Context.Request.Context.

//Make sure to pass c.Request.Context() as the context and not c itself
cur, findErr := collection.Find(c.Request.Context(), bson.D{})

So make sure that you pass the Context to the mongodb operation. Check out this issue for more info.

We now have our todo app ready and instrumented. It’s time to utilize OpenTelemetry to its full potential. Our ability to visualize traces is where the true troubleshooting power of this technology comes into play.

For visualization, we’ll be using the open-source Jaeger Tracing and Aspecto.

Visualization with Jaeger and Aspecto

The setup to export traces to Jaeger or Aspecto is relatively similar. Follow along with the Jaeger setup, then switch to Aspecto by changing a few lines of code.

OpenTelemetry Go and Jaeger Tracing: Export traces to Jaeger

Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.

Here’s what the setup looks like:

1) Install the Jaeger exporter

go get go.opentelemetry.io/otel/exporters/jaeger

2) Create a tracing folder and a jaeger.go file

3) Add the following code to the file

package tracing
import (
  "go.opentelemetry.io/otel/exporters/jaeger"
  "go.opentelemetry.io/otel/sdk/resource"
  sdktrace "go.opentelemetry.io/otel/sdk/trace"
  semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)
func JaegerTraceProvider() (*sdktrace.TracerProvider, error) {
  exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint("http://localhost:14268/api/traces")))
  if err != nil {
      return nil, err
  }
  tp := sdktrace.NewTracerProvider(
      sdktrace.WithBatcher(exp),
      sdktrace.WithResource(resource.NewWithAttributes(
          semconv.SchemaURL,
          semconv.ServiceNameKey.String("todo-service"),
          semconv.DeploymentEnvironmentKey.String("production"),
      )),
  )
  return tp, nil
}

4) Go back to the main.go file and modify our code to use the JaegerTraceProvider function we just created

func main() {
  tp, tpErr := tracing.JaegerTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  setupWebServer()
}

Next, we are going to hook up the instrumentations we installed.

5) Add the Mongo instrumentation. In our connectMongo function by adding this line

opts.Monitor = otelmongo.NewMonitor()

The function shold look like this

func connectMongo() {
  opts := options.Client()
  //Mongo OpenTelemetry instrumentation
  opts.Monitor = otelmongo.NewMonitor()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with some todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}

Now, add the Gin instrumentation.

6) Go to the startWebServer function and add this line right after we create the gin instance

r.Use(otelgin.Middleware("todo-service"))

The function should look like this

func startWebServer() {
  r := gin.Default()
  //Gin OpenTelemetry instrumentation
  r.Use(otelgin.Middleware("todo-service"))
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //make sure to pass c.Request.Context() as the context and not c itself
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}

Below is the complete main.go file. Now we’re finally ready to export to Jaeger.

package main
import (
  "context"
  "log"
  "net/http"
  "github.com/aspecto-io/opentelemerty-examples/tracing"
  "github.com/gin-gonic/gin"
  "go.mongodb.org/mongo-driver/bson"
  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
  "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
  "go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo"
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/propagation"
)
var client *mongo.Client
func main() {   //Export traces to Jaeger
  tp, tpErr := tracing.JaegerTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  startWebServer()
}
func connectMongo() {
  opts := options.Client()
  //Mongo OpenTelemetry instrumentation
  opts.Monitor = otelmongo.NewMonitor()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with some todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
func startWebServer() {
  r := gin.Default()
  //gin OpenTelemetry instrumentation
  r.Use(otelgin.Middleware("todo-service"))
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //Make sure to pass c.Request.Context() as the context and not c itself
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}

Export traces to Jaeger

Run the todo-service with go run main.go
Make an HTTP GET request to localhost:8080/todo to generate some traces in Go
Open Jaeger at http://localhost:16686/search to view those traces

You can now see the Jaeger UI. Select todo-service and click on Find traces. You should see your trace on the right:

By clicking the trace, you can drill down and see more details about it that allow you to further investigate on your own:

Visualization with OpenTelemetry Go and Aspecto

So now you know the basic concepts of OpenTelemetry. You used it on your own to instrument your Go libraries, create traces and export the data to Jaeger for visualization.

Let’s take our visualization capabilities to the next level with Aspecto.

Exporting and visualizing our data in Aspecto is easy (we will see that in a moment). You can try it yourself with the free-forever plan that has no limited features. Give this Live Playground a try to get a better idea.

This is how we do it:

Create a new free account at www.aspecto.io
Install the otlp exporter by running

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

Create a file called aspecto.go in your tracing folder and add the code below
Replace the {ASPECTO_AUTH} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

func AspectoTraceProvider() (*sdktrace.TracerProvider, error) {
  exp, err := otlptracegrpc.New(context.Background(),
      otlptracegrpc.WithEndpoint("collector.aspecto.io:4317"),
      otlptracegrpc.WithHeaders(map[string]string{
          "Authorization": "<ADD YOUR TOKEN HERE>",
      }))
  if err != nil {
      return nil, err
  }
  tp := sdktrace.NewTracerProvider(
      sdktrace.WithBatcher(exp),
      sdktrace.WithResource(resource.NewWithAttributes(
          semconv.SchemaURL,
          semconv.ServiceNameKey.String("todo-service"),
          semconv.DeploymentEnvironmentKey.String("production"),
      )),
  )
  return tp, nil
}

Back in the main.go file, replace the JaegerTraceProvider with AspectoTraceProvider.

The main function should now look like this

func main() {   //Export traces to Aspecto
  tp, tpErr := tracing.AspectoTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  startWebServer()}

Exporting traces to Aspecto

Run the todo-service with go run main.go
Make an HTTP GET request to localhost:8080/todo to generate some traces
Navigate to app.aspecto.io to view the traces

We also sent a few traces with errors to see what it looks like. Our traces should look something like this:

In the Trace Search view, using the left filters pane, simply filter out by the service name (you can also filter by error, HTTP method, and much more).

If we drill down into one of these traces, we can see in more detail how long each request took and clear visualization of the entire workflow.

In more complex workflows, traces would look something like the below:

That’s all folks! We hope this guide was informative and easy to follow. You can find all files ready to use in our Github repository.

If you have any questions or issues, feel free to reach out to us via our chat.

To learn more about it, check out our OpenTelemetry Bootcamp. It’s a 6 episode YouTube series that covers OpenTelemetry from zero to 100 including the implementation of opentelemetry in production, security, sampling, and more. Completely free and vendor-neutral.

Follow these guides below, if you want to install OpenTelemetry in more officially supported languages:

Guide to OpenTelemetry Distributed Tracing in Rust

Yoav Danieli — Tue, 30 Aug 2022 09:10:50 +0000

In this article, I will share my experience adding OpenTelemetry distributed tracing to a Rust application.

I will attempt to answer the following questions:

How to instrument Opentelemetry in Rust?
How to add manual and auto instrumentations to a Rust application?
How to use tracing to debug Rust applications?
How to visualize traces and spans?
How to preserve our span context in a multithreaded environment?
What are your must-know crates when it comes to tracing in Rust?

I will begin by saying that I love Rust as a programming language, and I’ve been experimenting with it for the past year. Although it’s innovative and has a great ecosystem, it is not a secret that Rust has quite a steep learning curve. So even though it’s an intro-level article, I will not dive deep into Rust features, syntax, or semantics unless it’s somehow connected to our subject (Opentelemetry and distributed tracing).

What to Expect

What is OpenTelemetry?
Opentelemetry and Distributed Tracing in Rust
- Tracing in Rust
- OpenTelemetry in Rust
- Distributed tracing in Rust
Practical Example
- Creating the service
- Create our tracing setup
- Adding Auto-Instrumentation for Actix-Web framework
- Visualizing traces with Aspecto
- Manually instrumenting Rust functions
- Adding Auto-Instrumentation for Diesel
- How to preserve our span context when executing closures on different threads?
Conclusion

What is OpenTelemetry?

OpenTelemetry is a collection of APIs and SDKs that allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

It is a CNCF community-driven open-source project (Cloud Native Computing Foundation, the folks in charge of Kubernetes).

In a cloud-native environment, we use OpenTelemetry (OTel for short) to gather data from our system operations and events. In other words, to instrument our distributed services. This data enables us to understand and investigate our software’s behavior and troubleshoot performance issues and errors.

OpenTelemetry serves as a standard observability framework that captures all data under a single specification. It provides several components, including:

APIs and SDKs per programming language for generating telemetry.
The OpenTelemetry Collector receives, processes, and exports telemetry data to different destinations.
OTLP protocol for shipping telemetry data

To get a deeper understanding of this technology, including its structure and the primary motivation, visit this guide.

For this OpenTelemetry Rust article, here are the terms you need to know:

Span: The most basic unit. A span represents an event in our system (e.g., an HTTP request or a database operation that spans over time). A span would usually be the parent of another span, its child, or both.
Trace: ‘Call-stacks’ for distributed services. Traces represent a tree of spans connected in a child/parent relationship. Traces specify the progression of requests across different services and components in our app (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.
Exporter: Once we create a span, the exporter handles sending the data to our backend (e.g., in memory, Jaeger Tracing, or console output)
Context propagation – The mechanism that allows us to correlate events across distributed services. Context is referred to as the metadata we collect and transfer. Propagation is how the context is packaged and transferred across services, often via HTTP headers. Context propagation is one of the areas where OpenTelemetry shines.
Instrumentation – instrumentation libraries gather the data and generate spans based on different libraries in our applications (Actix-Web, Postgres…).

To understand more of the OpenTelemetry jargon, visit the official documentation.

Opentelemetry and Distributed Tracing in Rust

Tracing in Rust

In Rust, we have a great crate simply called tracing. It is our core framework for tracing a Rust program. The crates documentation outlines its core concepts and APIs which are composed of these terms – spans, events, and subscribers.

I covered what Spans and Events are, but a Subscriber is a new term. To record spans and events, one has to implement the subscriber trait. That means simply implementing the methods:

enter: indicating entering a span.
exit: indicating ending a span.
event: indicating an event has occurred.

To start recording traces, we need to initialize a subscriber. The crate tracing_subscriber helps us to do exactly that.

We initialize the Registry struct:

let subscriber = tracing_subscriber::Registry::default()

This struct implements a subscriber and exposes us to another important feature of this crate – The Layer. Using layers, we can configure our Subscriber to apply specific behaviors when interacting with spans and events.

For example, if we want to filter some of our data, export it, format it or edit it, we can create or use an existing layer and compose it with the registry subscriber.

let subscriber = tracing_subscriber::Registry::default().with(SomeLayer)

We will see more examples of how to use these crates to instrument a Rust program in the practical section.

OpenTelemetry in Rust

The crate that provides support for OpenTelemetry is called – how convenient – OpenTelemetry [see here].

“OpenTelemetry provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application.”

Let’s go over some of its key APIs:

Tracer. Inside the crate, there is a module called traces. It introduces us to a trait called Tracer. This is what tracks and connects our spans to create traces. To use Tracer, we create a subscriber layer. This is how the subscriber knows where to send the spans and how to generate traces.
Global. This module provides an API so that no matter in what section of the code we currently are, we can access the subscriber, tracer, and propagation context.
SDK. This module provides an implementation for common use cases with OpenTelemetry. For example, it provides an exporter to send traces to stdout. It also implements context propagation methods that we can use instead of implementing ourselves.

The git repository called opentelemetry-rust contains implementations of several crates that expand the opentelemetry ecosystem. There you can find common instrumentations, exporters, and subscribers use them in a Rust program.

OpenTelemetry Distributed tracing in Rust

Using crates from the opentelemetry-rust repo, we can instrument our application across multiple services and send them to a distributed platform.

For example, we can use the opentelemetry-jaeger crate to send our traces to a remote Jaeger platform.

Another example is the OpenTelementy-otlp crate.

Traces in OTLP format can be exported to the OpenTelemetry Collector using the exporter in this crate. OpenTelemetry Collector accepts, processes, and exports traces in a vendor-agnostic manner.

Practical OpenTelemetry Rust Example

For this example, I wanted to build a simple system I could use to integrate and visualize traces. I found auto-instrumentations for Actix-Web Framework and Diesel, an ORM and Query builder utility for SQL-based databases.

I chose to use these tools and build a simple REST api service to create, read and delete users following the examples given by these frameworks.

Here is a link for the source code.

Creating the service

First, I followed the Actix web and diesel documentation and created a simple web server that exposes a user service and communicates with a local Postgres database:

mod models;
mod schema;
mod users;
#[macro_use]
extern crate diesel;
extern crate dotenv;
use dotenv::dotenv;
use actix_web::{web, App, HttpServer};
use diesel::r2d2::{self, ConnectionManager};
use diesel::pg::PgConnection
use telemetry::init_telemetry;
use users::get_users_service;
pub type Connection = PgConnection
pub type DbPool = r2d2::Pool<ConnectionManager<Connection>>;
#[actix_web::main]
async fn main() -> std::io::Result<()> {
  dotenv().ok();
  let database_url = std::env::var("DATABASE_URL").expect("DATABASE_URL must be set");
  let manager = ConnectionManager::<InstrumentedPgConnection>::new(database_url);
  let pool: DbPool = r2d2::Pool::builder()
      .build(manager)
      .expect("Failed to create pool.");
  HttpServer::new(move || {
      App::new()
          .app_data(web::Data::new(pool.clone()))
          .service(get_users_service())
  })
  .bind(("127.0.0.1", 3000))?
  .run()
  .await
}

For this blog, I will only show one endpoint and its flow. You can find the rest in the source code.

use crate::DbPool;
use crate::{models::NewUser};
use actix_web::{delete, get, post, web, Error, HttpResponse};
use uuid::Uuid;
#[post("")]
async fn create_user(
  db: web::Data<DbPool>,
  new_user: web::Json<NewUser>,
) -> Result<HttpResponse, Error> {
  let user = web::block(move || {
      let conn = db.get()?;
      db_operations::insert_new_user(&new_user.name, &conn)
  })
  .await?
  .map_err(actix_web::error::ErrorInternalServerError)?;
  Ok(HttpResponse::Ok().json(user))
}
pub fn get_users_service() -> actix_web::Scope {
  web::scope("/users")
      .service(get_users)
      .service(get_user_by_id)
      .service(create_user)
      .service(delete_user)
}
mod db_operations {
pub fn insert_new_user(nm: &str, conn: &PgConnection) -> Result<models::User, DbError> {
  use crate::schema::users::dsl::*;
  let new_user = models::User {
      id: Uuid::new_v4().to_string(),
      name: nm.to_owned(),
  };
  diesel::insert_into(users).values(&new_user).execute(conn)?;
  Ok(new_user)
}
}

Create our tracing setup

Now that I have a working service, I can add telemetry data. I will need to configure a tracing subscriber and its layers.

use opentelemetry::sdk::export::trace::stdout;
use opentelemetry::sdk::propagation::TraceContextPropagator;
use opentelemetry::sdk::{trace, Resource};
use opentelemetry::KeyValue;
use opentelemetry::{global};
use std::collections::HashMap;
use std::env;
use tracing_subscriber::Registry;
use tracing_subscriber::{prelude::*, EnvFilter};
pub fn init_telemetry() {
  // Define Tracer
  let stdout_tracer = stdout::new_pipeline().install_simple();
  let subscriber = Registry::default();
  // Layer to filter traces based on level - trace, debug, info, warn, error.
  let env_filter = EnvFilter::try_from_default_env().unwrap_or(EnvFilter::new("INFO"));
  // Layer to add our configured tracer.
  let tracing_leyer = tracing_opentelemetry::layer().with_tracer(stdout_tracer);
  // Setting a trace context propagation data.
  global::set_text_map_propagator(TraceContextPropagator::new());
  subscriber
      .with(env_filter)
      .with(tracing_leyer)
      .init();
}

This configuration adds an env_filter layer and a tracing layer. It prints to stdout all spans that their level is equal to or above the “info” log level.

Adding OpenTelemetry Instrumentation for Actix-Web framework

Now let’s add some spans by instrumenting the actions we take. Let’s start by instrumenting the operation taken by Actix web.

For that, I will use the actix_web_opentelemetry crate which provides a middleware called RequestTracing. As its name suggests, this middleware extracts trace information from the request and creates a span with the same context.

use actix_web_opentelemetry::RequestTracing;
// add the middleware to the server.
HttpServer::new(move || {
      App::new()
          .wrap(RequestTracing::new())
          .app_data(web::Data::new(pool.clone()))
          .service(get_users_service())

Now, when running the application and invoking the create user endpoint, we print spans to the terminal.

Trying to understand something from this print is quite hard. Besides, we are gathered here today to visualize traces, not to print them to the terminal.

So I changed the tracer to export traces to an OTLP collector. I configured the tracer to send traces to the Aspecto platform.

Visualizing traces with Aspecto

To follow along, you can open a new free-forever Aspecto account or log in to your existing one.

Below, make sure to replace the {ASPECTO_API_KEY} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

use opentelemetry_otlp::WithExportConfig;
let exporter = opentelemetry_otlp::new_exporter()
      .http()
      .with_endpoint("https://otelcol.aspecto.io/v1/traces")
      .with_headers(HashMap::from([(
          "Authorization".into(),
          env::var("ASPECTO_API_KEY").unwrap().to_string(),
      )]));
let otlp_tracer = opentelemetry_otlp::new_pipeline()
      .tracing()
      .with_exporter(exporter)
      .with_trace_config(
          trace::config().with_resource(Resource::new(vec![KeyValue::new(
              opentelemetry_semantic_conventions::resource::SERVICE_NAME,
              env::var("SERVICE_NAME").unwrap().to_string(),
          )])),
      )
      .install_batch(opentelemetry::runtime::Tokio)
      .expect("Error - Failed to create tracer.");
let tracing_leyer = tracing_opentelemetry::layer().with_tracer(otlp_tracer);

When running the application once more, I can view my traces in Aspecto:

Clicking the trace, I can view the spans and their data:

Manually instrumenting Rust functions

Let’s add more instrumentations to our service.

By using the macro #[instrument] on a function, we can create a span for this function.

It means that the span name will be the name of the function. Its attribute will include the module crate, the library of the function, and the function’s attributes.

By using instrument(skip(…), field(…)), we can skip some of the arguments of a function that are not important for us to record and manually insert new attributes.

Let’s give it a go:

#[instrument(skip(db))]
#[post("")]
async fn create_user(
  db: web::Data<DbPool>,
  new_user: web::Json<NewUser>,
) -> Result<HttpResponse, Error> {
  let user = web::block(move || {
      let conn = db.get()?;
      db_operations::insert_new_user(&new_user.name, &conn)
  })
  .await?
  .map_err(actix_web::error::ErrorInternalServerError)?;
  Ok(HttpResponse::Ok().json(user))
}

I now added instrumentation for the function create_user. I skipped the db parameters. Let’s view this new span in Aspecto:

We can clearly see the new span added to the trace. Clicking the new span we can see all the attributes I mentioned before:

Adding OpenTelemetry Instrumentation for Diesel

To get a full picture, I will add the instrumentation for the db_operation function:

#[instrument(skip(conn))]
pub fn insert_new_user(nm: &str, conn: &Connection) -> Result<models::User, DbError> {
  use crate::schema::users::dsl::*;
  let new_user = models::User {
      id: Uuid::new_v4().to_string(),
      name: nm.to_owned(),
  };
  diesel::insert_into(users).values(&new_user).execute(conn)?;
  Ok(new_user)
}

And for the diesel library which we are using. To add the diesel instrumentation we will need to use the diesel_tracing crate and replace our PgConnection with InstrumentedPgConnection:

use diesel_tracing::pg::InstrumentedPgConnection;
pub type Connection = InstrumentedPgConnection;
pub type DbPool = r2d2::Pool<ConnectionManager<Connection>>;

I ran the application once more and observed the traces in Aspecto:

We can see many unrelated traces. This is because the database connection struct communicates with the database periodically by calling the establish function to make sure the connection is still live.

How to preserve our span context when executing closures on different threads?

But there is a problem here. Can you see it? For some reason, the spans representing our request and the spans representing the database query are not grouped together. This is happening because of this code:

#[instrument(skip(db))]
#[post("")]
async fn create_user(
  db: web::Data<DbPool>,
  new_user: web::Json<NewUser>,
) -> Result<HttpResponse, Error> {
  let user = web::block(move || {
      let conn = db.get()?;
      db_operations::insert_new_user(&new_user.name, &conn)
  })
  .await?
  .map_err(actix_web::error::ErrorInternalServerError)?;
  Ok(HttpResponse::Ok().json(user))
}

We are invoking our db operation inside a closure we pass to the web::block function. We lose the span context of our execution flow. However, the web::block function is crucial.

It executes a blocking function on a thread pool dedicated to async blocking operations. So how can we keep the context of the trace?

We can do that by wrapping the web::block function with our own tracing function:

fn traced_web_block<F, R>(f: F) -> impl Future<Output = Result<R, BlockingError>>
where
  F: FnOnce() -> R + Send + 'static,
  R: Send + 'static,
{
  let current_span = tracing::Span::current();
  web::block(move || current_span.in_scope(f))
}

Here we get our current span and invoke the closure within its scope.

Let’s view our traces now:

We can now get a full picture of the flow of our application.

OpenTelemetry Distributed Tracing in Rust: Summary

We reviewed how to add distributed tracing with OpenTelemetry to the Rust application. We explored how to add both manual instrumentation and library-provided instrumentations.

We used Aspecto to visualize the flow of our system and learned what to do when the flow is broken.

To sum up, I enjoyed working on this example. I think the approach Rust is taking regarding tracing is interesting and worth considering when writing a Rust program. Furthermore, the ecosystem is growing so I am sure more libraries will provide instrumentations soon.

I hope you learned something new. Feel free to reach out with any questions you might have.

Distributed Tracing for Kafka with OpenTelemetry in Node

Tom Weiss — Tue, 30 Aug 2022 09:10:17 +0000

In this guide, you will learn how to run OpenTelemetry in Node to generate spans for different Kafka operations (e.g., consume and produce) and ultimately visualize your traces. We will also touch on the basics of Kafka, OpenTelemetry, distributed tracing, and how they all play out together.

If you’re already familiar with Kafka and OpenTelemetry, feel free to jump right to the practical part of this guide.

What to Expect

What is Apache Kafka
What is OpenTelemetry
What is Distributed Tracing
What is OpenTelemetry Instrumentation
How to use Kafka with OpenTelemetry in Node
Visualizing OpenTelemetry Traces
TroubleShooting OpenTelemetry Node Installation Issues
Additional OpenTelemetry Resources

What is Apache Kafka?

Kafka is an open-source distributed event stream platform, where we can send events to topics and consume them. You can learn more about Kafka here.

What is OpenTelemetry?

OpenTelemetry is an open-source project, a collection of APIs and SDKs. Led by the CNCF (Cloud Native Computing Foundation, same foundation responsible for Kubernetes), it allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

In a distributed software world, messaging systems (like Kafka) are key to enabling communication between independent microservices, making it easier to build complex systems.

But understanding the path each message has gone through, where it goes, why, and when is as critical as it is complicated.

Users need a way to visualize and troubleshoot the complexity of their distributed architecture.

OpenTelemetry is our tool to collect data from the different transactions within our services and components (including various message brokers like Kafka) and understand our software’s performance and behavior.

For this specific guide, we will focus on distributed traces, and you can take a deeper dive into OpenTelemetry in this short guide.

What is Distributed Tracing?

Distributed tracing defines spans and traces. A trace tracks the progression of requests across the services, message brokers, and other components in our system.

A trace is a tree of spans. A span is a characterization of an activity/process that occurred between our services. For example, a call to a database, external service, sending messages to a Kafka topic, etc.

Traces inform us of the length of each request, which components and services it interacted with, the latency introduced during each step, and other valuable information about the nature of that activity.

What is OpenTelemetry Instrumentation?

Instrumentation is a piece of code responsible for generating spans (which make up traces) and sending them to a dedicated backend of your choice. Later, depending on the tool you’re using, the end goal is to visualize them and leverage that visualization for troubleshooting our system.

We do that by using the SDKs provided by OpenTelemetry, which bundles useful instrumentations.

OpenTelemetry includes instrumentations for various libraries in different languages, including a Kafka instrumentation responsible for creating Kafka spans.

How to use Kafka with OpenTelemetry in Node

By now you should have all the theory you need, so let’s start writing some code. A note before we begin – This guide assumes you are running Kafka in your local machine – If you don’t, visit this quick guide (it shows how to run Kafka with docker in the beginning).

The Practical Part

First, let’s create a brand new Express app & install relevant packages:

npx express-generator kafkaotel
npm install
npm install kafkajs
npm install @opentelemetry/exporter-collector
npm install @opentelemetry/auto-instrumentations-node
npm install @opentelemetry/sdk-node @opentelemetry/api opentelemetry-instrumentation-kafkajs

Now create a tracing file that would create spans and traces for Kafka operations (like produce and consume). For this we use the opentelemetry kafkajs instrumentation:

/* tracing.js */
const { KafkaJsInstrumentation } = require('opentelemetry-instrumentation-kafkajs');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
// Require dependencies
const opentelemetry = require("@opentelemetry/sdk-node");
const tracerProvider = new NodeTracerProvider({
 // be sure to disable old plugin
 plugins: {
   kafkajs: { enabled: false, path: 'opentelemetry-plugin-kafkajs' }
 }
});
const sdk = new opentelemetry.NodeSDK({
 tracerProvider,
 traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
 instrumentations: [
   new KafkaJsInstrumentation({
     // see kafkajs instrumentation docs for available configuration
   })
 ]
});
sdk.start()

Let’s update package.json to require the tracing file so that opentelemetry would be able to instrument:

"scripts": {
 "start": "node --require './tracing.js' ./bin/www"
},

Now let’s update our main endpoint to produce and consume Kafka messages:

const express = require('express');
const router = express.Router();
const { Kafka } = require('kafkajs');
const kafka = new Kafka({
 clientId: 'my-app',
 brokers: ['localhost:9092'],
})
const sendKafkaMessage = async () => {
 const producer = kafka.producer();
 await producer.connect();
 await producer.send({
   topic: 'test-topic',
   messages: [
     { value: 'Hello KafkaJS user!' },
   ],
 })
 await producer.disconnect();
}
const consumeMessages = async () => {
 const consumer = kafka.consumer({ groupId: 'test-group' });
 await consumer.connect();
 await consumer.subscribe({ topic: 'test-topic', fromBeginning: true });
 await consumer.run({
   eachMessage: async ({ topic, partition, message }) => {
     console.log({
       value: message.value.toString(),
     })
   },
 })
}
/* GET home page. */
router.get('/', async function(req, res, next) {
 await sendKafkaMessage();
 await consumeMessages();
 res.render('index', { title: 'Express' });
});
module.exports = router;

Now you can run npm start and run go to localhost:3000 in your favorite browser.

Then you can see the producer span in the console like this one:

{
  traceId: '003cd8ece100e4bed2d05867d3a98dc0',
  parentId: undefined,
  name: 'test-topic',
  id: 'c83af592be261547',
  kind: 3,
  timestamp: 1646386704369183,
  duration: 28633,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'test-topic',
    'messaging.destination_kind': 'topic'
  },
  status: { code: 0 },
  events: []
}

And also the consumer span:

{
  traceId: '003cd8ece100e4bed2d05867d3a98dc0',
  parentId: 'c83af592be261547',
  name: 'test-topic',
  id: '875ea3c9e2b0dbdf',
  kind: 4,
  timestamp: 1646386726625784,
  duration: 562,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'test-topic',
    'messaging.destination_kind': 'topic',
    'messaging.operation': 'process'
  },
  status: { code: 0 },
  events: []
}

And that’s that! You now know how to use Kafka and OpenTelemetry together, and can create spans for Kafka operations that happen in your microservices.

So far we only printed outputs to our console. But OpenTelemetry has a lot to offer in terms of visualization, so let’s see how you can achieve that.

Visualizing OpenTelemetry Traces in Node

For the purposes of this guide, I chose to use Aspecto as my visualization tool. This is because Aspecto provides built-in support for visualizing messaging systems like Kafka (and, of course, any other part of our microservice architectures).

Before jumping in, you can try it yourself with the free-forever plan that has no limited features.

Give this Live Playground a try to get a better idea.

Here’s how it would look at the end of the process:

You can also click on a specific node and see tracing data:

You can follow along by creating a free account, then, obtain your Aspecto API key, and modify your tracing.js file:

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { KafkaJsInstrumentation } = require("opentelemetry-instrumentation-kafkajs");
const provider = new NodeTracerProvider({
 resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: 'kafka-service' // service name is required
 }),
});
provider.register();
provider.addSpanProcessor(
 new SimpleSpanProcessor(
   new CollectorTraceExporter({
     url: 'https://otelcol.aspecto.io/v1/trace',
     headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
     }
   })
 )
);
registerInstrumentations({
 instrumentations: [
   // add other instrumentations here for packages your app uses
   new KafkaJsInstrumentation({})
 ],
});

Then run the app with your Aspecto api key, like this:

ASPECTO_API_KEY=YOUR_API_KEY node --require './tracing.js' ./bin/www

When you go to localhost:3000 with a browser, it would produce and consume a Kafka message and send corresponding spans to aspecto. Allow a minute or 2 and you could see it under the trace search in aspecto, just like the above picture shows.

If you want to instrument libraries other than Kafka, update your tracing file like this(using the node auto instrumentations library that enables a variety of instrumentations by default):

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { KafkaJsInstrumentation } = require("opentelemetry-instrumentation-kafkajs");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const provider = new NodeTracerProvider({
 resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: 'kafka-service' // service name is required
 }),
});
provider.register();
provider.addSpanProcessor(
 new SimpleSpanProcessor(
   new CollectorTraceExporter({
     url: 'https://otelcol.aspecto.io/v1/trace',
     headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
     }
   })
 )
);
registerInstrumentations({
 instrumentations: [
   getNodeAutoInstrumentations(),
   new KafkaJsInstrumentation({})
 ],
});

But note: for Node, it’s recommended to use the Aspecto SDK which uses OpenTelemetry under the hood but also supports collecting actual payloads.

If you take the exact same service from scratch – without all the OpenTelemetry installations (make sure uninstall them to avoid any version conflicts), and run:

npm install @aspecto/opentelemetry

Then add the following code at the top of your app.js file:

require('@aspecto/opentelemetry')({
 aspectoAuth: process.env.ASPECTO_API_KEY
});

And now you can even see the payload of the Kafka message:

P.S. If you prefer to use an open-source to visualize your traces, you can try Jaeger Tracing. You can follow this guide on deploying Jaeger on AWS and make sure to scroll to the part on sending traces to Jaeger with the OpenTelemetry SDK (In NodeJS). Keep in mind that it may take a bit more time to set up the required environment.

If you’re not familiar with Jaeger and get a deeper understanding of it, we recommend visiting this guide.

I hope this has been a useful guide for you to learn how to use OpenTelemetry in Node to generate traces for Kafka operations. Feel free to reach out if you have any questions!

TroubleShooting OpenTelemetry Node Installation Issues

It is not uncommon to encounter several issues when trying to install OpenTelemetry. More specifically, trying to install OpenTelemetry in your Node application and not seeing any traces or some expected spans are missing.

To help you avoid these common pitfalls, we recommend going over this in-depth checklist of issues and solutions. We cover the most common mistakes and show you how to solve them.

Additional OpenTelemetry Resources

If you want to go deeper into learning about OpenTelemetry, visit The OpenTelemetry Bootcamp video library. It’s a free and vendor-neutral, in-depth course that will help you master OpenTelemetry. We cover everything from the very basics to security and deploying in production.