DEV Community: Yoav Danieli

OpenTelemetry Operator for Kubernetes: Practical Guide | Part 4

Yoav Danieli — Tue, 15 Nov 2022 15:39:00 +0000

In this article, you will learn how to use the OpenTelemetry Operator. We will explore what a Kubernetes Operator is, how to use it, and tackle common issues you might encounter when setting up the Opentelemetry Operator in your cluster.

This is the 4th part of our OpenTelemetry on Kubernetes series.

In the previous articles, I explained what the Opentelemetry Collector is, how to configure it, and how to set up a local Kubernetes cluster using Minikube and deploy the collector to that cluster. In addition, I described the deployment methods for a collector and how to use them (Collector as a Gateway and a Sidecar agent). Lastly, I showed for any deployment how to examine traces generated by our programs and visualize them using Aspecto.

Good to know before reading this article:

OpenTelemetry Collector on Kubernetes Basics | Part 1
OpenTelemetry Collector as an Agent on Kubernetes | Part 2
Opentelemetry Collector on Kubernetes Using Helm | Part 3

What is a Kubernetes Operator?

An operator extends Kubernetes to manage applications and their components using custom resources. Custom resources allow you to define new resources that group API objects with specific settings.

For example, we can create an application custom resource that accepts an image of a web server application and a port. This resource is composed of a deployment object and a service object that exposes this port.

To manage resources, Kubernetes uses the controller pattern. Notably the control loop. It means that Kubernetes runtime is always watching the current state of the resources in a cluster and comparing them to the desired state. Then it tries to reconcile to bridge the gap between the states if one exists.

Operators are custom controllers and custom resources put together. It is software running in your cluster, the same as the Kubernetes runtime, and watches the custom resources and their state. Operators allow complex clusters and systems to run automatically.

In short, Operators extend the cluster's behavior without modifying Kubernetes code.

What is OpenTelemetry Operator?

The OpenTelemetry Operator is an operator written by the OpenTelemetry community and aims to manage and simplify the way we incorporate observability into our system. It aims to solve the challenges of any developer that wants to add observability to their cluster encounters.

The first challenge is the necessity to configure and manage OpenTelemetry Collectors that answer the specific requirements of your system. The second challenge is instrumenting your core business logic and generating telemetry data to observe.

The OpenTelemetry Operator introduces two CRDs as a solution to these challenges:

OpentelemetryCollector -- This resource simplifies the configuration and maintenance of a Collector.
Instrumentation -- This resource can instrument your application for you. No change to your code is required.

In the remainder of this article, I will demonstrate how to use the Operator to build a working and instrumented web server.

OpenTelemetry Operator: The Practical Part

Setting up our application

First, we will create our application and deploy it to a local development cluster.

The application code is short. We are creating a simple express server that listens on port 3000 and returns a hello world message. The code is written in typescript.

Let's create tsconfig.ts, package.json, and a Dockerfile to build the image.

Here you will find the source code.

// application/index.ts

import express, { Request, Response } from 'express';

const PORT = process.env.PORT || 3000;

const app = express();
app.get('/', (req: Request, res: Response) => {
res.send('Hello World!');
});

app.listen(PORT, () => {
console.log(`Running on ${PORT}`);
});

Use the following Docker file to build the application:

FROM node:16
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
EXPOSE 3000
CMD [ "node", "index.js" ]

Now run:

docker build -t example-express-server ./application

Now it's time to set up our cluster. To set up a development environment please follow part 1 of this series guide.

Let's create a new cluster. We will clean up everything we did so far and start fresh:

minikube delete --purgeminikube start

Create a new file to define the resources for our application:

touch application.yml

And a Kubernetes namespace to isolate our application logic:

kubectl create namespace application

And a deployment resource and a service:

apiVersion: v1
kind: Service
metadata:
name: express-server
namespace: application
labels:
  app: application
  component: express-server
spec:
ports:
  - name: express # Default endpoint for OpenTelemetry gRPC receiver.
    port: 3000
    protocol: TCP
    targetPort: 3000
selector:
  component: express-server
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: express-server
namespace: application
labels:
  app: application
  component: express-server
spec:
selector:
  matchLabels:
    app: application
    component: express-server
replicas: 1
template:
  metadata:
    labels:
      app: application
      component: express-serve
  spec:
    containers:
      - name: express-server
        ports:
          - containerPort: 3000
        image: example-express-server
        imagePullPolicy: Never
        resources:
          limits:
            memory: '128 Mi'
            cpu: '500m'

We created a service of type Load balancer to expose the application to the outside of the cluster.

Note that we are using a locally built image, so we need to configure minikube not to look for this image elsewhere. Read about this issue here.

eval $(minikube docker-env)

Create a minikube tunnel to expose our cluster to our machine:

# run in a different shell and keep it open
> minikube tunnel

Now let's apply the changes and call the server to see that it is working:

> kubectl apply -f application.yml
> curl localhost:3000
Hello World!%

Collector Gateway with OpenTelemetry Operator

The server is up and running. Now it's time to design and extend our system by adding observability.

You can read more about deployment strategies in previous articles. For now, our setup will be as follows:

OpenTelemetry Collector gateway deployed as a Kubernetes deployment. This Collector will receive telemetry data on port 4317 and exports the data to a local Jaeger instance.
A Jaeger 'all-in-one' instance to observe telemetry data (traces)
The application we defined above will run with a collector agent as a sidecar. In addition, we will use the OpenTelemetry Operator Instrumentation resource to instrument the application without changing its code.

Create a namespace for our OpenTelemetry resources:

kubectl create namespace opentelemetry

Let's use the following manifest to deploy a Jaeger instance and network services:

# jaeger.yml

apiVersion: v1
kind: Service
metadata:
name: jaeger-all-in-one
namespace: opentelemetry
labels:
  app: opentelemetry
  component: otel-collector
spec:
ports:
  - name: collector
    port: 14250
    protocol: TCP
    targetPort: 14250
selector:
  component: otel-collector
---
apiVersion: v1
kind: Service
metadata:
name: jaeger-all-in-one-ui
namespace: opentelemetry
labels:
  app: opentelemetry
  component: otel-collector
spec:
ports:
  - name: jaeger
    port: 16686
    protocol: TCP
    targetPort: 16686
selector:
  component: otel-collector
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jaeger-all-in-one
namespace: opentelemetry
labels:
  app: opentelemetry
  component: otel-collector
spec:
replicas: 1
selector:
  matchLabels:
    app: opentelemetry
    component: otel-collector
template:
  metadata:
    labels:
      app: opentelemetry
      component: otel-collector
  spec:
    containers:
      - image: jaegertracing/all-in-one:1.35
        name: jaeger
        ports:
          - containerPort: 16686
          - containerPort: 14268
          - containerPort: 14250

Apply the resources in the cluster:

> kubectl apply -f jaeger.yml

Visit http://localhost:16686 and check that the Jaeger UI is running and available.

Now we will use the OpenTelemetryCollector CRD to deploy a collector as a gateway.

First, we need to install the OpenTelemetry Operator. Follow the official documentation for more information.

> kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

You will probably receive an error about needing a cert-manager installed. So Let's install that also:

> kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml
> kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

This installs the OpenTelemetry Operator control plane in your cluster under the namespace 'opentelemetry-operator-system'. You can check out all the extra software your cluster is now running with:

> kubectl get all -n opentelemetry-operator-system

Now that we have the Operator installed, we can use the custom resources it offers. Let's create a gateway.yml file with the following config:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-collector
namespace: opentelemetry
labels:
  app: opentelemetry
  component: otel-collector
spec:
mode: deployment
config: |
  receivers:
    otlp:
      protocols:
        grpc:
        http:
  exporters:
    jaeger:
      endpoint: jaeger-all-in-one:14250
      tls:
        insecure: true
    logging:
  processors:
    batch:
    resource:
      attributes:
        - key: test.key
          value: "test-value"
          action: insert
  extensions:
    health_check:
    zpages:
      endpoint: :55679
  service:
    telemetry:
      logs:
        level: "debug"
    extensions: [zpages, health_check]
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch, resource]
        exporters: [logging, jaeger]

Apply the changes:

> kubectl apply -f gateway.yml

This resource deploys the OpenTelemetry Collector with the defined config as a deployment. I will not get into the configuration file since we explained it in depth in previous articles. What is important to note here is the mode specification. The available values are:

Deployment
Sidecar
DeamonSet

In addition to the Collector deployment, this resource also handles running the necessary service to communicate with the collector. How simple is that? If you read my previous articles, you can see much larger manifest files. Here this is not the case. The Operator simplified all of this process for us.

Collector Agent with OpenTelemetry Operator

The next part will add the opentelemetry collector agent to run as a sidecar for our application. Create a new file for the sidecar:

# sidecar.yml

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: sidecar
namespace: application
spec:
mode: sidecar
config: |
  receivers:
    otlp:
      protocols:
        grpc:
        http:
  processors:
    batch:
  exporters:
    logging:
    otlp:
      endpoint: "http://otel-collector-collector.opentelemetry.svc.cluster.local:4317"
      tls:
        insecure: true
  service:
    telemetry:
      logs:
        level: "debug"
    pipelines:
      traces:
        receivers: [otlp]
        processors: []
        exporters: [logging, otlp]

As you can see, the mode is set to sidecar while the config is set to the agent config we defined in the previous article.

An important note that can cause issues is that we need to pass the telemetry data forward to the gateway collector. Now, Because they are running in different namespaces, we need to specify the full cluster DNS for the desired service.

Get the name of the Gateway service by typing:

> kubectl get svc -n opentelemetry

Choose the service that listens on port 4317.

The full DNS is composed as follows:

[service_name].[namespace].svc.cluster.[local | remote_url]:[port]

Let's wait with applying this (I will explain why after the next section). For now, continue and create the instrumentation for our application.

Auto Instrumentation with OpenTelemetry Operator

Create a new manifest file with the following resource:

# instrumentation.yml

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: nodejs-instrumentation
namespace: application
spec:
propagators:
  - tracecontext
  - baggage
  - b3
sampler:
  type: always_on
nodejs:

We are using the Instrumentation CRD. It lets us define all kinds of settings regarding how we want to instrument our application. The most important ones are the propagator, sampler, and programming language (or runtime) specification.

We will not get into each of the configurations available by the instrumentation since this requires a separate article. If you are interested in reading more about it, visit the official docs.

The Propagators let our injected instrumentation SDK know how to propagate telemetry data. Let's use the propagators defined in the OpenTelemetry repo example.

The sampler specifies the sampling strategy to be executed by this instrumentation. Go here to read more about sampling.

Lastly, we define the Nodejs part of the instrumentation to use the default values. The other programming languages supported are Python, .Net, and Java.

OpenTelemetry Operator: Putting it all together

The last thing we need to do is update our application resource and add annotations. It is how the Operator knows to inject the sidecar and the instrumentation into the deployment resource.

Update application.yml and add the following annotation:

spec:
template:
  metadata:
    annotations:
      instrumentation.opentelemetry.io/inject-nodejs: 'nodejs-instrumentation'
      sidecar.opentelemetry.io/inject: 'sidecar'

Notice that the instrumentation only works if it runs before the application. It needs to be configured and applied in the cluster before the application.

Shut down the application:

> kubectl delete -f application.yml

And apply all of the above configurations:

> kubectl apply -f instrumentation.yml
> kubectl apply -f sidecar.yml
> kubectl apply -f application.yml

Let's see what's going on by examining the application:

> kubectl logs deployments.apps/express-server -n application
Defaulted container "express-server" out of: express-server, otc-container, opentelemetry-auto-instrumentation (init)

We can see two more containers run inside the deployment. The otc-container which is the sidecar, and the instrumentation runs as an init container. It means it runs just before the other containers and is used for running commands and prefixing them to other image commands.

Let's invoke the application endpoint one more time:

> curl localhost:3000
Hello World!%

We can see that we got back the hello world message. Check out http://localhost:16686 and verify that the telemetry data reached its final destination.

That is it! Using the OpenTelemetry Operator was simple, intuitive, and with much fewer issues than configuring each Kubernetes object by ourselves.

The most important advantage is the ability to instrument applications without changing their code.

It was a fascinating subject to write about, and I hope you enjoyed it as much as I did.

If you have any questions or comments, feel free to CONTACT me on LinkedIn. As my wife says, I'm wrong quite a lot, so if you see a mistake, please let me know.

See you next time.

OpenTelemetry Collector on Kubernetes with Helm Chart – Part 3

Yoav Danieli — Sun, 13 Nov 2022 14:16:00 +0000

Written by Yoav Danieli @ Team Aspecto

In this article, you will learn how to use the OpenTelemetry Helm chart to deploy a collector as a gateway in a Kubernetes cluster.

In previous tutorials, I demonstrated how to deploy the OpenTelemetry Collector as both Gateway and Agent on a Kubernetes cluster. It took a lot of work and hours of research. Luckily there is an easier way. Enter Helm Charts.

Related Guides

What is Helm?

Before we start using Helm, let's give it a quick introduction.

Helm is an open-source package manager for Kubernetes. It allows software built for Kubernetes to be provided, used, and shared. It's also a graduate project from the Cloud Native Computing Foundation (CNCF), similar to Kubernetes, Jaeger, and OpenTelemetry (although OTel is still incubating).

Helm packages software uses a format called Charts. A chart is a collection of files that form a set of Kubernetes resources. We can use a single chart to deploy simple things like a single-pod hello-world program or enormously complex stuff like a full-stack web app with HTTP servers, databases, caches, and more.

For more information about Helm, visit Helm Docs.

Now let's dive right in and figure out how to use the Helm chart provided by OpenTelemetry.

Using OpenTelemetry Helm Chart

Installing Opentelemetry-Collector Helm Chart

1. Install Helm using homebrew.

brew install helm

Add the OpenTelemetry helm repository. A helm repository is an HTTP server from which we can find and install helm charts. The following command will establish a connection with the repository server and allow us to find and install its resources.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

3. Search the repo and find out which charts it offers:

helm search repo open-telemetry

This will produce the following output:

NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
open-telemetry/opentelemetry-collector  0.34.0          0.61.0          OpenTelemetry Collector Helm chart for Kubernetes
open-telemetry/opentelemetry-demo       0.3.1           0.4.0-alpha     opentelemetry demo helm chart
open-telemetry/opentelemetry-operator   0.13.0          0.60.0          OpenTelemetry Operator Helm chart for Kubernetes

The repository offers three charts:

OpenTelemetry-collector -- The chart that we will use to create the collector.
OpenTelemetry-demo -- This is a demo chart that creates an incredibly complex microservice system. It's written in many programming languages, and you can find good examples for almost any use case you need.
OpenTelemetry-operator -- This chart helps us set up an OpenTelemetry operator in our project. The operator provides the implementation of the OpenTelemetry collector and injects auto instrumentation to the services running inside our cluster. I will demonstrate the usage of the OpenTelemetry collector in future articles.

When using a helm chart, we are provided with a pre-defined set of parameters. These parameters, as well as their default values, can be found in a file called values.yml. To show the file, we can use the command:

helm show values open-telemetry/opentelemetry-collector

But it is easier to jump to the chart source code and look at the files in the browser. I will not elaborate on each of the values, instead, I will demonstrate how to create the same Gateway Collector deployment we created above. Hopefully, it would be much simpler.

First, clean up your cluster and remove all resources within it. Getting started with a clean slate

Let's install the chart as is. With its default values.

helm install opentelemetry-collector open-telemetry/opentelemetry-collector

We get an error.

[ERROR] 'mode' must be set.

As stated in the source code from the link above. The default value for mode is set to an empty string while valid values are 'deployment'. 'daemonset' or 'statefulset'.

Let's add a mode value.

Create a collector-values.yml file and set the mode to deployment, the same as our Gateway.

helm install opentelemetry-collector open-telemetry/opentelemetry-collector -f ./collector-values.yml

So now we have an up-and-running collector.

💡Pro Tip -- We can replace the command 'install' with 'template' to see the exact Kubernetes resources rendered and deployed to the cluster. Looking at the resources generated by the helm chart, we see similarities to the resources we used ourselves. And all we did was install the chart with a specific mode!

OpenTelemetry-Collector Helm Chart Configuration

To configure it the same way as the Gateway collector we deployed in the previous chapters, there are more values we need to set. Let's start with the configuration taken from the chart's examples under 'deployment-otlp-traces'.

mode: deployment

ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

config:
  receivers:
    jaeger: null
    prometheus: null
    zipkin: null
  service:
    pipelines:
      traces:
        receivers:
          - otlp
      metrics: null
      logs: null

Here we disable the ports we are not using and enable only the traces pipeline, with only the otlp receiver. This configuration will install a collector that receives telemetry data via grpc and http protocols in otlp format.

There are no exporters defined, so the chart will render the default exporter which is a simple logging. Let's add the OTLP exporter for Aspecto, our distributed tracing platform. And let's also add Jaeger exporter:

config:
 receivers:
   jaeger: null
   prometheus: null
   zipkin: null
 exporters:
   otlp/aspecto:
     endpoint: otelcol.aspecto.io:4317
     headers:
       Authorization: ${ASPECTO_API_KEY}
   jaeger:
     endpoint: jaeger-all-in-one:14250
     tls:
       insecure: true
   logging:
     loglevel: debug
 service:
   telemetry:
     logs:
       level: debug
   pipelines:
     traces:
       receivers:
         - otlp
       exporters:
         - otlp/aspecto
         - jaeger
         - logging
     metrics: null
     logs: null

Populate the ASPECTO_API_KEY with a secret, same as above. The OpenTelemetry collector chart supports it by using the extraEnv property:

extraEnvs:
 - name: ASPECTO_API_KEY
   valueFrom:
     secretKeyRef:
       name: aspecto
       key: api-key
       optional: false

We also need to expose our collector to the internet using the load balancer service:

service:
 type: LoadBalancer

The last thing we should do is tell the helm chart which OpenTelemetry collector image it should use and how. By default, it uses the latest image from the otel/opentelemetry-collector-contrib docker repo.

We want to change it to the stable collector-core image of our choosing tag.

image:
 repository: otel/opentelemetry-collector
 tag: "latest"

command:
 name: otelcol

Upgrade the deployment:

helm upgrade opentelemetry-collector open-telemetry/opentelemetry-collector -f ./collector-values.yml

And also deploy Jaeger UI, so we can visualize our traces:

kubectl apply -f jaeger.yml

After checking the logs and status of our Collector and Jaeger deployments and verifying everything is ready, we can start our test service to generate traces.

Once the service runs, you should see traces reaching the destinations configured in the exporter's section.

Final Thoughts and Aspecto Helm Chart

Using the OpenTelemetry-Collector Helm chart was simple and quick compared to manually defining each resource. I highly recommend using it in your clusters.

If you reach a point where you need to extend those resources or add properties that are not enabled via the helm chart, you can output the template resource with the helm template command and use it. Easy peasy.

In addition, some distributed tracing vendors offer their own helm chart to add a vendor-specific functionality to the Collector configuration.

Check Aspecto Helm charts here.

ASPECTO HELM CHART

Thanks for reading, and if you have any questions, I'm happy to connect and help.

P.S. You can find a lot of the concepts covered here in our latest live workshop on deploying the OpenTelemetry Collector on Kubernetes.

OpenTelemetry Collector as an Agent on Kubernetes – Part 2

Yoav Danieli — Fri, 11 Nov 2022 13:11:00 +0000

Written by Yoav Danieli @ Team Aspecto

In this article, you will learn how to deploy and configure the OpenTelemetry Collector as an agent on Kubernetes.

In part 1, we described how to set up a local Kubernetes environment with Minikube. We configured an OpenTelemetry collector and deployed it to a local Kubernetes cluster. We outlined and explained each of the Kubernetes resources we need to deploy the Opentelemetry Collector as a Gateway.

You can read this article as a stand-alone guide or as part 2 of our OpenTelemetry Collector on Kubernetes series.

Related Guides

Why Deploy the OpenTelemetry Collector as an Agent

We use the collector gateway as a central point for all our telemetry data from our distributed architecture. It means the collector usually receives a lot of traffic and therefore is responsible for handling this traffic.

Deploying specific collector instances for certain services can help us offload some of those responsibilities. For example, the agent can handle actions like batching, compression, retries, and more.

In addition, it improves the overall performance of the system. Since the agent is running in the same host as our services, there is no DNS resolving (it's just localhost). Thus it receives the data much faster.

Lastly, having a specific collector instance per service means we can add specific features and configurations to our telemetry data. We can add specific attributes and use any collector process configuration specific to this service.

Here is an illustration of running a collector gateway + agent as sidecar architecture:

There are a couple of strategies to deploy the collector as an agent.

We can deploy it as a DeamonSet, Sidecar, or StatefullSet. Each strategy has its use cases. To read more about deployment strategies, I recommend reading this article.

In this tutorial, I will deploy the OpenTelemetry collector agent as a sidecar.

Kubernetes Resource Configuration for Collector as an Agent

Let's create a new manifest file named business-application.yml. We will start by creating the agent's configuration.

One exporter and one receiver on OTLP protocols:

apiVersion: v1
kind: ConfigMap
metadata:
 name: otel-agent-conf
data:
 otel-agent-config: |
   receivers:
     otlp:
       protocols:
         grpc:
         http:
   processors:
     batch:
   exporters:
     otlp:
       endpoint: otel-collector:4317
       tls:
         insecure: true
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: []
         exporters: [otlp]
---

Secondly, we will create a deployment with two containers. The application and the sidecar.

For the application service, I will use a test service that sends an example span every 5 seconds to the agent's endpoint. For all code examples, please check out the source code.

I built it using:

docker build -t test-service ./test-service

Here is the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: test-service
spec:
 selector:
   matchLabels:
     app: test-service
 template:
   metadata:
     labels:
       app: test-service
   spec:
     containers:
       - name: test-service
         image: test-service
         env:
           - name: SERVICE_NAME
             value: test-service
         resources:
       - name: otel-agent
         image: otel/opentelemetry-collector:latest
         command:
           - '/otelcol'
           - '--config=/conf/otel-agent-config.yaml'
         resources:
         volumeMounts:
           - name: otel-agent-config-vol
             mountPath: /conf
     volumes:
       - configMap:
           name: otel-agent-conf
           items:
             - key: otel-agent-config
               path: otel-agent-config.yaml
         name: otel-agent-config-vol

Deploying the OpenTelemetry Collector Gateway + Agent locally

Let's apply the configuration we have so far to check the new service is working and deploy the agent and gateway to verify the new service is working.

You can find all gateway configurations in this gateway.yml file.

Deploy it using

kubectl apply -f gateway.yml

Next deploy the manifests we just wrote using:

kubectl apply -f business-application.yml

The application crashed.

Issue #1 -- Use locally built docker inside a Kubernetes cluster. Printing the log from the test-service pod we get the following message:

Defaulted container "test-service" out of: test-service, otel-agent
Error from server (BadRequest): container "test-service" in pod "test-service-5646dd8d77-fp27c" is waiting to start: trying and failing to pull image

The cluster can't build the image because it can't pull it. It searches the cluster's docker daemon for the image and can't find it. The solution, as described here, is to push the image to the local docker environment of minikube.

Following the documentation, we enter the following, change the imagePullPolicy to Never, and run it again:

eval $(minikube docker-env)

Let's verify that everything runs smoothly. The logs state that the exporter could not be found. It makes sense because we didn't create a service to enable this communication.

Add another service resource in the gateway manifest. It needs to be of type ClusterIP and have the gateways labels in its selector block:

apiVersion: v1
kind: Service
metadata:
 name: otel-collector-in
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 type: ClusterIP
 selector:
   component: otel-collector
 ports:
   - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
     port: 4317
     protocol: TCP
     targetPort: 4317
   - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
     port: 4318
     protocol: TCP
     targetPort: 4318

Let's run the application once more. Still no connection with the exporter.

Issue #2 -- Cross Namespace Communication in Kubernetes. When running pods in different namespaces the pod's DNS is suffixed with the name of the namespace. In our config map, we need to specify the full DNS in the exporter's endpoint:

endpoint: otel-collector.opentelemetry.svc.cluster.local:4317

Read the Kubernetes documentation regarding DNS to learn more.

After fixing the last issue and running the cluster once more, we can see the traces reaching the exports we configured in the Gateway. Success!

Final Thoughts

In this article, we configured a simple Collector agent that runs as a sidecar for an example span-generating application.

We outlined how to deploy and configure the OpenTelemetry Collector as an agent on Kubernetes.

Please feel free to check out the source code for this article. Thanks for reading and please let me know if you have any questions or comments.

The next part of this series will explain how to save us a lot of time and effort by using Helm.

OpenTelemetry Collector on Kubernetes – Part 1

Yoav Danieli — Thu, 10 Nov 2022 15:02:00 +0000

Written by Yoav Danieli @ Team Aspecto

In this guide, you will learn how to set up a working OpenTelemetry Collector on Kubernetes that is configured to receive data in OTLP format via HTTP or gRPC and send traces for visualization in both Jaeger and Aspecto.

The purpose of this tutorial is to simplify the process by going through the main steps and solving common issues.

Before we start I recommend reading the following blog posts that explain:

To demonstrate how the OpenTelemetry collector can be deployed on a Kubernetes cluster, I will use a similar configuration to that in the second article.

Also, a basic knowledge of Kubernetes could help. I will explain the Kubernetes commands and resources I'm using, but the inner workings of Kubernetes are out of this article's scope.

Next in this series

OpenTelemetry Collector on Kubernetes: Creating a Development Environment

We will use Minikube Kubernetes runtime in this article. There are many other choices for local development such as Kind, k3s, MicroK8s, and more. Each one has its unique setup, but the configuration files for the cluster are (almost) the same so feel free to play with your preferred runtime.

To set up a Kubernetes cluster we will first need to install:

Kubectl:

brew install kubectl

Then, install minikube:

brew install minikube

To start a cluster run:

minikube start

Minikube automatically configures kubectl to work with the newly created cluster.

A cool feature of minikube is the dashboard feature. By running minikube dashboard in a different terminal, you can open a convenient UI that you can use to examine your local cluster. It is helpful for beginners, but I would recommend relying on this sparingly.

Once you migrate to a cloud environment, you'll still need to rely on the command line.

Run OpenTelemerty Collector as a Gateway in Kubernetes cluster

Kubernetes Resource Configuration

Namespace

Now that we have a cluster running, we can start creating Kubernetes resources. Start by creating a namespace for all of our OpenTelemetry-related work. Although it is not a must, it's best practice to separate logical groups into namespaces.

Create a file named otel-collector-config.yml and add the namespace object:

# otel-collector-config.yml
apiVersion: v1
kind: Namespace
metadata:
 name: opentelemetry

To deploy the configuration to the cluster and create or update resources we use the command:

kubectl apply -f ./otel-collector-config.yml

By running kubectl get namespaces, we can verify that our 'opentelemetry namespace' was indeed created. We can also see namespaces used for the Kubernetes internal workings as well as a default namespace.

ConfigMap

Next, we want to create a ConfigMap resource. To pass the collector a configuration file, we need to store it inside the cluster.

We use a ConfigMap. In the config file, enter three dashes after the namespace --- This lets kubectl know where a new resource ends and another begins.

After that, enter the following ConfigMap resource:

---
apiVersion: v1
kind: ConfigMap
metadata:
 name: otel-collector-conf
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector-conf
data:
 otel-collector-config: |
   receivers:
     otlp:
       protocols:
         grpc:
         http:
   exporters:
     otlp/aspecto:
       endpoint: otelcol.aspecto.io:4317
       headers:
         Authorization: ${ASPECTO_API_KEY}
     jaeger:
       endpoint: localhost:14250
       tls:
         insecure: true
     logging:
   processors:
     Batch:
     resource:
       attributes:
         - key: test.key
           value: "test-value"
           action: insert
   extensions:
     health_check:
     zpages:
       endpoint: :55679
   service:
     telemetry:
       logs:
         level: "debug"
     extensions: [zpages, health_check]
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [logging, jaeger, otlp/aspecto]

Note this is a similar configuration to the one in the article I mentioned at the beginning. But let's go over it quickly to be sure:

Receivers -- We are defining the receivers to receive data in otlp format in both grpc and http protocols on their default endpoint. Which is the local host and the port 4317 for grpc and 4318 for http.
Exporters -- We define two exporters. Jaeger and Aspecto. Jaeger is an open-source visualization tool that we will need to set up locally from an image. And Aspecto is a distributed tracing platform, to use Aspecto you will need to create a free account and generate an Aspecto api key (Go to Setting -> Integrations -> Tokens)
Processors -- We are using a batch processor to send all telemetry data to our exporters in batches. We also use a custom resource to insert a test attribute.
Extensions -- Defining the health check and zPages extensions. We will use them once we deploy the collector.
Service -- Enabling the traces pipelines for the receivers, processors, exporters, and extensions we defined above. We also set the collector telemetry log level to be 'debug' so we can see all sorts of fascinating things when we run.

Service

To pass data to the collector, we need a network service. Let's create and open all the relevant ports.

Enter the following Service resource:

---
apiVersion: v1
kind: Service
metadata:
 name: otel-collector
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 ports:
   - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
     port: 4317
     protocol: TCP
     targetPort: 4317
   - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
     port: 4318
     protocol: TCP
     targetPort: 4318
   - name: zpages
     port: 55679
     protocol: TCP
     targetPort: 55679
   - name: health-check
     port: 13133
     protocol: TCP
     targetPort: 13133
 selector:
   component: otel-collector

Make sure you open all the ports the collector is using. Otherwise, we will not be able to receive data or communicate with the collector.

Deployment

Because we run the collector as a gateway, we will use a deployment. It will allow us to scale once we encounter an increase in traffic.

Enter the following configuration:

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: otel-collector
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 selector:
   matchLabels:
     app: opentelemetry
     component: otel-collector
 minReadySeconds: 5
 replicas: 1
 template:
   metadata:
     namespace: opentelemetry
     labels:
       app: opentelemetry
       component: otel-collector
   spec:
     containers:
       - command:
           - '/otelcol'
           - '--config=/conf/otel-collector-config.yaml'
         image: otel/opentelemetry-collector:latest
         name: otel-collector
         resources:
           limits:
             cpu: '1'
             memory: 2Gi
           requests:
             cpu: 200m
             memory: 400Mi
         ports:
           - containerPort: 4317   #otlp grpc
           - containerPort: 4318   # otlp http
           - containerPort: 55679  # zpages
           - containerPort: 13133  # health check
         volumeMounts:
           - name: otel-collector-config-vol
             mountPath: /conf
     volumes:
       - configMap:
           name: otel-collector-conf
           items:
             - key: otel-collector-config
               path: otel-collector-config.yaml
         name: otel-collector-config-vol

In this configuration, we define one container to run the opentelemetry-collector image. It receives a command line argument to run with a configuration file created from our ConfigMap resource.

The configuration file is created in the volumes section of the deployment and in the volumeMounts section inside the container.

We are setting the deployment to have exactly 1 replica and setting the container CPU and memory limits according to the minimum that was checked for performance in their docs.

Deploying the OpenTelemerty Collector Gateway locally

Now that we created our cluster configuration, it's time to apply it to the cluster.

Run:

Kubectl apply -f ./otel-collector-config.yml

Let's check the collector is alive by making a health check call:

curl localhost:13133

We got back the following response:

curl: (7) Failed to connect to localhost port 13133 after 5 ms: Connection refused

Issue #1 -- Specifying the right type of service: This is not a problem with the collector. This is because we forgot to specify to our service that we need these ports exposed outside of the cluster. This means we need a service of the type LoadBalancer. Let's add ' type: LoadBalancer' inside the service spec block.

Calling the health check again still brings the same result.

Issue #2 -- Opening minikube tunnel: In minikube, in order to communicate with our load balancer, we need to run the command 'minikube tunnel' in a different terminal. This tunnel needs to stay open as long as our cluster runs.

Calling the health check once more gives us the following response:

{
"status":"Server available", "upSince":"2022-10-08T04:55:17.253942835Z", "uptime":"2h19m34.104112936s"
}

Now let's start generating some traces for testing.

I created a simple Typescript program that sends a span every 5 seconds to a collector's endpoint using an otlp-http exporter.

import { trace } from '@opentelemetry/api';
import * as opentelemetry from '@opentelemetry/sdk-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const otelColEndpoint =
 process.env.OTEL_COL_OTLP_ENDPOINT || 'http://localhost:4318';

console.log(`Exporting to traces to ${otelColEndpoint}`);

const exporter = new OTLPTraceExporter({
 url: otelColEndpoint,
});

const sdk = new opentelemetry.NodeSDK({
 spanProcessor: new SimpleSpanProcessor(exporter),
 serviceName: process.env.SERVICE_NAME,
});

sdk.start();
console.log(`${process.env.SERVICE_NAME}: Start Tracing...`);

let spanCounter = 0;
setInterval(() => {
 trace.getTracer('My Tracer').startSpan('Span number: #' + spanCounter++).end();
}, 5000);

Here you might start having issues. In order to solve these issues it's useful to look at the collector's logs.

First set the current context so you look at the OpenTelemery namespace by default:

kubectl config set-context --current --namespace=opentelemetry

Secondly, print all the pods in the namespace:

kubectl get pods

Copy the collector pod id and print its logs:

kubectl logs otel-collector-647fc9c4fd-g958h

Do it all in a one-liner:

kubectl logs `kubectl get pods | awk '/otel-collector/ {print $1}'`

You can also watch new logs by adding the -f flag to the command.

Now we are running the trace generator service. The minikube cluster is also up and running. But both Jaeger nor Aspecto received any traces. Where is the problem now? Looking at the collector's logs, we see no trace for traces (pun intended). Our collector never received any telemetry data.

Issue #3 -- Traces being sent to the wrong collector's endpoint: Our trace generator sent traces to http://localhost:4318 but the collector expects to receive traces at /v1/traces. Changing the Collector endpoint inside the trace generator service worked.

We start seeing logs indicating that the trace was received by one of the receivers.

2022-10-08T07:14:16.204Z    info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 1}

Here is a log printed by our logging exporter.

Looking at our visualization platforms, we still do not see any traces. Solving one issue at a time:

Issue #4 -- Run Jaeger inside the cluster with its own service and deployment: Because localhost is a local interface IP address. Hosts, nodes, and pods have their own localhost interfaces and they are not connected to each other. This is also true for our local machine where the Jaeger image runs.

Let's deploy Jaeger inside the cluster with its own service and deployment.

Create a directory for all our Kubernetes manifests. Move the previous configuration and create a new jaeger.yml file:

mkdir otel-k8s
mv otel-collector-config.yml otel-k8s/otel-collector-k8s.yml
cd otel-k8s
touch jaeger.yml

In the Jaeger manifest, create a deployment with the same properties as the container we ran before with docker compose.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: jaeger-all-in-one
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 replicas: 1
 selector:
   matchLabels:
     app: opentelemetry
     component: otel-collector
 template:
   metadata:
     labels:
       app: opentelemetry
       component: otel-collector
   spec:
     containers:
       - image: jaegertracing/all-in-one:latest
         name: jaeger
         ports:
           - containerPort: 16686
           - containerPort: 14250
---

Make sure the labels and namespace are the same as the rest of our OpenTelemetry setup.

Add a headless service with a selector matching our deployments so they can communicate with each other.

apiVersion: v1
kind: Service
metadata:
 name: jaeger-all-in-one
 namespace: opentelemetry
 labels:
   app: opentelemetry
   component: otel-collector
spec:
 ports:
   - name: collector
     port: 14250
     protocol: TCP
     targetPort: 14250
 selector:
   app: opentelemetry
   component: otel-collector

Finally, add the Jaeger UI port 16686 to the load balancer and change the Jaeger exporter endpoint to the newly created deployment.

Add Jaeger to the load balancer list of ports:

 ports:
   - name: otlp-grpc
   ...
   - name: otlp-http
   ...
   - name: zpages
   ...
   - name: health-check
   ...
   - name: jaeger
     port: 16686
     protocol: TCP
     targetPort: 16686

Modify the Jaeger exporter:

exporters:
     otlp/aspecto: ...
     jaeger:
       endpoint: jaeger-all-in-one:14250
       tls:
         insecure: true
     logging:

Deploy by applying the entire directory:

kubectl apply -f ./

Now, let's look at Jaeger UI at localhost:16686, and voila:

We can see traces from our trace service.

Time to handle the issue with our second exporter

Issue #5 -- Set a Kubernetes Secret resource for your secret environment variables: In previous tutorials, we used environment variables to pass the remote tracing platform's credentials. But here it doesn't work. The correct way to handle sensitive information is with Secrets.

Let's create a secret using kubectl:

kubectl create secret generic aspecto --from-literal=api-key=$ASPECTO_API_KEY

Verify the secret was created and encrypted by running:

kubectl get secrets -o json

We get back a JSON with the api-key encrypted. Let's add the ASPECTO_API_KEY environment variable to the container:

...
containers:
 - command:
     - '/otelcol'
     - '--config=/conf/otel-collector-config.yaml'
   env:
     - name: ASPECTO_API_KEY
       valueFrom:
         secretKeyRef:
           name: aspecto
           key: api-key
           optional: false
   image: otel/opentelemetry-collector:latest
...

Now, after re-running our cluster we can see traces in both our visualization platforms:

We have a working local gateway for telemetry data.

Final Thoughts and Next Steps

Setting up your Kubernetes cluster to start collecting telemetry data could be a challenging task. In this guide, we set up a working OpenTelemetry Collector that is configured to receive data in OTLP format via Http or gRPC.

We then ran the collector using Kubernetes. Lastly, we attempted to solve some of the most common issues you might encounter as a beginner with OpenTelemetry and Kubernetes.

This guide is suitable for practice and learning. For real-world production systems, I recommend reading the following articles:

Thanks for reading, and please let me know if you have any questions about this article.

How to Instrument AWS Services with OpenTelemetry

Yoav Danieli — Mon, 19 Sep 2022 07:18:23 +0000

In this AWS OpenTelemetry guide, you will learn how to instrument your AWS services with OpenTelemetry. I will demonstrate how to instrument a simple microservice system running on AWS. We will use AWS SQS, DynamoDB, and Lambda.

First, we will create all the resources we require using Terraform and AWS. Secondly, we will instrument our code automatically using Opentelemetry node sdk. Lastly, we will deploy the system and visualize the traces it creates.

Let’s begin!

What is OpenTelemetry

OpenTelemetry is a community-driven open-source project and a unified specification for how we collect, generate, and export telemetry data to analyze our software’s behavior.

Sponsored by the CNCF (Cloud Native Computing Foundation), the OpenTelemetry project provides APIs and SDKs per programming language for generating telemetry, a centralized Collector that receives, processes, and exports data, and the OTLP protocol for shipping telemetry data.

In a cloud-native environment, we use OpenTelemetry (OTel for short) to gather data from our system operations and events. In other words, to instrument our distributed services. This data enables us to understand and investigate our software’s behavior and troubleshoot performance issues and errors.

To get a deeper understanding of this technology, watch our free OpenTelemetry Bootcamp video series that, in 6 episodes, cover OTel from end to end (it’s binge-worthy, really).

AWS + OpenTelemetry: Creating our application

We will create a simple Order taking application. It will be composed of the following components:

An order-api written with Node and deployed as an AWS Lambda.
AWS DynamoDB database
AWS SQS messaging queue
External service written with Node that listens on the SQS queue.

The application will receive orders and insert those orders into the database with status processing. It will publish an order received message to SQS then the external service in charge of processing these orders will receive this message and change the order status in the database to completed.

That is it! Let’s start by setting up AWS lambda.

Setup Disclaimer

There are many ways of interacting with the AWS platform. Each with its pros and cons. I like using Terraform and other PaaS code so that I can easily change, configure, replicate or migrate my work. So in this tutorial, I will create all the AWS services and resources I require using Terraform. Explanations regarding Terraform and AWS permissions and configurations are out of this article’s scope.

For the exact implementation details, you’re welcome to read the code example for this guide.

AWS Lambda Setup

The first component of our system is the order-api. Let’s create a new Typescript project and add the lambda code to the index.ts file:

import { SQS, DynamoDB } from "aws-sdk";
import { Context, APIGatewayEvent, APIGatewayProxyResult } from 'aws-lambda';
import { SendMessageRequest } from "aws-sdk/clients/sqs";
const sqs = new SQS();
const dynamo = new DynamoDB.DocumentClient();
const handler = async (event: APIGatewayEvent, context: Context): Promise<APIGatewayProxyResult> => {
   let body; ``
   let statusCode = 200;
   const routeKey = `${event.httpMethod} ${event.resource}`
   const headers = {
       "Content-Type": "application/json"
   };
   try {
       const tableName = process.env.DDB_TABLE_NAME as string
       const sqsUrl = process.env.SQS_URL as string
       if (!tableName) {
           return { statusCode: 500, body: 'Missing environment variable DDB_TABLE_NAME', headers }
       }
       if (!sqsUrl) {
           return { statusCode: 500, body: 'Missing environment variable SQS_URL', headers }
       }
       const id = event.pathParameters?.id
       switch (routeKey) {
           case "DELETE /items/{id}": // Delete Item...
           case "GET /items/{id}": // get item...
           case "GET /items": // get all items...
           case "PUT /items":
               let requestJSON = JSON.parse(event.body as string);
               await dynamo.put({
                   TableName: tableName,
                   Item: {
                       id: requestJSON.id,
                       price: requestJSON.price,
                       name: requestJSON.name,
                       status: "processing"
                   }
               }).promise();
               const params: SendMessageRequest = {
                   MessageBody: JSON.stringify({ id: requestJSON.id }),
                   QueueUrl: sqsUrl,
               }
               await sqs.sendMessage(params).promise();
               body = `Put item ${requestJSON.id}`;
               break;
           default:
               throw new Error(`Unsupported route: "${routeKey}"`);
       }
   } catch (err: any) {
       statusCode = 400;
       body = err.message;
   } finally {
       body = JSON.stringify(body);
   }
   return {
       statusCode,
       body,
       headers
   };
};
module.exports = { handler }

As you can see, the Put clause creates a new item with status “processing” and inserts it into DynamoDB. It then sends the newly created order’s id as a string to SQS.

Let’s compile the code using tsc and ensure a new dist folder was created.

Now add this to your terraform configuration:

resource "aws_lambda_function" "order_api" {
 function_name = "order_api"
 filename         = data.archive_file.lambdas_code.output_path
 source_code_hash = data.archive_file.lambdas_code.output_base64sha256
 role    = aws_iam_role.iam_for_lambda.arn
 handler = "index.handler"
 runtime = "nodejs12.x"
 timeout = 10
  environment {
   variables = {
     SQS_URL = # TBD
     DDB_TABLE_NAME = # TBD

   }
 }
}
data "archive_file" "lambdas_code" {
 type        = "zip"
 output_path = "${path.module}/dist.zip"
 source_dir  = "${PATH_TO_DIST_FOLDER}/dist"
}

Important Notes

Before deploying the Lambda, verify that the compiler target language is compatible with the Node runtime you specified. For compatibility check, visit here.
Check that the source dir specified in the archive_file resource contains index.js and node_modules.
In the example code for this blog, I also added an AWS api gateway so we can trigger the Lambda using a public URL. That is out of scope for this blog, but you can visit the source code and check it out.

AWS SQS Setup

Create the SQS queue and add the created queue URL as an environment variable to our lambda.

resource "aws_sqs_queue" "order_queue" {
 name = "orderQueue"
}

Change SQS_URL:

SQS_URL = aws_sqs_queue.order_queue.id

AWS DynamoDB

Create a DynamoDB table and add the table name as an environment variable to our lambda.

resource "aws_dynamodb_table" "order_table" {
 name           = "Orders"
 billing_mode   = "PROVISIONED"
 read_capacity  = 20
 write_capacity = 20
 hash_key       = "id"
 attribute {
   name = "id"
   type = "S"
 }
}

Change DDB_TABLE_NAME:

DDB_TABLE_NAME = aws_dynamodb_table.order_table.name

OpenTelemetry + AWS: External Service

The external service is a simple Node service. It can run anywhere as long as it has the permissions necessary to read messages from the queue and update items in the DynamoDB table. This example runs the service locally.

To receive the messages from the queue, we will use the sqs-consumer library. The service will receive messages describing newly created orders. After some processing, it will change the order status in the table to ‘completed’.

Create sqs-listner.ts:

import { DynamoDB } from "aws-sdk";
import { UpdateItemInput } from "aws-sdk/clients/dynamodb";
import { Consumer } from "sqs-consumer";
const dynamo = new DynamoDB.DocumentClient({ region: process.env.AWS_REGION as string });
async function dynamoUpdateCompleted(id: string) {
   const params: UpdateItemInput = {
       TableName: process.env.DDB_TABLE_NAME as string,
       Key: {
           // @ts-ignore
           id
       },
       ExpressionAttributeNames: {
           "#S": "status"
       },
       // This expression is what updates the item attribute
       ExpressionAttributeValues: {
           ":s": { S: "complete" }
       },
       UpdateExpression: "SET #S = :s",
       ReturnValues: "ALL_NEW",
   };
   await dynamo.update(params).promise()
}

const app = Consumer.create({
   queueUrl: process.env.SQS_URL as string,
   handleMessage: async (message) => {
       if (message.Body) {
           const parsedBody = JSON.parse(message.Body);
           // do some processing and then change status to complete
           await dynamoUpdateCompleted(parsedBody.id);
       }
   }
});
app.on('error', (err) => {
   console.error(err.message);
});
app.on('processing_error', (err) => {
   console.error(err.message);
});
app.start();

Important Notes

Make sure you have AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID configured on your machine.
The message is deleted from the queue once the handleMessage function is completed. That is important because it lets whoever uses this queue know that the message was received and acknowledged.

Instrumenting AWS services with OpenTelemetry and exporting telemetry data

To instrument Node services, we create a file containing the code and the configuration for our tracing.

We then require this file before any other code in our system runs so it can wrap the relevant function and create spans for any of the operations our system is performing.

On AWS, it is no different. Let’s create a tracing.ts file and add the OpenTelemetry configuration to it.

The OpenTelemetry is implemented and accessed by the application through the NodeSDK instance. So we will initiate it with our configuration.

First, let’s add the instrumentations. Our Lambda services use the aws-sdk library and the aws-lambda library. So any library that provides auto-instrumentations for these operations should be enough. Luckily when using Node, we can use the auto-instrumentation-node library.

It includes all Node auto instrumentations available to the public, including:

Important Configuration Notes

The lambda instrumentation is trying to use the X-Ray context headers by default (even when we’re not using X-Ray), causing us to have a non-sampled context and a NonRecordingSpan. To fix this, we use the disableAwsContextPropagation flag. More information about this can be found here and in the instrumentation docs.
The SQS queue we have created is configured by default to show message attributes in its payload. That is called ‘Raw Message Delivery.’ You can read more about it in AWS docs. When this is the case, we need to explicitly tell our program to receive its context from the payload. We do this by enabling sqsExtractContextPropagationFromPayload and setting it to true. Know that there will be performance implications because now the instrumentation will run JSON.parse to get the data.

Let’s process our spans using a batchSpanProcessor and a custom OTLPTtraceExporter. We can export our traces to the console, but to watch them, we will have to visit AWS CloudWatch (which will l be a bit messy).

Visualizing OpenTelemetry Data in Aspecto

AWS has good tools for tracing, but in this example, I will use another remote and distributed tracing platform – Aspecto.

To follow along, you can open a new free-forever Aspecto account or log in to your existing one.

Below, make sure to replace the {ASPECTO_API_KEY} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

Finally, let’s give our service a name that can be taken from either the SERVICE_NAME or the AWS_LAMBDA_FUNCTION_NAME environment variables.

Putting it all together, it looks something like that:

import { NodeSDK } from "@opentelemetry/sdk-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
const exporter = new OTLPTraceExporter({
   url: 'https://otelcol.aspecto.io/v1/traces',
   headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
   }
})
const sdk = new NodeSDK({
   spanProcessor: new BatchSpanProcessor(exporter),
   serviceName: process.env.SERVICE_NAME || process.env.AWS_LAMBDA_FUNCTION_NAME,
   instrumentations: [
       getNodeAutoInstrumentations({
           "@opentelemetry/instrumentation-aws-sdk": {
               sqsExtractContextPropagationFromPayload: true
           },
           "@opentelemetry/instrumentation-aws-lambda": {
               disableAwsContextPropagation: true
           }
       })
   ]
});
sdk.start()

To use OpenTelemetry Node SDK, we must load and run it before our application is loaded. So let’s compile it using tsc so it ends up inside the dist folder and be packaged together with the Lambda code.

After that, we need to add a NODE_OPTIONS environment variable to our Lambda, notifying the node runtime to run this process before the lambda code. Here’s the final Lambda configuration:

resource "aws_lambda_function" "order_api" {
 function_name = "order_api"
 filename         = data.archive_file.lambdas_code.output_path
 source_code_hash = data.archive_file.lambdas_code.output_base64sha256
 role    = aws_iam_role.iam_for_lambda.arn
 handler = "index.handler"
 runtime = "nodejs12.x"
 timeout = 10
  environment {
   variables = {
     ASPECTO_API_KEY = var.ASPECTO_API_KEY
     SQS_URL = aws_sqs_queue.order_queue.id
     DDB_TABLE_NAME = aws_dynamodb_table.order_table.name
     NODE_OPTIONS =  "--require tracing.js"
   }
 }
}

Important Notes

You don’t have to create an opentelemetry configuration file such as this for each of your lambdas. In fact, you shouldn’t. In AWS, you can use Lambda Layers. You can define the OpenTelemetry tracing piece of code as a Lambda layer and use it in any Lambda you want. Furthermore, OpenTelemetry went ahead and implemented this opentelemetry-lambda layer for us. All we need to do is use it with our config.

Running AWS Services and Visualizing OpenTelemetry Data in Aspecto

When running the SQS-listener service, remember to require the tracing file configuration.

Note that for this service, you can remove the disableAwsContextPropagation flag.

Let’s compile and run the service:

SERVICE_NAME=sqs-listener node --require path/to/tracing.js path/to/sqs-listener.js

Now the service is waiting for messages to be sent to the queue. Let’s deploy our Lambda so we can invoke it and send messages to the queue:

terraform -chdir=path/to/terraform/config apply --var ASPECTO_API_KEY=${ASPECTO_API_KEY}

You can invoke the Lambda using AWS CLI, or if you set up an api gateway, you can also make http requests.

curl --header "Content-Type: application/json" \
--request POST \
--data '{"id":"1","price":100, "name": "Awesome Item"}' \
http://your-api-gateway-lambda-invoke-url/items

You should get back a 200 response with the message “Put item 1”. Let’s check out our traces in Aspecto:

I received the following traces:

As you can see, our flow is divided into two traces. This happens because SQS sends messages in batches. I will explain more about it below. For now, let’s examine the traces.

Click the “order_api” lambda trace:

We can see 5 spans:

The Lambda trigger
AWS-SDK operation for dynamo DB + http request.
AWS-SDK operation for SQS + http request

Clicking the SQS span:

We can see that the message is linked and was received by another trace. A click on this link redirects us to the second trace:

This trace contains 7 spans. By clicking the Node service, I can see a more convenient view of the spans (on the left)

It seems that 4 spans were created from the AWS-SDK instrumentation, while the other 3 from the HTTP-instrumentation. We can disable the http instrumentations by passing the following flag in the configuration: suppressInternalInstrumentation.

By putting these two traces together we can get a full and clear view of our system.

OpenTelemetry AWS SQS: Batch Receiving

According to the opentelemetry specification for messaging systems, When a process receives messages in a batch it is impossible for this process to determine the parent span for the span that it is currently creating.

Since a span can only have one parent if it is propagated and the propagated trace and span IDs are unknown when the receiving span is started, the receiving span will not have a parent, and the processing spans are correlated with the producing spans via links.

Conclusion

In this guide, you hopefully learned how to create a simple application using AWS lambdas, SQS, DynamoDB, and instrument the application using OpenTelemetry. I also pointed out some tips and gotchas I encountered while working on this project.

Please feel free to contact me with any questions, comments, or suggestions.

Top 5 Jaeger Alternatives for Distributed Tracing

Yoav Danieli — Tue, 13 Sep 2022 11:16:00 +0000

In this article, we will explore 5 Jaeger alternatives you can test out immediately. We chose to cover these tools specifically because you can start using them today, and they provide a free version with enough room to play around with.

The tools we are covering:

Aspecto
Honeycomb
SigNoz
Lightstep
Logz.io

As you probably know, Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.

These Jaeger alternatives can replace the entire Jaeger suite, though we will focus mainly on trace visualization.

Another noteworthy aspect of these Jaeger alternatives is OpenTelemetry support. Most of these tools support OpenTelemetry whether you already have it running in your system or looking for an OpenTelemetry distro (an OpenTelmetry SDK with some vendor-specific customizations).

P.S. Shoutout to all of them for providing excellent OpenTelemetry education and resources.

Top Jaeger Alternatives in 2022

Before jumping in, make sure to check these checkboxes when looking for a Jaeger alternative:

A free version and you can test it out quickly
Supports OpenTelemetry
Provides the same or better trace visualization capabilities
Allows you to search and investigate trace data efficiently

Aspecto

Aspecto is an end-to-end distributed tracing platform that is fully compatible with OpenTelemetry. Aspecto allows developers to troubleshoot performance bottlenecks and errors within their microservices and correlate root causes across traces and logs.

Dev and observability teams use Aspecto as their OpenTelemetry infrastructure. From fully supporting the implementation to the consumption, management, scalability, and visualization of OpenTelemetry data.

Notable features

1) Advanced OpenTelemetry data search and filtering capabilities such as free text search, key value search, and filters by parameters.
2) Advanced service map visualization: Aspecto offers a service map visualization to improve your troubleshooting abilities and create a clear image of the relations between services.
3) Logs and traces integration: Integrate Aspecto and your logging solution to correlate logs with their matched traces, side-by-side, under a single view so you short-cut your debugging workflow.
4) Remote sampling configuration: Remotely configure new OpenTelemetry head and tail sampling rules from the Aspecto UI without changing your code.
5) End-to-end message broker visibility: Visualize and understand the entire journey messages go through (including Kafka, RabbitMQ, SQS, and more)

Pricing

Aspecto has a Free Forever Plan with unlimited features and users. Its paid plans start as low as $4 per 5M traces per month. Aspecto also offers a plan for R&D teams needing tailored OpenTelemetry implementation.

You can also give the free playground a try.

Product Overview

Honeycomb

Honeycomb is an observability tool designed for engineering teams providing visibility to troubleshoot distributed systems. Honeycomb provides an automatic instrumentation agent called Honeycomb Beelines and also supports OpenTelemetry.

Notable features

BubbleUp — Automatically detecting anomalies, highlighted and visualized on a heatmap
SLOs — Set up triggered alerts based on incident response time and other important service level objectives.
Metrics — View system metrics alongside application observability data

Pricing

Honeycomb offers a free plan, and its paid plan starts at $110. The pricing is based on event volume captured per month.

Product Overview

Lightstep

Lightstep offers two solutions — incident response and observability tool. Incident Response helps IT Operations and DevOps find their path to fast incident resolution. The Lightstep Observability platform combines metric and tracing data to gain observability to your system.

Notable features

Change Intelligence — This allows you to surface important changes in your system. Change Intelligence correlates changes found in traces with deviations in your data.
Notebook — Query and save both your infrastructure (metrics) and app performance (traces) data in one place
Create alerts for metric data — You can create alerts for your metric data by configuring thresholds against metric queries that when crossed, trigger the alert

Pricing

For its observability solution, Lightstep offers a Free version to help you get started. Its paid plan starts at $100 per month and is based on the number of monthly active services.

Product Overview

SigNoz

SigNoz is a full-stack open-source APM and observability tool. It captures both metrics and traces. Since SigNoz is an open-source project It can be self-hosted so you won’t need to send your data to any third party.

Notable features

Metrics support — visualize your Prometheus metrics.
Live view — test your app by debugging traces in real time
Service map visualization — visualize all services and the relations between them

Pricing

Signoz has a Free Plan for self-hosted apps and limited features. Its paid plans start from $200 which acquire you additional premium features such as hosting & managing your app by Signoz

Product Overview

Logz.io

Similar to Lightstep, Logz.io offers various features among them you can find Distributed tracing. Logz.io offers managed Jaeger Tracing as their distributed tracing solution and, on top of Jaeger, they offer additional features. This is a good option if you want to stick to Jaeger but looking for a managed solution.

Notable features

Managed Elasticsearch with Jaeger
Real-time alerting — identify high-priority production incidents and receive alerts via Slack, email, and other tools.
Service Performance Monitoring dashboard — View request rate, error rate, and latency under a single dashboard. The dashboard includes a breakdown of R.E.D data based on the operations running inside the chosen service.

Pricing

Logz.io offers a free plan for collecting small log volumes with advanced analytics and Up to 1 GB of log data. Its paid plan starts at 5$ Per million spans.

Product Overview

https://www.aspecto.io/wp-content/uploads/2022/07/Yalla-Zehooo.mp4

Jaeger and why you may want alternatives

If you’re reading this article, you probably have some experience with Jaeger and concluded that Jaeger is powerful, but it might not be the best solution for you.

We hope that the option we provided above, each unique in its way, will serve you well in finding your observability match.

Since Jaeger is so tied together with OpenTelemetry, and if you want to learn more, we recommend you check out our free OpenTelemtry Bootcamp which you can find here or on Youtube.

Getting Started with Aspecto

Aspecto has a free-forever tier and provides everything included in Jaeger and more. Like Jaeger on steroids. Feel free to try the playground and sign up for the free forever plan.

Guide to OpenTelemetry Distributed Tracing in Rust

Yoav Danieli — Tue, 30 Aug 2022 09:10:50 +0000

In this article, I will share my experience adding OpenTelemetry distributed tracing to a Rust application.

I will attempt to answer the following questions:

How to instrument Opentelemetry in Rust?
How to add manual and auto instrumentations to a Rust application?
How to use tracing to debug Rust applications?
How to visualize traces and spans?
How to preserve our span context in a multithreaded environment?
What are your must-know crates when it comes to tracing in Rust?

I will begin by saying that I love Rust as a programming language, and I’ve been experimenting with it for the past year. Although it’s innovative and has a great ecosystem, it is not a secret that Rust has quite a steep learning curve. So even though it’s an intro-level article, I will not dive deep into Rust features, syntax, or semantics unless it’s somehow connected to our subject (Opentelemetry and distributed tracing).

What to Expect

What is OpenTelemetry?
Opentelemetry and Distributed Tracing in Rust
- Tracing in Rust
- OpenTelemetry in Rust
- Distributed tracing in Rust
Practical Example
- Creating the service
- Create our tracing setup
- Adding Auto-Instrumentation for Actix-Web framework
- Visualizing traces with Aspecto
- Manually instrumenting Rust functions
- Adding Auto-Instrumentation for Diesel
- How to preserve our span context when executing closures on different threads?
Conclusion

What is OpenTelemetry?

OpenTelemetry is a collection of APIs and SDKs that allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

It is a CNCF community-driven open-source project (Cloud Native Computing Foundation, the folks in charge of Kubernetes).

In a cloud-native environment, we use OpenTelemetry (OTel for short) to gather data from our system operations and events. In other words, to instrument our distributed services. This data enables us to understand and investigate our software’s behavior and troubleshoot performance issues and errors.

OpenTelemetry serves as a standard observability framework that captures all data under a single specification. It provides several components, including:

APIs and SDKs per programming language for generating telemetry.
The OpenTelemetry Collector receives, processes, and exports telemetry data to different destinations.
OTLP protocol for shipping telemetry data

To get a deeper understanding of this technology, including its structure and the primary motivation, visit this guide.

For this OpenTelemetry Rust article, here are the terms you need to know:

Span: The most basic unit. A span represents an event in our system (e.g., an HTTP request or a database operation that spans over time). A span would usually be the parent of another span, its child, or both.
Trace: ‘Call-stacks’ for distributed services. Traces represent a tree of spans connected in a child/parent relationship. Traces specify the progression of requests across different services and components in our app (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.
Exporter: Once we create a span, the exporter handles sending the data to our backend (e.g., in memory, Jaeger Tracing, or console output)
Context propagation – The mechanism that allows us to correlate events across distributed services. Context is referred to as the metadata we collect and transfer. Propagation is how the context is packaged and transferred across services, often via HTTP headers. Context propagation is one of the areas where OpenTelemetry shines.
Instrumentation – instrumentation libraries gather the data and generate spans based on different libraries in our applications (Actix-Web, Postgres…).

To understand more of the OpenTelemetry jargon, visit the official documentation.

Opentelemetry and Distributed Tracing in Rust

Tracing in Rust

In Rust, we have a great crate simply called tracing. It is our core framework for tracing a Rust program. The crates documentation outlines its core concepts and APIs which are composed of these terms – spans, events, and subscribers.

I covered what Spans and Events are, but a Subscriber is a new term. To record spans and events, one has to implement the subscriber trait. That means simply implementing the methods:

enter: indicating entering a span.
exit: indicating ending a span.
event: indicating an event has occurred.

To start recording traces, we need to initialize a subscriber. The crate tracing_subscriber helps us to do exactly that.

We initialize the Registry struct:

let subscriber = tracing_subscriber::Registry::default()

This struct implements a subscriber and exposes us to another important feature of this crate – The Layer. Using layers, we can configure our Subscriber to apply specific behaviors when interacting with spans and events.

For example, if we want to filter some of our data, export it, format it or edit it, we can create or use an existing layer and compose it with the registry subscriber.

let subscriber = tracing_subscriber::Registry::default().with(SomeLayer)

We will see more examples of how to use these crates to instrument a Rust program in the practical section.

OpenTelemetry in Rust

The crate that provides support for OpenTelemetry is called – how convenient – OpenTelemetry [see here].

“OpenTelemetry provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application.”

Let’s go over some of its key APIs:

Tracer. Inside the crate, there is a module called traces. It introduces us to a trait called Tracer. This is what tracks and connects our spans to create traces. To use Tracer, we create a subscriber layer. This is how the subscriber knows where to send the spans and how to generate traces.
Global. This module provides an API so that no matter in what section of the code we currently are, we can access the subscriber, tracer, and propagation context.
SDK. This module provides an implementation for common use cases with OpenTelemetry. For example, it provides an exporter to send traces to stdout. It also implements context propagation methods that we can use instead of implementing ourselves.

The git repository called opentelemetry-rust contains implementations of several crates that expand the opentelemetry ecosystem. There you can find common instrumentations, exporters, and subscribers use them in a Rust program.

OpenTelemetry Distributed tracing in Rust

Using crates from the opentelemetry-rust repo, we can instrument our application across multiple services and send them to a distributed platform.

For example, we can use the opentelemetry-jaeger crate to send our traces to a remote Jaeger platform.

Another example is the OpenTelementy-otlp crate.

Traces in OTLP format can be exported to the OpenTelemetry Collector using the exporter in this crate. OpenTelemetry Collector accepts, processes, and exports traces in a vendor-agnostic manner.

Practical OpenTelemetry Rust Example

For this example, I wanted to build a simple system I could use to integrate and visualize traces. I found auto-instrumentations for Actix-Web Framework and Diesel, an ORM and Query builder utility for SQL-based databases.

I chose to use these tools and build a simple REST api service to create, read and delete users following the examples given by these frameworks.

Here is a link for the source code.

Creating the service

First, I followed the Actix web and diesel documentation and created a simple web server that exposes a user service and communicates with a local Postgres database:

mod models;
mod schema;
mod users;
#[macro_use]
extern crate diesel;
extern crate dotenv;
use dotenv::dotenv;
use actix_web::{web, App, HttpServer};
use diesel::r2d2::{self, ConnectionManager};
use diesel::pg::PgConnection
use telemetry::init_telemetry;
use users::get_users_service;
pub type Connection = PgConnection
pub type DbPool = r2d2::Pool<ConnectionManager<Connection>>;
#[actix_web::main]
async fn main() -> std::io::Result<()> {
  dotenv().ok();
  let database_url = std::env::var("DATABASE_URL").expect("DATABASE_URL must be set");
  let manager = ConnectionManager::<InstrumentedPgConnection>::new(database_url);
  let pool: DbPool = r2d2::Pool::builder()
      .build(manager)
      .expect("Failed to create pool.");
  HttpServer::new(move || {
      App::new()
          .app_data(web::Data::new(pool.clone()))
          .service(get_users_service())
  })
  .bind(("127.0.0.1", 3000))?
  .run()
  .await
}

For this blog, I will only show one endpoint and its flow. You can find the rest in the source code.

use crate::DbPool;
use crate::{models::NewUser};
use actix_web::{delete, get, post, web, Error, HttpResponse};
use uuid::Uuid;
#[post("")]
async fn create_user(
  db: web::Data<DbPool>,
  new_user: web::Json<NewUser>,
) -> Result<HttpResponse, Error> {
  let user = web::block(move || {
      let conn = db.get()?;
      db_operations::insert_new_user(&new_user.name, &conn)
  })
  .await?
  .map_err(actix_web::error::ErrorInternalServerError)?;
  Ok(HttpResponse::Ok().json(user))
}
pub fn get_users_service() -> actix_web::Scope {
  web::scope("/users")
      .service(get_users)
      .service(get_user_by_id)
      .service(create_user)
      .service(delete_user)
}
mod db_operations {
pub fn insert_new_user(nm: &str, conn: &PgConnection) -> Result<models::User, DbError> {
  use crate::schema::users::dsl::*;
  let new_user = models::User {
      id: Uuid::new_v4().to_string(),
      name: nm.to_owned(),
  };
  diesel::insert_into(users).values(&new_user).execute(conn)?;
  Ok(new_user)
}
}

Create our tracing setup

Now that I have a working service, I can add telemetry data. I will need to configure a tracing subscriber and its layers.

use opentelemetry::sdk::export::trace::stdout;
use opentelemetry::sdk::propagation::TraceContextPropagator;
use opentelemetry::sdk::{trace, Resource};
use opentelemetry::KeyValue;
use opentelemetry::{global};
use std::collections::HashMap;
use std::env;
use tracing_subscriber::Registry;
use tracing_subscriber::{prelude::*, EnvFilter};
pub fn init_telemetry() {
  // Define Tracer
  let stdout_tracer = stdout::new_pipeline().install_simple();
  let subscriber = Registry::default();
  // Layer to filter traces based on level - trace, debug, info, warn, error.
  let env_filter = EnvFilter::try_from_default_env().unwrap_or(EnvFilter::new("INFO"));
  // Layer to add our configured tracer.
  let tracing_leyer = tracing_opentelemetry::layer().with_tracer(stdout_tracer);
  // Setting a trace context propagation data.
  global::set_text_map_propagator(TraceContextPropagator::new());
  subscriber
      .with(env_filter)
      .with(tracing_leyer)
      .init();
}

This configuration adds an env_filter layer and a tracing layer. It prints to stdout all spans that their level is equal to or above the “info” log level.

Adding OpenTelemetry Instrumentation for Actix-Web framework

Now let’s add some spans by instrumenting the actions we take. Let’s start by instrumenting the operation taken by Actix web.

For that, I will use the actix_web_opentelemetry crate which provides a middleware called RequestTracing. As its name suggests, this middleware extracts trace information from the request and creates a span with the same context.

use actix_web_opentelemetry::RequestTracing;
// add the middleware to the server.
HttpServer::new(move || {
      App::new()
          .wrap(RequestTracing::new())
          .app_data(web::Data::new(pool.clone()))
          .service(get_users_service())

Now, when running the application and invoking the create user endpoint, we print spans to the terminal.

Trying to understand something from this print is quite hard. Besides, we are gathered here today to visualize traces, not to print them to the terminal.

So I changed the tracer to export traces to an OTLP collector. I configured the tracer to send traces to the Aspecto platform.

Visualizing traces with Aspecto

To follow along, you can open a new free-forever Aspecto account or log in to your existing one.

Below, make sure to replace the {ASPECTO_API_KEY} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

use opentelemetry_otlp::WithExportConfig;
let exporter = opentelemetry_otlp::new_exporter()
      .http()
      .with_endpoint("https://otelcol.aspecto.io/v1/traces")
      .with_headers(HashMap::from([(
          "Authorization".into(),
          env::var("ASPECTO_API_KEY").unwrap().to_string(),
      )]));
let otlp_tracer = opentelemetry_otlp::new_pipeline()
      .tracing()
      .with_exporter(exporter)
      .with_trace_config(
          trace::config().with_resource(Resource::new(vec![KeyValue::new(
              opentelemetry_semantic_conventions::resource::SERVICE_NAME,
              env::var("SERVICE_NAME").unwrap().to_string(),
          )])),
      )
      .install_batch(opentelemetry::runtime::Tokio)
      .expect("Error - Failed to create tracer.");
let tracing_leyer = tracing_opentelemetry::layer().with_tracer(otlp_tracer);

When running the application once more, I can view my traces in Aspecto:

Clicking the trace, I can view the spans and their data:

Manually instrumenting Rust functions

Let’s add more instrumentations to our service.

By using the macro #[instrument] on a function, we can create a span for this function.

It means that the span name will be the name of the function. Its attribute will include the module crate, the library of the function, and the function’s attributes.

By using instrument(skip(…), field(…)), we can skip some of the arguments of a function that are not important for us to record and manually insert new attributes.

Let’s give it a go:

#[instrument(skip(db))]
#[post("")]
async fn create_user(
  db: web::Data<DbPool>,
  new_user: web::Json<NewUser>,
) -> Result<HttpResponse, Error> {
  let user = web::block(move || {
      let conn = db.get()?;
      db_operations::insert_new_user(&new_user.name, &conn)
  })
  .await?
  .map_err(actix_web::error::ErrorInternalServerError)?;
  Ok(HttpResponse::Ok().json(user))
}

I now added instrumentation for the function create_user. I skipped the db parameters. Let’s view this new span in Aspecto:

We can clearly see the new span added to the trace. Clicking the new span we can see all the attributes I mentioned before:

Adding OpenTelemetry Instrumentation for Diesel

To get a full picture, I will add the instrumentation for the db_operation function:

#[instrument(skip(conn))]
pub fn insert_new_user(nm: &str, conn: &Connection) -> Result<models::User, DbError> {
  use crate::schema::users::dsl::*;
  let new_user = models::User {
      id: Uuid::new_v4().to_string(),
      name: nm.to_owned(),
  };
  diesel::insert_into(users).values(&new_user).execute(conn)?;
  Ok(new_user)
}

And for the diesel library which we are using. To add the diesel instrumentation we will need to use the diesel_tracing crate and replace our PgConnection with InstrumentedPgConnection:

use diesel_tracing::pg::InstrumentedPgConnection;
pub type Connection = InstrumentedPgConnection;
pub type DbPool = r2d2::Pool<ConnectionManager<Connection>>;

I ran the application once more and observed the traces in Aspecto:

We can see many unrelated traces. This is because the database connection struct communicates with the database periodically by calling the establish function to make sure the connection is still live.

How to preserve our span context when executing closures on different threads?

But there is a problem here. Can you see it? For some reason, the spans representing our request and the spans representing the database query are not grouped together. This is happening because of this code:

#[instrument(skip(db))]
#[post("")]
async fn create_user(
  db: web::Data<DbPool>,
  new_user: web::Json<NewUser>,
) -> Result<HttpResponse, Error> {
  let user = web::block(move || {
      let conn = db.get()?;
      db_operations::insert_new_user(&new_user.name, &conn)
  })
  .await?
  .map_err(actix_web::error::ErrorInternalServerError)?;
  Ok(HttpResponse::Ok().json(user))
}

We are invoking our db operation inside a closure we pass to the web::block function. We lose the span context of our execution flow. However, the web::block function is crucial.

It executes a blocking function on a thread pool dedicated to async blocking operations. So how can we keep the context of the trace?

We can do that by wrapping the web::block function with our own tracing function:

fn traced_web_block<F, R>(f: F) -> impl Future<Output = Result<R, BlockingError>>
where
  F: FnOnce() -> R + Send + 'static,
  R: Send + 'static,
{
  let current_span = tracing::Span::current();
  web::block(move || current_span.in_scope(f))
}

Here we get our current span and invoke the closure within its scope.

Let’s view our traces now:

We can now get a full picture of the flow of our application.

OpenTelemetry Distributed Tracing in Rust: Summary

We reviewed how to add distributed tracing with OpenTelemetry to the Rust application. We explored how to add both manual instrumentation and library-provided instrumentations.

We used Aspecto to visualize the flow of our system and learned what to do when the flow is broken.

To sum up, I enjoyed working on this example. I think the approach Rust is taking regarding tracing is interesting and worth considering when writing a Rust program. Furthermore, the ecosystem is growing so I am sure more libraries will provide instrumentations soon.

I hope you learned something new. Feel free to reach out with any questions you might have.

Distributed Tracing for RabbitMQ with OpenTelemetry

Yoav Danieli — Mon, 22 Aug 2022 12:55:44 +0000

In this guide, you will learn how to use OpenTelemetry to instrument RabbiMQ to create spans for different operations. (e.g., consume and produce). We will then see how to visualize your traces in Jaeger and Aspecto. I will use Node.js for all code examples.

Feel free to skip to the practical section of this guide if you are already familiar with RabbitMq and OpenTelemetry.

What to Expect

What is OpenTelemetry
- Using RabbitMq in this guide
The Practical: OpenTelemetry Node and RabbitMQ
- Step 1 – Create an application
- Step 2 – Add RabbitMq messaging code
- Step 3 – Instrument with Opentelemetry
Visualization with OpenTelemety
- OpenTelemetry, RabbitMQ, and Jaeger Tracing
- Advanced Visualization for Aspecto
Final note

What is OpenTelemetry

OpenTelemetry is a CNCF (cloud-native compute foundation) open source project that allows us to collect, export and generate telemetry data – logs metrics, and traces (which together make up the three pillars of observability).

OpenTelemetry provides each programming language with a single API and SDK with which you can instrument your application to generate telemetry. The OpenTelemetry Specification defines the cross-language requirements for the APIs and SDKs.

*But what does it have to do with RabbitMQ?
*
Well, there is a great deal of complexity in modern software applications. Their architecture is made up of many microservices that are independent of one another. For communication and information transfer, these microservices use messaging systems (like RabbitMq).

Distributed architectures are complex, which is why users need an easy way to visualize and troubleshoot them.

We use OpenTelemetry to collect data from different transactions within our services and components. Third-party tools, such as messaging systems like RabbitMq, are also included in those components.

Using the collected telemetry, we can gain a better understanding of how our software performs and behaves.

Check out this short guide for a deeper dive into OpenTelemetry.

As far as this OpenTelemetry js guide is concerned, these terms should be familiar to you:

** Span:** Spans represent actions/operations that have occurred in our system. An HTTP request or a database operation that spans over time (starts at X and has a duration of Y milliseconds). A span would usually be the parent and/or the child of another span.
Trace: Traces represent a tree of spans connected in a child/parent relationship. Traces specify the progression of requests across different services and components in our app (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.
Exporter: Once we create a span, we need to send it to a dedicated location (e.g., a collector). This is the component that sends telemetry data to that destination.
Instrumentation: The instrumentation libraries enable us to gather data and generate spans based on different libraries used in our applications, such as RabbitMQ, Mongo, Express, etc. Our app can be instrumented manually or automatically.
- Auto Instrumentation: Automatically create spans from the application libraries we use with ready-to-use OpenTelemetry libraries.
- Manual instrumentation: Writing specific code manually to define where each span begins and ends.

See the official documentation for more information on OpenTelemetry jargon.

Using RabbitMq in this OpenTelemetry Node guide

RabbitMq is a messaging broker that supports many messaging patterns. One of the most famous messaging patterns is the publisher/subscriber pattern. In this pattern, one service produces and sends the message (publisher), while other services consume the message (subscribers). It is possible for any service to subscribe to receive messages from the publisher.

In our guide, we will use RabbitMq’s publisher/subscriber pattern. For more information on RabbitMQ visit the official site.

The Practical: OpenTelemetry and RabbitMQ

Create an application
Add RabbitMq messaging code
Instrument with OpenTelemetry

Step 1 – Create a Node.js application

First, let’s create a basic node application. It will consist of two services. a publisher and subscribers. The publisher will listen to requests from the user. After receiving such a request it will publish a message to a topic (called exchange in RabbitMq terms). The services that will subscribe to this exchange will receive and print the message.

The publisher code:
/* publisher.js */
const express = require('express')
const app = express()
const port = 3000
app.get('/', (req, res) => {
  // TODO: Implement publish message
  res.send('Hello World!')
})
app.listen(port, () => {
  console.log(`Publisher app listening on port ${port}`)
})
/* consumer.js */
// TODO: implement consume messages

Step 2 – Add RabbitMq messaging code

Now let’s add the code we need to publish and consume messages on RabbitMq. for that we will need to install the amqplib library and run a local instance of RabbitMq

npm i amqplib
docker run -d --name rabbit rabbitmq:3-management

Now let’s add the code to our publisher:

/* publisher.js */
const amqplib = require('amqplib');
const express = require('express')
const app = express()
const port = process.env.PORT || 3000
let rabbitConnection;
/* An exchange is where the rabbitMq computation takes place. 
According to the messaging strategy defined by the exchange type, messages are sent to an exchange that distributes them to consumers. */
const exchange = 'logs'
const sendRabbitMqMessage = async (message) => {
  if (!rabbitConnection) {
    rabbitConnection = await amqplib.connect('amqp://localhost');
  }

  const channel = await rabbitConnection.createChannel();
/* Type "fanout" means sending the message to all consumers that subscribed to that exchange. */
  await channel.assertExchange(exchange , 'fanout')
/* Notice that we pass an empty string as the queue name. This means the queue will be defined per consumer. */
  await channel.publish(exchange, '', Buffer.from(message))
}
app.get('/', async (req, res) => {
  const message = 'Hello World!'
  console.log(`Send message: '${message}'`);
  await sendRabbitMqMessage(message);
  res.send(message)
})
app.listen(port, () => {
console.log(`${process.env.SERVICE} Running`)
})

Now let’s subscribe to messages by adding this code to the consumer:

/* consumer.js */
const amqplib = require('amqplib');
let rabbitConnection;
let exchange = 'logs'
const rabbitMqListenToMessages = async (callback) => {
  if (!rabbitConnection) {
      rabbitConnection = await amqplib.connect('amqp://localhost');
  }
  const channel = await rabbitConnection.createChannel();
  await channel.assertExchange(exchange, 'fanout')
  const q = await channel.assertQueue('');
  await channel.bindQueue(q.queue, exchange, '');
  await channel.consume(q.queue, (message) => callback(message.content.toString()), { noAck: true })
}
rabbitMqListenToMessages((message) => console.log(`Consumer received message: ${message}`))
console.log(`${process.env.SERVICE} Running`)

Let’s check if our messaging works properly. Run the application:

# terminal 1
SERVICE=publisher node ./publisher.js
> Publisher Running
# terminal 2
SERVICE=consumer-1 node ./consumer.js
> consumer-1 running
# We can also add another consumer just for fun
# terminal 2.1
SERVICE=consumer-2 node ./consumer.js
> consumer-2 running

After running the services we can open RabbitMQ management UI at http://localhost:15672/#/exchanges (username and password are both ‘guest’).

There we can see the exchange named ‘logs’ of the type fanout we created:

By navigating to the queues tab we can see the two queues that were created for the consumer services:

Now let’s call the get endpoint and trigger the flow of events:

# terminal 3
curl http://localhost:3000

Now we should see the following printed:

# on terminal 1
> Send message: 'Hello World!'
# on terminal 2 and 2.1
> Consumer received message: Hello World!

Congrats! We just wrote a system with services that can communicate with each other. The next step is to understand the workflow of the system by adding OpenTelemetry.

Step 3 – Instrument with Opentelemetry Node

So far, so good. We can now start examining our application behavior. For that, we will generate spans using amqplib instrumentation. Then view them in the console. Because we also use http to call on the publisher endpoint that is implemented using Express.js, Let’s add auto-instrumentations for this span as-well

Install the following packages:

npm install @opentelemetry/sdk-node @opentelemetry/instrumentation-amqplib @opentelemetry/instrumentation-http opentelemetry-instrumentation-express

Create a tracing.js file:

/* tracing.js */
// Require dependencies
const opentelemetry = require("@opentelemetry/sdk-node");
const { AmqplibInstrumentation } = require('@opentelemetry/instrumentation-amqplib');
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { ExpressInstrumentation } = require("opentelemetry-instrumentation-express");

const sdk = new opentelemetry.NodeSDK({
  traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
instrumentations: [
      new AmqplibInstrumentation(),
      new HttpInstrumentation(),
      new ExpressInstrumentation()
  ],
serviceName: process.env.SERVICE
});
sdk.start()

By running the application again and invoking the endpoint we can see the spans printed in the console:

# terminal 1
SERVICE=publisher node --require './tracing.js' ./publisher.js
# terminal 2
SERVICE=consumer node --require './tracing.js' ./consumer.js
# terminal 3
curl http://localhost:3000
# terminal 1 prints
> {
  traceId: 'fbbf2a11623d4230e712f6fd0c1d5912',
  parentId: undefined,
  name: 'HTTP GET',
  id: '72ed82cb81b3ee6d',
  kind: 1,
  timestamp: 1659068638640026,
  duration: 6384,
  attributes: {
    'http.url': 'http://localhost:3000/',
    'http.host': 'localhost:3000',
    'net.host.name': 'localhost',
    'http.method': 'GET',
    'http.target': '/',
    'http.user_agent': 'curl/7.79.1',
    'http.flavor': '1.1',
    'net.transport': 'ip_tcp',
    'net.host.ip': '::ffff:127.0.0.1',
    'net.host.port': 3000,
    'net.peer.ip': '::ffff:127.0.0.1',
    'net.peer.port': 54034,
    'http.status_code': 200,
    'http.status_text': 'OK'
  },
  status: { code: 0 },
  events: [],
  links: []
}
{
  traceId: 'fbbf2a11623d4230e712f6fd0c1d5912',
  parentId: '72ed82cb81b3ee6d',
  name: 'GET /',
  id: '802f16d65cc7f7a1',
  kind: 0,
  timestamp: 1659068638642200,
  duration: 4973,
  attributes: {
    'http.route': '',
    'express.route.configured': '',
    'express.route.params': '{}'
  },
  status: { code: 1 },
  events: [],
  links: []
}
{
  traceId: 'fbbf2a11623d4230e712f6fd0c1d5912',
  parentId: '72ed82cb81b3ee6d',
  name: 'logs ->  send',
  id: 'c78dc63639307344',
  kind: 3,
  timestamp: 1659068638659382,
  duration: 1086,
  attributes: {
    'messaging.protocol_version': '0.9.1',
    'messaging.url': 'amqp://localhost',
    'messaging.protocol': 'AMQP',
    'net.peer.name': 'localhost',
    'net.peer.port': 5672,
    'messaging.system': 'rabbitmq',
    'messaging.destination': 'logs',
    'messaging.destination_kind': 'topic',
    'messaging.rabbitmq.routing_key': ''
  },
  status: { code: 0 },
  events: [],
  links: []
}
# terminal 2 prints
> {
  traceId: 'fbbf2a11623d4230e712f6fd0c1d5912',
  parentId: 'c78dc63639307344',
  name: ' process',
  id: 'd69903f27d4c48b8',
  kind: 4,
  timestamp: 1659068638665048,
  duration: 729,
  attributes: {
    'messaging.protocol_version': '0.9.1',
    'messaging.url': 'amqp://localhost',
    'messaging.protocol': 'AMQP',
    'net.peer.name': 'localhost',
    'net.peer.port': 5672,
    'messaging.system': 'rabbitmq',
    'messaging.destination': 'logs',
    'messaging.destination_kind': 'topic',
    'messaging.rabbitmq.routing_key': '',
    'messaging.operation': 'process'
  },
  status: { code: 0 },
  events: [],
  links: []
}

At this point, we created spans and logged them to our console.

Visualization with OpenTelemety

Even though our spans look stunning in the console, this is not just about logging them but visualizing them. Our ability to visualize traces is where the true troubleshooting power of this technology comes into play.

For visualization, we’ll be using:

The open-source Jaeger Tracing
Aspecto

OpenTelemetry, RabbitMQ, and Jaeger Tracing

Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.

This is how it’s done:

Export to Jaeger

Run Jaeger locally with the following docker command

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  jaegertracing/all-in-one:1.30

We can view Jaeger by visiting http://localhost:16686.

As we can see, there are no traces to view yet. To see traces we need to add an Exporter to export our traces to Jaeger.

Install the following packages:

npm install @opentelemetry/exporter-jaeger @opentelemetry/sdk-trace-base

Edit our tracing.js file and add Jaeger exporter:

/* tracing.js */
const opentelemetry = require("@opentelemetry/sdk-node");
const { AmqplibInstrumentation } = require('@opentelemetry/instrumentation-amqplib');
const { SimpleSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { ExpressInstrumentation } = require("opentelemetry-instrumentation-express");
const exporter = new JaegerExporter();
const sdk = new opentelemetry.NodeSDK({
  spanProcessor: new SimpleSpanProcessor(exporter),
  instrumentations: [
      new AmqplibInstrumentation(),
      new HttpInstrumentation(),
      new ExpressInstrumentation()
  ],
  serviceName: process.env.SERVICE
});
sdk.start()

Now, let’s run the publisher and consumers services:

# terminal 1 - publisher
SERVICE=publisher node -r './tracing.js' publisher.js
> Publisher Running
# terminal 2 - consumer 1
SERVICE=consumer-1 node -r './tracing.js' ./consumer.js
> Consumer-1 Running
# terminal 3 - consumer 2
SERVICE=consumer-2 node -r './tracing.js' consumer.js
> Consumer-2 Running

Invoke the publisher’s endpoint:

# terminal 4
curl http://localhost:3000

Let’s see our traces in Jaeger UI. As you can see we now have 3 more services listed in the search input:

By selecting the publisher service and clicking ‘Find Traces’ we can see our trace with 3 spans created from the three services:

Clicking on the trace once more will show us the details of each span:

Advanced Visualization for OpenTelemetry Traces and RabbitMQ with Aspecto

Jaeger offers impressive visualization capabilities, so feel free to stop here if you’re satisfied.

However, you can take your tracing visualization to the next level with Aspecto. Try it yourself with the free-forever plan that has no limited features.

Sending traces to Aspecto takes a few minor modifications to the existing code. Give this Live Playground a try to get a better idea of what to expect.

Export to Aspecto

Here’s how it’s done:

Create a free account at www.aspecto.io or log in to your existing account
Install the following packages:

npm install @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto

Modify the tracing.js file

Make sure to replace the {ASPECTO_AUTH} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

/* tracing.js */
const opentelemetry = require("@opentelemetry/sdk-node");
const { AmqplibInstrumentation } = require('@opentelemetry/instrumentation-amqplib');
const { SimpleSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto');
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { ExpressInstrumentation } = require("opentelemetry-instrumentation-express");
const exporter = new OTLPTraceExporter({
  url: 'https://otelcol.aspecto.io/v1/traces',
  headers: {
      // Aspecto API-Key is required
      Authorization: process.env.ASPECTO_API_KEY
  }
})
const sdk = new opentelemetry.NodeSDK({
  spanProcessor: new SimpleSpanProcessor(exporter),
  instrumentations: [
      new AmqplibInstrumentation(),
      new HttpInstrumentation(),
      new ExpressInstrumentation()
  ],
  serviceName: process.env.SERVICE
});
sdk.start()

That’s it! Now run the application and invoke the endpoint once again:

# terminal 1 - publisher
SERVICE=publisher node -r './tracing.js' publisher.js
> Publisher Running
# terminal 2 - consumer 1
SERVICE=consumer-1 node -r './tracing.js' ./consumer.js
> Consumer-1 Running
# terminal 3 - consumer 2
SERVICE=consumer-2 node -r './tracing.js' consumer.js
> Consumer-2 Running

Invoke the publisher’s endpoint:

# terminal 4
curl http://localhost:3000

We can view our tracing on the Aspecto platform.

Drilling down to a specific trace, we can see a graph of the trace’s flow and a timeline, which makes it super convenient to understand the application’s workflow.

Quick note: the “gaps” you see in the timeline are where RabbitMQ was processing the message.

That’s about it for this OpenTelemetry Node with RabbitMQ guide, folks. If you have any questions or issues with any of these steps, feel free to reach out to us via chat or join our OpenTelemetry Slack channel (part of the CNCF Slack).

Final notes

If you’re interested, we also provide a simple way for wrapping all the instrumentations your node application needs with the Aspecto SDK. Simply import and invoke the following package at the beginning of your code (before all other imports).

require('@aspecto/opentelemetry')({
  aspectoAuth: process.env.ASPECTO_API_KEY
});

DEV Community: Yoav Danieli

OpenTelemetry Operator for Kubernetes: Practical Guide | Part 4

Good to know before reading this article:

What is a Kubernetes Operator?

What is OpenTelemetry Operator?

OpenTelemetry Operator: The Practical Part

Setting up our application

Collector Gateway with OpenTelemetry Operator

Collector Agent with OpenTelemetry Operator

Auto Instrumentation with OpenTelemetry Operator

OpenTelemetry Operator: Putting it all together

OpenTelemetry Collector on Kubernetes with Helm Chart – Part 3

Related Guides

What is Helm?

Using OpenTelemetry Helm Chart

Installing Opentelemetry-Collector Helm Chart

OpenTelemetry-Collector Helm Chart Configuration

Final Thoughts and Aspecto Helm Chart

OpenTelemetry Collector as an Agent on Kubernetes – Part 2

Related Guides

Why Deploy the OpenTelemetry Collector as an Agent

Kubernetes Resource Configuration for Collector as an Agent

Deploying the OpenTelemetry Collector Gateway + Agent locally

Final Thoughts

OpenTelemetry Collector on Kubernetes – Part 1

Next in this series

OpenTelemetry Collector on Kubernetes: Creating a Development Environment

Run OpenTelemerty Collector as a Gateway in Kubernetes cluster

Kubernetes Resource Configuration

Namespace

ConfigMap

Service

Deployment

Deploying the OpenTelemerty Collector Gateway locally

Final Thoughts and Next Steps

How to Instrument AWS Services with OpenTelemetry

What is OpenTelemetry

AWS + OpenTelemetry: Creating our application

Setup Disclaimer

AWS Lambda Setup

Important Notes

AWS SQS Setup

AWS DynamoDB

OpenTelemetry + AWS: External Service

Important Notes

Instrumenting AWS services with OpenTelemetry and exporting telemetry data

Important Configuration Notes

Visualizing OpenTelemetry Data in Aspecto

Important Notes

Running AWS Services and Visualizing OpenTelemetry Data in Aspecto

OpenTelemetry AWS SQS: Batch Receiving

Conclusion

Top 5 Jaeger Alternatives for Distributed Tracing

Top Jaeger Alternatives in 2022​

Aspecto

Notable features

Pricing

Product Overview

Honeycomb

Notable features

Pricing

Product Overview

Lightstep

Notable features

Pricing

Product Overview

SigNoz

Notable features

Pricing

Product Overview

Logz.io

Notable features

Pricing

Product Overview

Jaeger and why you may want alternatives

Getting Started with Aspecto

Guide to OpenTelemetry Distributed Tracing in Rust

What to Expect

What is OpenTelemetry?

Opentelemetry and Distributed Tracing in Rust

Top Jaeger Alternatives in 2022