DEV Community: Michael Haberman

How to Get Started with OpenTelemetry Go

Michael Haberman — Wed, 31 Aug 2022 07:28:19 +0000

In this guide, you will learn hands-on how to create and visualize traces with OpenTelemetry Go without prior knowledge.

We will start with creating a simple to-do app that uses Mongo and the Gin framework. Then, we will send tracing data to Jaeger Tracing and to Aspecto for visualization. You can find all the relevant files in this Github repository.

What to Expect

Intro to OpenTelemetry
Hello world: OpenTelemetry GO example
1. Create main.go file with Gin and Mongo
2. Install OpenTelemetry GO
3. Gin instrumentation: gin.Context
Visualization with Jaeger and Aspecto
1. OpenTelemetry Go and Jaeger Tracing
2. OpenTelemetry Go and Aspecto

Intro to OpenTelemetry

OpenTelemetry is a collection of APIs and SDKs that allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

It is a CNCF community-driven open-source project (Cloud Native Computing Foundation, the folks in charge of Kubernetes).

In a cloud-native environment, we use OpenTelemetry (OTel for short) to gather data from our system operations and events. In other words, to instrument our distributed services. This data enables us to understand and investigate our software’s behavior and troubleshoot performance issues and errors.

OpenTelemetry serves as a standard observability framework that captures all data under a single specification. It provides several components, including:

APIs and SDKs per programming language for generating telemetry
The OpenTelemetry Collector; receives, processes, and exports telemetry data to different destinations.
OTLP protocol for shipping telemetry data

To get a deeper understanding of this technology, including its structure and the primary motivation, visit this guide.

For this OpenTelemetry Golang guide, here are the terms you need to know:

Span: The most basic unit. A span represents an event in our system (e.g., an HTTP request or a database operation that spans over time). A span would usually be the parent of another span, its child, or both.
Trace: ‘Call-stacks’ for distributed services. Traces represent a tree of spans connected in a child/parent relationship. Traces specify the progression of requests across different services and components in our app (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.
Exporter: Once we create a span, the exporter handles sending the data to our backend (e.g., in memory, Jaeger Tracing, or console output)
Context propagation – The mechanism that allows us to correlate events across distributed services. Context is referred to as the metadata we collect and transfer*. Propagation* is how the context is packaged and transferred across services, often via HTTP headers. Context propagation is one of the areas where OpenTelemetry shines.
Instrumentation – instrumentation libraries gather the data and generate spans based on different libraries in our applications (Kafka, Mongo, Gin, etc.).

There are two ways to instrument our app – manually and automatically:

Auto instrumentation: Automatically create spans from the application libraries we use with ready-to-use OpenTelemetry libraries.
Manual instrumentation: Manually add code to your application to define the beginning and end of each span and the payload.

To understand more of the OpenTelemetry jargon, visit the official documentation.

Hello world: OpenTelemetry Go example

We will start by creating our to-do service and installing two libraries (Gin and Mongo) to understand how instrumentations work.

Step 1: Create main.go file for our to-do app

1) Install Gin and Mongo-driver

go get -u github.com/gin-gonic/gin
go get go.mongodb.org/mongo-driver/mongo

2) Set up gin and mongo to listen on “/todo”

3) Create some to-do’s to seed Mongo

package main
import (
  "context"
  "net/http"
  "github.com/gin-gonic/gin"
  "go.mongodb.org/mongo-driver/bson"
  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
)
var client *mongo.Client
func main() {
  connectMongo()
  setupWebServer()
}
func connectMongo() {
  opts := options.Client()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
func setupWebServer() {
  r := gin.Default()
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //Important: Make sure to pass c.Request.Context() as the context and not c itself - TBD
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}

Now that our small todo app is ready, let’s introduce OpenTelemetry.

Step 2: Install OpenTelemetry Go

We will be configuring OpenTelemetry to instrument our Go app.

1) To install the OTel SDK, run:

go get go.opentelemetry.io/otel /
go.opentelemetry.io/otel/sdk /

2) Instrument our Gin and Mongo libraries to generate traces.

3) Gin instrumentation: Install otelgin

go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

4) Mongo instrumentation: Install otelmongo

go get go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo

Gin instrumentation: gin.Context

We previously discussed the idea of context propagation – the way to transfer metadata between distributed services to correlate events in our system.

The Gin framework has its own type gin.Context which gets passed as a parameter to an HTTP handler. However, the context that should be passed down to the mongo operations is the standard Go library Context object, available in gin.Context.Request.Context.

//Make sure to pass c.Request.Context() as the context and not c itself
cur, findErr := collection.Find(c.Request.Context(), bson.D{})

So make sure that you pass the Context to the mongodb operation. Check out this issue for more info.

We now have our todo app ready and instrumented. It’s time to utilize OpenTelemetry to its full potential. Our ability to visualize traces is where the true troubleshooting power of this technology comes into play.

For visualization, we’ll be using the open-source Jaeger Tracing and Aspecto.

Visualization with Jaeger and Aspecto

The setup to export traces to Jaeger or Aspecto is relatively similar. Follow along with the Jaeger setup, then switch to Aspecto by changing a few lines of code.

OpenTelemetry Go and Jaeger Tracing: Export traces to Jaeger

Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.

Here’s what the setup looks like:

1) Install the Jaeger exporter

go get go.opentelemetry.io/otel/exporters/jaeger

2) Create a tracing folder and a jaeger.go file

3) Add the following code to the file

package tracing
import (
  "go.opentelemetry.io/otel/exporters/jaeger"
  "go.opentelemetry.io/otel/sdk/resource"
  sdktrace "go.opentelemetry.io/otel/sdk/trace"
  semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)
func JaegerTraceProvider() (*sdktrace.TracerProvider, error) {
  exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint("http://localhost:14268/api/traces")))
  if err != nil {
      return nil, err
  }
  tp := sdktrace.NewTracerProvider(
      sdktrace.WithBatcher(exp),
      sdktrace.WithResource(resource.NewWithAttributes(
          semconv.SchemaURL,
          semconv.ServiceNameKey.String("todo-service"),
          semconv.DeploymentEnvironmentKey.String("production"),
      )),
  )
  return tp, nil
}

4) Go back to the main.go file and modify our code to use the JaegerTraceProvider function we just created

func main() {
  tp, tpErr := tracing.JaegerTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  setupWebServer()
}

Next, we are going to hook up the instrumentations we installed.

5) Add the Mongo instrumentation. In our connectMongo function by adding this line

opts.Monitor = otelmongo.NewMonitor()

The function shold look like this

func connectMongo() {
  opts := options.Client()
  //Mongo OpenTelemetry instrumentation
  opts.Monitor = otelmongo.NewMonitor()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with some todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}

Now, add the Gin instrumentation.

6) Go to the startWebServer function and add this line right after we create the gin instance

r.Use(otelgin.Middleware("todo-service"))

The function should look like this

func startWebServer() {
  r := gin.Default()
  //Gin OpenTelemetry instrumentation
  r.Use(otelgin.Middleware("todo-service"))
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //make sure to pass c.Request.Context() as the context and not c itself
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}

Below is the complete main.go file. Now we’re finally ready to export to Jaeger.

package main
import (
  "context"
  "log"
  "net/http"
  "github.com/aspecto-io/opentelemerty-examples/tracing"
  "github.com/gin-gonic/gin"
  "go.mongodb.org/mongo-driver/bson"
  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
  "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
  "go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo"
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/propagation"
)
var client *mongo.Client
func main() {   //Export traces to Jaeger
  tp, tpErr := tracing.JaegerTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  startWebServer()
}
func connectMongo() {
  opts := options.Client()
  //Mongo OpenTelemetry instrumentation
  opts.Monitor = otelmongo.NewMonitor()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with some todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
func startWebServer() {
  r := gin.Default()
  //gin OpenTelemetry instrumentation
  r.Use(otelgin.Middleware("todo-service"))
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //Make sure to pass c.Request.Context() as the context and not c itself
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}

Export traces to Jaeger

Run the todo-service with go run main.go
Make an HTTP GET request to localhost:8080/todo to generate some traces in Go
Open Jaeger at http://localhost:16686/search to view those traces

You can now see the Jaeger UI. Select todo-service and click on Find traces. You should see your trace on the right:

By clicking the trace, you can drill down and see more details about it that allow you to further investigate on your own:

Visualization with OpenTelemetry Go and Aspecto

So now you know the basic concepts of OpenTelemetry. You used it on your own to instrument your Go libraries, create traces and export the data to Jaeger for visualization.

Let’s take our visualization capabilities to the next level with Aspecto.

Exporting and visualizing our data in Aspecto is easy (we will see that in a moment). You can try it yourself with the free-forever plan that has no limited features. Give this Live Playground a try to get a better idea.

This is how we do it:

Create a new free account at www.aspecto.io
Install the otlp exporter by running

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

Create a file called aspecto.go in your tracing folder and add the code below
Replace the {ASPECTO_AUTH} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

func AspectoTraceProvider() (*sdktrace.TracerProvider, error) {
  exp, err := otlptracegrpc.New(context.Background(),
      otlptracegrpc.WithEndpoint("collector.aspecto.io:4317"),
      otlptracegrpc.WithHeaders(map[string]string{
          "Authorization": "<ADD YOUR TOKEN HERE>",
      }))
  if err != nil {
      return nil, err
  }
  tp := sdktrace.NewTracerProvider(
      sdktrace.WithBatcher(exp),
      sdktrace.WithResource(resource.NewWithAttributes(
          semconv.SchemaURL,
          semconv.ServiceNameKey.String("todo-service"),
          semconv.DeploymentEnvironmentKey.String("production"),
      )),
  )
  return tp, nil
}

Back in the main.go file, replace the JaegerTraceProvider with AspectoTraceProvider.

The main function should now look like this

func main() {   //Export traces to Aspecto
  tp, tpErr := tracing.AspectoTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  startWebServer()}

Exporting traces to Aspecto

Run the todo-service with go run main.go
Make an HTTP GET request to localhost:8080/todo to generate some traces
Navigate to app.aspecto.io to view the traces

We also sent a few traces with errors to see what it looks like. Our traces should look something like this:

In the Trace Search view, using the left filters pane, simply filter out by the service name (you can also filter by error, HTTP method, and much more).

If we drill down into one of these traces, we can see in more detail how long each request took and clear visualization of the entire workflow.

In more complex workflows, traces would look something like the below:

That’s all folks! We hope this guide was informative and easy to follow. You can find all files ready to use in our Github repository.

If you have any questions or issues, feel free to reach out to us via our chat.

To learn more about it, check out our OpenTelemetry Bootcamp. It’s a 6 episode YouTube series that covers OpenTelemetry from zero to 100 including the implementation of opentelemetry in production, security, sampling, and more. Completely free and vendor-neutral.

Follow these guides below, if you want to install OpenTelemetry in more officially supported languages:

OpenTelemetry Java: Getting Started Guide

Michael Haberman — Mon, 29 Aug 2022 06:39:16 +0000

This is a practical guide that brings you just what you need to get started with OpenTelemetry Java. No prior OpenTelemetry knowledge is needed – we will cover the basics here.

For this OpenTelemetry Java example, we’ll be using the Spring Pet Clinic project, to save you some time setting up an example project.

What to Expect

Intro to OpenTelemetry
Hello World: OpenTelemetry Java
Visualization with Jaeger Tracing
Advanced Visualization with Aspecto
Keep Learning: Mastering OpenTelemetry

What is OpenTelemetry?

OpenTelemetry is a collection of SDKs and APIs – an open-source project – that allows us to collect, generate, and export logs, metrics, and traces (also known as the three pillars of observability).

OpenTelemetry enables us to instrument our distributed services. Instrumenting means capturing telemetry data from events and operations in our distributed system.

Ultimately, we use this data to understand and investigate our system’s behavior and troubleshoot and debug performance issues and errors.

Led by the CNCF (Cloud Native Computing Foundation, the foundation responsible for Kubernetes), it serves as a single library that gathers data under a single specification.

It allows us to be vendor-agnostic and not tied down to a single tool and ship our data to any dedicated location (e.g., backend, OpenTelemetry collector, supporting open sources, and more).

Since its implementation for most modern programming languages, it has been growing in popularity and usage by developers everywhere. It combines all three aspects needed for proper monitoring (leading with tracing, followed by metrics).

There is much more to learn about OpenTelemetry and if you want to dive deeper into its structure, follow this guide. For this guide, here are the key terms you need to know:

Span: A span represents an action/operation that occurred in our system. An HTTP request or a database operation that spans over time. A span would usually be the parent and/or the child of another span.
Trace: Represents a tree of spans connected in a child/parent relationship. Traces describe the progression of requests across different services and components in our system (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.

Instrumentation – instrumentation libraries are what allow us to gather the data and create spans based on the different libraries in our system such as Kafka, MySQL, Spring, etc. There are 2 ways to instrument our app – manually or automatically:
- Auto instrumentation: Automatically create spans from the application libraries we use with ready-to-use OpenTelemetry libraries.
- Manual instrumentation: Manually create spans by adding code to your application to define the beginning and end of each span.
Exporter: Once we create a span, we need to send it to a dedicated backend. It may be in memory, console output, vendor, or an open-source such as Jaeger Tracing. The exporter handles sending the data to our backend.

For more OpenTelemetry terminology, visit the official documentation.

Hello World: OpenTelemetry Java

Step 1: Download the latest OpenTelemetry Java agent

Download the latest Java agent ‘JAR’ from the official repo on GitHub and copy the opentelemetry-javaagent-all.jar file to your project.

Note: Make sure you cloned the Spring Pet Clinic project we mentioned before.

Step 2: Logging OpenTelemetry spans to console

1) Run the following commands in your terminal

We are setting up our service name my-service
Set the exporter to log spans to our console OTEL_TRACES_EXPORTER=logging

OTEL_SERVICE_NAME=my-service OTEL_TRACES_EXPORTER=logging java -javaagent:./opentelemetry-javaagent.jar -jar target/*.jar

We mentioned instrumentations earlier, the libraries that actually allow us to gather data from our apps. The above command will automatically instrument most common libraries. You can view the complete list here.

In terms of implementing OpenTelemetry Java instrumentation in our project to collect traces, we’re pretty much done (keep reading, there’s more to it).

Now, the final output should look something like the one below.

//Console output

INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - '/owners/{ownerId}' :
99af87eeaa19b83d014463b046884e56 632e2d2a5931bd17 SERVER [tracer: io.opentelemetry.tomcat-7.0:1.13.1-
alpha] AttributesMap{data={net.transport=ip_tcp, http.target=/owners/11, thread.id=120, http.flavor=1.1,
http.status_code=200, net.peer.ip=0:0:0:0:0:0:0:1, thread.name=http-nio-8080-exec-10,
http.host=localhost:8080, http.route=/owners/{ownerId}, http.user_agent=Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36,
http.method=GET, net.peer.port=50778, http.scheme=http}, capacity=128, totalAddedValues=13}

Note: This is not the best-looking console output 😬 We thought about making it prettier, however, this is the way it looks in the console so we decided to stick to the source to prevent any confusion.

If you look at the JSON, you can see that, for example, http.route=/owners/{ownerId} represents an https call to fetch an owner by ID.

At this point, we created spans and log them to our console. But we’re not here only to have beautiful spans in our console – it’s all about visualization.

Our ability to visualize trace data is where the true troubleshooting power of this technology comes into play.

For visualization, we’ll be using the open-source Jaeger Tracing and Aspecto.

Getting Started with OpenTelemetry Java and Jaeger Tracing

Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.

This is how we send traces to Jaeger:

Export to Jaeger

1) Run Jaeger locally with the following docker command

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.32

2) Previously, we set the exporter to log to our console. Now, we need to make the following changes and run this command:

Change the exporter to Jaeger’s OTEL_TRACES_EXPORTER=jaeger
Specify the Jaeger’s endpoint OTEL_EXPORTER_JAEGER_ENDPOINT

OTEL_SERVICE_NAME=my-service OTEL_TRACES_EXPORTER=jaeger OTEL_EXPORTER_JAEGER_ENDPOINT=http://localhost:14250 java -javaagent:./opentelemetry-javaagent.jar -jar target/*.jar

3) Run your Pet Clinic project and execute a few actions. For example, we created a new owner, then searched the owner by the last name.

4) Use your browser to view Jaeger UI at http://localhost:16686/

5) We can now view our trace in the Jaeger UI. Select ‘my-service’ from the search pane on the right and click on Find Traces.

You can drill down into a specific trace. For example, a request to search the owner by owner ID.

Advanced Visualization with OpenTelemetry Java and Aspecto

So now you know the basics of instrumentations, spans, traces, and how we can use OpenTelemetry to generate traces for our Java app.

You can take your tracing visualization to the next level with Aspecto.

Sending traces to Aspecto is as easy as what we were doing so far. Before jumping in, you can try it yourself with the free-forever plan that has no limited features.

Give this Live Playground a try to get a better idea.

Export to Aspecto

Here’s how it’s done:

1) Create a free account at www.aspecto.io or log in to your existing account

2) Previously, we set the exporter to send traces to Jaeger. Now, we need to make the following changes and run these commands:

Change the exporter to OTEL_EXPORTER_OTLP_HEADERS
Specify the Aspecto endpoint OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
Replace the {ASPECTO_AUTH} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)

OTEL_SERVICE_NAME=my-service 
OTEL_EXPORTER_OTLP_HEADERS=Authorization={ASPECTO_AUTH}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 java -javaagent:./opentelemetry-javaagent.jar -jar target/*.jar

3) Execute a few actions in your project. In your Aspecto account, the main Trace Search view should look something like this:

Using the left Filters pane, we can quickly find any error we have in our system (this error was embedded in the Pet Clinic project by default):

Then, drill down into this trace to investigate and troubleshoot the root cause:

We can also filter traces out by HTTP route name. For example, here we’re filtering out only the http.route=/owners/{ownerId} we mentioned before:

Our ability to quickly solve issues and fix errors in production relies heavily on how we visualize traces and the ease with which we can filter out and deep dive into our data.

That’s about it for this OpenTelemetry Java guide, folks. If you have any questions or issues with any of these steps, feel free to reach out to us via chat or join our OpenTelemetry Slack channel (part of the CNCF Slack).

Mastering OpenTelemetry

To keep your OpenTelemetry journey going, we created a 6-episode video series that brings you everything you need to know about OpenTelemetry. From zero to mastering, it contains everything from the very basics to advanced concepts such as sampling, security, and scaling for production.

The OpenTelemetry Bootcamp – free and vendor-neutral.

Use this as your OpenTelemetry playbook:

Episode 1: OpenTelemetry Fundamentals
Episode 2: Integrate Your Code (logs, metrics, and traces)
Episode 3: Deploy to Production + Collector
Episode 4: Sampling and Dealing with High Volumes
Episode 5: Custom Instrumentation
Episode 6: Testing with OpenTelemetry

Jaeger Tracing: The Ultimate Guide

Michael Haberman — Tue, 23 Aug 2022 11:05:16 +0000

In this guide, you’ll learn what Jaeger tracing is, what distributed tracing is, and how to set it up in your system. We’ll go over Jaeger’s UI and touch on advanced concepts such as sampling and deploying in production.

You’ll leave this guide knowing how to create spans with OpenTelemetry and send them to Jaeger tracing for visualization. All that, from scratch.

Jaeger Tracing: Contents

What is Distributed Tracing? Introduction
What is Jaeger Tracing?
Jaeger Tracing Architecture
Running Jaeger locally using Docker
Jaeger Tracing and OpenTelemetry
Jaeger Tracing Python Example
Jaeger Tracing UI Review
Jaeger Tracing Advanced Concepts
Jaeger Tracing Production Deployment
The Costs of Running Jaeger
Jaeger Tracing Glossary

What is Distributed Tracing? Introduction

Before we dive into explaining everything you need to know from 0 to 100 about Jaeger tracing, it’s important to understand the umbrella term that Jaeger is part of – distributed tracing.

In the world of microservices, most issues occur due to networking issues and the relations between the different microservices.

A distributed architecture (as opposed to a monolith) makes it a lot harder to get to the root of an issue.

To resolve these issues, we need to see which service sent what parameters to another service or a component (a DB, queue, etc.).

Distributed tracing helps us achieve just that by enabling us to collect data from the different parts of our system, to enable this desired observability into our system. You can think of it as ‘call-stacks’ for distributed services.

To learn more about the benefits of distributed tracing vs. logs, read this quick guide.

In addition, traces are a visual tool, allowing us to visualize our system to better understand the relationships between services, making it easier to investigate and pinpoint issues.

Distributed tracing visualization with Aspecto

What is Jaeger Tracing?

Now that you know what distributed tracing is, we can safely talk about Jaeger.

Jaeger is an open-source distributed tracing system created by Uber back in 2015.

The Jaeger data model is compatible with OpenTracing – which is a specification that defines how the collected tracing data would look, as well as libraries of implementations in different languages.

(More on OpenTracing and OpenTelemetry later)

As in most distributed tracing systems, Jaeger works with spans and traces, as defined in the OpenTracing specification.

A span represents an action (HTTP request, call to a DB, etc) and is Jaeger’s most basic unit of work. A span must have an operation name, start time, and duration.

A trace is a collection/list of spans connected in a child/parent relationship (and can also be thought of as a directed acyclic graph of spans). Traces specify how requests are propagated through our services and other components.

Jaeger Tracing Architecture

Here’s what Jaeger architecture looks like

It consists of a few parts, all of which I explain below:

Jaeger client: The Jaeger clients are libraries written in various programming languages, and are responsible for span creation. Note: they are being deprecated in favor of OpenTelemetry. Again, more on that later.
Jaeger Agent: Jaeger agent is a network daemon that listens for spans received from the Jaeger client over UDP. It gathers batches of them and then sends them together to the collector.
Jaeger Collector: The Jaeger collector is responsible for receiving traces from the Jaeger agent and performs validations and transformations. After that, it saves them to the selected storage backends.
Storage Backends: Jaeger supports various storage backends – which store the spans & traces for later retrieving them. Supported storage backends are In-Memory, Cassandra, Elasticsearch, and Kafka (only as a buffer to another storage like Cassandra or elasticsearch).
Jaeger Query: This is a service responsible for retrieving traces from the jaeger storage backend and making them accessible for the jaeger UI.
Jaeger UI: a React application that lets you visualize the traces and analyze them. Useful for debugging system issues.
Ingester: Ingester is relevant only if we use Kafka as a buffer between the collector and the storage backend. It is responsible for receiving data from Kafka and ingesting it into the storage backend. More info can be found in the official Jaeger Tracing docs.

Running Jaeger locally using Docker

Jaeger comes with a ready-to-use all-in-one docker image that contains all the components necessary for Jaeger to run.

It’s really simple to get it up and running on your local machine:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.30

Then you can simply open the jaeger UI on http://localhost:16686.

Below you’ll learn how to send data to it using Python and OpenTelemetry.

Jaeger Tracing and OpenTelemetry

Yes, you’re right. I did mention before that Jaeger’s data model is compatible with the OpenTracing specification.

You may already know that OpenTracing and OpenCensus have merged to form OpenTelemetry and are wondering why does Jaeger use OpenTracing and if you can use OpenTelemetry to report to jaeger instead.

As to why Jaeger uses OpenTracing – well, the reason is that Jaeger exists from before the above-mentioned merger.

To get a full understanding of OpenTelemetry, what is it, its components, and how you can use with, read this comprehensive guide.

Deprecation of Jaeger Client in favor of OpenTelemetry Distro:

I also mentioned that Jaeger clients are now deprecating.

You can find more info about this deprecation here, but essentially the idea is that you should now use the OpenTelemetry SDK in the programming language of your choice, alongside a Jaeger exporter.

This way created spans would be converted to a format Jaeger knows how to work with, passing all the way through to the Jaeger collector and then to the storage backend.

At the time of writing this, the OpenTelemetry collector is not considered a replacement for the Jaeger collector [1]. In the future, the idea is that the OpenTelemetry collector would be used instead of the Jaeger collector that would be deprecated [2].

If you can’t wait and want to try using the OpenTelemetry collector with Jaeger now – see this guide.

Jaeger Tracing Python Example

Here is a Python example of creating spans and sending them to Jaeger. Note that you could also use automatic instrumentations and still use the Jaeger exporter (assuming you’re running Jaeger locally like shown above):

# jaeger_tracing.py
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
trace.set_tracer_provider(
   TracerProvider(
       resource=Resource.create({SERVICE_NAME: "my-hello-service"})
   )
)
jaeger_exporter = JaegerExporter(
   agent_host_name="localhost",
   agent_port=6831,
)
trace.get_tracer_provider().add_span_processor(
   BatchSpanProcessor(jaeger_exporter)
)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("rootSpan"):
   with tracer.start_as_current_span("childSpan"):
           print("Hello world!")

This is what they would look like in the Jaeger UI:

To learn how to hands-on use OpenTelemetry in Python from scratch, read this guide.

Jaeger Tracing UI Review

The jaeger UI is a powerful tool for us to debug and understand our distributed services better.

Here’s what you need to know about it:

The search pane:

You can use the search pane to search for traces with specific properties.

Which service they come from, what operation was made, specific tags that were included within the trace (for example, the http status code), how long in the past to look for and result amount limiting.

When you’re done defining your search in this pane, click on Find Traces.

The search results section:

In this example, I chose to query the jaeger-query service. I can see my traces on a timeline or as a list. Click on the desired trace to drill down into it.

The specific trace view:

When you find a specific trace where you think there might be an issue and click on it, you’d see a screen that looks like this:

Here you can find specific information about execution times, which calls were made and their durations, specific properties like http status code, route path (in the case of an http call), and more.

Feel free to play around and investigate for yourself with your actual data.

Jaeger Tracing Advanced Concepts

Jaeger Sampling

Sampling is a complex aspect by itself.

You should know that there are 2 places where you can make the sampling decision: at the client code (at the distro level – head-based sampling), and at the collector level (tail-based sampling).

Sampling at the Distro (SDK) level

The deprecating Jaeger client has 4 sampling modes:

Remote: the default, and is used to tell the Jaeger client that sampling is taken from the Jaeger backend. More on this when we talk about tail based sampling.
Constant: either take all traces or take none. Nothing in between. Receives 1 for all and 0 for none
Rate limiting: choose how many traces would be sampled per second.
Probabilistic: choose a percentage of traces that would be sampled, for example – choose 0.1 to have 1 of each 10 traces to be sampled.

Collector level sampling

If we choose to enable sampling at the Jaeger collector, Jaeger supports 2 modes: file sampling and adaptive sampling.

File sampling – you specify a configuration file path to the collector that contains the per-service and per-operation sampling configuration.

Adaptive sampling – let Jaeger learn the amount of traffic each endpoint receives and calculate the most appropriate rate for that endpoint. Note that at the time of writing only Memory and Cassandra backends support this.

More info on Jaeger sampling can be found here: https://www.jaegertracing.io/docs/1.30/sampling/

Jaeger Tracing Production Deployment

All-in-one or separate containers ?

Jaeger all-in-one is a pre-built Docker image containing all the Jaeger components needed to get up and running quickly with Jaeger tracing by launching a single command.

A lot of people (including my past self) ask themselves what’s the correct way to launch Jaeger in production. If it’s safe to use Jaeger all-in-one in production, etc.

While at the time of writing I could not find any official answer to use or not to use it, I think the right answer is – you could, but you probably shouldn’t.

Using it as in production means you have a single source of failure which is not distributed.

Theoretically, an issue even with the Jaeger UI might crush the entire container and you wouldn’t be able to receive critical spans created by your system.

The best way to go about this would be to run each image separately, without the all-in-one.

The Costs of Running Jaeger (and why you may want an alternative)

As you can probably tell by now, Jaeger is a powerful but complex beast.

Deploying and maintaining it is time-consuming and can be costly as it involves storage, compute and would usually need constant uptime for a production use case.

Of course, there is no amount I could measure now and claim it fits all, as it varies depending on your cloud provider and infrastructure choices.

However, you may be glad to know you don’t have to do all this on your own.

Aspecto has a free-forever tier and provides everything included in Jaeger and more. Sort of like Jaeger on steroids.

Feel free to try the playground yourself and sign up for the free forever plan.

A quick look into Aspecto

If you have any questions, feel free to reach out to me on Twitter @thetomzach, and to join our #OpenTelemetry-Bootcamp slack channel to be on top of what’s happening in observability.

Jaeger Tracing Glossary

Span – a description of an action/operation that occurs in our system; an HTTP request or a database operation that spans over time (start at X and has a duration of Y milliseconds). Usually, it will be the parent and/or child of another span.

Trace – a tree/list of spans representing the progression of requests as it is handled by the different services and components in our system. For example, sending an API call to user-service resulted in a DB query to users-db. They are ‘call-stacks’ for distributed services.

Observability – a measure of how well we can understand the internal states of a system based on its external outputs. When you have logs, metrics, and traces you have the “3 pillars of observability”.

OpenTelemetry – OpenTelemetry is an open-source tool. A collection of tools, APIs, and SDKs, led by the CNCF (Cloud Native Computing Function). OpenTelemetry enables the automated collection and generation of traces, logs, and metrics with a single specification.

OpenTracing – an open-source project for distributed tracing. It was deprecated and “merged” into OpenTelemetry. OpenTelemetry offers backward compatibility for OpenTracing.

Get Started with OpenTelemetry Node: A Practical Guide for Devs

Michael Haberman — Sun, 24 Apr 2022 13:58:20 +0000

In this guide, you will learn exactly how to get started with OpenTelemetry in Node, from scratch, and without any prior knowledge of OpenTelemetry.

You will set up a simple to-do app that uses Express and MongoDB, then use OpenTelemetry to generate spans, and send them to Jaeger and Aspecto for visualization.

Intro to OpenTelemetry

OpenTelemetry is an open-source project, a collection of APIs and SDKs that allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

Led by the CNCF (Cloud Native Computing Foundation, the folks responsible for Kubernetes), we use OpenTelemetry to gather data from operations and events happening in our system. In other words, it enables us to instrument our distributed services.

This data ultimately helps us understand and investigate our software’s behavior and troubleshoot any performance issues and errors.

OpenTelemetry serves as a single library that captures all this information under a single specification and ships it to any dedicated location (e.g., backend, collector, supporting open sources, and more).

We covered OpenTelemetry in multiple posts so if you want to get a deeper understanding of the technology, including its structure, visit this guide.

But for this OpenTelemetry in node guide, here are the terms you need to know:

Span: A span represents an action/operation that occurred in our system. An HTTP request or a database operation that spans over time (starts at X and has a duration of Y milliseconds). A span would usually be the parent and/or the child of another span.
Trace: ‘Call-stacks’ for distributed services. Traces represent a tree of spans connected in a child/parent relationship. Traces specify the progression of requests across different services and components in our app (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.

Exporter: Once we create a span, we need to send it to a dedicated backend. It may be in memory, Jaeger Tracing, or even as console output. The exporter handles sending the data to our backend.
Instrumentation – instrumentation libraries are what allow us to gather the data and generate spans based on different libraries in our applications such as Kafka, Mongo, Express, etc. There are two ways to instrument our app – manually or automatically:
1) Auto instrumentation: Automatically create spans from the application libraries we use with ready-to-use OpenTelemetry libraries.
2) Manual instrumentation: Manually add code to your application to define the beginning and end of each span and the payload.

To understand more of the OpenTelemetry jargon, visit the official documentation.

Hello World: OpenTelemetry Node

Step 1: Create the tracing.js File to generate spans

We’ll start by installing the relevant packages and then create a tracing.js file, which contains everything you need to generate spans using OpenTelemetry.

1) Start with a new Node project (Node v10+ is supported by OpenTelemetry see here).

2) Install the following packages. Since our to-do app uses Express, HTTP, and MongoDB, notice we’re installing the instrumentations – @opentelemetry/instrumentation – for all three libraries.

npm install --save @opentelemetry/api
npm install --save @opentelemetry/sdk-trace-node
npm install --save opentelemetry-instrumentation-express
npm install --save @opentelemetry/instrumentation-mongodb
npm install --save @opentelemetry/instrumentation-http

npm install --save express
npm install --save mongodb

3) Create a tracing.js file with the following code:

// tracing.js


//OpenTelemetry
const { Resource } = require("@opentelemetry/resources");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");
const { ConsoleSpanExporter } = require('@opentelemetry/sdk-trace-base');
const { SimpleSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { trace } = require("@opentelemetry/api");

//instrumentations
const { ExpressInstrumentation } = require("opentelemetry-instrumentation-express");
const { MongoDBInstrumentation } = require("@opentelemetry/instrumentation-mongodb");
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");


//Exporter
module.exports = (serviceName) => {
  const exporter = new ConsoleSpanExporter();

  const provider = new NodeTracerProvider({
    resource: new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
    }),
  });
  provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

  provider.register();

  registerInstrumentations({
    instrumentations: [
      new HttpInstrumentation(),
      new ExpressInstrumentation(),
      new MongoDBInstrumentation(),
    ],
    tracerProvider: provider,
  });

  return trace.getTracer(serviceName);
};

Step 2: Create our to-do service

1) Create a file named index.js

2) Add the following code to your index.js file (we also created 3 new tasks so our to-do list isn’t empty):

const tracer = require("./tracing")("todo-service");
const express = require("express");
const { MongoClient } = require("mongodb");

const app = express();
app.use(express.json());
const port = 3000;
let db;
app.get("/todo", async (req, res) => {
  const todos = await db.collection("todos").find({}).toArray();
  res.send(todos);
});

app.get("/todo/:id", async (req, res) => {
  const todo = await db
    .collection("todos")
    .findOne({ id: req.params.id });
  res.send(todo);
});

const startServer = () => {
  MongoClient.connect("mongodb://localhost:27017", (err, client) => {
    db = client.db("todo");
db.collection("todos").insertMany([
     { id: "1", title: "Buy groceries" },
     { id: "2", title: "Install Aspecto" },
     { id: "3", title: "buy my own name domain" },
   ]);
 });
  });

  app.listen(port, () => {
    console.log(`Example app listening on port ${port}`);
  });
};
startServer();

3) Run Mongo

docker run -d -p 27017:27017 mongo

4) Execute a request to the todo service

curl http://localhost:3000/todo

At this point, we created spans and log them to our console. Now, even though our spans look absolutely gorgeous in the console, we’re here not only to log them but rather to visualize them.

Our ability to visualize traces is where the true troubleshooting power of this technology comes into play.

For visualization, we’ll be using the open-source Jaeger Tracing and Aspecto.

Getting Started with OpenTelemetry Node and Jaeger Tracing

Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.

This is how it’s done:

Export to Jaeger

1) Run Jaeger locally with the following docker command

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.32

2) Install the Jaeger exporter in your node project

npm install --save @opentelemetry/exporter-jaeger

To send the spans to Jaeger, in our tracing.js file, we’d use the OpenTelemetry JaegerExporter instead of the ConsoleSpanExporter we used before.

3) We’re going to modify your tracing.js file to use the Jaeger Exporter:

Add const { JaegerExporter } = require(“@opentelemetry/exporter-jaeger”);
Replace your exporter with the new imported JaegerExporter

//OpenTelemetry
const { Resource } = require("@opentelemetry/resources");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");
const { SimpleSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { trace } = require("@opentelemetry/api");

//exporter
const { JaegerExporter } = require("@opentelemetry/exporter-jaeger");

//instrumentations
const { ExpressInstrumentation } = require("opentelemetry-instrumentation-express");
const { MongoDBInstrumentation } = require("@opentelemetry/instrumentation-mongodb");
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");


//Exporter
module.exports = (serviceName) => {
 const exporter = new JaegerExporter({
   endpoint: "http://localhost:14268/api/traces",
 });

 const provider = new NodeTracerProvider({
   resource: new Resource({
     [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
   }),
 });
 provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

 provider.register();

 registerInstrumentations({
   instrumentations: [
     new HttpInstrumentation(),
     new ExpressInstrumentation(),
     new MongoDBInstrumentation(),
   ],
   tracerProvider: provider,
 });

 return trace.getTracer(serviceName);
};

4) Run your application and execute a few requests

5) Use your browser to view Jaeger UI at http://localhost:16686/

6) We can now view our trace in the Jaeger UI. Select our to-do service from the search pane on the right and click on Find Traces.

You can see more details about it (e.g., how long each operation took) which you can further investigate on your own:

Visualization with OpenTelemetry Node and Aspecto

So now you know the basics of spans, traces, instrumentations, and how we can use OpenTelemetry to create traces for code written in Node.

You can take your visualization capabilities from Jaeger to the next level using Aspecto.

Visualizing our data in Aspecto is super easy (we’ll see that in a moment), and you can try it yourself with the free-forever plan that has no limited features. Give this Live Playground a try to get a better idea.

Export to Aspecto

Here’s how to do it:

1) Create a new free account at www.aspecto.io

2) We’ll make two modifications to our tracing.js file (the complete file is below):

Install the following packages:

npm install --save @opentelemetry/exporter-trace-otlp-http

Replace your existing exporter import with the exporter-trace-otlp-http:

const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");

3) Set the environment variable ASPECTO_API_KEY to the token you got from Aspecto (Settings > Integrations > Tokens)

4) Configure the exporter to send data to the Aspecto endpoint:

const traceExporter = new OTLPTraceExporter({
  url: "https://collector.aspecto.io/v1/traces",
  headers: {
    Authorization: process.env.YOUR_ASPECTO_API_KEY,
  },
});

Your tracing.js file should look like this:

//OpenTelemetry
const { Resource } = require("@opentelemetry/resources");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");
const { SimpleSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { trace } = require("@opentelemetry/api");

//Exporter
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");

//instrumentations
const { ExpressInstrumentation } = require("opentelemetry-instrumentation-express");
const { MongoDBInstrumentation } = require("@opentelemetry/instrumentation-mongodb");
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");


//Exporter
module.exports = (serviceName) => {
 const exporter = new OTLPTraceExporter({
   url: "https://collector.aspecto.io/v1/traces",
   headers: {
     Authorization: process.env.ASPECTO_API_KEY,
   },
 });

 const provider = new NodeTracerProvider({
   resource: new Resource({
     [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
   }),
 });
 provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

 provider.register();

 registerInstrumentations({
   instrumentations: [
     new HttpInstrumentation(),
     new ExpressInstrumentation(),
     new MongoDBInstrumentation(),
   ],
   tracerProvider: provider,
 });

 return trace.getTracer(serviceName);
};

You should now be able to see these traces in our Aspecto account (notice, in the image below we sent many to-do requests, therefore we see multiple traces).

You can use filters and textual search to quickly find any errors. You can also use secondary dimension grouping to slice tracing data into which specific services, operations, or message brokers are the troublemakers.

While this is a simple to-do app, with few traces, in production, these capabilities are extremely powerful.

Quick note: The “startServer” trace you see in the image was created manually, follow the bonus section below to learn how to do that.

If you drill down into the specific to-do request, we are able to also see the request’s payload (to add that, follow the bonus section below).

Bonus: Creating Manual Span and Collecting Additional Data

Manual spans

In your to-do service file, modify the startServer function to create a manual span when your todo service is started:

// Manual span

const startServer = () => {
 tracer.startSpan("startServer").end();
 MongoClient.connect("mongodb://localhost:27017", (err, client) => {
   db = client.db("todo");
   db.collection("todos").insertMany([
     { id: "1", title: "Buy groceries" },
     { id: "2", title: "Install Aspecto" },
     { id: "3", title: "buy my own name domain" },
   ]);
 });

 app.listen(port, () => {
   console.log(`Example app listening on port ${port}`);
 });
};

Collecting the request and response

To collect the http body request you need to configure the ExpressInstrumentation and add it to the span, you should add this to registerInstrumentations:

new ExpressInstrumentation({
       requestHook: (span, requestInfo) => {
         span.setAttribute("http.request.body", JSON.stringify(requestInfo.req.body));
       },

And that’s it, folks! We hope this tutorial was helpful and practical for you. Reach out via chat to us if you have any questions.

You can find the complete project in this GitHub repository. We created three versions for the tracing.js file (for Aspecto, Jaeger, and console) to make it easier to use.

P.S. If you want to learn more about OpenTelemetry, watch this 6-episode video series – The OpenTelemetry Bootcamp (vendor-neutral and perfect for binge-watching). Use it as your OpenTelemetry roadmap, from basics to scaling, production, and security.

Episode 1: OpenTelemetry Fundamentals
Episode 2: Integrate Your Code (logs, metrics, and traces)
Episode 3: Deploy to Production + Collector
Episode 4: Sampling and Dealing with High Volumes
Episode 5: Custom Instrumentation
Episode 6: Testing with OpenTelemetry

What is OpenTelemetry and What is it Used For?

Michael Haberman — Mon, 14 Mar 2022 16:35:29 +0000

In this guide, you will learn everything you should know to get started with OpenTelemetry, why logs and metrics aren’t enough to debug our distributed services, what distributed tracing is, and why we would need it in a distributed environment.

We will then take a deep dive into OpenTelemetry, its structure, deployment methods, and some best practices.

Introduction

As developers, one of the biggest aspects of our job is to respond quickly to incidents that occur in production and to resolve these issues as fast as possible.

To be able to do that, we need to gather a lot of information fast so we can understand the full picture of what’s going on in production and tackle these incidents as soon as they arise.

But with the adoption of distributed architecture, and as the number of services in many organizations increases, understanding and troubleshooting our system has become more complex.

The tools we used to rely on the most – metrics and logs – fail to deliver the necessary visibility across our distributed system.

This is where OpenTelemetry and distributed tracing come into play.

Distributed tracing is slowly becoming a crucial tool to pinpoint, troubleshoot and solve performance issues, errors, and more in distributed systems.

OpenTelemetry is currently the standard open-source for collecting distributed traces, which eventually help us solve issues in our system (more on that later).

Logs and Metrics Aren’t Enough to Debug our Distributed System

Metrics

Metrics provide us with a high-level view of our system’s health and if it behaves within our desired boundaries.

Metrics are great at showing you when behavior has changed. However, since metrics are so high level, we only know which application is experiencing a change in behavior (e.g., database) and what metric was changed (e.g. high CPU).

We do not have the relevant information as to why it’s happening, what the root cause is, and how we can fix it.

Logs

Logs are the trail of breadcrumbs we leave within our application to read them later and understand how the application is behaving. For example, if your application failed to write to the database, you’d be able to read that in your logs.

But in distributed systems, your code is also getting distributed, and with that, your logs. You end up having a distributed trail of breadcrumbs, which is extremely difficult to follow.

If logs are the story each service tells us, in a distributed system, suddenly we have 3 different stories, which makes it difficult to understand where we should investigate.
Our ability to understand where a problem lies diminishes as we continue to distribute our applications.

To be more specific, we lose the ability to correlate where an operation has started, where a simple request came from and the process it went through.

Logs - Low-level data point - breadcrumbs trail we leave within our application to read them later and understand how the application behaved.
Metrics - High-level data point - High-level health and behavior of our entire system; something is not working as expected from the system point of view

So metrics are too high level and only provide us with the status of the overall system and logs (while helpful as they offer more details) are too low level and don’t provide the full picture.

This is where distributed tracing makes the difference.

What is Distributed Tracing?

Distributed tracing tells us what happens between different services and components and showcases their relationships. This is extremely important for distributed services architecture, where many issues are caused due to the failed communication between components.

Traces specify how requests are propagated through our services. It solves a lot of the gaps that we had when we relied solely on metrics and logs

Each trace consists of spans. A span is a description of an action/operation that occurs in our system; an HTTP request or a database operation that spans over time (start at X and has a duration of Y milliseconds). Usually, it will be the parent and/or child of another span.

A trace is a tree/list of spans representing the progression of requests as it is handled by the different services and components in our system. For example, sending an API call to user-service resulted in a DB query to users-db. They are ‘call-stacks’ for distributed services.

Traces tell us how long each request took, which components and services it interacted with, and the latency introduced during each step, giving you a complete picture, end-to-end.

To learn more about the benefits of distributed tracing, read this quick article.

Distributed Traces with Jaeger Tracing

Traces are visual instrumentation, allowing us to visualize our system to better understand the relationships between services, making it easier to investigate and pinpoint issues.

Here’s what traces look like in Jaeger Tracing – an open-source tool that allows you to visualize traces.

The Root Request and Event Order

This tracing data “tree” explains what the root request is and what was triggered by this action.

It essentially lays out the hierarchy and order in which each request was executed.

Within this tree, we have a parent-child relationship for each span. Each child span happened because of its parent span.

The very first request is made by the orders-service, which you can see is the very first span listed in the hierarchy. This is the “parent” span, which initiated the trace.
The order-service then communicates with the users-service to /users/verify.

Then the order-service sends an API call to stock-service to update the stock, which leads to the stock-service to run a .find operation on Mongoose.

Jaeger Timeline

This visualization details how long each request took, when it started, when it ended, and what happened in parallel or in sequence. That’s super important when we’re trying to identify performance issues and optimize different operations.

Each row within the timeline graph represents a span and the corresponding request listed in the hierarchy. You can now see which request took the most time.

It’s important to mention that each operation is able to produce its own logs and metrics but now, by adding traces, we’re getting the full story, the full and visual context of the whole process.

The Three Types of Telemetry

Telemetry is the data we use to monitor our applications. It’s a term meant to cover a wide range of data types, such as metrics, logs, and traces.

To be able to troubleshoot our distributed system as fast as possible, and to get complete visibility into the process happening, we want to have these three data types – logs, metrics, and traces.

Logs - Breadcrumbs trail we leave within our application to read them later and understand how the application behaved.
Metrics - High level; Health and behavior of our entire system. Great for showing when behavior has changed.
Distributed traces - The context of what and why things are happening and the story of the communication between services, allow us to visualize the progression of requests as they move throughout our system.

Having a holistic view of all three gives you the visibility you need to pinpoint problems in production and solve them as fast as possible.

What is OpenTelemetry?

OpenTelemetry is an open source project, which, among other things, allows you to collect and export traces, logs, and metrics (also known as the three pillars of observability). It is a set of APIs and SDKs.

Led by the CNCF (Cloud Native Computing Foundation, same foundation responsible for Kubernetes), it enables us to instrument our distributed services, meaning, to gather data from the events that happen in our systems, which ultimately help us understand our software’s performance and behavior.

OpenTelemetry is unique for 3 main reasons:

It is an open-source.
Collects the three pillars of observability (logs, metrics and traces) and acts as the glue that brings them together.
One specification respected by all vendors.

It is worth mentioning that at this time of writing, OpenTelemetry is still in its early stages, and bringing these telemetry data together is currently more of a vision than actual practice. You can follow the project’s status here. We also covered this topic in our Future of Observability article.

OpenTelemetry operates as a single library that captures all of this information, in a unified way and under a single specification, and ships it to your dedicated location (backend, collector, etc). This means that the OpenTelemetry data you collect can be distributed to many open sources (Jaeger and Prometheus) and vendors.

Since its implementation for most modern programming languages, it has been growing in popularity and usage by developers everywhere since it combines all three key aspects needed for proper monitoring (leading with tracing, followed by metrics).

As of today, many vendors are aligning to OpenTelemetry, which means:

You can be vendor agnostic and not tied down to a single tool or platform.
We’re able to send OpenTelemetry data to different vendors and open-sources for testing and comparison purposes with only a simple configuration change.
Use fewer platforms and get the most out of each one.

Let’s go over the OpenTelemetry stack, what components you need in order to run OpenTelemetry, and how the SDK works.

The OpenTelemetry Stack

Generally speaking, the OpenTelemetry stack consists of 3 layers:

Your application (in which you’d implement the OpenTelemetry SDK)
The OpenTelemetry collector
A visualization layer

Your application – As soon as the SDK is implemented in your application and traffic starts to flow, data from all your services will immediately be sent to the OpenTelemetry collector.

OpenTelemetry collector – Once gathered, you can choose to send the data to a dedicated location (in the image above, we are sending the data to a database).

Visualization – You can then work with a third-party or an open-source to visualize the traces, as we did above with Jaeger.

Depending on your deployment strategy, you may not need to use all aspects of the stack.

OpenTelemetry Deployment Strategies

When discussing deploying OpenTelemetry, there are two components we need to consider:

The SDK: in charge of collecting the data.
The Collector: the component responsible for how to receive, process, and export telemetry data.

For each component, you can choose between the open source-only path, the vendor path, or a combination of the two.

The first thing you should ask yourself is: am I going to work with a vendor or do I want to only work with open source solutions?

Using both Vendor and Open-Source

How to collect the data (Native SDK or Distro)

Vendor’s Distro – This option basically works by using the vendor’s OpenTelemetry distribution (also called distro) to instrument your services.

An OpenTelemetry distribution (distro) is a customized version of an OpenTelemetry component. Essentially, it is an OpenTelmetry SDK with some customizations that are vendor or end-user-specific.

OpenTelemetry Native SDK – if you prefer not to use a vendor for this part of the process, you can use the OpenTelemetry native SDK.

Where to send the data (Collector)

The OpenTelemetry collector receives the data that is being sent by the SDK. It then processes the data and sends it to any destination you’ve configured it to, like a database or a visualization tool.

If you choose to install the OpenTelemetry native SDK, you can send data directly to the vendor or send data to your own OpenTelemetry collector that then sends it to the vendor for visualization.

In any case, you must install some OpenTelemetry SDK in your services. It can be either the OpenTelemetry out-of-the-box SDK or a vendor distro.

Pro tip: go first with vendor neutral unless you get significant value from using the distro.

Using pure Open-source

If you choose to go with pure open-source, you will be using the native OpenTelemetry SDK, the native OpenTelemetry collector, and an open-source visualization layer, such as Jaeger.

Bear in mind that if you choose this path, you’ll need to run the whole stack on your own, which will allow you to be extremely flexible but will require a lot of management and manpower. Everything you need to do to run a backend application, you will need to do here as well.

How does the OpenTelemetry SDK work?

So now that you know what OpenTelemetry is, what the stack looks like, and which deployment options you have, it’s time to take a look at what’s under the hood.

Let’s say your application has two services: service A and service B.

Service A sends an API call to service B and once that happens, service B writes to the database.

Both services have the OpenTelemetry SDK and we’re using the OpenTelemetry collector.

Once service A makes an API call to Service B, Service A also sends to the collector a span that describes the call to service B, essentially letting it know that it sent an API call (making it the parent in the trace).

There’s now a “parent / child” relationship between Service A and Service B (Service A being the parent of all the actions that follow).

OpenTelemetry injects the details about the parent span within the API call to Service B. It uses the HTTP headers to inject the trace context (trace ID, span ID, and more) into a custom header.

Once service B receives the HTTP call, it extracts the same header. From that point, any following action Service B takes is reported as its child.

All this magic happens out of the box.

This action brings another key aspect of OpenTelemetry – context propagation.

Essentially, it’s the mechanism that allows us to correlate spans across services. The context is transferred over the network using metadata such as HTTP headers.

The Header will include a trace ID, which represents the sequence of HTTP calls that were performed. It will also include a span ID, which represents the event – or span – that just took place.

Because service B has the SDK implemented as well, it will also send data to the collector, informing it that it has received an API call from service A (its parent).

The same goes for Service B’s call to the DB. It’ll create another span with the same trace ID as the initial trace ID created by service A.

OpenTelemetry Auto and Manual Instrumentation

As mentioned, OpenTelemetry allows us to instrument our system, meaning, to gather data from different libraries in our applications (e.g, AWS, SDK, Redis Client) and produce spans that represent their behavior.

There are two ways spans are created – Automatic instrumentation and manual instrumentations.

Here’s a quick TL;DR

Auto instrumentations – you don’t need to write code to collect spans
Manual instrumentations – you do need to write code to collect spans

Auto Instrumentation

These are ready-to-use libraries developed by the OpenTelemetry community. They automatically create spans from your application libraries (e.g., Redis client, HTTP client).

Once you send an HTTP GET using an HTTP library, the instrumentation will automatically create a new span with the corresponding data.

Manual Instrumentation

Manual instrumentation works by manually adding code to your application in order to define the beginning and end of each span as well as the payload.

Pro tip: aim to use auto instrumentations – Check the OpenTelemetry registry to find all available instrumentations libraries.

However, there are a few common use cases for using manual instrumentations:

Unsupported auto instrumentations – Not all libraries in all languages have a ready-to-use instrumentation. In that case, you want to instrument these libraries manually and even create and contribute an instrumentation on your own.

Internal libraries – many organizations create their own libraries (for various reasons), which require you to create your own instrumentation. If that is the case, here’s what you can do:

Take inspiration from a similar open-source that has instrumentations.
Follow the OpenTelemetry specification so that your visualization tool would work as expected.

Before you start working on manual instrumentation, you should know that manual instrumentations:

Require good knowledge of OpenTelemetry
Are time-consuming and hard to maintain

Learn hands-on how to use instrumentations in the OpenTelemetry Bootcamp:

Episode 2 – How to run an auto instrumentation
Episode 5 – How to create your own custom instrumentation.

Get Started with OpenTelemetry

In this guide, we covered the very fundamentals of OpenTelemetry. There is a lot more to learn and understand, especially if you plan on implementing OpenTelemetry in your company.

If you want to learn more, you can check out this free, 6 episodes, OpenTelemetry Bootcamp (vendor-neutral).

The Bootcamp contains live coding examples where you can follow along.

It’s basically your OpenTelemetry playbook where you will learn everything, from very hands-on basics to scaling and deploying to production:

Episode 1: OpenTelemetry Intro and Basic Deployment
Episode 2: Integrate Your Code (logs, metrics and traces)
Episode 3: Deploy to Production + Collector
Episode 4: Sampling and Dealing with High Volumes
Episode 5: Custom Instrumentation
Episode 6: Testing with OpenTelemetry

OpenTelemetry Glossary

Visit the guide for the full glossary.

OpenTelemetry and Istio: Everything you need to know

Michael Haberman — Thu, 03 Feb 2022 15:38:35 +0000

There is a high correlation between R&D teams using Istio (k8s) and OpenTelemetry.

The reason is that OpenTelemetry shines when there are a lot of services communicating with each other and the best platform to host it is k8s.

Getting started with OpenTelemetry can be a challenge as it requires a lot of knowledge. Having a high entry point effort, it seems like a good idea to set up Istio to produce distributed traces because the setup is relatively fast and simple.

In this article, you will learn about the good, the bad, and the ugly of using Istio to collect traces. By the end, you will be able to decide whether using Istio is right for you.

OpenTelemetry and Istio: The Good

Easy to set up.
No code level changes are required.

Because Istio is responsible for managing traffic, it can also report logs, metrics, and traces that allow visibility to Istio and the application behavior.

Istio knows where each API call was originated from and where it is heading, making it easy to build inter-service communication (this comes with a caveat. More on that below).

The setup is quite easy:

--set tracing.enabled=true

Now, just point it to the right destination to enable tracing and you are good to go:

`--set values.global.tracer.Zipkin.address=<jaeger-collector-address>:9411`

This is easy compared to the other options for collecting trace data where you need to implement an SDK within your application code.

Implementing an SDK in our code is a far more complex and demeaning assignment (though it allows us to collect more data and provide more flexibility).

OpenTelemetry and Istio: The Bad

Single “hop” between services (“hop” explained below)
Only allows service to service communication (No databases, messaging systems, etc.)
Sampling is an issue.

Single “hop” between services

Since there is no code running within the application to collect data, you can collect partial data with partial context.

Take a look at the diagram below. When service A calls service B, Istio creates a span that represents this event. However, when service B calls service C, Istio cannot recognize that it’s the same continual trace originating from service A.

It creates a new trace (which is what I meant by using the term “hop”).

To solve this issue, you need to install OpenTelemetry SDK in each service to extract the context propagation from Istio and inject it into the downstream services.

Meaning, you probably have to install OpenTelemetry to get proper traces.

(Note that OpenTelemetry uses, by default, the W3C context propagation specification, while Istio uses the B3 context propagation specification – this can be modified).

Service to service communication

When visualizing traces with Istio, you only see HTTP and gRPC communication. Databases, cache, messaging systems are not visible.

Sampling

We use traces to debug issues in production when we cannot solve them with more traditional methods like logs and metrics. These are the cases where you need more granular data, and any piece of information is crucial.

Setting up Istio to collect traces is easy, but it may not solve the problem you set out to solve.

100% of the organizations I met that collect traces do not collect 100% of traffic. Each and every one of them uses sampling.

Sampling is a complex issue by itself, and you need lots of tools to address it. Unfortunately, Istio only gives you the ability to set the percentage of the traffic to collect. Features like different percentiles for different HTTP routes are not supported.

Here, again, OpenTelemetry comes in handy.

To allow yourself to make a better decision, you might want to dig deeper into sampling with OpenTelemetry. Here is an article that covers everything you should know about sampling.

OpenTelemetry and Istio: The Ugly

Istio supports Jaeger and Zipkin format, but both are sunsetting for OpenTelemetry to rise.

Most organizations today choose to use OTel to collect traces. This is because Jaeger (at least the collecting / SDK clients parts) is in sunset mode. The official project you should use is OpenTelemetry.

If you choose to run both Istio and OpenTelemetry, as mentioned above, make sure to configure OpenTelemetry to use B3 or write dedicated code that transforms B3 to W3C (unless you’re into broken traces. You do you, we don’t judge).

In any case, this is quite ugly (which makes sense since this is the ugly part of this article).

I assume it is just a point time, and Istio will be able to produce traces with the correct context propagation.

Btw, if at this point you feel like OpenTelemetry is for you and want to look into its different deployment strategies, here is a super quick guide for you.

Get Started with OpenTelemetry

Almost all organizations I know did try Istio’s distributed traces. However, within a short time, they either added OpenTelemetry or removed Istio for using only OpenTelemetry implemented within the code.

Running both is possible but requires some configuration. It also increases the amount of data being collected – meaning a single HTTP request has spans from both Istio and OpenTelemetry SDK.

Try it for yourself, and see what works best for you.

P.S. If you want to learn more about OpenTelemetry, you can check out this free, 6 episodes, OpenTelemetry Bootcamp (vendor-neutral).

It’s basically your OpenTelemetry playbook where you will learn everything, from the very basics to scaling and deploying to production:

Episode 1: OpenTelemetry Fundamentals
Episode 2: Integrate Your Code (logs, metrics and traces)
Episode 3: Deploy to Production + Collector
Episode 4: Sampling and Dealing with High Volumes
Episode 5: Custom Instrumentation
Episode 6: Testing with OpenTelemetry

OpenTelemetry SDK and Collector Deployment Strategies

Michael Haberman — Tue, 07 Dec 2021 17:09:55 +0000

So you are starting your journey with OpenTelemetry, congrats!

When looking back on my OpenTelemetry journey, it wasn’t always smooth sailing. So I’m writing this to try and help.

In this article, I will walk you through the different deployment methods. More specifically, what options are available for us when it comes to the SDK and Collector.

By the end of reading this, you will hopefully have a clearer view of which option you should choose.

OpenTelemetry Components

The first thing that you should do is get yourself familiarized with the different OpenTelemetry components and their roles. So let’s jump right into it.

Similar to any full-stack system, you have a frontend, a backend, and a database. OpenTelemetry is a bit like a full-stack system:

SDK / Distro – Collecting telemetry data from your services.
Collector – Ingest the telemetry data and persist it.
DB – Stores all the telemetry data.
Visualization – UI to display traces.

Your end goal is to visualize quality traces and have the ability to understand your system and troubleshoot issues.

Here is the TL;DR version of how it works: the OpenTelemetry SDK sends data to the collector that saves the data in a database. The visualization layer then queries the DB and displays your beautiful traces.

Once you have this setup, you are a happy engineer!

When it comes to the visualization layer, you have two options: a vendor (like Aspecto) or an open-source (Jaeger).

Deployment Methods

When it comes to collecting and ingesting the data, there are two components in play:

The SDK: in charge of collecting the data.
The Collector: the component responsible for how to receive, process, and export telemetry data.

You can choose between the open source-only path, the vendor path, or a combination of the two.

Choosing a vendor / an open source-vendor combo

There are a few ways to go about it:

How you collect the data (SDK / Distro)

Vendors may tell you to install their OpenTelemetry distribution (also referred to as distro), or you can install the native out-of-the-box OpenTelemetry SDK.

An OpenTelemetry distribution (distro) is a customized version of an OpenTelemetry component. Essentially, it is an OpenTelemetry SDK with some customizations that are vendor, or end-user-specific.

Where to send the data (Collector)

If you choose to install the OpenTelemetry native SDK, you can send data directly to the vendor or send data to your OpenTelemetry collector that then sends it to the vendor.

In any case, you must install some OpenTelemetry SDK in your services. It can be either the OpenTelemetry out-of-the-box SDK or a vendor distro.

Pro tip: always prefer going vendor-neutral unless you get significant value for using the vendor distro.

As for the OpenTelemetry Collector, you do not have to use it. As I mentioned above, you can send data directly to the vendor from the OpenTelemetry SDK or send it to a collector you manage and from there to the vendor.

Choosing an open source

If you choose the open-source-only path, you must run the whole stack yourself (i.e., OpenTelemetry SDK, Collector, Jaeger as a visualization layer). It has now become your operational responsibility (scaling, securing, DRP, etc.) – Are you ready for the challenge?

Using a Vendor Distro: Pros & Cons

Pros

The easiest and simplest way. Many times it ends with a few lines of code, and the vendor does the heavy lifting for you. You also have the vendor’s full support if anything goes wrong.

Cons

Vendor lock. If you ever decide to change vendors, make sure you have enough capacity to do so and that the new vendor’s offer is worth it because it will take plenty of time and effort.

Managing Collector on Your Own: Pros & Cons

Pros

This option gives you the most flexibility to decide where you want to send your data.

Cons

Managing a collector on your own takes time and knowledge – do you have that?

Should You Manage an OpenTelemetry Collector?

Here are a few tips to help you decide if you should manage a collector on your own:

The collector receives a lot of data. For every API call, DB query, cloud vendor interaction, or 3rd party API call your back end is handing, the collector will receive one event (one span in OpenTelemetry terminology). Are you experienced in managing data on a large scale? Do you have time to learn how to optimize the OpenTelemetry Collector? If yes, consider running it yourself.
Do you have the capacity to make a disaster recovery plan for when things go sideways with the collector? If the answer is no, I would suggest not running it on your own. Telemetry data becomes the data to debug/troubleshoot your production, which means it has to be available at all times.
Are you running OpenTelemetry primarily for traces, or are you interested in metrics and logs as well? OpenTelemetry can collect logs, traces, and metrics. Sending them all to one backend and then distributing them to different vendors / open sources gives you a lot of power.

Running your collector allows you to ship data anywhere and change it as you see fit. For example, you can start with one vendor and flip it in a simple configuration change. You can even run multiple POCs with vendors and open sources at the same time quite easily. The price is operational.

The Bottom Line

Here is a quick summary of our deployment options:

Use a vendor’s distro.
Use the OpenTelemetry native SDK, and send data directly to the vendor’s collector.
Use an OpenTelemetry SDK, send data to your collector, and then to the collector provided by the vendor.

It all mainly comes down to whether you have the capacity and resources to set up and manage all the components on your own – in this case, you should give the open-source path a try.

Trace-Based Testing with OpenTelemetry: Meet Open Source Malabi

Michael Haberman — Thu, 12 Aug 2021 14:11:40 +0000

Originally published on the CNCF blog. Written by Yuri Shkuro, creator and maintainer of Jaeger, and Michael Haberman, Co-Founder & CTO of Aspecto.

If you deal with distributed applications at scale, you probably use tracing. And if you use tracing data, you already realize its crucial role in understanding your system and the relationships between system components, as many software issues are caused by failed interactions between these components.

Most of you reading this article are well aware of how powerful tracing is, but here’s the thing, you are not using tracing to its full potential. Let us explain why and also introduce you to Malabi, the new open-source tool for trace-based testing.

How We Use Tracing Today

Companies these days use tracing data for critical functions such as application performance monitoring, allowing DevOps teams and SREs to find and fix issues in production after they happen. Developers usually use it when trying to debug a complex issue that happened in production.

In most cases, they are being reactive to issues, trying to understand what went wrong. But there’s another use case for tracing data: utilizing trace output to test and validate your application’s behavior.

What is trace-based testing

Trace-based testing is a method that allows us to improve assertion capabilities by leveraging traces data and make it accessible while setting our expectations from a test. That enables us to validate essential relationships between software components that otherwise are put to the test only in production.

Trace-based validation enables developers to become proactive to issues instead of reactive.

Why trace-based testing

Trace-based testing answers the complexities of current testing strategies and techniques for distributed applications. We will later see how trace-based testing can help us overcome these challenges.

To set the stage for the example that follows (and since there’s always a terminology mess around testing), we will discuss all types of tests that make actual network calls: integration tests, system tests, service tests, end-to-end tests, contract tests, API tests, you-name-it tests. Unit tests and usage of mocks are out of scope.

It is also important to mention that you can leverage trace data for testing without the need to add another layer to your existing tests.

Why traditional testing techniques aren’t enough
Let’s take a look at a use case where “regular” tests fall short and what we can do to combat these challenges. We’ll use a real use case that many people can easily relate to ordering food via an app such as Uber Eats or DoorDash.
Here is a pseudo-code representing the logic when ordering food:

// This function is called when you order food via an app
async function orderFoodDelivery(order: Order) {
 if (!valid(order)) {
   throw new Error('Bad order')
 }

 const orderDetails = await restaurantApproval(order);

 if(orderDetails.failed){
   throw new Error('Restaurant too busy')
 }

 Promise.all([
   // Send a notification to mobile app
   notifyClient(orderDetails),

   // Start delivery
   enqueueDelivery(orderDetails),

   // Call stripe and if successful upload invoice to s3.
   // if failed, put in queue for retry
   paymentAndInvoice(orderDetails),
 ]);
}

This code flow is pretty simple.

The backend gets the order details, validates them, and asks the restaurant to deliver the food.

Then we need to notify the customer, find a delivery person, and charge the customer.

One major concern we can all agree on is that getting someone else’s food could make you mad. Really mad!

We want the delivery person, customer, and restaurant to be all aligned with the order details. Making it happen with “traditional” testing techniques can be a significant hassle.

Essentially, the problem is we’re not fully testing the whole process. Usually, we can only validate the response, which is, by design, limiting us from peeking into its internal workflow and does not offer the level of reliability we’re looking for when testing.

For example, the fact that we got an order approval from the restaurant, doesn’t mean the delivery person received our correct address.

How can trace-based testing help?

There are a few common cases in which traces are useful for testing internal workflows:

When dealing with multiple components, your test needs to be familiar with different APIs and configure many clients. Testing against trace uses a single uniform API.
A trace can capture a transient state which is difficult to retrieve after the workflow has finished. For instance, there may be no record anywhere other than logs that an error occurred in a given component.
Each component may not even expose its internal state in sufficient detail to validate the workflow. Tracing provides us with granular data to overcome this.

Trace-based testing validates that the proper interaction between components occurs as part of the tested workflow. We want to validate there’s complete sync between the customer, restaurant, and delivery.

We mentioned we’d introduce you to a new open-source for trace-based testing called Malabi. Take a look at the following integration test to see how we use Malabi to easily validate this sync:

// Nothing worse than getting the wrong food.
it('Making sure the right person get the food.', ()=>{
 const order = {id:1, userId};
 orderFoodDelivery(order);

// Using Malabi to make sure the internal calls to Kafka and WS are synced with the same order ID.
 expect(malabi.kafka({topic:'match-delivery-order'}).toMatch(id:order.id})
 expect(malabi.websocket({room:order.userId}}).toMatch({id:order.id})
})

Meet “Malabi”

In the code example above, you can see how Malabi helps us by making traces accessible in the assertion phase while setting our expectations from the test. We can then validate the internals of the API call.

Malabi is an open-source Javascript framework (still in its early days) based on OpenTelemetry that allows you to leverage trace data and improve assertion capabilities.

(It is also a delicious milk pudding dessert made of rice, sugar, rice flour, and milk).

Credit: @malabi_marshmallow

This is how open source Malabi works

Let’s review the diagram below that shows what happens in Malabi when running the integration test from above. We are validating that the Order Service submitted correct messages to Kafka and to a Websocket.

The Malabi lib that runs inside the Test Runner is fetching traces collected in the OpenTelemetry SDK. This is done by adding a custom exporter to OTEL SDK that stores the trace in memory (the exporter is already provided by Malabi). You can add the custom exporter by yourself, or by using the OpenTelemetry SDK we created for you (with a good amount of auto instrumentations from both OTEL contrib and Aspecto’s repo).

Currently, Malabi supports validating spans created by the service under test (order service in the diagram above). For supporting downstream services and async message brokers, you will need to spin up some backend (Jaeger, OTEL Collector) and store those spans as well. That is in Malabi’s roadmap.

By collecting the output traces of each test and serving them to the developer in the assertion phase we’re easily getting deeper visibility into the workflow’s internal processes.

You can check Malabi’s repo for more technical details and a demo you can run locally: https://github.com/aspecto-io/malabi ⭐️

A quick code example for the assertion (this is how we use it internally):

it('happy flow - pull_request opened event is written to db and sqs', async () => {
      const event = 'pull_request';
      const eventId = 'my_eventId';
      const payload = {
          action: 'opened',
          name: 'Rick',
          lastName: 'Sanchez',
          phrase: 'Wubba Lubba Dub-Dub',
          organization: {
              login: 'aspecto-io',
          },
      };
      const res = await client.post('/api/v1/webhook', payload, {
          headers: {
              [sigHeaderName]: getGithubSignature(payload),
              'X-Github-Event': event,
              'X-GitHub-Delivery': eventId,
           },
      });

      expect(res.status).toBe(200);
      const spans = await getMalabiExtract();

      // Assert DB write
      expect(spans.mongo().length).toBe(1);
      const mongo = spans.mongo().first;
      expect(mongo.hasError).toBeFalsy();
      expect(mongo.dbOperation).toBe('save');
      expect(mongo.mongoCollection).toBe('github-events');
      const doc = JSON.parse(mongo.dbResponse);
      expect(doc.payload).toEqual(payload);
      expect(doc.eventId).toEqual(eventId);
      expect(doc.type).toEqual(event);
      // Make sure timestamps are written
      expect(doc.createdAt).toBeDefined();
      expect(doc.updatedAt).toBeDefined();

      // Assert SQS Send
      expect(sqs.hasError).toBeFalsy();
      expect(spans.awsSqs().length).toBe(1);
      const sqs = spans.awsSqs().first;
      expect(sqs.rpcMethod).toBe('sendMessage');
      expect(JSON.parse(sqs.messagingPayload)).toEqual({
          type: 'git-hub-pr',
          data: payload,
      });

      const incomingHttpSpan = spans.http().incoming().first;
      expect(incomingHttpSpan.hasError).toBe(false);
  });

Getting started with Malabi

Malabi’s installation is easy and requires only two parts:

An OpenTelemetry SDK is installed in the service under test.
A test assertion NPM package is installed in the test runner.

Calling all early adopters and the bottom line

It seems that there’s a whole world of opportunities to optimize and increase the effectiveness of testing – by getting more visibility into the processes running across our entire system and by leveraging available tracing data to increase test reliability and possibly preventing issues early in the development cycle.

This is what Malabi is meant to do.

Malabi is an early-stage open source and there is much work to be done – this is where you, brilliant people, come in! We would love your support in this project, feel free to help in any shape or form (contribute code, knowledge, ideas, or just share it. Oh yeah! GitHub stars make traces better).

If that sounds interesting, leave your email and one of our developers, in love with OpenTelemetry, will reach out to you.

Distributed Tracing: Build vs Buy

Michael Haberman — Thu, 05 Aug 2021 13:57:12 +0000

As microservices observability becomes more and more of a challenge for engineers, companies are turning to distributed tracing solutions.

Now, they have to decide whether to buy or to build their distributed tracing infrastructure. While building a solution from scratch might sound compelling, and can even be supported with open source Jaeger and OpenTelemetry, buying solutions provide more features and easier usage.

Let’s see how the two compare and when you should choose each one. But first, let’s remind ourselves about what distributed tracing is.

What is Distributed Tracing?

Logging is a difficult and time-consuming practice, which doesn’t always provide the relevant information for solving performance issues and regressions. Traces complement logs, by presenting the relationships between services and components and tracking requests across them. It’s no surprise, then, that many developers are adding tracing to their workflows, and engineering departments are using it to reduce their MTTR.

But we live in a world of microservices, and we need tracing to adapt.

That’s where distributed tracing comes in. Distributed tracing is tracing that is suited for microservices. It provides observability into the microservices architecture. For example, it shows why failed integrations between components occur. These insights are used for faster troubleshooting and accelerating time-to-market.

Read more about distributed tracing, its advantages, and when to use it in this blog post.

Building vs. Buying Distributed Tracing

So you’ve determined your organization needs a distributed tracing solution. Congratulations! Now you need to decide whether you’re going to build one or to buy one.

Building a solution means the product is built internally, from the ground up. This includes researching the problem, designing the tracing solution with open source solutions like Jaeger, using OpenTelemetry to instrument, gather and analyze data, allocating resources to build it, and continuously maintaining it.

One of the most famous examples of building an internal distributed testing solution is Netflix, whose developers wrote a blog post about the progress they’ve made four years after they started building it.

But even if you’re not Netflix, you can probably still build your own basic tracing solution.

What if you need a tracing solution that can scale and provide lots of added value? Then, you might want to look into buying a solution.

This means purchasing a ready-made out-of-the-box distributed tracing solution. You can either choose a managed solution, i.e a solution that manages open source (like Jaeger) for you. Or, an unmanaged solution, i.e a distributed tracing solution from a vendor, with unique features and capabilities.

Whichever you choose, buying a solution enables immediate implementation, observability, and insights.

Let’s dive deeper into how these two types of solutions compare.

Building vs. Buying Distributed Tracing – a Comparison

1. Maintenance and Scalability

You’re on the quest for data. Luckily, you’ve come to the right place. However, maintaining and storing all that data isn’t always easy, or easy on your resources. Scaling your tracing operation is another huge challenge.

Ingesting and storing large amounts of data from many services requires a reliable infrastructure. This infrastructure should enable scaling the amount of traces collected, processing them, and storing them in a database like ElasticSearch.

Building a solution requires constant operation and maintenance as you scale your data collections. Buying a solution ensures your data is scalable and available, no matter the load or requirements.

2. Features and Capabilities

Built and bought solutions offer different features and capabilities. With built solutions, you can design the system according to your specific needs and requirements. On the other hand, buying solutions provide out-of-the-box capabilities that are meant to give you value beyond your basic needs. There is usually a wide variety, and they are built on industry best practices and requirements from multiple companies.

3. Cost

Built solutions require many resources for configuring, designing, managing, maintaining, and operating. While this might seem more cost-effective, the truth is that when you build your own, you know where you start – but not where you’ll end up. You have to factor in the time developers need to learn and build the solution, as well as the cloud resources cost.

Buying a solution, on the other hand, has a fixed cost. The price might seem higher, but everything is taken care of. You can also always start out with a free trial or free tier to make sure the solution answers your needs.

4. Security

Security is shifting left, making it more and more the responsibility of developers. When building a distributed tracing solution, you also have to take security controls into consideration. Data privacy, reducing the attack surface and mitigating vulnerabilities have to be designed. Purchased solutions, on the other hand, do the security heavy lifting. They take security concerns off your plate and offer it out-of-the-box.

5. Business Focus

Sure, your super developer powers can build anything. But should they? When developers build a distributed tracing solution internally, they’re diverted from the core business. Why not let other developers, who work at companies where distributed tracing is the core business, build the solution for you?

6. Democratization and Accessibility

When distributed tracing solutions are built internally, the engineers who built them control the knowledge and accessibility to the product. Buying a solution enables more democratization since access and training is available for all.

Buy vs. Build Distributed Tracing Comparison Table

Distributed Tracing Solutions

Unless you are a large company with available resources and unique architecture, or an individual looking to test the water of distributed tracing (in which case maybe a hybrid model would also work for you), we believe buying is the way to go.

Here are some of the tools you can choose from for both building and buying a distributed tracing solution:

1. Open Source: Jaeger

Jaeger is an open-source, distributed tracing tool that you can use as the basis for building your own distributed tracing solution. You can use Jaeger for transaction monitoring, latency optimization, root cause analysis, dependency analysis, and context propagation. Jaeger can run on any infrastructure and leverage OpenTelemetry regardless of the code language.

2. Managed Distributed Tracing Solution

If you’re into Jaeger but not into getting your hands dirty with it, managed solutions will bring the hand sanitizer and do it for you. With such solutions, you get Jaeger’s capabilities, but the vendor packs them up for you in dashboards that already provide you with insights and observability.

It’s also easier to get started than having to download Jaeger and building your infrastructure from scratch.

3. Aspecto

Aspecto is a distributed tracing platform with plenty of new features and a unique UI. Aspecto helps developers find, fix, and prevent distributed application issues across the entire development cycle.

It’s the Chrome DevTools for your distributed applications.

OpenTelemetry based, Aspecto allows developers to prevent issues before they reach production by implementing telemetry data that learns the system, then compares what they do locally to the production, staging, or other locals baseline data. You can easily install Aspecto, for free, with a one-liner SDK, or give the Live Playground a spin.

And if you’re already using the OpenTelemetry collector, you can use it to export traces to the Aspecto OTEL collector.

Conclusion

Building your distributed tracing solution might sound like a nice challenge. Unless you’re a huge company with endless resources and unique architecture – you might be embarking on a very costly and ineffective adventure.

Buying a tracing solution offers you all the capabilities and observability you need right when you get started. You can buy Jaeger-based managed solutions or get more feedback and visibility with tons of additional features through solutions like Aspecto.

We recommend getting started with distributed tracing as soon as you can, no matter your choice. Make your microservices management and troubleshooting a lot more effective with tracing.

Logging vs Tracing: Why Logs Aren’t Enough to Debug Your Microservices

Michael Haberman — Thu, 24 Jun 2021 15:10:51 +0000

When debugging microservices, it can be challenging for developers to identify the root cause of issues. Not to mention how frustrating it is to search through endless logs across multiple services and the time it takes them.

With all these challenges, however, there is a silver lining — distributed tracing.

Distributed tracing can help your developers with tracking requests across services (but more on that later).

Let’s dive into what is distributed tracing, its benefits, and the role it plays in your teams’ system. Then we’ll cover which tools developers can use to implement distributed tracing in a cloud native environment.

But first, to understand where tracing fits in your microservices debugging process and why you might even need them in the first place, let’s identify the challenges that debugging with logs pose.

Log Debugging Challenges

Logs can be very useful when we are trying to understand an unexpected response or a production failure. However, logs don’t have unlimited capabilities. Here are some of the challenges they pose for your developers when they are debugging microservices:

1. Logging Is a Manual Time-Consuming Process

Adding logs is not an automatic process, and it requires a lot of meticulous, manual work. Identifying all the potential information that will be needed for debugging, adding the logs, removing them if necessary – these all take a long time and require a lot of effort. Also, the process is error-prone. Developers might be spending a lot of time adding logs but will still miss the exact information they need in production.

2. It’s Hard to Find the Right Balance

Developers need to ensure they have enough logs for debugging, but not too many logs so that the code is too heavy and they waste too much time on adding and analyzing them. It’s hard to create this balance. If they haven’t logged enough information, they’ll miss data for debugging. If they logged too much, the process becomes resource-intensive and makes log analysis much more difficult.

3. Tracking Logs across Services Is Difficult

Tracking and analyzing log entries across multiple services, containers, and processes is challenging. The developer has to be able to make sense of the relationship between all the different logs, which requires understanding the code flow in different services and correlating them to logs. They have to go through the process of transforming raw text (logs) into visualization in their minds.

This takes a very, very long time.

Even companies that have added unique identifiers to their instrumentation to enable tracking have difficulties maintaining and updating them. Not to mention ensuring all developers are up to speed about their homegrown identifier conventions.

4. Logs Aren’t Standardized

Logs do not have a structured format, meaning that any developer can create messages and events according to their style. While this provides flexibility and freedom, it can be challenging and counter-productive for your team to try to understand someone else’s logs or to explain them.

Also, lack of standardization leaves more room for human error.

Log Debugging Fail

As a result, logs won’t always provide the required information to solve performance and regressions. There are many solutions out there that try to overcome these challenges. These include standardization conventions, best practices, analysis tools, and more. But, maybe we need to realize that logging has its limitations and that your team needs another solution for debugging microservices.

And that solution is tracing.

What Is Distributed Tracing?

Traces complement logs. While logs provide information about what happened inside the service, distributed tracing tells you what happened between services/components and their relationships. This is extremely important for microservices, where many issues are caused due to the failed integration between components.

Also, logs are a manual developer tool and can be used for any level of activity – a specific low-level detail, or a high-level action. This is also why there are many logging best practices available for developers to learn from. On the other hand, traces are generated automatically, providing the most complete understanding of the architecture.

Distributed tracing is tracing that is adapted to a microservices architecture. Distributed tracing is designed to enable request tracking across autonomous services and modules, providing observability into cloud native systems.

Distributed Tracing Advantages
Where logging is bounded, distributed tracing thrives. Let’s see how distributed tracing answers logging limitations when it comes to debugging microservices.

1. Visualization

Traces are visual instrumentation. As opposed to text logs, with traces, developers don’t have to imagine the communication flows and make up an image in their minds. Instead, they can see it right before their eyes. This makes it easier for developers to understand the relationships between services and to resolve issues, like performance bottlenecks.

2. Automation

Unlike logs, traces are automatic. Developers don’t have to make the manual effort of adding logs to get the complete picture. Instead, they automatically get a visualization of what happened. This also solved the standardization problem. With automated traces, the standardization is hard-coded in.

3. Accelerate Time-to-Market

Distributed tracing provides observability and a clear picture of the services. This improves productivity because it enables developers to spend less time trying to locate errors and debugging them, as the answers are more clearly presented to them. As a result, productivity is increased, and developers can spend more time developing features, (or taking a break), while you accelerate time-to-market.

4. Tracking Requests Across Services

Microservices interactions span multiple services. Distributed tracing enables understanding the system and the relationships between components. This is done by tracking and recording all these requests through unique IDs that are passed to the services handling them. As a result, developers can see the flow and progression of the request across the entire architecture, which is often the hardest to understand when debugging. Your team’s code quality will improve immensely.

5. Easy to Use and Implement

With the right setup, developers can work with multiple applications and across different programming languages. This is unique for distributed tracing and saves your team a lot of time and headaches, by not restricting you to one language or certain apps.

6. Insightful

Distributed tracing provides the developer with a lot of insightful information. This includes request time, information about components, latency, application health, and more. All this info can be useful when debugging and during root cause analysis, for improving code quality and resolving customer issues quickly.

When Should We Use Distributed Tracing?

Great question! Here are the three main use cases in which distributed tracing can be helpful for you and your team.

1. For a Distributed-Application Architecture

If your department is using a distributed infrastructure, we highly recommend implementing distributed tracing. As you can see, this is the best method for tracking requests across services, with many teams involved and when you have complex processes in place.

It makes sure you don’t waste your time trying to investigate issues across machines or, search through endless logs.

2. When You Don’t Know Which Problem to Look for

One of the reasons developers end up with too many logs is that they want to cover themselves and make sure they have information for all and any scenario that could go wrong. But that’s the wrong approach. This is exactly what traces are for. Traces provide you with all the heaps of information you need to analyze yourself, without the disadvantages of logs. So if you don’t know what the problem is, you can analyze until you do.

3. When You Need Observability

Distributed traces provide you with visibility into the system and across all services and the relationships between them. You can see the journey requests went through, how long they took, insights into system health, and more. You can use distributed tracing not only for identifying why a problem occurred, but also to avoid problems with ongoing observability and tracking.

Distributed Tracing Tools

Hopefully, by now you’re convinced that distributed tracing can make your life easier, or at least shorten your debugging time. To get you started, here are three tools for your team to look into. These tools use an open-source called OpenTelemetry, an observability framework for microservices and a member of the Cloud Native Computing Foundation.

Here are the tracing tools that will complement your logging efforts, especially in a microservices architecture:

1. Jaeger

Jaeger is an open-source, distributed tracing tool. It enables transaction monitoring, latency optimization, and advanced data analysis. Jaeger supports most common languages and requires running Kubernetes. You can check it out here.

2. Zipkin

Zipkin, an open-source tool very similar to Jaeger, and also provides all distributed tracing capabilities. For implementation, Zipkin doesn’t require containers. You can use Docker, but you don’t have to. The difference between the two is minor, and in the end, it comes to personal preferences and specific technology stack needs.

3. Aspecto

Aspecto is like the Chrome DevTools for your distributed applications, helping developers find, fix, and prevent distributed application issues across the entire development cycle. Starting with their local dev environment all the way to production.

Aspecto is OpenTelemetry based, and the way it allows developers to prevent issues before they reach production is by implementing telemetry data that learns the system, then compares what they do locally to the production, staging, or other locals baseline data.

This helps you to validate changes and prevent issues, live, while you develop.

It’s super easy to install with a one-liner SDK, and you can give the Live Playground a spin.

Conclusion

Debugging with logs can only get you so far. By implementing distributed tracing, you can see your requests and services, and spend less time debugging. Try distributed tracing with an open-source tool, like Jaeger or Zipkin and if you’re looking for that extra boost of predicting the effects of your changes, give Aspecto a try, for faster feedback and more visibility.

Choosing a Message Broker: Kafka vs RabbitMQ vs AWS SQS/SNS

Michael Haberman — Tue, 01 Jun 2021 15:05:10 +0000

Microservice applications rely heavily on messaging and asynchronous communications to keep everything working smoothly.

Choosing the right message broker is one of the first critical choices you must make when developing the services that need to communicate with each other.

Making the “right” choice can be a battle of features and edge cases that can be challenging to differentiate between.

In this article, I’ll help provide a bit of guiding light by providing an overview of a few of the more well-known message brokers – Kafka, RabbitMQ, and AWS SQS/SNS.

I’ll explore the driving forces behind them, the general messaging model they follow and do my best to provide some guidance on selecting the broker that is right for you.

Apache Kafka

Kafka is an open-source message broker developed and maintained primarily by the Apache Software Foundation, with the assistance of the open-source community.

Key Features

Focus on streamable content, working with large data streams
Message persistence and reprocessing capabilities are core features
On-site hosting with third party options

Kafka provides optimized stream-based processing of events, with a publish/subscribe model driving the communications between consumers.

These events can be subdivided into topics, allowing for greater organization of your distributed application’s communication patterns, and are partitioned onto multiple servers within a cluster, allowing for a resilient and highly performant message delivery system.

Technical Details and Deployment

Apache provides SDKs in several different languages.

Kafka is designed to be deployed on-site, within your own application’s architecture. This can be on a set of stand-alone servers, a virtual machine, or a Docker container.

Multiple vendors offer Kafka hosting as a service, such as AWS, CloudKarafka, and Aiven, or in virtual machines.

Below is some sample JS code for getting started with Apache Kafka events:

const { Kafka } = require('kafkajs')
const kafka = new Kafka({
 clientId: 'my-app',
 brokers: ['localhost:9092']
})

// this produces a message
async function produce() {
 const producer = kafka.producer()
 await producer.connect()
 await producer.send({
   topic: 'TOPIC_NAME',
   messages: [
     { key: 'key1', value: 'hello world' },
   ],
 })
}

async function consume() {
 const consumer = kafka.consumer({ groupId: 'my-group' })
 await consumer.connect()
 await consumer.subscribe({ topic: 'TOPIC_NAME' })
 await consumer.run({
   eachMessage: async ({ topic, partition, message }) => {
     console.log({
       key: message.key.toString(),
       value: message.value.toString(),
       headers: message.headers,
     })
   },
 })
}

Strengths & Weaknesses

Kafka has a high focus on data stream throughput, something that shows in their performance statistics.

This focus on processing streams of data results in a system with high throughput, allowing for complex processing of large data streams.

Kafka’s routing capabilities for those streams of data are relatively limited when compared to other message brokers – a gap that is continually getting smaller as these products improve.

To summarize, Kafka is a powerful solution that can provide robust and fault-tolerant high-performance message streaming, letting you confidently drive your application’s behavior.

Depending on your bandwidth and resources, you can abstract away as much or as little of the hosting as you feel comfortable, making Kafka a solid choice that will scale with your traffic.

RabbitMQ

Like Kafka, RabbitMQ is another open-source message broker. Originally developed by Rabbit Technologies, the technology has through a series of acquisitions ended up under the ownership of VMWare.

Key Features

Focus on messaging-based communication, with support for large data streams
Complex routing capabilities provided as a core feature
On-site hosting with third party options

RabbitMQ uses the publish/subscribe model as well, sending message objects in their binary form to different named queues, which can be dynamically created and destroyed.

RabbitMQ is designed to operate both in isolation and as part of a cluster, providing enough configurable power to drive any set of redundancy or data safety needs.

Technical Details and Deployment

RabbitMQ provides several client libraries in a wide variety of languages.

It can be deployed on-site, on anything from a full server to a container, or on one of several cloud providers:

The following sample code, written in Node.js using the AMQPLIB package, should provide a small sample of what it might be like to work with RabbitMQ:

const amqp = require('amqplib/callback_api');
amqp.connect('amqp://localhost', function(error0, connection) {
 if (error0) {
   throw error0;
 }
 connection.createChannel(function(error1, channel) {
   if (error1) {
     throw error1;
   }
   const queue = 'hello-queue';
   const msg = 'Hello world!';

   channel.assertQueue(queue, {
     durable: false
   });

   // Sending message to queue
   channel.sendToQueue(queue, Buffer.from(msg));
   console.log("Sent message", msg);

   // Consuming messages
   channel.consume(queue, function(msg) {
     console.log("Received message", msg.content.toString());
   }, { noAck: true });
 });
});

Strengths & Weaknesses

RabbitMQ has the power to handle workloads of nearly any size, and can effectively scale alongside your application as your user base grows.

With a focus on messaging-based delivery and complex routing scenarios, RabbitMQ is extremely adaptable to any application architecture.

While originally there was no great support for data stream processing, and messages were generally only processed once without the capacity to reprocess a stream, both of these gaps have closed as RabbitMQ has continued to grow.

With the ability to take ownership of the things you want and outsource the rest, RabbitMQ can fit into any appropriate role in your application’s infrastructure.

Amazon Web Services SQS/SNS

SNS and SQS represent two different ways of looking at distributed messaging.

SNS is highly focused on message delivery, offering a publish-subscribe model to quickly distribute a message to an array of clients (e.g., mobile devices, HTTPS endpoints, other AWS services).

SQS, by comparison, is focused on the successful delivery and processing of messages by individual clients.

Key Features

Two products allowing for both broadcast messaging and pub/sub
Quick setup and configuration using AWS
No hosting available outside of AWS

While SNS will broadcast the same message to an array of recipients, SQS will distribute queue messages to single subscribers for processing.

SNS takes a push-based approach to notifications, allowing for automated responses to notification activity, while SQS tends to focus more on a polling-style mechanism with some additional event-driven functionality supported.

Technical Details

AWS provides a general SDK with access to most AWS services (SQS and SNS included) in several popular languages.

The below sample code uses the AWS SDK to demonstrate the process of working with SNS and SQS:

// SNS - publish
const AWS = require('aws-sdk');
AWS.config.update({ region: 'REGION' });

const publishParams = {
 Message: 'MESSAGE_TEXT',
 TopicArn: 'TOPIC_ARN'
};

const publishTextPromise = new AWS.SNS({ apiVersion: '2010-03-31' }).publish(publishParams).promise();

publishTextPromise.then(
 function(data) {
   console.log(`Message ${publishParams.Message} sent to topic ${publishParams.TopicArn}`);
 }).catch(
 function(err) {
   console.error(err, err.stack);
 });

// SNS - Subscribe
const subscribeParams = {
 TopicArn : 'TOPIC_ARN'
}
const subscribePromise = new AWS.SNS({ apiVersion: '2010-03-31' }).listSubscriptionsByTopic(subscribeParams).promise();
subscribePromise.then(
 function(data) {
   console.log(data);
 }).catch(
 function(err) {
   console.error(err, err.stack);
 }
);

// SQS - send
const sqs = new AWS.SQS({ apiVersion: '2012-11-05' });
const queueURL = "SQS_QUEUE_URL";

const sendParams = {
 DelaySeconds: 10,
 MessageAttributes: {
   "Title": {
     DataType: "String",
     StringValue: "Some String"
   }
 },
 MessageBody: "Something!",
 QueueUrl: queueURL
};

sqs.sendMessage(sendParams, function(err, data) {
 if (err) {
   console.log("Error sending to SQS", err);
 } else {
   console.log("Success sending to SQS", data.MessageId);
 }
});

// SQS - receive
const receiveParams = {
 AttributeNames: [
   "SentTimestamp"
 ],
 MaxNumberOfMessages: 10,
 MessageAttributeNames: [
   "All"
 ],
 QueueUrl: queueURL,
 VisibilityTimeout: 20,
 WaitTimeSeconds: 0
};

sqs.receiveMessage(receiveParams, function(err, data) {
 if (err) {
   console.log("Receive Error", err);
 } else if (data.Messages) {
   console.log("Received messages:", JSON.stringify(data.Messages))
 }
});

Strengths & Weaknesses

AWS SQS and SNS, together, can be used to build the backbone of a highly scalable, highly resilient distributed application.

With ties into many other AWS services (such as AWS Lambda), these two tools can help you easily grow your application’s communications, while providing you with all the tools you need to manage the underlying complexities of the service interactions.

If your web application is already running on AWS, set-up time is effectively zero, with much less complexity than many of the other systems. This does potentially come at the expense of a larger AWS bill as the number of messages grows.

While Kafka and RabbitMQ don’t provide a default message size limit, AWS provides some limits around SQS and SNS messages – converting the messages into S3 objects after they reach a certain size.

We published a detailed article on how you can overcome this size limit – I highly recommend browsing it to get a feel for how SQS in particular manages large messages.

[You can find our SQS/SNS Producer/Consumer Library inside. It provides an ability to pass payloads through s3]

SQS and SNS being cloud-first do add the additional complication of being vendor-locked to a specific service, whereas other message brokers resolve this by providing local installation and maintenance capabilities.

Choosing The Right Message Broker

Generally speaking, there are two considerations you should be thinking about when choosing a broker:

Consideration #1: The Type of Messages You Send

The first step to choosing a message broker is determining what messages you’ll be sending, and what their general format will be.

The characteristics of these messages will drive the questions that need to be asked about each platform’s offering, though most will be roughly equivalent in terms of feature set – meaning that at a general level, every solution listed above supports the functionality needed to serve as a message broker for a scalable distributed application.

From a pure feature perspective, one solution is as good as another.

Consideration #2: Your Daily Work and Applications’ Infrastructure

This is where secondary considerations come into play. Think about your day-to-day work and systems and ask yourself:

Are you building an application exclusively in AWS? Maybe SQS and SNS make the most sense for establishing your inter-service communication
Are you more interested in writing your application than maintaining the piped data between its components? Then a third-party hosted solution might be the best option for allowing you to focus on your strengths while growing your codebase
Are you highly focused on speed of delivery and minimal latency? Then Kinesis might be right up your alley (we’ll go over Kinesis on another article, so stay tuned), while an application more focused on verified delivery and redundancy might choose a different technology.

The requirements of your application’s infrastructure and behavioral patterns will drive the choice at this level.

With the above taken into the account, and with the caveat that it is hard – and somewhat unfair – to reduce these large tech products down to a couple of lines of recommendations, here are some guidelines on choosing the right message broker:

If you care about message retention and being able to easily re-process data, Kafka is likely your best option
If you are more concerned with being able to maintain and implement a complex set of routing rules, RabbitMQ is likely your best choice
If you’re a small startup looking to get up and running quickly, with minimal overhead, AWS SQS/SNS is a great option given its quick setup and cost structure

Getting End-to-end Visibility Into a Message Journey

One element you’ll want to evaluate is how to best maintain the final product. Once your application is sending messages, how do we track things down when they go wrong?

OpenTelemetry is a set of SDKs and tools that can be used to set up observability for your distributed application, providing you the means to troubleshoot the distributed messaging driving your application when things go wrong.

Here’s a quick step-by-step guide on implementing OpenTelemetry in your distributed applications, allowing you to achieve end-to-end visibility into a message journey. The guide demonstrates how to do so with Kafka as the message broker.

[PS: grab your OpenTelemetry Kafkajs Instrumentation for Node.js inside.]

Conclusion

If you’re building an application that is “distributed” in any way, odds are you’ll need to handle asynchronous communications between components of your application at some point.

Messages – and the brokers that deliver them – will play a critical role in the infrastructure driving your application.

The above summary is by no means exhaustive – I would probably need another thousand words just to make a solid start at capturing the complete message broker landscape – but should hopefully provide some quality information that you can use when making your decision.

The key is to fully understand the requirements for your application, and how those requirements fit into the capabilities of the message brokers you are evaluating.

There is ultimately no “wrong” answer to which message broker you choose, but hopefully, the above information helps point you in the right direction.

About Aspecto

Aspecto is an OpenTelemetry-based troubleshooting platform 
that helps developers prevent distributed application 
issues from their local dev environment, 
and across the entire development cycle.

You can think of it as the 
Chrome DevTools for your distributed applications.

Aspecto is used for detecting and 
troubleshooting microservices-based distributed systems, 
and preventing software failures before deployment.

Visit us at Aspecto.io
for more microservices tutorials