OpenTelemetry SDK and Collector Deployment Strategies

So you are starting your journey with OpenTelemetry, congrats!

When looking back on my OpenTelemetry journey, it wasn’t always smooth sailing. So I’m writing this to try and help.

In this article, I will walk you through the different deployment methods. More specifically, what options are available for us when it comes to the SDK and Collector.

By the end of reading this, you will hopefully have a clearer view of which option you should choose.

OpenTelemetry Components

The first thing that you should do is get yourself familiarized with the different OpenTelemetry components and their roles. So let’s jump right into it.

Similar to any full-stack system, you have a frontend, a backend, and a database. OpenTelemetry is a bit like a full-stack system:

• SDK / Distro – Collecting telemetry data from your services.
• Collector – Ingest the telemetry data and persist it.
• DB – Stores all the telemetry data.
• Visualization – UI to display traces.

Your end goal is to visualize quality traces and have the ability to understand your system and troubleshoot issues.

Here is the TL;DR version of how it works: the OpenTelemetry SDK sends data to the collector that saves the data in a database. The visualization layer then queries the DB and displays your beautiful traces.

Once you have this setup, you are a happy engineer!

When it comes to the visualization layer, you have two options: a vendor (like Aspecto) or an open-source (Jaeger).

Deployment Methods

When it comes to collecting and ingesting the data, there are two components in play:

1. The SDK: in charge of collecting the data.
2. The Collector: the component responsible for how to receive, process, and export telemetry data.

You can choose between the open source-only path, the vendor path, or a combination of the two.

Choosing a vendor / an open source-vendor combo

There are a few ways to go about it:

How you collect the data (SDK / Distro)

Vendors may tell you to install their OpenTelemetry distribution (also referred to as distro), or you can install the native out-of-the-box OpenTelemetry SDK.

An OpenTelemetry distribution (distro) is a customized version of an OpenTelemetry component. Essentially, it is an OpenTelemetry SDK with some customizations that are vendor, or end-user-specific.

Where to send the data (Collector)

If you choose to install the OpenTelemetry native SDK, you can send data directly to the vendor or send data to your OpenTelemetry collector that then sends it to the vendor.

In any case, you must install some OpenTelemetry SDK in your services. It can be either the OpenTelemetry out-of-the-box SDK or a vendor distro.

Pro tip: always prefer going vendor-neutral unless you get significant value for using the vendor distro.

As for the OpenTelemetry Collector, you do not have to use it. As I mentioned above, you can send data directly to the vendor from the OpenTelemetry SDK or send it to a collector you manage and from there to the vendor.

Choosing an open source

If you choose the open-source-only path, you must run the whole stack yourself (i.e., OpenTelemetry SDK, Collector, Jaeger as a visualization layer). It has now become your operational responsibility (scaling, securing, DRP, etc.) – Are you ready for the challenge?

Using a Vendor Distro: Pros & Cons

Pros

The easiest and simplest way. Many times it ends with a few lines of code, and the vendor does the heavy lifting for you. You also have the vendor’s full support if anything goes wrong.

Cons

Vendor lock. If you ever decide to change vendors, make sure you have enough capacity to do so and that the new vendor’s offer is worth it because it will take plenty of time and effort.

Managing Collector on Your Own: Pros & Cons

Pros

This option gives you the most flexibility to decide where you want to send your data.

Cons

Managing a collector on your own takes time and knowledge – do you have that?

Should You Manage an OpenTelemetry Collector?

Here are a few tips to help you decide if you should manage a collector on your own:

1. The collector receives a lot of data. For every API call, DB query, cloud vendor interaction, or 3rd party API call your back end is handing, the collector will receive one event (one span in OpenTelemetry terminology). Are you experienced in managing data on a large scale? Do you have time to learn how to optimize the OpenTelemetry Collector? If yes, consider running it yourself.

2. Do you have the capacity to make a disaster recovery plan for when things go sideways with the collector? If the answer is no, I would suggest not running it on your own. Telemetry data becomes the data to debug/troubleshoot your production, which means it has to be available at all times.

3. Are you running OpenTelemetry primarily for traces, or are you interested in metrics and logs as well? OpenTelemetry can collect logs, traces, and metrics. Sending them all to one backend and then distributing them to different vendors / open sources gives you a lot of power.

Running your collector allows you to ship data anywhere and change it as you see fit. For example, you can start with one vendor and flip it in a simple configuration change. You can even run multiple POCs with vendors and open sources at the same time quite easily. The price is operational.

The Bottom Line

Here is a quick summary of our deployment options:

1. Use a vendor’s distro.
2. Use the OpenTelemetry native SDK, and send data directly to the vendor’s collector.
3. Use an OpenTelemetry SDK, send data to your collector, and then to the collector provided by the vendor.

It all mainly comes down to whether you have the capacity and resources to set up and manage all the components on your own – in this case, you should give the open-source path a try.