DEV Community

Aspecto profile image Tom Weiss
Tom Weiss for Aspecto

Posted on • Updated on • Originally published at aspecto.io

Distributed Tracing for Kafka with OpenTelemetry in Node

Distributed Tracing for Kafka with OpenTelemetry in Node

In this guide, you will learn how to run OpenTelemetry in Node to generate spans for different Kafka operations (e.g., consume and produce) and ultimately visualize your traces. We will also touch on the basics of Kafka, OpenTelemetry, distributed tracing, and how they all play out together.

If you’re already familiar with Kafka and OpenTelemetry, feel free to jump right to the practical part of this guide.

What to Expect

What is Apache Kafka?

Kafka is an open-source distributed event stream platform, where we can send events to topics and consume them. You can learn more about Kafka here.

What is OpenTelemetry?

OpenTelemetry is an open-source project, a collection of APIs and SDKs. Led by the CNCF (Cloud Native Computing Foundation, same foundation responsible for Kubernetes), it allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

In a distributed software world, messaging systems (like Kafka) are key to enabling communication between independent microservices, making it easier to build complex systems.

But understanding the path each message has gone through, where it goes, why, and when is as critical as it is complicated.

Users need a way to visualize and troubleshoot the complexity of their distributed architecture.

OpenTelemetry is our tool to collect data from the different transactions within our services and components (including various message brokers like Kafka) and understand our software’s performance and behavior.

For this specific guide, we will focus on distributed traces, and you can take a deeper dive into OpenTelemetry in this short guide.

What is Distributed Tracing?

Distributed tracing defines spans and traces. A trace tracks the progression of requests across the services, message brokers, and other components in our system.

A trace is a tree of spans. A span is a characterization of an activity/process that occurred between our services. For example, a call to a database, external service, sending messages to a Kafka topic, etc.

Traces inform us of the length of each request, which components and services it interacted with, the latency introduced during each step, and other valuable information about the nature of that activity.

OpenTelemetry Tracing Spans and Traces

What is OpenTelemetry Instrumentation?

Instrumentation is a piece of code responsible for generating spans (which make up traces) and sending them to a dedicated backend of your choice. Later, depending on the tool you’re using, the end goal is to visualize them and leverage that visualization for troubleshooting our system.

We do that by using the SDKs provided by OpenTelemetry, which bundles useful instrumentations.

OpenTelemetry includes instrumentations for various libraries in different languages, including a Kafka instrumentation responsible for creating Kafka spans.

How to use Kafka with OpenTelemetry in Node

By now you should have all the theory you need, so let’s start writing some code. A note before we begin – This guide assumes you are running Kafka in your local machine – If you don’t, visit this quick guide (it shows how to run Kafka with docker in the beginning).

The Practical Part

First, let’s create a brand new Express app & install relevant packages:

npx express-generator kafkaotel
npm install
npm install kafkajs
npm install @opentelemetry/exporter-collector
npm install @opentelemetry/auto-instrumentations-node
npm install @opentelemetry/sdk-node @opentelemetry/api opentelemetry-instrumentation-kafkajs
Enter fullscreen mode Exit fullscreen mode

Now create a tracing file that would create spans and traces for Kafka operations (like produce and consume). For this we use the opentelemetry kafkajs instrumentation:

/* tracing.js */
const { KafkaJsInstrumentation } = require('opentelemetry-instrumentation-kafkajs');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
// Require dependencies
const opentelemetry = require("@opentelemetry/sdk-node");
const tracerProvider = new NodeTracerProvider({
 // be sure to disable old plugin
 plugins: {
   kafkajs: { enabled: false, path: 'opentelemetry-plugin-kafkajs' }
 }
});
const sdk = new opentelemetry.NodeSDK({
 tracerProvider,
 traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
 instrumentations: [
   new KafkaJsInstrumentation({
     // see kafkajs instrumentation docs for available configuration
   })
 ]
});
sdk.start()
Enter fullscreen mode Exit fullscreen mode

Let’s update package.json to require the tracing file so that opentelemetry would be able to instrument:

"scripts": {
 "start": "node --require './tracing.js' ./bin/www"
},
Enter fullscreen mode Exit fullscreen mode

Now let’s update our main endpoint to produce and consume Kafka messages:

const express = require('express');
const router = express.Router();
const { Kafka } = require('kafkajs');
const kafka = new Kafka({
 clientId: 'my-app',
 brokers: ['localhost:9092'],
})
const sendKafkaMessage = async () => {
 const producer = kafka.producer();
 await producer.connect();
 await producer.send({
   topic: 'test-topic',
   messages: [
     { value: 'Hello KafkaJS user!' },
   ],
 })
 await producer.disconnect();
}
const consumeMessages = async () => {
 const consumer = kafka.consumer({ groupId: 'test-group' });
 await consumer.connect();
 await consumer.subscribe({ topic: 'test-topic', fromBeginning: true });
 await consumer.run({
   eachMessage: async ({ topic, partition, message }) => {
     console.log({
       value: message.value.toString(),
     })
   },
 })
}
/* GET home page. */
router.get('/', async function(req, res, next) {
 await sendKafkaMessage();
 await consumeMessages();
 res.render('index', { title: 'Express' });
});
module.exports = router;
Enter fullscreen mode Exit fullscreen mode

Now you can run npm start and run go to localhost:3000 in your favorite browser.

Then you can see the producer span in the console like this one:

{
  traceId: '003cd8ece100e4bed2d05867d3a98dc0',
  parentId: undefined,
  name: 'test-topic',
  id: 'c83af592be261547',
  kind: 3,
  timestamp: 1646386704369183,
  duration: 28633,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'test-topic',
    'messaging.destination_kind': 'topic'
  },
  status: { code: 0 },
  events: []
}
Enter fullscreen mode Exit fullscreen mode

And also the consumer span:

{
  traceId: '003cd8ece100e4bed2d05867d3a98dc0',
  parentId: 'c83af592be261547',
  name: 'test-topic',
  id: '875ea3c9e2b0dbdf',
  kind: 4,
  timestamp: 1646386726625784,
  duration: 562,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'test-topic',
    'messaging.destination_kind': 'topic',
    'messaging.operation': 'process'
  },
  status: { code: 0 },
  events: []
}
Enter fullscreen mode Exit fullscreen mode

And that’s that! You now know how to use Kafka and OpenTelemetry together, and can create spans for Kafka operations that happen in your microservices.

So far we only printed outputs to our console. But OpenTelemetry has a lot to offer in terms of visualization, so let’s see how you can achieve that.

Visualizing OpenTelemetry Traces in Node

For the purposes of this guide, I chose to use Aspecto as my visualization tool. This is because Aspecto provides built-in support for visualizing messaging systems like Kafka (and, of course, any other part of our microservice architectures).

Before jumping in, you can try it yourself with the free-forever plan that has no limited features.

Give this Live Playground a try to get a better idea.

Here’s how it would look at the end of the process:

Kafka message with OpenTelemetry in Node

You can also click on a specific node and see tracing data:

Aspecto UI seeing OpenTelemetry tracing data in Node

You can follow along by creating a free account, then, obtain your Aspecto API key, and modify your tracing.js file:

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { KafkaJsInstrumentation } = require("opentelemetry-instrumentation-kafkajs");
const provider = new NodeTracerProvider({
 resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: 'kafka-service' // service name is required
 }),
});
provider.register();
provider.addSpanProcessor(
 new SimpleSpanProcessor(
   new CollectorTraceExporter({
     url: 'https://otelcol.aspecto.io/v1/trace',
     headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
     }
   })
 )
);
registerInstrumentations({
 instrumentations: [
   // add other instrumentations here for packages your app uses
   new KafkaJsInstrumentation({})
 ],
});
Enter fullscreen mode Exit fullscreen mode

Then run the app with your Aspecto api key, like this:

ASPECTO_API_KEY=YOUR_API_KEY node --require './tracing.js' ./bin/www
Enter fullscreen mode Exit fullscreen mode

When you go to localhost:3000 with a browser, it would produce and consume a Kafka message and send corresponding spans to aspecto. Allow a minute or 2 and you could see it under the trace search in aspecto, just like the above picture shows.

If you want to instrument libraries other than Kafka, update your tracing file like this(using the node auto instrumentations library that enables a variety of instrumentations by default):

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { KafkaJsInstrumentation } = require("opentelemetry-instrumentation-kafkajs");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const provider = new NodeTracerProvider({
 resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: 'kafka-service' // service name is required
 }),
});
provider.register();
provider.addSpanProcessor(
 new SimpleSpanProcessor(
   new CollectorTraceExporter({
     url: 'https://otelcol.aspecto.io/v1/trace',
     headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
     }
   })
 )
);
registerInstrumentations({
 instrumentations: [
   getNodeAutoInstrumentations(),
   new KafkaJsInstrumentation({})
 ],
});
Enter fullscreen mode Exit fullscreen mode

But note: for Node, it’s recommended to use the Aspecto SDK which uses OpenTelemetry under the hood but also supports collecting actual payloads.

If you take the exact same service from scratch – without all the OpenTelemetry installations (make sure uninstall them to avoid any version conflicts), and run:

npm install @aspecto/opentelemetry
Enter fullscreen mode Exit fullscreen mode

Then add the following code at the top of your app.js file:

require('@aspecto/opentelemetry')({
 aspectoAuth: process.env.ASPECTO_API_KEY
});
Enter fullscreen mode Exit fullscreen mode

And now you can even see the payload of the Kafka message:

Payload of the Kafka message with OpenTelemetry in Node

P.S. If you prefer to use an open-source to visualize your traces, you can try Jaeger Tracing. You can follow this guide on deploying Jaeger on AWS and make sure to scroll to the part on sending traces to Jaeger with the OpenTelemetry SDK (In NodeJS). Keep in mind that it may take a bit more time to set up the required environment.

If you’re not familiar with Jaeger and get a deeper understanding of it, we recommend visiting this guide.

I hope this has been a useful guide for you to learn how to use OpenTelemetry in Node to generate traces for Kafka operations. Feel free to reach out if you have any questions!

TroubleShooting OpenTelemetry Node Installation Issues

It is not uncommon to encounter several issues when trying to install OpenTelemetry. More specifically, trying to install OpenTelemetry in your Node application and not seeing any traces or some expected spans are missing.

To help you avoid these common pitfalls, we recommend going over this in-depth checklist of issues and solutions. We cover the most common mistakes and show you how to solve them.

Additional OpenTelemetry Resources

If you want to go deeper into learning about OpenTelemetry, visit The OpenTelemetry Bootcamp video library. It’s a free and vendor-neutral, in-depth course that will help you master OpenTelemetry. We cover everything from the very basics to security and deploying in production.

Top comments (0)

Read next

bugmagnet profile image

Awesome COBOL!

Bruce Axtens -

opensourcee profile image

The never-ending all-in-one software game...

OpenSource -

nicosleonardo9 profile image

🤿 Looking for Waterpolo API for Research Consultant Project? 🌊

Nicos -

aralroca profile image

SPA-Like Navigation Preserving Web Component State

Aral Roca -