DEV Community: Tom Weiss

Distributed Tracing for Kafka with OpenTelemetry in Python

Tom Weiss — Mon, 05 Sep 2022 07:35:03 +0000

In this tutorial, I will cover Apache Kafka, OpenTelemetry, and how they play out together with practical examples in Python from 0 to 1.

You will learn how to enable OpenTelemetry tracing in Python to generate spans and visualize traces for various Kafka operations.

What to Expect

What is Apache Kafka?
What is OpenTelemetry?
OpenTelemetry Traces
What are OpenTelemetry Instrumentations?
How to Use Kafka with OpenTelemetry in Python
Visualizing OpenTelemetry Tracing Data

This guide has two parts:

A “hello world” kind of intro to OpenTelemetry and Kafka in Python in which you will end up seeing the outputs in a console.
In the second part, you will learn to visualize traces to troubleshoot your microservices.

To generate spans for the different Kafka operations (e.g., consume and produce), we will use OpenTelemetry libraries.

If you’re already familiar with Kafka and OpenTelemetry, feel free to jump right to the practical part of this guide.

What is Apache Kafka?

Kafka is an open-source distributed event stream platform, where we can send events to topics and consume them. You can learn more about Kafka here.

What is OpenTelemetry?

In a world where microservices are becoming a standard architecture and distributed messaging systems like Kafka play a core part in enabling the independent microservices to communicate, the world needed a way to visualize and troubleshoot the complexity of a distributed architecture.

Led by the CNCF (Cloud Native Computing Foundation), OpenTelemetry is an open-source project, a set of APIs and SDKs, that allows us to collect, export, and generate traces, logs, and metrics.

With OpenTelemetry, we gather data from the events happening within the different components in our systems (including all sorts of message brokers like Kafka), which ultimately help us understand our software’s performance and behavior and visualize our microservices architecture.

For this specific guide, we will focus on traces, you can learn more about OpenTelemetry in this friendly guide for developers.

OpenTelemetry Traces

A trace tracks the progression of a single request, as it is handled by the different microservices/ moving parts.

The atomic unit of work in a trace is called a span, for example, a call to a database, external service, send message to Kafka topic, etc.

Each trace contains a collection of 1 or more spans and more information like how much time the entire trace took to complete.

What are OpenTelemetry Instrumentations?

So now you know what traces and spans mean, but how do you create them?

That can be done using the SDKs provided to us by OpenTelemetry, which bundles useful instrumentations.

What’s an instrumentation? A piece of code that is responsible for creating spans and traces and sending them to some backend, for later visualizing them.

OpenTelemetry contains instrumentations for a lot of different libraries, including ones for Kafka.

How to Use Kafka with OpenTelemetry in Python

By now you should have all the theory you need, so let’s start writing some code.

This guide assumes you are running Kafka in your local (if you don’t, visit this quick guide).

The Practical Part

First, let’s install the relevant libraries:

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install opentelemetry-instrumentation-kafka-python
pip install kafka-python

Then, let’s add our tracing.py file:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    SimpleSpanProcessor,
    ConsoleSpanExporter,
)
from opentelemetry.instrumentation.kafka import KafkaInstrumentor
def instrument(*args, **kwargs):
    provider = TracerProvider()
    simple_processor = SimpleSpanProcessor(ConsoleSpanExporter())
    provider.add_span_processor(simple_processor)
    trace.set_tracer_provider(provider)
    KafkaInstrumentor().instrument()

This file is responsible for using OpenTelemetry SDKs to enable the creation of spans and traces.

Notice we have a console span exporter which means we will be sending the spans to the console output.

As you can see we use the KafkaInstrumentor to create spans for Kafka-related operations, like consume and produce.

This is all the OpenTelemetry code we need to create Kafka spans.

Let’s create the producer file:

from tracing import instrument
instrument()
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
for _ in range(1):
   producer.send('foobar', b'some_message_bytes')
   producer.flush()

And create the consumer file:

from tracing import instrument
instrument()
from kafka import KafkaConsumer
consumer = KafkaConsumer('foobar', bootstrap_servers='localhost:9092')
for msg in consumer:
   print(msg)

Notice that both files have these 2 lines:

from tracing import instrument
instrument()

This is the calling of the instrument function and enables the creation of the spans when we run the consumer and producer files using the following commands:

python producer.py
python consumer.py

Since we use console exporter – we can see the spans of the consumer and producer on the console output:

{
    "name": "foobar receive",
    "context": {
        "trace_id": "0x436c0e9807b2f61b4a72bc63e5205954",
        "span_id": "0xdda310172f7297cd",
        "trace_state": "[]"
    },
    "kind": "SpanKind.CONSUMER",
    "parent_id": "0x711a6a5d5bb977d8",
    "start_time": "2022-03-02T10:40:36.695519Z",
    "end_time": "2022-03-02T10:40:36.720864Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "messaging.system": "kafka",
        "messaging.destination": "foobar",
        "messaging.kafka.partition": 0,
        "messaging.url": "\"localhost:9092\""
    },
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.9.1",
        "service.name": "unknown_service"
    }
}

{
    "name": "foobar send",
    "context": {
        "trace_id": "0x436c0e9807b2f61b4a72bc63e5205954",
        "span_id": "0x711a6a5d5bb977d8",
        "trace_state": "[]"
    },
    "kind": "SpanKind.PRODUCER",
    "parent_id": null,
    "start_time": "2022-03-02T10:40:36.609135Z",
    "end_time": "2022-03-02T10:40:36.610410Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "messaging.system": "kafka",
        "messaging.destination": "foobar",
        "messaging.url": "\"localhost:9092\""
    },
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.9.1",
        "service.name": "unknown_service"
    }
}

But I have a feeling that you want to get a better visualization than just a console log. If you do – read on 🙂

Visualizing OpenTelemetry Tracing Data

For this article, I will be using Aspecto to visualize my traces. You can follow along by quickly creating a free account.

Aspecto has built-in support for visualizing async messaging platforms like Kafka so it fits perfectly with this guide’s purpose.

Once you’re signed in, you get an API key and all you need to do is change a few lines of code, environment variables and install some packages. I’ll be showing you how to do that below.

The end result should look like this:

First, let’s install:

pip install opentelemetry-exporter-otlp-proto-grpc

Then let’s modify our tracing.py file so that it uses the otlp exporter instead of the console exporter, to send data to Aspecto:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
   BatchSpanProcessor,
   SimpleSpanProcessor,
   ConsoleSpanExporter,
)
from opentelemetry.instrumentation.kafka import KafkaInstrumentor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
def instrument(*args, **kwargs):
   provider = TracerProvider()
   simple_processor = SimpleSpanProcessor(OTLPSpanExporter())
   provider.add_span_processor(simple_processor)
   trace.set_tracer_provider(provider)
   KafkaInstrumentor().instrument()

To run the consumer

OTEL_SERVICE_NAME=kafka-consumer OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 OTEL_EXPORTER_OTLP_HEADERS=Authorization=YOUR_ASPECTO_API_KEY_HERE python consumer.py

To run the producer

OTEL_SERVICE_NAME=kafka-producer OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 OTEL_EXPORTER_OTLP_HEADERS=Authorization=YOUR_ASPECTO_API_KEY_HERE python producer.py

And that’s it! When you enter the Aspecto platform and click on Trace Search, after a minute or two you should be able to see your traces.

Feel free to click on the chat button and contact us, we’d be glad to help you figure everything out and answer any questions you may have.

P.S. If you prefer to use an open-source to visualize your traces, check out Jaeger Tracing. Follow this guide to learn how to use OpenTelemetry to send traces to Jaeger. Keep in mind that it takes a bit more time to set up the required environment.

Hope this has been a useful guide for you, feel free to reach out if you have any questions!

Distributed Tracing for Kafka with OpenTelemetry in Node

Tom Weiss — Tue, 30 Aug 2022 09:10:17 +0000

In this guide, you will learn how to run OpenTelemetry in Node to generate spans for different Kafka operations (e.g., consume and produce) and ultimately visualize your traces. We will also touch on the basics of Kafka, OpenTelemetry, distributed tracing, and how they all play out together.

If you’re already familiar with Kafka and OpenTelemetry, feel free to jump right to the practical part of this guide.

What to Expect

What is Apache Kafka
What is OpenTelemetry
What is Distributed Tracing
What is OpenTelemetry Instrumentation
How to use Kafka with OpenTelemetry in Node
Visualizing OpenTelemetry Traces
TroubleShooting OpenTelemetry Node Installation Issues
Additional OpenTelemetry Resources

What is Apache Kafka?

Kafka is an open-source distributed event stream platform, where we can send events to topics and consume them. You can learn more about Kafka here.

What is OpenTelemetry?

OpenTelemetry is an open-source project, a collection of APIs and SDKs. Led by the CNCF (Cloud Native Computing Foundation, same foundation responsible for Kubernetes), it allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

In a distributed software world, messaging systems (like Kafka) are key to enabling communication between independent microservices, making it easier to build complex systems.

But understanding the path each message has gone through, where it goes, why, and when is as critical as it is complicated.

Users need a way to visualize and troubleshoot the complexity of their distributed architecture.

OpenTelemetry is our tool to collect data from the different transactions within our services and components (including various message brokers like Kafka) and understand our software’s performance and behavior.

For this specific guide, we will focus on distributed traces, and you can take a deeper dive into OpenTelemetry in this short guide.

What is Distributed Tracing?

Distributed tracing defines spans and traces. A trace tracks the progression of requests across the services, message brokers, and other components in our system.

A trace is a tree of spans. A span is a characterization of an activity/process that occurred between our services. For example, a call to a database, external service, sending messages to a Kafka topic, etc.

Traces inform us of the length of each request, which components and services it interacted with, the latency introduced during each step, and other valuable information about the nature of that activity.

What is OpenTelemetry Instrumentation?

Instrumentation is a piece of code responsible for generating spans (which make up traces) and sending them to a dedicated backend of your choice. Later, depending on the tool you’re using, the end goal is to visualize them and leverage that visualization for troubleshooting our system.

We do that by using the SDKs provided by OpenTelemetry, which bundles useful instrumentations.

OpenTelemetry includes instrumentations for various libraries in different languages, including a Kafka instrumentation responsible for creating Kafka spans.

How to use Kafka with OpenTelemetry in Node

By now you should have all the theory you need, so let’s start writing some code. A note before we begin – This guide assumes you are running Kafka in your local machine – If you don’t, visit this quick guide (it shows how to run Kafka with docker in the beginning).

The Practical Part

First, let’s create a brand new Express app & install relevant packages:

npx express-generator kafkaotel
npm install
npm install kafkajs
npm install @opentelemetry/exporter-collector
npm install @opentelemetry/auto-instrumentations-node
npm install @opentelemetry/sdk-node @opentelemetry/api opentelemetry-instrumentation-kafkajs

Now create a tracing file that would create spans and traces for Kafka operations (like produce and consume). For this we use the opentelemetry kafkajs instrumentation:

/* tracing.js */
const { KafkaJsInstrumentation } = require('opentelemetry-instrumentation-kafkajs');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
// Require dependencies
const opentelemetry = require("@opentelemetry/sdk-node");
const tracerProvider = new NodeTracerProvider({
 // be sure to disable old plugin
 plugins: {
   kafkajs: { enabled: false, path: 'opentelemetry-plugin-kafkajs' }
 }
});
const sdk = new opentelemetry.NodeSDK({
 tracerProvider,
 traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
 instrumentations: [
   new KafkaJsInstrumentation({
     // see kafkajs instrumentation docs for available configuration
   })
 ]
});
sdk.start()

Let’s update package.json to require the tracing file so that opentelemetry would be able to instrument:

"scripts": {
 "start": "node --require './tracing.js' ./bin/www"
},

Now let’s update our main endpoint to produce and consume Kafka messages:

const express = require('express');
const router = express.Router();
const { Kafka } = require('kafkajs');
const kafka = new Kafka({
 clientId: 'my-app',
 brokers: ['localhost:9092'],
})
const sendKafkaMessage = async () => {
 const producer = kafka.producer();
 await producer.connect();
 await producer.send({
   topic: 'test-topic',
   messages: [
     { value: 'Hello KafkaJS user!' },
   ],
 })
 await producer.disconnect();
}
const consumeMessages = async () => {
 const consumer = kafka.consumer({ groupId: 'test-group' });
 await consumer.connect();
 await consumer.subscribe({ topic: 'test-topic', fromBeginning: true });
 await consumer.run({
   eachMessage: async ({ topic, partition, message }) => {
     console.log({
       value: message.value.toString(),
     })
   },
 })
}
/* GET home page. */
router.get('/', async function(req, res, next) {
 await sendKafkaMessage();
 await consumeMessages();
 res.render('index', { title: 'Express' });
});
module.exports = router;

Now you can run npm start and run go to localhost:3000 in your favorite browser.

Then you can see the producer span in the console like this one:

{
  traceId: '003cd8ece100e4bed2d05867d3a98dc0',
  parentId: undefined,
  name: 'test-topic',
  id: 'c83af592be261547',
  kind: 3,
  timestamp: 1646386704369183,
  duration: 28633,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'test-topic',
    'messaging.destination_kind': 'topic'
  },
  status: { code: 0 },
  events: []
}

And also the consumer span:

{
  traceId: '003cd8ece100e4bed2d05867d3a98dc0',
  parentId: 'c83af592be261547',
  name: 'test-topic',
  id: '875ea3c9e2b0dbdf',
  kind: 4,
  timestamp: 1646386726625784,
  duration: 562,
  attributes: {
    'messaging.system': 'kafka',
    'messaging.destination': 'test-topic',
    'messaging.destination_kind': 'topic',
    'messaging.operation': 'process'
  },
  status: { code: 0 },
  events: []
}

And that’s that! You now know how to use Kafka and OpenTelemetry together, and can create spans for Kafka operations that happen in your microservices.

So far we only printed outputs to our console. But OpenTelemetry has a lot to offer in terms of visualization, so let’s see how you can achieve that.

Visualizing OpenTelemetry Traces in Node

For the purposes of this guide, I chose to use Aspecto as my visualization tool. This is because Aspecto provides built-in support for visualizing messaging systems like Kafka (and, of course, any other part of our microservice architectures).

Before jumping in, you can try it yourself with the free-forever plan that has no limited features.

Give this Live Playground a try to get a better idea.

Here’s how it would look at the end of the process:

You can also click on a specific node and see tracing data:

You can follow along by creating a free account, then, obtain your Aspecto API key, and modify your tracing.js file:

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { KafkaJsInstrumentation } = require("opentelemetry-instrumentation-kafkajs");
const provider = new NodeTracerProvider({
 resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: 'kafka-service' // service name is required
 }),
});
provider.register();
provider.addSpanProcessor(
 new SimpleSpanProcessor(
   new CollectorTraceExporter({
     url: 'https://otelcol.aspecto.io/v1/trace',
     headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
     }
   })
 )
);
registerInstrumentations({
 instrumentations: [
   // add other instrumentations here for packages your app uses
   new KafkaJsInstrumentation({})
 ],
});

Then run the app with your Aspecto api key, like this:

ASPECTO_API_KEY=YOUR_API_KEY node --require './tracing.js' ./bin/www

When you go to localhost:3000 with a browser, it would produce and consume a Kafka message and send corresponding spans to aspecto. Allow a minute or 2 and you could see it under the trace search in aspecto, just like the above picture shows.

If you want to instrument libraries other than Kafka, update your tracing file like this(using the node auto instrumentations library that enables a variety of instrumentations by default):

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { KafkaJsInstrumentation } = require("opentelemetry-instrumentation-kafkajs");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const provider = new NodeTracerProvider({
 resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: 'kafka-service' // service name is required
 }),
});
provider.register();
provider.addSpanProcessor(
 new SimpleSpanProcessor(
   new CollectorTraceExporter({
     url: 'https://otelcol.aspecto.io/v1/trace',
     headers: {
       // Aspecto API-Key is required
       Authorization: process.env.ASPECTO_API_KEY
     }
   })
 )
);
registerInstrumentations({
 instrumentations: [
   getNodeAutoInstrumentations(),
   new KafkaJsInstrumentation({})
 ],
});

But note: for Node, it’s recommended to use the Aspecto SDK which uses OpenTelemetry under the hood but also supports collecting actual payloads.

If you take the exact same service from scratch – without all the OpenTelemetry installations (make sure uninstall them to avoid any version conflicts), and run:

npm install @aspecto/opentelemetry

Then add the following code at the top of your app.js file:

require('@aspecto/opentelemetry')({
 aspectoAuth: process.env.ASPECTO_API_KEY
});

And now you can even see the payload of the Kafka message:

P.S. If you prefer to use an open-source to visualize your traces, you can try Jaeger Tracing. You can follow this guide on deploying Jaeger on AWS and make sure to scroll to the part on sending traces to Jaeger with the OpenTelemetry SDK (In NodeJS). Keep in mind that it may take a bit more time to set up the required environment.

If you’re not familiar with Jaeger and get a deeper understanding of it, we recommend visiting this guide.

I hope this has been a useful guide for you to learn how to use OpenTelemetry in Node to generate traces for Kafka operations. Feel free to reach out if you have any questions!

TroubleShooting OpenTelemetry Node Installation Issues

It is not uncommon to encounter several issues when trying to install OpenTelemetry. More specifically, trying to install OpenTelemetry in your Node application and not seeing any traces or some expected spans are missing.

To help you avoid these common pitfalls, we recommend going over this in-depth checklist of issues and solutions. We cover the most common mistakes and show you how to solve them.

Additional OpenTelemetry Resources

If you want to go deeper into learning about OpenTelemetry, visit The OpenTelemetry Bootcamp video library. It’s a free and vendor-neutral, in-depth course that will help you master OpenTelemetry. We cover everything from the very basics to security and deploying in production.

OpenTelemetry Collector: A Friendly Guide for Devs

Tom Weiss — Wed, 24 Aug 2022 13:19:53 +0000

In this guide, you will learn everything you need to know about the OpenTelemetry Collector.

Before reading about it myself, the collector felt like a complex beast. But as I was learning, I realized it was not so complicated after all.

In a nutshell, the collector receives telemetry data and sends it to wherever you configure it. It can also manipulate the data as you see fit. These are the key concepts you need to understand.

We will later go through a practical example where we configure the collector and visualize data.

Now that you know this, we can further elaborate.

What to Expect

How does the OpenTelemetry Collector work?
- Receivers
- Processors
- Exporters
The Architecture of Running an OpenTelemetry Collector
- Gateway
- Agent
OpenTelemetry Collector Deployment Methods
Configuring OpenTelemetry Collector Gateway with Jaeger for Visualization
Configuring Your Collector to Send Data to an Observability Vendor
OpenTelemetry Collector Extensions

How does the OpenTelemetry Collector work?

Images speak louder than words. So here’s one that’d help you figure this one out:

So what is happening here?

Looking at data ingestion to the OTel collector, you should know you can send logs, metrics, and traces using the OTLP protocol, which supports those data types.

The OpenTelemetry collector has three main components:

Receivers

As their name suggests, they are in charge of receiving spans from our span emitting applications. We can use a native OTEL SDK to create spans and export them to the receiver that listens for calls in a specified port on the collector.

We can configure our receivers to accept both gRPC and HTTP protocols.

For a list of receivers, you can use this link.

Processors

The processor’s goal is to enable us to manipulate the data the collector receives right before we export it to our DB or backend. Common use cases:

Sampling

Assume we have a lot of data we do not actually want to store in our backend. We can define a processor responsible for implementing the sampling logic and deciding which spans would be sent to the backend/DB.

Note – the collector still receives all traces using the receiver, but then the processor comes and picks the relevant ones.

Data alteration

Say you want to remove sensitive data before it reaches the distributed tracing backend (to avoid tokens or other info from leaking).

You could use a processor to do this for you, but note that this would be a processor that has to be marked as such that is able to mutate your data, thus causing the creation of a copy of your entire data for this processor. Meaning, that every processor marked as mutatesData: true would work with a copy of the entire data as it received it.

That is not a problem in most cases, but you should be aware of it. It is called data ownership, and you can read more about it here.

You could also use this to add attributes (for example, calculate new ones based on others).

Exporting metrics from spans

Using the Span Metrics Processor, we can aggregate data from spans into metrics and then export it to a relevant backend. For example, this processor could gather data about the HTTP path and response status code, enabling you to see which routes have the most 500 errors. You can read more about this here.

You could take a look at the OTEL Collector Contrib to see a list of additional processors you can use.

Exporters

The goal of the exporters is to take the telemetry data as it is represented in the collector (OTEL data), convert it to a different format when needed (like Jaeger), and then send it to the endpoint you define. The sending part is done using the OTLP format, over either HTTP or gRPC.

You can export directly to Elasticsearch, Jaeger, Zipkin, and other vendors to enable distributed services visualization. Visit the OTEL Collector Contrib to see a list of additional exporters you can use.

Why should you use the OpenTelemetry collector?

From the capabilities described above, you can understand that the collector is helpful in the following scenarios:

You want to control the data leaving your company
You want to convert data before it leaves for another system
You want to add abilities you would not otherwise have, e.g., extracting metrics from spans and other data manipulations.
You want to send your data to multiple destinations as easily as adding a few lines of configuration (like different vendors)
You want to control the sampling levels yourself (even though some vendors like Aspecto do let you do this without the need to set up your own collector).

But all this goodness comes with a price: you must know that maintaining a collector has its own complexities to take care of, like security, infrastructure management & cost (which could go up or down depending on your needs), etc.

If you want to go about it by yourself, the next part talks about the architecture of running an OTEL collector.

The Architecture of Running an OpenTelemetry Collector

The collector binary has two modes of running:

As an Agent
As a Gateway

Gateway

When we use the collector as a Gateway, it is run as a standalone machine independent from any other services or serves as a central point at which the telemetry data of an entire distributed architecture comes in.

It provides more advanced capabilities than the agent, such as tail-based sampling (in which the collector only exports spans that have errors, for example).

It can also help with API token management and reduce the number of egress points required to send data.

You should know that each collector instance is independent. This means that if you want to scale, you could set up a “gateway cluster” with various collector gateway instances behind a load balancer.

Agent

The agent runs at the same host as your app server (whether it runs as a container or not).

Advantages:

It can add insightful information about the host (IP, hostname, etc.).
Receives the data faster since there’s no DNS resolving to do (it’s just localhost).
Offload responsibilities otherwise belonging to the app like batching, compression, retry, and more.

After doing all the above, it would send the data to the Gateway Collector. For more information about all the above, visit Getting Started | OpenTelemetry Collector.

Now that you’re familiar with the two modes of operation of the collector, we can talk about deployment methods.

OpenTelemetry Collector Deployment Methods

I would suggest two options:

Option 1

All the microservices of your app interact directly with a single collector instance as a collector gateway:

Option 2

Each microservice writes to an agent running at the same host as the microservice. Then, the agent writes to a central collector gateway:

Despite the benefits of using the agent mode (listed above), I would only recommend this approach when you’ve reached a level of maturity with your app and OpenTelemetry, as maintaining it introduces infrastructure overhead. In the beginning, you could probably do just as well with the simple single collector mode.

And that’s the reason why I will be showing a practical example of the first option (gateway).

Configuring OpenTelemetry Collector Gateway with Our App and Jaeger for Visualization

Here’s a diagram of what we will be building:

What we want to achieve: we run a collector and Jaeger using docker-compose, set up OTLP receivers, add some data using a processor, and export to Jaeger for visualization.

Setting up the collector gateway

First, let’s set up a configuration file for our collector.

Collector-gateway.yaml:

receivers:
 otlp:
   protocols:
     http:
       endpoint: 0.0.0.0:4318
     grpc:
       endpoint: 0.0.0.0:4317
processors:
 batch:
   timeout: 1s
 resource:
   attributes:
     - key: test.key
       value: "test-value"
       action: insert
exporters:
 logging:
   loglevel: info
 jaeger:
   endpoint: jaeger-all-in-one:14250
   insecure: true
extensions:
 health_check:
 pprof:
   endpoint: :1888
 zpages:
   endpoint: :55679
service:
 extensions: [pprof, zpages, health_check]
 pipelines:
   traces:
     receivers: [otlp]
     processors: [batch, resource]
     exporters: [logging, jaeger]

So what do we see in the file above?

First, we set up OTLP receivers and told our collector to add HTTP & gRPC endpoints at ports 4318 & 4317.

Then, we set up a batch processor that batches up the spans together and every 1 second sends the batch forward. In production, you would want more than 1 second, but I set this here to 1 second for instant feedback in Jaeger.

Then we add another processor that inserts a new attribute on each span that passed through it. The key: “test.key”, the value: “test-value”.

After that, we define two exporters: one to log everything to the console and the other to export the spans to Jaeger.

We do it insecurely now, but you would not want to do that for production purposes.

Jaeger-all-in-one is the container’s name in docker-compose. You will see its setup below.

Ignore the extensions part, for now (we will discuss it later).

The service is the orchestrator of all the above. We set up a tracing pipeline (though you also could add a metrics/logs one). In our tracing pipeline, we tell the collector to use all the definitions from above.

Now that we have the collector config ready – let’s use it in the collector.

Docker-compose.yaml:

version: "2"
services:
 # Jaeger
 jaeger-all-in-one:
   image: jaegertracing/all-in-one:latest
   ports:
     - "16686:16686"
     - "14268"
     - "14250"
 # Collector
 collector-gateway:
   image: otel/opentelemetry-collector:0.29.0
   volumes:
     - ./collector-gateway.yaml:/etc/collector-gateway.yaml
   command: [ "--config=/etc/collector-gateway.yaml" ]
   ports:
     - "1888:1888"   # pprof extension
     - "13133:13133" # health_check extension
     - "4317:4317"        # OTLP gRPC receiver
     - "4318:4318"        # OTLP HTTP receiver
     - "55670:55679" # zpages extension
   depends_on:
     - jaeger-all-in-one

So we configure Jaeger to run in the relevant UI and data collection ports and call it Jaeger-all-in-one, as mentioned before.

Then, we define a collector-gateway container to use the otel collector binary and map the volume so that the container would take the otel config file we created above and use it.

The command below is where the collector takes that file from within the container.

Then you can see the ports for each receiver/extension and the dependency on the existence of the jaeger container.

Now let’s run with the following command:

docker compose up

We need to start sending data to the collector and then see it from Jaeger.

Let’s set up a simple nodejs-express app with OpenTelemetry enabled for this.

Let’s use an express-generator to generate our initial code:

npx express-generator otel-collector-sender

This structure should have been automatically created for you:

Let’s install the relevant dependencies:

npm install @opentelemetry/sdk-node @opentelemetry/api opentelemetry-instrumentation-express
 @opentelemetry/exporter-trace-otlp-http

Now your package.json should look like this:

{
 "name": "otel-collector-sender",
 "version": "0.0.0",
 "private": true,
 "scripts": {
   "start": "node ./bin/www"
 },
 "dependencies": {
   "@opentelemetry/api": "^1.0.4",
   "@opentelemetry/exporter-trace-otlp-http": "^0.28.0",
   "@opentelemetry/instrumentation-http": "^0.28.0",
   "@opentelemetry/sdk-node": "^0.28.0",
   "cookie-parser": "~1.4.4",
   "debug": "~2.6.9",
   "express": "~4.16.1",
   "http-errors": "~1.6.3",
   "jade": "~1.11.0",
   "morgan": "~1.9.1",
   "opentelemetry-instrumentation-express": "^0.27.1"
 }
}

Launch the app to see that it works as expected. If you run “npm start” and go to localhost:3000 you should see this, which means our Express app is up:

Create a tracing.js file to enable tracing

/* tracing.js */
// Require dependencies
const opentelemetry = require("@opentelemetry/sdk-node");
const { diag, DiagConsoleLogger, DiagLogLevel } = require('@opentelemetry/api');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { ExpressInstrumentation } = require('opentelemetry-instrumentation-express');
// For troubleshooting, set the log level to DiagLogLevel.DEBUG
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.INFO);
const exporter = new OTLPTraceExporter({
 // optional - url default value is http://localhost:55681/v1/traces
 url: 'http://localhost:4318/v1/traces',
 // optional - collection of custom headers to be sent with each request, empty by default
 headers: {},
});
const sdk = new opentelemetry.NodeSDK({
 traceExporter: exporter,
 instrumentations: [new ExpressInstrumentation()]
});
sdk.start()

Now, this code has to be run before our app does, so that traces could be created.

The way to do so is by modifying the start script in our package.json to contain -r:

{
 "name": "otel-collector-sender",
 "version": "0.0.0",
 "private": true,
 "scripts": {
   "start": "node -r ./tracing.js ./bin/www"
 },
 "dependencies": {
   "@opentelemetry/api": "^1.0.4",
   "@opentelemetry/exporter-trace-otlp-http": "^0.28.0",
   "@opentelemetry/instrumentation-http": "^0.28.0",
   "@opentelemetry/sdk-node": "^0.28.0",
   "cookie-parser": "~1.4.4",
   "debug": "~2.6.9",
   "express": "~4.16.1",
   "http-errors": "~1.6.3",
   "jade": "~1.11.0",
   "morgan": "~1.9.1",
   "opentelemetry-instrumentation-express": "^0.27.1"
 }
}

Restart the app and refresh your browser at localhost:3000.

Head over to Jaeger, refresh your page, and select unknown_service:node from the panel on your left. Then hit Find Traces, and you should see a list of traces that have gone from the OTEL JS SDK to the collector to Jaeger.

Would look like this:

Pretty nice, isn’t it?

Now you may ask – “but you promised to add some attributes, where are they?” – I am glad you asked that.

Let’s see the HTTP GET trace, for example, pick any span, and check out the process area:

At the bottom, we have a test.key and test-value just like we defined in the collector.

You could use this to add any data you want, not just test-value. And of course, you could use any other type of processor as mentioned above.

But now, you might ask, what if I want to use a vendor instead of Jaeger? Before I show you how to add a vendor into the loop, let’s talk about usage with vendors.

Usage with observability vendors

Since the OpenTelemetry collector works with the standard OpenTelemetry specification, it is vendor agnostic. You could configure the collector to send data to any vendor that supports OpenTelemetry data by switching or adding another exporter to your collector.

Configuring Your Collector to Send Data to an Observability Vendor

Modify your collector-gateway.yaml file to look like this:

receivers:
 otlp:
   protocols:
     http:
       endpoint: 0.0.0.0:4318
     grpc:
       endpoint: 0.0.0.0:4317
processors:
 batch:
   timeout: 1s
 resource:
   attributes:
     - key: test.key
       value: "test-value"
       action: insert
exporters:
 logging:
   loglevel: info
 jaeger:
   endpoint: jaeger-all-in-one:14250
   insecure: true
 otlphttp:
   endpoint: https://otelcol-fast.aspecto.io
   headers:
     Authorization: YOUR-ASPECTO-TOKEN

extensions:
 health_check:
 pprof:
   endpoint: :1888
 zpages:
   endpoint: :55679
service:
 extensions: [pprof, zpages, health_check]
 pipelines:
   traces:
     receivers: [otlp]
     processors: [batch, resource]
     exporters: [logging, jaeger, otlphttp]

At Aspecto, you can sign up for free and use our generous free-forever plan (no limited features).

Just grab your token and paste it where it says YOUR-ASPECTO-TOKEN (https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens).

Now re-run using docker-compose up and refresh your localhost:3000 browser page.

Then, in the Aspecto app, the traces would look like this:

Drill down into the trace below, it would look like this:

Below you can see the test.key: “test-value” we added.

It sent it both to Aspecto and Jaeger, and we have a high level of visibility this way 🙂

If you want to see payloads and not just standard otel data, you can use Aspecto Node SDK written on top of OTEL. It adds this and other capabilities.

OpenTelemetry Collector Extensions

This article wouldn’t be complete without talking about the extensions you’ve seen earlier.

OpenTelemetry collector extensions provide additional capabilities that are not provided in the standard collector. In addition, these abilities are not part of the logs, metrics, and traces pipeline.

Let’s talk about the extensions you’ve seen above:

health_check – responsible for responding to health check calls on behalf of the collector.
PProf – fetches the collector’s performance data
zPages – serves as an http endpoint that provides live debugging data about instrumented components.

For more information on the above, read here.

That would sum up everything you need to know to get started with the OpenTelemetry collector.

If you want to learn more about the collector and OpenTelemetry in general, watch our OpenTelemetry Bootcamp. It’s a free and vendor-neutral video series (6-episodes) that you can use as your OpenTelemetry playbook. It contains everything you need from the basics to production deployment.

Episode 1: OpenTelemetry Fundamentals
Episode 2: Integrate Your Code (logs, metrics, and traces)
Episode 3: Deploy to Production + Collector
Episode 4: Sampling and Dealing with High Volumes
Episode 5: Custom Instrumentation
Episode 6: Testing with OpenTelemetry

Get Started with OpenTelemetry Python: A Practical Guide

Tom Weiss — Wed, 19 Jan 2022 16:51:44 +0000

This is a practical guide that gives you just what you need to get started with OpenTelemetry in Python without any prior knowledge in OpenTelemetry.

Intro to OpenTelemetry

OpenTelemetry is a CNCF project (same folks responsible for Kubernetes), which, among other things, allows the collection of traces, logs, and metrics (also known as the three pillars of observability).

It enables us to instrument our distributed services, meaning, to collect data from the events that happen in our systems which ultimately help us understand our software’s performance and behavior.

OpenTelemetry has been widely covered in various posts – You can learn more about OpenTelemetry and distributed tracing here.

For this guide (and for using OpenTelemetry), here are the relevant terms you must be familiar with:

Span: The basic building block (I like to call it the “atom”) of OpenTelemetry.
A span is an action that occurs in our system, like a POST/GET request or a db.insert operation.

Trace: A trace is a tree of spans representing the progression of a single request as it is handled by the different services of your app.

Exporter: Once we have created a span we need to send it to some backend. It may be in memory, Jaeger, or even as console output. The exporter handles sending the data to our backend.

Manual / Automatic instrumentation:

Manual: manually creating a span inside the application code
Automatic: using instrumentation libraries (like pymongo for mongodb), to automatically create spans for us and send them to the backend through the exporter.

If you want to learn more terms, you can see the official documentation related to this here: https://opentelemetry.io/docs/concepts/data-sources/

If you want to learn about the advantages of tracing and their comparison to logs, check out this guide.

Hello World: OpenTelemetry Python

Create spans and see them in the console output:

Let’s begin by writing some simple code that creates manual spans and logs them to console output.

1.Start a new python project (python 3.6+ is supported by OpenTelemetry)

2.Make the following installs

pip install opentelemetry-api
pip install opentelemetry-sdk

3.Create a tracing.py file with 2 manual spans created – rootSpan and childSpan.

# tracing.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
   BatchSpanProcessor,
   ConsoleSpanExporter,
)

provider = TracerProvider()
processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("rootSpan"):
   with tracer.start_as_current_span("childSpan"):
           print("Hello world!")

Launch the tracing.py file and this is the output you should see – 2 manual spans:

{
    "name": "childSpan",
    "context": {
        "trace_id": "0x6a37f0f0678f07485a01ba001b1119b0",
        "span_id": "0x4c162caa4e6d10c4",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0xdaf18d32c6af7c38",
    "start_time": "2022-01-03T14:10:46.601440Z",
    "end_time": "2022-01-03T14:10:46.601490Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.8.0",
        "service.name": "unknown_service"
    }
}
{
    "name": "rootSpan",
    "context": {
        "trace_id": "0x6a37f0f0678f07485a01ba001b1119b0",
        "span_id": "0xdaf18d32c6af7c38",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2022-01-03T14:10:46.601349Z",
    "end_time": "2022-01-03T14:10:46.601515Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.8.0",
        "service.name": "unknown_service"
    }
}

Getting Started with OpenTelemetry Python and Jaeger – Advancing

Even though our newly created spans are really beautiful in the console, you are (rightfully) not satisfied just by having them. You most likely want to get some visualization of how they play out together.

That’s probably what got you interested in distributed tracing in the first place.

You’ll be happy to know about the open-source Jaeger, which is a storage backend for telemetry data that also contains a basic UI for visualizing spans and traces.

You’ll even be happier to know that exporting spans to Jaeger is almost as easy as it was to send to our console output.

To send the spans to Jaeger, we’d use the OpenTelemetry Jaeger exporter instead of the console span exporter we used before.

Here’s how it’s done:

1.Start a new python project (or keep the same one, as you wish)

2.Run installs:

pip install opentelemetry-exporter-jaeger

3.Run Jaeger locally

4.Create a jaeger_tracing.py file with this content:

# jaeger_tracing.py
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(
   TracerProvider(
       resource=Resource.create({SERVICE_NAME: "my-hello-service"})
   )
)

jaeger_exporter = JaegerExporter(
   agent_host_name="localhost",
   agent_port=6831,
)

trace.get_tracer_provider().add_span_processor(
   BatchSpanProcessor(jaeger_exporter)
)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("rootSpan"):
   with tracer.start_as_current_span("childSpan"):
           print("Hello world!")

5.Run the jaeger_tracing.py file

6.Use a browser to go to http://localhost:16686/

7.You can now see the Jaeger UI. Select my-hello-service and click on Find traces. You should see your trace with rootSpan and childSpan here on the right:

8.After clicking on our rootSpan from the list you can see more details about it which you can further investigate on your own:

An auto instrumentation example

In real life, you would most likely use auto instrumentation more than you would use manual ones. I chose to start with the manual ones as it’s simpler, to begin with, and understand.

Let’s say we have a small script that writes data to a MongoDB database using the PyMongo library.

For us to create and visualize these spans in Jaeger, we would use an automatic instrumentation library forPyMongo.

In this case, it’s called opentelemetry-pymongo-instrumentation.

First, let’s start mongo locally using docker:

docker run -d -p 27017:27017 mongo

I have updated our script from before to connect to a DB I created called pytest, and saved a post document in a collection called posts.

After that, it tries to find the exact same document using mongo find_one.

This is the updated code:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
   BatchSpanProcessor,
   ConsoleSpanExporter,
)
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from pymongo import MongoClient
from opentelemetry.instrumentation.pymongo import PymongoInstrumentor
import datetime

# Setup tracing
provider = TracerProvider(
       resource=Resource.create({SERVICE_NAME: "my-mongo-service"})
)
jaeger_exporter = JaegerExporter(
   agent_host_name="localhost",
   agent_port=6831,
)
processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))
PymongoInstrumentor().instrument(tracer_provider=provider)

client = MongoClient('mongodb://localhost:27017')
db = client.pytest

posts = db.posts

# insert post
post = {"author": "Tom",
       "text": "My blog post",
       "date": datetime.datetime.utcnow()}
print('inserting post')
post_id = posts.insert_one(post).inserted_id
print('Inserted post with ID:', post_id)

# find our newly created post
found_post = posts.find_one({"_id":post_id})
print('post is', found_post)

If we head back to our Jaeger UI and query for ‘my-mongo-service’ service, we will see 2 spans have been created. One for inserting the post, and another for finding it.

All are created automatically by the pymongo instrumentation library.

Why are they not in the same trace?

Well, simply because there was no piece of code that was meant to create any span for our script runner.

In real life, you would most likely use some web app framework like Django and instrument it, so that a span would be created for a call to your endpoint, becoming the root span and containing these 2 spans under it. But that use-case is out of the scope of this tutorial.

This is what it would look like if we selected the find.posts span in Jaeger:

Bonus: Advanced Visualization with Aspecto

By now you should have a basic understanding of what a span is and how OpenTelemetry can be used to add distributed tracing for code written in Python.

But you probably reached distributed tracing because you wanted to visualize your distributed services and understand them better, which is exactly what Aspecto does.

Give our Live Playground a try to get a better idea – it’s free and no sign-up is required.

At the time of writing this, Aspecto has a free forever plan that you could start using today.

Here’s how to do it:

1.First, create a new free account at www.aspecto.io

2.Then, let’s install the following packages:

pip install opentelemetry-instrumentation
pip install opentelemetry-distro
pip install opentelemetry-exporter-otlp-proto-grpc

3.Modify your python file like this:

# main.py
from pymongo import MongoClient
import datetime

client = MongoClient('mongodb://localhost:27017')
db = client.pytest

posts = db.posts

# insert post
post = {"author": "Tom",
       "text": "My blog post",
       "date": datetime.datetime.utcnow()}
print('inserting post')
post_id = posts.insert_one(post).inserted_id
print('Inserted post with ID:', post_id)

# find our newly created post
found_post = posts.find_one({"_id":post_id})
print('post is',found_post)

Go to Aspecto settings and copy your API keys.

Run like this so that spans are sent to Aspecto:

OTEL_SERVICE_NAME=your-service-name 
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 
OTEL_EXPORTER_OTLP_HEADERS=Authorization=your-aspecto-api-key-here opentelemetry-instrument python main.py

You got yourself a clean list of traces with easy-to-use filters.

Now let’s dive into one of the traces by selecting it:

See how you got a nice visualization that your service (a python file in this case) has made a call to mongo with the query ‘find.posts’?

Just imagine how it would look in your production, giving you complete visibility on all your microservices as you troubleshoot issues 🤯

And that would be it! If you have any questions, feel free to reach out at any time.

P.S. If you want to learn more about OpenTelemetry, you can check out this free, 6 episodes, OpenTelemetry Bootcamp (vendor-neutral).

It’s basically your OpenTelemetry playbook where you will learn everything, from the very basics to scaling and deploying to production:

Episode 1: OpenTelemetry Fundamentals
Episode 2: Integrate Your Code (logs, metrics and traces)
Episode 3: Deploy to Production + Collector
Episode 4: Sampling and Dealing with High Volumes
Episode 5: Custom Instrumentation
Episode 6: Testing with OpenTelemetry

How To Use OpenTelemetry With AWS Lambda

Tom Weiss — Thu, 11 Nov 2021 16:54:44 +0000

Lambda is the AWS solution for serverless functions.

OpenTelemetry is an open-source meant to create traces and send them to a backend and gain visibility.

The observability-aware developer which has serverless lambdas as part of his stack will surely tackle the need to connect OpenTelemetry with lambda.

If you are such a developer, this guide is for you.

Today, I’ll show you exactly how to deploy a tracing-enabled lambda with OpenTelemetry.

This article is part of the Aspecto Hello World series, where we tackle distributed services-related topics for you. Our team searches the web for common issues, then we solve them ourselves and bring you complete how-to guides. Aspecto is an OpenTelemetry-based distributed tracing platform for developers and teams of distributed applications.

Setting Up

Create a new directory for your project, and add the following package.json (or this packages to your existing project):

{
 "name": "lambda-otel-post",
 "version": "1.0.0",
 "description": "",
 "main": "handler.js",
 "dependencies": {
   "@opentelemetry/api": "1.0.2",
   "@opentelemetry/instrumentation": "0.25.0",
   "@opentelemetry/auto-instrumentations-node": "0.25.0",
   "@opentelemetry/instrumentation-aws-lambda": "0.25.0",
   "@opentelemetry/instrumentation-http": "0.25.0",
   "@opentelemetry/sdk-trace-base": "0.25.0",
   "@opentelemetry/sdk-trace-node": "0.25.0",
   "axios": "^0.24.0"
 },
 "devDependencies": {},
 "scripts": {
   "test": "echo \"Error: no test specified\" && exit 1"
 },
 "author": "",
 "license": "ISC"
}

Run installs:

npm install

Add the handler.js

This code is a simple lambda entry point that contains a call to an external API and returns a message.

Later on, we will want to make sure that a span has been created for this HTTP call, and also for the actual lambda invocation.

'use strict';

const axios = require("axios");

module.exports.hello = async (event) => {
 const todoItem = await axios('https://jsonplaceholder.typicode.com/todos/1');

 return {
   statusCode: 200,
   body: JSON.stringify(
     {
       message: 'Some Message Here',
       input: event,
     },
     null,
     2
   ),
 };

 // Use this code if you don't use the http event with the LAMBDA-PROXY integration
 // return { message: 'Go Serverless v1.0! Your function executed successfully!', event };
};

Add the lambda wrapper file that enables tracing with OpenTelemetry

Let’s add the following lambda-wrapper.js file:

const { SimpleSpanProcessor, ConsoleSpanExporter } = require("@opentelemetry/sdk-trace-base");
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { AwsLambdaInstrumentation } = require('@opentelemetry/instrumentation-aws-lambda');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()))
provider.register();

registerInstrumentations({
 instrumentations: [
   getNodeAutoInstrumentations(),
   new AwsLambdaInstrumentation({
     disableAwsContextPropagation: true
   })
 ],
});

Notice I am using ConsoleSpanExporter, which writes all the telemetry data to the console.

In production, you would probably want to have this sent to some other tool like Jaeger or an observability vendor.

For this blog post, however, this exporter will do.

A note on `disableAwsContextPropagation`

Another thing you’re probably wondering is why I added disableAwsContextPropagation:true.

The reason for this is the lambda instrumentation is trying to use the X-Ray context headers by default (even when we’re not using X-Ray), causing us to have a non-sampled context and a NonRecordingSpan.

To fix this, we use the disableAwsContextPropagation flag.

More information about this can be found here and in the instrumentation docs.

Deploy the lambda

There are various ways of deploying lambda to S3 and this is not the scope of the tutorial.

I chose to use a serverless framework, but you can also use AWS CLI / other forms to do this.

If you use serverless, this is the serverless.yml file.

Do not forget to set the correct region & function name.

service: lambda-otel-post

# You can pin your service to only deploy with a specific Serverless version
# Check out our docs for more details
frameworkVersion: '2'

provider:
 name: aws
 runtime: nodejs12.x
 lambdaHashingVersion: 20201221
 environment:
   NODE_OPTIONS: --require lambda-wrapper



 region: eu-west-2

functions:
 tom-otel-lambda-post:
   handler: handler.hello

Add environment variable

For the tracing code to run, we need to make sure Node requires it before any other file has been required.

That’s why we need to add this value for the NODE_OPTIONS environment variable: “–require lambda-wrapper”.

If you use serverless with the file above, it is done for you automatically.

If not, head to the configuration section of the deployed lambda and set it:

The reason for this necessity is that the wrapper file must be included before any other file for the OpenTelemetry instrumentations to work properly.

Calling the lambda function

Now when you run your lambda(I used the built-in AWS console’s test utility), you should expect to see 2 spans being created – one for the lambda invocation, and the other for the outgoing HTTP call.

Indeed, that’s what we get:

This is the outgoing HTTP span

{
  traceId: '4f373b61315c23fa47605a72b94ab59e',
  parentId: '7ce4ab2283755eda',
  name: 'HTTPS GET',
  id: '54c07955525dad7f',
  kind: 2,
  timestamp: 1635332193754154,
  duration: 82864,
  attributes: {
    'http.url': 'https://jsonplaceholder.typicode.com/todos/1',
    'http.method': 'GET',
    'http.target': '/todos/1',
    'net.peer.name': 'jsonplaceholder.typicode.com',
    'net.peer.ip': '104.21.4.48',
    'net.peer.port': 443,
    'http.host': 'jsonplaceholder.typicode.com:443',
    'http.response_content_length_uncompressed': 83,
    'http.status_code': 200,
    'http.status_text': 'OK',
    'http.flavor': '1.1',
    'net.transport': 'ip_tcp'
  },
  status: { code: 1 },
  events: []
}

And the lambda invocation span:

{
  traceId: '4f373b61315c23fa47605a72b94ab59e',
  parentId: undefined,
  name: 'lambda-otel-post-dev-tom-otel-lambda-post',
  id: '7ce4ab2283755eda',
  kind: 1,
  timestamp: 1635332193747990,
  duration: 93019,
  attributes: {
    'faas.execution': 'ed075caa-4d54-44f8-96b4-b96085acbf9a',
    'faas.id': 'arn:aws:lambda:eu-west-2:MY-AWS-ID:function:lambda-otel-post-dev-tom-otel-lambda-post',
    'cloud.account.id': 'MY-AWS-ID'
  },
  status: { code: 0 },
  events: []
}

That would be it for today folks, you can now export those spans to wherever you like.

P.S. If you don’t have an easy way of visualizing these traces just yet, feel free to check out Aspecto (it’s free). This is what a single trace would look like:

How to Improve Your Integration Tests Using OpenTelemetry

Tom Weiss — Thu, 28 Oct 2021 15:12:32 +0000

This article is part of the Aspecto Hello World series, where we tackle microservices-related topics for you. Our team searches the web for common issues, then we solve them ourselves and bring you complete how-to guides. Aspecto is an OpenTelemetry-based distributed tracing platform for developers and teams of distributed applications.

Note: this tutorial assumes you are familiar with OpenTelemetry, traces, and spans. If you want to learn more about OpenTelemetry, check out The OpenTelemetry Bootcamp. This is a free, vendor-neutral, six-episode video series that brings you everything you need to know to get started with OpenTelemetry, from zero to hero.

Introduction

The evolution of OpenTelemetry (OTEL) in recent years has made it a lot easier for developers that are interested in better understanding their microservices – to instrument their services and gain that desired view.

But so far, the usage has been mainly for debugging production issues.

What if I told you there’s a way of utilizing OpenTelemetry’s power to prevent production issues, by using it in your integration test environment?

Sounds interesting? read on as I show you how it can be done easily.

What it would look like to use OpenTelemetry in an integration test

The end goal is to instrument our service under test while the test is running and make assertions on the created spans.

We call this: Trace-Based Testing.

Now you might be starting to think practically on such an implementation and say – “Oh, so I need to integrate the OpenTelemetry SDK now in my test run? Where do I store the spans? How do I get them while the test is already running so that I can assert?”

Well, those questions are legit indeed.

Luckily, you don’t need to implement all that on your own.

This use case is exactly what has led to the creation of Malabi, an open-source that wraps the OpenTelemetry SDK and does all this setup for you so that you can simply add it to your project and start asserting.

P.S. You can read more about the concept of Trace-Based testing and Malabi here.

How does Malabi help?

(Practical guide below, this is the theoretical part)

The way it works is simple.

You add Malabi to your project by using npm or yarn and add 3 lines of code at the top of the main file of the service.

Then, Malabi uses OpenTelemetry SDK for you and creates spans as your service is run (in the context of an integration test – for example in CI).

Malabi then stores these spans in memory and exposes an endpoint that lets you access these spans, and gives you utility functions to extract the data that you need for your assertions.

The practical part – how to take your existing NodeJs microservice and utilize OpenTelemetry to make assertions in an integration test

The following code is the ExpressJS code of the microservice that we want to test.

It is a simple service that uses SQLite as an in-memory database to store & retrieve data about users. It also stores some of the fetched data in a redis cache for faster retrieval.

Here is the code in the index.ts file:

Part 1 – The microservice code

* Note that you can find the complete code in the Malabi examples folder:

index.js file:

import * as malabi from 'malabi';
malabi.instrument();
malabi.serveMalabiFromHttpApp(18393);

import axios from 'axios';
import express from 'express';
import body from "body-parser";
import User from "./db";
import { getRedis } from "./redis";
import Redis from "ioredis";
let redis: Redis.Redis;

getRedis().then((redisConn) => {
   redis = redisConn;
   app.listen(PORT, () => console.log(`service-under-test started at port ${PORT}`));

})
const PORT = process.env.PORT || 8080;

const app = express();
app.use(body.json())
app.get('/',(req,res)=>{
   res.sendStatus(200);
})
app.get('/todo', async (req, res) => {
   try {
       const todoItem = await axios('https://jsonplaceholder.typicode.com/todos/1');
       res.json({
           title: todoItem.data.title,
       });
   } catch (e) {
       res.sendStatus(500);
       console.error(e, e);
   }
});

app.get('/users', async (req, res) => {
   try {
       const users = await User.findAll({});
       res.json(users);
   } catch (e) {
       res.sendStatus(500);
       console.error(e, e);
   }
});

app.get('/users/:firstName', async (req, res) => {
   try {
       const firstName = req.param('firstName');
       if (!firstName) {
           res.status(400).json({ message: 'Missing firstName in url' });
           return;
       }

       let users = [];
       users = await redis.lrange(firstName, 0, -1);
       if (users.length === 0) {
           users = await User.findAll({ where: { firstName } });
           if (users.length !== 0) {
               await redis.lpush(firstName, users)
           }
       }

       res.json(users);
   } catch (e) {
       res.sendStatus(500);
       console.error(e, e);
   }
});

app.post('/users', async (req, res) => {
   try {
       const { firstName, lastName } = req.body;
       const user = await User.create({ firstName, lastName });
       res.json(user);
   } catch (e) {
       res.sendStatus(500);
   }
})

In the above file, you see all the endpoints of the microservice. Mostly self-explanatory – fetching, storing data in SQLite DB / Redis as cache.

But notice the top three lines where the Malabi magic happens:

import * as malabi from 'malabi';
malabi.instrument();
malabi.serveMalabiFromHttpApp(18393);

Basically, we require Malabi (after running npm install –save-dev malabi of course).

Then, Malabi instruments our service – meaning it will create spans (in memory) as it runs.

At that point, we tell it to serve the created spans from port 18393.

In part 2, you will see how we use Malabi util functions to query this endpoint and use Jest to make assertions on them. But first, let’s continue to understand our service.

This db.ts file that handles SQLite with Sequelize:

import { Sequelize, DataTypes } from 'sequelize';

const sequelize = new Sequelize({
   dialect: 'sqlite',
   storage: ':memory:'
});

const User = sequelize.define('User', {
   firstName: {
       type: DataTypes.STRING,
       allowNull: false
   },
   lastName: {
       type: DataTypes.STRING
   }
});

User.sync({ force: true }).then(() => {
   User.create({ firstName: "Rick", lastName: 'Sanchez' });
})

export default User;

The redis.ts file:

import { RedisMemoryServer } from 'redis-memory-server';
import Redis from "ioredis";
const redisServer = new RedisMemoryServer();

export async function getRedis() {
   const host = await redisServer.getHost();
   const port = await redisServer.getPort();
   const redis = new Redis(port, host);
   return redis;
}

Part 2 – The Test Code

service-under-test.spec.ts file:

This file will be run using Jest.

Notice we have the port of the service itself (to call the actual running service, which you would run in the CI environment or locally).

We also have the Malabi utility functions – fetchRemoteTelemetry, clearRemoteTelemetry that like their name suggests – fetch the spans from the endpoint for assertions and clear the in-memory cache (which is useful to clean up between tests to maintain a clean slate each time).

Take a look at the code, more info follows below:

const SERVICE_UNDER_TEST_PORT = process.env.PORT || 8080;
import axios from 'axios';
import { fetchRemoteTelemetry, clearRemoteTelemetry } from 'malabi';
const getTelemetryRepository = async () => await fetchRemoteTelemetry({ portOrBaseUrl: 18393 });

describe('testing service-under-test remotely', () => {
   beforeEach(async () => {
       // We must reset all collected spans between tests to make sure spans aren't leaking between tests.
       await clearRemoteTelemetry({ portOrBaseUrl: 18393 });
   });

   it('successful /todo request', async () => {
       // call to the service under test - internally it will call another API to fetch the todo items.
       const res = await axios(`http://localhost:${SERVICE_UNDER_TEST_PORT}/todo`);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       // Validate internal HTTP call
       const todoInteralHTTPCall = telemetryRepo.spans.outgoing().first;
       expect(todoInteralHTTPCall.httpFullUrl).toBe('https://jsonplaceholder.typicode.com/todos/1')
       expect(todoInteralHTTPCall.statusCode).toBe(200);
   });

   it('successful /users request', async () => {
       // call the service under test
       const res = await axios.get(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users`);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       // Validating that /users had ran a single select statement and responded with an array.
       const sequelizeActivities = telemetryRepo.spans.sequelize();
       expect(sequelizeActivities.length).toBe(1);
       expect(sequelizeActivities.first.dbOperation).toBe("SELECT");
       expect(Array.isArray(JSON.parse(sequelizeActivities.first.dbResponse))).toBe(true);
   });

   it('successful /users/Rick request', async () => {
       // call the service under test
       const res = await axios.get(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users/Rick`);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       const sequelizeActivities = telemetryRepo.spans.sequelize();
       expect(sequelizeActivities.length).toBe(1);
       expect(sequelizeActivities.first.dbOperation).toBe("SELECT");

       const dbResponse = JSON.parse(sequelizeActivities.first.dbResponse);
       expect(Array.isArray(dbResponse)).toBe(true);
       expect(dbResponse.length).toBe(1);
   });

   it('Non existing user - /users/Rick111 request', async () => {
       // call the service under test
       const res = await axios.get(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users/Rick111`);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       const sequelizeActivities =  telemetryRepo.spans.sequelize();
       expect(sequelizeActivities.length).toBe(1);
       expect(sequelizeActivities.first.dbOperation).toBe("SELECT");

       const dbResponse = JSON.parse(sequelizeActivities.first.dbResponse);
       expect(Array.isArray(dbResponse)).toBe(true);
       expect(dbResponse.length).toBe(0);

       expect(telemetryRepo.spans.httpGet().first.statusCode).toBe(200);
   });

   it('successful POST /users request', async () => {
       // call the service under test
       const res = await axios.post(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users`,{
           firstName:'Morty',
           lastName:'Smith',
       });

       expect(res.status).toBe(200);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       // Validating that /users created a new record in DB
       const sequelizeActivities =  telemetryRepo.spans.sequelize();
       expect(sequelizeActivities.length).toBe(1);
       expect(sequelizeActivities.first.dbOperation).toBe("INSERT");
   });


   /* The expected flow is:
       1) Insert into db the new user (due to first API call; POST /users).
       ------------------------------------------------------------------
       2) Try to fetch the user from Redis (due to the second API call; GET /users/Jerry).
       3) The user shouldn't be present in Redis so fetch from DB.
       4) Push the user object from DB to Redis.
   */
   it('successful create and fetch user', async () => {
       // Creating a new user
       const createUserResponse = await axios.post(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users`,{
           firstName:'Jerry',
           lastName:'Smith',
       });
       expect(createUserResponse.status).toBe(200);

       // Fetching the user we just created
       const fetchUserResponse = await axios.get(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users/Jerry`);
       expect(fetchUserResponse.status).toBe(200);

       // get spans created from the previous calls
       const telemetryRepo = await getTelemetryRepository();
       const sequelizeActivities = telemetryRepo.spans.sequelize();
       const redisActivities =  telemetryRepo.spans.redis();

       // 1) Insert into db the new user (due to first API call; POST /users).
       expect(sequelizeActivities.first.dbOperation).toBe('INSERT');
       // 2) Try to fetch the user from Redis (due to a second API call; GET /users/Jerry).
       expect(redisActivities.first.dbStatement).toBe("lrange Jerry 0 -1");
       expect(redisActivities.first.dbResponse).toBe("[]");
       // 3) The user shouldn't be present in Redis so fetch from DB.
       expect(sequelizeActivities.second.dbOperation).toBe("SELECT");
       //4) Push the user object from DB to Redis.
       expect(redisActivities.second.dbStatement.startsWith('lpush Jerry')).toBeTruthy();
   });
});

Once we have fetched the spans, we can use Jest’s expect function to make assertions, as we would in any other test regardless of trace-based testing.

Examined example 1 – test named “successful /users request”:

Let’s examine the above code – for example, the test named “successful /users request”.

First, we call the service to fetch all users.

Then, we use the fetchRemoteTelemetry wrapped by the getTelemetryRepository function to get the spans from Malabi.

After that, we use the sequelize accessor to filter only sequelize spans.

Once we have the sequelize spans at hand, we can assert to have only 1 as we only fetch the DB once.

We also know it’s a SELECT operation, so we assert that it’s a SELECT operation.

Examined example 2 – test named “successful create and fetch user”:

Let’s now examine a slightly more complicated test.

In the indicated test, we create a new user using POST /users. Then, we try to query for that user using GET /users/:firstName.

As expected, we assert for 200 as you would in any other test.

Now here again we use Malabi utilities to fetch relevant spans, and store them in variables – one for redis spans and another for sequelize:

const telemetryRepo = await getTelemetryRepository();
const sequelizeActivities = telemetryRepo.spans.sequelize();
const redisActivities =  telemetryRepo.spans.redis();

The first assertion – we want to make sure that the initial POST operation had caused a DB INSERT operation:

expect(sequelizeActivities.first.dbOperation).toBe('INSERT');

Since the user was just created, we expect it to not exist in redis, so we assert the redis query to be as we want it and expect the response from redis to be an empty array:

expect(redisActivities.first.dbStatement).toBe("lrange Jerry 0 -1");
expect(redisActivities.first.dbResponse).toBe("[]");

Since the user was not present in redis, we expect to have fetched the DB – so assert that the second DB operation select

expect(sequelizeActivities.second.dbOperation).toBe("SELECT");

And now we expect redis to have received push command to make sure in real life(not in test runs since we clean up everything), subsequent runs would not have to fetch the DB (but take from redis):

expect(redisActivities.second.dbStatement.startsWith('lpush Jerry')).toBeTruthy();

That would be it! I hope you can see how simple it can be to use OpenTelemetry & Malabi to write powerful integration tests, in a much easier way than before.

P.S. Malabi is a relatively new library implementing a new approach, and its authors (myself included) would love to hear your thoughts on it and hear any improvements/suggestions you have. So feel free to open a discussion in GitHub or contact me via Twitter DMs.

How To Write Integration Tests Easily Using Trace-Based Testing

Tom Weiss — Wed, 29 Sep 2021 14:19:29 +0000

This article is part of the Aspecto Hello World series, where we tackle microservices-related topics for you. Our team searches the web for common issues, then we solve them ourselves and bring you complete how-to guides. Aspecto is an OpenTelemetry-based distributed tracing platform for developers and teams of distributed applications.

Introduction

Integration tests are never fun to write.

Setting up an environment, finding a way to reproduce the needed state of it, populating it with relevant data, etc – It’s all work no play.

In this blog post, I will show you how to write an integration test and talk about why you want to do it.

The motivation for writing an integration test

Let’s begin by talking about why we even want to write one.

So you finished writing all of the code of your task and everything is working.

How do you make sure it stays this way and does not break when another developer changes just one tiny thing that happens to break your feature? (Not on purpose of course, or so we hope).

The solution: Instead of moving onto the next feature right away, you can write an integration test to make sure that your team gets notified if something breaks (by running it in a CI/CD pipeline that alerts you).

The role of the integration test is to verify that not only does each function you wrote works as a standalone (that’s unit tests), but also to verify that all of the functions play out well together.

Spoiler alert: in this guide, I will also show you how to make sure that a database received the relevant params.

What we will build:

I like to keep everything as simple as possible, giving you only what you need to get on with your day.

Therefore, I will build a simple to-do app: a nodejs express service that authenticates the user, with endpoints for creating & viewing his tasks.

The integration test will verify the following:

Login
Create a new TODO item
Verify the todo item was saved with the relevant email & todo text was saved to the database
Fetch all todo items
Verify that the relevant email was used to query the database

The tools I will be using:

NodeJS + Express
Jest – a framework for authoring and executing nodejs tests
Malabi – a Trace-Based Testing library that lets me make assertions on real data passed to moving parts like mongodb / elasticsearch and more (I will further elaborate on this below).

Part 1 – Setting up the tested microservice

Note: If you already have a service you wish to test, feel free to skip to part 2 where I show how to write the actual integration test.

Step 1 – Set up the project

Let’s use express-generator to generate the initial code for our express app, with the pug view engine.

Also, let’s install the packages we will be using:

npx express-generator -v pug
npm i --save jsonwebtoken mongoose passport passport-jwt axios

We will be storing the data in a MongoDB database, which you can run locally using docker like this:

docker pull mongo
docker run -d -p 27017:27017 mongo

Step 2 – create the views

We’ll add a very simple UI for our imaginary users to add TODO items.

login.pug

extends layout

block content
 body
   form(action='/auth/login?redirectToTodos=true', method='POST')
     p
       | username:
       input(type='text', name='username', value='tom@a.com')
       |          <br/>password:
       input(type='password', name='password', value='password')
     input(type='submit', value='Submit')

app.pug

extends layout

block content
   p Welcome to the app screen <b>#{user}</b>
   form(action='/todos/', method='POST')
       p
           | new todo text:
           input(type='text', name='username', value='hi')
       input(type='submit', value='Submit')
   ul
     each val in todos

Step 3 – Adding routes

In the routes folder, let’s delete the users’ file, we won’t be using it.

Now place the following code in the index.js file.

Index.js

const var express = require('express');
const router = express.Router();
const jwt = require('jsonwebtoken');
const TOKEN_SECRET = 'SECRET';

router.get('/login', function(req, res, next) {
 res.render('login', { title: 'Express' });
});

router.post('/auth/login', function(req, res, next) {
 const email = req.body.username;
 const token = jwt.sign({ email }, TOKEN_SECRET, {
   expiresIn: 60 * 60,
 });
 res.cookie('auth', token, { httpOnly: true });
 req.query.redirectToTodos ? res.redirect('/todos') : res.json({ success: true });
});


module.exports = router;

Here we have two endpoints – one for rendering the login view for our users, and the other one for performing the login with a username & password. We’re not verifying passwords here, as authentication is not the focus of the blog.

The way it works in general: login, sign a JWT token with email, send to client with the response.

I added a small modification here – redirectToTodos param, to avoid writing client-side code.

In real life, I would not add it and have the client handle the JSON response. For the purpose of this post, it’s good enough.

Let’s add another routes file called todos.js:

const express = require('express');
const router = express.Router();
const passport = require('passport');
const mongoose = require('mongoose');

async function connectToMongoose() {
 await mongoose.connect('mongodb://localhost:27017/test');
}

connectToMongoose().catch(err => console.log(err));

const todoSchema = new mongoose.Schema({
 text: String,
 email: String
});

const Todo = mongoose.model('Todo', todoSchema);

router.get('/',
 (req, res, next) => {
   passport.authenticate('jwt', { session: false },  async (err, user, info) => {
     if (err) {
       console.log('error is', err);
       res.status(500).send('An error has occurred, we cannot greet you at the moment.');
     }
     else {
       console.log('user is', user);
       console.log('info is', info);
       const { email } = user;
       const todos = await Todo.find({ email });

       res.render('app',{ success: true, user: email, todos });
     }
   })(req, res, next);
 });

router.post('/',
 (req, res, next) => {
   passport.authenticate('jwt', { session: false },  async (err, user, info) => {
     if (err) {
       console.log('error is', err);
       res.status(500).send('An error has occurred, we cannot greet you at the moment.');
     }
     else {
       console.log('user is', user);
       console.log('info is', info);
       const todo = new Todo({ text: 'Hey that is my new todo', email: user.email });
       await todo.save();
       res.json({ success: true });
     }
   })(req, res, next);
 });


module.exports = router;

In this file I define the mongoose Todo model and add two endpoints: GET for retrieving all todos for the current user, and POST for creating one.

I use the passport-jwt strategy to decode the user’s email (which is sent with the request as a cookie)

Step 4 – Adding app.js

The app.js file contains different setups for authentication & routing. I won’t dive into this deeply. If you’re curious feel free to check out my guide to microservices authentication strategies.

var createError = require('http-errors');
var express = require('express');
var path = require('path');
var cookieParser = require('cookie-parser');
var logger = require('morgan');

var indexRouter = require('./routes/index');
var todoRouter = require('./routes/todos');

var app = express();

// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'pug');

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

const passport = require('passport');
const JwtStrategy = require('passport-jwt').Strategy,
 ExtractJwt = require('passport-jwt').ExtractJwt;

app.use(passport.initialize());
app.use(passport.session());

app.use('/', indexRouter);
app.use('/todos', todoRouter);

// catch 404 and forward to error handler
app.use(function(req, res, next) {
 next(createError(404));
});

// error handler
app.use(function(err, req, res, next) {
 // set locals, only providing error in development
 res.locals.message = err.message;
 res.locals.error = req.app.get('env') === 'development' ? err : {};

 // render the error page
 res.status(err.status || 500);
 res.render('error');
});

const cookieExtractor = function(req) {
 let token = null;
 if (req && req.cookies)
 {
   token = req.cookies['auth'];
 }
 return token;
};

const TOKEN_SECRET = 'SECRET';

const opts = {
 jwtFromRequest: ExtractJwt.fromExtractors([cookieExtractor]),
 secretOrKey: TOKEN_SECRET,
};

passport.use(
 'jwt',
 new JwtStrategy(opts, (jwt_payload, done) => {
   try {
     console.log('jwt_payload', jwt_payload);
     done(null, jwt_payload);
   } catch (err) {
     done(err);
   }
 }),
);

module.exports = app;

Step 5 – Running the app

Now you can simply run the app using and go to localhost:3000/login, login and submit the form to create a todo item and see that it is working.

npm start

Part 2: Writing an integration test

In this part, we will be using jest & Malabi to write our integration test.

Jest is a library that lets you author and execute tests for nodejs.

As for Malabi – an explanation follows.

Malabi is an open-source Javascript framework based on OpenTelemetry that allows you to leverage trace data and improve assertion capabilities. This library introduces a new way of testing services: Trace-based testing (TBT).

Here’s a quick tutorial on Malabi and how it works:

Trace based testing

So what is Trace Based Testing anyway? Good thing you ask.

It is a new approach that utilizes OpenTelemetry to our advantage in testing.

Opentelemetry gives us SDKs that let us instrument our application, meaning to create data of what calls are being made to endpoints, databases and essentially is aimed at giving us visibility on our microservices.

You can read more about Malabi and Trace-based testing in this blog post.

By instrumenting our tested microservice, we’re able to make assertions on what API calls were made, with what parameters.

We can also assert how a database was queried – in our case, which exact parameter did Mongo receive when it was asked for all todos by the current user. We want to assert the current user and know we don’t have a bug causing us to log in with one user but query another user’s data.

Malabi wraps the opentelemetry SDK, creates spans (actions performed in the service), and exposes an HTTP endpoint for you to query & assert on.

Enough theory for now – Let’s begin.

Step 1 – perform installs

npm install --save-dev jest malabi

Add the following lines at the top of our app.js file

const malabi = require('malabi');
malabi.instrument();
malabi.serveMalabiFromHttpApp(18393);

This tells Malabi to create spans for us (actions we can assert on, for example – a MongoDB query and an HTTP request are 2 examples of possible spans).

It also tells malabi to expose an API endpoint at port 18393 that lets the test runner get these spans and then assert on them.

Step 2- add a new folder & test file: tests/integ.spec.js

const axios = require('axios').default;
const { fetchRemoteTelemetry, clearRemoteTelemetry } = require('malabi');
const getMalabiTelemetryRepository = async () => await fetchRemoteTelemetry({ portOrBaseUrl: 18393 });

describe('testing service-under-test remotely', () => {
 beforeEach(async () => {
   // We must reset all collected spans between tests to make sure span aren't leaking between tests.
   await clearRemoteTelemetry({ portOrBaseUrl: 18393 });
 });

 it('successful /todo request', async () => {
   // simulate login
   const loginRes = await axios.post(`http://localhost:3000/auth/login`, { username: 'tom@a.com', password: 'password' });
   const authCookie = loginRes.headers['set-cookie'];

   // Create a new todo item
   const newTodoRes = await axios.post(`http://localhost:3000/todos`, {}, { headers: {
       Cookie: authCookie
   } });
   // console.log('newTodoRes', newTodoRes);

   // call to the service under test - internally it will call another API to fetch the todo items.
   await axios(`http://localhost:3000/todos/`, { headers: {
       Cookie: authCookie
   } });

   // Get instrumented spans
   const repo = await getMalabiTelemetryRepository({ portOrBaseUrl: 13893 });

   // This is the span that holds data from the creation of the new todo item
   const postTodoSpan = repo.spans.mongo().first;
   const emailUsedForCreatingNewTodo = JSON.parse(postTodoSpan.dbStatement).document.email;
   console.log('emailUsedForCreatingNewTodo', emailUsedForCreatingNewTodo);
   // Assert the email was saved equals the email we logged in with
   expect(emailUsedForCreatingNewTodo).toEqual("tom@a.com")

   // This is the span that holds data from the fetching of todos from mongo for current user
   const getTodosSpan = repo.spans.mongo().second;
   const emailUsedInMongoQuery = JSON.parse(getTodosSpan.dbStatement).condition.email;

   // Assert the email was used for querying equals the email we logged in with
   expect(emailUsedInMongoQuery).toEqual("tom@a.com")
 });
});

A few things to note about the above file:

getMalabiTelemetryRepository – fetches spans created in the current run, from the malabi endpoint.

We can then use the received object to query for specific span types, like all Mongo spans, all http requests, etc.

We also have clearRemoteTelemetry that cleans up the in-memory cache of spans between test runs. This helps maintain our test clean & not polluted between test runs.

Notice we have made 2 assertions: one that the post request saved the todo item with the relevant email (the one we logged in with), and the second one verifies that we fetched only todos that belong to that user.

You can now run the test using the “jest” command, and see that everything is working as expected.

As you can see, this is a very simple and clean way we can make sure the connections between our different functions and moving parts are working as expected.

Of course, in production, you would most likely set this up as part of a CI/CD workflow, but this is not the focus of this guide.

That’s basically it, I hope this guide was useful for you, feel free to reach out to me @magnificoder for any questions you may have!

Feel free to check out some of my other articles, like this one: How to Deploy Jaeger on AWS: a Comprehensive Step-by-Step Guide.

How to Deploy Jaeger on AWS: a Comprehensive Step-by-Step Guide

Tom Weiss — Thu, 26 Aug 2021 14:08:29 +0000

Introduction

In this tutorial, I will be showing you how to host jaeger on AWS ECS. We will do so step by step: set up an AWS Elasticsearch service domain and use it as a storage backend. For this purpose, we will use the jaeger all-in-one image inside our own ECS cluster & service.

(Note: for a production use case you’d probably want to use the jaeger images separately and not the all-in-one. We’re doing this to simplify the blog post).

What is Jaeger

If you’re here you probably already know this, but just in case: jaeger is an open-source distributed tracing system, originally developed by Uber.

Essentially it stores traces & spans (in a storage backend) and hosts a UI that gives us visibility on these traces and spans (if you’re not familiar with the OpenTelemetry jargon, you can get more info here: https://opentelemetry.io/).

A small note about security

Before we dive in – it is important to know that the blog post assumes you are running your ES & jaeger in private subnets inside a secured VPC. For additional security measures, go to https://www.jaegertracing.io/docs/1.25/security/ and check out the official recommendations from Jaeger.

Setting up AWS Elasticsearch instance

Since the jaeger collector is persistent – it requires a storage backend. You could use memory as a storage backend but it is not suitable for production use cases.

So we will begin by creating a new elasticsearch domain to serve as a storage backend for our Jaeger.

Go to AWS ES console.
Click on “Create a new domain”
Select custom & latest version of ES and hit Next.
Let’s set the following settings (but you can of course change them as needed):

Name: as you wish
No auto-tuning (you may want this in production)
1 AZ, 1 Node (note that in production you would probably want at least 2 AZ & nodes for redundancy)
Instance type: T2.small
No master node (you may want this in production)

After everything is set and done – hit Next.

5.Configuring access & security:

Since most likely you’re going to send sensitive data to your Jaeger, you want to make sure it is stored securely. Therefore, it is recommended to only allow access within your VPC. AWS recommends a dedicated subnet for each elasticsearch domain. For this tutorial’s purposes, I am choosing an existing private subnet.

As for security groups, I created a security group that allows open access inside the VPC. For this blog post – I’m ok with it since the elasticsearch is only accessible from private subnets. In production, you may want to be stricter and enable fine-grained access control / different & IP, port settings.

For domain access policy – to simplify the installation and since we’re inside the VPC – I chose to allow open access to the domain. But here also, for production, you may want to be stricter and allow access from a specific IP address or IAM ARN (like the jaeger IP address for example).

As for encryption – I’m leaving the defaults, but feel free to modify it to fit your needs.

6.At this point, we’re done. Let’s hit next as needed and let AWS create our ES domain.

Deploying Jaeger

In order to deploy Jaeger, we will be using AWS ECS. Jaeger provides us with an all-in-one docker image that contains everything that Jaeger needs to work.

Let’s create an ECS service running that image.

Creating our new ECS cluster

Go to ECS Cluster -> Create cluster
Select EC2 Linux + Networking
Configure the cluster:
As for subnets – since I have an AWS VPN client enabled, I put the Jaeger on two private subnets, so that no one from outside the VPC can access its data.
For the security group – let’s create a new one and add inbound rules for the following ports from within the VPC. Replace 10.100.0.0/16 with your own VPC address range. You can learn more about the Jaeger ports here: https://www.jaegertracing.io/docs/1.25/getting-started/.
Change anything else if you need it and hit create.

Creating a Task Definition

For us to create an ECS Service we need to define a Task Definition. Head over to Task Definition and hit Create.

Since we chose EC2, we will choose EC2 as for type compatibility:
Select “Configure from JSON”. Paste the following JSON in the relevant field:

{
 "requiresCompatibilities": [
   "EC2"
 ],
 "inferenceAccelerators": [],
 "containerDefinitions": [    {
   "dnsSearchDomains": [],
   "environmentFiles": null,
   "logConfiguration": null,
   "entryPoint": [],
   "portMappings": [
     {
       "hostPort": 14269,
       "protocol": "tcp",
       "containerPort": 14269
     },
     {
       "hostPort": 14268,
       "protocol": "tcp",
       "containerPort": 14268
     },
     {
       "hostPort": 6832,
       "protocol": "udp",
       "containerPort": 6832
     },
     {
       "hostPort": 6831,
       "protocol": "udp",
       "containerPort": 6831
     },
     {
       "hostPort": 5775,
       "protocol": "udp",
       "containerPort": 5775
     },
     {
       "hostPort": 14250,
       "protocol": "tcp",
       "containerPort": 14250
     },
     {
       "hostPort": 16685,
       "protocol": "tcp",
       "containerPort": 16685
     },
     {
       "hostPort": 5778,
       "protocol": "tcp",
       "containerPort": 5778
     },
     {
       "hostPort": 16686,
       "protocol": "tcp",
       "containerPort": 16686
     },
     {
       "hostPort": 9411,
       "protocol": "tcp",
       "containerPort": 9411
     }
   ],
   "command": [
     "--collector.zipkin.host-port",
     "9411"
   ],
   "linuxParameters": null,
   "cpu": 1024,
   "environment": [
     {
       "name": "ES_SERVER_URLS",
       "value": "https://some-es-name.some-es-region.es.amazonaws.com"
     },
     {
       "name": "SPAN_STORAGE_TYPE",
       "value": "elasticsearch"
     }
   ],
   "resourceRequirements": null,
   "ulimits": null,
   "dnsServers": [],
   "mountPoints": [],
   "workingDirectory": null,
   "secrets": null,
   "dockerSecurityOptions": [],
   "memory": 1024,
   "memoryReservation": null,
   "volumesFrom": [],
   "stopTimeout": null,
   "image": "jaegertracing/all-in-one:1.25.0",
   "startTimeout": null,
   "firelensConfiguration": null,
   "dependsOn": null,
   "disableNetworking": null,
   "interactive": null,
   "healthCheck": null,
   "essential": true,
   "links": [],
   "hostname": null,
   "extraHosts": null,
   "pseudoTerminal": null,
   "user": null,
   "readonlyRootFilesystem": null,
   "dockerLabels": null,
   "systemControls": [],
   "privileged": null,
   "name": "tom-jaeger-new"
 }],
 "volumes": [],

 "networkMode": null,
 "memory": "2048",
 "cpu": "2048",
 "placementConstraints": [],
 "family": "tom-jaeger-new",
 "taskRoleArn": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID:role/ecsTaskExecutionRole",
 "executionRoleArn": "arn:aws:iam::YOUR_AWS_ACCOUNT_ID:role/ecsTaskExecutionRole",
 "tags": []
}

Make sure to make the relevant modifications like your own account id, and copy the correct elasticsearch URL from the AWS ES console, and put it as an environment variable called ES_SERVER_URLS.

Creating the ECS Service

Now we’re ready to create the ECS service that runs Jaeger.

Go back to your Jaeger cluster -> services -> create.
This is the configuration you want (be sure to select the newly created task definition):

3.Unlike what the screenshot suggests, instead of minimum 100, maximum 200 -we use 0-100 so that we only have 1 instance running and no port issues. Again, we do this to simplify the tutorial.

4.From here on you can hit next until creating the service. For this tutorial I chose not to create a load balancer. If you feel you need one feel free to create it.

Accessing Jaeger UI

Now Jaeger should be up and running. Let’s go to the ECS cluster -> ECS instances -> click on the instance id inside the ECS instances column. That should lead you to the corresponding EC2 instance.

Clicking on its ID should give you info about the instance. What you are looking for is the instance IP.

I’m assuming that you are inside the VPC. If you’re not, you may have to find a different way of obtaining network access to your Jaeger like placing it in a public subnet & allowing access to your IP (it is not recommended from a security standpoint).

Copy the private IP address of the instance, and head over to it in the browser with port 16686. For example: 10.100.30.224:16686

You now have access to the Jaeger UI:

Sending traces to Jaeger

At this point you have the jaeger UI running. Now we need to start sending traces to it.

If you take a look at Kibana, you can already see that Jaeger created its own jaeger-span-DATE-FORMAT. Currently, it only contains internal Jaeger spans, but let’s send our own.

Note: for the simplicity of this tutorial we did not implement any index rollover, but you may want to do so to optimize resources allocated to indices. You can read more about this here: https://www.jaegertracing.io/docs/1.25/deployment/#elasticsearch-rollover

Sending traces to Jaeger with the OpenTelemetry SDK (In NodeJS)

Step 1:

npx express-generator

Step 2: Perform npm install

npm install

Step 3: Install OpenTelemetry libraries:

npm install --save @opentelemetry/instrumentation-http 
@opentelemetry/instrumentation-express @opentelemetry/api 
@opentelemetry/node @opentelemetry/exporter-jaeger

Step 4: Create a tracing.js file (replace the relevant IP to your EC2 IP)

'use strict';

const opentelemetry = require('@opentelemetry/api');

// Not functionally required but gives some insight what happens behind the scenes
const { diag, DiagConsoleLogger, DiagLogLevel } = opentelemetry;
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);

const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { NodeTracerProvider } = require('@opentelemetry/node');
const { SimpleSpanProcessor } = require('@opentelemetry/tracing');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes: ResourceAttributesSC } = require('@opentelemetry/semantic-conventions');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');

const Exporter = JaegerExporter;

const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');

module.exports = () => {
 const serviceName = 'serviceName';

 const provider = new NodeTracerProvider({
   resource: new Resource({
     [ResourceAttributesSC.SERVICE_NAME]: serviceName,
   }),
 });
 registerInstrumentations({
   tracerProvider: provider,
   instrumentations: [
     // Express instrumentation expects HTTP layer to be instrumented
     HttpInstrumentation,
     ExpressInstrumentation,
   ],
 });

 const exporter = new Exporter({
   host: '10.100.40.132',
   port: 6832,
 });

 provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

 // Initialize the OpenTelemetry APIs to use the NodeTracerProvider bindings
 provider.register();

 return opentelemetry.trace.getTracer('express-example');
};

Step 5: At the top of the app.js file, add this line to enable tracing:

require('./tracing')();

Step 6: Run npm start & go to http://localhost:3000/ in the browser

Step 7: Back in the Jaeger UI – you can see this trace now:

Bonus (faster route) – Sending traces to Jaeger with the Aspecto SDK:

Aspecto provides a free and easy-to-use SDK that can be configured to export traces to Jaeger (and to Aspecto, that enables additional abilities that Jaeger does not have) with only one line of code.

Step 1: Create a new express app using

npx express-generator

Step 2: Perform npm installs

npm install
npm install @aspecto/opentelemetry

Step 3: Register for free at www.aspecto.io, and obtain your API key.

Step 4: At the top of your app.js file, add this (before any import):

require('@aspecto/opentelemetry')({
  local:true,
  aspectoAuth: '*your-aspecto-api-key-goes-here*',
  customZipkinEndpoint: 'http://10.100.40.132:9411/api/v2/spans',
  otCollectorEndpoint: 'http://10.100.40.132:9411/v1/trace',
});

Do not forget to change the IP to your own EC2 IP address.

Step 5: Let’s modify the route in index.js to /hello and open the browser at localhost:3000/hello.

Step 6: And voila! our Jaeger is showing the span we just sent:

You can see how this looks in Aspecto UI (click on the Aspecto link in your terminal):

If you choose to use it, you don’t even need to maintain Jaeger at all but simply use Aspecto. Feel free to learn more here.

That’s it, I hope that was helpful for you and wish you years of happy tracing! 😁 Reach out to me @magnificoder for any questions you may have.

Feel free to check out some of my other articles, like this one: Microservices Authentication Strategies: Theory to Practice.

Why I started my own bootstrapped software business

Tom Weiss — Sat, 31 Jul 2021 15:24:39 +0000

It all started off at the age of 7:

In hindsight, it all started very early. At the age of 7, I decided I need to find ways to make money. I went with my sister at summer time to sell lemonades at the beach. The business model was great: right at my own home I found 2 investors who happen to be our parents, and have funded the whole adventure. That meant that every dollar that we earned stay as our very own profit.

Some time later, I decided to extend my services, that included cleaning services, haircuts, deliveries, writing songs: all for a few dollars. Here’s an image of a “marketing brochure” I created at the time describing my services, hoping to count the dollars apparently. Not the best marketing image selection was it? ;)

My first “Marketing brochure”, in Hebrew

Helping my dad with his business

It was at 2009 when my dad decided to quit his job at a hospital as the manager of the technical department, to pursue his own business of selling medical equipment.

At the time I was only beginning my career as a software engineer, and had a full time job.

Him being excited doing his own thing, and me being able to see it first hand was the first time I realized that one day I want to do the same.

He also needed my help. At weekends & evenings when I got back from work I used to work with him: building stock management software for his physical products, went to AdWords courses in google, learned about paid advertising, SEO and everything in between. We even flew abroad to conferences together and negotiated deals with equipment manufacturers.

It was a great experience, and even though not long after my dad decided having a business was not for him, the hunger for my own one stayed with me.

First international trip: enter the digital nomad dream

An image I took at the outskirts of Playa Del Carmen

It was late 2015 when I left my first job and went on my first trip that did not include a ticket back home. Amongst other places, I went to Playa Del Carmen, Mexico, where I stayed for over a month.

There, at a beach hostel I saw something I never thought I’d ever see: a person working while being on vacation.

I was shocked, and remember being “angry” at him, saying — why are you working now? You’re here to have fun!

He said: Yeah, at my job they don’t care where I’m at as long as work is being done. I’d just rather be traveling than staying at home all the time.

Little did I know this had a name: a Digital Nomad, and little did he know that from that day I swore to do the same.

Since I fell in love with the culture & beauty of Latin America, I made a rule that until I get my dream come true I have to be there at least for a month every year. Spoiler alert: That did happen.

Working at small startups

It was 2016 when I landed at my first tiny startup job. I remember asking recruiters to not send me any company bigger than 15 employees, since I knew that one day I’d start my own and thought it would be wise to learn how it is first hand first.

And boy did I learn the hard way: 10 months later the first startup ran out of money, so I had to find another one.

So I did, and again it was one of about the same size, ~15 employees.

Photo by Proxyclick Visitor Management System on Unsplash

2019: Forming my own business, and making the nomad dream come true

After almost 2 years working at the second startup, I started feeling the itch of traveling.

I had asked my boss to try and work remotely, but he said he does not believe in this type of working. I just knew it was time to resign, even though I loved everything about that job.

So I did resign, and have then created my own business. I packed my laptop and booked a one way ticket to Latin America. No plan, I just knew that once you begin you find your way with things.

One week before my flight, I met someone very interesting. He runs a site for greetings that is ranked very high in google. He monetizes this site using advertising. And said he knows he could probably make more money but doesn’t want to invest any time in this site.

I said, “hey: I’m a developer and am free. Let’s do something and split the money.”

Long story short: a day later I’m building a site that lets users take a picture & a greeting text, edit & send to their loved ones for birthdays, holidays, etc. I had done so in the airplane and after and couldn’t stop until it’s ready due to the excitement of having something that generates a passive income.

About a month later, it was ready, and is generating a few passive dollars every day to this day.

Seeing my business partner doing this, I just knew bootstrapping was possible. Since then I was determined to get to a point where I have a steady income from my own websites, as fast as possible.

That meant I had to start my own projects, and combined with inspiration from the indie hackers community it has given me the ideas that led me to 2 indie projects I’m currently working on: one marketplace, one Saas, which when the time comes(very soon) I will share more about.

As for the DN life — I had stayed a total of 6 months being a digital nomad, building my own things and trying to see if I can find remote work opportunities at websites.

Then came COVID, which made it easy to find remote work, but less to travel. But it’s ok since I’m back home focusing on growing my business.

Since the beginning of COVID and to this day, I am working part time as a freelance software engineer, while I pursue my indie dreams. Feel free to follow me on Twitter where I document the continuation of my journey.

Microservices Authentication Strategies: Theory to Practice

Tom Weiss — Thu, 15 Jul 2021 16:12:11 +0000

In this article, we will walk through common ways of implementing authentication microservices.

We will have 2 parts:

1.The theoretical part talking about OpenID Connect, OAuth 2.0, JWT, etc.

Here I try to save you time wandering through the web and giving you all the basics you need to understand in order to start coding.

2.The practical part, where we will implement two Node.js microservices, one responsible for user authentication via google login, another responsible for greeting the user that has a token created by the previous service. Plus, we add a react JS app to interact with those services.

The Theory

What is authentication?

Authentication is the answer users give us when we ask them “Who are you?”. For us to believe users, they need to go through a process providing some proof.

For example – by providing a username & password or by using a social login provider.

What is authorization?

Authorization is usually relevant when we already know who the user is, thus the user is authenticated (unless we allow anonymous access, but we won’t get into that use case here).

Our users want to perform certain actions in our system, and the process of checking if they are allowed to do it or not is called authorization.

(Note: The reason we’re talking about this authorization in an authentication article is that these terms are often confused, and we need to understand it to understand concepts like OAuth 2.0 & OpenID Connect)

A good real-world analogy for both of the above would be while checking in a hotel room. Authentication is your passport, and authorization is if I’m allowed to enter a certain room (because I booked it).

OAuth 2.0

OAuth 2.0 is an authorization protocol. To understand it best, let’s remember the days when it did not exist. In the image below you can see a Facebook screen asking for our Gmail password to search for our contacts on Facebook and add them as friends.

Facebook from the days before OAuth, source: https://oauth.net/videos/

Think of what this means: Facebook developers would have access to your Gmail password, and essentially all your Gmail data including emails.

OAuth enables an app like Gmail to grant access solely to specific resources from that app, in this case, your contacts. It does so by creating an access token that can talk to an API and retrieve this data.

OAuth provides us with 2 tokens: a refresh toke and an access token.

The access token is short-lived and enables you to access the restricted API.

The refresh token’s role is to enable us to obtain new access tokens, without requiring the user to log in each time our access token expires, which results in a better user experience.

OpenID Connect (OIDC)

OpenID is an authentication protocol built on top of OAuth 2.0 and its main addition is the ID token. The ID token is intended for use with client-side applications, whereas the access token provided by OAuth 2.0 is meant to be used with the resource server (the API).

OAuth token audiences, source: https://oauth.net/videos/

JWT

JWT – Json Web Token is a standard method for representing claims securely between two parties. The information in the token is digitally signed to avoid tampering.

While OAuth doesn’t enforce token type, a lot of implementations use JWT tokens to store the refresh & access tokens. OpenID Connect on the other hand – defines that the token must be in the
JWT format.

NodeJS Microservices Authentication Strategies

Now that we have a basic understanding of the relevant terms (and before we dive into practice) we can start exploring possibilities to implement authentication in our microservices:

1.The obvious way: use a database to store user data, write your logic for creating users, registration, store passwords, etc. You can then create a form in your client-side where the user logs in, and once logged in you can store user information wherever you see fit (cookies, app state, etc).

2.OpenID Connect: you can use services like Google & Facebook that would enable your users to log in using their corresponding accounts. Then, you create a corresponding user in your database. Here you don’t need to implement any user creation UI or store passwords.

When your user logs in you can store the JWT token in a cookie, and your microservice could know who the user is according to that token. It could also allow or forbid certain actions based on that, but that’s out of the scope of this post.

You could of course combine this with option one, having some of the users created via identity providers like Google/Facebook and others in your own system.

3.Use an identity as a service tool like Auth0 / Okta, which essentially helps with both use cases above and can save you time implementing everything on your own. I won’t dive into this one, but you can check their websites for more info.

For the practical part of this guide I have selected option 2, because I feel it gives the most benefit in understanding authentication in Node.js and you will most likely use it anyway, whether directly or under the hood.

The Practice

Let’s begin implementing our service with OpenID connect enabled.

Here’s what we’re going to create:

account-service – a rest API that handles user creation. We will be using passport with passport-google-oauth strategy, which is based on OpenID Connect.
greeting-service – a simple rest api that greets the user.
A React app that lets the user login with google by consuming account-service and greets the user consuming the greeting-service.

Step 1 – Setting up Google project for login

1.Go to https://console.cloud.google.com/ and register if you haven’t done so yet

2.Create a new project

Google OAuth project creation

3.Select the newly created project, and click on Create Credentials

4.Select OAuth Client ID

5.Select Configure Consent Screen

6.Select Internal & Hit Create

7.Fill in the name and emails (you can leave the rest blank for now, we will get to it later)

8.In the scopes screen, click add scopes, and then select userinfo.email

Updating selected Google scopes

9.Hit Continue. Now we have our consent screen. Let’s go back to credentials & hit create credentials and choose OAuth Client ID

10.Choose your name, and for authorized redirect URI – add “http://localhost:5000/auth/google/callback” and hit create

11.You should now receive a pop-up with client id & client secret. Keep them, we will be using them soon

Step 2 – Creating account-service

Note: throughout this tutorial, for the sake of simplicity we’re using the default code generated with express-generator. So no good-looking typescript-like things to see here.

1. Create the project with express-generator & perform the installs needed

npx express-generator --no-view account-service
cd account-service
npm install
npm install --save jsonwebtoken passport passport-google-oauth cors

2. Express generator created some default routes.

We won’t be using the users’ one, but use the index.js. So I’m deleting references to it. The initial project looks like this.

3. In bin/www let’s change the port from 3000 to 5000

var port = normalizePort(process.env.PORT || '5000');

4. Add passport

Passport is authentication middleware for Nodejs. It’s very simple to use and supports all the options we need so I chose to use it. Passport uses strategies to handle certain types of login. passport-google-login is a strategy for logging in with Google and is based on OpenID Connect.

In our app.js file – let’s add the following code so that it looks like this:

const express = require('express');
const path = require('path');
const cookieParser = require('cookie-parser');
const logger = require('morgan');
const cors = require('cors');
const passport = require('passport');
const GoogleStrategy = require('passport-google-oauth').OAuth2Strategy;

const indexRouter = require('./routes/index');

const app = express();
// This is here for our client side to be able to talk to our server side. you may want to be less permissive in production and define specific domains.
app.use(cors());

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));


app.use(passport.initialize());
app.use(passport.session());

app.use('/', indexRouter);

passport.serializeUser(function(user, cb) {
 cb(null, user);
});

passport.deserializeUser(function(obj, cb) {
 cb(null, obj);
});

passport.use(new GoogleStrategy({
   clientID: 'your-google-client-id',
   clientSecret: 'your-google-client-secret',
   callbackURL: "http://localhost:5000/auth/google/callback"
 },
 function(accessToken, refreshToken, profile, done) {
   // here you can create a user in the database if you want to
   return done(null, profile);
 }
));

module.exports = app;

Notice we have a few interesting things here.

One – we added cors for the client-side (localhost:3000) to be able to make requests from our server-side. Having security in mind – you probably want to only allow specific domains in production.

Serialize & deserialize user – these are functions responsible for serializing & deserializing the user to and from the session.

GoogleStrategy – this is how we tell Passport we would be using google authentication.

Remember the client id & secret you saved earlier? Now is a good time to insert them.

5. Adding authentication routes

Now let’s go to the routes/index.js file and add the relevant routes.

const express = require('express');
const router = express.Router();
const passport = require('passport');
const jwt = require('jsonwebtoken');

router.get('/', function(req, res, next) {
 res.render('index', { title: 'Express' });
});

const TOKEN_SECRET = 'SECRET';

router.get('/auth/google',
 passport.authenticate('google', { scope : ['profile', 'email'] }));

router.get('/auth/google/callback',
 passport.authenticate('google', { failureRedirect: '/error' }),
 function(req, res) {
   const token = jwt.sign({ id: req.user.sub, name: req.user.name }, TOKEN_SECRET, {
     expiresIn: 60 * 60,
   });
   res.cookie('auth', token, { httpOnly: true });
   res.redirect('http://localhost:3000/');
});

module.exports = router;

TOKEN_SECRET – we will be using this to sign our JWT token.

/auth/google – the actual google login route.

Users are redirected to Google. Once they are done, they are redirected back to our server at /auth/google/callback. There we can create our JWT token.

Once created, we add it on the request as an httpOnly cookie, so that it is not accessible to javascript code (which is good in terms of security). You’ll soon see how this works on the client end.

When ready, we redirect back to the client-side.

Side note: we store the name inside the JWT for demonstration purposes, but you probably don’t need it, and may not be a best practice in terms of security.

Now that we’re done with our account service, let’s go on to the client-side.

Step 3: Create a client-side react application

npx create-react-app auth-strategies-client
cd auth-strategies-client/
yarn add axios

We now have a default react app. Let’s modify the app js file to contain a link to google authentication.

import logo from './logo.svg';
import './App.css';

function App() {
 return (
   <div className="App">
     <header className="App-header">
       <img src={logo} className="App-logo" alt="logo" />
       <a href="http://localhost:5000/auth/google">Sign in with Google</a>
     </header>
   </div>
 );
}

export default App;

And after running it with yarn start, it looks like this:

Click on “sign in with Google”. After doing so you should be redirected to Google for authentication.

(Make sure you run npm start on accounts-service for it to run in port 5000).

Let’s take a look at what happened after our call to the accounts-service/auth/google/callback:

1.accounts-service made a POST request to google, which returned an access token & id token.

Zoom in to the response

Zoom in to the services

[P.S. These images are generated using Aspecto’s live flow viewer. If you’d like to visualize your services the way I did, you should try Aspecto (for free). It takes 2 minutes to start sending traffic].

2.It used that token to make another GET request to google to get the user’s personal info.

Zoom in to the response

3.Our microservice has redirected us back to the client-side, with set-cookie in the response for our auth cookie creation.

Therefore – after that, you will be redirected back to the exact same screen, with one difference: If you open your browser’s DevTools, you should see an auth cookie that is http only:

Now we’re ready to make use of this token. That leads us to our greeting-service. Keep the client-side handy as we will modify it soon.

Step 4 – Setting up the greetings-service

1. Let’s create our service

npx express-generator --no-view greetings-service
cd greetings-service
npm install
npm install --save passport passport-jwt cors

2. Remove user.js file and routes in app.js, this is our fresh start:

const express = require('express');
const path = require('path');
const cookieParser = require('cookie-parser');
const logger = require('morgan');

const indexRouter = require('./routes/index');

const app = express();

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

app.use('/', indexRouter);

module.exports = app;

3. Modify port to 5001 in bin/www

var port = normalizePort(process.env.PORT || '5001');

4. Set up passport to read our JWT token by modifying app.js:

const express = require('express');
const path = require('path');
const cookieParser = require('cookie-parser');
const logger = require('morgan');
const cors = require('cors');

const indexRouter = require('./routes/index');

const app = express();
app.use(cors({ credentials: true, origin: 'http://localhost:3000' }));


app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

const passport = require('passport');
const JwtStrategy = require('passport-jwt').Strategy,
 ExtractJwt = require('passport-jwt').ExtractJwt;

app.use(passport.initialize());
app.use(passport.session());

app.use('/', indexRouter);

const cookieExtractor = function(req) {
 let token = null;
 if (req && req.cookies)
 {
   token = req.cookies['auth'];
 }
 return token;
};

const TOKEN_SECRET = 'SECRET';

const opts = {
 jwtFromRequest: ExtractJwt.fromExtractors([cookieExtractor]),
 secretOrKey: TOKEN_SECRET,
};

passport.use(
 'jwt',
 new JwtStrategy(opts, (jwt_payload, done) => {
   try {
     console.log('jwt_payload', jwt_payload);
     done(null, jwt_payload);
   } catch (err) {
     done(err);
   }
 }),
);

module.exports = app;

Here we need to pay attention to a few interesting things.

cookieExtractor is responsible for reading the token from the httpOnly cookie we created earlier and will be passed along with the request (more on that later).

Notice we must use the same TOKEN_SECRET we used to create the token in order to read it or we will get an invalid signature error when reading.

The extractor is then passed to the JwtStrategy, which is responsible for providing us with the jwt_payload. We could be fetching more info about the user from the database if we were to add a database, but for the sake of simplicity, I decided not to.

Now we’ll add our greeting route in index.js:

const express = require('express');
const router = express.Router();
const passport = require('passport');

router.get('/', function(req, res, next) {
 res.render('index', { title: 'Express' });
});

router.get('/greetme', (req, res, next) => {
 passport.authenticate('jwt', { session: false }, (err, user, info) => {
   if (err) {
     console.log('error is', err);
     res.status(500).send('An error has occurred, we cannot greet you at the moment.');
   }
   else {
     res.send({ success: true, fullName: `${user.name.givenName} ${user.name.familyName}` })
   }
 })(req, res, next);
});


module.exports = router;

What happens here is that Passport extracts the info from the JWT for us, and all we do is return it to the client.

Start the greetings-service in port 5001:

npm start

Step 5 – Modify the client side to send the httpOnly cookie

Since we want the client JWT token to not be accessible to any malicious javascript code, we have stored it in an httpOnly cookie.

(Side note: in real life, you may want to also make it secure so that it’s only accessible via HTTPS).

So, we want to perform our greeting request to the greetings-service. For that, we need to send the contents of the cookie to the server. Let’s do that then.

Back in our client-side react application, we modify App.js by adding a button:

import React, { useState } from 'react';
import axios from "axios";
import logo from './logo.svg';
import './App.css';

function App() {
 const [name, setName] = useState('');
 return (
   <div className="App">
     <header className="App-header">
       <img src={logo} className="App-logo" alt="logo" />
       <a href="http://localhost:5000/auth/google">Sign in with Google</a>
       <br />
       <button onClick={async () => {
         const result = await axios.get('http://localhost:5001/greetme', {
           withCredentials: true
         });
         setName(result.data.fullName);
       }}>Greet me please</button>
       {name && <span>{`Hi, ${name}`}</span>}
     </header>
   </div>
 );
}

export default App;

Now once we get a response with the current user’s full name, we would see “Hi, full name”.

Notice we added a basic axios with withCredentials: true – that is what makes the cookies pass alongside our request, for the server to extract.

And this emphasizes what happened here behind the scenes:

A simple GET request that returns a JSON with the user’s full name as it came from google and stored in the JWT token.

Here is what we get after clicking the button:

That’s it!

We have successfully created the account service for registration & JWT creation, and the greetings service that knows how to read the JWT token and provide data about the user.

I hope this helped you get a better understanding of authentication in general, and specifically while implementing it in nodejs.

Check out some of my other articles, like this one: Lerna Hello World: How to Create a Monorepo for Multiple Node Packages.

Lerna Hello World: How to Create a Monorepo for Multiple Node Packages

Tom Weiss — Wed, 12 May 2021 14:05:26 +0000

In this post, I will walk you through how to use Lerna to manage, and publish, two packages under the same monorepo. Publishing will be done to my private GitHub repository under the GitHub packages registry.

I decided to keep it as simple as possible, Lerna-only. No yarn workspaces to be found here.

Intro & Motivation For Using Lerna

Using a monolith, you have a single code base.

It is usually quite easy to share code between the different parts of the monolith, just import from the relevant file.

When it comes to microservices, however, by definition – you would have more than one microservice.

Most likely, you would have shared logic between the microservices, whether it is for everyday authentication purposes, data access, etc.

Then, one might (rightfully) suggest – let’s use a package. Where do you store that package? Yet another repo.

So far so good, but what happens when you have 35 shared packages between 18 different microservices?

You’d agree that it can be quite a hassle to manage all of these repos.

That is the part where Lerna comes in.

A tool that enables us to manage (and publish) as many npm packages as we want in a single repository.

1. Github Repository Creation

Create a new private github repository (I called mine learna but call it as you see fit).

2. Install Lerna & Setup the Project Locally

In order to set up Lerna in our project, we first need to install it globally, create a git repository locally and run lerna init:

npm install --global lerna
git init learna && cd learna
lerna init

Note: there are two modes for initializing the Lerna repo independent and fixed. We’re going to use the default one for simplicity reasons. Essentially what it means is all version numbers are tied together and managed in top-level lerna.json.

Read more about it here: https://github.com/lerna/lerna#how-it-works

Now let’s link this to our GitHub repository (replace names accordingly):

git remote add origin git@github.com:aspectom/learna.git

3. Create Lerna managed packages

Create two packages, hello-world and aloha-world (with the default options):

lerna create hello-world
lerna create aloha-world

lerna create is Lerna’s way to help us create packages managed by a Lerna initialized repo.

Inside both of the packages, modify the corresponding js files to have them greet as we want them to:

aloha-world.js

'use strict';

module.exports = alohaWorld;

function alohaWorld() {
 console.log('Aloha World');
}

hello-world.js

'use strict';

module.exports = helloWorld;

function helloWorld() {
 console.log('Hello World');
}

Now we have to make a modification in our package.json to contain the GitHub username of our account / organization:

{
 "name": "@aspectom/aloha-world",
 "version": "0.0.0",
 "description": "> TODO: description",
 "author": "Tom Z <tom@aspecto.io>",
 "homepage": "",
 "license": "ISC",
 "main": "lib/aloha-world.js",
 "directories": {
   "lib": "lib",
   "test": "__tests__"
 },
 "files": [
   "lib"
 ],
 "repository": {
   "type": "git",
   "url": "git@github.com:aspectom/learna.git"
 },
 "scripts": {
   "test": "echo \"Error: run tests from root\" && exit 1"
 }
}

Do this for both aloha-world and hello-world, and make sure to replace my GitHub username with your own.

PS: While we’re making managing multiple repos easier, here’s how you can make running multiple microservices locally feels like a walk in the park. It’s a simple, easy-to-use hack we, at Aspecto, came up with to make this process less messy – It’s called the local router.

At this point you should have a directory structure that looks like this:

At the root of the repository, add an empty LICENSE.md.

This will be necessary later to avoid this error when publishing:

lerna WARN ENOLICENSE Packages aloha-world and hello-world are missing a license.
lerna WARN ENOLICENSE One way to fix this is to add a LICENSE.md file to the root of this repository.
lerna WARN ENOLICENSE See https://choosealicense.com for additional guidance.

Let’s make our initial commit to GitHub.

git add .  
git commit -m 'Initial commit'
git push -u origin master

4. Generating a GitHub Personal Access Token

First, create a GitHub personal access token to publish and read packages:

Go to https://github.com/settings/profile,
Click on developer settings
Click on personal access token
Select write & read packages, which should also mark the repo automatically
Add a note so that you remember what it’s about and click on generate the token.

Now, go to your .npmrc file and add the following lines (can be local .npmrc in each repo or global ~/.npmrc, but beware – better to not commit this file):

//npm.pkg.github.com/:_authToken=TOKEN
@aspectom:registry=https://npm.pkg.github.com/

Do not forget to replace TOKEN with the token you have just created, and aspectom with your own GitHub account.

5. Publishing The Packages to GPR

Now let’s publish these packages to the GitHub package registry so that we can use them in a different project:

lerna publish --registry=https://npm.pkg.github.com/

If you had the following error, you probably omitted the registry part from lerna publish:

? Are you sure you want to publish these packages? Yes
lerna info execute Skipping releases
lerna info git Pushing tags...
Enter passphrase for key '/Users/tom/.ssh/aspecto_id_rsa': 
lerna info publish Publishing packages to npm...
lerna info Verifying npm credentials
lerna http fetch GET 401 https://registry.npmjs.org/-/npm/v1/user 1370ms
401 Unauthorized - GET https://registry.npmjs.org/-/npm/v1/user

Since it tries to go to npm registry instead of GitHub packages.

And if you had this error:

lerna info publish Publishing packages to npm...
lerna notice Skipping all user and access validation due to third-party registry
lerna notice Make sure you're authenticated properly ¯\_(ツ)_/¯
lerna http fetch PUT 404 https://npm.pkg.github.com/hello-world 694ms
lerna ERR! E404 404 Not Found - PUT https://npm.pkg.github.com/hello-world

You probably forgot to use @YOUR_GITHUB/package-name in one of your package.json files under the “packages” folder.

In my case – it was the hello-world package.

After resolving issues (if any) you should receive a success message, and looking at the repository you can see you have 2 packages:

Any time you want to publish, you have to make a change and commit it otherwise lerna will say that there’s no change.

You can make the change or force Lerna to publish by adding --force-publish to the lerna publish command, like this:

lerna publish --registry=https://npm.pkg.github.com/ --force-publish

6. Using The Packages in a Different Project

First, create a project to consume the aloha-world and hello-world packages:

mkdir use-lerna-repo
cd use-lerna-repo/
yarn init

Assuming you’ve used global .npmrc, no further steps needed to consume the packages with yarn or npm install.

If you used local npmrc in your lerna repo, copy it to the use-lerna-repo root folder.

yarn add @aspectom/aloha-world
yarn add @aspectom/hello-world

Create an index.js file:

const helloWorld = require('@aspectom/hello-world');
const alohaWorld = require('@aspectom/aloha-world');

helloWorld();
alohaWorld();

Package.json for this project:

{
 "name": "use-lerna-repo",
 "version": "1.0.0",
 "main": "index.js",
 "license": "MIT",
 "scripts": {
   "start": "node index.js"
 },
 "dependencies": {
   "@aspectom/aloha-world": "^0.0.4",
   "@aspectom/hello-world": "^0.0.4"
 }
}

Then, run node index.js and you should get the following output:

$ node index.js
Hello World
Aloha World

And voila! We have just finished creating, publishing, and consuming our lerna-managed packages in the one monorepo.

Good luck, we at Aspecto wish you years of happy packaging and a lot of downloads!

DEV Community: Tom Weiss

Distributed Tracing for Kafka with OpenTelemetry in Python

What to Expect

What is Apache Kafka?

What is OpenTelemetry?

OpenTelemetry Traces

What are OpenTelemetry Instrumentations?

How to Use Kafka with OpenTelemetry in Python

The Practical Part

Visualizing OpenTelemetry Tracing Data

Distributed Tracing for Kafka with OpenTelemetry in Node

What to Expect

What is Apache Kafka?

What is OpenTelemetry?

What is Distributed Tracing?

What is OpenTelemetry Instrumentation?

How to use Kafka with OpenTelemetry in Node

The Practical Part

Visualizing OpenTelemetry Traces in Node

TroubleShooting OpenTelemetry Node Installation Issues

Additional OpenTelemetry Resources

OpenTelemetry Collector: A Friendly Guide for Devs

What to Expect

How does the OpenTelemetry Collector work?

Receivers

Processors

Sampling

Data alteration

Exporting metrics from spans

Exporters

The Architecture of Running an OpenTelemetry Collector

Gateway

Agent

OpenTelemetry Collector Deployment Methods

Configuring OpenTelemetry Collector Gateway with Our App and Jaeger for Visualization

Setting up the collector gateway

Create a tracing.js file to enable tracing

Usage with observability vendors

Configuring Your Collector to Send Data to an Observability Vendor

OpenTelemetry Collector Extensions

Get Started with OpenTelemetry Python: A Practical Guide

Intro to OpenTelemetry

Hello World: OpenTelemetry Python

Create spans and see them in the console output:

Getting Started with OpenTelemetry Python and Jaeger – Advancing

An auto instrumentation example

Why are they not in the same trace?

Bonus: Advanced Visualization with Aspecto

How To Use OpenTelemetry With AWS Lambda

Setting Up

Add the handler.js

Add the lambda wrapper file that enables tracing with OpenTelemetry

A note on disableAwsContextPropagation

Deploy the lambda

Add environment variable

Calling the lambda function

This is the outgoing HTTP span

How to Improve Your Integration Tests Using OpenTelemetry

Introduction

What it would look like to use OpenTelemetry in an integration test

How does Malabi help?

The practical part – how to take your existing NodeJs microservice and utilize OpenTelemetry to make assertions in an integration test

Part 1 – The microservice code

Part 2 – The Test Code

service-under-test.spec.ts file:

Examined example 1 – test named “successful /users request”:

Examined example 2 – test named “successful create and fetch user”:

How To Write Integration Tests Easily Using Trace-Based Testing

Introduction

The motivation for writing an integration test

Part 1 – Setting up the tested microservice

Step 1 – Set up the project

Step 2 – create the views

Step 3 – Adding routes

Step 4 – Adding app.js

Step 5 – Running the app

Part 2: Writing an integration test

Trace based testing

Step 1 – perform installs

Step 2- add a new folder & test file: tests/integ.spec.js

A note on `disableAwsContextPropagation`