Exploring OpenTelemetry and Jaeger

#dotnet #docker #jaeger #opentelemetry

Have you followed my tutorial regarding the development of forum-based web applications? I recommend following the series before we start this tutorial. You can check the applications in this repository.

bervProject / forum-api-microservices

Forum API Microservices

forum-api-microservices

Forum API Microservices

Directory Structure

/app : Microservices Source Code
- Currently we have Auth Service, User Service, and Thread Service.
docker-compose.yml : Containerize MongoDB & Redis. Will help for development.

Microservices Development

You will need to copy or modify docker-compose.yml to ignore the deployment of microservices.
Run Redis & MongoDB using docker compose up -d.
Go to the microservice you want to update and read the README.md of each directory to understand how to run them.

Development

Build Images of Microservices: docker compose build
Run all: docker compose up -d

Software Architecture

License

MIT

MIT License
Copyright (c) 2022 Bervianto Leo Pratama's Personal Projects

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute,

…

View on GitHub

Preparation

You can start from this commit.
Please check this pull request for the changes.
In short, you only need Docker and the source codes in this tutorial.

Overview

In short, we will call the tracing API in Jaeger from AuthService, ThreadService, and UserService. ThreadService and UserService will call in port 14268, but AuthService will call in port 4318. ThreadService and UserService will use the Jaeger exporter library, but AuthService will use the OTLP exporter library.

Let's Coding

We will add Jaeger in our previous docker-compose.yml. You can choose to expose all the ports or just expose the UI port, which is 16686. We will communicate with the internal network of containers.

  jaeger:
    image: jaegertracing/all-in-one:1.45
    networks:
      - backend
    environment:
      - COLLECTOR_ZIPKIN_HTTP_PORT=:9411
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - 6831:6831/udp
      - 6832:6832/udp
      - 5778:5778
      - 16686:16686
      - 14268:14268
      - 14269:14269
      - 14250:14250
      - 9411:9411
      - 4317:4317
      - 4318:4318

Add these packages to ThreadService and UserService.

<PackageReference Include="OpenTelemetry.Exporter.Jaeger" Version="1.4.0" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.4.0" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.14" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.14" />

Add these codes to the UserService (Program.cs).

using System.Diagnostics;

using OpenTelemetry.Exporter;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
        tracerProviderBuilder

.AddSource(DiagnosticsConfig.ActivitySource.Name)
            .ConfigureResource(resource => resource

.AddService(DiagnosticsConfig.ServiceName))
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddJaegerExporter());

// after UseCors()
app.Use(async (context, next) =>
{
    context.Response.Headers.Add("trace-id", Activity.Current?.TraceId.ToString());
    await next();
});
// before MapGet("/users", ...

// after app.Run()
public static class DiagnosticsConfig
{
   public const string ServiceName = "UserService";
   public static ActivitySource ActivitySource = new ActivitySource(ServiceName);
}

Add these codes to the ThreadService (Program.cs). The changes are the same with UserService. The difference is the ServiceName.

using System.Diagnostics;

using OpenTelemetry.Exporter;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
        tracerProviderBuilder

.AddSource(DiagnosticsConfig.ActivitySource.Name)
            .ConfigureResource(resource => resource

.AddService(DiagnosticsConfig.ServiceName))
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddJaegerExporter());

// after UseCors()
app.Use(async (context, next) =>
{
    context.Response.Headers.Add("trace-id", Activity.Current?.TraceId.ToString());
    await next();
});
// before MapGet("/users", ...

// after app.Run()
public static class DiagnosticsConfig
{
   public const string ServiceName = "ThreadService";
   public static ActivitySource ActivitySource = new ActivitySource(ServiceName);
}

Update the package.json of AuthService. Please run yarn to update the yarn.lock.

"scripts": {
    "start": "node --require ./lib/instrumentation.js lib/index.js",
    "start:dev": "ts-node-dev --require ./instrumentation.ts src/index.ts",
    "compile": "tsc",
    "lint": "eslint .",
    "fix": "eslint . --fix"
  },
  "dependencies": {
    "@opentelemetry/api": "^1.4.1",
    "@opentelemetry/exporter-metrics-otlp-proto": "^0.39.1",
    "@opentelemetry/exporter-trace-otlp-proto": "^0.39.1",
    "@opentelemetry/instrumentation-express": "^0.32.2",
    "@opentelemetry/instrumentation-http": "^0.39.1",
    "@opentelemetry/sdk-metrics": "^1.13.0",
    "@opentelemetry/sdk-node": "^0.39.1",

Update index.ts in the AuthService.

import api from '@opentelemetry/api';

  // add these before logger.info("Getting Request", requestMeta);
  const activeSpan = api.trace.getSpan(api.context.active());
res.header("trace-id", activeSpan?.spanContext().traceId);

Add intrumentation.ts in the AuthService.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { ExpressInstrumentation } from "@opentelemetry/instrumentation-express";
import { Resource } from "@opentelemetry/resources";

const jaegerHost = process.env.JAEGER_URI || "http://localhost:4318";

const resource =
    Resource.default().merge(
        new Resource({

[SemanticResourceAttributes.SERVICE_NAME]: "AuthService",
        })
    );

const sdk = new NodeSDK({
    resource: resource,
    traceExporter: new OTLPTraceExporter({
        url: `${jaegerHost}/v1/traces`,
    }),
    instrumentations: [
        new HttpInstrumentation(),
        new ExpressInstrumentation(),
    ]
});

sdk.start()

Update the docker-compose.yml for thread-services and user-services section.

depends_on:
  - jaeger
  - mongodb
  - redis
environment:
    # add these envs
  OTEL_EXPORTER_JAEGER_ENDPOINT: http://jaeger:14268/api/traces
  OTEL_EXPORTER_JAEGER_AGENT_HOST: jaeger

Update the docker-compose.yml for auth-services section.

depends_on:
  - jaeger
  - mongodb
  - redis
environment:
   JAEGER_URI: http://jaeger:4318

Let's build the docker image for those services (AuthService, ThreadService and UserService). You can use docker compose build.
Start the services. docker compose up -d. Ensure all the services are running. docker compose ps -a.
You can start to call the APIs. So you can see the tracing.
I recommend exploring the tracing when creating the thread and login.