Tracing with OpenTelemetry

Tracing gives you great insights into certain bottlenecks within your application. I'll go over the steps needed to enable OpenTelemetry within a Fastify application on Google Cloud Run. I got inspired by the Nearform webinar from Daily.Dev.

https://www.youtube.com/watch?v=UKaJDmwIIpE

By default Cloud Run already has tracing enabled on requests that enter your cloud run instances. So if you go to traces in the console, you should see some traces for every HTTP endpoint of your application. You can see an example below:

Now we can start adding additional instrumentation for our application. There are multiple libraries proposed by Google:

• OpenTelemetry: recommended by Google
• OpenCensus: only alpha support and NodeJS support not even mentioned by Google
• Google Client library

OpenTelemetry

As described on the website of OpenTelemetry

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

It's still in beta, but general availability should arrive soon.

Installing

Make sure to initialize all OpenTelemetry packages at the start of your application so that it can patch the necessary calls to underlying libraries. In this example, we add HTTP, TypeORM and Postgress instrumentation with the following instrumentation packages:

• @opentelemetry/instrumentation-http
• @opentelemetry-instrumentation-typeorm
• @opentelemetry/instrumentation-pg

import opentelemetry, { DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { TraceExporter } from '@google-cloud/opentelemetry-cloud-trace-exporter';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { TypeormInstrumentation } from 'opentelemetry-instrumentation-typeorm';

export function tracing() {
  // Enable OpenTelemetry exporters to export traces to Google Cloud Trace.
  // Exporters use Application Default Credentials (ADCs) to authenticate.
  // See https://developers.google.com/identity/protocols/application-default-credentials
  // for more details.
  const provider = new NodeTracerProvider();

  // Initialize the exporter. When your application is running on Google Cloud,
  // you don't need to provide auth credentials or a project id.
  const exporter = new TraceExporter();

  // Configure the span processor to send spans to the exporter
  provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
  provider.register();
  opentelemetry.trace.setGlobalTracerProvider(provider);
  registerInstrumentations({
    instrumentations: [
      new HttpInstrumentation(),
      new PgInstrumentation(),
      new TypeormInstrumentation({
        // see under for available configuration
      }),
    ],
  });
  return { provider };
}

The Google Cloud trace exporter automatically uses the Cloud Run service account, so make sure this service account has access to utilize the Tracing API. You can configure or verify this in the IAM Console.

Once you have this added to your application and deployed on Cloud Run, you should be able to see more in-depth traces. The trace ID can always be found in the response header of your request: 'X-Cloud-Trace-Context'.

Google Coud - Daily analysis reports

If you feed Google Cloud with this tracing data, you will also get more detailed insights. An example where we had a significant impact on specific API calls is also noticed automatically by Google Cloud.

Summary

Easy to implement and can give you more insights over time. The Google Cloud Trace dashboard is quite simple and easy to use, so the entry barrier is relatively low.