DEV Community: austin

Ask Austin: Putting The IR into ObseRvabIlity

austin — Fri, 29 Apr 2022 14:17:48 +0000

If you've listened to me on a podcast, or spoken to me online, you might have heard me claim that 'observability is the foundation that you build on.' What does that mean, though? In this edition of 'Ask Austin,' I'll share some insights I’ve picked up talking to our customers about how observability is a foundational component of DevOps practices, such as incident response.

Schrodinger's U+1F408

The most crucial currency of a startup is time — you're under immense pressure to ship features and write reliable, resilient code. Into every life a little rain must fall, though, and failure seems inevitable. Some people might tell you that failure can be boxed away where it won't ever happen, but it really is impossible to create something perfect. We have to understand, tolerate, and ameliorate failure rather than shy away from it.

Observability is absolutely critical to this effort. Without observability, you can't quantify failure and know how to prioritize it. More specifically, without observability, you aren't able to connect system state to business outcomes using SLO's, which means you lack the ability to understand failure in the context of what you're actually trying to do.

This is where things get tricky, though — [your SLO's tell you when performance is impacting users (https://go.lightstep.com/register-the-secret-to-high-quality-slos-on-demand-webinar.html), but how do you start to tolerate, and ameliorate that failure? This is where observability alone can't help — you need to adopt best practices and tools that can aid in communication and recovery efforts.

Building a Culture of Resiliency

In order to respond and mitigate failure in your system, it's not enough to be aware of it. Your ability to respond starts before failures even happen, though — before you even write the first line of code in a service. You need to start out the way you intend to keep going. I'm going to list three strategies you should adopt in order to build a culture of resiliency in your engineering team:

Monitoring and observability should be 'day zero' problems. Ensure that every new feature or service has appropriate tracing, metrics, and logging telemetry being output and captured through OpenTelemetry tools, and test these telemetry outputs as part of your CI/CD process.
Effective incident response relies on documentation and communication. Create a defined process for building playbooks that include links to dashboards, explanations of alerts and SLOs, and service ownership. Keep them up to date as part of your recurring engineering workload — especially as the team grows or contracts. If you're writing everything down as you go along, you're more resilient to unexpected changes in team composition.
Understand alerts before you build them. Ideally, you should only be alerting off of SLOs for application behavior! I've seen too many people get burnt out by blindly setting up alerts for their infrastructure resources and spend hours or days chasing down failures that didn't actually matter that much to their customers. Alerts need tending and grooming, just like anything else — you can try it yourself with your existing alerts. Spend a week on-call noting if a triggered alert actually impacted the customer experience in some way.

Incidents Happen

It's inevitable that things will break, so we need to build systems that tolerate failure. Observability and incident response tools are helpful in this quest. Observability tells you where, and why, a failure occurred. Incident response tools help you manage and respond to those failures. The real question you need to ask yourself is how much time you want to spend on the management of your observability and incident response platforms.

While tools such as Prometheus and Upptime will let you monitor, alert, and communicate incidents to your users, there's a lot more than just tracking metrics and pinging your page to really understand your system. The cost of stitching together your own bespoke solution in terms of time alone can be weeks or months, which can strain already over-extended engineers. Moreover, wouldn't you rather be working on things that provide value to your end-users rather than spinning up Grafana again?

Traditional paid services are an option, but many of them incur a management cost. Seat-based pricing that controls how many people can access the tool is a pain, especially since most companies make single-sign on an 'enterprise' feature. This isn't universally true — Lightstep has no limits on the amount of users of our tools, because your data is useful to all of your engineers, not just a few. If you’re already feeling underwater on managing alert fatigue, try us out and see the difference for yourself.

If this all seems overwhelming, I understand — especially if you're at a smaller company or on a smaller team, some of this probably feels like stuff you haven't started to really deal with in earnest yet. The best advice I can give you is to find your peers and build community with them. All of the advice in the world is great, but finding someone that's been where you are and can help direct you along the way is even better. If you're interested in being a part of ours, you can join us on our Discord.

I hope this answered a few of y'alls questions about observability and incident response. If you've got more burning questions, feel free to hit me up on our Discord server or on Twitter and I'll answer them next time!

OpenTelemetry Python: All you need to know

austin — Thu, 03 Dec 2020 14:13:48 +0000

Hi all, tedsuo back again, dropping a knowledge bomb and a bunch of stale-yet-crunchy pop culture references. Last week we covered Node; this week we are going to dive into Python.
If you crack open OpenTelemetry, you’ll quickly discover that there’s a lot there. But, as a developer applying OpenTelemtry to your application, 99% of what’s in there doesn’t matter.

TL;DR

All you need to know is:

Initialization: How to start and shutdown cleanly.
Tracer methods: get_tracer, get_current_span, startSpan, and withSpan.
Span methods: setAttribute, addEvent, recordException, setStatus, and end.

Seriously, that’s it. If you want to try it out, follow the guide below. A heavily commented version of the finished tutorial can be found at https://github.com/tedsuo/otel-python-basics, please use it as a reference when you get started with instrumenting your own application.

Python: Off to Docker

(If you already have a python setup you’re fine with, just skip this bit).

This time, let’s do our local development in docker. Managing Python installations can be a bit of a snake’s nest, especially on a mac, where the current default python3 installation has a bit of an issue with psutil, which we depend on.

First, make a directory for your application:

mkdir otel-py-demo && cd otel-py-demo

Install docker, then grab a python image and start a container that mounts your application directly. If you’re in your app directory, the following starts a python container with your current directory mounted at /app, and logs you into a bash shell within the container.

docker run -it -v $PWD:/app python:3.8 bash

Whenever you need to log into a new terminal, find the container ID, and then use it to exec into a bash shell.

% docker ps
CONTAINER ID   IMAGE         COMMAND    ETC...
bc1e3c27f4d0   python:3.7   "python3"   More columns
% docker exec -it bc1e3c27f4d0 bash

And that’s “all you need to know” about docker. 😁

Hello, World

Once again, it is time to say Hello to this cruel World.
First, exec into your docker container and install the required dependencies. For this basic app, we’re going to use flask for the server, and requests for the client.

cd /app
pip install flask
pip install requests

Create a file named server.py and make the world’s simplest app.

#!/usr/bin/env python3

from flask import Flask
from time import sleep

PORT = 8000
app = Flask(__name__)

@app.route("/hello")
def hello():
   sleep(30 / 1000)
   return "hello world\n"

if __name__ == "__main__":
   app.run(host="0.0.0.0", port=PORT)

Amazing. Add a client at client.py which makes five requests in a loop:

import requests

for i in range(5):
   r = requests.get("http://localhost:8000/hello")
   print(r.text)

In one docker terminal, start the server:

> export FLASK_ENV=development
> python server.py
 * Serving Flask app "server" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)

In another, run the client:


> python client.py
Hello World
Hello World
Hello World
Hello World
Hello World

It works!

Install OpenTelemetry

Ok so if you know python that was all very boring. Here’s the stuff you came for: installing opentelemetry.

First, you need to pick the analysis tool you want to target. I work on Lightstep, and we have a free Community account specifically for trying out OpenTelemetry like this. These instructions assume you have one of those. If you’d like to set up Jaeger instead, you can find installation instructions here.

To connect to Lightstep, install the Lightstep distro for OpenTelemetry, the OpenTelemetry launcher. Lightstep is OpenTelemetry native, all the launcher does is install the relevant packages and make the configuration simpler.

pip install --use-feature=2020-resolver opentelemetry-launcher

The launcher will install the core opentelemetry components, plus the currently available instrumentation. Just to unpack it a bit, there are three critical packages, beyond the launcher itself, which are worth understanding as they explain how OpenTelemetry is structured.

opentelemetry-api: the API package contains the opentelemetry instrumentation API. This package only contains interfaces, no implementation. It is safe to bring into any package without concern that a large dependency chain may follow it.
opentelemetry-sdk: the SDK package contains the standard implementation for opentelemetry. This implementation is a framework written in python, allowing for various exporters, samplers, and lifecycle hooks to be plugged in, so that a wide variety of analysis tools can be supported.
opentelemetry-instrumentation: this package contains two command line tools for automatically instrumenting your application: opentelemetry-bootstrap and opentelemetry-instrument. We’re going to use both of them now.

Install Automatic Instrumentation

The first command to learn is opentelemetry-bootstrap. This will inspect the currently installed site-packages, and detect any packages we have instrumentation available for. By default, it prints out the packages to be copied into a requirements file, but it can also install them for you. For this example, let’s run it in installation mode.

opentelemetry-bootstrap --action=install

And that’s it for installation!

Run with OpenTelemetry

The easiest way to run OpenTelemetry is via the opentelemetry-instrument command, using env vars for configuration. You can find a list of available configurations here, but there are only two which are required:

LS_SERVICE_NAME - The name for this type of service. We’ll use hello-sever and hello-client, respectively.
LS_ACCESS_TOKEN - You can find this one by first logging into your Lightstep account (or create one), then going to the settings page. Use the clipboard button to copy the access token.

To run opentelemetry with Lightstep, first log into your account and find your access token on the Settings page. Use the clipboard button to copy the access token.

export FLASK_ENV=development
export LS_SERVICE_NAME=hello-server
export LS_ACCESS_TOKEN=<ACCESS TOKEN>
opentelemetry-instrument python server.py

Do the same for the client.

export LS_SERVICE_NAME=hello-client
export LS_ACCESS_TOKEN=<ACCESS TOKEN>
opentelemetry-instrument python client.py

Check for data by clicking on the explorer:

Huzzah! We see some spans. Click into one and check out the trace.

Let’s pause for a second and review the data we are looking at. There are two spans, one from the requests package on the client, and one from the flask package on the server. Clicking on a span, you can see that it is already rich with data.

http.* and net.* – these conventions describe everything about the request.
instrumentation.name – this describes the instrumentation package which generated the span.
span.kind - either client, server, or internal.

The representation of common concepts like HTTP are standardized across languages. so that analysis tools can automatically interpret the data they are looking at. We refer to these standardized attributes as Semantic Conventions. The complete list can be found here.

No code required

The biggest, most important note is that we added OpenTelemetry to our service, but didn’t write any code. Everything could be done from the command line. This means that OpenTelemetry can potentially be added to a service by an operator, with a simple modification to deployment.

I highly recommend this approach as a first pass, before adding any additional detail. OpenTelemetry needs to be installed in every service in order for distributed tracing to work. It is more important to get every service instrumented at a high level than it is to dig in and deeply instrument the application code in a particular service. Library level instrumentation (flask, requests, redis, etc) will give you enough information to set up alerting root causing issues.

If this is the first time you’ve added distributed tracing to your system, don’t be surprised if a number of latency-related issues immediately become visible! After you’ve done a wide-scale rollout, you can dig in selectively and add detail where needed. Converting your existing logs to span events is another great way to add detail without having to write a lot of code.

Using the OpenTelemery Python API

Okay, automation is great, but eventually, you are going to want to add detail. Spans are already decorated with standardized attributes, but once you’re settled in, you will want to start adding more detail.

The most important details to add are application-level attributes critical to segmenting your data. For example, a projectID allows you to differentiate between errors that are affecting everyone connecting to a service, vs errors that are localized to a handful of accounts. Those would be two very different scenarios, and you would probably start looking in different places based on that feedback.

Also, logs. They are a thing. OpenTelemetry has a structured logging facility, we just call it events.

Adding data to the current span

To add additional data to your trace, you need access to the currently active span. Since context propagation is already set up, thanks to the automatic instrumentation, there is already a span available to you.

Attributes are simply key value pairs. Events consist of a message and a dictionary of attributes.

from opentelemetry import trace

@app.route("/hello")
def hello():
   # get the current span, created by flask
   span = trace.get_current_span()
   # add more attributes to the server span
   span.set_attribute("http.route", "some_route")
   # add events (AKA structured logging)
   span.add_event("event message",
                  {"event_attributes": 1})

   sleep(20 / 1000)
   return "hello"

It’s best to add data to existing spans, rather than create child spans. This keeps all of the attributes grouped together, which makes for better indexing.

Creating a child span

Of course, you are going to want to create child spans on some occasions. A span represents a distinct operation - not an individual function, but an entire operation, such as a database query. Generally, this means you shouldn't be creating spans in your application code, they should be managed as part of the framework or library you are using.

But, that said, here is how you do it. First, create a tracer. A tracer is just a namespace - it lets you know which package created the span, via the instrumentation.name attribute (you can also add a version as a second parameter).

Span management has two parts - the span lifetime and the span context. The lifetime is managed by starting the span with a tracer and adding it to a trace by assigning it a parent.

# Start the span with a name and a parent span
child = tracer.start_span("my_operation", parent=parent)

try:
   # pass the span around as a parameter
   do_work(span=child)
finally:
   # End the span, which measures the span duration and
   # triggers the span data to be exported.
   # WARNING: failing to end a span will create a leak.
   child.end()

Using spans directly like this is cumbersome. Instead, we want to create a new context where the span is active so that it can be accessed by get_current_span instead of passing it around. In almost all cases, the easiest way to manage a span is by calling start_as_current_span.

from opentelemetry import trace

# create a tracer and name it after your package
tracer = trace.get_tracer(__name__)

@app.route("/hello")
def hello():
   # add latency to the parent span
   sleep(20 / 1000)

   # always create a new context when starting a span
   with tracer.start_as_current_span("server_span") as span:
     # add an event to the child span
     span.add_event("event message", 
                    {"event_attributes": 1})
     # get_current_span will now return the same span
     trace.propagation.get_current_span().add_attribute()
     # add latency to the child span
     sleep(30 / 1000)
     return "hello"

If you ever need to create a span in your application code, I strongly recommend using the above pattern.

Recording errors

Ok, one final bit. We’ve covered spans, attributes, and events. But what about exceptions? Exceptions are reported as events, but they should be properly formatted. As a convenience, OpenTelemetry provides a record_exception method for capturing them correctly.

from opentelemetry.trace.status import StatusCode

@app.route("/hello")
def hello():
   span = trace.propagation.get_current_span()
   try:
       1 / 0
   except ZeroDivisionError as error:
       # record an exception
       span.record_exception(error)
       # fail the operation
       span.set_status(StatusCode.ERROR)
       print("caught zero division error")

Uncaught exceptions are automatically recorded as errors.

Conclusion

And that is that. All you need to know to get started with tracing in Python.

Hopefully, it’s clear that If you stick with the above patterns, you can get a great deal of visibility with very little work. Of course, there are many more details and options; you can check out the API documentation for more information. I also have a more involved getting started guide; it works as a handy reference for all of the procedures described above.

OpenTelemetry is still in beta due to API changes, but it is also already in production across many organizations. If you stick to a Distro and automated instrumentation, you can use OpenTelemetry today without much fear of a breaking change affecting you.

Also: consider joining our community! There are plenty of libraries left to instrument. You can find us on GitHub, or say hi on gitter.

OpenTelemetry NodeJS: All you need to know

austin — Thu, 03 Dec 2020 14:13:15 +0000

Hi all, tedsuo here. We’re passing an important milestone on OpenTelemetry: the tracing specification is about to be frozen, and release candidates for OpenTelemetry tracing implementations will be coming soon, with metrics following in the next couple of months.

While we are putting our core documentation together, I thought now would be a good time to point out how simple it is to actually use distributed tracing in JavaScript. OpenTelemetry is a large framework, it has a lot of options and a lot of surface area. But, as an end user, you don’t have to worry about all of that. So, forget the details: this walkthrough contains all you need to know to actually use OpenTelemetry in Node. Apply this walkthrough to your application, and you are good to go.

TL;DR

All you need to know is:

Initialization: How to start and shutdown cleanly.
Tracer methods: getTracer, getCurrentSpan, startSpan, and withSpan.
Span methods: setAttribute, addEvent, recordException, setStatus, and end.

Seriously, that’s it. If you want to try it out, follow the guide below. A heavily commented version of the finished tutorial can be found at https://github.com/tedsuo/otel-node-basics.

Hello, world

For this tutorial, we’re going to make a very, very simple application: an express service that responds to http://localhost:9000/hello with “Hello World.” It’s as basic as it is original!

First, make a directory to contain your project, and install express:

npm i express

Once we have that, let’s get to coding. Make a file called server.js and serve up some hello world:

const express = require('express');

const app = express();

app.get('/hello', (req, res) => {
 res.status(200).send('Hello World');
});

app.listen(9000);

Then, make a file called client.js which sends 5 requests to the server and then exits.

const http = require('http');

function makeRequest() {
   http.get({
     host: 'localhost',
     port: 9000,
     path: '/hello',
   }, (response) => {
     const body = [];
     response.on('data', (chunk) => body.push(chunk));
     response.on('end', () => {
       console.log(body.toString());
     });
   });
}

for (let i = 0; i < 5; i++) {
 makeRequest();
}

Boot up server and check that it works:

> node server.js

With the server running, test the client in another tab:

> node client.js
Hello World
Hello World
Hello World

OpenTelemetry Architecture in 30 seconds

Ok, I said no details, but here is one that is actually helpful. OpenTelemetry clients have two major components: the SDK and the API. The SDK is the actual framework, the API is what you use to instrument your code.

This separation provides loose coupling: your application code only depends on the API, which has virtually no dependencies and acts like a no-op when the SDK is not installed. This allows packages to add instrumentation without automatically pulling in the implementation’s dependency chain (think grpc, etc). This separation of concerns is especially helpful for OSS libraries that want to bake in instrumentation, but don’t want to create overhead or dependency conflicts when OpenTelemetry is not being used.

Tip: Never reference any SDK package outside of installation and setup. All other packages and application code should only depend on the API.

In case you were wondering, while there are two Javascript SDKs - one for NodeJS and one for the browser - there is only one Javascript API. Instrumented code remains portable between both environments.

Pick an OpenTelemetry backend

Ok, let’s add OpenTelemetry to this application. To test our tracing, you’ll need a place to send the data.

At Lightstep, we created free-for-life community accounts specifically for making OpenTelemetry easy to experiment with. If you don’t already have one, please grab an account.

If you’d like to use Zipkin or Jaeger instead, this getting started guide will walk you through the setup. Once you’re set-up, you can come back here and follow the rest of the tutorial.

Install the NodeJS OpenTelemetry Launcher

Since we’re connecting to Lightstep, we’ll also be using the Lightstep Distro of OpenTelemetry, the OpenTelemetry Launchers. Distros package up any plugins and configuration needed to talk to a particular backend. At the moment, we’re still fleshing out the full definition of a Distro (what is allowed, and what isn’t), but the basic point is to make getting started easier by reducing configuration boilerplate. If you want more detail, you can check out this blog post where I initially proposed the concept.

Installing the OpenTelemetry Launcher package will also install OpenTelemetry, plus all currently available instrumentation.

npm i lightstep-opentelemetry-launcher-node

Create an OpenTelemetry initialization file

To instrument your server, you need to start the OpenTelemetry SDK before loading your application. As of v0.12, OpenTelemetry NodeJS loads asynchronously, This is actually the trickiest bit of OpenTelemetry right now, and in future versions will move to a simpler, synchronous startup. However, for now you can copy and paste the approach below, and it will work for any application.

Create a file called server_init.js. This will serve as your new entry point. You can copy and paste the below code.

const {
 lightstep,
 opentelemetry,
} = require('lightstep-opentelemetry-launcher-node');

const sdk = lightstep.configureOpenTelemetry({
 accessToken: '<ACCESS_TOKEN>',
 serviceName: 'hello-server-1',
 serviceVersion: 'v1.2.3',
 propagators: 'tracecontext,b3',
});

sdk.start().then(() => {
 require('./server');
});

function shutdown() {
 sdk.shutdown().then(
   () => console.log("SDK shut down successfully"),
   (err) => console.log("Error shutting down SDK", err),
 ).finally(() => process.exit(0))
};

process.on('exit', shutdown);
process.on('SIGINT', shutdown);
process.on('SIGTERM', shutdown);

Configure the launcher with your Lightstep Access Token (You can find your Access Token on the settings page). Create a client_init.js file in the same manner, only change the serviceName to ‘hello-client’ and the required startup file to ‘./client’.

Use the launcher to start the SDK. Once the SDK has completed its setup, require your original entry point to start your application.

Why load your application in two phases like this? If your application begins requiring packages (or running) before OpenTelemetry is set up, it can create issues. By initializing OpenTelemetry in a separate file, and only requiring the rest of your application after the SDK is started, OpenTelemetry has an opportunity to automatically apply any available instrumentation, as well as auto-detect any available system resources before your application starts to run. It also ensures that your application loads normally.

Run your application with OpenTelemetry

Start your newly auto-instrumented server and client. Let’s also turn the debug logs on, so we can see what OpenTelemetry is doing.

export OTEL_LOG_LEVEL=debug
node server_init.js

export OTEL_LOG_LEVEL=debug
node client_init.js

At startup, the debug logs will print out the configuration, and list every successfully loaded instrumentation library. Every time the tracer flushes data, all of the spans which have been exported are printed out. This can be really helpful for debugging when you are setting up.

Check out what automatic instrumentation gives you

Switch over to Lightstep, or your backend of choice, and confirm the spans were received:

Yup, we see spans. Click through and look at a trace:

Notice that we see a client span from hello-client, a server span from hello-server, and several internal spans representing built-in express components. Also, notice that the client and server spans are already populated with HTTP, network, and other attributes.

All of this common information is standardized across instrumentation as semantic conventions. An HTTP request will always be described with the same keys and values, regardless of what language or package it comes from.

This is a lot of really useful information. We already have a complete trace, with a lot of detail, and we haven’t written any instrumentation yet. When rolling out OpenTelemetry, this is the approach I recommend. Get OpenTelemetry installed into every service and ensure that context is propagating correctly, before adding any further detail. This will be enough information to set up error monitoring and identify latency issues.

The OpenTelemetry Javascript API

Ok, so the out-of-the-box experience will get you a long way, but of course, you will eventually want to add additional application data.
Spans should ideally be managed by your application framework. In this case, the express framework manages the span for you. In your application code, you can continue to decorate these spans with more information. There are two primary types of data you will want to add: attributes and events.

Span attributes are indexes for segmenting your data. For example, you may want to add project.id or account.id in order to understand if slow requests and errors are specific to a certain set of accounts, or affecting everyone.

Fine grain logging can be added as span events. Events are a form of structured logging - use them like you would logs. The advantage with span events is that you can automatically find all of the logs associated with a particular transaction, rather than having to go hunting with a bunch of searches and filters. As you scale up, this becomes a lifesaver (or, at least, a big time saver).

First, require the OpenTelemetry API. At the package level, create a tracer and name it after your package:

const opentelemetry = require('@opentelemetry/api');
const express = require('express');

// create a tracer and name it after your package
const tracer = opentelemetry.trace.getTracer('@otel-node-basics/server');

const app = express();

The name of the tracer appears on every span as the instrumentation.name attribute. This is useful for investigating instrumentation issues.

Once you have a tracer, you can use it to access the server span created by the express instrumentation. Calling tracer.getCurrentSpan() will return the span for the current context. Once you have access to the span, you can add attributes and events.

const app = express();

app.get('/hello', (req, res) => {
 // access the span created by express instrumentation
 span = tracer.getCurrentSpan();
  // add an attribute to segment your data by projectID
 span.setAttribute('projectID', '123');
 // log an event and include some structured data.
 span.addEvent('setting timeout', { sleep: 300 });

 setTimeout(()=> {
   span.addEvent(responding after timeout);
   res.status(200).send('Hello World');
 }, 300);
});

app.listen(9000);

You can also chain these methods, which can be a little more concise.

app.get('/hello', (req, res) => {
 tracer.getCurrentSpan()
       .setAttribute('projectID', '123')
       .addEvent('setting timeout', { sleep: 300 });

 setTimeout(()=> {
   tracer.getCurrentSpan().addEvent('sending response');
   res.status(200).send('Hello World');
 }, 300);
});

Run your server and client again, and you will see these new attributes and events show up on the same spans.

Creating your own spans

You can also create your own spans. These spans will automatically become children of the current span and added to the trace.

Span management involves three steps: starting the span, setting it as the current span, and ending the span.

To start a child span, grab the tracer again, and call tracer.startSpan( name ). Name the span after the operation you are measuring. Advice on naming can be found in the tracing specification.

IMPORTANT: make sure to end the span when your operation finishes, or you will have a leak!

After span.end() is called, Spans are queued up to be exported in the next flush. Calls to setAttribute and addEvent become no-ops after span.end() is called.

app.get('/hello', (req, res) => {
  // start a new span named “sleeper”
 const childSpan = tracer.startSpan("sleeper");

 setTimeout(()=> {
   // childSpan works normally when referenced
   childSpan.addEvent('finished sleeping');
   // However, starting a span does not automatically
   // set it to the current span. getCurrentSpan still 
   // returns the parent span.
   tracer.getCurrentSpan();
   res.status(200).send('Hello World');
   // Ending the span is a requirement. It measures the duration 
   // of the operation, and then sends the span to the exporter.
   childSpan.end();
 }, 300);
});

So, the above “works,” except the child span has not been set as the current span. In almost all circumstances, this is critical. You want the rest of your code to be able to access the span without handing it around as a parameter. And unless you set the new span as current, getCurrentSpan will return the parent span, which would be incorrect.

So, after you start a span, create a closure in which the span is active by calling tracer.withSpan(span, cb). Within the callback, the new span will now be active.

app.get('/hello', (req, res) => {
  // start a new span named “sleeper”
 const childSpan = tracer.startSpan("sleeper");

 // use withSpan to create a new context
 tracer.withSpan(childSpan,()=> {
   setTimeout(()=> {
     // getCurrentSpan now correctly returns childSpan
     const span = tracer.getCurrentSpan();
     span.addEvent('sending response');
     res.status(200).send('Hello World');
     span.end();
   }, 300);
 });
});

My advice is to avoid creating child spans, except when you truly require a new context - seprating out a dabatase operations from application code, for example. Ideally, span management should happen in some kind of framework, rather than scattered about your application code. Favor adding events over creating child spans. If you pool all of your attributes on to the same span, you will get better indexing.

Error Handling

There is one final type of event that deserves special attention: exceptions. In OpenTelemetry, exceptions are recorded as events. But, to ensure that the exception is properly formatted, the span.recordException(error) method should be used instead of addEvent.

app.get('/hello', (req, res) => {
 try {
   throw ("ooops");
 } catch (error) {
   // Add the exception as a properly formatted event.
   span.recordException(error);

   // Set the status code to make the exception count 
   // as an error.
   span.setStatus({ code: 
     opentelemetry.CanonicalCode.UNKNOWN });
 }

By default, exceptions do not count as errors. In OpenTelemetry, an error means that the overall operation did not complete. Plenty of exceptions are expected, and a handled exception does not automatically mean the entire operation failed to complete. In other cases, an operation could fail without an exception being thrown.

In order to declare an operation a failure, call span.setStatus() and pass in an error code. Status codes are used by analysis tools to automatically trigger alerting, measure error rates, etc.

Note: status codes will be simplified in the next version of OpenTelemetry.

That’s all, folks!

And that is that. All you need to know to get started with tracing in NodeJS. Hopefully, that was pretty straight forwards, and clears up any mysteries about how to use OpenTelemetry.

If you stick with the above patterns, you can get a great deal of visibility with very little work. Of course, there are many more details and options; you can check out the API documentation for more information. I also have a more involved getting started guide; it works as a handy reference for all of the procedures described above.

If you are writing manual instrumentation during the beta, consider creating helper functions that simplify the API for your use cases, and give you a centralized place to manage any potential breakage.

Also: consider joining our community! There are plenty of libraries left to instrument. You can find us on GitHub, or say hi on gitter.

Understanding Static Site Performance with OpenTelemetry

austin — Wed, 24 Jun 2020 16:38:42 +0000

When you're building a static website, performance can be hard to measure. There are hundreds of variables and confounding conditions that can impact page load speed, everything from mundane considerations like the size of a PNG to the complex vagaries of Content Delivery Networks. How are you supposed to figure out if your users, the people that are trying to load your page, are having a good experience? This becomes trickier when you're using popular managed hosting services, like Netlify, Contentful, or GitHub Pages. You need some sort of telemetry data from the perspective of your end-users in order to accurately gauge how long things take to load.

OpenTelemetry can help you solve this problem! OpenTelemetry is an open source project that promises to make high-quality telemetry a "built-in" feature of cloud-native software. To this end, the project maintains a variety of 'automatic instrumentation' plugins for popular frameworks, runtimes, and libraries that offer you the ability to drop OpenTelemetry into your existing code without major changes in order to profile the performance of your software in production. In simpler terms, OpenTelemetry collects data on the performance of your software or web site from your end-user's perspective, and send that telemetry data to one of many open source or proprietary tools that allow you to persist and analyze it.

In this tutorial, we'll go through the entire process of setting up OpenTelemetry on a static site using Hugo and Webpack, then configuring a deployment of the OpenTelemetry collector in Kubernetes to receive data from our site.

Prerequisites

In order to add OpenTelemetry to a Hugo site, you'll need a Hugo site - tautological, I know, but them's the breaks. If you already have one, then you can use that -- if you're trying to follow along without an existing site, I'd check out the Victor Hugo boilerplate generator, as it'll get you started on the right foot with Hugo and Webpack. You'll also need the following:

Node.JS
NPM

We'll continue this tutorial assuming you're using Victor Hugo, with some notes from my experience instrumenting the opentelemetry.io site. I'll also assume that you're familiar with the basics of using Git and GitHub, and the basics of HTML, CSS, and JavaScript.

If you'd like to follow along with deploying an OpenTelemetry Collector, you'll need a Kubernetes cluster as well - we'll use Google Kubernetes Engine in this tutorial, but the steps should work on any Kubernetes cluster.

Getting Started

First, create a fork (or simply clone) the Victor Hugo boilerplate, and checkout your source code. We'll start by adding OpenTelemetry to our project -

$ npm install --save @opentelemetry/core @opentelemetry/tracing @opentelemetry/web @opentelemetry/plugin-document-load @opentelemetry/exporter-collector

This will install and save several OpenTelemetry components, including the core API and SDK components, automatic browser instrumentation and plugins, and an exporter to the OpenTelemetry Collector. Let's now add a new file to our repository where we'll import OpenTelemetry at /src/tracing.js. You don't need to add anything here for now, but before we forget, let's import it in the main JS file. Open /src/index.js and modify it as so

// JS Goes here - ES6 supported
import "./tracing.js";
import "./css/main.css";

// Say hello
console.log("🦊 Hello! Edit me in src/index.js");

Now that we've got the skeleton of our project set up, it's time to add OpenTelemetry itself.

Adding OpenTelemetry-Web

In the previous step, we created a new file, called tracing.js to hold our OpenTelemetry code. Switch to that in your editor, and you're ready to install and configure OpenTelemetry. First, we'll need to import a few packages that we installed earlier. Add the following --

import { SimpleSpanProcessor, ConsoleSpanExporter } from '@opentelemetry/tracing';
import { WebTracerProvider } from '@opentelemetry/web';
import { DocumentLoad } from '@opentelemetry/plugin-document-load';

Let's briefly touch on what our imports are doing here. First, we're importing a Span Processor which is registered to our Tracer Provider. This component is responsible for handling span data that's generated by the tracer, customarily by exporting it. Our ConsoleSpanExporter will write span data to the browser console for now. Finally, the DocumentLoad plugin extends the capabilities of our tracer provider, allowing it to automatically instrument (read: generate spans for) our page load.

Spans are the building blocks of traces. They represent work being done by your process, or page.

Complete the setup of OpenTelemetry by adding the following code to this file --

const provider = new WebTracerProvider({
  plugins: [
    new DocumentLoad()
  ]
});

provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));
provider.register();

This code will create our provider and plugin, register the span
processor to the provider, and start the provider. Amazingly enough, this is all you need to do for now! In your terminal, start your page preview with npm run preview and open http://localhost:3000 in a web browser. Open your JavaScript console, and refresh the page, you should see output similar to the following.

[HMR] Waiting for update signal from WDS... log.js:24:12
🦊 Hello! Edit me in src/index.js index.js:9:9
Object { traceId: "16b18f5cef76bc6c4fd1578bd0df53d9", parentId: "741587dc317632f9", name: "documentFetch", id: "53ea6e17e3389a01", kind: 0, timestamp: 1592582737016000, duration: 57000, attributes: {…}, status: {…}, events: (8) […] }
ConsoleSpanExporter.js:64:21
Object { traceId: "16b18f5cef76bc6c4fd1578bd0df53d9", parentId: "741587dc317632f9", name: "http://localhost:3000/main.js", id: "ffd85307d05068f5", kind: 0, timestamp: 1592582737140000, duration: 17000, attributes: {…}, status: {…}, events: (8) […] }
ConsoleSpanExporter.js:64:21
Object { traceId: "16b18f5cef76bc6c4fd1578bd0df53d9", parentId: "741587dc317632f9", name: "http://localhost:3000/main.css", id: "278b38cfa637b67c", kind: 0, timestamp: 1592582737140000, duration: 19000, attributes: {…}, status: {…}, events: (8) […] }
ConsoleSpanExporter.js:64:21
Object { traceId: "16b18f5cef76bc6c4fd1578bd0df53d9", parentId: undefined, name: "documentLoad", id: "741587dc317632f9", kind: 0, timestamp: 1592582737016000, duration: 252000, attributes: {…}, status: {…}, events: (9) […] }
ConsoleSpanExporter.js:64:21

Briefly, let's take a look at one of the objects we see here --

{
  "traceId": "16b18f5cef76bc6c4fd1578bd0df53d9",
  "name": "documentLoad",
  "id": "741587dc317632f9",
  "kind": 0,
  "timestamp": 1592582737016000,
  "duration": 252000,
  "attributes": {
    "component": "document-load"
  },
  "status": {
    "code": 0
  },
  "events": [
    {
      "name": "fetchStart",
      "time": [
        1592582737,
        16000105
      ]
    },
    // more events...
  ]
}

This is a JSON representation of a span, which is what OpenTelemetry is creating for you using the DocumentLoad plugin. Spans include more information than you see here, but this is most of the important parts: a name, a trace identifier, a span identifier, timestamp, and duration. We can also see attributes and events -- these are, respectively, properties that help categorize the span, and events that occurred during the span's lifetime.

Let's add some more attributes to our spans in order to make them a bit more useful. Since our ultimate goal is to understand the performance of our page loads, there's two things I can immediately think of that would be useful -- the language of the user's browser, and the path that our users are accessing. We can add both of these properties to our traces by creating some default attributes. In tracing.js, add a new object and modify your provider initialization as follows:

const locale = {
  "browser.language": navigator.language,
  "browser.path": location.pathname
}

const provider = new WebTracerProvider({
  plugins: [
    new DocumentLoad()
  ],
  defaultAttributes: locale
});

Our locale object reads a few values from the browser runtime (namely, the language that the browser is set to, and the current path) and assigns them to our provider as default attributes, which means they'll be applied to all spans created by our tracer. If you refresh your page, you can prove this to yourself by looking at the attribute key in the console output. We'll use these later to get an idea of what pages people are looking at, and roughly where they're from in the world (or at least, we'll be able to use the browser language as a rough proxy for where in the world they are).

Now that we've added OpenTelemetry, we need to actually get the data somewhere other than the browser console. There's a few wrinkles to attend to here as well. First, on modern browsers, OpenTelemetry uses the Beacon API to forward telemetry to a collector service in order to reduce latency for end-users. We also need a place to send that data to. You can either directly export telemetry data to a backend service, or send it to a collector to be aggregated and forwarded.

You can discover the wide range of exporters available to OpenTelemetry at the OpenTelemetry Registry.

There's pros and cons to each of these methods, ones that we won't fully elaborate on due to space considerations, but for the purposes of this tutorial we'll be setting up an OpenTelemetry collector in order to receive our telemetry data. This provides a useful separation of concerns between the generation of telemetry, and the dispensation of that telemetry - for example, if we want to send our telemetry data elsewhere, we can do so by modifying our collector, without having to redeploy our site.

Deploying the OpenTelemetry Collector

The collector itself is a fairly straightforward piece of software with a few moving parts to understand. Generally, though, it allows you to define one or more Receivers, which are endpoints that can receive telemetry data in a specific format. This telemetry data is then sent to another system for analysis and storage through an Exporter. Receivers and Exporters are part of one or more Pipelines, which also allow for the configuration of Processors that can modify the telemetry data in some way. Finally, the collector supports several Extensions which add new features and functionality.

In our case, we don't need anything terribly complicated in terms of collector configuration. We're going to receive data in the OpenTelemetry Format (henceforth referred to as OTLP), and export it to Lightstep using OTLP as well. We'll add some processors to control the amount of memory our collector instances use, and to allow for batching and retrying of exports. One other consideration we must address, however, is TLS (Transport Layer Security). If we deploy our site using HTTPS (and you are using HTTPS in 2020, aren't you?) then our collector also needs to be served over HTTPS. Since we're using Kubernetes, we can take advantage of the Ingress resource to handle this for us -- we'll be using nginx-ingress along with cert-manager to automate the process of creating and provisioning SSL certificates.

Note that the OTLP receiver can handle TLS on its own, but for ease of certificate maintenence, we'll be using our Kubernetes Ingress to terminate TLS and send data to collector instances in plaintext.

I'm going to split this next part into a few discrete steps because there's a lot going on. I'm going to assume that your cluster is basically pristine - here's what mine looked like when I started.

Master version: 1.16.8-gke.15.
3 nodes, type n1-standard-1 with autoscaling.

Set your kubectl context to your Kubernetes cluster before continuing.

Preparing our Kubernetes Cluster

Without getting into a ton of extraneous details, we're going to be using nginx-ingress as our Ingress resource provider in lieu of the GKE Ingress. This is mostly because of how health checks work on GKE Ingress and how the OTLP Receiver in the controller functions (in short, GKE expects the / route to return HTTP 200 OK on GET even if your container readiness probe specifies something else entirely), so we'll start by installing nginx-ingress and cert-manager to our cluster.

Not using GKE? Find more installation instructions for NGINX Ingress here and for cert-manager here.

You'll need to first initialize your user as a cluster administrator by running the following command.

$ kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole cluster-admin \ --user $(gcloud config get-value account)

After this, install nginx-ingress by running this command.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml

You should see a variety of resources being configured and created on your cluster. You can validate that the installation worked by executing $ kubectl get pods -n ingress-nginx, you should see something similar to the following:

NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-9hv54        0/1     Completed   0          22h
ingress-nginx-admission-patch-ddjfp         0/1     Completed   0          22h
ingress-nginx-controller-579fddb54f-zjhq7   1/1     Running     0          22h

Now, lets install cert-manager. Run the following command.

$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.yaml

Again, you'll see a lot of output as resources are created on your cluster. Validate the installation by running $ kubectl get pods -n cert-manager, your result should look something like this:

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-9b8969d86-zrgpg               1/1     Running   0          22h
cert-manager-cainjector-8545fdf87c-pfvxd   1/1     Running   0          22h
cert-manager-webhook-8c5db9fb6-4bdpq       1/1     Running   0          22h

We're now ready to configure our deployment of the OpenTelemetry Collector.

Creating a Collector Configuration

Our first order of business will be to configure the collector itself. We'll store our configuration as a Kubernetes ConfigMap which will be mounted into each pod, and the collector will read this file at startup to configure itself. This makes reconfiguring our collector as simple as updating the ConfigMap, then restarting the pods.

In our case, we expect fairly light load on the collectors, so we're not going to go overboard in resourcing them. Here's the ConfigMap I used, I'll explain some of the more esoteric parts below.

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-collector-conf
  labels:
    app: opentelemetry-collector
    component: web-collector-conf
data:
  web-collector-config: |
    receivers:
      otlp:
        endpoint: "0.0.0.0:55680"
    processors:
      batch:
      memory_limiter:
        ballast_size_mib: 700
        limit_mib: 1500
        spike_limit_mib: 100
        check_interval: 5s
      queued_retry:
    extensions:
      health_check: {}
    exporters:
      otlp:
        endpoint: "ingest.lightstep.com:443"
        headers:
          "lightstep-access-token": <insert access token>
    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch, queued_retry]
          exporters: [otlp]

The configuration file for the collector is also a YAML file, which makes it compose neatly with Kubernetes YAML syntax. Of note in this are really two things - first, the memory_limiter processor and the otlp exporter. I'm going to link to the documentation on the memory limiter but in short, these options assist us in managing the memory usage of the collector process in order to prevent it running out of memory. On the exporter, I've set the endpoint to forward traces to Lightstep, and I'm passing in an access token (you'd find this in your Lightstep project under 'Settings') as a header.

Not already a Lightstep user? Get a free developer account here and follow along!

If we wanted to add another exporter to this pipeline, it'd be very simple -- create a new exporter, and add it to the array of exporters in the pipelines section of our configuration. We could also define a metrics pipeline and send that data to Prometheus or any other desired system. This is, really, one of the advantages of using the collector - you can manage where stuff goes, completely independent of how its generated.

Set up the Kubernetes Deployment and Service

Now that our configuration is settled, it's time to deploy the collector to our Kubernetes cluster and expose it so that it's reachable by our Ingress. At this point, I'd suggest you reference this GitHub repository as a source for the Kubernetes YAML, as I'm going to just point out things you should be aware of -- we're not doing anything too different from a bog-standard deployment. First, let's check out deployment.yaml.

One very important thing to note is the command being passed to the container. The flag --mem-ballast-size-mib must match the ballast_size_mib value defined in the ConfigMap. Other than this, it's a fairly straightforward deployment. The livenessProbe and readinessProbe are accessing port 13133 because that's the default (you enable this by adding the health_check extension in the collector configuration). Finally, take note of the image -- in this case, we're using a development build of the collector, but you may wish to use a stable release or the opentelemetry-collector-contrib container. See this link for information about what's contained in the contrib collector -- it's usually exporters and plugins that aren't in the "main line" collector. On to service.yaml. We're simply mapping port 55680 to port 80 on a ClusterIP, which is how our Ingress will connect to it.

At this point, you're ready to start deploying this to your cluster. Execute $ kubectl apply -f configmap.yaml -f deployment.yaml -f service.yaml, and the cluster will be on its way. After a few moments (slightly more if you've never pulled these containers before), running $ kubectl get pods should display something similar to this:

NAME                                        READY   STATUS    RESTARTS   AGE
web-collector-deployment-79cfc8797c-7vvln   1/1     Running   0          23h
web-collector-deployment-79cfc8797c-vzslm   1/1     Running   0          23h

Creating your Ingress and Certificates

You're halfway there, and if you were only going to be sending telemetry data to your collector from inside this cluster, you'd be ready to roll. However, we want to send data from outside the cluster, so we need to expose this service to the world.

Important Note! You'll need a domain name that you control in order to create a certificate using ACME, so be ready.

First, we're going to deploy our Ingress service in order to determine the external IP address we need to assign to our domain name. We'll actually be deploying this ingress in two steps, so rather than simply applying the existing ingress.yaml, take a look at this version:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: web-collector-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - <YOUR FQDN>
    secretName: web-collector-tls
  rules:
  - host: <YOUR FQDN>
    http:
      paths:
      - path: /
        backend:
          serviceName: web-collector-svc
          servicePort: 80

For <YOUR FQDN>, you would want to use whatever domain name will point to your collector (in my case, I used 'otelwebtelemetry.com', but you could use a subdomain, such as 'collector.mysite.com'). Save this file and apply it using kubectl, and wait several minutes. Run $ kubectl get ingress and you should see something similar to the following:

NAME                    HOSTS                  ADDRESS           PORTS     AGE
web-collector-ingress   otelwebtelemetry.com   104.198.132.223   80, 443   22h

In your DNS management, set your host to the ADDRESS you see in your kubectl output. Note that DNS changes can take some time to propagate around the internet, so you may need to wait up to 30 minutes (or longer) - a good way to see if it's ready is to run $ dig @8.8.8.8 <your domain> and see if the answer section has correctly associated your domain name with the IP address of your ingress controller.

Meanwhile, you should verify that the Ingress controller is functioning properly. The easiest way to do so, for the collector, is to run curl against the OTLP receiver path.

$ curl -kivL -X POST -H 'Host: <YOUR FQDN>' 'http://<YOUR IP>/v1/trace'

This command will provide verbose output, follow redirects, show TLS headers, and not give an error on an insecure certificate as it makes a POST request to the OTLP endpoint. If you get a 200 OK response, everything is working, and we can set up certificate management through Let's Encrypt.

Refer to the le-staging-issuer.yaml and le-prod-issuer.yaml files in the repository. You should start with the staging one, as Let's Encrypt aggressively rate limits connections - once everything works, you'll switch to the production (prod) issuer.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: your@email.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class:  nginx

In both this and the production issuer, make sure to change the email field to one that you control. You can then apply this to your cluster with $ kubectl apply -f le-staging-issuer.yaml. Verify that the issuer was successfully created and registered by running $ kubectl describe clusterissuer letsencrypt-staging and verify that the Type field is set to Ready.

In your ingress.yaml, add two new annotations:

annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    acme.cert-manager.io/http01-ingress-class: "nginx"

Now, run $ kubectl apply -f ingress.yaml once again. After a few moments, run $ kubectl get certificate, and you should see a certificate that's set to the value of secretName in your ingress (in my case, it's web-collector-tls). Run $ kubectl describe certificate <name> and you should see 'Ready' under Type, as well as several events (one of which should say 'Certificate issued successfully').

The last step, then, is to switch from the staging Let's Encrypt issuer to the production one. In your ingress, change the "cert-manager.io/cluster-issuer" annotation value to "letsencrypt-prod" and the secretName so that it won't conflict with the staging secret (you can just add -prod). Deploy the production issuer by running $ kubectl apply -f le-prod-issuer.yaml, and then redeploy your ingress again. You should now have an OpenTelemetry Collector deployed to the public internet! Verify this with $ curl -vL -X POST https://<your domain>/v1/trace, if you see a 200 response code with empty braces as the body, then you're good to go!

Configure OpenTelemetry Web with the Collector Exporter

That was a lot, I know - but we're back to something more straightforward now. Only one more step to go! Back to our tracing.js file, add a new import and configure the Collector Exporter. The following is what your file should look like after we're done:

import { SimpleSpanProcessor } from '@opentelemetry/tracing';
import { WebTracerProvider } from '@opentelemetry/web';
import { DocumentLoad } from '@opentelemetry/plugin-document-load';
import { CollectorExporter } from '@opentelemetry/exporter-collector';

const exporter = new CollectorExporter({
  serviceName: '<your website name>',
  url: 'https://<your domain name>/v1/trace'
})

const locale = {
  "browser.language": navigator.language,
  "browser.path": location.pathname
}

const provider = new WebTracerProvider({
  plugins: [
    new DocumentLoad()
  ],
  defaultAttributes: locale
});

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

Now, if you've configured everything correctly up to this point, you should be able to refresh your page a few times in order to generate data and then open up Lightstep. You should see some data in the Explorer, corresponding to page loads from your site!

From here, you can simply deploy your site to the internet using Netlify, GitHub Pages, or your own personal hosting and start to see exactly how people are using your site in new and interesting ways. Want to know how page load speeds are for users with the Russian language, grouped by the pages they're viewing? Lightstep makes that easy!

Summary

We've been through a lot in this tutorial, so I think it's best to do a quick recap of everything we learned today.

Integrating OpenTelemetry into your static site is as straightforward as adding some packages and configuring a tracing module. You don't have to change any of your existing code, just make sure that you import the tracing module first!
Setting up an OpenTelemetry Collector is a great way to collect trace and metric data from your services, be they front-end or back-end, and can be done through Kubernetes.
Once you're using OpenTelemetry, Lightstep is a great way to get started analyzing your trace data, but by no means are you locked in. You can use OpenTelemetry to export data to a variety of open source and proprietary analysis tools!

Thanks for sticking with me through this, I know it's a lot to take in - but I know that once you try it, you'll find something to love. I firmly believe that OpenTelemetry solves one of the biggest problems that plague people who run and build software for a living, the eternal question, "What the $#@* is it doing?" As someone that's asked that question many times over the years, usually in the middle of the night, I'm really excited to see the progress it's made towards providing easier and better answers to that question.

How-To: Tracing React + Node in under 50 lines of code!

austin — Tue, 23 Jun 2020 16:19:10 +0000

When you're building a React app, it can be difficult to know what's going on behind the scenes in terms of performance. When something's slow, how do you figure out why? There could be a hundred things at fault -- a bug in your reducer, a slow API, problems with the database, a network failure -- and tracking down these problems can be extremely time-consuming and frustrating.

What if there was an easier way to get answers? OpenTelemetry to the rescue! In less than 50 lines of code, you can trace every step in a request - from your React code, down to your database - in production with little additional overhead. In this video, I'll show you how to integrate OpenTelemetry into a React SPA and a Node.JS server built on Express, how to deal with common configuration problems, and how to verify that everything is working.

Next time, we'll look at some deployment options for our application and explore building custom metrics in React. Join me then!

Hugo + OpenTelemetry - How fast is my site? (Video, Pt. 2)

austin — Fri, 19 Jun 2020 13:13:01 +0000

We love Hugo for building blazing-fast static sites, but it can be tricky to understand how well they perform for everybody. There can be hundreds or thousands of variables from an end-user's browser, to their ISP, to the vagaries of routing and content delivery networks around the world. Wouldn't it be nice if there was a simple way to quickly understand the performance of your static site? OpenTelemetry makes that easy! You can see how to add OpenTelemetry in my previous video -

Hugo + OpenTelemetry - How fast is my site? (Video, Pt. 1)

austin ・ Jun 16 ・ 1 min read

#opentelemetry #tutorial #javascript

Once you've integrated OpenTelemetry into your site, how do you actually get the data somewhere that you can analyze and review it? In this video, we'll walk through the configuration and deployment of the OpenTelemetry Collector in a Kubernetes cluster, and expose it to the world via HTTP. I'll show you how to add some interesting attributes to the automatic instrumentation that you can analyze later, and we'll see how you can easily connect your OpenTelemetry Collector to Lightstep in order to visualize the traces being emitted.

We're almost there - the last step is getting ready for production by configuring HTTPS on our Collector. Keep an eye out for the wrap-up post where I'll write up the entire journey, and offer some suggestions on how to add OpenTelemetry to your site!

Links and References

open-telemetry / opentelemetry-js

OpenTelemetry JavaScript API and SDK

open-telemetry / opentelemetry-collector

OpenTelemetry Collector

Questions? Comments? Want to watch the next show live?
Follow me on Twitter for all the latest!
Follow On-Call Me Maybe on Twitch!

Hugo + OpenTelemetry - How fast is my site? (Video, Pt. 1)

austin — Tue, 16 Jun 2020 15:41:07 +0000

OpenTelemetry can help! In this video series, we'll walk through how you can start instrumenting a static web page generated with Hugo (or the static site generator of your choice) using OpenTelemetry's automatic instrumentation, send that telemetry data somewhere, and then use it to analyze the performance of your page and debug in production.

Stay tuned for part two, where we'll get our telemetry out of the browser and into an analysis system like Lightstep in order to visualize the performance of our site.

Links and References

open-telemetry / opentelemetry-js

OpenTelemetry JavaScript API and SDK

open-telemetry / opentelemetry.io

The OpenTelemetry website and documentation

gohugoio / hugo

The world’s fastest framework for building websites.

jgthms / bulma

Modern CSS framework based on Flexbox

Questions? Comments? Want to watch the next show live?
Follow me on Twitter for all the latest!
Follow On-Call Me Maybe on Twitch!

Your System is Deeper Than You Think

austin — Wed, 08 Jan 2020 20:07:56 +0000

Trying to parse my career trajectory from reading my resume is like relying on a pirate’s riddle to find treasure. Instead of “walk four paces from the crooked tree, west of skull rock,” though, it’s “spend eight months dropping frozen chicken tenders into a deep fryer, then move eight hundred miles to write automated QA tests for software.”

But there’s one constant in all the varied jobs I’ve had: they are defined by systems that are largely outside of our control, but that we’re ultimately responsible for.

The person who prepares your food has about as much control over the process that delivered them the ingredients as the on-call developer has — getting paged at 3 a.m. because some tangentially related system on the Internet has broken. It’s the people, though, who are ultimately responsible for the immediate crisis, and who bear the burden of fixing the immediate problem.

Even Simple Systems Are Complex

The thing that these systems all have in common is that they’re largely more complex than it seems from the outside. Similar to an iceberg, where a small precipice breaking through the waves can hide a massive chunk of floating ice below the waterline (at least until climate change gets really going and we’re all living in Gas Town waiting for our turn in Thunderdome Plus), the systems that we’re responsible for in our professional lives are often more massive and cumbersome than they appear at first glance. Why is this? Some of it is easily explained, some more difficult, but the usual answer is quite simply ‘inertia’. This organizational and technical inertia isn’t simply something that affects large technical companies, however — it’s something that touches nearly everyone working with software today.

One form of this hidden depth is, of course, other people’s computers, better known as ‘the cloud’. Cloud services have revolutionized the software industry, giving us access to infinitely scalable hardware resources, durable managed services, and a whole host of convenient ways to accidentally delete your entire cloud stack because someone fat-fingered a terraform apply. I’ve seen that last one happen: a new team member at a prior job accidentally deleted every resource in our AWS account due to a combination of poor internal documentation, flawed pairing practices, and overly permissive IAM roles.

But, even if you’ve built an internal system that’s resilient to human beings, how confident are you that every external service you rely on is also so circumspect? You may have a small application with only a handful of services (or even only one service), but every external API you rely on, be it from a cloud provider, or some other SaaS product, is a potential point of complexity, failure, and hair-rending frustration when it goes down and takes your application with it.

Your Dependencies, Their Dependencies, and (Lest We Forget) What’s Dependent on Them

You probably didn’t write your own HTTP stack, or networking stack, or even string comparison library. While it’s easy (and cheap) to go after left-pad or other, similar stories (such as Docker migrating their Go packages to a new GitHub organization, before gomod), the biggest threat to the performance and security of your application may simply be bugs in your third-party dependencies.

Either through benign logic errors, or malicious intent, every module you import is a potential landmine and a source of complexity that you have little control over.

As open source becomes more integral to the art of software development, the potential impact becomes even more widespread — you may vet all of your dependencies, after all, but are the authors of your dependencies taking that same level of care with their dependencies? This, of course, isn’t simply something you need to consider with direct code dependencies — your CI system, your package and container repositories, your deployment pipeline — these are all uncontrolled sources of entropy and complexity that you need to contend with.

While we often think about our software systems strictly in terms of technical depth and complexity, it’s important to remember the organizational and human systems that underpin them.

I think most people can relate to the feeling of helplessness that comes from fighting endless battles against organizational dynamics that don’t seem to make a lot of sense or have misaligned priorities. Maybe your organization rewards new features, while maintenance work is seen as “less important”. Perhaps you’re trapped in a ‘sales-driven development’ pattern where your work shifts from project to project, relentlessly adding new checkboxes to a system without a lot of concern for the overall scalability or maintainability of the application?

Long-term vendor contracts can tie us to particular pieces of technology, forcing hacks and workarounds to development. There are a million other pieces of organizational detritus that float around our teams, regardless of the size or complexity of the actual software we work on. This hidden depth is possibly the most pernicious, as it’s difficult to understand how you can even begin to tackle it, but it’s still a source of complexity as you build and maintain software.

Burning(Out) Man: We’ve All Been There, Some of Us Are Just More Vocal About It

Unfortunately, we don’t have the luxury of sitting back and saying “eh, we’ll fix it tomorrow” when it comes to addressing these issues. You may have a brilliant team of developers, designers, PMs, and more — but you can’t afford the human costs associated with unreliable software. Posts about burnout litter technical communities across the web and recent studies indicate three in five employees feel burnt out by their job at least once a month.

The stress-induced by trying to debug and analyze failures, especially those that aren’t in services or external systems under your direct control, can contribute to burnout. Teams that are suffering from burnout find themselves in situations that can rapidly escalate out of control, and the consequences can be dire — as failures pile on and multiply, more and more time is spent firefighting rather than dealing with the root cause of failures, which leads to more failure and late-night pages.

The Truth About Deep Systems

We’ve been talking a lot about deep systems recently here at LightStep, and I’d encourage you to read some of that material and think about it in the context I’ve presented here.

In short, deep systems are architectures where there are at least four layers of stacked, independently operated services, including cloud or SaaS dependencies. Perhaps a better way to think about deep systems is not so much an explicit definition, but rather what they “sound like”:

“I’m too busy to update the runbook.”
“Where’s Chris? I’m dealing with a P0, and they’re the only one who knows how to debug this.”
“We have too many dashboards!”

I’ve spoken with a lot of developers who think that their system isn’t ‘deep’ because they’ve only got a handful of services, or because they’re a small team. I’d argue that this isn’t the case at all — as demonstrated above, there’s an awful lot of ways your system can have hidden depth that contributes to stress, burnout, and unreliable software. The solution isn’t to despair, but it’s to embrace observability as a core practice of how you build and run software.

This way, when something breaks, you’ll have the context you need to understand what happened, who is responsible, and how to best resolve the issue — even if the regression or error is deep in the stack or the result of a third-party dependency.

OpenTelemetry 101: What is an Exporter?

austin — Mon, 28 Oct 2019 14:57:26 +0000

OpenTelemetry is an open-source observability framework for generating, capturing, and collecting telemetry data for cloud-native software. Prior posts in this series have covered the definition of observability, as it applies to OpenTelemetry, and a dive into the tracing and metrics API. There’s a third critical component, though, that you’ll want to understand before you get started using OpenTelemetry, and that’s how to actually get data out of it! In this post, we’ll talk about the OpenTelemetry exporter model and the OpenTelemetry Collector, along with several basic deployment strategies.

Note - Some of the information in this post is subject to change as the specification for OpenTelemetry continues to mature.

To understand how OpenTelemetry’s exporter model works, it is useful to generally understand a little bit about how instrumentation is integrated to service code. Generally, you can have instrumentation at three different points: Your service, its library dependencies, and its platform dependencies. Integrating at the service level is fairly straightforward, as you would declare a dependency in your code on the appropriate OpenTelemetry package and deploy it with your code. Library dependencies are similar, except that your libraries would generally only declare a dependency on the OpenTelemetry API. Platform dependencies are a more unusual case. When I say ‘platform dependency’, what I mean are the pieces of software you run to provide services to your service, things like Envoy and Istio. These will deploy their own copy of OpenTelemetry, independent of your actions, but will also generally emit trace context that your service will want to be a part of.

In every case, the trace and metric data that your service or its dependencies emit are of limited use unless you can actually collect that data somewhere for analysis and alerting. The OpenTelemetry component responsible for batching and transporting telemetry data to a backend system is known as an exporter. The exporter interface is implemented by the OpenTelemetry SDKs, and uses a simple plug-in model that allows for telemetry data to be translated to whatever format a backend system requires, and transmit it to that backend system. Exporters can be composed and chained together, allowing for common functionality to be shared (like tagging data before export, or providing a queue to ensure consistent performance) across multiple protocols.

To put this in more concrete terms, let’s compare OpenTelemetry to OpenTracing. In OpenTracing, if you wanted to switch what system you were reporting data to, you’d need to replace the entire tracer component with another -- for example, swapping out the Jaeger client library with the LightStep client library. In OpenTelemetry, you simply need to change the export component, or even just add the new one and export to multiple backend systems simultaneously. This makes it a lot easier to try out new analysis tools or send your telemetry data to different analysis tools in different environments.

While the exporter model is very convenient, there are instances when you don’t have the ability to actually redeploy a service in order to add a new exporter. In some organizations, there’s a disconnect between the people writing the instrumented code and the people running the observability platform, which can impair the velocity of rolling out changes to where data goes. In addition, some teams may prefer to abstract the entire exporter model out from their code, and into a separate service. This is where the OpenTelemetry Collector comes in useful. The collector is a separate process that is designed to be a ‘sink’ for telemetry data emitted by many processes, which can then export that data to backend systems. The Collector has two different deployment strategies -- either running as an agent alongside a service, or as a remote application. You’d generally think about using both: the agent would be deployed with your service and run as a separate process, or in a sidecar. The collector would be deployed separately, as its own application running in a container or virtual machine. Each agent would forward telemetry data to the collector, which could then export it to a variety of backend systems such as LightStep, Jaeger, Prometheus, and more.

Regardless of how you choose to instrument or deploy OpenTelemetry, exporters provide a lot of powerful ways to report telemetry data. You can directly export from your service, you can proxy through the collector, or you can aggregate into standalone collectors -- or even a mix of these! Ultimately, what’s important is that you’re getting that telemetry data into an observability platform that can help you analyze and understand what’s going on in your system.

OpenTelemetry 101: What Are Metrics?

austin — Tue, 15 Oct 2019 00:21:24 +0000

OpenTelemetry is an open-source observability framework for generating, capturing, and collecting telemetry data for cloud-native software. In the previous posts in this series, I discussed what observability is, as it relates to OpenTelemetry, and what tracing is. Today, I’d like to cover the second major area of the OpenTelemetry API: metrics.

OpenTelemetry’s metrics API supports reporting diagnostic measurements from a service using the three basic kinds of instruments. These instruments are commonly known as counters, gauges, and measures. Developers use these objects in order to gain visibility into operational metrics about their service.

Most developers are familiar with metrics in some fashion. It’s extremely common, for instance, to monitor metric values such as process memory utilization or error rate, and to create alerts to indicate when a service is violating a predetermined threshold. In addition to these common measurements, metrics events streaming to these instruments can be applied in other unique ways, including by being aggregated and recorded by tracing or logging systems. With that in mind, let’s look at the instruments available through the OpenTelemetry Metrics API and discuss how they can be used.

The API distinguishes between the metric instruments available through their semantic meaning rather than the eventual type of the value they export. This is somewhat unconventional, and it stems from the design of OpenTelemetry itself -- the separation between the API and the SDK forces the SDK to ultimately determine what should happen with any specific metric event, and could potentially implement a given instrument in an non-obvious or non-standard way. If you’re familiar with existing metrics APIs (such as the Prometheus API), this explains why there’s no method to export a histogram or summary distribution. These are considered to be measures, and the SDK can be configured to export a histogram or a summary from a measure.

From a developer’s point of view, the exact backing instrument is somewhat opaque by design, as the API is designed for portability. Developers use the Add, Set, and Record methods with optional declarations to set restrictions on the specific measure (for example, to allow a counter to support positive and negative values) and the SDK takes care of the rest. That said, which of these should you use for any given scenario?

If you’re trying to record a count of something -- such as the sum of errors over time -- then you should use the Add method, which is supported by a counter. A counter is, by default, monotonic, which means it only expects positive values. This property allows a counter by default to be interpreted as a rate measurement. Rates are a comparison between two quantities -- bytes transferred per second, for example. Non-monotonic counters are also supported, which can be useful for reporting changes in a quantity (like the number of elements in a set as they’re added and removed).

What about circumstances where you need a count, but not a quantity? For example, if you’re measuring something over an arbitrary interval or you don’t care about the rate of change, just the current value? This is when you would use a gauge, by calling Set. You can think about a gauge like a thermostat, or a ratio (such as memory consumption vs. total memory available). As you may expect, the default for these instruments is non-monotonic, but a monotonic option is available as well. Why? To report a change, versus a sum. If you’re trying to report a sum you should use a gauge, but if you’re reporting increments then you should use a counter.

Finally, what should you use when you care about individual measurements? Consider a circumstance where you want to know both the count and sum of a given event, or when you’re interested in grouping events into quantiles? You would use Record for these values, as it expresses a measure. A common application of measures is to create a histogram or summary of values. You could use these to calculate averages, like the average size of a particular response, or the average duration of HTTP requests.

This is getting rather long, so let’s do a quick review. There are three instruments you can use in OpenTelemetry, each defined by the method you call to send a metric event. They are:

Counters, which you Add a value to. These are good for values that you’d like to think of as a rate, or changes in a quantity.
Gauges, which you Set the value of. You can think of these as either a car’s odometer (a monotonic gauge, it never decreases) or a car’s speedometer (a non-monotonic gauge, as it can go up and down.)
Measures, to which you Record a value. These are useful to build histograms or summaries, metric projections that let you calculate averages of many values.

The exact mechanism by which you’ll use each of these instruments is a bit language-specific -- OpenTelemetry, by design, is trying to allow each language SIG to implement the API in a way that is conventional for the language it’s being implemented in. This means the exact details of creating a new metric event may not match the specification precisely, you should consult the documentation for your particular OpenTelemetry implementation for more information. At a high level, however, here’s how it works.

First, you’ll need to create an instrument of the appropriate kind, and give it a descriptive name. Each instrument name needs to be unique inside its process. You can also provide label keys, which are optional key values that are used to optimize the metric export pipeline. You’ll also need to initialize a LabelSet, which is a set of labels (both keys and values) that correspond to attributes being set on your metric events. What does this look like?

// in our init or in middleware...
requestKeys = ["path", "host"] // label keys

// now we initialize instruments for our keys, a Count and a Measure
requestBytes = NewIntCounter("request.bytes", WithKeys(requestKeys), withUnit(unit.Bytes))
requestLatency = NewFloatMeasure("request.latency", WithKeys(requestKeys), withUnit(unit.Second))

// then, define the labels for each key (and add more if required)
labels = meter.DefineLabels({“path”: “/api/getFoo/{id}”, “host”: “host.name”})

Again, the specifics are going to be slightly different for each language, but the basic gist is the same -- early on, you create actual metric instruments by giving them a name, and telling them what keys they’ll be seeing. After that, either explicitly add labels for your instruments (or get a blank set of labels, if appropriate). Once you’ve accomplished this, actually recording metric events is fairly straightforward.

requestBytes.Add(labels, req.bytes)
requestLatency.Record(labels, req.latency)

Another option, for high-performance scenarios, is to utilize a handle. This effectively skips a step, by precomputing the instrument and labels (after all, you can add arbitrary labels to a label set, and calling an instrument through the label set will add that event to each combination of label + instrument), useful if you’re in a performance-critical section of code.

requestBytesHandle = requestBytes.GetHandle(labels)
requestLatencyHandle = requestLatency.GetHandle(labels)

for req in requestBatch {
    ...
    requestBytesHandle.Add(req.bytes)
    requestLatencyHandle.Record(req.latency)
    ...
}

One thing to remember with handles, however, is that you’re responsible for ‘cleaning up’ after you’re done by freeing the handle and deallocating it!

So, to summarize one more time:

Metric events are recorded through instruments.
You need to create a unique instrument for each metric you want to record, and give it a unique name.
You can apply additional metadata to your instrument through labels.
Recording metric events is performed either by calling the appropriate method on the instrument itself, or by getting a precomputed handle and calling the method there.

Hopefully this has given you a better understanding of how the OpenTelemetry metrics API functions. It’s a lot to take in! The real power of OpenTelemetry, however, isn’t just that it provides a tracing and metrics API for you to use. In my next post, I’ll cover the exporter model, and the OpenTelemetry Collector, which are the ways you’ll get data out of OpenTelemetry and into analysis systems such as LightStep.

OpenTelemetry 101: How to Start Contributing

austin — Thu, 10 Oct 2019 17:25:43 +0000

The strength and durability of any open source project is, quite often, measured by its contributor base. Projects with a large and active community – not just users, but also individuals who give back by opening issues, fixing bugs, writing documentation, and so forth – are generally ones that produce higher quality software. If you’ve been reading about OpenTelemetry then you may be asking yourself how you can get involved. It’s a great question, and it’s also a great time to dig in and start making pull requests during Hacktoberfest (you can get a free t-shirt!), so let me tell you the best way to get started.

Join The Discussion

Nearly all communication related to OpenTelemetry is public, and if you want to get started contributing, then it’s easy to join in the discussion. Every SIG (Special Interest Group) has regular meetings, all of which are published on a public calendar (links to which are as follows: gCal, iCal, web). Recordings of past SIG meetings can be found on our YouTube channel. Want to just dip your toe in on the community in general? Everyone is encouraged to join in the monthly meetings, held on the second Wednesday of every month. You can also join the mailing list to keep track of discussions and questions. The mailing list will publish announcements, project updates, and other important news in an easy-to-follow format. If you just want to chat, check out our Gitter!

Honestly, if all you feel like you can contribute is your awareness, then that’s great! Being aware of what’s happening in the project and making your voice heard about decisions like the API and design of the SDK are valuable – we love hearing from people with experience instrumenting their code for telemetry, or running observability systems. Your expertise not only matters, it can be used to help push the state of the art forward.

Join A SIG

If you’re interested in participating more concretely, then you should think about joining a SIG. We organize our community members that want to work on specific aspects of the project into SIGs by language, or area of interest. For example, if you wanted to help implement the OpenTelemetry API specification into code, you could join the Java or .NET SIG. If you want to debate the finer points of the cross-language specification, then the Specification SIG might be more your speed. If you want to support the community by maintaining the website, then there’s a SIG for that too!

There’s nothing special that you need to do to join a SIG – just drop in to a weekly meeting and go from there!

Your First Contribution

If you’re looking to contribute, then the easiest way to jump in is by doing. Find a repository in the OpenTelemetry organization that you’d like to make a commit to, then make a fork of it into your account. Want to know what to work on? Check the issues tab and filter the list by labels such as “up for grabs” or “good first issue”. These will be some places you can start contributing immediately! Be sure to post in the issue and let a maintainer know that you’re taking the issue for yourself so they can update the issue accordingly.

If you want to add new functionality, it’s generally a good idea to create an issue and discuss it with that SIG’s maintainers first, to make sure that your idea is in line with the overall project goals. You can also create issues in order to report bugs, make suggestions, or give feedback to the SIG maintainers.

You don’t have to limit yourself to contributing code, though. The project needs more examples, sample code, documentation, and other resources to make it as easy to use as possible. You can find issues for building documentation in the website repository that are ready-to-go for Hacktoberfest, with more to come. Finally, you can contribute by communicating about OpenTelemetry itself. Tell your friends! Tell your coworkers! We’re extremely excited about our alpha releases that are coming out over the next few weeks, and would love to get more users to look at them and give their impressions.

So, again, if you want to contribute to OpenTelemetry, there’s three things I’d suggest:

Subscribe to the mailing list, join the Gitter, and follow/star the project on GitHub.
Check out the public calendar and attend SIG meetings, and/or the monthly community meeting.
Open an issue, make a PR, write a tweet, or create some samples that demonstrate how to use OpenTelemetry – anything, and everything, is appreciated!

OpenTelemetry 101: What Is Tracing?

austin — Thu, 03 Oct 2019 17:58:08 +0000

OpenTelemetry is an open-source observability framework for generating, capturing, and collecting telemetry data for cloud-native software. In my previous post, I covered what observability means for software, and talked about the three types of telemetry data that comprise observability signals -- tracing, metrics, and logs. Today, I’ll be taking a deeper look at the first of these, tracing.

When we refer to tracing in OpenTelemetry, we’re generally referring to distributed tracing (or distributed request tracing). Traditionally, tracing is a low-level practice used to profile and analyze application code by developers through a combination of specialized debugging tools (such as dtrace on Linux or ETW on Windows) and programming techniques. By contrast, distributed tracing is an application of these techniques to modern, microservice-based architectures.

Microservices introduce significant challenges to tracing a request through an application, thanks to the distributed nature of microservices deployments. Consider a traditional monolithic application: since your code is centralized onto a single host, diagnosing a failure can be as simple as following a stack trace. When your application consists of tens, hundreds, or thousands of services running across many hosts, you can’t rely on a single stack trace -- you need something that represents the entire request as it moves from service to service, component to component. Distributed tracing solves this problem, providing powerful capabilities such as anomaly detection, distributed profiling, workload modeling, and diagnosis of steady-state problems.

Much of the terminology and mental models that we use to describe distributed tracing can trace their origin to systems such as Magpie, X-Trace, and Dapper. Dapper, particularly, has been highly influential to modern distributed tracing, and many of the mental models and terminology that OpenTelemetry uses can trace their origin to there. The goal of these distributed tracing systems was to profile requests that moved across processes and hosts, and generate data about those requests suitable for analysis.

The above diagram represents a sample trace. A trace is a collection of linked spans, which are named and timed operations that represent a unit of work in the request. A span that isn’t the child of any other span is the parent span, or root span, of the trace. The root span, typically, describes the end-to-end latency of the entire trace, with child spans representing sub-operations.

To put this in more concrete terms, let’s consider the request flow of a system that you might encounter in the real world, such as a ride sharing app. When a user requests a ride, multiple actions begin to take place -- information is passed between services in order to authenticate and authorize the user, validate their payment information, locate nearby drivers, and dispatch one of them to pick up the rider. A simplified diagram of this system, and a trace of a request through it, appears in the following figure. As you can see, each operation generates a span to represent the work being done during its execution. These spans have implicit relationships (parent-child) both from the beginning of the entire request at the client, but also from individual services in the trace. Traces are composable in this way, where a valid trace is comprised of valid sub-traces.

Each span in OpenTelemetry encapsulates several pieces of information, such as the name of the operation it represents, a start and end timestamp, events and attributes that occurred during the span, links to other spans, and the status of the operation. In the above diagram, the dashed lines connecting spans represent the context of the trace. The context (or trace context) contains several pieces of information that can be passed between functions inside a process or between processes over an RPC. In addition to the span context, identifiers that represent the parent trace and span, the context can contain other information about the process or request, like custom labels.

One important feature of spans, as mentioned before, is that they encapsulate other information. Much of this information is required -- the operation name, the start and stop timestamps, for example -- but some is optional. OpenTelemetry offers two data types, Attribute and Event which are incredibly valuable as they help to contextualize what happened during the execution measured by a single span. Attributes (known as tags in OpenTracing) are key-value pairs that can be freely added to a span to help in analysis of the trace data. You can think of attributes as data that you’d like to eventually aggregate or use to filter your trace data, like a customer identifier, process hostname, or anything else you can imagine. Events (known as logs in OpenTracing) are time-stamped strings that can be attached to a span, with an optional set of Attributes that further describe it. OpenTelemetry additionally provides a set of semantic conventions of reserved attributes and events for operation or protocol specific information.

Spans in OpenTelemetry are generated by the Tracer, an object that tracks the currently active span and allows you to create (or activate) new spans. Tracers are configured with Propagator objects that support transferring the context across process boundaries. The exact mechanism of creating and registering a tracer is dependent on your implementation and language, but you can generally expect there to be a global Tracer capable of providing a default tracer for your spans, and/or a Tracer provider capable of granting access to the tracer for your component. As spans are created and completed, the tracer dispatches them to the OpenTelemetry SDK’s Exporter, which is responsible for sending your spans to a backend system for analysis.

To recap, let’s summarize:

A span is the basic building block of a trace. A trace is a collection of linked spans.
Spans are objects that represent a unit of work, which is a named operation such as the execution of a microservice or a function call.
A parentless span is known as the root span or parent span of a trace.
Spans contain attributes and events, which describe and contextualize the work being done under a span.
A tracer is used to create and manage spans inside a process, and across process boundaries through propagators.

In my next post in this series, I plan to discuss the OpenTelemetry metrics data source, and how it interacts with the traces. Stay tuned!