Getting Started with OpenTelemetry in C# - Part 1

Software applications grow in complexity and monitoring and troubleshooting becomes more critical than developing. OpenTelemetry, an open-source observability framework, provides a common way to collect and export telemetry data, such as traces, metrics, and logs, from your applications.

How We Used to Do It: A Look at the Past

Before OpenTelemetry or other modern observability tools existed, developers had to manually generate correlation IDs for tracking requests across services. This often involved-

• Generating a unique ID at the start of a request
• Adding it to API response headers
• Passing the ID across service calls to track the request flow
• Logging the ID manually in different parts of the system

While this approach worked, it required additional effort and was prone to inconsistencies. OpenTelemetry simplifies this by automatically generating trace IDs, propagating them across services, and exporting them in a structured way, making debugging and monitoring much easier.

What is OpenTelemetry?

Like any other telemetry tool, OpenTelemetry provides a set of APIs, libraries, agents, and instrumentation that enables you to capture distributed traces, metrics, and logs from your application.

All you have to do is gain a comprehensive view of your application's behavior, making it easier to detect issues, optimize performance, and troubleshoot in real time.

Why OpenTelemetry then?

Ans - a unified way for ALL !!
You manage observability data from different sources, without vendor lock-in. You can

-Monitor system health and performance
-Detect bottlenecks and reduce downtime
-Trace the flow of requests across microservices
-Ensure seamless integration with popular backend observability -platforms like Prometheus, Grafana etc

What the heck is vendor lock-in?

Vendor lock-in refers to a situation where a customer becomes dependent on a particular vendor's products or services, making it difficult or expensive to switch to another vendor. This dependency is often created because the customer has integrated the vendor's specific technologies, tools, or services into their infrastructure.

Vendor lock-in can occur when
Proprietary Tools
Data Portability Issues
Custom Integrations

Vendor lock-in can be risky for businesses because it limits flexibility, increases long-term costs, and may make it harder to adopt better or more cost-effective solutions in the future. Open standards and open-source tools, like OpenTelemetry, are often used to avoid this issue, as they provide greater interoperability across different platforms and vendors.

Spans and Activities

Each request is associated with a trace, represented as a tree of System.Diagnostics.Activity instances. The first Activity serves as the root, tracking the overall duration and success/failure of the request. Child activities can be created to monitor specific steps, such as database queries, allowing independent tracking of their duration and outcome.

Activities also store metadata, including:

OperationName (type of work)
Tags (descriptive parameters)
Events (timestamped logs)

NOTE: In distributed tracing, these units of work are commonly called Spans, but .NET historically uses the term Activity instead.

Activity IDs and Parent-Child Relationships

.NET uses Activity objects to track operations in distributed tracing. Parent-child relationships are established through unique IDs. .NET supports two ID formats:

Each trace has a globally unique 16-byte trace ID (Activity.TraceId), and each activity within the trace has an 8-byte span ID (Activity.SpanId). Parent-child relationships are defined using Activity.ParentSpanId, allowing tracking across different processes.

A span is the fundamental unit of a trace, representing a distinct part of a distributed system's workflow. Multiple spans form a trace, which is structured like a tree, showing the start and end times of each span.

In .NET, a span is represented by an Activity. The OpenTelemetry client for .NET leverages the existing Activity class, allowing applications to generate OpenTelemetry-compatible traces using the .NET Runtime itself.

To create a span in .NET, an ActivitySource is first defined:

private static readonly ActivitySource Activity = new(nameof(MyBestSideProjectIdeaEver));

Activity Lifecycle

Each thread has a Current activity that flows with synchronous and asynchronous calls.
Starting a new Activity makes it the current activity until it stops, after which the previous activity is restored.
Activities record StartTimeUtc and calculate Duration upon stopping.
Tracking Across Process Boundaries
To maintain trace continuity across services, parent IDs are transmitted via HTTP headers:

W3C TraceContext format uses standard headers. Hierarchical format uses a custom request-id header. .NET automatically encodes and decodes Activity IDs in HTTP requests, reducing the need for manual implementation.

Activity vs ActivitySource

Activity

• Represents a single unit of work (span) within a trace.
• Tracks operations such as HTTP requests, database calls, or background tasks.
• Captures metadata like start time, duration, tags, events, and parent-child relationships.

Example:

using (var activity = new Activity("ProcessOrder"))
{
    activity.Start();
    // Perform work
    activity.Stop();
}

ActivitySource

• Acts as a factory for creating Activity instances in an organized and efficient manner.
• Enables filtering and sampling before an Activity is created, reducing unnecessary instrumentation overhead.
• Encourages structured tracing by defining named sources for activities.

private static readonly ActivitySource ActivitySource = new("BestEverServiceName");

using (var activity = ActivitySource.StartActivity("ProcessOrder"))
{
    // Work inside this block is recorded in the Activity
}

Attributes

Attributes in OpenTelemetry are key-value pairs that provide additional context to a trace. In .NET, these are referred to as Tags and can be added to an Activity to capture metadata about operations.

activity?.SetTag("user_agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)");
activity?.SetTag("client_ip", "192.168.1.100");
activity?.SetTag("name", "godzilla");

Collecting Traces

Activities must be collected and stored for later analysis. Tools like Application Insights, OpenTelemetry, or third-party APM solutions help with this.
In order to make a system observable, it must be instrumented: That is, code from the system’s components must emit traces, metrics, and logs.

Using OpenTelemetry, you can instrument your code in two primary ways:
Code-based solutions via official APIs and SDKs for most languages
Zero-code solutions

Code-based solutions allow you to get deeper insight and rich telemetry from your application itself. They let you use the OpenTelemetry API to generate telemetry from your application, which acts as an essential complement to the telemetry generated by zero-code solutions.

Zero-code solutions are great for getting started, or when you can’t modify the application, you need to get telemetry out of. They provide rich telemetry from libraries you use and/or the environment your application runs in. Another way to think of it is that they provide information about what’s happening at the edges of your application.

You can use both solutions simultaneously.

For a deeper dive into OpenTelemetry concepts, refer to the official documentation.

Key Takeaways

• OpenTelemetry simplifies observability by providing a standardized way to collect traces, metrics, and logs across distributed applications.
• Vendor lock-in is a major challenge with proprietary observability tools. OpenTelemetry ensures flexibility and interoperability.
• Before OpenTelemetry, manual tracking was tedious—developers had to generate and propagate correlation IDs themselves. OpenTelemetry automates this process.
• Spans (Activities in .NET) are the building blocks of tracing and help track request flows across services.
• ActivitySource in .NET provides a structured way to create traces efficiently while reducing unnecessary instrumentation overhead.
• Attributes (Tags) enrich traces by adding contextual metadata, making debugging more effective.
• Instrumentation can be done via code-based or zero-code solutions, or both for deeper insights into system performance.
• OpenTelemetry integrates seamlessly with popular observability platforms like Prometheus, Grafana, Application Insights, AWS Cloudwatch.

Let's now shift our focus to the practical implementation of OpenTelemetry in a distributed .NET environment. In Part 2, we will look at the code in the simplest way and break it down and understand how to do it.