CloudWatch OTLP metrics exporter using .Net Lambda

We will configure the open telemetry protocol(OTLP) metrics exporter to AWS CloudWatch using AWS Lambda with .Net 8 runtime. We will start from infrastructure using AWS CDK.

Lambda infrastructure

To support lambda for open telemetry we need to configure the next thing. Add a lambda layer for AWS Distro for OpenTelemetry(ADOT); layer versions are in GitHub repository. Also, add collector.yaml path to the environment variables.

const string version = "ver-0-102-1:1";
var lambda = new Function("otlp-lambda", new FunctionProps
{
  ...
  Runtime = Runtime.DOTNET_8,
  Environment = new Dictionary<string, string>
  {
    ["OPENTELEMETRY_COLLECTOR_CONFIG_FILE"] = "/var/task/collector.yaml";
     },
  Layers = [LayerVersion.FromLayerVersionArn(
     this,
     "otel-lambda-layer",
     Fn.Join("", ["arn:aws:lambda:", Aws.REGION, $":901920570463:layer:aws-otel-collector-arm64-{version}"]))
  ]
}

Collector config

The collector file is a configuration to receive, process, and export telemetry data. An important thing is NodeName that we will path later, during OTLP configuration. ADOT collector configuration here.

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  logging:
    verbosity: normal
  awsemf:
    log_group_name: '/aws/lambda/{NodeName}'
service:
  pipelines:
    metrics:
      receivers: [ otlp ]
      exporters: [ awsemf ]

.Net part

First of all, we need to register open telemetry metrics that we want to collect and add exporter. AWS_LAMBDA_FUNCTION_NAME - environment variable that is managed by AWS.

.AddOpenTelemetry()
  .WithMetrics(builder =>
  {
    var lambdaName = Environment.GetEnvironmentVariable("AWS_LAMBDA_FUNCTION_NAME")
      ?? "unknown";
    var resourceBuilder = ResourceBuilder.CreateDefault()
      .AddService("otlp-metrics-sample")
      .AddAttributes([new KeyValuePair<string, object>("NodeName", lambdaName!)]);

    builder
    .SetResourceBuilder(resourceBuilder)
    .AddMeter(api.test.operation);
    .AddOtlpExporter((_, readerOptions) =>
    {
        readerOptions.TemporalityPreference = MetricReaderTemporalityPreference.Delta;
    })
  });

Next, we need to integrate metrics with the DI container .AddMetrics().
The last thing - record our metric

app.MapPost("api/metric", (IMeterFactory meterFactory) =>
{
  var metter = meterFactory.Create("api.test.operation");
  var instrument = metter.CreateCounter<int>("sample_counter");
  instrument.Add(1);

  return Results.Ok();
});

CloudWatch

ADOT collector will create a separate stream in the AWS Lambda log group with the prefix otel-stream- and put all metrics there. After that, we can use those metrics with CloudWatch tools(dashboards, alarms, etc).

"OTelLib": "api.test.operation",
"Version": "1",
"_aws": {
  "CloudWatchMetrics": [
    {
      "Namespace": "otlp-metrics-sample",
      "Dimensions": [
        [ "OTelLib" ]
      ],
      "Metrics": [
        {
          "Name": "sample_counter"
        }
      ]
    }
  ],
  "Timestamp": 1722233676723
},
"sample_counter": 2

Conclusion

1. Open telemetry - language agnostic protocol, and using the built-in exporter we may easily migrate from AWS CloudWatch to other supported exporters.
2. Using .Net abstraction IMeterFactory for OTLP we may easily migrate from AWS Lambda to other computed services(Any Cloud, On-Prem, etc), with no vendor lock.

Help links

• https://aws-otel.github.io/docs/getting-started/lambda/lambda-dotnet#lambda-layer
• https://aws-otel.github.io/docs/getting-started/dotnet-sdk/manual-instr
• https://aws-otel.github.io/docs/getting-started/cloudwatch-metrics
• https://learn.microsoft.com/en-us/dotnet/core/diagnostics/metrics-instrumentation