Prevent .NET Application Insights telemetry loss

If you're using the .NET Application Insights SDK to instrument and monitor your ASP.NET Core or worker service applications, this blog post is for you.
We'll explore some potential causes for data loss in your telemetry, and how to fix them:

1. Losing telemetry data on application shutdown
2. Not capturing application logs
3. Failing to capture end-user information in the ASP.NET Core authentication middleware
4. Not capturing personal identifiable information (for internal web applications only)
5. Capturing too much personal identifiable information (for external web applications only)
6. Data retention not long enough
7. Using the wrong Application Insights SDK NuGet package for your workload

Losing telemetry data on application shutdown

The .NET Application Insights SDK typically sends telemetry data every 30 seconds or whenever the internal buffer is full (500 items by default).
This periodic flush happens in the background and is automatic, so you don't have to do anything.
You can learn how it works by looking at the SDK source code: TelemetryBuffer.cs.

However, when your application shuts down, either due to a graceful shutdown or an unexpected error, the .NET Application Insights SDK might not send the buffered telemetry data.
TelemetryBuffer does try to listen for application shutdown in order to flush the remaining telemetry data, but the IApplicationLifecycle that is supposed to provide the Stopping event listener is null in modern .NET applications (ASP.NET Core 6+, .NET 6+ workers).

This .NET DevBlogs post from 2020 provides the required code to send the buffered telemetry data on shutdown (using TelemetryClient.Flush() and Thread.Sleep()), but it is outdated.

Since April 22, 2021, you can use the new TelemetryClient.FlushAsync() task-based API with a custom IHostedService to ensure that you don't lose any telemetry data:

internal sealed class ApplicationInsightsShutdownFlushService : IHostedService
{
    private readonly TelemetryClient _telemetryClient;

    public ApplicationInsightsShutdownFlushService(TelemetryClient telemetryClient)
    {
        this._telemetryClient = telemetryClient;
    }

    public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        // Flush the remaining telemetry data when application shutdown is requested.
        // Using "CancellationToken.None" ensures that the application doesn't stop until the telemetry data is flushed.
        //
        // If you want to use the "cancellationToken" argument, make sure to configure "HostOptions.ShutdownTimeout" with a sufficiently large duration,
        // and silence the eventual "OperationCanceledException" exception. Otherwise, you will still be at risk of losing telemetry data.
        var successfullyFlushed = await this._telemetryClient.FlushAsync(CancellationToken.None);
        if (!successfullyFlushed)
        {
            // Here you can handle the case where transfer of telemetry data to server has failed with non-retriable HTTP status.
        }
    }
}

Don't forget to register this custom hosted service in your dependency injection services:

services.AddHostedService<ApplicationInsightsShutdownFlushService>();

Not capturing application logs

You've added Application Insights to your dependency injection services, and it's now capturing traces for incoming HTTP requests and external dependencies such as outgoing HTTP requests and database queries:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddApplicationInsightsTelemetry();

// [...]

However, you might not be capturing application logs. To do so, you need to register the Application Insights logging provider:

builder.Logging.AddApplicationInsights();

Once you've added the logging provider, you can configure it through the dedicated Logging:ApplicationInsights configuration section. Here's an example using appsettings.json:

{
  "ApplicationInsights": {
    "ConnectionString": "<YOUR_CONNECTION_STRING>"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    },
    "ApplicationInsights": {
      "LogLevel": {
        // Configure log levels specific to Application Insights to better
        // control the volume of logs being sent
        "Default": "Information"
      }
    },
    "Console": {
      // ...
    }
  }
}

Failing to capture end-user information in the ASP.NET Core authentication middleware

Setting the Context.User.AuthenticatedUserId (documentation) on ITelemetry items is essential for several reasons:

• Identifying and tracking individual user behavior across multiple sessions and devices.
• Pinpointing errors or problems specific to certain users.
• Recognizing differences in behavior between authenticated and anonymous users.
• Maintaining an audit trail of user activity.

Unfortunately, the Application Insights SDK for ASP.NET Core applications doesn't automatically handle this. To remedy this, you must register a custom ITelemetryInitializer that extracts the current authenticated user ID (typically from the current ClaimsPrincipal, but it can come from elsewhere) and assigns it to telemetry items. Here's an example that extracts the authenticated user's name:

internal sealed class PrincipalTelemetryEnrichment : TelemetryInitializerBase
{
    public PrincipalTelemetryEnrichment(IHttpContextAccessor httpContextAccessor)
        : base(httpContextAccessor)
    {
    }

    protected override void OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
    {
        if (!string.IsNullOrEmpty(telemetry.Context.User.AuthenticatedUserId))
        {
            return;
        }

        if (platformContext.User.Identity is { IsAuthenticated: true, Name: { Length: > 0 } username })
        {
            telemetry.Context.User.AuthenticatedUserId = username;
        }
    }
}

You can register this ITelemetryInitializer as a singleton:

services.TryAddEnumerable(ServiceDescriptor.Singleton<ITelemetryInitializer, PrincipalTelemetryEnrichment>());

The issue here is that HttpContext.User, which is a ClaimsPrincipal, is only set at the end of the ASP.NET Core authentication middleware. It's common to implement events like OpenIdConnectEvents.OnTokenValidated or JwtBearerEvents.OnTokenValidated to register custom behavior just before the end of the authentication process. However, if an error occurs at this stage, the associated telemetry will not contain the user ID being authenticated.

To fix this, you can store the ID of the user being authenticated in a location such as:

• The HttpContext.Items property,
• A custom scoped service,
• A custom AsyncLocal-based property.

Then, modify the ITelemetryInitializer to retrieve this stored user ID:

services.AddAuthentication().AddOpenIdConnect(options =>
{
    options.Events.OnTokenValidated = context =>
    {
        // Store the ID of the user being authenticated for it to be read by the telemetry initializer
        if (context.HttpContext.User.Identity is { IsAuthenticated: true, Name: { Length: > 0 } username })
        {
            context.HttpContext.Items["AuthenticatedUserId"] = username;
        }

        // [...] Do something like register the user in a database, or whatever
        return Task.CompletedTask;
    };
});

// Change the telemetry initializer to read that stored authenticated user ID
internal sealed class PrincipalTelemetryEnrichment : TelemetryInitializerBase
{
    public PrincipalTelemetryEnrichment(IHttpContextAccessor httpContextAccessor)
        : base(httpContextAccessor)
    {
    }

    protected override void OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
    {
        if (!string.IsNullOrEmpty(telemetry.Context.User.AuthenticatedUserId))
        {
            return;
        }

        if (platformContext.User.Identity is { IsAuthenticated: true, Name: { Length: > 0 } identityName })
        {
            telemetry.Context.User.AuthenticatedUserId = identityName;
        }
        else if (platformContext.Items.TryGetValue("AuthenticatedUserId", out var usernameObj) && usernameObj is string authUserId)
        {
            // If the authentication process failed, at least we had a chance to retrieve the user ID
            telemetry.Context.User.AuthenticatedUserId = authUserId;
        }
    }
}

Not capturing personal identifiable information (for internal web applications only)

🛑 This section only concerns internal enterprise web applications.

In an ASP.NET Core application, the .NET Application Insights SDK, by default, captures the user IP address. It then performs a geolocation lookup to populate the fields client_City, client_StateOrProvince, and client_CountryOrRegion. The IP address is subsequently discarded, and 0.0.0.0 is written to the client_IP field.

For internal enterprise web applications, discarding the IP address might be seen as a loss of valuable information. Fortunately, you can configure the Azure Application Insights resource to enable both IP collection and storage with the DisableIpMasking property.

Refer to this documentation page to learn how to change this property using infrastructure as code or directly in the Azure Portal.

Capturing too much personal identifiable information (for external web applications only)

On the other hand, if you're building an application for external customers, you should respect their privacy and avoid collecting personal identifiable information (PII). The default city, state or province, and country or region fields populated by Application Insights could be considered as PII.

To disable the collection of these fields, you can register a custom ITelemetryInitializer that removes the collected IP address:

internal sealed class RemoveGeolocationTelemetryInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        // Prevents Application Insights from extracting geolocation
        telemetry.Context.Location.Ip = "0.0.0.0";
    }
}

Data retention not long enough

By default, Application Insights retains your telemetry data for 90 days. However, in some cases, you might need to store the data for longer periods, either for compliance reasons or deeper analysis. Make sure to check your data retention settings:

• Navigate to your Log Analytics workspace.
• In the left menu, click on "Usage and estimated costs".
• Click on on the "Data Retention" button.
• Move the slider to adjust the retention duration between 30 and 730 days, according to your needs.

Using the wrong Application Insights SDK NuGet package for your workload

This final piece of advice is not specifically about the loss of telemetry data. However, not using the appropriate Application Insights SDK NuGet package could result in missing the opportunity to register the right built-in telemetry initializers for your workload.

Essentially, if you're building an ASP.NET Core web application, you'll want to install the Microsoft.ApplicationInsights.AspNetCore NuGet package. It comes with numerous built-in web-specific telemetry initializers.

However, if you're building a console application, a background worker, or any non-web-based application, you should use the Microsoft.ApplicationInsights.WorkerService NuGet package, which doesn't include references to ASP.NET Core.

Comments

[Freddy Hidalgo-Monchez]

Good tips overall. I found myself in a situation where the telemetry data stopped being sent yet the application was still running. I'll keep this post in mind if the issue arises again. Thanks :)

[Xavier Abelaira Rueda]

Since I'm currently migrating one app to. Net7 from 4.8 and should be well monitored I think I'll check most of these tips, very useful!

Thanks!