HttpClient Observability: Logs, Metrics, and Traces Done Right

Hey observability enthusiasts! 👋

Your HttpClient is making calls, but can you see what's happening? Today we'll add proper logging, metrics, and distributed tracing to your HTTP calls!

The Built-in Logging

.NET's IHttpClientFactory includes logging out of the box. Just configure your log levels:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "System.Net.Http.HttpClient": "Information"
    }
  }
}

This gives you:

info: System.Net.Http.HttpClient.MyClient.LogicalHandler[100]
      Start processing HTTP request GET https://api.example.com/users
info: System.Net.Http.HttpClient.MyClient.ClientHandler[100]
      Sending HTTP request GET https://api.example.com/users
info: System.Net.Http.HttpClient.MyClient.ClientHandler[101]
      Received HTTP response headers after 145.2ms - 200

🔍 Fun Fact: HttpClient emits events at two levels: LogicalHandler (before/after your handlers) and ClientHandler (actual network call). This helps you see what your handlers add!

Custom Logging Handler

Want more control? Build a logging handler:

public class HttpLoggingHandler : DelegatingHandler
{
    private readonly ILogger<HttpLoggingHandler> _logger;

    public HttpLoggingHandler(ILogger<HttpLoggingHandler> logger)
    {
        _logger = logger;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var requestId = Guid.NewGuid().ToString("N")[..8];
        var sw = Stopwatch.StartNew();

        _logger.LogInformation(
            "[{RequestId}] → {Method} {Uri}",
            requestId,
            request.Method,
            request.RequestUri);

        if (request.Content != null && _logger.IsEnabled(LogLevel.Debug))
        {
            var body = await request.Content.ReadAsStringAsync(cancellationToken);
            _logger.LogDebug("[{RequestId}] Request body: {Body}", requestId, body);
        }

        try
        {
            var response = await base.SendAsync(request, cancellationToken);
            sw.Stop();

            _logger.LogInformation(
                "[{RequestId}] ← {StatusCode} in {ElapsedMs}ms",
                requestId,
                (int)response.StatusCode,
                sw.ElapsedMilliseconds);

            return response;
        }
        catch (Exception ex)
        {
            sw.Stop();
            _logger.LogError(
                ex,
                "[{RequestId}] ✗ Failed after {ElapsedMs}ms: {Message}",
                requestId,
                sw.ElapsedMilliseconds,
                ex.Message);
            throw;
        }
    }
}

Metrics with .NET 8

.NET 8 introduced native metrics for HttpClient:

public class HttpMetricsHandler : DelegatingHandler
{
    private readonly IMeterFactory _meterFactory;
    private readonly Histogram<double> _duration;
    private readonly Counter<long> _requests;

    public HttpMetricsHandler(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("MyApp.HttpClient");

        _duration = meter.CreateHistogram<double>(
            "http.client.request.duration",
            unit: "ms",
            description: "HTTP request duration");

        _requests = meter.CreateCounter<long>(
            "http.client.request.count",
            description: "Total HTTP requests");
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var sw = Stopwatch.StartNew();
        var tags = new TagList
        {
            { "http.method", request.Method.Method },
            { "server.address", request.RequestUri?.Host ?? "unknown" }
        };

        try
        {
            var response = await base.SendAsync(request, cancellationToken);

            tags.Add("http.status_code", ((int)response.StatusCode).ToString());
            tags.Add("http.response.status_code", ((int)response.StatusCode).ToString());

            return response;
        }
        catch (TaskCanceledException)
        {
            tags.Add("error.type", "timeout");
            throw;
        }
        catch (HttpRequestException ex)
        {
            tags.Add("error.type", ex.GetType().Name);
            throw;
        }
        finally
        {
            sw.Stop();
            _duration.Record(sw.Elapsed.TotalMilliseconds, tags);
            _requests.Add(1, tags);
        }
    }
}

Export to Prometheus/OpenTelemetry

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation() // Built-in HttpClient metrics!
            .AddMeter("MyApp.HttpClient")   // Our custom metrics
            .AddPrometheusExporter();
    });

Distributed Tracing

Trace requests across services:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation(options =>
            {
                options.RecordException = true;
                options.EnrichWithHttpRequestMessage = (activity, request) =>
                {
                    activity.SetTag("http.request.body_size", 
                        request.Content?.Headers.ContentLength ?? 0);
                };
                options.EnrichWithHttpResponseMessage = (activity, response) =>
                {
                    activity.SetTag("http.response.body_size",
                        response.Content.Headers.ContentLength ?? 0);
                };
            })
            .AddJaegerExporter();
    });

💡 Pro Tip: EventSource for Deep Debugging

.NET's HttpClient emits detailed events via EventSource:

public class HttpEventListener : EventListener
{
    protected override void OnEventSourceCreated(EventSource eventSource)
    {
        if (eventSource.Name == "System.Net.Http" ||
            eventSource.Name == "System.Net.Sockets" ||
            eventSource.Name == "System.Net.Security")
        {
            EnableEvents(eventSource, EventLevel.Verbose);
        }
    }

    protected override void OnEventWritten(EventWrittenEventArgs eventData)
    {
        Console.WriteLine($"[{eventData.EventSource.Name}] {eventData.EventName}");

        if (eventData.Payload != null)
        {
            for (int i = 0; i < eventData.Payload.Count; i++)
            {
                Console.WriteLine($"  {eventData.PayloadNames?[i]}: {eventData.Payload[i]}");
            }
        }
    }
}

// Enable in your app
using var listener = new HttpEventListener();

This shows DNS resolution, socket connections, TLS handshakes — everything!

Health Checks

Monitor your downstream dependencies:

services.AddHealthChecks()
    .AddUrlGroup(
        new Uri("https://api.example.com/health"),
        name: "example-api",
        tags: new[] { "ready" })
    .AddUrlGroup(
        new Uri("https://api.payment.com/ping"),
        name: "payment-api",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "ready" });

Custom Health Check

public class ApiHealthCheck : IHealthCheck
{
    private readonly HttpClient _client;

    public ApiHealthCheck(IHttpClientFactory factory)
    {
        _client = factory.CreateClient("HealthCheck");
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            var sw = Stopwatch.StartNew();
            var response = await _client.GetAsync("/health", cancellationToken);
            sw.Stop();

            var data = new Dictionary<string, object>
            {
                { "responseTime", sw.ElapsedMilliseconds },
                { "statusCode", (int)response.StatusCode }
            };

            if (response.IsSuccessStatusCode)
            {
                return sw.ElapsedMilliseconds > 1000
                    ? HealthCheckResult.Degraded("Slow response", data: data)
                    : HealthCheckResult.Healthy("OK", data);
            }

            return HealthCheckResult.Unhealthy(
                $"Status: {response.StatusCode}", 
                data: data);
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy(ex.Message, ex);
        }
    }
}

Complete Setup

// Program.cs
builder.Services.AddTransient<HttpLoggingHandler>();
builder.Services.AddTransient<HttpMetricsHandler>();

builder.Services.AddHttpClient<IMyApiClient, MyApiClient>()
    .AddHttpMessageHandler<HttpLoggingHandler>()
    .AddHttpMessageHandler<HttpMetricsHandler>();

builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m
        .AddHttpClientInstrumentation()
        .AddMeter("MyApp.HttpClient")
        .AddPrometheusExporter())
    .WithTracing(t => t
        .AddHttpClientInstrumentation()
        .AddJaegerExporter());

builder.Services.AddHealthChecks()
    .AddCheck<ApiHealthCheck>("downstream-api");

What to Monitor

Metric	Why
Request duration	Spot slowdowns early
Error rate	Know when things break
Status code distribution	Understand failure patterns
Request count	Capacity planning
Connection pool usage	Detect exhaustion

Wrapping Up

Observability isn't optional — it's how you know your app is healthy. With .NET's built-in instrumentation plus custom handlers, you can:

• Debug production issues faster
• Track SLOs and SLAs
• Correlate requests across services
• Catch problems before users do

Start with the built-in AddHttpClientInstrumentation() and add custom metrics as needed.

Happy observing! 👁️🚀