DEV Community

AI Engineering Foundry
AI Engineering Foundry

Posted on

How to Debug AI Agent Failures in .NET Using OpenTelemetry and Distributed Tracing

#dotnet #csharp #ai #debugging

Debug AI Agent Failures in .NET Using OpenTelemetry

AI agent observability in .NET: learn how to debug AI agent failures with OpenTelemetry, distributed tracing, custom C# spans, metrics, and LLM monitoring.

ai-agent-observability-dotnet-opentelemetry

AI agent observability in .NET

Introduction

AI agent observability in .NET starts with a simple problem: an agent can return a wrong answer even when every API responds successfully. A database query runs without errors, the LLM request returns HTTP 200, and the vector database returns results. Yet the final response is incomplete, irrelevant, or slow.

Traditional logging can tell you a request failed. But an agent does several things before it answers:

  1. Receive a user request.
  2. Retrieve conversation history.
  3. Generate embeddings and search a vector database.
  4. Call an LLM.
  5. Execute one or more tools.
  6. Generate the final response.

When something goes wrong in that workflow, you need visibility into each operation, its duration, its dependencies, and its errors. That's what OpenTelemetry and distributed tracing provide.

In this tutorial we'll build an observability foundation for an AI-powered ASP.NET Core application using C#, OpenTelemetry, custom spans, metrics, and structured logging.

1. Why Traditional Logging Is Not Enough

A user asks a support agent, "Find the latest order and summarize its delivery status." The agent searches the customer database, retrieves the order, calls a shipping API, sends the data to an LLM, and writes a response. The whole request takes 8 seconds, and a traditional log shows this:

2026-11-03 10:30:00 Information: Request received
2026-11-03 10:30:08 Information: Response generated
Enter fullscreen mode Exit fullscreen mode

It doesn't say where the eight seconds went. Was the database slow? The shipping API? Did the LLM retry? A distributed trace answers that:

AI Agent Request                 8000 ms
│
├── Load Conversation             100 ms
├── Search Customer Database      250 ms
├── Shipping API                 1200 ms
├── LLM Request                  5800 ms
└── Generate Final Response       650 ms
Enter fullscreen mode Exit fullscreen mode

These timings are illustrative, not benchmark results.

2. The Three Signals: Logs, Metrics and Traces

  • Logs record individual events: an LLM request failed, a database connection timed out, RAG retrieval returned no documents.
  • Metrics are numbers over time: request count, average duration, LLM failures, tool execution time.
  • Traces show how one request travels through components. A trace is made of spans, and each span is one operation, such as an HTTP request, a database call, an LLM call, a vector search, or a tool execution.

Together they make AI agent observability in .NET practical. Microsoft documents the approach here: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/observability-with-otel

3. Set Up the Project 

dotnet new webapi -n AIObservabilityDemo
cd AIObservabilityDemo

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
Enter fullscreen mode Exit fullscreen mode

These provide hosting integration, console export, ASP.NET Core instrumentation, and outgoing HttpClient instrumentation. For production, add an OTLP exporter and a telemetry backend, and check package versions first.

Configure tracing and metrics in Program.cs:

using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: "AIObservabilityDemo", serviceVersion: "1.0.0"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("AIObservabilityDemo.Agent")
        .AddConsoleExporter())
    .WithMetrics(metrics => metrics
        .AddMeter("AIObservabilityDemo.Agent")
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter());

builder.Services.AddHttpClient();
builder.Services.AddScoped<AiAgentService>();
builder.Services.AddSingleton<AgentMetrics>();

var app = builder.Build();
app.UseHttpsRedirection();
app.MapControllers();
app.Run();
Enter fullscreen mode Exit fullscreen mode

AddAspNetCoreInstrumentation() captures incoming requests, AddHttpClientInstrumentation() captures outgoing ones, AddSource() registers our custom ActivitySource, and ConfigureResource() names the service. The console exporter is for local development.

4. Add Custom Spans for Agent Operations

Automatic instrumentation sees HTTP requests, but it doesn't know about prompt construction, RAG retrieval, tool selection, or agent execution. Add those with an ActivitySource:

using System.Diagnostics;

public static class AgentTelemetry
{
    public static readonly ActivitySource ActivitySource =
        new("AIObservabilityDemo.Agent");
}
Enter fullscreen mode Exit fullscreen mode

Then wrap each stage in a nested activity:

public class AiAgentService
{
    public async Task<string> ProcessAsync(string question)
    {
        using var activity = AgentTelemetry.ActivitySource.StartActivity("AI Agent Execution");
        activity?.SetTag("agent.type", "CustomerSupport");

        try
        {
            using var retrieval = AgentTelemetry.ActivitySource.StartActivity("RAG Retrieval");
            var documents = await SearchDocumentsAsync(question);
            retrieval?.SetTag("rag.documents_count", documents.Count);

            using var llm = AgentTelemetry.ActivitySource.StartActivity("LLM Request");
            var response = await GenerateResponseAsync(question, documents);

            activity?.SetStatus(ActivityStatusCode.Ok);
            return response;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.AddException(ex);
            throw;
        }
    }

    // Simulated operations, not real vector database or LLM integrations.
    private async Task<List<string>> SearchDocumentsAsync(string question)
    {
        await Task.Delay(200);
        return ["Document one", "Document two"];
    }

    private async Task<string> GenerateResponseAsync(string question, List<string> documents)
    {
        await Task.Delay(500);
        return "This is an example AI response.";
    }
}
Enter fullscreen mode Exit fullscreen mode

The trace now shows the parent agent operation with its child operations, so you can see which stage to investigate. Send a POST /api/agent/ask request from any simple controller that calls ProcessAsync, and read the spans in the console.

5. Trace Tool Calls and LLM Requests

Agents use tools to reach databases, APIs and services, and one slow tool can slow the whole response. A reusable wrapper makes each tool call its own span:

public class AgentToolExecutor
{
    public async Task<T> ExecuteAsync<T>(string toolName, Func<Task<T>> operation)
    {
        using var activity = AgentTelemetry.ActivitySource.StartActivity("AI Agent Tool");
        activity?.SetTag("tool.name", toolName);

        try
        {
            var result = await operation();
            activity?.SetStatus(ActivityStatusCode.Ok);
            return result;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.AddException(ex);
            throw;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Use the same pattern around LLM calls, and record the model name, duration, and any failure. Useful LLM signals include request duration, success and failure counts, retry count, provider status, timeouts, and token usage. For token usage, read the usage data in the provider's response. Don't estimate it from response length. And don't retry a failed LLM call blindly: use bounded retries, timeouts, and provider-specific rate-limit handling.

Security: keep complete prompts, customer records, tokens, API keys and confidential tool arguments out of telemetry. Record only the metadata you need for debugging.

6. Add Custom Metrics

Traces help you investigate one request. Metrics show behavior across many:

using System.Diagnostics.Metrics;

public class AgentMetrics
{
    private readonly Meter _meter = new("AIObservabilityDemo.Agent");
    private readonly Counter<long> _requests;
    private readonly Counter<long> _failures;
    private readonly Histogram<double> _duration;

    public AgentMetrics()
    {
        _requests = _meter.CreateCounter<long>("ai.agent.requests");
        _failures = _meter.CreateCounter<long>("ai.agent.failures");
        _duration = _meter.CreateHistogram<double>("ai.agent.duration", unit: "ms");
    }

    public void RecordRequest() => _requests.Add(1);
    public void RecordFailure() => _failures.Add(1);
    public void RecordDuration(double ms) => _duration.Record(ms);
}
Enter fullscreen mode Exit fullscreen mode

Avoid high-cardinality labels such as user IDs, conversation IDs or raw prompts. They inflate telemetry storage and cost.

7. Use Structured Logging

Use ILogger<T> with message templates, like _logger.LogError(ex, "Agent operation failed: {Operation}", operation), instead of writing to the console. Structured logs are easy to filter by operation name and other safe metadata, and you can correlate them with traces through trace identifiers. Don't log sensitive data just because it might help debugging.

8. Export to Jaeger or Grafana

Console output is fine for development, but production needs centralized storage and visualization. A typical setup:

ASP.NET Core AI Application
          |
     OpenTelemetry
          |
 OpenTelemetry Collector
          |
   Jaeger / Prometheus
          |
       Grafana
Enter fullscreen mode Exit fullscreen mode

Jaeger visualizes traces, Prometheus stores metrics, and Grafana builds dashboards. To export with OTLP:

dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
Enter fullscreen mode Exit fullscreen mode 
.WithTracing(tracing => tracing
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddSource("AIObservabilityDemo.Agent")
    .AddOtlpExporter())
Enter fullscreen mode Exit fullscreen mode

Set the endpoint through configuration, for example OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317. It must match your collector's receiver and protocol, and production should use secured transport. Docs: https://opentelemetry.io/docs/languages/dotnet/exporters/

9. Debug Common Failures With Traces

  • The agent is slow. Likely causes: a slow LLM response, unnecessary tool calls, slow queries, too many retrievals. Compare span durations inside one trace.
  • RAG returns irrelevant results. Likely causes: wrong embedding configuration, poor chunking, missing metadata filters. Record safe retrieval metadata: result count, retrieval duration, filter identifiers.
  • A tool fails. Likely causes: an external API timeout, a database connection failure, invalid arguments, an authorization failure. Check the tool span's status, exception, and dependency telemetry.
  • LLM requests fail intermittently. Likely causes: rate limits, provider availability, timeouts. Watch error rates, duration, provider status, and retries.
  • The answer is wrong but nothing failed. Tracing shows which retrieval and tool operations ran, but it can't judge whether the answer is correct. Combine observability with a separate evaluation process that checks relevance, grounding, and expected behavior.

10. Best Practices for AI Agent Observability in .NET

  • Use meaningful span names like Agent.Execution, RAG.Retrieval, LLM.Completion, and Agent.ToolExecution, not Operation1.
  • Mark failures by capturing exceptions and setting error status on failed spans.
  • Correlate everything through trace context.
  • Keep sensitive data out of telemetry: keys, tokens, personal information, full prompts, confidential results.
  • Configure sampling. Tracing every request gets expensive at scale, so consider tail-based sampling to keep failed traces.
  • Set explicit timeouts and bounded retries on LLM and external API calls.
  • Separate monitoring from evaluation. Observability explains what happened. Evaluation measures whether the result was good.

Conclusion

Debugging agents in production needs visibility into the whole execution workflow. A success or failure log doesn't tell you whether a problem happened in retrieval, model inference, database access, or tool execution.

AI agent observability in .NET combines OpenTelemetry's automatic instrumentation with custom ActivitySource spans, structured logging, and application-specific metrics, so you can see how each operation behaves. Build it into the architecture from the start, not after the first production failure.

More practical AI engineering guides for .NET developers: AI Engineering Foundry

FAQ

What is OpenTelemetry in .NET?
An open-source observability framework that lets .NET applications generate, collect, and export traces, metrics, and logs.

Why does distributed tracing matter for AI agents?
It shows how an agent processes a request across components, so you can find slow operations, failed tool calls, and dependency problems.

Can OpenTelemetry monitor LLM API calls and RAG pipelines?
Yes. Instrument outgoing HTTP calls and add custom spans around embeddings, vector searches, retrieval, and LLM generation to capture duration, errors, and safe metadata.

Does OpenTelemetry evaluate answer quality?
No. It monitors execution. Answer quality needs separate evaluation and testing.

Which tools visualize OpenTelemetry data?
Jaeger for traces, Prometheus for metrics, Grafana for dashboards, and compatible APM platforms.

References

  1. OpenTelemetry .NET documentation: https://opentelemetry.io/docs/languages/dotnet/
  2. Microsoft Learn, .NET Observability with OpenTelemetry: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/observability-with-otel
  3. OpenTelemetry .NET Exporters: https://opentelemetry.io/docs/languages/dotnet/exporters/

Top comments (0)