Manoranjan Rajguru

Posted on Sep 22

Observability in Microsoft Foundry: Tracing Agent Runs, Continuous Evaluation, and the OpenTelemetry Data Plane

#azure #ai #opentelemetry #observability

Day 10 of the Microsoft Foundry 100 Days / 100 Blogs series.

You shipped an agent. It calls a model, invokes two tools, retrieves a few documents, and returns an answer. It works in your dev loop. Three weeks later, a support ticket lands on your desk: "the assistant gave a wrong price for SKU-4471." You have no idea which of the six internal steps produced that number, whether the tool returned stale data, whether the model hallucinated over a truncated retrieval result, or whether a retry silently doubled a side effect. You have logs, but logs are flat — they don't tell you which LLM call belongs to which tool result, nested inside which user turn.

This is the problem agent tracing exists to solve, and it's the problem this article is about: how Microsoft Foundry captures, stores, and lets you query the execution anatomy of an agent run — not as a marketing feature, but as a distributed-systems observability pipeline built on OpenTelemetry (OTel) semantic conventions, backed by Azure Monitor Application Insights, and wired into a continuous evaluation loop that can automatically grade production traffic.

Why This Matters

Agents are not stateless request/response functions. A single "agent run" can fan out into a tree of operations: a planning call to the model, a tool call to an MCP server, a retrieval call to a vector index, a second model call to synthesize the tool result, and possibly a handoff to another agent. Each of those steps has its own latency, its own token cost, its own failure mode, and its own opportunity to introduce an error that only becomes visible several hops later at the top of the tree.

Without structured tracing, debugging an agent regresses to grep-ing logs and guessing. With structured tracing, you get:

A causal call graph — you can see that the wrong price came from a tool call that returned a cached response older than your cache TTL, not from the model.
Cost attribution — you can see exactly which span in the tree consumed 80% of your tokens.
Regression detection — when average latency jumps from 2s to 9s after a deployment, you can pinpoint the exact span type (tool call vs. model call vs. retrieval) responsible.
A hook for automated evaluation — because the trace is structured data, you can sample it and run quality/safety evaluators against it continuously, without a human ever opening a trace viewer.

Foundry treats this as a first-class capability area, not an afterthought bolted onto logging. It sits on three pillars: evaluation, monitoring, and tracing — and this article focuses primarily on tracing and its downstream monitoring/evaluation consumers, because that's where the architecturally interesting decisions live.

Core Concepts: Traces, Spans, Attributes
Foundry's Observability Architecture
How Tracing Actually Works at Runtime
Setting Up Tracing: Server-Side vs. Client-Side
Implementation: Instrumenting a Real Agent
Reading a Trace: The Waterfall View
From Traces to Judgments: Continuous Evaluation
The Agent Monitoring Dashboard
Multi-Agent Tracing and the Emerging Semantic Conventions
Production Considerations
Security Considerations
Cost Considerations
Common Mistakes and Pitfalls
Alternatives and Trade-offs
Practical Recommendations
Conclusion
References

1. Core Concepts: Traces, Spans, Attributes

Foundry's tracing model is not a proprietary format — it's built directly on OpenTelemetry, the CNCF standard for distributed tracing, metrics, and logs. If you've instrumented a microservice with OTel before, the mental model transfers almost directly:

Trace: the entire journey of one request through your system — in this case, one agent run (a user turn, or a background task execution). It's uniquely identified by a trace_id.
Span: a single unit of work inside that trace — an LLM call, a tool invocation, a retrieval query. Spans have a start time, an end time, a parent span (for nesting), and a set of key-value attributes.
Attributes: structured metadata attached to a span — model name, token counts, tool name, arguments, HTTP status, error flags. Foundry populates these using the OpenTelemetry GenAI semantic conventions, a community-driven spec (co-developed with contributions from Microsoft and Cisco Outshift for multi-agent scenarios) that standardizes attribute names like gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.tool.name so that tooling built for one GenAI framework can render traces from another.
Trace exporter: the component that ships span data out of the process boundary to a storage/analysis backend. In Foundry, the backend is Azure Monitor Application Insights.

The reason semantic conventions matter architecturally: they decouple the producer of telemetry (your agent code, or Foundry's own hosted runtime) from the consumer (the Foundry portal's trace viewer, Application Insights, or a third-party OTel-compatible tool like Grafana Tempo or Honeycomb). As long as both sides speak the same attribute vocabulary, you can swap the visualization layer without touching instrumentation.

2. Foundry's Observability Architecture

At a high level, the data plane looks like this:

Three things are worth calling out about this architecture:

Application Insights is the single source of truth. Foundry doesn't maintain a separate proprietary trace store — it stores spans as Application Insights dependency/request telemetry, which means anything you already know about querying Log Analytics (KQL) works here too. This is a deliberate trade-off: you inherit Application Insights' retention, cost model, and RBAC — for better (mature tooling, familiar ops model) and worse (you now own an Application Insights bill, and access requires a real IAM setup, not just "Foundry project contributor").
The portal is a read-through view, not a separate database. The Traces tab in the Foundry portal queries the same Application Insights resource your team already has access to. There's no data duplication to reconcile.
Monitoring and evaluation are consumers of the trace stream, not separate instrumentation paths. The Agent Monitoring Dashboard's token/latency/success-rate charts are aggregations over the same span data. Continuous evaluation rules sample from the same event stream (response.completed events) rather than requiring a second instrumentation pass. This is the architectural insight that makes the whole system compose well: instrument once, consume three ways (debug, dashboard, auto-grade).

3. How Tracing Actually Works at Runtime

When tracing is enabled and an agent runs, the sequence is roughly:

Span creation. The Foundry Agent Service runtime (for Prompt Agents and Hosted Agents — Workflow and external agents are still in preview for this feature) opens a root span for the run, then opens child spans for each major operation: the initial model call, each tool invocation, each retrieval query, and any nested sub-agent delegation.
Attribute population. Each span is enriched with GenAI semantic-convention attributes: model deployment name, input/output token counts, tool name and arguments, retrieval query and top-k results, latency, and error status if the operation failed.
Context propagation. Parent-child relationships are preserved via OTel's trace-context propagation, so a tool call made inside an LLM's function-calling turn is correctly nested under that LLM span, which is nested under the run's root span — even if the tool call physically executes in a different process (e.g., a remote MCP server) as long as trace context is forwarded.
Export. Spans are flushed to the configured OTLP exporter, which routes to your project's connected Application Insights resource.
Ingestion delay. There's a short (typically sub-minute) ingestion lag between a run completing and the trace being queryable in Application Insights / Log Analytics — worth knowing so you don't panic when a just-completed run doesn't show up instantly.

Because server-side tracing is enabled by connecting an Application Insights resource to the project — not by changing agent code — this works uniformly for both Prompt Agents (defined via PromptAgentDefinition) and Hosted Agents (custom runtimes deployed behind the Responses/Invocations protocols), which is a meaningfully different design point from "add an SDK decorator to every function," the model most bespoke agent frameworks use.

4. Setting Up Tracing: Server-Side vs. Client-Side

Foundry gives you two complementary instrumentation paths, and the recommended sequence is deliberate:

Server-side traces (start here)

This requires zero code changes. You connect an Application Insights resource to your Foundry project (via the Agents → Traces → Connect flow, or Manage → Project details → Connected resources), and Foundry automatically starts logging traces for any Prompt Agent, Hosted Agent, or workflow running in that project. You get 90 days of out-of-the-box trace history the moment it's wired up.

# There's no CLI step for the connection itself (it's a portal action today),
# but you can verify the Application Insights resource exists and is linked
# via Azure CLI as part of your provisioning pipeline:
az monitor app-insights component show \
  --app my-foundry-project-insights \
  --resource-group rg-foundry-prod \
  --query "{name:name, connectionString:connectionString}" \
  -o table

Client-side traces (add when you need visibility into your own code)

If your application wraps the Foundry SDK with custom orchestration logic — retries, pre/post-processing, business rule branching — you'll want spans for that code too, not just what happens inside the agent runtime. This is standard OpenTelemetry instrumentation layered on top of the Azure SDK's tracing plugin:

pip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry

# main.py — client-side tracing for custom orchestration code around a Foundry agent call
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter

# 1. Wire up an OTel TracerProvider that exports to the same
#    Application Insights resource connected to your Foundry project.
provider = TracerProvider()
exporter = AzureMonitorTraceExporter(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
    # 2. Wrap your own business logic in a span. This nests correctly
    #    alongside the server-side spans Foundry emits for the agent call,
    #    because both use the same OTel trace-context propagation.
    with tracer.start_as_current_span("pricing_lookup_orchestration") as span:
        span.set_attribute("customer.tier", "enterprise")
        span.set_attribute("sku.id", "SKU-4471")

        response = project_client.get_openai_client().responses.create(
            model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
            input="What is the current price for SKU-4471?",
        )

        span.set_attribute("response.id", response.id)
        print(response.output_text)

The important architectural detail: because the Azure SDK's tracing plugin (azure-core-tracing-opentelemetry) and your manual span both register against the same global TracerProvider, the resulting trace has your custom span as a parent (or sibling) of the SDK's auto-generated GenAI spans — you get one coherent tree, not two disconnected traces you have to mentally stitch together.

There's also a Foundry Toolkit for VS Code extension that spins up a local OTLP collector so you can view traces during development without needing an Application Insights resource at all — useful for the inner dev loop before you've provisioned cloud infrastructure.

5. Implementation: Instrumenting a Real Agent

Here's a more complete example — a Prompt Agent with a tool call, instrumented end-to-end, followed by a script that queries the resulting trace back out of Application Insights using KQL.

# create_traced_agent.py
# Production-adjacent pattern: create an agent, run it, and confirm
# the run is traceable. Requires AZURE_AI_PROJECT_ENDPOINT and
# AZURE_AI_MODEL_DEPLOYMENT_NAME to already be connected to an
# Application Insights resource in the Foundry portal.
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import PromptAgentDefinition

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
model = os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
    project_client.get_openai_client() as openai_client,
):
    agent = project_client.agents.create_version(
        agent_name="pricing-assistant",
        definition=PromptAgentDefinition(
            model=model,
            instructions=(
                "You are a pricing assistant. Use the get_price tool "
                "to answer questions about SKU pricing. Never guess a price."
            ),
            tools=[
                {
                    "type": "function",
                    "name": "get_price",
                    "description": "Look up the current price for a SKU.",
                    "parameters": {
                        "type": "object",
                        "properties": {"sku_id": {"type": "string"}},
                        "required": ["sku_id"],
                    },
                }
            ],
        ),
    )

    response = openai_client.responses.create(
        model=agent.name,
        input="What is the current price for SKU-4471?",
        extra_body={"agent": {"name": agent.name, "type": "agent_reference"}},
    )

    # response.id is the correlation key you'll search for in
    # Foundry's Traces tab or in Application Insights.
    print(f"Response ID (search this in Traces tab): {response.id}")

To pull the resulting trace back out programmatically (useful for CI gates that assert "no run in this test suite exceeded 5 seconds of model latency"), query Application Insights with KQL:

// Find all GenAI spans for a given response/run, ordered by start time,
// showing the causal shape of the run.
dependencies
| where customDimensions["gen_ai.response.id"] == "resp_abc123..."
| project timestamp, name, duration, target,
          model = tostring(customDimensions["gen_ai.request.model"]),
          inputTokens = tostring(customDimensions["gen_ai.usage.input_tokens"]),
          outputTokens = tostring(customDimensions["gen_ai.usage.output_tokens"]),
          toolName = tostring(customDimensions["gen_ai.tool.name"])
| order by timestamp asc

// Aggregate latency by span type over the last 24 hours to spot
// which stage of the pipeline is driving a regression.
dependencies
| where timestamp > ago(24h)
| where customDimensions has "gen_ai"
| extend spanKind = tostring(customDimensions["gen_ai.operation.name"])
| summarize p50 = percentile(duration, 50), p95 = percentile(duration, 95), count() by spanKind
| order by p95 desc

6. Reading a Trace: The Waterfall View

Once telemetry lands in Application Insights, the Foundry portal's Traces tab renders it as a waterfall — a horizontal timeline where nested bars represent the parent/child span hierarchy:

This view answers the debugging question directly: in the pricing example from the introduction, you'd see the root span (the full run, ~4.2s), a planning LLM call, a get_price tool-call span with its actual returned arguments and result, and a final synthesis LLM call. If the tool span shows a 0.6s duration but the displayed answer is wrong, you immediately know the bug isn't latency-related — it's either in the tool's data or in how the model interpreted the tool's result. If instead you see an 800ms gap between a tool call finishing and the next span starting, that's a retry or a queuing delay, not a model problem. This is the entire value proposition of structured tracing over flat logs: the shape of the trace itself is diagnostic, before you've read a single attribute value.

You can also pivot from a trace to its Conversation view, which shows the response ID, ordered run steps, and full input/output payloads between user and agent — useful when the question isn't "what was slow" but "what did the model actually see."

7. From Traces to Judgments: Continuous Evaluation

This is where Foundry's observability stack stops being "a nicer log viewer" and becomes an actual quality-control system. Because every agent response is a structured event (response.completed) with an attached trace, Foundry lets you attach evaluators — the same built-in quality/safety/RAG-specific evaluators used in offline evaluation — to a live sampling rule that runs continuously against production traffic.

There are two flavors:

Scheduled evaluation: runs on a fixed recurrence (e.g., daily at 9am) against a batch of recent traces, good for periodic regression checks and dashboards that don't need to be real-time.
Continuous evaluation: samples live traffic as it happens, gated by a max_hourly_runs throttle so you don't accidentally run (and pay for) an evaluator on every single production call.

from azure.ai.projects.models import (
    EvaluationRule,
    ContinuousEvaluationRuleAction,
    EvaluationRuleFilter,
    EvaluationRuleEventType,
)

# 1. Define what "good" means: an evaluator config, here checking for violent content.
data_source_config = {"type": "azure_ai_source", "scenario": "responses"}
testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "violence_detection", "evaluator_name": "builtin.violence"}
]
eval_object = openai_client.evals.create(
    name="Continuous Evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

# 2. Wire that evaluator to a live sampling rule: run it on every
#    response.completed event for this agent, capped at 100 runs/hour
#    to bound evaluation cost.
continuous_eval_rule = project_client.evaluation_rules.create_or_update(
    id="my-continuous-eval-rule",
    evaluation_rule=EvaluationRule(
        display_name="My Continuous Eval Rule",
        description="Runs a safety evaluator on live agent responses",
        action=ContinuousEvaluationRuleAction(eval_id=eval_object.id, max_hourly_runs=100),
        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
        filter=EvaluationRuleFilter(agent_name="pricing-assistant"),
        enabled=True,
    ),
)

The architectural point worth internalizing: the evaluation rule doesn't re-run the agent — it consumes the already-captured trace and response as the input to the evaluator, meaning it adds evaluator inference cost but not agent re-execution cost. This is a materially cheaper design than "shadow-run every production request through an offline eval pipeline," and it's why continuous evaluation is viable at meaningful sample rates in production, not just in staging.

Setting this up requires the project's managed identity to hold the Foundry User role (recently renamed from Azure AI User) on the project — a detail that trips people up because the evaluation rule runs under the project's identity, not the caller's, so RBAC has to be granted ahead of time or the rule silently fails to execute.

8. The Agent Monitoring Dashboard

The Monitor tab in the Foundry portal turns the raw trace stream into the four numbers you actually check daily:

Metric	What a bad number means
Token usage	Verbose prompts/responses; a candidate for prompt or context-window optimization
Latency (p50/p95)	Above ~10s often indicates model throttling, heavy tool calls, or network issues
Run success rate	Below ~95% warrants investigating failed runs — this is your first-line SLO
Evaluation scores	Built-in and custom evaluator scores sampled from continuous evaluation rules

It also surfaces red team scan results (adversarial testing for risks like data leakage or prohibited actions) and lets you configure alerts on latency, token usage, evaluation-score thresholds, or red-team findings — turning what would otherwise be a manual "check the traces tab" habit into an actual paging/notification system. All of this is still marked preview at the time of writing, which matters for anyone deciding whether to build a hard production dependency on the dashboard UI itself versus querying Application Insights directly (the latter is GA and stable; the dashboard is a convenience layer on top).

9. Multi-Agent Tracing and the Emerging Semantic Conventions

Single-agent tracing is a solved problem in most GenAI observability tooling at this point. Multi-agent tracing is not, and it's an area Microsoft is actively investing standards effort into. Foundry, in collaboration with Cisco Outshift, contributes to semantic conventions for multi-agent systems that extend the base OpenTelemetry GenAI agent/framework spans — the goal being a standard way to represent things like "which agent delegated to which sub-agent," "which agent owns a given tool call," and "how did a task hand off across an A2A boundary" as first-class span attributes rather than framework-specific ad hoc fields.

This matters because as you move from single-agent Prompt Agents toward orchestrated multi-agent systems (Sequential/Concurrent/Handoff/GroupChat/Magentic patterns via the Microsoft Agent Framework — see Day 7 of this series on the Workflows-to-Agent-Framework migration), the trace tree gets a lot deeper and a lot wider, and without standardized attribution, you end up with an opaque blob of nested LLM calls with no way to answer "which agent introduced this error," only "which span." Standardized multi-agent semantic conventions are what let a trace viewer render an agent-boundary-aware view (grouping spans by owning agent) instead of a flat operation tree.

10. Production Considerations

Ingestion is asynchronous. Don't build synchronous logic that depends on a trace being queryable immediately after a run completes — build a short polling/backoff window if you need programmatic confirmation (e.g., a CI gate that checks "did this test run get traced").
Traces are retained per your Application Insights/Log Analytics configuration, not a Foundry-specific retention policy — plan your data lifecycle (and cost) accordingly, and don't assume the 90-day portal window is your only retention horizon; you can retain longer (and pay more) or shorter.
Workflow and external-agent tracing are preview. If you've built on the visual Workflow designer (being deprecated — see Day 7) or bring-your-own-hosting external agents, validate tracing coverage explicitly before relying on it for incident response.
Alerts are preview but worth piloting now. Wire latency and evaluation-score alerts into your existing on-call tooling (Action Groups → PagerDuty/Teams/webhook) rather than relying on someone remembering to check the dashboard.
RBAC is a day-one blocker, not a day-30 cleanup task. Log Analytics Reader (and, for protected tables, Privileged Monitoring Data Reader) needs to be granted before anyone on your team can view a trace — bake this into your project provisioning IaC (Bicep/Terraform role assignments) rather than doing it manually per engineer.

11. Security Considerations

Traces capture exactly what makes them useful for debugging — full inputs, outputs, and tool arguments — which is also exactly what makes them a data-exfiltration and compliance risk if mishandled:

Don't let secrets flow into spans. If a tool call takes an API key or a customer's PII as an argument, that value can end up as a span attribute verbatim unless you actively redact it before the call, or configure attribute-level scrubbing.
Treat trace data as production telemetry with the same access controls as your logs. This means it should NOT be broadly readable by every developer with "Contributor" on the resource group — grant Log Analytics Reader deliberately, and audit it.
Prompt injection risk extends to observability. A malicious tool result or retrieved document could contain content designed to look like a legitimate log entry or to poison downstream evaluator judgments if evaluators consume raw trace content without sanitization — worth keeping in mind if you're building custom evaluators that parse trace attributes as trusted input.
Entra-authenticated trace ingestion is available (as opposed to connection-string-based ingestion) for teams that need to avoid distributing a long-lived Application Insights connection string across services — prefer this for anything beyond a quick prototype.

12. Cost Considerations

Tracing cost is not a Foundry line item — it's an Application Insights / Log Analytics ingestion and retention cost, billed per GB ingested and per GB-month retained (verify current rates before budgeting; pricing changes over time — verify this stat before publishing). This has two practical implications:

High-volume agents can generate meaningfully more telemetry volume than you expect, especially if you're capturing full prompt/response payloads on every span for a high-QPS production agent. Consider sampling strategies (trace a percentage of runs at full fidelity, the rest at summary-only) if ingestion cost becomes a concern.
Continuous evaluation has a second, separate cost axis: every sampled run triggers actual evaluator model inference (an LLM-as-judge call, in most built-in evaluators), on top of the trace ingestion cost. The max_hourly_runs throttle on evaluation rules exists specifically to bound this — set it deliberately rather than leaving it at a default that could surprise you on a high-traffic agent.

13. Common Mistakes and Pitfalls

Assuming server-side tracing covers your own code. It only covers what happens inside the Foundry agent runtime. If your application does meaningful work around the agent call — retries, business logic, multi-step orchestration outside the agent boundary — you need client-side instrumentation too, or that logic is invisible in the trace.
Forgetting RBAC and then concluding "tracing is broken." The single most common failure mode reported is "I don't see any traces," and the most common cause is a missing Log Analytics Reader role, not a broken pipeline.
Treating the Agent Monitoring Dashboard as a stable production dependency when it's still marked preview — fine for internal visibility, risky as the sole mechanism for a customer-facing SLA today.
Not redacting sensitive data before it enters a span, then discovering months later that PII has been sitting in Application Insights logs with broad read access.
Over-sampling continuous evaluation on high-traffic agents without setting max_hourly_runs deliberately, leading to surprise evaluator inference costs.
Conflating trace retention with agent memory/conversation persistence. A trace being retained for 90 days in the portal doesn't mean the underlying conversation state is retained that long in the agent's own storage — these are separate systems with separate retention semantics.

14. Alternatives and Trade-offs

If you're not deep in the Foundry ecosystem, or you need a single pane of glass across non-Foundry services too, you have real alternatives, because Foundry's tracing is standard OTel underneath:

Bring your own OTel collector + backend (Grafana Tempo, Honeycomb, Datadog, Jaeger): since Foundry emits standard OTel GenAI semantic-convention spans, you can point the exporter at any OTLP-compatible backend instead of (or in addition to) Application Insights, if your org has already standardized elsewhere. You lose the tight Foundry-portal trace viewer integration and the native continuous-evaluation wiring, which are Application-Insights-specific today.
LangSmith / other framework-native tracing, if you're building on LangChain/LangGraph on top of Foundry-hosted models — Foundry explicitly supports tracing for these frameworks, so you can choose whether the framework's native tracing or Foundry's server-side tracing is your primary lens (or run both, since they're not mutually exclusive).
Rolling your own structured logging with correlation IDs is always possible, but you forfeit the semantic-convention interoperability, the automatic waterfall visualization, and the direct evaluator-rule integration — you'd be re-building a worse version of what tracing already gives you for free once Application Insights is connected.

The trade-off in choosing Foundry's native path is mostly about lock-in vs. leverage: you get tight integration with evaluation and monitoring at the cost of your telemetry backend being Application Insights specifically (rather than a vendor-neutral OTel backend of your choice) for the highest-value features like continuous evaluation.

15. Practical Recommendations

Enable server-side tracing on day one of any new Foundry project, before you write a line of custom orchestration code. It's a portal click, not an engineering task, and it's the highest-leverage debugging tool you'll have.
Add client-side instrumentation only for the orchestration logic that lives outside the agent boundary — don't try to manually re-instrument what the runtime already gives you for free.
Bake RBAC (Log Analytics Reader, Foundry User for eval rules) into your IaC, not into a runbook someone forgets to follow.
Start continuous evaluation with a narrow, cheap evaluator (e.g., a single safety check) and a conservative max_hourly_runs, then expand scope once you've validated cost and signal quality.
Treat trace payloads as sensitive by default. Redact before you regret, not after an audit.
Use the KQL layer, not just the portal UI, for anything you want to gate CI/CD on or alert against — the portal is for humans debugging interactively; KQL queries are for automation.

Conclusion

Observability in Microsoft Foundry isn't a dashboard bolted on top of an agent platform — it's a data-plane decision: emit OpenTelemetry GenAI-convention spans from the runtime, store them in Application Insights, and let three different consumers (a human debugging in the Traces tab, an aggregation layer in the Monitoring dashboard, and an automated evaluator sampling live traffic) read from the same stream. That single-source-of-truth design is what makes it possible to go from "a customer says the agent was wrong" to "here is the exact span, with the exact tool arguments, that produced that answer" — and, increasingly, to catch that class of error automatically before a customer ever notices, via continuous evaluation.

If you're running Foundry agents in anything beyond a demo, connecting Application Insights and enabling server-side tracing is not optional infrastructure — it's the difference between debugging with a flashlight and debugging with a floor plan.

Call to action: If you haven't connected an Application Insights resource to your Foundry project yet, do it before your next deploy — it's a five-minute portal action that will save you hours the first time an agent misbehaves in production. Then come back tomorrow for Day 11 of this series.

References

This is Day 10 of the Microsoft Foundry 100 Days / 100 Blogs series — one deep technical dive into a different corner of the Foundry ecosystem every day. Previous entries covered long-running agent resilience, the Responses vs. Invocations protocols, Autopilot identity, Foundry Local, the Agent Optimizer, MCP toolbox governance, the Workflows-to-Agent-Framework migration, Code Interpreter internals, and Voice Agents.

DEV Community