How to trace and evaluate an AI agent in Microsoft Foundry, from the first trace to production monitoring

Microsoft Foundry splits agent observability into three capabilities, and each one answers a different question. Tracing shows what the agent did during a run, evaluation scores how good its answers were against criteria you choose, and monitoring keeps asking both questions once the agent is in front of users. This post goes through them in the order you'll usually need them, using the official Python samples, and it says which parts Microsoft labels as generally available and which as preview, because that label decides how much weight each piece can carry.

What's generally available and what isn't

Tracing is generally available for prompt agents and hosted agents, while workflow agents and external agents are still in preview. Microsoft's Build announcement from June 3, 2026 says tracing and evaluations reached general availability in the spring, but the reference pages still attach preview labels to individual pieces. On the agent evaluators page, Task Adherence, Task Completion, Intent Resolution, Customer Satisfaction, Output Quality and Tool Use Quality are labeled preview, while the tool evaluators such as Tool Call Accuracy don't carry any label at all. On the monitoring page, agent metrics, recurring evaluations, red team scans and alerts are labeled preview, and the page says Microsoft doesn't recommend its preview items for production workloads. The GitHub Action for evaluations is in preview too. I checked these labels on October 14, 2026, and since they change quickly, it's worth reopening the linked pages before you build on any of them.

Step 1, connect Application Insights and read your first trace

Tracing for prompt agents and hosted agents works on the server side, so you don't need to change anything in your code. With the New Foundry toggle turned on, open your project, go to Agents, select Traces and choose Connect, then attach an existing Application Insights resource or create a new one. From that point the runs of an agent hosted in Foundry are recorded through OpenTelemetry, including the model calls, the tool invocations, the decisions the agent took and its dependencies on other services, as described in the observability overview.

To read the traces you'll need the Log Analytics Reader role on the connected Application Insights resource, plus Privileged Monitoring Data Reader when the underlying Log Analytics tables are protected. After you run the agent once, the Traces tab lets you search, filter and sort the traces from the last 90 days, open one to step through its spans, or search by Response ID or Trace ID and open a Conversation ID to see the history, token counts, run steps, tool calls, inputs and outputs. If nothing shows up, the setup page recommends waiting a few minutes and refreshing before you head to its troubleshooting section. That page also covers client side tracing with azure-core-tracing-opentelemetry, which is optional and meant for the code of your own application.

Step 2, evaluate the agent with a small dataset

A trace tells you what happened but not whether the answer was good, and that's the job of evaluation. The sample below follows the official page on evaluating your AI agents, with the rubric generation step removed so it only uses evaluators that ship with Foundry. You'll need the Foundry User role on the project, which Microsoft renamed from Azure AI User without changing the role IDs, and a chat completion model deployment that acts as the judge for the evaluators that use a model.

That page asks for azure-ai-projects 2.4.0 or later, while other pages in the same documentation list lower minimums, so it's safer to install the latest release.

pip install "azure-ai-projects>=2.4.0" azure-identity

Set the environment variables AZURE_AI_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME, then create a JSONL file called test-queries.jsonl where each line holds a query field. These three lines are only example questions, so swap them for the questions your agent gets.

{"query": "What is the difference between an Azure region and an availability zone?"}
{"query": "When should I use Azure Functions instead of a virtual machine?"}
{"query": "How do I stop a storage account from being publicly accessible?"}

The first part of the script creates the clients and uploads the dataset to your project.

import os
import time

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
model_deployment = os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"]
AGENT_NAME = "my-agent"  # replace with the name of your agent

project_client = AIProjectClient(endpoint=endpoint, credential=DefaultAzureCredential())
client = project_client.get_openai_client()

dataset = project_client.datasets.upload_file(
    name="agent-test-queries",
    version="1",
    file_path="./test-queries.jsonl",
)

The second part defines what to measure and creates the evaluation. Coherence is scored by the judge deployment, while Violence comes from the risk and safety group, which the official sample runs without a judge deployment.

from azure.ai.projects.models import TestingCriterionAzureAIEvaluator
from openai.types.eval_create_params import DataSourceConfigCustom

testing_criteria = [
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="Coherence",
        evaluator_name="builtin.coherence",
        initialization_parameters={"deployment_name": model_deployment},
        data_mapping={
            "query": "{{item.query}}",
            "response": "{{sample.output_text}}",
        },
    ),
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="Violence",
        evaluator_name="builtin.violence",
        data_mapping={
            "query": "{{item.query}}",
            "response": "{{sample.output_text}}",
        },
    ),
]

data_source_config = DataSourceConfigCustom(
    type="custom",
    item_schema={
        "type": "object",
        "properties": {"query": {"type": "string"}},
        "required": ["query"],
    },
    include_sample_schema=True,
)

evaluation = client.evals.create(
    name="Agent Quality Evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

The third part starts the run. With the azure_ai_target_completions source, Foundry sends every query to the agent named in the target and evaluates the answers it gets back, which is why the mappings above read from sample.output_text.

eval_run = client.evals.runs.create(
    eval_id=evaluation.id,
    name="Agent Evaluation Run",
    data_source={
        "type": "azure_ai_target_completions",
        "source": {"type": "file_id", "id": dataset.id},
        "input_messages": {
            "type": "template",
            "template": [
                {
                    "type": "message",
                    "role": "user",
                    "content": {"type": "input_text", "text": "{{item.query}}"},
                }
            ],
        },
        "target": {"type": "azure_ai_agent", "name": AGENT_NAME},
    },
)

while True:
    run = client.evals.runs.retrieve(run_id=eval_run.id, eval_id=evaluation.id)
    if run.status in ["completed", "failed"]:
        break
    time.sleep(5)

print(f"Status: {run.status}")
print(f"Report URL: {run.report_url}")

In the response, result_counts gives the total, passed, failed and errored rows, per_testing_criteria_results breaks the same numbers down by evaluator, and each row carries a score, a label, a reason, a threshold and a passed flag for every evaluator. The report_url opens the same results in the Foundry portal, where they also show up under the Evaluations tab, and the reason field is where I'd start reading when a row fails.

If your agent calls tools, the tool use evaluators are the ones worth adding next. Tool Call Accuracy checks that the agent made the right calls with correct parameters and without redundancy, Tool Selection checks that it chose the tools it needed and no extra ones, Tool Input Accuracy checks the parameters, Tool Output Utilization checks whether the agent used the results correctly, and Tool Call Success checks that the calls ended without technical errors. Four of them take tool_definitions as an input, and Tool Call Accuracy scores on a scale of one to five before a threshold turns that score into a pass or a fail.

What it costs and where it's restricted

Risk and safety evaluations, which include the Violence evaluator used above, and agents playground evaluations are billed on consumption, and playground evaluations are enabled by default for every Foundry project. To turn them off, clear all the evaluators in the playground's metrics selector. Some features also have regional restrictions, namely rubric generation, synthetic and trace based dataset creation, and the risk and safety evaluators, so check the supported regions before you run the sample in a project you already have. The monitoring data itself lives in the connected Application Insights resource, so its retention and billing follow however you've configured that resource.

Step 3, keep evaluating after release

The monitoring page describes continuous evaluation, which runs an evaluation on sampled traffic while the agent serves users. The code below creates an evaluation with an azure_ai_source and attaches a rule that runs it whenever a response from the named agent completes. The project managed identity needs the Foundry User role for continuous evaluation rules to run, and the rule is capped by max_hourly_runs, which the sample sets to 100 and which Microsoft documents as the default limit per hour. Once the cap is reached, further runs are skipped until the next hour or until you raise the value.

import os

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
    EvaluationRule,
    ContinuousEvaluationRuleAction,
    EvaluationRuleFilter,
    EvaluationRuleEventType,
)

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
agent_name = os.environ["AZURE_AI_AGENT_NAME"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
    project_client.get_openai_client() as openai_client,
):
    eval_object = openai_client.evals.create(
        name="Continuous Evaluation",
        data_source_config={"type": "azure_ai_source", "scenario": "responses"},
        testing_criteria=[
            {
                "type": "azure_ai_evaluator",
                "name": "violence_detection",
                "evaluator_name": "builtin.violence",
            }
        ],
    )

    rule = project_client.evaluation_rules.create_or_update(
        id="my-continuous-eval-rule",
        evaluation_rule=EvaluationRule(
            display_name="My Continuous Eval Rule",
            description="Runs on agent response completions",
            action=ContinuousEvaluationRuleAction(
                eval_id=eval_object.id, max_hourly_runs=100
            ),
            event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
            filter=EvaluationRuleFilter(agent_name=agent_name),
            enabled=True,
        ),
    )
    print(f"Rule created (id: {rule.id})")

To read the results, open the agent in the Foundry portal and select Monitor. The dashboard shows token usage, latency, run success rate, evaluation metrics and red teaming results for the time range you pick, and the documentation gives two starting points for interpreting it. Latency above ten seconds may point to model throttling, complex tool calls or network issues, and a run success rate below 95 percent warrants an investigation of the failed runs.

Running the same evaluation on every commit

You can also run the evaluation from Step 2 in a pipeline through the microsoft/ai-agent-evals@v3-beta GitHub Action, which the documentation labels as preview. The step below goes after the checkout and an Azure login step that uses OpenID Connect, so the workflow also needs the id-token: write permission and the client, tenant and subscription IDs stored as GitHub Actions variables.

      - name: Run Evaluation
        uses: microsoft/ai-agent-evals@v3-beta
        with:
          azure-ai-project-endpoint: "<your-ai-project-endpoint>"
          deployment-name: "<your-deployment-name>"
          agent-ids: "<your-ai-agent-ids>"
          data-path: ${{ github.workspace }}/path/to/your/data-file

The agent-ids input takes values such as my-agent:1,my-agent:2, an optional baseline-agent-id sets the version used as the comparison, and the results show up in the workflow run summary with the score of each metric, confidence intervals and, when you compare several agents, a pairwise statistical comparison that tells you whether a difference is meaningful or just random variation. That makes it a reasonable quality gate for changes to an agent's instructions or model, since the same dataset runs against both versions.

If you'd rather see the three pieces working together before you try them, the Azure Friday episode Observe and evaluate agents for quality and safety in Microsoft Foundry walks through evaluation, tracing and monitoring with a live demo in the Foundry portal and in Visual Studio Code.

Which of those three questions is the hardest to answer about your agents today, what they did, how good their answers were, or whether they stay that good after release?