Once an agent runs correctly, there is still room to improve its prompts, model choice, and tool descriptions. Microsoft Foundry's Agent Optimizer wraps these experiments in a single evaluation flow: generate candidate configurations, compare them against the current version, and let us decide whether to adopt them.
The video below walks through the full workflow with a Foundry agent (hosted on Bilibili). This post doesn't replay every screen; it covers the details the video leaves out — the ones that decide whether optimization succeeds.
Tell the Three Models Apart Before You Optimize
Three model roles show up in an optimization task, and mixing them up leads to misreading the results:
| Model in the UI | What it does |
|---|---|
| The agent's current model and candidate models | Run the actual business tasks — they are what gets compared |
| Optimization model | Analyzes results and generates improved configurations |
| Evaluation model | Scores answers against the evaluation criteria |
Changing the Evaluation model changes the grader; it does not swap the agent's model. For the current list of supported models, refer to the Agent Optimizer documentation. Also note that Maximum candidates in the wizard's Target step directly drives time and cost: every candidate runs the full dataset once.
Optimization Actually Executes Your Tools
This is the easiest point to miss: optimization is not a static analysis of your configuration. The optimizer runs the baseline and every candidate through the entire dataset, entry by entry, and the agent's external tool calls — APIs, databases, third-party services — execute for real in every evaluation. That works out to roughly (number of candidates + 1) × dataset entries tool executions, and billing, state changes, and rate limiting all scale with it. The documentation's advice: point tools with side effects at test endpoints, or swap in mock implementations while optimizing. (How the Agent Optimizer works)
For the data-compliance agent in the video, this means that if its scanning tools are connected to production data sources, they will be executed repeatedly during optimization.
The Dataset and Criteria Define "Better"
The Optimizer can only aim at the dataset: the dataset supplies the tasks, and the evaluation criteria decide which behavior earns a higher score. If you generate the dataset from traces, check that it is representative and that failure cases are covered. When uploading data, column names must match what the evaluator requires — the wizard currently has no column mapping. (Dataset and evaluation criteria setup)
How to Read the Optimization Results
The best candidate in the video, candidate_3, compared with the baseline:
| Metric | Baseline | candidate_3 |
|---|---|---|
| Evaluation score | 0.542 | 0.607 |
| Average tokens | 944.5 | 1.9K |
| Average latency | 2.0 s | 2.9 s |
The composite score is the average of each evaluator's raw scores rescaled to 0–1. Per the documentation, the grading model scores each task once per criterion and returns binary scores, so small fluctuations should be treated as noise. The official documentation provides this scale for interpreting gains: (Interpreting score improvements)
| Gain | Interpretation |
|---|---|
| Below 0.03 | Noise — not a meaningful improvement |
| 0.03 – 0.10 | Moderate improvement, worth deploying |
| 0.10 – 0.20 | Significant improvement |
| Above 0.20 | Major improvement — usually a sign the baseline itself was weak |
On this scale, the video's 0.065 counts as a "moderate improvement, worth deploying" — provided you accept the increase in tokens and latency. More tokens does not necessarily mean a higher bill, because candidate models may have different unit prices; likewise, the cost estimate on the Review page (Min/Est/Max) is not a cap on spend. (Optimizer costs and tokens)
Check View Changes Before You Promote
The recommended candidate rewrote the system prompt and switched the model from gpt-4o to gpt-6-luna — so the gain cannot be attributed entirely to the model swap. Clicking Promote candidate creates a new version; if the agent is pinned to a specific version, you also have to switch the active version manually before it receives traffic. (Promoting a candidate and version behavior)
The Boundaries of Tool-Description Optimization
The Tool description optimization task in the second half of the video fails with:
AgentCandidate must set at least one tool for mutate_tools to work
That error is the capability boundary at work: this optimization rewrites only the descriptions of function-calling tools and their parameters — it does not modify parameter types, defaults, or required fields — and MCP and Toolbox descriptions are out of scope. Agents that use MCP tools can still optimize prompts and models, though. As a side note, the Optimizer also supports code-defined Hosted Agents through the Foundry Toolkit / azd CLI rather than the portal wizard, with an additional Skill improvement target. (Optimization targets and agent types)
Model Router Is a Different Layer: Runtime Model Selection
Agent Optimizer compares configurations in a single offline evaluation; Model Router selects the underlying model per request at runtime, taking the system message, user input, tool definitions, and conversation history into account. Different requests to the same agent may be handled by different models. (How Model Router works)
The video shows the Router's deployment UI but doesn't cover the trade-offs behind these settings:
- Routing mode: Balanced (the default, balancing quality and cost), Quality, or Cost. A mode expresses a routing preference, not a guarantee — change one variable at a time if you want results you can actually explain. (Routing mode configuration)
- Models subset: "All" in Route to all supported models means the pool of models the Router supports and that meet deployment requirements — not every model already deployed in your project. Selecting a subset restricts which models participate in routing, and newly supported models do not join a subset automatically. You usually don't need to deploy underlying models separately for the Router; Claude is the exception. (Model subset configuration)
- Version and deployment limits: For long-context tasks, plan against the smallest context window in your subset; TPM and content filtering are deployment-level settings, and the values in the video are demo values only; pinning a Router version number does not mean the underlying model pool stays fixed forever. (Deployment settings)
How to validate: log the model field in responses to see which model actually served each request. Keep prompts, tools, and the test set unchanged; validate the Optimizer's recommended fixed model version first, then add the Router as a comparison — only then can you tell whether a change came from configuration or from routing. The video does not show an evaluation comparing the Router against a fixed model, so it cannot support the conclusion that enabling the Router is always cheaper or more accurate. (How to evaluate Model Router fairly)
Takeaways
From the video — one complete optimization loop:
- Agent Optimizer generated and re-evaluated candidates for the data-compliance agent; candidate_3 won 0.542 → 0.607 and was promoted as the official version.
- The Tool-description optimization attempt failed because no tool was set to mutate.
- A Model Router deployment handed model selection to runtime.
Added by this post — the trade-offs the video leaves out:
- The Optimizer compares candidates offline; the Router picks a model per request at runtime — two layers, validated with the same controlled comparison.
- Gains below 0.03 are noise, and 0.065 is a "moderate improvement, worth deploying." But the prompt and the model changed at once (check View Changes first) and tokens and latency rose with them — be careful with attribution and trade-offs.
- The dataset and criteria define "better" — optimization results are only as trustworthy as the evaluation process, so invest there first.
- Optimization executes your tools for real, roughly (candidates + 1) × dataset entries calls — isolate side effects (mocks or test endpoints) before submitting.
- Tool-description optimization covers function-calling tools only (MCP and Toolbox are out); for pinned agents, switch the active version manually after Promote, and keep the status quo when every candidate loses to the baseline.
When the dataset is too small, lacks failure samples, or the scores are unstable, manual iteration and accumulating data from traces pays off better. The optimizer amplifies the value of your evaluation process — if that process isn't trustworthy, the optimizer just produces untrustworthy scores faster.
You can outsource your thinking, but you cannot outsource your understanding.
Top comments (0)