Meta Description: Can Azure AI Foundry's evaluations, guardrails, and red teaming reach agents running on third-party clouds or on-prem? Here's the real architecture, limits, and what actually works.
Picture the average enterprise AI estate in 2026: a customer-support agent built on LangChain and deployed on AWS Lambda, an internal copilot running on a Kubernetes cluster in a private data center, and a shiny new Foundry-native agent in Azure — all three expected to meet the same safety bar before legal will sign off. If your governance tooling only sees the one running in Azure, you don't have a safety program. You have a safety hobby for a third of your estate.
This is the exact tension pulling teams toward Azure AI Foundry's external agent registration feature, and it's generating a question I hear in almost every architecture review right now: if Foundry can "see" an agent running anywhere, does that mean evaluations, guardrails, and red teaming all just... work, regardless of where the agent lives?
The honest answer is no — not uniformly. And the distinction matters enough that getting it wrong could leave a production agent you think is protected completely exposed. This post walks through exactly which Foundry capabilities extend to third-party and on-prem agents, which don't, and the architecture that explains why.
Table of Contents
- The Quick-Answer Matrix
- Foundrys Three Observability Pillars
- How External Agent Registration Actually Works
- Evaluation on Third-Party and On-Prem Agents: The Real Yes
- Guardrails: Why Content Safety Doesnt Automatically Reach External Agents
- Red Teaming: A Conditional Yes, Gated by Reachability
- The Building Blocks: Wiring It All Together
- A Responsibility Checklist Before You Ship
- Conclusion
The Quick-Answer Matrix
Before the deep dive, here's the shape of the answer in one picture. Azure AI Foundry's governance surface splits cleanly into three buckets when the agent in question lives outside Azure: one that works out of the box, one that works conditionally, and one that requires you to build the connective tissue yourself.
The rest of this post exists to justify that table — because in AI governance, "it depends" without the "on what" is just a shrug.
Foundrys Three Observability Pillars
Microsoft Foundry organizes its AI observability story into three core capabilities that work together but serve distinct purposes:
Evaluation measures the quality, safety, and reliability of AI responses using built-in evaluators — general-purpose metrics like coherence and fluency, RAG-specific metrics like groundedness and relevance, safety metrics like hate/unfairness and violence detection, and agent-specific metrics like tool call accuracy and task completion.
Monitoring is the production layer — real-time dashboards tracking token consumption, latency, error rates, and quality scores, integrated with Azure Monitor Application Insights, with alerting when outputs breach quality thresholds.
Tracing is distributed tracing built on OpenTelemetry standards, capturing the execution flow of LLM calls, tool invocations, and agent decision chains.
Notice what's conspicuously absent from that list: guardrails and red teaming aren't core "observability" primitives in Foundry's own framing. Guardrails (Azure AI Content Safety) are a content moderation and policy enforcement service. Red teaming (the AI Red Teaming Agent) is a proactive adversarial testing service built on Microsoft's open-source PyRIT framework. All three — evaluation, guardrails, red teaming — are related but architecturally distinct, which is exactly why they don't extend to external agents in the same way.
How External Agent Registration Actually Works
This is the feature that makes any of this possible for non-Foundry-hosted agents, so it's worth understanding precisely what it does — and, just as importantly, what it explicitly does not do.
Foundry lets you register an agent that runs on any cloud, on-premises, or other host so you can use Foundry's trace view and evaluation experiences. Critically, Foundry stores only registration metadata for these agents — a name, a description, and an otel_agent_id. It does not host, proxy, or invoke the agent's runtime. Your agent keeps its existing endpoint; no AI Gateway is required.
The mechanism is refreshingly low-tech: your external agent is instrumented with OpenTelemetry, emitting spans tagged with a gen_ai.agent.id attribute to an Application Insights resource connected to your Foundry project. A separate registration call creates the agent record in Foundry. The Foundry portal then matches incoming traces to that registration by agent ID and displays them in the trace view — and once traces are flowing, you can run trace-based evaluations against them directly.
This is worth sitting with for a second: the entire bridge between "your agent" and "Foundry's governance tools" is a telemetry pipe, not a traffic proxy. That single design decision is the root cause of everything that follows — it's why evaluation travels cleanly across cloud boundaries, and why guardrails fundamentally cannot.
Evaluation on Third-Party and On-Prem Agents: The Real Yes
Of the three capabilities, evaluation is the one that genuinely, unambiguously extends to agents running anywhere. Here's why the mechanics hold up.
Once your external agent is instrumented — using the Microsoft OpenTelemetry distro (available for Python, .NET, and JavaScript) or a compliant OpenTelemetry setup of your own — every span it emits during normal operation carries the gen_ai.agent.id attribute. Those spans land in Application Insights regardless of whether the agent is sitting in AWS, GCP, a colocation facility, or under someone's desk, as long as it can make an outbound HTTPS call to the Application Insights ingestion endpoint.
From there, Foundry's trace-based evaluation runs the OpenAI-compatible evals API directly over the captured telemetry. No separate dataset construction is required — Foundry resolves traces by matching (project, agent_id) over a lookback window. You can evaluate individual interactions (inputs, outputs, tool calls, latency) and even multi-turn conversations, provided the agent emits a stable gen_ai.conversation.id.
This means a support bot running entirely outside Azure can still be scored for groundedness, coherence, task completion, and safety-adjacent metrics like hate/unfairness or protected material exposure — using the exact same evaluators Microsoft applies to Foundry-native agents. For organizations with genuinely heterogeneous agent fleets, this is the single most useful capability in the entire external-agent story: one evaluation pane of glass, regardless of hosting location.
The caveat worth flagging: evaluation here is strictly after the fact, run against captured telemetry. It cannot block, modify, or reject an agent's output in real time. That is a guardrail's job, and guardrails are where the story changes.
Guardrails: Why Content Safety Doesnt Automatically Reach External Agents
This is the section that trips people up, because "guardrails" and "evaluation" feel like they should travel together. They don't, and the reason is architectural, not a licensing restriction.
Azure AI Content Safety — the service behind Prompt Shields (jailbreak detection), groundedness detection, protected material scanning, and harm-category filtering — is designed to sit inline, in the request/response path of a model or agent call. It's a synchronous API call that happens before a response reaches a user, or before a prompt reaches a model. That positioning is what lets it actually block, redact, or flag content before damage is done.
External agent registration, by contrast, is explicitly architected around that path. Recall the core design principle: Foundry "doesn't host, proxy, or invoke the runtime." There is no gateway sitting between your on-prem agent and its users that Foundry controls. Your agent's existing endpoint is left completely untouched. That's a deliberate trade-off — it's precisely what makes external registration so lightweight to adopt, since it needs only telemetry emission rather than network re-architecture. But it also means there is no interception point where Content Safety filtering can be injected automatically.
Microsoft's own documentation draws this distinction explicitly: Control Plane custom agents route traffic through an AI Gateway, and that Gateway is the component that enables inline guardrail enforcement and fleet-wide governance. External agents deliberately skip the Gateway to keep integration friction low. The official recommendation for production safety on agents outside that Gateway path is to implement your own safety guardrails — calling the Content Safety API directly inside your agent's code, or using Microsoft's safety system message templates — because Foundry isn't going to insert that protection for you from the outside.
So, concretely: if you want Content Safety-style guardrails protecting a third-party or on-prem agent, you have exactly two real options. Front it with Foundry's Control Plane AI Gateway, which changes your architecture and effectively brings it back under Azure's traffic path, or call the Content Safety REST API yourself, inline, inside your own agent's request handling. There is no third option where registering the agent with Foundry silently grants it inline protection. (If your organization is evaluating which path fits your compliance posture, this is worth a dedicated architecture conversation before go-live.)
Red Teaming: A Conditional Yes, Gated by Reachability
Red teaming sits in the middle of the spectrum, and understanding why requires understanding what red teaming actually does mechanically: it's not a passive telemetry consumer like evaluation, and it's not an inline filter like guardrails. It's an active prober — Foundry's AI Red Teaming Agent, built on Microsoft's open-source PyRIT framework, has to reach out and call your agent or model endpoint with adversarial prompts, then capture and score the responses.
That single fact is the whole story: red teaming requires a door to knock on. If your third-party or on-prem agent exposes an HTTP(S) endpoint that the Foundry red-teaming service can reach, with appropriate authentication, it can be targeted. Architecturally, this is no different from red-teaming any external API — Foundry doesn't need to host the agent, it just needs connectivity.
The nuance comes in at the risk-category level. Model-level and simpler agent risk categories — hateful/unfair content, violent content, self-harm content, protected material, code vulnerability, ungrounded attributes — support both local and cloud red teaming and work against "model and agents" broadly, external or not. Local runs execute PyRIT directly against your endpoint from wherever you invoke it; cloud runs execute from Foundry's managed service.
But the more sophisticated agentic risk categories — prohibited actions, sensitive data leakage, task adherence — are cloud-only, because they require Foundry to observe not just the final output but tool calls and intermediate reasoning, inside a minimally sandboxed environment. These categories work best, and in practice are easiest to run, against agents that are either Foundry-hosted or at minimum expose a fully instrumented, reachable endpoint. A fully air-gapped on-prem agent with no externally callable interface simply cannot be reached by the cloud red-teaming service — your fallback there is running PyRIT locally and directly, outside the Foundry cloud product entirely.
One more detail worth knowing if you're red-teaming anything customer-facing: Foundry redacts harmful or adversarial inputs from the resulting red-teaming reports, and for agentic categories targeting Foundry-hosted agents specifically, runs are transient so harmful data isn't logged or stored. Whether that same transient guarantee applies when the target is a genuinely external, non-Foundry-hosted agent is a detail worth confirming directly with your Microsoft account team before you point adversarial probing at anything handling regulated data (verify this behavior for external targets specifically before running production red-team campaigns).
The Building Blocks: Wiring It All Together
Stepping back, the full picture is best understood as five stacked layers — and which governance capability reaches you depends entirely on which layers you've built.
Layers 1 through 4 are what you need to unlock evaluation and trace visibility — an agent anywhere, instrumented with OpenTelemetry, feeding Application Insights, registered in a Foundry project. That's the whole requirement.
Red teaming needs those same four layers plus a reachable, authenticated endpoint at Layer 1 — and for the deeper agentic risk categories, ideally a Foundry-hosted or sandboxable execution context.
Guardrails are the odd one out: they don't live in this stack at all unless you explicitly insert them — either by adding an AI Gateway layer in front of Layer 1 (bringing traffic under Control Plane governance) or by calling Content Safety directly from within your Layer 1 agent code. No amount of OpenTelemetry instrumentation substitutes for that missing inline hook.
A Responsibility Checklist Before You Ship
If you're taking a third-party or on-prem agent into production and want to lean on Foundry's governance tooling, run through this before go-live:
-
Instrumentation: Is the agent emitting OpenTelemetry spans with a consistent
gen_ai.agent.id, and isAPPLICATIONINSIGHTS_CONNECTION_STRINGcorrectly pointed at the Application Insights resource connected to your Foundry project? -
Registration: Has the agent been registered as an
externalagent in Foundry (via portal or SDK), and does the registeredotel_agent_idmatch what the running agent actually emits? - Evaluation cadence: Have you scheduled recurring trace-based evaluations rather than a one-time check, given this is telemetry-driven and only as good as your traffic sample?
- Reachability for red teaming: Does the agent expose an authenticated endpoint Foundry's red-teaming service can call? Have you confirmed which risk categories (local-only vs. cloud, model-only vs. agentic) actually apply to your use case?
- Guardrails, explicitly: Have you either routed the agent through Foundry's Control Plane AI Gateway, or implemented your own inline calls to Content Safety (Prompt Shields, harm-category filtering) inside the agent's own request path? Don't assume registration alone provides this.
- Compliance ownership: Is someone on your team formally responsible for reviewing what data crosses organizational and geographic boundaries when telemetry and evaluation calls flow between your external host and Azure?
Conclusion
The question "can Azure AI Foundry's evaluations, guardrails, and red teaming reach my third-party or on-prem agents?" doesn't have a single yes-or-no answer — it has three answers, and they diverge for a precise architectural reason: evaluation and red teaming work over a telemetry and reachability model that cares nothing about where your agent lives, while guardrails require an inline enforcement point Foundry deliberately doesn't insert for external agents by default.
That's genuinely good news for anyone running a heterogeneous agent fleet — you can get real observability and adversarial testing coverage across your entire estate without migrating every agent's runtime into Azure. But it also means "I registered it with Foundry" is not the same sentence as "it's protected." If guardrails matter for your use case — and for anything customer-facing or regulated, they almost certainly do — you need to deliberately build that inline layer yourself, via the AI Gateway or a direct Content Safety integration.
Before your next agent goes to production outside Azure, don't just ask whether Foundry can see it. Ask whether it can reach it, and whether anything sits in front of it that can actually say no. Start by instrumenting one external agent with OpenTelemetry this week, register it in Foundry, and run your first trace-based evaluation — that single step will tell you more about your real governance posture than any architecture diagram, including this one.
Sources: Register external agents for observability and evaluation — Microsoft Learn, Observability in Generative AI — Microsoft Foundry, AI Red Teaming Agent — Microsoft Foundry, What is Azure AI Content Safety?
Top comments (0)