DEV Community: Lukas Walter

Trust Boundaries Around AI Features

Lukas Walter — Fri, 14 Aug 2026 15:30:00 +0000

If an action requires approval, a prompt is never the control that enforces it.

A prompt can tell the model to ask, "Should I close this ticket?" That helps the conversation. It does not guarantee that the model asks, that the user sees the exact operation, or that the arguments stay unchanged after confirmation. A deterministic execution boundary must verify mandatory approval before execution.

I use the same rule for other AI output: the model may propose. The application decides whether that proposal becomes behavior.

A trust boundary is where authority, security domain, data exposure, or an established trust property changes. Some boundaries protect an action. Others protect data as it moves to a model provider, log sink, browser, plugin, or another tenant.

Trust depends on the property you need

"Trusted" and "untrusted" are too blunt on their own.

An authenticated user has a verified identity, but their text is still untrusted input. An internal document may have known provenance while being stale or outside the current user's permissions. Model output can match a JSON schema and still request an action the caller cannot perform.

For each value that crosses a boundary, write down which properties have been established:

Identity and delegation: who initiated, approved, and executes the request, and under whose authority does each step run?
Authorization: what may each identity access or change?
Provenance and integrity: where did the data come from, and could it have changed outside the expected process?
Freshness and state: is the data current enough, and does the target still satisfy the expected conditions?
Validity: does the value satisfy its syntax and domain rules?
Approval: did an authorized reviewer approve this exact operation?

These properties do not substitute for one another. Authentication does not make input safe. Schema validation does not grant access. Approval does not freeze permissions or resource state.

Trace the whole feature

Consider a support assistant that can search customer records, summarize a ticket, draft a response, and request that a ticket be closed.

The action path branches after policy classification:

authenticated support user
    -> application resolves identity and tenant
    -> trusted data-access path constrains retrieval
    -> selected content crosses to the model provider
    -> model proposes a response or action
    -> application parses and validates the proposal
    -> application loads the target inside the caller's tenant
    -> authorization evaluates caller and resource
    -> application policy classifies the canonical action

        Deny
            -> stop and record the decision

        Allow
            -> execution boundary re-checks current state, authorization, and policy
            -> executor atomically persists the action and Allow decision while claiming a durable Executing attempt

        RequireApproval
            -> approval service stores the canonical operation
            -> authorized reviewer approves or rejects that operation
            -> execution boundary re-checks current state, authorization, and policy
            -> executor atomically moves Approved to Executing and creates a durable attempt

    -> executor performs the effect
    -> executor records the execution outcome
    -> application applies destination-specific output handling

Logs and traces cross several parts of this path. They can capture prompts, retrieved documents, tool arguments, resource identifiers, approval details, and model responses. Treat telemetry as its own data boundary. Decide which fields may leave the application, redact sensitive values, and avoid turning full prompt capture into the default.

The call to the model provider is another boundary. Send only the data the task needs. Application policy first determines the permitted routing set from the request, data classification, and security context. A model may then choose among non-security-sensitive options such as fast or reasoning within that set. Model output must not select arbitrary providers, deployments, regions, retention settings, or credentials.

The model takes part in the flow, but it owns none of these security decisions. The same design applies to a chat UI, an agent framework, a background worker, or custom orchestration code.

Map the boundaries that change risk

I pay closest attention to places where data gains authority because that is where a plausible string can turn into a real side effect. Data exposure matters just as much when confidential information crosses into a less trusted domain.

Transition	What remains untrusted or unresolved	Application control
Request to application	Text, uploads, requested identifiers	Authentication, request validation, size and rate limits
Application to retrieval	Search text and model-selected identifiers	Tenant and resource authorization in the retrieval or data-access path
Retrieved content to model	Documents, emails, pages, tool results	Source labeling, separation from instructions, constrained tool access, downstream validation
Application to model provider	Prompts and selected business data	Data minimization, approved deployment and region, retention policy, scoped credentials
Model output to application	Text, structured output, tool arguments	Parsing, allow lists, type-specific validation, controlled failure
Application to protected resource	Requested action and resource ID	Current-state lookup and resource-based authorization
Approval client to approval service	Reviewer input and untrusted generated explanations	Reviewer authentication and authorization, request integrity, safe rendering, replay protection
Approval service to executor	Stored operation and approval artifact	Durable state transitions, version checks, reauthorization, current policy, concurrency control
Model output to client	Generated Markdown, HTML, links, or code	Output encoding, sanitization, content policy, data-loss checks
Application to telemetry	Prompts, arguments, responses, decisions	Data minimization, redaction, access control, retention limits

I would not label any one row "prompt injection defense." No filter can prove that retrieved text is harmless. The useful boundary limits what that text can influence and validates every privileged transition after the model has processed it.

Keep security context out of model arguments

The model may propose an operation. It may also provide an identity or scope as target data when the operation needs it, such as asking to view configuration for a named tenant. It must never supply or override the authenticated identity, tenant scope, delegated authority, or authorization result under which the operation runs.

Do not expose a tool contract like this:

Task CloseTicketAsync(
    string tenantId,
    string userId,
    string ticketId,
    bool userIsAuthorized);

The model controls every argument. The function cannot distinguish a real authorization result from a convincing boolean.

Use authenticated application context for the trusted values. Normalize the proposal into a canonical action before policy evaluates it:

public enum ActionDisposition
{
    Allow,
    Deny,
    RequireApproval
}

public sealed record CloseTicketProposal(
    string? TicketId,
    string? ResolutionSummary);

public sealed record CloseTicketArguments(
    string ResolutionSummary);

public sealed record ProposedAction(
    string Operation,
    string RequesterId,
    string TenantId,
    string Environment,
    string ResourceId,
    long ResourceVersion,
    CloseTicketArguments Arguments);

public sealed record ApprovalRequirement(
    string ReviewerPolicy,
    string RiskClass,
    bool AllowSelfApproval,
    TimeSpan ValidFor);

public sealed record ActionDecision
{
    private ActionDecision(
        ActionDisposition disposition,
        ProposedAction? action,
        ApprovalRequirement? approval,
        string decisionSourceId,
        string? policyVersionId,
        string? reason)
    {
        Disposition = disposition;
        Action = action;
        Approval = approval;
        DecisionSourceId = decisionSourceId;
        PolicyVersionId = policyVersionId;
        Reason = reason;
    }

    public ActionDisposition Disposition { get; }
    public ProposedAction? Action { get; }
    public ApprovalRequirement? Approval { get; }
    public string DecisionSourceId { get; }
    public string? PolicyVersionId { get; }
    public string? Reason { get; }

    public static ActionDecision Allow(
        ProposedAction action,
        string decisionSourceId,
        string? policyVersionId = null) =>
        new(
            ActionDisposition.Allow,
            action,
            null,
            decisionSourceId,
            policyVersionId,
            null);

    public static ActionDecision Deny(
        ProposedAction? action,
        string reason,
        string decisionSourceId,
        string? policyVersionId = null) =>
        new(
            ActionDisposition.Deny,
            action,
            null,
            decisionSourceId,
            policyVersionId,
            reason);

    public static ActionDecision RequireApproval(
        ProposedAction action,
        ApprovalRequirement approval,
        string reason,
        string decisionSourceId,
        string? policyVersionId = null) =>
        new(
            ActionDisposition.RequireApproval,
            action,
            approval,
            decisionSourceId,
            policyVersionId,
            reason);
}

The boundary loads the ticket through a tenant-scoped path and checks resource authorization. Summary validation is deliberately small here. A real support system may restrict formatting, require a resolution code, or check for sensitive data before storage. The reviewer UI must still encode the summary as untrusted text.

public sealed class TicketActionBoundary(
    TicketStore tickets,
    CurrentRequest currentRequest,
    IAuthorizationService authorization,
    TicketActionPolicy policy)
{
    private const string DecisionSourceId = "ticket-action-boundary:v1";

    public async Task<ActionDecision> EvaluateAsync(
        CloseTicketProposal proposal,
        CancellationToken cancellationToken)
    {
        string summary = proposal.ResolutionSummary?.Trim() ?? string.Empty;

        if (string.IsNullOrWhiteSpace(proposal.TicketId) ||
            summary.Length is 0 or > 500)
        {
            return ActionDecision.Deny(
                null,
                "Invalid proposal.",
                DecisionSourceId);
        }

        SupportTicket? ticket = await tickets.FindAsync(
            currentRequest.TenantId,
            proposal.TicketId,
            cancellationToken);

        if (ticket is null)
        {
            return ActionDecision.Deny(
                null,
                "Ticket not found.",
                DecisionSourceId);
        }

        AuthorizationResult result = await authorization.AuthorizeAsync(
            currentRequest.User,
            ticket,
            "CloseSupportTicket");

        if (!result.Succeeded)
        {
            return ActionDecision.Deny(
                null,
                "Action is not allowed.",
                DecisionSourceId);
        }

        var action = new ProposedAction(
            Operation: "support-ticket.close",
            RequesterId: currentRequest.UserId,
            TenantId: currentRequest.TenantId,
            Environment: currentRequest.Environment,
            ResourceId: ticket.Id,
            ResourceVersion: ticket.Version,
            Arguments: new CloseTicketArguments(summary));

        return policy.Classify(ticket, action);
    }
}

ASP.NET Core supports this resource-based check through IAuthorizationService.AuthorizeAsync(user, resource, policyName). The authorization handlers evaluating the policy receive the authenticated principal and the loaded ticket as the resource. The model sees neither an authorization flag nor a tenant selector.

ProposedAction is the application's canonical representation of the operation. It includes application-owned security context and execution preconditions. A denial before canonicalization has no action. Once an action has been canonicalized, a policy denial retains it so the audit record shows what was evaluated and rejected.

Each deterministic decision point attaches its own stable source ID. TicketActionBoundary identifies validation and authorization denials. TicketActionPolicy identifies classification decisions and records its policy version separately. A caller cannot supply a stale policy version before the decision happens.

In this sample, CurrentRequest.UserId and GetUserId() return the same application-owned canonical user ID. If an application compares external identities directly, it must include the issuer and relevant tenant or security domain with the subject value.

Make approval an application-policy decision

Mandatory approval always belongs to application policy or another deterministic execution boundary. The prompt can ask for confirmation to make the interaction clearer. That conversational answer is not an approval artifact and must not bypass policy.

The policy needs at least three outcomes:

public sealed class TicketActionPolicy
{
    public const string DecisionSourceId = "ticket-action-policy";
    public const string PolicyVersionId = "support-ticket-close:v3";

    public ActionDecision Classify(
        SupportTicket ticket,
        ProposedAction action)
    {
        if (ticket.Status is not TicketStatus.Resolved)
        {
            return ActionDecision.Deny(
                action,
                "Only resolved tickets can be closed.",
                DecisionSourceId,
                PolicyVersionId);
        }

        if (ticket.IsEscalated ||
            ticket.Priority is TicketPriority.Critical)
        {
            return ActionDecision.RequireApproval(
                action,
                new ApprovalRequirement(
                    ReviewerPolicy: "ApproveSupportTicketClosure",
                    RiskClass: "high",
                    AllowSelfApproval: false,
                    ValidFor: TimeSpan.FromMinutes(15)),
                "Escalated or critical tickets require review.",
                DecisionSourceId,
                PolicyVersionId);
        }

        return ActionDecision.Allow(
            action,
            DecisionSourceId,
            PolicyVersionId);
    }
}

This example allows an authorized user to close an ordinary resolved ticket automatically. It requires review for a critical or escalated ticket and denies closure while work is unresolved.

Real policies can also consider the environment, destination, amount, quantity, bulk scope, reversibility, and compliance impact. Read-only is not automatically low risk. Exporting every customer record or retrieving a secret may deserve a stricter policy than a reversible write.

Do not require approval merely because a model selected a tool. Require it when the operation's disclosure, external, financial, destructive, privilege, or compliance impact exceeds the application's automatic-execution policy.

If risk classification or policy lookup fails for a consequential action, stop. Do not fall back to automatic execution.

Store the operation that was approved

The approval service should persist the canonical ProposedAction, policy decision, and ApprovalRequirement. Another option is to persist one canonical representation with a consistency digest. The reviewer client receives an approval ID and display data. It does not reconstruct the operation for execution.

Canonical serialization makes a digest reproducible. It does not make a plain cryptographic hash tamper-resistant. An attacker who can modify both the operation and its digest can simply recompute the hash. Use a plain hash only to detect accidental inconsistency inside an already protected store.

If the value must protect integrity across a trust boundary, use an HMAC with a protected server-side key, a digital signature, or a digest stored in a separately protected trust domain. Compute it on the server from one documented serialization format. Property order, number and timestamp formats, Unicode normalization, null handling, and collection ordering must be deterministic.

A useful approval record includes:

approval ID and one-time nonce
operation name and normalized arguments
requester, tenant, environment, and security domain
target resource and expected version or state
decision source ID, policy version, and the typed approval requirement
reviewer, decision, and the reviewer policy that was evaluated
whether self-approval is allowed and any separation-of-duties rule
creation, expiry, decision, and execution-state timestamps
intended executor or delegated identity when relevant
an idempotency or deduplication key for execution

The reviewer needs authorization too. Before accepting the decision, check whether this reviewer may approve this operation for its tenant, environment, resource, and risk class. For some operations, the requester must not approve their own proposal.

public sealed record ApprovalAuthorizationContext(
    ProposedAction Action,
    ApprovalRequirement Requirement);

var approvalContext = new ApprovalAuthorizationContext(
    pendingApproval.ProposedAction,
    pendingApproval.Requirement);

AuthorizationResult reviewerAccess = await authorization.AuthorizeAsync(
    reviewer,
    approvalContext,
    pendingApproval.Requirement.ReviewerPolicy);

if (!reviewerAccess.Succeeded ||
    (!pendingApproval.Requirement.AllowSelfApproval &&
     pendingApproval.ProposedAction.RequesterId == reviewer.GetUserId()))
{
    return ApprovalResult.Denied("Reviewer is not authorized.");
}

return await approvals.RecordDecisionAsync(
    pendingApproval.ApprovalId,
    reviewer.GetUserId(),
    approved,
    cancellationToken);

Reviewer authorization may depend on current resource attributes that are not part of the canonical action, such as ownership, business unit, or classification. Load those attributes through the trusted resource path before evaluating the reviewer policy. Pass the resulting application-defined context to AuthorizeAsync. Do not ask the reviewer client or model to supply it.

RecordDecisionAsync must atomically move the record from Pending to Approved or Rejected. It rejects expired, already decided, or otherwise invalid records. Use a transaction, row version, compare-and-swap update, or an equivalent concurrency mechanism so two reviewers cannot overwrite each other's decision. A check in the client or controller is not enough.

An unauthorized submission is denied. It is not a rejection decision. Reserve Rejected for an authorized reviewer who explicitly chooses not to approve the operation.

The approval UI is a trust boundary. Render the model-generated summary as untrusted text and keep it separate from fields loaded by the application, such as the requester, ticket ID, current priority, environment, and expected effect. Generated markup must not impersonate authoritative UI or conceal part of the operation.

Approval binds to the canonical operation, not to the persuasiveness of the explanation.

For browser-based approval endpoints that use ambient credentials such as authentication cookies, apply the same request-integrity and anti-forgery protections as other state-changing endpoints.

Re-check everything that can change

An automatic Allow and an approved action enter the same execution boundary with different admission evidence. Neither path freezes authorization, policy, or resource state.

The two paths should converge like this:

automatically allowed action
    -> verify the Allow decision and canonical action
    -> reload the resource through the trusted tenant path
    -> re-authorize the requester
    -> compare resource version and expected state
    -> evaluate the current action policy
    -> atomically persist the action and current Allow decision while claiming a durable Executing attempt

approved action
    -> verify binding, expiry, evidence of decision-time reviewer authorization, and decision state
    -> reload the resource through the trusted tenant path
    -> re-authorize every identity whose current authority execution depends on
    -> compare resource version and expected state
    -> evaluate the current action policy and approval compatibility
    -> atomically move Approved to Executing and create a durable attempt

both paths
    -> perform the effect with idempotency or explicit duplicate handling
    -> move Executing to Completed, OutcomeUnknown, FailedRetryable, or FailedFinal
    -> record a result without leaking sensitive content

If a resource change can invalidate execution, condition the claim on the version that was just checked. An optimistic concurrency condition is one option. Another is to prevent incompatible changes until the point of commitment. For an external effect, define that point and decide how to coordinate conflicting state changes.

The durable-attempt and idempotency rules also apply to automatically allowed actions when their side effects require them. Approval changes the admission path, not the reliability requirements of execution.

The automatic path does not need a separate durable Allowed state. Its execution claim atomically persists the canonical action and current Allow decision as admission evidence with the durable attempt.

On the approval path, reviewer authorization is mandatory when the decision is made. Revalidating the reviewer during execution depends on policy. Some systems treat an approval as a valid historical act after the reviewer changes roles. Others require that authority to remain in place until execution. Record which rule your system uses.

The automatic path must still receive Allow when it re-evaluates policy. On the approval path, a current Deny stops execution. A changed approval requirement invalidates the stored approval unless the current policy explicitly accepts approvals created under its recorded version. A later Allow must not silently remove a control that the stored decision required.

The transaction design depends on the side effect. A database update can often move the execution state and apply the change in one transaction with an optimistic concurrency token. An external email or payment cannot share that transaction.

For external effects, use a durable execution record or outbox and an idempotency key when the provider supports one. Recovery must treat an abandoned or stale Executing attempt as potentially applied. Reconcile the external result or rely on a safe idempotency mechanism before retrying. There is no generic exactly-once guarantee across a local database and an arbitrary external system.

Approval-state transitions and idempotency solve different problems. The state machine stops an approval decision or execution claim from being replayed. Idempotency prevents duplicate effects for the same operation. The resource-version check detects changes covered by the resource's versioning policy. Explicit expected-state checks determine whether the security- or business-relevant preconditions still hold.

Before initiating the effect, a failure in reauthorization, current-state loading, policy evaluation, durable-attempt persistence, or approval validation when required must prevent execution. After an external effect may have occurred, a persistence failure cannot turn that effect into a denial. Move the attempt to OutcomeUnknown when possible, block blind retries, and reconcile the external result instead of marking the operation Pending or Rejected.

Structured output narrows the interface

Structured output makes the boundary easier to inspect. It can reject missing properties, wrong types, and unsupported shapes before business logic runs.

It cannot establish intent or permission. This is valid JSON:

{
  "ticketId": "ticket-from-another-tenant",
  "resolutionSummary": "Customer confirmed resolution."
}

Do not ask the model whether the ticket belongs to the current tenant. Load it through the trusted tenant path and evaluate authorization in application code.

Different model-controlled values need different controls. Prefer predefined queries or query builders over arbitrary generated SQL.

For URLs, restrict schemes and destinations. The network endpoint used for the connection must satisfy the DNS/IP policy. Do not validate one resolution and then allow the HTTP client to perform a second, unchecked resolution. Disable redirects or repeat the destination and DNS/IP checks for every redirect target. HttpClientHandler.AllowAutoRedirect defaults to true, so configure it intentionally for server-side fetches.

Resolve file paths against an allowed root and handle traversal and links. Apply destination and data-loss policy to recipient addresses. Map configuration choices to allow-listed identifiers.

There is no generic ValidateModelOutput() call that makes all of those values safe.

Retrieved content and tool results remain data

Enforce tenant and resource authorization inside the trusted retrieval or data-access path before the caller or model can observe protected content or result metadata. A metadata filter is one option. Row-level security, ACL-aware repositories, separate indexes, per-tenant storage, and scoped service identities can enforce the same boundary.

Remove unauthorized candidates inside the trusted authorization boundary. Their content, identifiers, scores, counts, and metadata must not reach an unauthorized component, model, client, cache, log, or trace.

Keep that check as close to candidate generation as practical. A trusted application or retrieval service may create a broader internal candidate set and apply authoritative security trimming, but only when that broader access is legitimate and nothing unauthorized leaves the trusted path.

A matching document can contain malicious instructions, inaccurate statements, or stale facts. Preserve source and version metadata, but do not mistake labels for enforcement. Retrieved content may inform a proposal. It does not authorize the next operation.

Tool results have the same problem. An internal function may return text from an email, website, issue tracker, or database field that another user controls. The function call came from trusted code. Its returned text did not become a trusted instruction because of that.

Indirect prompt injection is one way this boundary fails. No prompt or content filter removes the risk completely, so downstream capabilities still need deterministic controls.

Least privilege limits what a failure can do

A document summarizer does not need permission to send email. A support assistant that drafts refunds may not need permission to issue them. Separate read and write capabilities where practical, and put sensitive operations behind narrow application services instead of exposing broad infrastructure clients to the model-facing layer.

Prefer credentials and data-layer controls that make cross-tenant access impossible. Some systems still use a shared service identity with broad data access. When that is unavoidable, centralize tenant enforcement and test it so model-generated identifiers or queries cannot bypass the trusted path.

Execution identity also needs an explicit design. The application can act with the requester's delegated authority, or it can use a service identity after checking requester authorization and approval. Those choices behave differently when permissions are revoked and when tokens expire. They also produce different audit trails. Record who requested, approved, and executed the operation rather than collapsing all three into "the user."

OWASP describes excessive agency as excessive functionality, permissions, or autonomy. Reducing any one of them limits the damage when the model makes a bad decision or untrusted content influences it.

Review boundaries through failure cases

An architecture diagram names components. A boundary review should say what happens when an assumption fails.

For each transition, record:

Which values cross it?
Which property must be true first?
Which deterministic component checks that property?
What happens when the check fails?
What evidence is logged without exposing protected content?
What can the component still do if this check has a bug?

For the ticket example:

Input	Required property	Enforcement	Failure behavior
Ticket ID	Exists inside the caller's tenant	Tenant-scoped repository	Reject without revealing another tenant's ticket
Resolution summary	Meets storage and display rules	Application validator	Return a controlled validation error
Close request	Requester may close this resource	Resource authorization policy	Reject and audit the denial
Approval decision	Reviewer may approve this risk class	Reviewer authorization policy	Reject the decision
Approval decision	Record is `Pending` and unchanged	Approval service with atomic transition	Reject expired or already decided records
Current ticket	Version and state still match	Repository and action policy	Invalidate the stale approval
Execution attempt	Allowed or approved operation is claimed once	Durable execution state machine	Recover, reconcile, return the recorded result, or block retry

That is more useful than writing "validate AI output" beside a model box. It names the property, owner, and failure contract.

When this level of control is useful

Map explicit trust boundaries when an AI feature:

retrieves protected or tenant-scoped data
consumes uploaded, external, or user-controlled content
turns model output into routing or workflow decisions
sends data to an external provider, tool, plugin, or telemetry system
writes application state or communicates with people
affects money, permissions, production resources, or regulated data

A small drafting feature with no protected data and no side effects needs fewer controls. It still needs safe rendering and appropriate storage rules.

Sometimes the honest answer is that an LLM should not make the decision. If deterministic checks cannot make the transition safe enough for its consequence, keep that operation outside the AI feature.

Practical boundary checklist

Before shipping, check that:

identity, delegation, and tenant scope come from authenticated application context
retrieval authorization runs inside the trusted authorization path before protected content or result metadata becomes observable outside it
model-provider and telemetry boundaries minimize and protect sensitive data
retrieved content and tool results remain untrusted data
structured output receives validation appropriate to the value type and destination
authorization runs against the current protected resource
application policy chooses Allow, Deny, or RequireApproval
mandatory approval is verified outside the prompt
the reviewer is authorized for the operation and cannot bypass separation-of-duties rules
approval binds to the canonical operation, resource version, policy decision, typed requirement, and expiry
approval decisions use an atomic Pending to Approved or Rejected transition
automatically allowed and approved actions enter the execution boundary through a durable attempt
recovery treats stale Executing attempts as potentially applied and blocks unsafe retries
the executor re-checks current authorization, resource state, and current policy
when approval was required, the executor also validates policy compatibility and approval state
retries are idempotent or deduplicated where possible, and blocked when safe retry cannot be guaranteed
execution credentials follow least privilege and tenant enforcement cannot be influenced by model input
approval and policy failures before effect initiation stop consequential execution
logs record decisions without copying sensitive context by default
each client applies output handling for its destination, such as contextual encoding, restricted Markdown, or URL validation

The model can interpret, rank, summarize, and propose. The application owns the decision to act.

Why Retrieval Exists

Lukas Walter — Wed, 12 Aug 2026 15:30:00 +0000

Retrieval exists because the model is not your application's knowledge layer.

A model can answer from patterns encoded in its parameters and from context supplied with the current request. It has no live access to your application data unless the application provides it. Internal documents, current product rules, tenant-specific policies, and recently changed runbooks do not become available just because the application can call a language model.

The model's built-in knowledge lives in its weights. When the application relies on those weights, the model generates likely output rather than querying a versioned source of truth. The application cannot reliably ask where a fact came from, whether it reflects the approved version for a given date, or whether it applies to the current user. Even if a fact appeared in training data, the model might not recall it correctly.

Retrieval lets the application select evidence from identifiable sources. When the pipeline preserves the necessary metadata, the result can retain source ownership, version, provenance, and access information. The model can still produce a bad answer, but the application no longer depends on model weights as its only knowledge source.

Sending every possible document with every request is not a durable solution either. The application needs a pipeline that manages source freshness, determines which records the user may access, finds relevant evidence, and decides what is worth placing in the model context.

In this article, retrieval means relevance-based evidence selection from a candidate corpus:

Select useful evidence from multiple candidates at runtime.

In RAG, the application makes that selected evidence available to the model as context.

Retrieval does not make the evidence true or the model deterministic. It does not enforce permissions on its own.

The model has a knowledge boundary

A language model draws on patterns stored in its weights and the context supplied with the current request. Neither one owns your application's source of truth.

Built-in model knowledge is the wrong place to own application facts because it is:

not live: it does not automatically reflect a policy changed yesterday or the current state of a customer's order
not authoritative: the model does not own your policies, runbooks, or business state
not reliably attributable: the application usually cannot trace a parametric fact back to an approved source
not guaranteed to be recalled correctly: training exposure does not guarantee an accurate answer
not dynamically access-scoped: model weights do not enforce the current user's tenant, role, or document permissions at request time
not version-addressable: the application cannot reliably request the approved policy version for a particular date

Changing to a newer model may change what the model can answer from its built-in knowledge. It does not connect the model to your live application data.

You can place additional information into the request. That works well when the input is already available and reasonably small. For example, a model can summarize a document the user has just uploaded without a retrieval system.

The problem changes when the application owns thousands of possible documents and only a few of them matter for one question. At that point, context is a selection problem.

Retrieval is runtime context selection

A minimal standard retrieval-augmented generation flow looks like this:

user question
-> retrieve relevant evidence
-> assemble the allowed context
-> generate an answer from that context

A common preparation flow for document-based retrieval looks like this:

source documents
-> parse and chunk
-> add metadata
-> index

The runtime side of that standard RAG flow selects information for one request:

question
-> search and filter
-> rank results
-> select context
-> call the model

Several application layers sit between the source data and the model call. Source ownership determines which system is authoritative. Ingestion and synchronization determine which version is available for retrieval. Authorization determines which records are eligible. Retrieval and ranking find relevant evidence within that set. Context assembly decides what enters the model request.

This is why RAG is an engineering concern rather than a prompt trick. Each stage affects the answer and fails in a different way.

Microsoft's RAG design guidance separates these stages for the same reason: document preparation, retrieval, and generation have different failure modes and need to be evaluated separately.

Retrieval is not the same as vector search

Retrieval means finding useful information for the request. Vector search is one way to do that.

A retrieval path might use:

lexical or full-text search for terms and phrases
metadata filters for tenant, product, language, or version
vector search for semantic similarity
hybrid search that combines text and vector results
a relational database with suitable text or vector search
an internal API that already exposes a search capability

The information-retrieval phase of a RAG system may use full-text, vector, hybrid, filtered, or multi-query search. Embeddings are important, but they are not the definition of retrieval.

In ordinary language, an API call that fetches application state also retrieves data. I use retrieval more narrowly here to mean selection from a candidate corpus. A structured lookup or query starts with a known key or explicit predicates and returns application state. Corpus retrieval starts with an information need and ranks relevant evidence among multiple candidates. They have different failure modes and should not be forced through the same search path.

Without this distinction, teams often send every question through a vector database because the feature has an LLM in it.

If a user asks for order A-1842, semantic similarity is not the main problem. The application already has an exact identifier. Query the system of record.

If a user asks, "Which runbook explains intermittent timeouts after a deployment?", semantic retrieval may be useful because the wording in the question might not match the document title.

Choose the retrieval method from the shape of the question and the data, not from the AI label attached to the feature.

What retrieval gives the application

Corpus retrieval becomes useful when the required knowledge is outside the current request and must be selected from many candidates.

Access to private and domain-specific knowledge

Internal policies, product documentation, support notes, architecture decisions, and operational runbooks belong to the application or organization. Retrieval makes selected parts of that corpus available at request time.

The source system still owns the knowledge. The model receives only the material selected from that corpus.

Selection from more information than one request needs

A large knowledge base may contain millions of words, but a question might depend on three paragraphs.

In a standard RAG pipeline, retrieval reduces the candidate set before the model call. A smaller, focused context is usually cheaper and easier to inspect than repeatedly sending the entire corpus. Agentic systems may search several times between model calls, but each search still performs the same selection job.

A larger model context window can move the limit, but it does not remove the selection problem. More available space does not tell the application which document is current or which section answers the question.

Independent updates

Application knowledge changes on its own schedule.

A runbook can be corrected, a policy can expire, or a product version can be retired without changing the model. A retrieval system can reindex that source data and expose the updated material at runtime.

This does not make freshness automatic. The ingestion and deletion paths still have to work. Keeping knowledge outside the model allows it to change independently. Retrieval makes updated material selectable from a corpus without retraining the model.

Permission-aware context

The application can apply tenant, role, product, region, or document-level constraints while it queries the corpus.

Those constraints must come from trusted application context. Similarity is not authorization, and a model-supplied tenant ID is not an access-control decision.

Keep eligibility filters separate from semantic ranking. Authorization should narrow the eligible records using trusted application context. Retrieval can then rank those records by relevance. I use the same boundary in Keep vector search filters separate from semantic ranking.

Inspectable evidence

Retrieved results can carry document IDs, chunk IDs, versions, headings, and source URLs into the generation step.

That gives the application a traceable retrieval path:

question
-> retrieval query
-> retrieved chunk
-> source document

This metadata is the basis for an inspectable evidence path, but it does not prove that a generated claim came from a particular chunk. The model may combine sources, rely on its weights, cite the wrong chunk, or add an unsupported claim. To trace a claim back to evidence, the application has to create an explicit citation or attribution and validate it. Groundedness evaluation answers a separate question: whether the supplied evidence supports the generated claims. Keep RAG citations tied to retrieved chunks covers the first part of that work. Microsoft's RAG evaluation guidance treats retrieval quality and response groundedness as separate measurements.

One assistant, six different data problems

Consider an internal support assistant. It receives these requests:

Request	Best starting point	Why
"Explain eventual consistency."	Model knowledge or a supplied reference	This is general knowledge unless the company has a specific definition or requires approved sources.
"What is our current cancellation policy for German customers?"	Filtered document retrieval, if the governed policy corpus owns the answer	A policy service or rules engine may be the better source when it owns the current policy.
"What is the status of order A-1842?"	Direct API or database lookup	The request identifies one structured record. Similarity search adds ambiguity.
"Which orders have been delayed for more than three days?"	Structured API or database query	The request applies explicit filters to application state. Relevance ranking adds no value.
"Calculate the SLA deadline for this incident."	Deterministic application code	Business rules and date calculations should not depend on generated reasoning.
"Summarize the incident report I attached."	Put the report directly in context if it fits	Retrieval may still help when the report is too long or the task needs section selection.

All six requests could be sent to a language model. They should not all use the same data path.

The model can still provide the conversational interface. The application decides whether the request needs general generation, document retrieval, a structured query, deterministic code, or direct context.

That routing decision is part of the application architecture.

What retrieval does not solve

It does not solve:

missing knowledge in the source corpus
stale or contradictory documents
poor chunk boundaries
weak metadata
incorrect permission filters
bad ranking
excessive or noisy context
unsupported claims added during generation
malicious instructions in retrieved content

If the only indexed policy is obsolete, retrieval can return it perfectly and the final answer will still be wrong.

Poor chunk boundaries can prevent the retrieval pipeline from returning enough context to preserve the author's meaning.

Authorization should be enforced as part of the retrieval query or before results leave the trusted retrieval path. Filtering only after ranking can expose sensitive content to downstream components, logs, caches, or telemetry. It can also leave too few eligible results in the original top-k set. Azure AI Search security filters are one example of applying document-level constraints during the query.

Retrieval does not make content safe. Documents, web pages, and user-provided files can contain misleading instructions or indirect prompt injections. Treat retrieved content as data, not as trusted instructions. Keep tool permissions and authorization decisions outside the model. Indirect Prompt Injection Is a Trust Boundary Problem explains this boundary in more detail.

If you evaluate only the final answer, you may blame the prompt for a retrieval miss.

When corpus retrieval is the wrong solution

Building a dedicated corpus retrieval system is work. Depending on what already exists, the application may have to own ingestion, indexing, synchronization, ranking, evaluation, and operations. Build one when the use case needs runtime selection from a knowledge corpus.

Use a structured lookup or query for application state

Questions about a known customer, invoice, shipment, subscription, or ticket often belong in an authenticated API or database query.

The same applies to filters and aggregations over many records. A request for all delayed orders or the number of cancellations last month belongs in SQL, an API with structured filters, or analytics infrastructure.

Return the structured result to the application. Let the model explain it only if natural-language presentation adds value.

Vector similarity should not decide which customer's invoice the user meant when the request already contains the invoice ID.

Use deterministic code for rules and calculations

Totals, eligibility checks, deadlines, authorization decisions, and state transitions should use application logic.

A policy document may be retrieved to explain why a rule exists. The actual decision should still come from the code or policy engine that owns it.

Retrieval provides evidence. It does not replace deterministic execution.

Use direct context when the relevant input is already known

If the user uploads one short document and asks for a summary, pass that document to the model.

If every request needs the same small, stable set of instructions, keeping them in a versioned prompt or configuration file may be simpler than building a search index.

Retrieval becomes useful when selection is real. Do not manufacture a search problem where none exists.

Fix behavior problems at the behavior layer

Retrieval is not the first tool for changing tone, output shape, refusal behavior, or tool-selection policy.

Use prompts, structured output, validation, application logic, or, in suitable cases, fine-tuning for those concerns.

Adding more documents will not repair a response-format contract.

Do not treat an ungoverned corpus as authoritative

Retrieval can work over public web pages, user uploads, support tickets, or sources that disagree. The application still needs to preserve provenance and account for the quality and trust level of each source.

A research assistant can report that two sources conflict. An internal policy assistant should not present an outdated or unowned document as company policy. When the use case requires an authoritative answer, establish the source owner, update path, and access model before building retrieval around it.

When retrieval is enough without generation

Sometimes the user needs a list of matching documents, not a generated answer.

In that case, return search results with titles, snippets, filters, and links. Generation adds cost and can blur the distinction between what a source says and what the model inferred.

RAG is useful when synthesis adds value. Search remains a valid product on its own.

A practical decision sequence

Before adding retrieval to an AI feature, ask these questions in order.

1. Does the task require application-owned or source-grounded information that is not already in the request?

If no, retrieval is not needed. Use the model, direct context, or deterministic code according to the task.

If yes, identify the system that owns or provides the missing information.

2. Is the need a structured query, deterministic computation, or relevance-based evidence selection?

Structured lookup or query: call the database or API using explicit keys, filters, or aggregations.
Calculation or business decision: run deterministic code.
Relevance-based evidence selection from a corpus: consider retrieval.

Some requests need more than one path. An assistant might retrieve the policy explanation and call an API for the customer's current account state.

3. What authority does the source corpus have?

Identify who owns it, how it changes, how deletions propagate, and whether the use case needs authoritative or merely discoverable sources.

If the answer must be authoritative but the corpus has no freshness or ownership model, the work starts before embeddings.

4. Can the application enforce scope before generation?

Define tenant, user, role, region, product, and document constraints. Apply them before the selected context reaches the model.

Do not treat a good semantic match as permission to reveal the content.

5. Can you evaluate retrieval separately?

Keep test questions with expected sources or relevant chunks. For routine telemetry, record document and chunk identifiers, scores, applied filters, and context-selection decisions. Keep raw retrieved content out of logs unless the application's privacy, security, and retention rules allow it.

If a response is wrong, you should be able to distinguish:

no relevant source existed
the source existed but was not retrieved
the right source was retrieved but excluded during context assembly
the right context reached the model but the generated answer was still wrong

Without that separation, every failure looks like a model failure.

When to use retrieval

Several conditions make corpus retrieval worth considering:

the answer depends on private, domain-specific, or changing knowledge that must be selected from multiple candidates
the available corpus is larger than the useful context for one request
the application must select evidence using relevance ranking and trusted metadata constraints
the selected evidence should remain traceable to its source
the source and retrieval paths can be maintained and evaluated

Do not use retrieval when:

the required input is already present and small enough to send directly
the request needs a structured lookup, filter, or aggregation over application state
deterministic code owns the calculation or decision
the problem is response behavior rather than missing knowledge

Retrieval can still help discover material in an ungoverned corpus. Do not use that corpus as the authoritative basis for an answer.

Do not add generation when plain retrieval results solve the user need.

Use corpus retrieval when the application must select relevant evidence from many candidates at runtime. When the application supplies that evidence to a model for synthesis, retrieval is the context-selection stage of RAG.

Start with the system that owns the information. If the input is already known, use the simpler path.

The Model Is Only One Dependency. Map the Rest.

Lukas Walter — Thu, 06 Aug 2026 15:30:00 +0000

A model endpoint can be healthy while the feature around it is broken.

A valid model response says nothing about whether the application used authorized data, checked current business facts, or completed a requested state change.

The model does not own most of the feature's contract. Application code must enforce account scope, choose the right systems of record, validate hard constraints, and report whether state changes succeeded.

A dependency register adds behavioral contracts to the interactions shown in an architecture diagram.

Start with enforceable properties

"Recommend a suitable guitar from our current catalog" is a useful product promise, but it is not precise enough for an operational contract.

Translate it into properties the application can enforce:

Every recommended product ID resolves against the accepted catalog version used for the request.
Every displayed price comes from the designated pricing source and is no older than the allowed maximum when the response is produced.
The feature says "in stock" only when the inventory observation is recent enough for that claim.
Every customer record matches the authenticated account scope before the application uses it.
The response reports a successful save only after the shortlist write is confirmed.

Not every part of the product promise becomes a runtime invariant. Product existence, account scope, freshness, and save outcomes can be enforced deterministically. Whether a recommendation is genuinely useful or subjectively suitable remains an evaluation problem. It needs quality criteria, representative test cases, and feedback; the application cannot generally prove subjective suitability for each request.

Freshness limits and time budgets belong to the product and its service objectives. Name them, assign an owner, and enforce them.

For the guitar assistant, the request path might look like this:

authenticated shopper
    -> resolve account scope
    -> authorize profile access
    -> read preferences and optional purchase history when enabled
    -> resolve the accepted catalog version
    -> verify that the version is still acceptable
    -> read product records and specifications
    -> read current price observations
    -> read inventory observations when availability matters
    -> resolve prompt, schema, route, deployment, and model versions
    -> call the model
    -> validate schema and selected product references
    -> resolve authoritative prices and availability when relevant
    -> revalidate budget and applicable availability constraints
    -> compose authoritative facts into the response
    -> write a shortlist when requested
    -> return recommendation and save outcome

observability: signals across the complete path

Catalog-version selection and request-time freshness validation are separate because they fail differently.

For exact business facts, the model can rank product IDs while application code resolves price and relevant availability from their systems of record. If the final price exceeds the budget or required availability no longer satisfies the request, reject that candidate. Use another validated candidate, make one bounded regeneration when the contract allows it, or state that the product-backed request cannot be completed.

The register applies established reliability thinking to any distributed feature. AI makes it especially useful because a successful model call can still produce an invalid result. Generated references, probabilistic quality, token and quota limits, content filters, and fallback compatibility are especially easy to miss when teams rely mainly on service diagrams.

Separate three dependency roles

Not every dependency participates in the immediate response. I separate three roles before deciding how critical an interaction is.

Role	Purpose	Guitar assistant example
Execution	Produces or validates the result or a requested state change	Account authorization, price read, model request, shortlist write
Prerequisite	Creates usable state before requests can consume it	Catalog ingestion and accepted-version publication
Operational	Detects, supports, audits, or recovers the feature	Telemetry export, security-event delivery, recovery scheduler

An operational dependency may not affect one response, yet remain required for audit, billing, safety, or recovery. Analytics may be best effort; required security-event delivery is not.

Classify interactions, not services

A capability rarely has one relationship to the request: catalog publication happens beforehand, version validation inline, and recovery later. Give each identifiable interaction a row with its role, condition, criticality and horizon, timing, and contract reference. Split it again at another owned boundary or operating lifecycle; keep different failure classes of one call inside its failure contract.

Timing does not imply importance. Criticality has three useful values:

Required: The declared capability, claim, or operational obligation cannot be fulfilled without the interaction.
Degradable: The system may omit or reduce the affected capability or operational behavior under an explicitly defined degraded mode. Any user-visible limitation must be made clear.
Best effort: Failure invalidates neither the user-visible result nor any required operational obligation, although it may reduce analytical or operational value.

Criticality describes whether the affected capability or operational obligation may be omitted under the stated condition. It never relaxes security, integrity, authorization, or correctness constraints when an interaction is attempted. "Live inventory is required" is too broad. "A current inventory observation is required for this request when the response claims an item is in stock" is operationally useful.

For the guitar recommendation, an initial classification could look like this:

Interaction	Role	Condition	Criticality and horizon	Timing	Contracts and details
Resolve and enforce account scope	Execution	Stored customer data is read	Required for that read	Inline	Profile access policy
Read purchase history	Execution	Optional purchase-history personalization is enabled	Degradable for this recommendation	Inline	Profile read · feature detail
Publish an accepted catalog version	Prerequisite	Product-backed recommendations remain available	Required over the freshness window	Pre-request	Catalog publication
Verify accepted catalog version	Execution	Product-backed recommendation	Required for this request	Inline	Catalog acceptance
Read product record and specifications	Execution	Response recommends a catalog product	Required for this request	Inline	Catalog read
Read authoritative price	Execution	Response displays a price or enforces a budget	Required for that claim	Inline	Pricing read
Read inventory	Execution	Response makes an availability claim	Required for that claim	Inline	Inventory read
Resolve compatible prompt and output-schema versions	Execution	Model-generated recommendation	Required for this request	Inline	Prompt and schema contract
Resolve model route and version	Execution	Model-generated recommendation	Required for this request	Inline	Model routing
Call the model deployment	Execution	Model-generated recommendation	Required for this request	Inline	Model gateway
Validate schema and product references	Execution	Model-generated recommendation	Required for this request	Inline	Output contract
Revalidate budget and applicable availability constraints	Execution	Response enforces a budget or makes an availability claim	Required for that constraint or claim	Inline	Recommendation constraints
Attempt shortlist save	Execution	Shopper asks to save	Required for the save-success claim	Inline	Shortlist write · feature detail
Discover eligible unresolved shortlist operations	Operational	Inline attempt lease expired and operation remains non-terminal	Required over the recovery objective	Deferred	Recovery discovery
Resume or reconcile non-terminal shortlist operation	Execution	Eligible operation remains unresolved	Required until terminal, when promised by the persistence contract	Deferred	Shortlist recovery · feature detail
Export recommendation analytics	Operational	Product analytics is enabled	Best effort unless another obligation requires it	Deferred	Analytics export

In a working register, the last cell links the shared contract and any feature detail. An explicit promise to use stored purchase history makes that read required unless the request supplies the same information.

Do not combine catalog, pricing, and inventory reads when they are separate calls or owned contracts. Conversely, DNS failure, throttling, content filtering, and malformed output may be separate failure classes of one model call rather than four register entries.

Stop at an owned boundary

Stop at an observable boundary with a documented owner and support contract. Internals stay outside the register unless your application interacts with them separately or they have a distinct operating lifecycle.

Expand only risky interactions

The classification table is the feature register. Expand rows that handle protected data, change state, return ambiguous outcomes, permit degradation, or need a distinct operational response. Other rows can reference a shared contract.

Artifact	Owns
Feature register	Interaction-specific condition, criticality, time horizon, and degraded user behavior
Shared capability contract	API or schema version, owner, general retry semantics, SLO, and compatibility policy
Runbook or threat model	Operational response, escalation details, and trust-boundary analysis

Store shared facts once; feature entries contain only operation-specific behavior and overrides.

Use this expanded template only where the risk warrants it:

### <Interaction name>

- Feature register row: <link>
- Capability contract: <link and version>
- Data classification: <organization's classification>
- Freshness or completion requirement: <maximum age or completion target>
- Per-attempt and total time budget: <attempt and end-to-end allocation>
- Failure behavior: <response by material failure class>
- Retry override: <feature-specific difference>
- Degraded mode: <valid reduced behavior, or none>
- Consistency and idempotency: <read or write guarantees>
- Concurrency control: <lease claims and conditional transitions>
- Diagnostic signals: <outcome, duration, attempt, controlled identifiers>
- Escalation path: <owner or runbook>

Read authorized purchase history

- Feature register row: Execution, inline, and degradable when optional purchase-history personalization is enabled; authorization remains required
- Capability contract: Versioned profile-read contract with enforced authenticated account scope
- Data classification: Organization's protected-customer-data class
- Freshness or completion requirement: No feature-specific freshness override
- Per-attempt and total time budget: All attempts fit inside the request's profile-read allocation
- Failure behavior:
  - Unavailable or timed out: continue without purchase-history personalization
  - Authorization denied: do not read; continue only if an unpersonalized result is valid
  - Scope or cache-partition mismatch: suppress the result and start the security incident path
  - Malformed response: reject the data and record a contract failure
- Retry override: Do not retry authorization denials, scope mismatches, malformed data, or after caller cancellation
- Degraded mode: Omit personalization without claiming otherwise. Suspected cross-account exposure has no degraded mode
- Consistency and idempotency: Use account-partitioned cache keys and preserve account scope through every call
- Diagnostic signals: Outcome, duration, attempt, authorization reference, and controlled scope identifier; exclude purchase-history contents
- Escalation path: Profile runbook for availability; security incident runbook for suspected cross-account access

This distinction is easy to miss: purchase-history personalization is degradable, but correct authorization is not. Missing data can produce a valid reduced result. Data from the wrong account invalidates the entire operation and may indicate that protected data was exposed.

Attempt shortlist save

- Feature register row: Execution, inline, and required for the save-success claim when the shopper asks to save
- Capability contract: Versioned idempotent write state machine with status lookup by operation ID
- Command durability: `PendingDispatch` stores the immutable save command, or a durable reference sufficient to reconstruct and verify it, plus its fingerprint, operation ID, idempotency key, attempt lease, and `nextAttemptAt`
- Command identity: Reusing the operation ID or idempotency key with a different command fingerprint is rejected
- Data classification: Organization's applicable customer-data class
- Freshness or completion requirement: Report success only after the write is confirmed
- Per-attempt and total time budget: Write and inline lookup share the request's save allocation
- Failure behavior:
  - Confirmed success: report that the shortlist was saved
  - Confirmed rejection: return the recommendation with a separate save failure
  - Timeout or lost response: mark `OutcomeUnknown` and query while inline time remains; if still unknown, return the operation ID with an unconfirmed save status
- Retry override: After an unknown outcome, retry only with the same idempotency key when safe deduplication is guaranteed
- Degraded mode: The recommendation may still succeed, but the save result remains separate
- Consistency and idempotency: One operation ID, idempotency key, and command fingerprint identify the logical save
- Concurrency control: Dispatch requires an atomic conditional update that acquires the lease and transitions the eligible state to `Submitting` using the expected version. Terminal updates require the expected version and attempt identity so an older attempt cannot overwrite a newer result
- Diagnostic signals: Operation ID, outcome, duration, and attempt; exclude shortlist contents
- Escalation path: Persistence errors use the write runbook; the recovery scheduler discovers eligible unresolved operations

A confirmed rejection and a timeout with an unknown outcome are different states. The inline interaction ends with the response; unresolved recovery does not.

Resume or reconcile non-terminal shortlist operations

- Feature register row: Execution, deferred, and required until the operation reaches a terminal state when the persistence contract promises recovery
- Capability contract: State machine, authoritative status lookup, and idempotent retry contract for the original operation ID
- Data classification: Same class as the original save
- Freshness or completion requirement: Reach a terminal outcome within the recovery objective
- Per-attempt and total time budget: Scheduled attempts use the recovery budget, not the request budget
- Recovery eligibility: The inline attempt lease has expired and `nextAttemptAt` is due
- Concurrency control: The worker atomically acquires the lease and transitions the expected state and version before acting. Terminal updates require the active attempt identity and expected version
- Recovery behavior:
  - `PendingDispatch`: dispatch the stored command with its existing operation ID and idempotency key
  - `Submitting` or `OutcomeUnknown`: perform authoritative lookup or a safe idempotent retry of the stored command when permitted by the shared contract
  - Confirmed saved: record the terminal success
  - Confirmed rejected, or authoritatively absent after the contract's visibility window: record the terminal failure
  - Non-authoritative absence, unknown, or unavailable: reschedule within the recovery policy; escalate when its deadline is exhausted
- Retry override: Reuse the original operation ID, idempotency key, and command fingerprint; never create a new logical save
- Degraded mode: None for saving; the returned recommendation remains valid
- Consistency and idempotency: Lookup, retry, and recovery refer to one logical operation
- Result publication: Persist the terminal status so the shopper and support tooling can resolve the original operation ID
- Diagnostic signals: Operation ID, outcome class, attempt, age, and terminal result; exclude shortlist contents
- Escalation path: Exhausted recovery uses the persistence-reconciliation runbook

Recovery resolves the requested operation to a terminal outcome; it does not always complete the state change. It is deferred execution work. The discovery scheduler is operational: it scans eligible unresolved operations and exposes recovery lag. The feature's operating or admission policy decides whether a breached recovery objective stops new saves, changes save behavior, restricts traffic, or triggers operator intervention.

Fallback routes and cross-region or cross-provider attempts also consume the same remaining budget unless the feature contract explicitly defines another execution model.

Preserve the contract when degrading

Another model is not automatically a safe fallback. It may behave differently with structured output, context limits, tools, latency, or safety controls. An older catalog version may restore access while violating the freshness limit. Skipping a failed validation step is not graceful degradation.

For a general comparison, valid degradation may omit optional purchase-history personalization or unverified availability claims. When the request explicitly requires products that are available now, return a clearly identified partial result or state that the availability-backed request cannot be completed. A recommendation may still return with a separate save failure.

The feature may return less, but it must not silently change what success means.

Use existing artifacts as inputs

The register synthesizes existing decisions. Diagrams identify interactions and trust boundaries; failure analysis supplies failure classes; SLOs supply budgets; threat models and runbooks supply controls and escalation paths.

Register entries can also inform trace design. Use stable interaction names and record duration, outcome, retry count, and controlled correlation identifiers. Do not log prompts, retrieved documents, purchase history, or tool arguments indiscriminately. Observability has its own access and retention boundaries.

Apply it one operation at a time

Start with one user-visible operation:

Separate enforceable properties from quality criteria.
Map and classify the interactions; split distinct boundaries and lifecycles.
Link shared contracts and expand only risky rows.
Test slow, stale, malformed, unauthorized, duplicated, unavailable, and unknown-outcome paths.

Use it for protected data, retrieval, tools, state changes, authoritative claims, or operational obligations. A disposable local spike may need only basic error handling.

A useful dependency register determines runtime behavior before an incident rather than merely explaining the incident afterward. A successful model response does not override a violated account boundary, stale price, unverified availability claim, or unconfirmed write.

Stop Letting Provider SDKs Define Your .NET AI Architecture

Lukas Walter — Mon, 03 Aug 2026 15:30:00 +0000

.NET developers can call language models through several provider SDKs. That part is solved.

The harder decision is where provider-specific code ends and application code begins.

Microsoft.Extensions.AI introduces shared abstractions such as IChatClient, common message and response types, and a middleware pipeline that fits normal .NET dependency injection patterns. Provider SDKs still matter, but they can stay near the composition root instead of defining every application service.

Models and providers still behave differently. The abstraction gives the application one stable place to handle the behavior it owns.

The first generation was organized around the SDK

A small AI feature often starts by injecting a provider client directly into the class that needs it.

public sealed class TicketSummarizer(
    AzureOpenAIClient azureClient,
    IConfiguration configuration)
{
    public async Task<string> SummarizeAsync(
        string ticket,
        CancellationToken cancellationToken)
    {
        ChatClient client = azureClient.GetChatClient(
            configuration["AZURE_OPENAI_DEPLOYMENT"]!);

        ClientResult<ChatCompletion> response =
            await client.CompleteChatAsync(
                $"Summarize this support ticket:\n\n{ticket}",
                cancellationToken: cancellationToken);

        return response.Value.Content[0].Text;
    }
}

This code is fine for a spike or a feature that needs provider-specific behavior.

The architectural problem appears when this shape becomes the default across the application:

application services know provider client types
deployment names and provider options leak into business code
streaming, tools, and structured output use different provider-specific shapes
testing requires provider SDK objects or network calls
logging, telemetry, caching, and retries are reimplemented around individual calls
changing a provider becomes an application-wide migration

At that point, the code follows the provider's model instead of the application's use case.

`IChatClient` creates an application boundary

Microsoft.Extensions.AI defines IChatClient as the common chat abstraction. Provider integrations adapt their concrete clients to that interface.

The code samples use the current Microsoft.Extensions.AI and Azure.AI.OpenAI APIs available at the time of writing. Check package versions before copying them into an application.

The provider client still exists, but it is created at the edge of the application:

using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Extensions.AI;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

AzureOpenAIClient azureClient = new(
    new Uri(builder.Configuration["AZURE_OPENAI_ENDPOINT"]!),
    new DefaultAzureCredential());

IChatClient providerClient = azureClient
    .GetChatClient(builder.Configuration["AZURE_OPENAI_DEPLOYMENT"]!)
    .AsIChatClient();

builder.Services
    .AddChatClient(providerClient)
    .UseOpenTelemetry(
        sourceName: builder.Environment.ApplicationName);

Application services now depend on the capability they need:

public sealed class TicketSummarizer(IChatClient chatClient)
{
    public async Task<string> SummarizeAsync(
        string ticket,
        CancellationToken cancellationToken)
    {
        ChatResponse response = await chatClient.GetResponseAsync(
            [
                new ChatMessage(
                    ChatRole.System,
                    "Summarize support tickets without inventing facts."),
                new ChatMessage(ChatRole.User, ticket)
            ],
            cancellationToken: cancellationToken);

        return response.Text;
    }
}

The summarizer no longer knows how the provider authenticates, which deployment name is configured, or which middleware surrounds the request.

That split is what matters. The application service owns summarization. The composition root owns the provider connection.

The abstraction is more than provider switching

Provider portability gets most of the attention. I think it is the smaller benefit.

IChatClient gives application code a shared shape for:

chat messages and roles
complete and streaming responses
tools and function calls
response metadata and usage when the provider supplies it
dependency injection
test doubles
middleware composition

Teams can build application behavior once around that interface.

A scripted IChatClient can test a summarizer, router, approval flow, or agent without calling a live model. Telemetry middleware can wrap whichever provider is configured, and integration tests or local development can supply another client without changing the application service.

This is ordinary dependency inversion applied to model access.

Middleware turns cross-cutting behavior into composition

The Microsoft.Extensions.AI package provides a chat-client builder that can compose behavior around the underlying client.

builder.Services
    .AddChatClient(providerClient)
    .UseOpenTelemetry(
        sourceName: builder.Environment.ApplicationName)
    .UseFunctionInvocation(
        configure: client =>
            client.MaximumIterationsPerRequest = 5);

The pattern should look familiar to anyone who has worked with HTTP handlers or ASP.NET Core middleware.

The function-invocation layer removes a particularly repetitive piece of provider-specific code. Without it, the application has to inspect each model response for tool calls, invoke the requested functions, add their results to the conversation, and call the model again until it produces a final response or reaches a stop condition. UseFunctionInvocation() handles that loop around the inner client. The iteration limit in the example makes one of those stop conditions explicit.

The host can add shared behavior once instead of repeating it inside every use case:

telemetry
function invocation
caching
logging
resilience policies
request or response inspection
custom policy middleware

Order still matters. Middleware is not a bag of attributes. A cache placed outside telemetry observes different behavior from telemetry placed outside the cache. Function invocation introduces additional model calls and tool executions that need their own budgets and traces.

A useful pipeline makes runtime behavior easier to inspect. If the order is difficult to explain, the composition is too clever.

Keep use-case policy outside the pipeline. Authorization, approval rules, data access, argument validation, and side-effect safety still need explicit application boundaries.

Shared interfaces do not erase provider differences

An abstraction can standardize a programming model. It cannot make models equivalent.

Providers and model families still differ in areas such as:

tool and function-calling behavior
structured-output guarantees
multimodal input support
token accounting
safety systems
streaming details
response metadata
model-specific options
rate limits and regional availability

Application code can depend on IChatClient while the composition layer still chooses a concrete provider and exposes required provider-specific capabilities deliberately.

If a feature depends on a provider-only API, represent that dependency honestly. Do not force it through a generic interface until the important behavior disappears.

Keep provider coupling visible and contained. A generic interface that hides provider-only capabilities gives the application a misleading kind of portability.

One application may need more than one client

Registering one global IChatClient is enough for a small application. Larger systems often need clients with different purposes:

a fast model for classification
a stronger model for complex generation
a restricted client for sensitive workflows
a local model for development
a separate embedding generator for retrieval

Do not hide those differences behind runtime string comparisons throughout the application.

Use explicit typed services, keyed DI registrations, or a small routing abstraction at the composition boundary. Keep stable application names separate from provider deployment names.

IChatClient classificationClient = azureClient
    .GetChatClient(
        builder.Configuration["AZURE_OPENAI_CLASSIFICATION_DEPLOYMENT"]!)
    .AsIChatClient();

builder.Services.AddKeyedChatClient(
    "classification",
    classificationClient);

builder.Services.AddTransient<SupportClassifier>();

public sealed class SupportClassifier(
    [FromKeyedServices("classification")] IChatClient chatClient)
{
    // The application depends on a stable purpose,
    // not a provider deployment name.
}

Use a key such as classification because it describes the application's purpose. Configuration decides which provider and model serve that purpose.

This helps during local development too. The classification registration can use a local model container during development and an Azure-backed client in the cloud. SupportClassifier stays unchanged because its dependency is the application's purpose rather than the environment's provider.

Agents and frameworks sit above this foundation

Not every AI feature needs an agent.

A direct IChatClient call is often enough for summarization, classification, extraction, translation, or another bounded inference task.

Agent and orchestration frameworks become useful when the application needs session state, dynamic tool selection, multi-step workflows, approvals, handoffs, or coordination between agents.

Microsoft Agent Framework uses the Microsoft.Extensions.AI abstractions as a lower layer:

application use case
    -> agent or workflow when needed
    -> Microsoft.Extensions.AI abstractions
    -> provider integration
    -> model endpoint

Start at the lowest layer that solves the problem. Reach for an agent only when the use case needs agent behavior.

A practical migration path

A full platform rewrite is unnecessary. Move one boundary at a time.

1. Find provider clients outside the composition root

Search for application services, handlers, and controllers that construct or depend directly on provider SDK clients.

Do not change provider-specific code that genuinely needs provider-specific APIs. Identify the calls that only need normal chat behavior.

2. Adapt the provider client to `IChatClient`

Create the provider client and deployment configuration in the host. Adapt it once, then register the resulting IChatClient.

3. Move prompts and use-case policy into application services

Create clients in the composition root. Keep business prompts, authorization rules, and output policy in application services.

4. Add one cross-cutting behavior through the pipeline

I would start with telemetry. It shows which calls already pass through the new boundary before caching, retries, tools, or custom middleware complicate the pipeline.

5. Replace network-heavy tests with a fake client

Test deterministic application behavior with a scripted IChatClient. Keep a smaller set of provider integration tests and separate evaluations for model quality.

This migration produces value even if the provider never changes. The application becomes easier to compose, test, and operate.

When to use the shared abstraction

Use IChatClient when application code needs common chat, streaming, tool, or structured-response capabilities and should participate in normal .NET dependency injection and middleware composition.

It is a strong default for reusable libraries, application services, tests, and cross-cutting telemetry.

When to keep the provider SDK visible

Use the provider SDK directly when the shared abstraction cannot represent a capability cleanly. Examples include provider-specific batch APIs, file management, fine-tuning operations, deployment administration, and specialized request options.

Keep that code close to an adapter or infrastructure service. Do not let one specialized requirement pull provider types through the rest of the application.

Practical takeaway

Provider SDKs connect .NET applications to model platforms. Keep that role at the edge of the application.

Create the provider client at the edge. Let application code depend on IChatClient or another purpose-specific interface. Compose telemetry, tools, caching, and other cross-cutting behavior through explicit middleware. Keep provider differences visible where they matter.

The result looks like the rest of a maintainable .NET application. Services own use cases, while provider setup and shared runtime behavior stay centralized in the host.

Sources

Microsoft Learn: Microsoft.Extensions.AI libraries
Microsoft Learn: Use the IChatClient interface
Microsoft Learn: AI tool calling
Microsoft Learn: AddChatClient
Microsoft Learn: AddKeyedChatClient
Microsoft Learn: Microsoft.Extensions.AI.Evaluation libraries
Microsoft Learn: Build an AI chat app with .NET
Microsoft Learn: Application design for AI workloads on Azure

Calling a Model Is Easy. Running an AI System Is Not

Lukas Walter — Fri, 31 Jul 2026 15:30:00 +0000

A successful model call proves that one request reached one model and produced one response. It does not prove that the feature is reliable, secure, observable, affordable, or useful.

That gap contains most of the engineering work.

The model call might be ten lines of code. The production feature is everything that happens before, around, and after those ten lines. It still has to authenticate the user, retrieve authorized data, manage time and cost, validate the result, handle failures, record what happened, and decide whether the output may trigger an action.

Do not design around the model call. Design around the complete application flow that contains it.

A successful response proves very little

The first working version of an AI feature often looks like this:

ChatResponse response = await chatClient.GetResponseAsync(
    "Summarize this support ticket.",
    cancellationToken: cancellationToken);

return response.Text;

This is a useful spike. It verifies credentials, connectivity, request formatting, and basic model compatibility.

It does not answer the questions that determine whether the feature can run in production:

Was the caller allowed to read the ticket?
Which ticket data was sent to the model?
Was sensitive content removed or retained?
What happens when retrieval returns incomplete context?
How long may the operation run?
How many retries are allowed?
What is the token and cost budget?
How is the output checked before it is displayed or used?
What happens when the provider is unavailable?
Can an operator explain why this request failed?
How will a model, prompt, or retrieval change be evaluated?

The model response is one intermediate value in a larger application flow.

Trace the complete request path

Take a support-reply feature. A user asks the application to prepare a response for a ticket.

The actual path is closer to this:

authenticated user
    -> request validation
    -> ticket authorization
    -> ticket and customer context
    -> prompt construction
    -> model call
    -> output validation
    -> policy decision
    -> draft persistence
    -> response to the user
    -> traces, metrics, and audit data

Each arrow is a boundary where latency, failure, cost, or incorrect assumptions can enter the system.

The model may answer successfully while the feature still fails because the wrong customer context was retrieved. A technically valid response may contain an unsupported claim. A good draft may be stored under the wrong tenant. A timeout may leave the user unsure whether the request is still running. A retry may create a duplicate side effect.

This is why production readiness cannot be measured by asking whether the model returned text.

The model call is not the architecture. It is one step inside a request path whose other parts can fail even when the model succeeds.

Give each boundary an explicit contract

The application becomes easier to reason about when the model call is surrounded by normal software boundaries.

For the support-reply example, I would separate at least these responsibilities:

Boundary	Contract
Request	The input is valid, bounded, and associated with an authenticated caller.
Authorization	The caller may access this ticket and its supporting data.
Context	The application can identify which data was retrieved and why it was included.
Model interaction	The request has an explicit model, prompt version, options, timeout, and budget.
Output	The result can be parsed, checked, and rejected before use.
Policy	Application code decides whether the output may be displayed, stored, or acted on.
Persistence	Repeated processing does not create inconsistent or duplicate state.
Operations	Traces and metrics reveal where time, tokens, failures, and retries occurred.

A few named responsibilities and clear enforcement points are enough. A large framework is optional.

Keep the model behind an application service

Application code should not scatter model calls across endpoints, background workers, and controllers. Put the use case behind a service that owns the complete flow.

public sealed class SupportReplyService(
    ITicketContextReader contextReader,
    IChatClient chatClient,
    IReplyPolicy replyPolicy,
    IReplyDraftStore draftStore)
{
    public async Task<SupportReplyResult> PrepareAsync(
        Guid ticketId,
        ClaimsPrincipal user,
        CancellationToken cancellationToken)
    {
        TicketContext context = await contextReader.ReadAuthorizedAsync(
            ticketId,
            user,
            cancellationToken);

        ChatResponse response = await chatClient.GetResponseAsync(
            BuildMessages(context),
            cancellationToken: cancellationToken);

        ReplyDecision decision = replyPolicy.Evaluate(
            context,
            response.Text);

        if (!decision.IsAccepted)
        {
            return SupportReplyResult.Rejected(decision.Reason);
        }

        Guid draftId = await draftStore.SaveAsync(
            ticketId,
            response.Text,
            cancellationToken);

        return SupportReplyResult.Prepared(draftId, response.Text);
    }
}

This example is deliberately incomplete. A production implementation still needs telemetry, budgets, failure mapping, prompt versioning, and tests.

The ownership boundary is the point. The endpoint asks for a support reply. The application service coordinates the use case. The model client is only one dependency inside it.

That gives you a place to add policy without teaching the UI, controller, or provider SDK about business rules.

Reliability is a set of decisions, not a retry policy

Retries are useful for some transient failures. They are not a complete reliability strategy.

A retry can make the situation worse when:

the original request is still running
the model call is expensive
the operation already changed state
the provider is throttling the entire workload
the failure is caused by invalid input or an unsupported capability
the next attempt silently uses different context

For every dependency, decide what the system should do when it is slow, unavailable, or returns an unusable result.

Failure	Possible system decision
Model timeout	Cancel the attempt and return a retryable application result.
Provider throttling	Apply bounded backoff or queue the work when the result can arrive later.
Retrieval miss	Return an explicit no-context result instead of asking the model to guess.
Invalid structured output	Reject the output or make one bounded repair attempt.
Policy violation	Stop before persistence or tool execution.
Persistence failure	Do not report success; retry only through an idempotent operation.
Telemetry outage	Continue only if telemetry is not part of an audit requirement.

The correct response depends on the use case. Make those decisions before production traffic makes them for you.

Quality, latency, and cost are one operating envelope

AI features do not have one definition of success.

A response can be accurate but too slow. It can be fast but insufficiently grounded. It can satisfy the user while consuming an unacceptable number of tokens. It can pass an offline evaluation while failing under production concurrency.

Define an operating envelope for the complete feature. For a retrieval-backed support reply, that means an acceptable answer quality and retrieval miss rate, a maximum useful latency, a token or cost budget, and strict limits on tool calls or side effects. It also means deciding what happens after a timeout, which fallback is acceptable, and what data the application retains in telemetry.

These constraints interact. Adding more context may improve one evaluation score while increasing latency and cost. Adding retries may improve completion rate while amplifying throttling. A fallback model may improve availability while changing behavior.

Treat those as system tradeoffs, not model settings.

Evaluation and observability answer different questions

Evaluation asks whether the system behaves well enough across representative cases.

Observability asks what happened during a specific execution in a running environment.

You need both.

An evaluation set can detect that a prompt change reduced groundedness for refund questions. A trace can show that one production request spent six seconds in retrieval, made two model calls, invoked a tool, and then failed during persistence.

Logs and traces do not prove answer quality, and a quality score does not explain a production timeout. Unit tests cover deterministic application behavior, but they do not measure nondeterministic result quality. One successful manual prompt establishes neither quality nor reliability.

Keep deterministic application tests around authorization, routing, parsing, policy, and state changes. Add evaluations for model and retrieval behavior. Add telemetry across the boundaries that operators must diagnose.

Define production readiness at the feature level

Before calling an AI feature production-ready, I would want explicit answers to these questions:

Behavior

Which user outcome does the feature produce?
Which examples show acceptable and unacceptable behavior?

Boundaries

Where are authentication and authorization enforced?
Which inputs and outputs are treated as untrusted?
Which actions require deterministic policy or human approval?

Runtime

What are the time, token, cost, retry, and tool-call budgets?
Can the caller cancel the work, and should it stay in the request, stream, or become a background job?

Failure handling

Which failures are retryable?
What does the user see when a dependency is unavailable?
Can partial work be resumed without repeating a side effect?

Operations

Can traces separate retrieval, model, tool, and persistence time?
Are sensitive inputs and outputs excluded or deliberately governed?
Can operators detect degraded quality, latency, cost, or error rate?

Change control

How are model, prompt, retrieval, and tool changes evaluated before release?
Can the application roll back without changing stored data contracts?
Are provider-specific capabilities explicit, or are they hidden behind an abstraction that pretends all models behave the same?

If these questions do not have answers, the system may still be a useful prototype. It is not ready to be operated as a dependable product feature.

When this framing is useful

This framing matters whenever an AI result affects a user workflow, accesses private data, depends on retrieval, invokes tools, changes state, has meaningful cost, or creates an operational promise. At that point, the model call is no longer the architecture. The complete request path is.

When a direct model call is enough

A small direct call is reasonable for a local experiment, an internal spike, or a low-risk disposable tool where failure has little consequence and no sensitive data or side effects are involved.

Do not build a platform before you have a use case. Once the feature creates a real product promise, stop treating the model call as the architecture.

Practical takeaway

Start with one important AI request in your application and draw its complete path.

Mark every dependency, policy decision, data boundary, side effect, and operational signal. Then define what should happen when each part is slow, unavailable, or wrong.

The model call will probably remain one of the smallest boxes in the diagram.

Sources

Microsoft Learn: AI workloads on Azure
Microsoft Learn: Design principles for AI workloads on Azure
Microsoft Learn: Application design for AI workloads on Azure

Microsoft Foundry: When Agent Framework Agents Become Managed Resources

Lukas Walter — Wed, 29 Jul 2026 15:30:00 +0000

This is Part 19 of my series on the Microsoft Agent Framework. You can read the original post over on lukaswalter.dev.

In the previous article, we treated an agent as an application with testable boundaries: model calls, tools, structured output, routing, workflows, approvals, and failure handling.

The testing approach assumes the agent belongs to your application. C# code creates it, the process owns the session, your application stores the conversation, and your deployment platform runs the service.

Foundry gives you another place to put some of that responsibility. The agent can become a managed cloud resource with its own identity, endpoint, version, conversation state, access policy, and operational lifecycle. C# may still define its behavior, but the agent now exists outside your application process as well.

The ownership decision comes before the hosting details. You can keep the agent inside your application and call Foundry services, deploy the Agent Framework code as a hosted agent, or let Foundry own more of the definition through a prompt agent. Each choice puts the runtime, state, identity, and business policy in a different place.

Foundry does more than expose another model endpoint. Its control plane can handle enterprise identity, RBAC, governance, deployment, observability, and persistent state. You pay for that convenience with more resource boundaries, permissions, lifecycle decisions, and a tighter dependency on the platform.

My rule is simple:

Keep agent behavior and business policy in code when they need normal software engineering control.
Use Foundry when the agent itself needs managed identity, durable state, controlled sharing, centralized governance, or a managed runtime.

Choose the ownership model first

Start by separating model access, agent runtime, conversation state, identity, and business authorization. Choosing Foundry for one of them does not force you to hand over the others.

The current Foundry Agent Service documentation describes two main agent types: prompt agents and hosted agents. There is also a useful third deployment choice for Agent Framework applications: keep the code in your own process and call the Foundry Responses API without creating a managed agent resource.

Approach	Where the main behavior lives	What Foundry manages	What your application still manages
Prompt agent	Instructions, model selection, and tools configured through the portal, SDK, or REST API	Runtime, endpoint, scaling, identity, conversations, tools, and lifecycle	Product integration, end-user authorization, and domain policy
Hosted agent	Your Agent Framework code, packaged as a container or source bundle	Hosted runtime, endpoint, scaling, dedicated agent identity, session persistence, observability, and lifecycle	Agent logic, tools, business policy, tests, and the release artifact
In-process Agent Framework agent using Foundry	Your web app, worker, or service	Models, platform tools, and the project-scoped Responses API	Hosting, scaling, application endpoint, state mapping, and deployment lifecycle

Foundry visual workflows are a separate orchestration capability, not a third agent type alongside prompt and hosted agents. Microsoft has announced that the visual workflow experience will retire on December 1, 2026 and recommends Microsoft Agent Framework for new workflow implementations. I treat workflows as migration context here, not as another ownership model.

An application may need Foundry model and tool access while keeping the runtime in its own process. Create a managed agent resource only when you want Foundry to operate that runtime.

A local Agent Framework agent starts as application state

In the local version, the application creates the AIAgent, creates or restores the AgentSession, invokes the model through IChatClient, and owns the surrounding endpoint, storage, authorization, and observability.

Concern	Local Agent Framework application
Agent definition	C# code, configuration, and registered tools
Agent lifetime	The process creates and uses the agent
Session lifetime	In memory by default, or stored by an application-owned provider
Conversation identity	An application concept such as a user or chat ID
Authentication	The application authenticates to the model and downstream services
Authorization	Application code and tool policy decide what the user may do
Deployment	Your web app, worker, container, or job platform
Versioning	Source control, package versions, and your deployment pipeline
Observability	Your OpenTelemetry, logs, and monitoring setup

This keeps the agent close to the code that owns its business rules. Your team also owns state, user mapping, deployment, scaling, endpoint security, versioning, and observability.

Foundry adds an agent control plane

Foundry puts resource boundaries above the running agent. The request path then looks roughly like this:

Application-owned boundary
--------------------------
User interface or API
        |
        v
Application API
- authentication
- end-user authorization
- tenant policy
- conversation mapping
        |
        v

Foundry-managed control plane
-----------------------------
Foundry project endpoint
        |
        v
Prompt agent or hosted Agent Framework runtime
        |
        +--> model deployment
        +--> tools and connections
        +--> conversation or session state
        +--> traces, metrics, versions, and lifecycle metadata

The new URL is the least interesting part. What matters is that the agent now has a life outside one application process.

Foundry can provide:

a project boundary for agents, files, evaluations, and related assets
managed hosting and scaling
a dedicated hosted-agent identity and a separate project identity
RBAC for creating, managing, and invoking agents
versioned agent endpoints and optional publishing or distribution lifecycle
managed conversation or session state
tracing and monitoring integrations
networking, storage, and encryption options for enterprise requirements

With the control plane in place, platform and security teams can manage the agent as an operational resource instead of finding it buried inside an application deployment.

Hosted agents keep the code, but move the runtime

For an existing Agent Framework application, a hosted agent is usually the shortest path into Foundry. Your orchestration, tools, approval logic, and workflow can remain in C#. Foundry takes over the deployment and runtime around that code.

A minimal Responses host still creates an AIAgent, but exposes it through the Foundry hosting boundary:

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

AIAgent agent = new AIProjectClient(
        new Uri(builder.Configuration["FOUNDRY_PROJECT_ENDPOINT"]!),
        new DefaultAzureCredential())
    .AsAIAgent(
        model: builder.Configuration["AZURE_AI_MODEL_DEPLOYMENT_NAME"]!,
        instructions: """
        You help operators inspect deployments.
        Use read-only tools automatically.
        Request approval before production side effects.
        """,
        name: "deployment-agent",
        description: "An operations assistant for deployment checks.");

builder.Services.AddFoundryResponses(agent);

WebApplication app = builder.Build();
app.MapFoundryResponses();
app.Run();

Responses fits conversational agents because it supports streaming and message continuity through previous_response_id or a conversation ID. Invocations is better suited to non-conversational work or a custom request and response shape.

Do not confuse a hosted session with a conversation. A hosted session identifies sandbox compute and persisted filesystem state such as $HOME and /files; a Responses conversation stores message, tool call, and response history. Reusing an agent_session_id routes work to the same sandbox but does not replay prior messages. Invocations leaves conversation history to your application.

A hosted deployment also needs an azure.yaml file that declares the agent protocol, source directory, project, model deployment, and deployment mode. The current azd ai agent init flow can generate this from existing Agent Framework code.

The azd extension surface is still moving quickly, so I would treat the official hosted-agent guide as the source of truth for exact commands. The command sequence is not the architectural point. What matters is that the same C# agent now has a deployed Foundry version, a dedicated endpoint, and a runtime identity.

Treat persistent agents as compatibility context

Documentation searches may still surface Azure.AI.Agents.Persistent and the older agent, thread, message, and run model. That API is real, but it belongs to the classic Azure AI Agents surface. Microsoft has deprecated Agents (classic) and lists March 31, 2027 as its retirement date.

For new systems, choose the current Foundry Agent Service surface first: Responses API for in-process Agent Framework integrations, hosted agents when Foundry should run your Agent Framework code, and prompt agents when Foundry should own the agent definition. Use the persistent client mainly when maintaining or migrating an existing classic implementation.

A managed conversation is still an application data record

With a local AgentSession, you decide whether the session is a short-lived object, a serialized row, or a custom ChatHistoryProvider. Responses, hosted sessions, and classic persistent threads can move part of that durable record into Foundry.

The convenience is real, but state now crosses an application boundary.

The application now needs an explicit mapping such as:

application user and tenant
        |
        +--> application conversation ID
                    |
                    +--> Foundry conversation, response, or session ID

Do not treat a raw service ID as authorization. The request handler must still verify that the authenticated caller may access the mapped conversation.

A simple repository boundary keeps that decision in your application:

public sealed record ConversationBinding(
    string ApplicationConversationId,
    string TenantId,
    string UserId,
    string FoundryStateId,
    string FoundryStateKind,
    string AgentVersion);

public interface IConversationBindingStore
{
    Task<ConversationBinding?> FindAsync(
        string tenantId,
        string userId,
        string applicationConversationId,
        CancellationToken cancellationToken);

    Task SaveAsync(
        ConversationBinding binding,
        CancellationToken cancellationToken);
}

The Foundry ID and its protocol-specific kind are implementation details. Keep an application-owned record so you have somewhere to enforce tenant isolation, user ownership, agent version selection, retention decisions, and audit correlation.

Before calling managed conversation state "memory solved," answer these questions:

How long should inactive conversations remain available?
Can a user export or delete their conversation?
Can support staff inspect it, and under which role?
What happens when the agent version changes?
Which data may be stored in the managed service?

Foundry provides storage and retention building blocks. In the basic agent setup, files, conversations, and related state can use Microsoft-managed multitenant storage with logical separation. The standard agent setup lets you bring your own Azure resources for customer data, including files, conversations, and vector stores.

Your product remains responsible for retention, deletion, export, and support access policies.

RBAC introduces a second authorization plane

In a local application, developers may have repository access and an application identity may have access to the model endpoint. The application then decides what each end user can do.

Foundry adds a second access plane around the resource and project.

The current Foundry architecture separates control-plane actions from data-plane actions:

control plane: create resources, projects, model deployments, and security configuration
data plane: build agents, run evaluations, upload files, invoke agents, and work with project assets

Those permissions can be scoped at the Foundry resource and project levels.

The role names are changing too. Current documentation uses names such as Foundry User, Foundry Project Manager, Foundry Account Owner, and Foundry Agent Consumer. Older pages and existing role assignments may still show the previous names.

For that reason, a role table in an architecture document should record the role purpose and scope instead of copying a role name alone:

Actor	Typical responsibility	Important boundary
Platform administrator	Create the Foundry resource, networking, model deployments, and shared connections	Resource or subscription scope
Project manager	Configure and deploy project agents and hosted versions	Project or resource scope, depending on operation
Agent developer	Build and test agents, tools, evaluations, and prompts	Prefer project scope
Agent consumer	Invoke an agent endpoint without broad development access	Project or individual-agent scope
Application managed identity	Call the project endpoint from the application	Only the required project or agent permissions
Agent identity	Reach models, search, storage, MCP servers, or APIs	Downstream resource scope
End user	Use an agent through the product	Application authorization, not automatically Azure RBAC

Every hosted agent receives a dedicated Microsoft Entra identity and endpoint when deployed. That agent identity is the runtime identity. The project managed identity handles project-level infrastructure operations and cannot stand in for the hosted agent's identity. The agent identity still needs explicit role assignments for any external Azure resources it uses.

Use Microsoft Entra authentication when you need this granularity.
Key-based authentication does not provide the same role-level separation because the key grants broad access to the service.

Broad owner permissions are a tempting shortcut when the first setup fails. They may unblock a prototype while quietly turning every developer into a resource administrator.

The subtler mistake is treating the agent's managed identity as end-user authorization. It is not.

Suppose the agent can read an Azure AI Search index. The current user may still be allowed to see only some of its documents. User and tenant filters belong in the retrieval boundary. The same rule applies to tools that read tickets, invoices, deployments, or customer records.

Foundry RBAC controls who can manage or invoke the agent resource.
Your application still needs to control what a particular user may ask that agent to do.

Governance becomes part of the architecture

When an agent lives only in code, the repository is usually the best record of its definition. Once it becomes a managed resource, its effective behavior may also depend on:

instructions configured in the portal or resource
model and deployment selection
tool definitions and connection settings
content filters and guardrails
project-level permissions
managed identity assignments
network configuration
stored files and conversation state
deployed version, endpoint routing, and optional publishing settings

The C# class is now only part of the system. Foundry can add versioned endpoints, optional publishing and distribution, centralized monitoring, and resource-level governance. It can also introduce configuration drift.

A portal edit to instructions or tools is enough to make the repository disagree with the agent running in production.

I would treat the Foundry configuration as code or as a versioned deployment artifact wherever possible:

source code and prompt definitions
        |
        v
tests and evaluations
        |
        v
Foundry project configuration
        |
        v
deployed agent version
        |
        v
dedicated agent endpoint
        |
        v
optional publishing or distribution

For a prompt agent, I would create and update the definition through an SDK or REST API in the deployment pipeline rather than make production-only portal edits. With a hosted Agent Framework agent, the C# code remains the main behavior definition and the Foundry resource describes its runtime and deployment.

Whichever model you use, pick one authoritative layer.

Treat instructions, tools, model deployment names, connections, identity assignments, and protocol versions as release inputs. Promote tested versions through development, test, and production instead of editing the production agent casually.

Keep trace correlation across both ownership planes. A production trace should connect the application request, tenant, user, application conversation, Foundry agent version, Foundry state ID, model and tool spans, approvals, and side-effect IDs. Without that chain, Foundry may show a healthy agent run that you still cannot connect to a customer's request.

Split responsibility deliberately

Foundry can take over operations around the agent. Domain responsibility stays with your application.

Concern	Keep in application code	Foundry can manage
Agent behavior	Business rules, tool contracts, structured results, and deterministic workflow transitions	Prompt-agent configuration or the runtime around hosted code
Authorization	User, tenant, and record-level access checks	RBAC for managing and invoking Foundry resources and endpoints
Side effects	Validation, approval policy, idempotency, and audit records	Agent identity and secure access to downstream resources
Conversation data	Product conversation ID, ownership mapping, retention policy, and domain data	Response history, hosted sessions, or managed conversation storage, depending on the protocol
Quality	Deterministic tests for tools, routing, workflows, approvals, and failures	Evaluations, tracing, metrics, and managed monitoring
Deployment	Source review, release inputs, and promotion policy	Hosted runtime, scaling, endpoint, agent versions, and platform lifecycle
Governance	The authoritative definition of business behavior	Project boundaries, identities, connections, networking, encryption, and resource inventory

A prompt such as "only show records the user can access" is not an authorization system. The API or tool must enforce user, tenant, and permission filters before returning data to the agent.

The same applies to side effects. Foundry governance does not make a model-authorized deployment, refund, email, or permission change safe by itself. The application must still decide whether the action is allowed and whether it needs human approval, as described in the human-in-the-loop article.

Foundry evaluations complement deterministic tests. They do not replace them. The service conversation schema is an integration contract, not your product's domain model.

Pick the side of the table based on the operational work your team wants to own. Keep the domain responsibilities that cannot be delegated in application code.

When I would use Foundry

I would consider Foundry when:

the agent needs a managed runtime instead of another application-owned container
multiple products or teams need to invoke a shared agent
the agent needs durable conversations or managed sessions
platform teams need project boundaries and centralized RBAC
an agent needs a dedicated identity for downstream resources
security and compliance teams need network, storage, encryption, or audit controls
the organization needs versioned endpoints, optional distribution, monitoring, and an agent inventory

Hosted agents fit an existing Agent Framework application well: the orchestration stays in code and the runtime becomes a managed resource.

Managed hosting may cost more than expected. Hosted-agent runtime billing is based on CPU and memory consumed across active sessions, so concurrency and oversized sandboxes multiply the cost. An in-process agent can reuse existing application compute, while leaving scaling, isolation, and operations with your team. Compare the full operating cost of both models, including the engineering work outside the compute bill.

When I would stay local

I would keep the agent primarily in the application when:

it is a small feature used by one product
the conversation state already belongs in the product database
the agent must run locally or in multiple clouds
the application owns complex domain workflows and side effects
the agent changes frequently and needs tight source-level review
a Foundry project would add more permissions and deployment ceremony than value

Using Foundry models and platform tools through the Responses API does not force you to create a managed agent. Model access and agent lifecycle are separate decisions.

Conclusion

Microsoft Agent Framework gives you the .NET runtime abstractions. Foundry can put a managed control plane around them. Once it does, the agent definition, conversation state, runtime identity, deployed version, and operational lifecycle may no longer belong to one application team.

I would keep Agent Framework in the application when the agent should share the product's host, state, and release cycle. A hosted agent fits when the C# orchestration should remain intact but the runtime needs a managed endpoint, identity, and lifecycle. A prompt agent makes sense when Foundry should own the definition too.

None of these options lets the model decide business authorization or side-effect policy. Keep deterministic controls and application conversation IDs in code. Version the complete release input, and correlate Foundry state with the product request that created it.

All three options use code somewhere. The decision is about what happens after deployment: which team owns the agent, and where do they need to operate it?

When an AI Request Should Become a Background Job

Lukas Walter — Mon, 27 Jul 2026 15:30:00 +0000

LLM calls introduce unpredictable network latency, but latency alone does not decide the execution model. An AI request should become a background job when its result still matters after the HTTP request has ended.

A response that takes 30 seconds may still belong in an interactive streaming request. A five-second operation may already need a background job if it triggers durable work, must survive a deployment, or needs retries and duplicate handling outside the original HTTP request.

The boundary is ownership:

A synchronous request is owned by the current HTTP exchange.
A streaming request is still owned by that exchange, but returns partial output while work continues.
A background job is owned by the application after the request has been accepted.

That distinction changes the API contract, cancellation model, retry behavior, state storage, and operational responsibilities.

Start with the lifecycle decision

Before adding a queue, ask what should happen when the client disconnects or the application restarts.

Execution model	Good fit	Important limitation
Synchronous request	Short, bounded work whose result is needed immediately	The caller and server must keep the request alive
Streaming request	Interactive output where partial progress improves the experience	Streaming does not make the work durable
Background job	Work that must survive disconnects, expose status, or continue independently	Requires persistent state and an explicit job lifecycle

Keep the operation inside the request when all of these are true:

the caller needs the result immediately
the work has a bounded execution time
a disconnected caller no longer needs the result
retrying the complete request is safe
the hosting path can keep the connection open long enough

Move it to a background job when one or more of these are true:

the result must survive a browser refresh, network interruption, or deployment
the operation contains multiple model, retrieval, or tool steps
the caller needs progress or a status page
retries must happen without resubmitting the original HTTP request
the workload needs queue-based load leveling or concurrency limits
the operation creates an artifact that can be retrieved later
duplicate submissions could create duplicate cost or side effects

This is not limited to chat. Document extraction, large summarization jobs, batch classification, evaluation runs, ingestion pipelines, audio transcription, and report generation are common candidates.

Streaming is not a durability mechanism

Streaming solves a user-experience problem. It lets the caller see tokens, events, or progress before the complete result is available.

It does not answer what happens when:

the browser closes
a reverse proxy ends the connection
the application instance restarts
the client retries after losing the final response
the workflow must continue for several minutes
another device needs to retrieve the result later

A streaming operation can also be backed by a durable job, but that is a separate design. The stream then becomes one view over persisted job events rather than the only place where progress exists.

Use streaming when the interaction itself is the product. Use a job when completing and recording the work is the product.

The request-bound version is simple, but fragile

The direct version of a document-analysis endpoint is easy to write:

app.MapPost(
    "/documents/{documentId:guid}/analysis",
    async (
        Guid documentId,
        DocumentRepository documents,
        IChatClient chatClient,
        AnalysisRepository analyses,
        CancellationToken cancellationToken) =>
    {
        var document = await documents.GetAsync(
            documentId,
            cancellationToken);

        var response = await chatClient.GetResponseAsync(
            $"Analyze this document:\n\n{document.Text}",
            cancellationToken: cancellationToken);

        var analysis = await analyses.SaveAsync(
            documentId,
            response.Text,
            cancellationToken);

        return Results.Ok(analysis);
    });

This is a valid design when the caller is expected to wait and the operation can be safely repeated.

It becomes a poor fit when the analysis must outlive the request. Passing the request cancellation token through the complete operation correctly stops work when the caller leaves, but that is the opposite of durability. Ignoring the token is not a durable solution either. The work still has no stable identifier, no persisted status, and no reliable recovery path after a process restart.

The endpoint needs a different contract.

Return a job resource, not an unfinished result

For asynchronous processing, the initial request should validate and accept the work, create a stable job identifier, and return quickly.

The common HTTP shape is:

POST /documents/{documentId}/analysis-jobs
    -> 202 Accepted
    -> Location: /analysis-jobs/{jobId}
    -> Retry-After: 5

GET /analysis-jobs/{jobId}
    -> Pending | Running | Succeeded | Failed | Canceled

202 Accepted does not mean the analysis succeeded. It means the server accepted responsibility for processing it.

A minimal endpoint can make that contract explicit:

app.MapPost(
    "/documents/{documentId:guid}/analysis-jobs",
    async (
        Guid documentId,
        HttpContext httpContext,
        AnalysisJobSubmissionService submissions,
        CancellationToken cancellationToken) =>
    {
        var idempotencyKey =
            httpContext.Request.Headers["Idempotency-Key"].ToString();

        if (string.IsNullOrWhiteSpace(idempotencyKey))
        {
            return Results.BadRequest(new
            {
                error = "An Idempotency-Key header is required."
            });
        }

        var submission = await submissions.SubmitAsync(
            documentId,
            httpContext.User,
            idempotencyKey,
            cancellationToken);

        var statusUrl = $"/analysis-jobs/{submission.JobId}";
        httpContext.Response.Headers["Retry-After"] = "5";

        return Results.Accepted(statusUrl, new
        {
            submission.JobId,
            submission.Status,
            StatusUrl = statusUrl
        });
    })
    .RequireAuthorization();

The AnalysisJobSubmissionService is the important application boundary. It should:

Verify that the authenticated caller may analyze the document.
Scope the idempotency key to the caller or tenant and operation.
Bind the key to a request fingerprint.
Check whether that scoped key already belongs to an existing submission.
Return the existing job only when the stored and incoming fingerprints match.
Reject the request when the key exists with a different fingerprint.
Persist the new job as Pending.
Arrange for the job identifier to reach the worker.
Return the job to the endpoint.

The fingerprint should cover every input that changes the meaning of the work, such as the documentId, operation type, prompt or workflow version, and relevant options. Reusing the same key for a different fingerprint should produce a conflict response instead of returning an unrelated job. Enforce the scoped key with a unique database constraint so concurrent submissions cannot both create a job.

Persisting the row and publishing a queue message are two separate writes. If losing a job between them is unacceptable, use an outbox or another design that can reliably recover pending jobs. A database commit followed by an unprotected queue send leaves a failure window.

Persist the job lifecycle

The queue message should not be the only record that the job exists. Keep a persistent job resource that the API and worker can both use.

A practical job record usually contains:

job ID
owner or tenant ID
operation type
input reference
idempotency key
request fingerprint
state
creation and update timestamps
application processing attempt count
prompt or workflow version
model or deployment identifier
result reference
safe error code and message
cancellation request timestamp

Do not place raw prompts, private documents, or credentials in the queue message unless the queue is explicitly designed and governed for that data. A small message containing a job ID is easier to retry, inspect, and secure. The worker can load the authorized input from the system of record.

Use a closed set of states. For example:

public enum AnalysisJobStatus
{
    Pending,
    Running,
    Succeeded,
    Failed,
    CancellationRequested,
    Canceled
}

Terminal states should remain terminal. If a duplicate queue delivery reaches a job that is already Succeeded, Failed, or Canceled, the processor should exit without running the model again.

Keep the worker separate from the HTTP request

The worker owns execution after submission. A BackgroundService is a useful .NET hosting boundary, but the durability comes from the queue and job store, not from BackgroundService itself.

public sealed class AnalysisJobWorker(
    IAnalysisJobQueue queue,
    IServiceScopeFactory scopeFactory,
    ILogger<AnalysisJobWorker> logger)
    : BackgroundService
{
    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        await foreach (var delivery in queue.ReadAllAsync(stoppingToken))
        {
            await using var scope = scopeFactory.CreateAsyncScope();
            var processor = scope.ServiceProvider
                .GetRequiredService<AnalysisJobProcessor>();

            try
            {
                await processor.ProcessAsync(
                    delivery.JobId,
                    stoppingToken);

                await delivery.CompleteAsync(stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                using var settlementTimeout =
                    new CancellationTokenSource(TimeSpan.FromSeconds(10));

                await delivery.ReleaseAsync(settlementTimeout.Token);
                throw;
            }
            catch (Exception exception)
            {
                logger.LogError(
                    exception,
                    "Analysis job {JobId} failed",
                    delivery.JobId);

                using var settlementTimeout =
                    new CancellationTokenSource(TimeSpan.FromSeconds(10));

                await delivery.FailAsync(exception, settlementTimeout.Token);
            }
        }
    }
}

The sample uses application abstractions because the queue semantics matter. CompleteAsync acknowledges successful processing. FailAsync should map the exception to the queue's retry or dead-letter behavior. ReleaseAsync makes an interrupted delivery available again when the host shuts down. Settlement uses a short, independent timeout because the host shutdown token may already be canceled. Choose the timeout and settlement behavior for the backing broker.

The backing queue determines what those operations can guarantee:

A bounded Channel<T> can provide backpressure inside one process, but queued items disappear when that process stops.
A durable broker such as Azure Service Bus can survive restarts and support redelivery, but the processor must handle duplicate delivery.
A database-backed queue can keep job state and dispatch close together, but needs safe claiming and concurrency control.

Do not describe an in-memory queue as durable. It is appropriate only when the work is disposable or can be reconstructed from the persistent job store.

Broker delivery count and application processing attempts are related but different signals. Redelivery can happen before business processing starts, while one delivery may contain multiple internal attempts. Use broker metadata for transport diagnostics and record the processing attempts that matter to the workflow in the job store.

The worker is registered as a singleton hosted service. Creating a dependency-injection scope for each job lets the processor use scoped services such as DbContext without keeping one instance alive for the complete worker lifetime.

Make processing idempotent

There are two places where duplication can occur:

The caller retries the submission because it did not receive the first response.
The queue redelivers a job because processing failed before acknowledgement.

The submission idempotency key handles the first case. The persistent job state and an idempotent processor handle the second.

Before making an expensive model call, the processor should load the job and decide whether work is still required. Before committing a side effect, it should verify that the same effect has not already been applied.

That check must include an atomic claim. Move the job from Pending to Running with a conditional update guarded by the current state, an optimistic concurrency token such as a row version, or a time-bound lease. If the claim fails, another worker owns the job and the current delivery should exit or be released. A lease also needs an expiry and recovery rule so a crashed worker does not leave the job stuck in Running forever.

For model-only work, repeating a call may only duplicate cost. For jobs that send messages, update records, or invoke tools, repeating the job can also duplicate business effects. Those effects need their own idempotency boundary. The model should not be responsible for deciding whether an action already happened.

Separate request cancellation from job cancellation

The cancellation token on the submission endpoint belongs to the HTTP request. It is useful while validating and persisting the submission. It should not become the lifetime of the accepted job.

Once the API returns 202 Accepted, cancellation becomes an application operation. A typical API exposes something like:

POST /analysis-jobs/{jobId}/cancellation

That endpoint records CancellationRequested. The worker checks that state before expensive boundaries and passes a job-specific cancellation signal to model, retrieval, and tool calls where possible.

DELETE /analysis-jobs/{jobId} is also defensible when deleting the resource intentionally means requesting cancellation. If the job remains available and only changes state, an explicit cancellation operation makes that contract clearer.

Cancellation is cooperative. The application still needs to decide what happens to partial results and external side effects. A completed email cannot be canceled. A partially written report may need cleanup. A tool call may need compensation rather than cancellation.

Host shutdown is different again. The stoppingToken tells the worker that the process is stopping. With a durable queue, unfinished work should be released for redelivery rather than silently marked as a business failure.

Expose useful status without leaking internals

The status endpoint is part of the public contract, not an operational log viewer.

Return information the caller can act on:

{
  "jobId": "c7d8d96c-45ef-4c40-9a96-d6d12b9db4ec",
  "status": "Running",
  "createdAt": "2026-07-19T08:30:00Z",
  "lastUpdatedAt": "2026-07-19T08:30:07Z",
  "resultUrl": null,
  "error": null
}

Do not return raw exception messages, model payloads, credentials, or internal queue details. Map failures to stable error codes and safe descriptions. Keep trace IDs in the response when they help support teams correlate the public failure with internal telemetry.

Authorize every status and cancellation request against the job owner or tenant. Knowing a job ID must not be enough to read another user's inputs, results, or failure details.

Push completion without making notifications durable state

Polling the status URL is the simplest client contract, but it is not the only user experience. SignalR or WebSockets can push progress and completion events to connected clients, including events sent by a background worker.

The persisted job resource must remain the source of truth. A client can disconnect before receiving an event, reconnect after the event was sent, or miss a notification during deployment. Treat push as a fast notification path, then have the client read the status resource to confirm the current state. Do not make delivery of a SignalR or WebSocket event the durability boundary.

Common implementation mistakes

Returning `202` without a status resource

The caller needs a durable identifier and a place to observe the outcome. A bare 202 only moves uncertainty from the server to the client.

Starting `Task.Run` from the endpoint

Task.Run does not persist work, provide backpressure, survive a process restart, or create a retry contract. It is not a replacement for a job store and queue.

Using only the queue as state

Queue delivery state is not the same as business state. Persist whether the job is pending, running, complete, failed, or canceled.

Holding a database transaction open during the model call

Persist the state transition, commit it, call the remote model outside that transaction, and persist the result in a later short transaction. Do not hold database locks while waiting on an unpredictable network dependency.

Retrying every failure automatically

Retry transient transport failures deliberately. Do not retry invalid input, authorization failures, a request that exceeds the model's context limit, a non-retryable content-policy rejection, or deterministic tool errors as if they were temporary network problems.

Classify these terminal failures before applying a generic retry policy. Persist the job as Failed with a stable, safe error code. In a broker-backed design, dead-letter the delivery immediately when it represents poison input or requires operator inspection; otherwise complete the delivery after recording the terminal business failure. Either way, do not spend the full retry budget repeating a request that cannot succeed unchanged. A dead-letter queue also needs monitoring and an explicit inspect, correct, and resubmit process.

When to use a background job

Use this pattern when:

the work must continue after the request ends
the result needs a stable URL or status page
the operation is expensive enough to require queueing or concurrency control
retries and duplicate delivery need explicit handling
users may return later for the result
the workflow produces a durable artifact

When not to use a background job

Keep the normal request or streaming model when:

the interaction is short and the caller needs the immediate answer
partial output is useful and losing the stream should stop the work
the operation has no durable result or side effect
adding a job store, queue, worker, and status API would create more complexity than the workload justifies

Do not move work to a queue only because model calls feel slow. First decide whether the application needs durability, independent retries, progress, or load leveling. If it does not, a bounded synchronous or streaming request is usually easier to operate.

Practical takeaway

The decision is not "HTTP or queue." It is who owns the work after the request ends.

If the request owns the work, keep it synchronous or stream the response and propagate cancellation correctly.

If the application owns the work, persist a job, return 202 Accepted with a status URL, claim it atomically, process it through an explicit worker boundary, and make retries, cancellation, and terminal failure handling part of the application contract.

Testing Microsoft Agent Framework Applications

Lukas Walter — Fri, 24 Jul 2026 15:30:00 +0000

This is Part 18 of my series on the Microsoft Agent Framework. You can read the original post over on lukaswalter.dev.

In the previous article, we looked at observability for agents.
The main idea was to make a run visible as a chain of model calls, tool calls, approvals, and workflow events.

Testing starts from the same idea.

An agent run is not one answer string.
It is a small application flow with several boundaries:

user input
-> prompt and context
-> model request
-> model response
-> tool selection
-> tool arguments
-> tool execution
-> structured result or final answer
-> routing or workflow state

If the only test is an end-to-end prompt against a live model, all of those boundaries are mixed together.
When the test fails, you do not know whether the problem is the prompt, the model, the tool schema, the router, the workflow, or the real dependency behind the tool.

The solution is not to pretend that an LLM is deterministic.
The solution is to test each boundary at the level where it is deterministic, then add a smaller number of evaluation-style tests for behavior that genuinely depends on the model.

This post covers:

fake model clients
tool contract tests
structured output tests
routing tests
workflow tests
eval-style regression checks

The examples use xUnit-style assertions, but the testing approach does not depend on xUnit.
The snippets focus on the relevant testing boundary and omit some application-specific factory and workflow setup.

Do not start with the live model

A live model test is useful.
It is also expensive, slow, sometimes flaky, and difficult to diagnose.

That makes it a poor replacement for normal unit and integration tests.

I use a testing pyramid for agent applications:

                    evals
          realistic model and user examples

              application integration tests
       agent + tools + storage + workflow boundaries

              deterministic component tests
        fake model client, tools, schemas, routing

The bottom layer should be the largest.
It should catch ordinary programming mistakes without contacting Azure OpenAI, OpenAI, Foundry, or another provider.

The middle layer proves that the pieces work together.
It can use a fake IChatClient, an in-memory repository, and a real workflow execution.

The top layer checks behavior that cannot be completely specified with ordinary assertions:

whether an answer is useful
whether a route is appropriate for an ambiguous request
whether a summary preserves important facts
whether an agent follows a policy in realistic conversations

That last layer is closer to evaluation than to unit testing.
It belongs in the test strategy, but it should not carry the whole strategy.

Put the model behind `IChatClient`

The most important test seam is the model client.

Microsoft Agent Framework builds on the Microsoft.Extensions.AI abstractions, including IChatClient.
That means an application can construct an agent with a real provider in production and a scripted client in tests.

The fake client does not need to simulate intelligence.
It only needs to return the response that the test scenario requires and record what the application sent to it.
I prefer this small fake over mocking IChatClient with Moq or NSubstitute.
Response streams and ChatResponse to ChatResponseUpdate conversions are tedious to configure correctly, while the fake exercises the real interface and conversion path.

Here is a small scripted client:

using Microsoft.Extensions.AI;

public sealed class ScriptedChatClient : IChatClient
{
    private readonly Queue<ChatResponse> responses;

    public ScriptedChatClient(params string[] responseTexts)
    {
        responses = new Queue<ChatResponse>(
            responseTexts.Select(text =>
                new ChatResponse(
                    new ChatMessage(ChatRole.Assistant, text))));
    }

    public List<IReadOnlyList<ChatMessage>> Requests { get; } = [];

    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        IReadOnlyList<ChatMessage> request = messages.ToList();
        Requests.Add(request);

        if (responses.Count == 0)
        {
            throw new InvalidOperationException(
                "The scripted client has no response left.");
        }

        return Task.FromResult(responses.Dequeue());
    }

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        [System.Runtime.CompilerServices.EnumeratorCancellation]
        CancellationToken cancellationToken = default)
    {
        ChatResponse response = await GetResponseAsync(
            messages,
            options,
            cancellationToken);

        foreach (ChatResponseUpdate update in response.ToChatResponseUpdates())
        {
            yield return update;
        }
    }

    public object? GetService(Type serviceType, object? serviceKey = null)
        => null;

    public void Dispose()
    {
    }
}

This is intentionally simple.
It gives the test three useful capabilities:

It never calls a model provider.
It fails when the application makes more model calls than the scenario expects.
It lets the test inspect the messages and options sent to the model boundary.

For a real test project, I would usually add a few more features:

scripted responses that include usage metadata
a response factory that can inspect the request
a separate queue for streaming updates
a flag that records whether the request asked for JSON schema output
cancellation support for timeout tests

Do not make the fake client clever enough to become a second model implementation.
The point is to control the scenario, not to reproduce provider behavior.

Test the agent boundary

Once the client is injectable, the agent can be built exactly as it is in the application.

For example, this factory keeps the model client outside the test itself:

using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;

public static class SupportAgentFactory
{
    public static AIAgent Create(IChatClient chatClient)
    {
        return chatClient.AsAIAgent(
            name: "support-agent",
            instructions: """
            You are a support assistant.
            Answer using the available support tools when current data is required.
            Do not invent ticket status.
            """);
    }
}

A basic test can now verify the agent's application boundary without a network call:

[Fact]
public async Task Agent_returns_scripted_answer()
{
    var client = new ScriptedChatClient(
        "The ticket is waiting for customer information.");

    AIAgent agent = SupportAgentFactory.Create(client);

    AgentResponse response = await agent.RunAsync(
        "What is the status of ticket 42?");

    Assert.Equal(
        "The ticket is waiting for customer information.",
        response.Text);

    Assert.Contains(
        client.Requests.Single(),
        message => message.Role == ChatRole.User
            && message.Text!.Contains("ticket 42", StringComparison.OrdinalIgnoreCase));
}

This test does not prove that a real model would answer the same way.
That is not what it is trying to prove.

It proves that:

the agent can be constructed with the configured client
the input reaches the model boundary
the response is returned through the application API

That is already valuable.

You can also use the recorded requests to test prompt construction, context providers, chat reducers, or session behavior.
For example, a test can assert that a tenant id is present in the request and that a reduced history does not exceed a known message count.

Avoid asserting the entire serialized prompt unless the prompt itself is the contract.
Small instruction changes should not break every test.
Assert the important invariant instead:

Assert.Contains(
    client.Requests.Single().Select(message => message.Text),
    text => text?.Contains("tenant-42", StringComparison.Ordinal) == true);

Tool tests have two different jobs

A tool has at least two contracts:

The C# contract that executes the function.
The AI contract that describes the function to the model.

Test both.

Test the function as normal application code

The function itself should have ordinary unit tests.
It should not require an LLM just because it is exposed as an AIFunction.

public sealed record TicketStatus(
    int TicketId,
    string State,
    bool NeedsCustomerReply);

public sealed class TicketService
{
    public Task<TicketStatus> GetStatusAsync(
        int ticketId,
        CancellationToken cancellationToken = default)
    {
        return Task.FromResult(
            new TicketStatus(
                TicketId: ticketId,
                State: "WaitingForCustomer",
                NeedsCustomerReply: true));
    }
}

Test the service directly:

[Fact]
public async Task Ticket_service_returns_current_status()
{
    var service = new TicketService();

    TicketStatus result = await service.GetStatusAsync(42);

    Assert.Equal(42, result.TicketId);
    Assert.Equal("WaitingForCustomer", result.State);
    Assert.True(result.NeedsCustomerReply);
}

This catches business logic errors, authorization mistakes, invalid mappings, and dependency failures.
It is much easier to diagnose than an end-to-end test that only says the agent gave a bad answer.

Test the generated AI contract

Now expose the method as a function.
Keep the service as trusted instance state rather than adding it to the model-controlled argument list:

using System.ComponentModel;
using Microsoft.Extensions.AI;

public sealed class SupportTools(TicketService service)
{
    [Description("Gets the current status of a support ticket.")]
    public Task<TicketStatus> GetTicketStatusAsync(
        [Description("The numeric support ticket identifier.")]
        int ticketId,
        CancellationToken cancellationToken = default)
    {
        return service.GetStatusAsync(ticketId, cancellationToken);
    }
}

AIFunction tool = AIFunctionFactory.Create(
    (Func<int, CancellationToken, Task<TicketStatus>>)
        new SupportTools(new TicketService()).GetTicketStatusAsync,
    new AIFunctionFactoryOptions
    {
        Name = "get_ticket_status"
    });

The test should create the same AIFunction that the agent receives and inspect its declaration.
The TicketService instance is stored by SupportTools and captured by the delegate, so it does not appear in the AI-facing JSON schema.

[Fact]
public void Ticket_tool_exposes_a_narrow_contract()
{
    AIFunction tool = CreateTicketTool();

    Assert.Equal("get_ticket_status", tool.Name);
    Assert.Contains("support ticket", tool.Description,
        StringComparison.OrdinalIgnoreCase);
    Assert.Contains("ticketId", tool.JsonSchema.GetProperty("properties").EnumerateObject()
        .Select(property => property.Name));
    Assert.NotNull(tool.ReturnJsonSchema);
}

The important assertions are not necessarily the exact generated JSON string.
They are the properties that affect model behavior:

the tool has the intended name
the description says what it actually does
required parameters are present
parameters have the intended types
internal services are not exposed as model parameters
the result schema is present when the model needs it

The generated JSON schema is part of the prompt boundary.
A rename, a missing Description, or an accidentally exposed service parameter can change tool selection even when the C# function still compiles.

Test side effects separately from tool selection

A tool contract test does not prove that an agent will select the tool.
It proves that the tool is described correctly.

That distinction matters for side effects.

For a tool that sends an email, deletes data, deploys a release, or charges a card, add tests for:

authorization
validation of every argument
idempotency behavior
approval handling
cancellation
duplicate calls
failure and retry behavior

Then test that the tool is wrapped with the expected approval boundary.
The model can request the function, but the function should not execute just because the model requested it.

Structured output needs shape tests and meaning tests

Structured output gives application code a type instead of an unstructured string.
It does not make the model's values correct.

That gives us two test categories.

Test successful deserialization

The Microsoft.Extensions.AI structured-output extensions can request a typed ChatResponse<T> from an IChatClient.
With a fake client, the response can be controlled as JSON:

public sealed record IntentResult(
    string Intent,
    double Confidence,
    string Reason);

[Fact]
public async Task Structured_output_is_deserialized_into_the_expected_type()
{
    var client = new ScriptedChatClient(
        """
        {
          "intent": "billing",
          "confidence": 0.91,
          "reason": "The user asks about an invoice."
        }
        """);

    ChatResponse<IntentResult> response =
        await client.GetResponseAsync<IntentResult>(
            "Classify this request.",
            useJsonSchemaResponseFormat: false);

    IntentResult result = response.Result;

    Assert.Equal("billing", result.Intent);
    Assert.Equal(0.91, result.Confidence, precision: 2);
}

The useJsonSchemaResponseFormat: false argument keeps this test focused on deserialization.
A separate integration test can verify that the provider supports native JSON schema response formatting.

Also test malformed output:

[Fact]
public async Task Structured_output_test_fixture_can_expose_invalid_model_output()
{
    var client = new ScriptedChatClient(
        "{ \"intent\": \"billing\", \"confidence\": \"certain\" }");

    ChatResponse<IntentResult> response =
        await client.GetResponseAsync<IntentResult>(
            "Classify this request.",
            useJsonSchemaResponseFormat: false);

    Assert.False(response.TryGetResult(out _));
}

A model is not guaranteed to honor the requested schema.
Your application still needs a policy for a response that cannot be parsed:

retry with a repair request
return a controlled error
route to a human
use a safe fallback

The test should prove that the policy is applied.

Test business validation after deserialization

Do not stop at response.Result.

For the router, a valid IntentResult can still be unsafe:

var result = new IntentResult(
    Intent: "delete-production-data",
    Confidence: 0.99,
    Reason: "The user wants cleanup.");

The JSON is valid.
The intent is not one of the allowed routes.

Keep the next validation step in normal C#:

public enum UserIntent
{
    Support,
    Billing,
    Other
}

public sealed record ValidatedRoute(
    UserIntent Intent,
    double Confidence);

public static ValidatedRoute ValidateRoute(IntentResult result)
{
    if (!Enum.TryParse<UserIntent>(
            result.Intent,
            ignoreCase: true,
            out UserIntent intent))
    {
        return new ValidatedRoute(UserIntent.Other, 0);
    }

    double confidence = Math.Clamp(result.Confidence, 0, 1);
    return new ValidatedRoute(intent, confidence);
}

Test invalid enum values, missing fields, out-of-range confidence, and a low-confidence result.
These tests protect the application boundary even when the model produces a structurally valid object.

Routing tests should mostly be ordinary C# tests

The manual routing article used a small intent agent followed by a C# switch statement.
That division makes routing easier to test.

The model classifies.
The application routes.

The route function can be tested with a table of known decisions:

[Theory]
[InlineData(UserIntent.Support, "support-agent")]
[InlineData(UserIntent.Billing, "billing-agent")]
[InlineData(UserIntent.Other, "fallback")]
public void Route_uses_the_application_mapping(
    UserIntent intent,
    string expectedAgent)
{
    string actualAgent = intent switch
    {
        UserIntent.Support => "support-agent",
        UserIntent.Billing => "billing-agent",
        _ => "fallback"
    };

    Assert.Equal(expectedAgent, actualAgent);
}

The real application method might return a delegate, an agent, or a command object.
The testing principle stays the same: once the model has produced a validated route, routing should be deterministic.

You should also test route boundaries:

low confidence asks for clarification
unknown intent uses the safe fallback
a restricted tenant cannot select a privileged agent
a request requiring approval cannot bypass the approval path
the specialist receives the original user input and the minimum necessary context
the fallback does not accidentally invoke an expensive specialist

Then test the model-based classifier separately with scripted responses:

[Theory]
[InlineData("Where is my invoice?", "billing")]
[InlineData("The login link is broken.", "support")]
public async Task Classifier_uses_the_typed_model_result(
    string userMessage,
    string expectedIntent)
{
    var client = new ScriptedChatClient(
        $$"""
        {
          "intent": "{{expectedIntent}}",
          "confidence": 0.95,
          "reason": "Test fixture"
        }
        """);

    AIAgent classifier = CreateClassifier(client);
    AgentResponse<IntentResult> response =
        await classifier.RunAsync<IntentResult>(userMessage);

    Assert.Equal(expectedIntent, response.Result.Intent);
}

This checks the classifier integration.
It does not claim that the model will classify every real user message correctly.
That is what the eval set is for.

Workflow tests should assert events and outputs

A workflow is more than its final output.
The previous workflow articles covered explicit executors, edges, events, and human-in-the-loop pauses.
Those are all testable.

For a simple workflow, run it in-process and inspect the emitted events:

using Microsoft.Agents.AI.Workflows;

[Fact]
public async Task Workflow_emits_the_expected_steps_and_output()
{
    Workflow workflow = CreateReviewWorkflow(
        chatClient: new ScriptedChatClient(
            "The document is safe to publish.",
            "The final review is approved."));

    await using Run run = await InProcessExecution.Lockstep.RunAsync(
        workflow,
        new ReviewRequest("document-42"));

    Assert.Contains(
        run.OutgoingEvents,
        eventItem => eventItem is WorkflowOutputEvent output
            && output.Data?.ToString() == "The final review is approved.");

    Assert.DoesNotContain(
        run.OutgoingEvents,
        eventItem => eventItem is WorkflowErrorEvent);
}

The exact event types and workflow output depend on the workflow.
The useful assertions are usually:

the expected executor ran
the output has the expected type
a branch was or was not taken
an error event was emitted when a step failed
the workflow stopped at an approval request
resuming with the response continues from the pause instead of restarting everything

Lockstep execution is useful for tests because it makes event ordering deterministic.
That is especially helpful when the production workflow uses streaming execution and events arrive while the workflow is running.

For a workflow with a branch, assert the branch rather than only the final text:

Assert.Contains(
    run.OutgoingEvents,
    eventItem => eventItem is ExecutorCompletedEvent completed
        && completed.ExecutorId == "security-review");

Assert.DoesNotContain(
    run.OutgoingEvents,
    eventItem => eventItem is ExecutorCompletedEvent completed
        && completed.ExecutorId == "publish-document");

That test says something meaningful about the process.
An assertion such as Assert.Contains("safe", finalText) would be much weaker.

Test pauses and resumptions

A human approval workflow should have at least three tests:

The workflow requests approval before the side effect.
Rejection ends the flow without executing the side effect.
Approval resumes the flow and executes only after the decision.

Keep the side effect behind a spy or fake service:

public sealed class RecordingPublisher : IPublisher
{
    public List<string> PublishedDocuments { get; } = [];

    public Task PublishAsync(
        string documentId,
        CancellationToken cancellationToken = default)
    {
        PublishedDocuments.Add(documentId);
        return Task.CompletedTask;
    }
}

Then assert the important safety property:

Assert.Empty(publisher.PublishedDocuments);

after the rejected approval.

The model's text is not the approval boundary.
The workflow event and the application response are.

Eval-style regression checks are a separate layer

Some behavior cannot be tested with exact string assertions.

Suppose the support agent should:

answer from the ticket data
avoid inventing a resolution
ask for missing information
use the billing route for invoices
never expose another tenant's data

There may be many acceptable answers.
An exact string assertion would reject good changes and accept bad ones that happen to match the fixture.

Agent Framework now includes an evaluation framework for this layer.
It provides local evaluators for fast checks and integrations with model-based evaluators when you need a semantic quality judgment.

For this kind of behavior, create a small evaluation set and run it through the built-in APIs.

For example, a local evaluator can check required words and tool usage without another model call:

using Microsoft.Agents.AI;

var local = new LocalEvaluator(
    EvalChecks.KeywordCheck("ticket"),
    EvalChecks.ToolCalledCheck("get_ticket_status"));

AgentEvaluationResults results = await agent.EvaluateAsync(
    ["What is the current status of ticket 42?"],
    local);

results.AssertAllPassed();

For quality dimensions such as relevance, coherence, groundedness, or task adherence, use a configured evaluator such as the Azure AI Foundry evaluator or a Microsoft.Extensions.AI.Evaluation evaluator.
Those checks are still regression tests, but they have model and evaluator variance, so they should usually run in a separate CI or release-evaluation stage.

The framework can also evaluate workflows and report per-agent results, which is useful when the final answer looks acceptable but one specialist or route has degraded.

The evaluation set remains an application responsibility.
The framework provides the execution and result types, but you still need to define representative cases, expected routes, expected tool calls, safety constraints, and acceptable quality thresholds.

public sealed record AgentCase(
    string Name,
    string Input,
    string ExpectedRoute,
    string[] RequiredFacts,
    string[] ForbiddenPhrases);

static readonly AgentCase[] Cases =
[
    new(
        Name: "invoice question",
        Input: "Why was I charged twice for invoice 42?",
        ExpectedRoute: "billing",
        RequiredFacts: ["invoice"],
        ForbiddenPhrases: ["I refunded your payment"]),

    new(
        Name: "missing ticket context",
        Input: "Can you fix my issue?",
        ExpectedRoute: "support",
        RequiredFacts: ["ticket", "more information"],
        ForbiddenPhrases: ["ticket is fixed"])
];

Each case can be run against a pinned model and evaluated with a mix of checks:

deterministic route assertions
required and forbidden content checks
structured output validation
tool-call assertions
human review for a small sample
an evaluator model for semantic criteria

The evaluator should not be the only judge.
If a deterministic check can express the requirement, use the deterministic check.

For example, the route is usually an enum and can be compared exactly.
Whether an explanation is helpful may need a rubric.

public sealed record EvaluationResult(
    string CaseName,
    bool RoutePassed,
    bool RequiredFactsPassed,
    bool ForbiddenPhrasesPassed,
    double HelpfulnessScore,
    string Notes);

Store the result with enough metadata to compare runs later:

model identifier and deployment
Agent Framework and Microsoft.Extensions.AI package versions
prompt or instruction version
tool manifest version
evaluation-set version
timestamp
token usage and latency
pass/fail result and evaluator notes

Do not compare a new run with an old run while silently changing all of those inputs.
If the model, prompt, tools, or data changed, record the change.

What belongs in a regression gate?

Not every score should block a pull request.

I would use three levels:

pull request
-> deterministic tests
-> fake-client integration tests
-> small, stable eval smoke set

nightly or release candidate
-> larger eval set
-> live provider runs
-> human review for sampled cases

A pull request gate can reasonably fail when:

structured output no longer parses
a required tool disappears from the schema
a route changes for a locked fixture
a workflow stops emitting a required event
a forbidden action is invoked
token usage exceeds a deliberately chosen budget

A nightly eval can track softer changes such as helpfulness, completeness, or groundedness.

Do not turn a noisy evaluator score into a hard gate before you understand its variance.
Otherwise developers will learn to ignore the test suite.

Test the failure paths on purpose

Agent systems have failure modes that ordinary CRUD applications do not have.
Add explicit scenarios for them.

The model fails

Script the model client to throw a provider exception or return an empty response.
Assert the application returns a controlled error and does not execute a side effect.

The model returns invalid structured output

Return malformed JSON, missing required properties, an unknown enum, or a value outside the allowed range.
Assert the retry, fallback, or escalation policy.

The model requests the wrong tool

If you can construct a provider-neutral function-call fixture, use it to prove that an unsupported tool call is rejected.
Also verify that the tool layer validates arguments instead of trusting the model to be polite.

The tool fails

Throw a timeout, an authorization exception, and a domain validation exception from the fake service.
Assert what the agent sees and whether another model iteration is allowed.

The agent loops

Configure a scripted client with repeated tool requests or repeated repair responses.
Assert that the configured iteration cap ends the run.
An agent test that can hang forever is a test suite problem.

The user cancels

Cancel the token during a model call, tool call, or workflow run.
Assert that the cancellation reaches the dependency and that the application does not report a successful side effect.

The context is too large

Use a recording client to count messages or inspect approximate input size.
This catches a disabled reducer, an over-eager retriever, or a tool that returns too much data.

These tests are not about making the model predictable.
They are about making the application predictable when the model or one of its dependencies is not.

Keep test data inside the trust boundary

Agent tests often contain realistic tickets, documents, prompts, and tool arguments.
Treat those fixtures as application data.

Do not put production conversations into a repository just because they are useful examples.
Redact names, emails, tokens, internal URLs, credentials, and customer-specific identifiers.

For live evaluation runs, decide:

which data may leave the test environment
which provider and region receives it
whether prompts and outputs are retained
who can inspect the results
how long artifacts are stored
whether the evaluator itself sees sensitive content

The Agent Framework safety guidance treats model calls, chat history, context providers, and tools as separate trust boundaries.
Your test pipeline crosses the same boundaries.

A practical test plan

When I add a new agent feature, I usually start with this checklist:

1. Test the normal C# service or tool directly.
2. Test the generated AIFunction name, description, and schemas.
3. Test the agent with a scripted IChatClient.
4. Test structured output parsing and business validation.
5. Test routing with normal C# values and low-confidence cases.
6. Test workflow outputs, branches, errors, and approval pauses.
7. Add a small eval set for behavior that exact assertions cannot express.
8. Run a larger live evaluation outside the fast unit-test loop.

This ordering gives fast feedback first.
It also keeps failures local.

If a tool contract test fails, inspect the tool declaration.
If a routing test fails, inspect the switch or policy.
If a structured output test fails, inspect parsing and validation.
If an eval fails while all deterministic tests pass, inspect the prompt, model, data, or rubric.

That is a much better debugging experience than a single red test named Agent_should_answer_correctly.

When I would use this approach

I would use this layered approach when the agent has tools or external side effects, returns structured data, uses multiple routes or workflows, needs approval handling, or is moving beyond a prototype.
I would start with the fake client and tool tests even for a small prototype because they are cheap and keep the model boundary replaceable.

When I would not overbuild it

I would not begin with a large evaluation platform while the agent has no stable behavior, no meaningful tools or routes, or no agreed definition of a good answer.
Start with a scripted IChatClient, a few failure cases, and one or two realistic examples.
Add a larger evaluation set when the behavior is stable enough for the results to mean something.

Conclusion

An agent is not an untestable black box.
The model is one nondeterministic component inside a larger .NET application, while the application still owns deterministic boundaries such as tool behavior and schemas, structured output validation, routing, workflow events, approvals, and failure policies.

Use fake model clients for fast tests, ordinary C# tests for routing and business rules, workflow assertions for process behavior, and evals for quality that needs realistic language examples.
The goal is not to make every agent answer identical text.
It is to know which parts of the system are correct, which parts are model-dependent, and what changed when a test or evaluation fails.

The next article moves from agents that live mainly in code to agents managed as cloud resources in Microsoft Foundry.
We will compare local Agent Framework development with Foundry's persistent agents and threads, RBAC, governance, lifecycle management, and the enterprise trade-offs that come with adding a cloud control plane.

Generate Useful Test Data in .NET with Bogus

Lukas Walter — Wed, 22 Jul 2026 15:30:00 +0000

Most applications need more test data than a few hand-written objects can provide. A single unit test may need one customer, but an integration test often needs hundreds of customers, orders, timestamps, statuses, and relationships that look plausible together.

Bogus is a .NET library for generating that kind of fake data. You describe how an object should be created, and Bogus generates one object or a collection of objects from those rules.

The useful distinction is this: Bogus generates data. It does not replace mocks, assertions, contract tests, or production data-quality checks. It is a tool for creating controlled input for development and testing.

What Bogus does

Bogus provides two things:

Faker<T> describes how to build instances of a type.
Data sets such as Name, Address, Internet, Commerce, Date, and Random provide reusable generators.

The main API is fluent. A rule maps a property to either a fixed value or a callback that receives a Bogus faker instance:

var faker = new Faker<Customer>()
    .RuleFor(customer => customer.FirstName, f => f.Name.FirstName())
    .RuleFor(customer => customer.LastName, f => f.Name.LastName());

Customer customer = faker.Generate();
List<Customer> customers = faker.Generate(100);

Generate() creates one object. Generate(count) creates a collection. Each call evaluates the rules again, so the generated values normally differ.

Install Bogus

At the time of writing, NuGet lists version 35.6.5; pin the version in real projects when repeatability matters, and update it deliberately when upgrading dependencies.

dotnet add package Bogus --version 35.6.5

The package is usually most useful in test projects, development-only seeders, demo applications, and tools that need a large amount of sample data. Keep it out of production application paths unless generating sample data is an explicit product feature.

Start with a simple faker

Here is a small model for the examples in this post:

public enum CustomerStatus
{
    Trial,
    Active,
    Suspended
}

public sealed class Customer
{
    public int Id { get; set; }
    public string FirstName { get; set; } = "";
    public string LastName { get; set; } = "";
    public string Email { get; set; } = "";
    public string CountryCode { get; set; } = "";
    public CustomerStatus Status { get; set; }
    public DateTime CreatedAtUtc { get; set; }
}

Now define the data rules in one place:

using Bogus;

var nextCustomerId = 1;

var customerFaker = new Faker<Customer>("en")
    .StrictMode(true)
    .RuleFor(customer => customer.Id, _ => nextCustomerId++)
    .RuleFor(customer => customer.FirstName, f => f.Name.FirstName())
    .RuleFor(customer => customer.LastName, f => f.Name.LastName())
    .RuleFor(customer => customer.Email, f => f.Internet.ExampleEmail())
    .RuleFor(customer => customer.CountryCode, f => f.Address.CountryCode())
    .RuleFor(customer => customer.Status, f => f.PickRandom<CustomerStatus>())
    .RuleFor(customer => customer.CreatedAtUtc,
        f => f.Date.PastOffset(2).UtcDateTime);

Customer customer = customerFaker.Generate();
List<Customer> customers = customerFaker.Generate(100);

The counter makes this faker stateful. Avoid sharing stateful faker instances across parallel tests. Prefer creating a fresh faker per test or per fixture operation when rules close over counters such as nextCustomerId.

The f parameter is a Faker instance. It exposes the built-in data sets and randomization helpers. For example:

f.Name.FirstName()
f.Address.CountryCode()
f.Date.PastOffset(2)
f.Random.Int(1, 10)
f.PickRandom<CustomerStatus>()

StrictMode(true) is valuable for reusable test fixtures. It makes Bogus complain when a writable property has no rule, so adding a property to the model does not silently leave the fixture incomplete. You can omit it for quick prototypes, but explicit rules make larger fixtures easier to review.

Rules can depend on the object being generated

Some values should be related to other values on the same object. Use the overload that receives both the faker and the object under construction:

var customerFaker = new Faker<Customer>("en")
    .RuleFor(customer => customer.FirstName, f => f.Name.FirstName())
    .RuleFor(customer => customer.LastName, f => f.Name.LastName())
    .RuleFor(customer => customer.Email,
        (f, customer) => f.Internet.Email(
            customer.FirstName,
            customer.LastName));

This produces an email address that matches the generated name. For test systems that must never accidentally send mail, ExampleEmail() is a safer default because it uses a reserved example domain. If the system under test needs a particular email format, prefer a reserved non-resolving domain such as test.invalid and assert that the application treats it as test data.

You can also use a custom callback for domain-specific rules:

public sealed class Subscription
{
    public string Plan { get; set; } = "";
    public int Seats { get; set; }
}

var planNames = new[] { "Starter", "Team", "Enterprise" };

var subscriptionFaker = new Faker<Subscription>()
    .RuleFor(subscription => subscription.Plan,
        f => f.PickRandom(planNames))
    .RuleFor(subscription => subscription.Seats,
        (f, subscription) => subscription.Plan switch
        {
            "Starter" => f.Random.Int(1, 3),
            "Team" => f.Random.Int(4, 25),
            _ => f.Random.Int(26, 500)
        });

The important part is that the generated values follow the rules your application cares about. A random number between 1 and 500 is not automatically realistic for every plan.

Compose related objects

Real test scenarios rarely contain isolated objects. A customer may have orders, and an order should refer to an existing customer rather than a random integer that happens to look like an ID.

Generate the parent objects first, then use them when defining the child faker:

public sealed class Order
{
    public int Id { get; set; }
    public int CustomerId { get; set; }
    public decimal Total { get; set; }
    public DateTime CreatedAtUtc { get; set; }
}

var customers = customerFaker.Generate(25);
var nextOrderId = 1;

var orderFaker = new Faker<Order>()
    .StrictMode(true)
    .RuleFor(order => order.Id, _ => nextOrderId++)
    .RuleFor(order => order.CustomerId,
        f => f.PickRandom(customers).Id)
    .RuleFor(order => order.Total,
        f => f.Finance.Amount(10, 2_000))
    .RuleFor(order => order.CreatedAtUtc,
        f => f.Date.PastOffset(2).UtcDateTime);

var orders = orderFaker.Generate(100);

This pattern is often more useful than generating a large object graph in a single expression. Each faker has one responsibility, and the relationship between the generated collections is visible in the test setup.

For a development database, the same approach can be used in a dedicated seeding step:

var customers = customerFaker.Generate(1_000);
var orders = orderFaker.Generate(5_000);

db.Customers.AddRange(customers);
db.Orders.AddRange(orders);
await db.SaveChangesAsync();

Keep this code separate from production startup. A development seeder should be deliberate, repeatable, and impossible to run against a production connection by accident.

If the database owns identity values, let EF Core generate the keys or build child records after the parent keys are known. Explicit integer IDs can conflict with identity and primary-key configuration, depending on how the model is mapped.

Make generated data repeatable

Random data is useful for finding unexpected cases, but a failing test is much easier to investigate when the same input can be generated again. Bogus supports repeatable output through a randomizer seed:

using Bogus;

Randomizer.Seed = new Random(12345);

var customers = customerFaker.Generate(10);

Use a seed when you need to reproduce a failing scenario or keep a development dataset stable while debugging. Do not make tests depend on exact generated names or on the exact order of every random value. Test the properties that matter:

For unit tests, prefer local seeding with UseSeed when you want a specific generated object to be repeatable without coupling other tests to a global random sequence:

var customer = customerFaker
    .UseSeed(12345)
    .Generate();

Local seeding controls Bogus randomness, not external state captured by your rules. If a faker closes over counters such as nextCustomerId, create a fresh faker before generating the seeded object.

var customer = customerFaker.Generate();

Assert.NotEqual(0, customer.Id);
Assert.False(string.IsNullOrWhiteSpace(customer.Email));
Assert.Matches("^[A-Z]{2}$", customer.CountryCode);

Randomizer.Seed is global state. Set it deliberately, preferably at the boundary of a test or data-generation operation, and be aware that other tests using Bogus may be affected. Also remember that values generated outside Bogus, such as Guid.NewGuid() or DateTime.UtcNow, are not made deterministic by the Bogus seed.

Even with a seed, generated values can change when you upgrade Bogus or its locale data. Avoid asserting exact fake names unless that exact value is the behavior under test. For maximum unit-test stability, stay within the same major version and treat dependency upgrades as a reason to review deterministic fixtures.

For fully stable timestamp fixtures, use a fixed date range or inject a known reference time instead of relying on a Past value relative to now.

Use custom constructors when necessary

Some domain types cannot be created with a parameterless constructor. CustomInstantiator lets the faker create the object using the constructor you need:

public sealed class Account
{
    public Account(Guid id)
    {
        Id = id;
    }

    public Guid Id { get; }
    public string Name { get; set; } = "";
}

var accountFaker = new Faker<Account>()
    .CustomInstantiator(f => new Account(f.Random.Guid()))
    .RuleFor(account => account.Name, f => f.Company.CompanyName());

var account = accountFaker.Generate();

This keeps the fixture aligned with the production construction rules instead of weakening the domain model just to make test data easier to create.

Use locales when the scenario needs them

Bogus includes locale-specific data sets. Pass a locale to the faker when names, addresses, or other generated values need to resemble a particular region:

var germanCustomerFaker = new Faker<Customer>("de")
    .RuleFor(customer => customer.FirstName, f => f.Name.FirstName())
    .RuleFor(customer => customer.LastName, f => f.Name.LastName())
    .RuleFor(customer => customer.CountryCode, _ => "DE");

Locales improve the shape of sample data, but they do not make it valid for every business or regulatory scenario. If a postal code, tax number, phone number, or address must satisfy a specific country’s rules, add an explicit domain generator and validate it independently.

When Bogus is a good fit

Bogus is a strong fit when you need:

Unit-test inputs with varied values.
Integration-test datasets for APIs, databases, or message handlers.
Seed data for local development and demos.
Pagination, filtering, sorting, and search scenarios with enough rows to expose mistakes.
Load-test input prepared before the run, where values need to look like the domain model.
Reproducible examples that should not contain real customer data.

The biggest benefit is lower fixture maintenance. Instead of copying a large JSON document into every test, you can describe the few invariants that matter and generate the rest.

It also makes edge cases easier to express. A fixture can generate a normal customer by default, then override one rule for a specific scenario:

var suspendedCustomer = CustomerFixtures.Create()
    .RuleFor(customer => customer.Status, _ => CustomerStatus.Suspended)
    .RuleFor(customer => customer.Email, _ => "suspended-user@test.invalid")
    .Generate();

This reuses the default rules from the shared fixture pattern shown below, then overrides only the values relevant to the scenario. Use fixed values for the behavior under test and randomness for the fields that are incidental.

When Bogus is not the right tool

Do not use Bogus as a substitute for the following:

A mock library. Bogus creates objects and values. It does not configure a mocked service or verify an interaction.
A contract fixture. If a test documents an exact API payload, an explicit hand-written object is often clearer than generated data.
Production customer data. Fake values can look realistic, but they are not anonymized copies of real data and may still create operational risk if sent to external systems.
A domain validator. Bogus may generate a plausible value without satisfying all of your business rules. Test the validator with intentional valid and invalid cases.
A security or payment simulator. A generated card number, account number, or identity number is not permission to call a real provider. Keep integrations behind test doubles or provider sandboxes.
A replacement for boundary cases. Random generation may never produce the exact empty, maximum, duplicate, expired, or malformed value your test needs. Add explicit examples for those cases.

Randomness also makes a poor foundation for a test that should be completely obvious to the next person reading it. If the important input is three strings and one status, write those values directly.

A practical fixture shape

For a larger test suite, keep faker definitions in a dedicated factory or fixture class instead of rebuilding them inside every test:

public static class CustomerFixtures
{
    public static Faker<Customer> Create()
    {
        var nextId = 1;

        return new Faker<Customer>("en")
            .StrictMode(true)
            .RuleFor(customer => customer.Id, _ => nextId++)
            .RuleFor(customer => customer.FirstName, f => f.Name.FirstName())
            .RuleFor(customer => customer.LastName, f => f.Name.LastName())
            .RuleFor(customer => customer.Email, f => f.Internet.ExampleEmail())
            .RuleFor(customer => customer.CountryCode, f => f.Address.CountryCode())
            .RuleFor(customer => customer.Status, f => f.PickRandom<CustomerStatus>())
            .RuleFor(customer => customer.CreatedAtUtc,
                f => f.Date.PastOffset(2).UtcDateTime);
    }
}

Tests can then start with a clear default and override only what matters:

var customer = CustomerFixtures.Create()
    .RuleFor(c => c.Status, _ => CustomerStatus.Suspended)
    .Generate();

This is a good balance between convenience and control. The shared fixture owns the defaults; the test still makes its scenario-specific assumptions visible.

Benefits and tradeoffs

The benefits are practical:

Less repetitive setup code.
More variation than a small set of hand-written fixtures.
Better-looking local development data.
Explicit relationships between generated objects.
Optional repeatability when debugging.
Reusable rules across unit tests, integration tests, and seeders.

The tradeoffs matter too:

A fixture can become difficult to understand if it contains too many hidden rules.
Random output can make failures harder to reproduce if you do not capture a seed.
Generated data may be plausible without being valid for your business domain.
Large generated graphs can make tests slow and obscure the behavior under test.
Global seeding can introduce coupling between tests.

The solution is to keep fakers small, use explicit overrides for important behavior, capture seeds when diagnosing failures, and assert invariants instead of snapshots of random output.

Conclusion

Bogus is most useful when the problem is not “how do I mock this dependency?” but “how do I create enough controlled input to exercise this code?”

Start with a Faker<T>, add rules for the fields that matter, enable StrictMode for shared fixtures, compose related objects from existing generated values, and use a seed when you need to reproduce a failure.

Use it for test data, development data, demos, and load-test inputs. Do not use it as evidence that your business rules, external integrations, or security boundaries work. Those still need explicit tests.

References

Observability for Agents with OpenTelemetry, Aspire, and Application Insights

Lukas Walter — Mon, 20 Jul 2026 15:30:00 +0000

This is Part 17 of my series on the Microsoft Agent Framework. You can read the original post over on lukaswalter.dev.

In the previous article, we looked at AG-UI and the frontend boundary for agents.
The main point was that agent behavior should not be flattened into plain chat text.
Tool calls, approvals, state updates, and cancellations are separate events.

The same rule applies in production.

Do not treat an agent run as one opaque log line.
An agent run is a chain of model calls, tool calls, retries, approvals, state changes, and external dependencies.
If you cannot see that chain, you cannot debug cost, latency, failure, or unsafe behavior.

I want the trace to look more like this:

HTTP request
-> agent run
-> model call
-> tool call
-> model call
-> approval request
-> tool execution
-> final response

From that trace, I want to answer normal production questions:

Which model was called?
How many input and output tokens were used?
Which tool ran?
How long did each step take?
Did the tool fail or retry?
Did a human approve the side effect?
Was sensitive prompt or tool data captured?
Can I inspect the same flow locally before sending it to the cloud?

OpenTelemetry is the shared telemetry format.
Aspire gives me the local loop.
Application Insights is where the same signals become searchable in Azure, including the agent-focused views in Azure Monitor.

Start with the questions

Before adding packages, decide what the telemetry must explain.

For most agent systems, I would start with these questions:

Question	Signal
Is the agent slow?	Trace duration by run, model call, tool call, dependency call
Is the agent expensive?	Token usage by agent, model, route, tenant, or feature
Is the agent failing?	Exceptions, failed tool calls, failed model calls, finish reasons
Is the agent looping?	Model call count, tool call count, max-iteration exits
Is the agent using the wrong tool?	Tool name, tool risk, tool result class, approval decision
Is sensitive data being exported?	Explicit content-capture setting, redaction policy, sampling policy
Can support find one incident?	Conversation id, run id, trace id, approval id, user-safe correlation id

Notice what is not in the table:

Log the whole prompt and hope it helps later.

This can help in a development environment.
In production, it is often the wrong default.
Prompts, tool arguments, retrieved documents, and tool results may contain customer data, secrets, internal tickets, file contents, or personally identifiable information.

Observability does not mean storing everything.
Treat it like production data, with the same access and retention discipline you would apply anywhere else.

The basic OpenTelemetry setup

Agent Framework builds on Microsoft.Extensions.AI.
That matters because Microsoft.Extensions.AI provides an OpenTelemetry middleware for chat clients.
The extension method is UseOpenTelemetry on ChatClientBuilder.

Start with the chat client:

using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

AzureOpenAIClient azureClient = new(
    new Uri(builder.Configuration["AZURE_OPENAI_ENDPOINT"]!),
    new DefaultAzureCredential());

IChatClient chatClient = azureClient
    .GetChatClient(builder.Configuration["AZURE_OPENAI_DEPLOYMENT"]!)
    .AsIChatClient()
    .AsBuilder()
    .UseOpenTelemetry(
        sourceName: builder.Environment.ApplicationName,
        configure: telemetry =>
        {
            telemetry.EnableSensitiveData =
                builder.Environment.IsDevelopment()
                && builder.Configuration.GetValue<bool>(
                    "AI:Telemetry:EnableSensitiveData");
        })
    .UseFunctionInvocation()
    .Build();

WebApplication app = builder.Build();

AIAgent agent = chatClient.AsAIAgent(
    name: "deployment-agent",
    instructions: """
    You help operators inspect deployment state and prepare safe release actions.
    Use tools when current deployment state is needed.
    Ask for approval before production changes.
    """,
    tools:
    [
        AIFunctionFactory.Create(CheckStagingStatusAsync),
        AIFunctionFactory.Create(DeployToProductionAsync)
    ],
    services: app.Services);

Two details matter here.

First, the sourceName is deliberate.
OpenTelemetry tracing only collects activity sources that the tracer provider listens to.
Aspire service defaults already add the application name as a trace source.
Using builder.Environment.ApplicationName keeps the chat client activity source aligned with the application telemetry source.

Second, sensitive data is disabled unless development configuration explicitly enables it.
By default, the OpenTelemetry chat client records metadata such as model information and token counts, but not raw prompts, raw outputs, tool arguments, or tool results.
That default is boring in the best possible way.

Note: The exact telemetry attributes follow the OpenTelemetry Generative AI semantic conventions. Those conventions are still moving, so verify attribute names against your package version before you bake them into dashboards or KQL queries.

Add the AI source to Service Defaults

If you use Aspire, your app probably has a ServiceDefaults project with a ConfigureOpenTelemetry method.
The starter template already configures ASP.NET Core, HttpClient, runtime metrics, OTLP export, and trace collection.

For agent telemetry, make sure the same source name used by UseOpenTelemetry is collected for both traces and metrics.

In the service defaults project, extend the OpenTelemetry setup:

using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

public static TBuilder ConfigureOpenTelemetry<TBuilder>(
    this TBuilder builder)
    where TBuilder : IHostApplicationBuilder
{
    builder.Logging.AddOpenTelemetry(logging =>
    {
        logging.IncludeFormattedMessage = true;
        logging.IncludeScopes = true;
    });

    builder.Services.AddOpenTelemetry()
        .WithMetrics(metrics =>
        {
            metrics
                .AddMeter(builder.Environment.ApplicationName)
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation();
        })
        .WithTracing(tracing =>
        {
            tracing
                .AddSource(builder.Environment.ApplicationName)
                .AddAspNetCoreInstrumentation(options =>
                {
                    options.Filter = context =>
                        !context.Request.Path.StartsWithSegments("/health")
                        && !context.Request.Path.StartsWithSegments("/alive");
                })
                .AddHttpClientInstrumentation();

            if (builder.Environment.IsDevelopment())
            {
                tracing.SetSampler(new AlwaysOnSampler());
            }
        });

    builder.AddOpenTelemetryExporters();

    return builder;
}

These two lines matter:

.AddMeter(builder.Environment.ApplicationName)
.AddSource(builder.Environment.ApplicationName)

The chat client creates an ActivitySource and a Meter.
If your OpenTelemetry configuration does not listen to them, the code can be instrumented and still produce nothing in the dashboard.

What the chat client gives you

The OpenTelemetry chat client is the easiest part of the setup.
It sits in the IChatClient pipeline and observes calls made through that client.

On a normal agent run, it can record model metadata such as:

operation name
requested model
response model
provider name
server address
streaming flag
finish reason
response id
input token count
output token count
cached input token count, when available
reasoning output token count, when available
operation duration
time to first streamed chunk
time between streamed chunks

With that alone, you can answer the first set of production questions.

For example:

Why did this request take 34 seconds?
-> The first model call took 1.4 seconds.
-> The search tool took 29 seconds.
-> The final model call took 2.1 seconds.

Why did this request cost more than usual?
-> The input token count jumped after retrieval.
-> The agent included eight document chunks instead of two.
-> The route used a larger model than the normal path.

Why did streaming feel broken?
-> The model call started quickly.
-> Time to first chunk was 18 seconds.
-> The delay is provider-side or prompt-side, not frontend rendering.

A trace turns "the agent was slow" into something you can actually fix.

Token usage is an operational signal

Token usage is billing data, but it is also behavior data.

High input tokens can mean:

the chat reducer is not running
retrieval returned too much context
the agent is carrying stale state
the frontend is resending too much history
tool results are too verbose
an inner agent is dumping its full conversation into the outer agent

High output tokens can mean:

the agent is overexplaining
the instructions are too broad
structured output is not constrained
the model is stuck in a repair loop
the user asked for a large artifact

The OpenTelemetry middleware records token usage when the provider returns usage information.
You can also inspect usage directly from responses when you need a local guardrail:

AgentResponse response = await agent.RunAsync(
    "Check release-2026-07-08 and tell me whether it is ready.",
    session,
    cancellationToken: cancellationToken);

if (response.Usage is { } usage)
{
    logger.LogInformation(
        "Agent run used {InputTokens} input tokens and {OutputTokens} output tokens.",
        usage.InputTokenCount,
        usage.OutputTokenCount);
}

I would not turn that into the main production telemetry path.
Use OpenTelemetry for durable traces and metrics.
Use direct response usage checks for local assertions, tests, or immediate guardrails.

For example, an integration test can fail when a supposedly small route starts consuming a huge prompt:

response.Usage?.InputTokenCount.Should().BeLessThan(8_000);

That catches prompt and retrieval regressions before they turn into a cloud bill.

Tool calls need their own spans

Model telemetry is not enough.
Many agent failures happen outside the model:

a search index is slow
an MCP server is down
a deployment API rejects a change ticket
a database query times out
a tool result is too large
approval is granted but execution fails

If your tool is just normal C#, instrument it like normal C#.

using System.Diagnostics;
using Microsoft.Extensions.DependencyInjection;

public static class AgentTelemetry
{
    public static readonly ActivitySource ActivitySource =
        new("DeploymentAgent");
}

public static async Task<DeploymentStatus> CheckStagingStatusAsync(
    string releaseId,
    IServiceProvider services,
    CancellationToken cancellationToken)
{
    using Activity? activity = AgentTelemetry.ActivitySource.StartActivity(
        "tool deployment.check_staging",
        ActivityKind.Internal);

    activity?.SetTag("agent.name", "deployment-agent");
    activity?.SetTag("ai.tool.name", "CheckStagingStatusAsync");
    activity?.SetTag("ai.tool.risk", "read_only");
    activity?.SetTag("deployment.environment", "staging");

    var deployments = services.GetRequiredService<IDeploymentService>();

    try
    {
        DeploymentStatus status = await deployments.CheckStagingStatusAsync(
            releaseId,
            cancellationToken);

        activity?.SetTag("deployment.status", status.State);
        activity?.SetStatus(ActivityStatusCode.Ok);

        return status;
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
        activity?.AddException(ex);
        throw;
    }
}

Then collect the tool source:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddSource(builder.Environment.ApplicationName)
            .AddSource("DeploymentAgent")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation();
    });

Use low-cardinality tags for dashboards.
For example, deployment.environment=production is fine.
Raw releaseId, user messages, customer names, email addresses, and ticket descriptions need more care.

When you need correlation without raw values, log a stable hash or an internal id that your support process can resolve under access control.

Approval telemetry is separate from tool telemetry

In the human-in-the-loop article, the rule was:

The model requests the action.
The application decides whether the action continues.

Telemetry should preserve that split.

The approval request, human decision, and eventual tool execution are related, but they are not the same event.

I usually want these approval fields:

approvalId
conversationId
agentRunId
traceId
toolName
toolArgumentsHash
requestedBy
approvedBy
decision
decisionReason
createdAt
decidedAt
executedAt
executionResult

An approval log might look like this:

logger.LogInformation(
    "Agent tool approval decided. ApprovalId={ApprovalId} ToolName={ToolName} Decision={Decision} TraceId={TraceId}",
    approval.Id,
    approval.ToolName,
    decision.Approved ? "approved" : "rejected",
    Activity.Current?.TraceId.ToString());

Do not log full tool arguments by default.
For production, I usually prefer:

logger.LogInformation(
    "Approval request created. ApprovalId={ApprovalId} ToolName={ToolName} ArgumentsHash={ArgumentsHash}",
    approval.Id,
    approval.ToolName,
    argumentsHash);

The user interface can show exact arguments at approval time.
That does not mean the telemetry backend should keep those arguments forever.

Sensitive data is a feature flag, not a casual toggle

The OpenTelemetry chat client has an EnableSensitiveData setting.
There is also an environment variable:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

When enabled, telemetry can include raw inputs and outputs such as:

system instructions
user messages
assistant messages
tool definitions
tool call arguments
tool call results
additional request and response properties

This helps during local debugging.
It can also become a data-retention problem in production.

I would treat it as an explicit environment policy:

.UseOpenTelemetry(
    sourceName: builder.Environment.ApplicationName,
    configure: telemetry =>
    {
        telemetry.EnableSensitiveData =
            builder.Environment.IsDevelopment()
            && builder.Configuration.GetValue<bool>(
                "AI:Telemetry:EnableSensitiveData");
    })

Make the configuration explicit:

{
  "AI": {
    "Telemetry": {
      "EnableSensitiveData": false
    }
  }
}

For production, decide this with security, privacy, and compliance input.
If you enable prompt capture, also decide:

which environments may capture it
who can query it
how long it is retained
whether sampling applies
which fields are redacted before export
whether customer data can cross region or tenant boundaries
whether tool results include secrets or confidential documents

The default should be metadata only.
Turn on content capture only when the debugging value justifies the data risk.

Redact before export when possible

OpenTelemetry processors can modify telemetry before it leaves the process.
For agent systems, this matters because sensitive content often appears in attributes, logs, or exception messages.

A minimal custom processor can remove known high-risk attributes:

using System.Diagnostics;
using OpenTelemetry;

public sealed class AgentRedactionProcessor : BaseProcessor<Activity>
{
    private static readonly HashSet<string> SensitiveTags =
    [
        "gen_ai.input.messages",
        "gen_ai.output.messages",
        "gen_ai.system.instructions",
        "gen_ai.tool.definitions"
    ];

    public override void OnEnd(Activity activity)
    {
        foreach (string tagName in SensitiveTags)
        {
            activity.SetTag(tagName, null);
        }
    }
}

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddSource(builder.Environment.ApplicationName)
            .AddProcessor(new AgentRedactionProcessor())
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation();
    });

This is just a sketch.
Real redaction usually needs to happen earlier, closer to the application boundary.
I prefer this order:

avoid collecting raw data
-> redact known sensitive fields
-> sample high-volume telemetry
-> restrict access in the telemetry backend
-> set retention intentionally

Do not rely on one processor as your entire privacy story.

Local debugging with Aspire

Aspire is a good local loop for agent observability.

With an AppHost and service defaults, you can run the agent backend locally and inspect:

resources
environment variables
logs
traces
metrics
outbound HTTP calls
failed dependencies
model-call spans
tool-call spans, if you added them

What I like about this setup is that the local flow uses OTLP.
Your app emits OpenTelemetry data.
The Aspire dashboard receives it.
You can inspect the same telemetry before sending it to Azure Monitor.

A minimal AppHost might look like this:

var builder = DistributedApplication.CreateBuilder(args);

var api = builder.AddProject<Projects.DeploymentAgent_Api>("deployment-agent-api")
    .WithEnvironment("AI__Telemetry__EnableSensitiveData", "true");

builder.Build().Run();

Then run the AppHost:

dotnet run --project src/DeploymentAgent.AppHost

In the dashboard, inspect one request end to end:

POST /chat
-> deployment-agent model call
-> CheckStagingStatusAsync tool span
-> deployment API HTTP dependency
-> deployment-agent model call
-> approval request log

This is what I want to see locally.
If you only see ASP.NET Core request spans, the app is observable as a web API but not yet observable as an agent.

If spans are missing, check the boring parts first:

UseOpenTelemetry is in the IChatClient pipeline before creating the agent.
sourceName matches an AddSource(...) entry.
The chat client's meter name is added with AddMeter(...) if you want token metrics.
Custom tool ActivitySource names are added with AddSource(...).
OTEL_EXPORTER_OTLP_ENDPOINT is set by Aspire or by your local environment.
The provider actually returns usage information if token counts are missing.

Also remember that the Aspire dashboard can display sensitive configuration and telemetry.
If you enable prompt capture locally, treat the dashboard as sensitive too.

Cloud monitoring with Application Insights

For Azure Monitor and Application Insights, the usual .NET path is the Azure Monitor OpenTelemetry distro.
For ASP.NET Core applications, install:

dotnet add package Azure.Monitor.OpenTelemetry.AspNetCore

Then enable Azure Monitor export:

using Azure.Monitor.OpenTelemetry.AspNetCore;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

if (!string.IsNullOrWhiteSpace(
        builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"]))
{
    builder.Services.AddOpenTelemetry()
        .UseAzureMonitor();
}

In production, set the connection string through configuration, not as a hard-coded value:

APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=...;IngestionEndpoint=...

The Aspire service defaults template already includes a commented Azure Monitor exporter section.
I usually wire it there so the same service defaults project controls local OTLP export and cloud export.

Once telemetry reaches Application Insights, you can use the normal views:

Application map
Transaction search
End-to-end transaction details
Failures
Performance
Logs
Metrics
Workbooks
Alerts

In a workspace-based Application Insights setup, the underlying trace and log data is queryable in the connected Log Analytics workspace.
That is where the KQL queries for custom dashboards, incident review, and agent-specific investigations usually live.

For AI systems, Application Insights also has an Agents view.
It is built around OpenTelemetry Generative AI semantics and is meant to help inspect agent runs, model usage, tool calls, errors, token usage, and cost-related behavior.

Still, keep normal application telemetry.
Agent observability should sit beside request, dependency, database, queue, cache, and runtime telemetry.
The agent is part of the application, not a separate universe.

What to alert on

Start with boring alerts.

For agents, I would usually begin with:

model call failure rate
tool call failure rate
high run duration
high model duration
high tool duration
high input token count
high output token count
repeated max-iteration exits
approval rejection spike
approval timeout spike
missing telemetry from an expected agent service

Avoid alerting on every weird model response.
That belongs in evaluation and product analytics more than infrastructure alerting.

I split it this way:

Concern	Better tool
Service is down	Application Insights alert
Model calls are failing	Application Insights alert
Tool latency is high	Application Insights alert
Token usage suddenly doubled	Metric alert or workbook
Answer quality regressed	Evaluation pipeline
Agent chose a poor plan	Trace review plus evaluation
User disliked the answer	Product analytics plus evaluation

For token spikes, tie the alert back to the meter you configured earlier with AddMeter(...).
If the OpenTelemetry pipeline exports the chat client token metrics to Azure Monitor, you can use Azure Monitor metric alerts for the fast signal and a workbook or KQL query for the investigation view.

Observability tells you what happened.
Evaluation tells you whether the behavior was good.
You need both.

Production checklist

For a production agent, I would want this before launch:

Every service has service.name or an equivalent cloud role name.
The agent has a stable name.
UseOpenTelemetry is configured on the chat client.
The chat client source is collected by tracing.
The chat client meter is collected by metrics.
Tool spans exist for important application tools.
Tool spans include low-cardinality tags such as tool name, risk, and result class.
Approval decisions are logged with correlation ids.
Raw prompt and tool content capture is disabled by default.
Any enabled content capture has a documented retention and access policy.
The Aspire dashboard shows a full local agent run.
Application Insights receives traces, metrics, and logs in the target environment.
A failed tool call can be found from a trace id.
A high-token request can be found and explained.
Alerts exist for failure rate, latency, and missing telemetry.

A decent first version is small:

UseOpenTelemetry on the chat client
AddSource and AddMeter for the AI source
custom spans around important tools
metadata-only telemetry by default
Aspire locally
Application Insights in Azure

This is enough to stop saying "the agent acted weird" and start looking at the actual execution path.

When I would use this

I would add this setup when:

the agent calls real tools
the agent runs in a web API or worker service
token cost matters
support needs to debug user incidents
approval flows need auditability
the agent has multiple model calls per request
retrieval or tool latency affects user experience
the system is moving from demo to production

The earlier you add it, the easier it is.
Retrofitting observability after a production incident usually means you first have to guess where the system is opaque.

When I would not start here

I would not start with full production telemetry when:

the agent is still a throwaway prototype
prompts contain data you are not allowed to store
no one has decided telemetry retention and access rules
the team has no baseline for normal token usage
the agent has no stable tool boundaries yet
the main problem is answer quality, not runtime behavior

In those cases, start smaller.
Log request ids, model names, token usage, and tool names locally.
Use Aspire for development.
Add cloud export once the boundaries are stable enough to monitor intentionally.

Conclusion

Agent observability is more than "turn on logs".
You need traces for the execution path, metrics for volume and cost, logs for decisions, and a clear policy for sensitive data.

For .NET agents, I usually start here:

Microsoft.Extensions.AI UseOpenTelemetry
-> Agent Framework agent
-> Aspire dashboard locally
-> Application Insights in Azure

Then fill the gaps that the model-call telemetry cannot know about:

custom tool spans
approval logs
application-specific correlation ids
redaction and retention rules
alerts that match real operational failure modes

Do not store every thought-like artifact an agent produces.
Store enough evidence to explain failures, cost, latency, and side effects without guessing.

The follow-up article moves from observing agent behavior to testing it.
I will look at how to test Microsoft Agent Framework applications without treating the model as an untestable black box: fake model clients, tool contract tests, structured output tests, routing tests, workflow tests, and eval-style regression checks.

Run Python in-process from .NET with CSnakes

Lukas Walter — Wed, 15 Jul 2026 15:30:00 +0000

So I definitely don't do everything using C# and .NET. I also use Python, but sometimes I just do not want another tiny Python service.

I want one Python library inside an otherwise normal .NET app. No container boundary. No HTTP wrapper. No extra deployment for something that is really just a function call.

CSnakes is interesting for that case. It runs Python in the same process and generates C# bindings from typed Python functions. The Python code stays in Python, but the .NET side can call it through a normal-looking method.

That makes it a good fit for quick demos, proofs of concept, internal tools, or the one feature where Python has exactly the library you need.

The shape

Install the packages:

dotnet add package CSnakes.Runtime
dotnet add package Microsoft.Extensions.Hosting

The project file shape is still small. Add the package references and include your Python files as analyzer additional files. That is how the source generator finds them:

<ItemGroup>
  <PackageReference Include="CSnakes.Runtime" Version="1.2.1" />
  <PackageReference Include="Microsoft.Extensions.Hosting" Version="10.0.9" />
</ItemGroup>

<ItemGroup>
  <AdditionalFiles Include="python/*.py" SourceItemType="Python">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </AdditionalFiles>
  <None Include="requirements.txt" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>

Then wire up Python at startup:

using CSnakes.Runtime;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var builder = Host.CreateApplicationBuilder(args);
var pythonHome = Path.Join(AppContext.BaseDirectory, "python");

builder.Services
    .WithPython()
    .WithHome(pythonHome)
    .FromRedistributable();

var app = builder.Build();
var env = app.Services.GetRequiredService<IPythonEnvironment>();

FromRedistributable() can download and use an isolated Python runtime. If your environment already standardizes on Conda or system Python, CSnakes can use those too.

A small example

Imagine a .NET app that needs one fuzzy matching function from Python. That is small enough that a separate API would feel silly.

Add a Python dependency in requirements.txt:

rapidfuzz

For a real app, pin the package version. Treat it like any other dependency you care about.

We can update our startup code to use CSnakes' uv integration to handle this automatically:

var requirements = Path.Join(AppContext.BaseDirectory, "requirements.txt");

builder.Services
    .WithPython()
    .WithHome(pythonHome)
    .FromRedistributable()
    .WithVirtualEnvironment(Path.Join(AppContext.BaseDirectory, ".venv"))
    .WithUvInstaller(requirements);

The first run may need network access because CSnakes can download the redistributable Python runtime and uv installs the packages from requirements.txt.

Now add python/text_tools.py:

from rapidfuzz import fuzz

def match_score(left: str, right: str) -> float:
    return float(fuzz.token_set_ratio(left, right))

Because the function has type hints, CSnakes can generate a typed C# wrapper. Python's match_score shows up as MatchScore in C#.

var textTools = env.TextTools();

var score = textTools.MatchScore(
    "Azure OpenAI invoice processing",
    "invoice processing with Azure AI");

Console.WriteLine(score);

The .NET code does not shell out to python. It also does not serialize an HTTP request just to call one local function.

What to keep in mind

Keep the Python boundary boring.

I would expose a few typed functions like match_score, extract_features, or rank_candidates. I would not let half the domain model move into Python because it was convenient in the moment.

Also remember that this is in-process. That is the point, but it changes the failure model. A Python dependency problem is now your application startup problem. A heavy Python call can affect your .NET process. The Python environment needs the same care as the rest of your runtime.

Deployment is the other boundary to check early. CSnakes can work with Native AOT when you use the source-generated bindings shown above, but not every interop path is AOT friendly. Manual Python binding still runs into the usual AOT limits around reflection and dynamic type handling. A self-contained .NET publish also does not magically include Python, your packages, or the .venv directory. Those still need to be bundled with the app.

When to use it

I would reach for CSnakes when:

the .NET app only needs a small slice of Python
HTTP would add more ceremony than value
the Python function has clear typed inputs and outputs
the integration belongs inside the app, not beside it

I would still use a separate Python service when the Python side has its own scaling needs, GPU runtime, deployment lifecycle, or team ownership. In-process interop is useful, but it is not a replacement for a real service boundary when you actually need one.

Agent Frontends with AG-UI: Streaming, Tool Calls and Human Approval

Lukas Walter — Mon, 13 Jul 2026 15:30:00 +0000

This is Part 16 of my series on the Microsoft Agent Framework. You can read the original post over on lukaswalter.dev.

In the previous article, we looked at human-in-the-loop agents.
The main point was that approval is a system boundary.
The model can request an action, but the application must decide whether that action is allowed to run.

That immediately creates a frontend problem.

A real agent UI cannot only show a chat transcript.
It also has to show:

streamed text while the agent is still answering
tool calls before and after execution
long-running tool progress
approval requests with exact arguments
state updates that change the visible application
errors, retries, cancellations, and run status

AG-UI exists for this layer.
It gives agent backends and frontends a protocol for exposing agent behavior as events instead of forcing everything through plain assistant text.

A simple rule is:

Use chat for language. Use events for agent behavior.

Plain chat is the wrong frontend boundary

A text-only chat box is fine for a simple assistant.
The user sends a message.
The assistant streams a response.
The interaction ends.

That model breaks down once the agent can do real application work.

Imagine a deployment assistant:

User: Deploy release-2026-06-24 to production using CHG-1042.

The frontend should not wait silently until the final answer arrives.
It should be able to show:

Run started
Checking staging status...
Tool call: CheckStagingStatusAsync(release-2026-06-24)
Tool result: staging healthy
Approval required: DeployToProductionAsync(...)
User approved
Deploying...
Deployment complete
Run finished

Those are not all assistant messages.
Some are lifecycle events.
Some are tool events.
Some are approval requests.
Some are state changes.

If you flatten all of that into text, the frontend loses structure.
It cannot reliably render progress, approval forms, state panels, or tool-specific UI.
It has to parse generated prose, which is exactly the kind of boundary we have avoided throughout this series.

What AG-UI adds

AG-UI is a protocol for agent user interfaces.
In Agent Framework, the integration connects an agent backend to web clients over HTTP with Server-Sent Events.

The useful part is not only "streaming text".
We already had RunStreamingAsync for that.

The useful part is that the stream can carry richer agent events:

run lifecycle
text message chunks
tool calls
tool results
approval requests
state snapshots
state deltas
errors

That gives the frontend enough structure to render the agent as an application workflow instead of a typing indicator with a text box.

The Agent Framework AG-UI documentation currently describes support for several protocol features, including agentic chat, backend tool rendering, human-in-the-loop approval, generative UI, shared state, and predictive state updates.
The protocol is still evolving, so treat implementation details as something to verify against the current package version before shipping.

The backend shape in .NET

For a .NET application, the basic backend shape is an ASP.NET Core endpoint.

Install the AG-UI hosting package:

dotnet add package Microsoft.Agents.AI.Hosting.AGUI.AspNetCore --prerelease

Then expose an Agent Framework agent through MapAGUI.

using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Hosting.AGUI.AspNetCore;
using Microsoft.Extensions.AI;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.Services.AddAGUI();

AzureOpenAIClient azureClient = new(
    new Uri(builder.Configuration["AZURE_OPENAI_ENDPOINT"]!),
    new DefaultAzureCredential());

IChatClient chatClient = azureClient.AsChatClient("gpt-5-mini");

AIAgent agent = chatClient.AsAIAgent(
    name: "deployment-agent",
    instructions: """
    You help operators inspect deployment state and prepare safe release actions.
    Explain tool usage clearly.
    Ask for approval before risky actions.
    """);

WebApplication app = builder.Build();

app.MapAGUI("/", agent);

await app.RunAsync();

AddAGUI registers the AG-UI services.
MapAGUI exposes the agent through an HTTP endpoint and streams responses using Server-Sent Events.

That endpoint is now the frontend boundary.
The browser does not need to know the provider SDK.
It talks to an AG-UI endpoint.

Conceptually:

React or web client
-> HTTP POST with messages
-> ASP.NET Core MapAGUI endpoint
-> Agent Framework agent
-> model, tools, approvals, state
-> Server-Sent Events back to the client

A .NET client can consume it as an agent

Agent Framework also has an AG-UI client package:

dotnet add package Microsoft.Agents.AI.AGUI --prerelease
dotnet add package Microsoft.Agents.AI --prerelease

The client side can create an AGUIChatClient, convert it to an agent, and consume streaming updates.

using Microsoft.Agents.AI;
using Microsoft.Agents.AI.AGUI;
using Microsoft.Extensions.AI;

string serverUrl =
    Environment.GetEnvironmentVariable("AGUI_SERVER_URL")
    ?? "http://localhost:8888";

using HttpClient httpClient = new();

AGUIChatClient aguiClient = new(httpClient, serverUrl);

AIAgent remoteAgent = aguiClient.AsAIAgent(
    name: "agui-client",
    description: "Client-side proxy for the AG-UI agent endpoint.");

AgentSession session = await remoteAgent.CreateSessionAsync();

List<ChatMessage> messages =
[
    new(ChatRole.User, "Check release-2026-06-24 and tell me whether it is ready.")
];

await foreach (AgentResponseUpdate update in remoteAgent.RunStreamingAsync(
                   messages,
                   session))
{
    foreach (AIContent content in update.Contents)
    {
        if (content is TextContent text)
        {
            Console.Write(text.Text);
        }
    }
}

This is useful for console demos or service-to-service tests.
For a real web frontend, you might use a React client such as CopilotKit or your own AG-UI event consumer.

The important design point is the same either way:

Do not treat the endpoint as "an API that returns a string".
It is an event stream.

Stream text, but do not stop there

Streaming text is the minimum viable agent frontend feature.

It solves the waiting problem.
The user sees output while the model is still generating.

But once tools are involved, text streaming alone is not enough.
The frontend also needs to understand the run lifecycle.

At minimum, keep separate UI state for:

public enum AgentRunStatus
{
    Idle,
    Running,
    WaitingForApproval,
    RunningTool,
    Completed,
    Failed,
    Cancelled
}

Then update it from events rather than from generated text.

For example:

public sealed record AgentRunViewModel
{
    public AgentRunStatus Status { get; init; }
    public string? ConversationId { get; init; }
    public string? RunId { get; init; }
    public string AssistantText { get; init; } = string.Empty;
    public List<ToolCallViewModel> ToolCalls { get; init; } = [];
    public ApprovalRequestViewModel? PendingApproval { get; init; }
}

That separation matters.

The assistant text is content.
The run status is application state.
Tool calls are structured events.
Approvals are interactive decisions.

If everything is appended to one markdown string, the frontend cannot behave like a product UI.

Show backend tool progress

In AG-UI backend tool rendering, tools are defined and executed on the server.
The frontend receives tool call and tool result updates.

That is a good fit for actions that belong inside your application boundary:

search internal docs
check deployment status
calculate a quote
query inventory
create a support ticket
call an authorized backend service

A simple server-side tool looks like this:

using System.ComponentModel;
using Microsoft.Extensions.DependencyInjection;

[Description("Checks current deployment status for a release.")]
public static Task<DeploymentStatus> CheckDeploymentStatusAsync(
    [Description("Release identifier, for example release-2026-06-24.")]
    string releaseId,
    IServiceProvider services,
    CancellationToken cancellationToken)
{
    var deployments = services.GetRequiredService<IDeploymentService>();

    return deployments.CheckStatusAsync(
        releaseId,
        cancellationToken);
}

AITool[] tools =
[
    AIFunctionFactory.Create(CheckDeploymentStatusAsync)
];

AIAgent agent = chatClient.AsAIAgent(
    name: "deployment-agent",
    instructions: "Use tools when deployment state is needed.",
    tools: tools,
    services: app.Services);

app.MapAGUI("/", agent);

On the client side, handle tool call and tool result content separately from text.

await foreach (AgentResponseUpdate update in remoteAgent.RunStreamingAsync(
                   messages,
                   session,
                   cancellationToken: cancellationToken))
{
    foreach (AIContent content in update.Contents)
    {
        switch (content)
        {
            case TextContent text:
                AppendAssistantText(text.Text);
                break;

            case FunctionCallContent call:
                MarkToolStarted(
                    call.CallId,
                    call.Name,
                    call.Arguments);
                break;

            case FunctionResultContent result:
                MarkToolFinished(
                    result.CallId,
                    result.Result,
                    result.Exception);
                break;
        }
    }
}

The actual UI can then show tool progress in a side panel, status bar, timeline, or inline expandable detail.

The point is not to expose every internal detail to the user.
The point is to make the system understandable.

Good frontend text might be:

Checking staging deployment status...

Bad frontend text is:

The model is thinking.

The first tells the user what the system is doing.
The second tells them nothing operationally useful.

Backend tools and frontend tools are different

AG-UI can also support frontend tools.

A frontend tool is registered on the client and executes in the client environment.
The server can see the tool declaration and request the call, but the implementation runs on the client.

That is useful for client-local capabilities:

read browser-local preferences
access device location after user permission
inspect UI selection
copy something to clipboard
trigger a client-side view change
read state that only exists in the frontend

For example:

[Description("Gets the user's current client-side location after permission.")]
public static Task<string> GetUserLocationAsync(
    CancellationToken cancellationToken)
{
    // In a real frontend, this would call browser or device APIs.
    return Task.FromResult("Berlin, Germany");
}

AITool[] frontendTools =
[
    AIFunctionFactory.Create(GetUserLocationAsync)
];

AIAgent remoteAgent = aguiClient.AsAIAgent(
    name: "agui-client",
    description: "AG-UI client with frontend tools.",
    tools: frontendTools);

The server receives the frontend tool schema.
The model can request the tool.
The client executes it and returns the result.

Do not use frontend tools for privileged operations.

If the action changes server data, spends money, changes permissions, or calls internal systems, make it a backend tool with normal authentication, authorization, validation, and audit logging.

Frontend tools are for client capabilities.
Backend tools are for application capabilities.

Approval requests need real UI

The previous article covered ApprovalRequiredAIFunction on the backend.
AG-UI brings that approval request across the frontend boundary.

In the .NET pattern documented for Agent Framework, approval-required tools use ApprovalRequiredAIFunction.
Middleware converts FunctionApprovalRequestContent into an AG-UI client tool call, commonly named request_approval.
The client renders an approval UI and sends the decision back as a tool result.
Middleware then converts that response back into FunctionApprovalResponseContent so the agent can continue.

Conceptually:

agent requests risky function
-> FunctionApprovalRequestContent
-> AG-UI client tool call: request_approval
-> frontend shows approval UI
-> user approves or rejects
-> frontend returns tool result
-> FunctionApprovalResponseContent
-> agent continues

The frontend must show the actual operation.
Not just the model's explanation.

For a deployment approval, show:

Tool: DeployToProductionAsync
Release: release-2026-06-24
Environment: production
Change ticket: CHG-1042
Requested by: lukas@example.com
Expires: 2026-06-24 15:30 UTC

A view model might look like this:

public sealed record ApprovalRequestViewModel
{
    public required string ApprovalId { get; init; }
    public required string FunctionName { get; init; }
    public required IReadOnlyDictionary<string, object?> Arguments { get; init; }
    public required string RequestedBy { get; init; }
    public DateTimeOffset ExpiresAt { get; init; }
    public string? AgentExplanation { get; init; }
}

The frontend should not approve vague text such as:

The agent wants to proceed.

It should approve a specific function call with specific arguments.

This is where AG-UI matters.
Approval becomes an interactive event in the frontend, not a paragraph in the chat.

State updates are separate from messages

Some agent applications are not only conversations.
They have visible application state.

Examples:

a trip plan
a deployment checklist
a draft support reply
a generated recipe
a remediation plan
a document outline
a shopping cart

The agent may update that state while it works.

AG-UI state management exists so the client and server can keep a synchronized view of application state.
The Agent Framework docs describe state snapshots, bidirectional state synchronization, and predictive state updates.

For a deployment frontend, state might be:

public sealed class DeploymentReviewState
{
    public string ReleaseId { get; set; } = string.Empty;
    public string Environment { get; set; } = "production";
    public string CurrentStage { get; set; } = "not-started";
    public List<DeploymentCheck> Checks { get; set; } = [];
    public bool RequiresApproval { get; set; }
}

public sealed class DeploymentCheck
{
    public string Name { get; set; } = string.Empty;
    public string Status { get; set; } = "pending";
    public string? Details { get; set; }
}

Then the UI can render a real checklist instead of hoping the model produces a readable markdown list every time.

AG-UI state events let the frontend receive updates as structured data.
The exact implementation depends on the client stack and package version.
In the current .NET docs, one C# pattern emits DataContent with application/json; the AG-UI hosting layer converts that into state snapshot events.

Conceptually:

agent or middleware produces structured state
-> JSON state snapshot
-> AG-UI state event
-> frontend replaces or patches local state
-> UI rerenders structured panels

The key rule is:

Do not make the frontend parse state out of assistant text.

If the agent updates a checklist, send a checklist object.
If it updates a plan, send a plan object.
If it updates approval state, send approval state.

Predictive state updates need discipline

Predictive state updates are attractive because the frontend can update while the model is still generating tool arguments.
That can make generative UI feel much faster.

But optimistic state is still speculative.

The model may change its arguments while streaming.
The user may reject the approval.
The tool may fail validation.
The final backend state may differ from the predicted state.

So treat predictive updates as provisional.

For example:

public enum StateConfidence
{
    Predicted,
    Confirmed,
    Rejected
}

public sealed record UiStateEnvelope<T>(
    T Value,
    StateConfidence Confidence,
    string? SourceRunId);

A good UI can show predicted state differently from confirmed state.
For example:

dim predicted checklist items
show "drafting..." on generated sections
require approval before applying risky state changes
rollback predicted changes if the tool call is rejected

This matters most when predictive state combines with human approval.

If the frontend shows a deployment plan while the model streams tool arguments, that does not mean the plan has been approved or executed.
The UI needs separate states:

predicted
pending approval
approved
executed
failed

Do not let optimistic UI blur the boundary between "the model proposed this" and "the system did this".

AG-UI does not replace product design

AG-UI gives you a protocol.
It does not decide what your product should show.

You still need to design the interaction.

For each agent feature, decide:

What should be visible to the user?
What should stay in developer traces?
Which tool calls deserve inline status?
Which tool calls should be hidden behind a simple progress label?
Which actions require approval?
Which state updates are provisional?
Which errors need user action?
Which errors should only be logged?

For a deployment assistant, I would show:

current run status
deployment checklist state
meaningful tool progress
approval request details
final result

I would not show:

every token event
every internal message
raw prompt text
secrets or internal identifiers the user is not allowed to see
model reasoning traces

The frontend is a boundary too.
It should expose useful system behavior without leaking internal implementation details.

A practical event handling shape

A frontend can treat the AG-UI stream as an event reducer.

The backend emits events.
The frontend reduces those events into UI state.

For example:

public sealed record AgentFrontendState
{
    public AgentRunStatus RunStatus { get; init; } = AgentRunStatus.Idle;
    public string AssistantText { get; init; } = string.Empty;
    public List<ToolCallViewModel> ToolCalls { get; init; } = [];
    public ApprovalRequestViewModel? PendingApproval { get; init; }
    public DeploymentReviewState? DeploymentState { get; init; }
    public string? ErrorMessage { get; init; }
}

Then each event updates one part of state.

public static AgentFrontendState Reduce(
    AgentFrontendState state,
    AIContent content)
{
    return content switch
    {
        TextContent text =>
            state with
            {
                AssistantText = state.AssistantText + text.Text
            },

        FunctionCallContent call =>
            state with
            {
                RunStatus = AgentRunStatus.RunningTool,
                ToolCalls = AddToolCall(state.ToolCalls, call)
            },

        FunctionResultContent result =>
            state with
            {
                ToolCalls = CompleteToolCall(state.ToolCalls, result)
            },

        FunctionApprovalRequestContent approval =>
            state with
            {
                RunStatus = AgentRunStatus.WaitingForApproval,
                PendingApproval = ToApprovalViewModel(approval)
            },

        DataContent data when data.MediaType == "application/json" =>
            state with
            {
                DeploymentState = ParseDeploymentState(data)
            },

        _ => state
    };
}

This example uses Agent Framework content types because it fits the .NET client surface.
A browser client may work with raw AG-UI protocol event names instead.
The design is the same:

Events update structured UI state.
Text is only one part of that state.

Cancellation belongs in the frontend contract

Agent frontends also need cancellation.

Users close tabs.
They navigate away.
They reject an approval.
They realize the prompt was wrong.

The frontend should be able to cancel the current run and the backend should pass cancellation through the model call and tool calls.

In .NET, keep the CancellationToken flowing:

await foreach (AgentResponseUpdate update in remoteAgent.RunStreamingAsync(
                   messages,
                   session,
                   cancellationToken: cancellationToken))
{
    Render(update);
}

For backend tools:

public static Task<DeploymentStatus> CheckDeploymentStatusAsync(
    string releaseId,
    IServiceProvider services,
    CancellationToken cancellationToken)
{
    var deployments = services.GetRequiredService<IDeploymentService>();
    return deployments.CheckStatusAsync(releaseId, cancellationToken);
}

If the browser disconnects from the SSE stream, the server should not blindly keep doing expensive work unless the product explicitly requires background continuation.

This is especially important with tools.
A cancelled text generation is one thing.
A cancelled deployment workflow is another.
Make the cancellation behavior explicit per operation.

When I would use AG-UI

Use AG-UI when:

the frontend needs streaming text
users should see tool progress
approval requests need real UI
agent state should update visible components
frontend tools need to run in the client
backend tools should expose progress and results
multiple clients should talk to the same agent endpoint shape
you want a protocol boundary instead of custom ad-hoc streaming JSON

This is especially useful when the agent is part of an application, not just a chat demo.

Examples:

deployment assistant with approval and checklist state
support copilot that drafts replies and creates tickets
document review UI with extracted findings and reviewer actions
planning assistant that updates a structured itinerary
operations assistant that streams tool status while querying systems

When I would not use AG-UI

Do not add AG-UI just because an agent exists.

It may be unnecessary when:

the application only needs a simple text response
there are no tools, approvals, or shared state
a normal backend endpoint already gives enough structure
the UI is internal and short-lived
you do not want to take a dependency on a still-evolving protocol
the frontend team cannot support event-driven state handling yet

For a background job, webhook processor, batch summarizer, or simple API endpoint, direct Agent Framework usage may be cleaner.

Use RunAsync or RunStreamingAsync directly when the frontend does not need the extra protocol semantics.

Conclusion

Agent frontends need more than a plain chat box once agents start doing application work.

Streaming text solves only one part of the experience.
The frontend also needs structured events for tool progress, approval requests, state updates, errors, cancellation, and run lifecycle.

AG-UI gives Agent Framework applications a protocol boundary for that.
On the backend, MapAGUI exposes an agent as an HTTP and SSE endpoint.
On the client, AG-UI events can become structured UI state instead of being flattened into assistant text.

The design I would carry forward is:

render text as text
render tool calls as tool progress
render approvals as approval forms
render state as structured UI
keep frontend tools separate from backend tools
treat predictive state as provisional
keep cancellation and errors explicit

That turns the frontend from a passive transcript into part of the agent system.

In the next post, I will move from frontend behavior to production visibility.
Once the UI can show streaming, tools, approvals, and state, the backend also needs traces that explain what happened across model calls, tool execution, AG-UI events, and user decisions.

DEV Community: Lukas Walter

Trust Boundaries Around AI Features

Trust depends on the property you need

Trace the whole feature

Map the boundaries that change risk

Keep security context out of model arguments

Make approval an application-policy decision

Store the operation that was approved

Re-check everything that can change

Structured output narrows the interface

Retrieved content and tool results remain data

Least privilege limits what a failure can do

Review boundaries through failure cases

When this level of control is useful

Practical boundary checklist

Further reading

Why Retrieval Exists

The model has a knowledge boundary

Retrieval is runtime context selection

Retrieval is not the same as vector search

What retrieval gives the application

Access to private and domain-specific knowledge

Selection from more information than one request needs

Independent updates

Permission-aware context

Inspectable evidence

One assistant, six different data problems

What retrieval does not solve

When corpus retrieval is the wrong solution

Use a structured lookup or query for application state

Use deterministic code for rules and calculations

Use direct context when the relevant input is already known

Fix behavior problems at the behavior layer

Do not treat an ungoverned corpus as authoritative

When retrieval is enough without generation

A practical decision sequence

1. Does the task require application-owned or source-grounded information that is not already in the request?

2. Is the need a structured query, deterministic computation, or relevance-based evidence selection?

3. What authority does the source corpus have?

4. Can the application enforce scope before generation?

5. Can you evaluate retrieval separately?

When to use retrieval

Further reading

The Model Is Only One Dependency. Map the Rest.

Start with enforceable properties

Separate three dependency roles

Classify interactions, not services

Stop at an owned boundary

Expand only risky interactions

Read authorized purchase history

Attempt shortlist save

Resume or reconcile non-terminal shortlist operations

Preserve the contract when degrading

Use existing artifacts as inputs

Apply it one operation at a time

Related reading

Further reading

Stop Letting Provider SDKs Define Your .NET AI Architecture

The first generation was organized around the SDK

IChatClient creates an application boundary

The abstraction is more than provider switching

Middleware turns cross-cutting behavior into composition

Shared interfaces do not erase provider differences

One application may need more than one client

Agents and frameworks sit above this foundation

A practical migration path

1. Find provider clients outside the composition root

2. Adapt the provider client to IChatClient

3. Move prompts and use-case policy into application services

4. Add one cross-cutting behavior through the pipeline

5. Replace network-heavy tests with a fake client

When to use the shared abstraction

When to keep the provider SDK visible

Practical takeaway

Related reading

Sources

Calling a Model Is Easy. Running an AI System Is Not

A successful response proves very little

Trace the complete request path

Give each boundary an explicit contract

`IChatClient` creates an application boundary

2. Adapt the provider client to `IChatClient`

Returning `202` without a status resource

Starting `Task.Run` from the endpoint

Put the model behind `IChatClient`