Lukas Walter

Posted on Aug 14 • Originally published at lukaswalter.dev

Trust Boundaries Around AI Features

#ai #architecture #dotnet #csharp

If an action requires approval, a prompt is never the control that enforces it.

A prompt can tell the model to ask, "Should I close this ticket?" That helps the conversation. It does not guarantee that the model asks, that the user sees the exact operation, or that the arguments stay unchanged after confirmation. A deterministic execution boundary must verify mandatory approval before execution.

I use the same rule for other AI output: the model may propose. The application decides whether that proposal becomes behavior.

A trust boundary is where authority, security domain, data exposure, or an established trust property changes. Some boundaries protect an action. Others protect data as it moves to a model provider, log sink, browser, plugin, or another tenant.

Trust depends on the property you need

"Trusted" and "untrusted" are too blunt on their own.

An authenticated user has a verified identity, but their text is still untrusted input. An internal document may have known provenance while being stale or outside the current user's permissions. Model output can match a JSON schema and still request an action the caller cannot perform.

For each value that crosses a boundary, write down which properties have been established:

Identity and delegation: who initiated, approved, and executes the request, and under whose authority does each step run?
Authorization: what may each identity access or change?
Provenance and integrity: where did the data come from, and could it have changed outside the expected process?
Freshness and state: is the data current enough, and does the target still satisfy the expected conditions?
Validity: does the value satisfy its syntax and domain rules?
Approval: did an authorized reviewer approve this exact operation?

These properties do not substitute for one another. Authentication does not make input safe. Schema validation does not grant access. Approval does not freeze permissions or resource state.

Trace the whole feature

Consider a support assistant that can search customer records, summarize a ticket, draft a response, and request that a ticket be closed.

The action path branches after policy classification:

authenticated support user
    -> application resolves identity and tenant
    -> trusted data-access path constrains retrieval
    -> selected content crosses to the model provider
    -> model proposes a response or action
    -> application parses and validates the proposal
    -> application loads the target inside the caller's tenant
    -> authorization evaluates caller and resource
    -> application policy classifies the canonical action

        Deny
            -> stop and record the decision

        Allow
            -> execution boundary re-checks current state, authorization, and policy
            -> executor atomically persists the action and Allow decision while claiming a durable Executing attempt

        RequireApproval
            -> approval service stores the canonical operation
            -> authorized reviewer approves or rejects that operation
            -> execution boundary re-checks current state, authorization, and policy
            -> executor atomically moves Approved to Executing and creates a durable attempt

    -> executor performs the effect
    -> executor records the execution outcome
    -> application applies destination-specific output handling

Logs and traces cross several parts of this path. They can capture prompts, retrieved documents, tool arguments, resource identifiers, approval details, and model responses. Treat telemetry as its own data boundary. Decide which fields may leave the application, redact sensitive values, and avoid turning full prompt capture into the default.

The call to the model provider is another boundary. Send only the data the task needs. Application policy first determines the permitted routing set from the request, data classification, and security context. A model may then choose among non-security-sensitive options such as fast or reasoning within that set. Model output must not select arbitrary providers, deployments, regions, retention settings, or credentials.

The model takes part in the flow, but it owns none of these security decisions. The same design applies to a chat UI, an agent framework, a background worker, or custom orchestration code.

Map the boundaries that change risk

I pay closest attention to places where data gains authority because that is where a plausible string can turn into a real side effect. Data exposure matters just as much when confidential information crosses into a less trusted domain.

Transition	What remains untrusted or unresolved	Application control
Request to application	Text, uploads, requested identifiers	Authentication, request validation, size and rate limits
Application to retrieval	Search text and model-selected identifiers	Tenant and resource authorization in the retrieval or data-access path
Retrieved content to model	Documents, emails, pages, tool results	Source labeling, separation from instructions, constrained tool access, downstream validation
Application to model provider	Prompts and selected business data	Data minimization, approved deployment and region, retention policy, scoped credentials
Model output to application	Text, structured output, tool arguments	Parsing, allow lists, type-specific validation, controlled failure
Application to protected resource	Requested action and resource ID	Current-state lookup and resource-based authorization
Approval client to approval service	Reviewer input and untrusted generated explanations	Reviewer authentication and authorization, request integrity, safe rendering, replay protection
Approval service to executor	Stored operation and approval artifact	Durable state transitions, version checks, reauthorization, current policy, concurrency control
Model output to client	Generated Markdown, HTML, links, or code	Output encoding, sanitization, content policy, data-loss checks
Application to telemetry	Prompts, arguments, responses, decisions	Data minimization, redaction, access control, retention limits

I would not label any one row "prompt injection defense." No filter can prove that retrieved text is harmless. The useful boundary limits what that text can influence and validates every privileged transition after the model has processed it.

Keep security context out of model arguments

The model may propose an operation. It may also provide an identity or scope as target data when the operation needs it, such as asking to view configuration for a named tenant. It must never supply or override the authenticated identity, tenant scope, delegated authority, or authorization result under which the operation runs.

Do not expose a tool contract like this:

Task CloseTicketAsync(
    string tenantId,
    string userId,
    string ticketId,
    bool userIsAuthorized);

The model controls every argument. The function cannot distinguish a real authorization result from a convincing boolean.

Use authenticated application context for the trusted values. Normalize the proposal into a canonical action before policy evaluates it:

public enum ActionDisposition
{
    Allow,
    Deny,
    RequireApproval
}

public sealed record CloseTicketProposal(
    string? TicketId,
    string? ResolutionSummary);

public sealed record CloseTicketArguments(
    string ResolutionSummary);

public sealed record ProposedAction(
    string Operation,
    string RequesterId,
    string TenantId,
    string Environment,
    string ResourceId,
    long ResourceVersion,
    CloseTicketArguments Arguments);

public sealed record ApprovalRequirement(
    string ReviewerPolicy,
    string RiskClass,
    bool AllowSelfApproval,
    TimeSpan ValidFor);

public sealed record ActionDecision
{
    private ActionDecision(
        ActionDisposition disposition,
        ProposedAction? action,
        ApprovalRequirement? approval,
        string decisionSourceId,
        string? policyVersionId,
        string? reason)
    {
        Disposition = disposition;
        Action = action;
        Approval = approval;
        DecisionSourceId = decisionSourceId;
        PolicyVersionId = policyVersionId;
        Reason = reason;
    }

    public ActionDisposition Disposition { get; }
    public ProposedAction? Action { get; }
    public ApprovalRequirement? Approval { get; }
    public string DecisionSourceId { get; }
    public string? PolicyVersionId { get; }
    public string? Reason { get; }

    public static ActionDecision Allow(
        ProposedAction action,
        string decisionSourceId,
        string? policyVersionId = null) =>
        new(
            ActionDisposition.Allow,
            action,
            null,
            decisionSourceId,
            policyVersionId,
            null);

    public static ActionDecision Deny(
        ProposedAction? action,
        string reason,
        string decisionSourceId,
        string? policyVersionId = null) =>
        new(
            ActionDisposition.Deny,
            action,
            null,
            decisionSourceId,
            policyVersionId,
            reason);

    public static ActionDecision RequireApproval(
        ProposedAction action,
        ApprovalRequirement approval,
        string reason,
        string decisionSourceId,
        string? policyVersionId = null) =>
        new(
            ActionDisposition.RequireApproval,
            action,
            approval,
            decisionSourceId,
            policyVersionId,
            reason);
}

The boundary loads the ticket through a tenant-scoped path and checks resource authorization. Summary validation is deliberately small here. A real support system may restrict formatting, require a resolution code, or check for sensitive data before storage. The reviewer UI must still encode the summary as untrusted text.

public sealed class TicketActionBoundary(
    TicketStore tickets,
    CurrentRequest currentRequest,
    IAuthorizationService authorization,
    TicketActionPolicy policy)
{
    private const string DecisionSourceId = "ticket-action-boundary:v1";

    public async Task<ActionDecision> EvaluateAsync(
        CloseTicketProposal proposal,
        CancellationToken cancellationToken)
    {
        string summary = proposal.ResolutionSummary?.Trim() ?? string.Empty;

        if (string.IsNullOrWhiteSpace(proposal.TicketId) ||
            summary.Length is 0 or > 500)
        {
            return ActionDecision.Deny(
                null,
                "Invalid proposal.",
                DecisionSourceId);
        }

        SupportTicket? ticket = await tickets.FindAsync(
            currentRequest.TenantId,
            proposal.TicketId,
            cancellationToken);

        if (ticket is null)
        {
            return ActionDecision.Deny(
                null,
                "Ticket not found.",
                DecisionSourceId);
        }

        AuthorizationResult result = await authorization.AuthorizeAsync(
            currentRequest.User,
            ticket,
            "CloseSupportTicket");

        if (!result.Succeeded)
        {
            return ActionDecision.Deny(
                null,
                "Action is not allowed.",
                DecisionSourceId);
        }

        var action = new ProposedAction(
            Operation: "support-ticket.close",
            RequesterId: currentRequest.UserId,
            TenantId: currentRequest.TenantId,
            Environment: currentRequest.Environment,
            ResourceId: ticket.Id,
            ResourceVersion: ticket.Version,
            Arguments: new CloseTicketArguments(summary));

        return policy.Classify(ticket, action);
    }
}

ASP.NET Core supports this resource-based check through IAuthorizationService.AuthorizeAsync(user, resource, policyName). The authorization handlers evaluating the policy receive the authenticated principal and the loaded ticket as the resource. The model sees neither an authorization flag nor a tenant selector.

ProposedAction is the application's canonical representation of the operation. It includes application-owned security context and execution preconditions. A denial before canonicalization has no action. Once an action has been canonicalized, a policy denial retains it so the audit record shows what was evaluated and rejected.

Each deterministic decision point attaches its own stable source ID. TicketActionBoundary identifies validation and authorization denials. TicketActionPolicy identifies classification decisions and records its policy version separately. A caller cannot supply a stale policy version before the decision happens.

In this sample, CurrentRequest.UserId and GetUserId() return the same application-owned canonical user ID. If an application compares external identities directly, it must include the issuer and relevant tenant or security domain with the subject value.

Make approval an application-policy decision

Mandatory approval always belongs to application policy or another deterministic execution boundary. The prompt can ask for confirmation to make the interaction clearer. That conversational answer is not an approval artifact and must not bypass policy.

The policy needs at least three outcomes:

public sealed class TicketActionPolicy
{
    public const string DecisionSourceId = "ticket-action-policy";
    public const string PolicyVersionId = "support-ticket-close:v3";

    public ActionDecision Classify(
        SupportTicket ticket,
        ProposedAction action)
    {
        if (ticket.Status is not TicketStatus.Resolved)
        {
            return ActionDecision.Deny(
                action,
                "Only resolved tickets can be closed.",
                DecisionSourceId,
                PolicyVersionId);
        }

        if (ticket.IsEscalated ||
            ticket.Priority is TicketPriority.Critical)
        {
            return ActionDecision.RequireApproval(
                action,
                new ApprovalRequirement(
                    ReviewerPolicy: "ApproveSupportTicketClosure",
                    RiskClass: "high",
                    AllowSelfApproval: false,
                    ValidFor: TimeSpan.FromMinutes(15)),
                "Escalated or critical tickets require review.",
                DecisionSourceId,
                PolicyVersionId);
        }

        return ActionDecision.Allow(
            action,
            DecisionSourceId,
            PolicyVersionId);
    }
}

This example allows an authorized user to close an ordinary resolved ticket automatically. It requires review for a critical or escalated ticket and denies closure while work is unresolved.

Real policies can also consider the environment, destination, amount, quantity, bulk scope, reversibility, and compliance impact. Read-only is not automatically low risk. Exporting every customer record or retrieving a secret may deserve a stricter policy than a reversible write.

Do not require approval merely because a model selected a tool. Require it when the operation's disclosure, external, financial, destructive, privilege, or compliance impact exceeds the application's automatic-execution policy.

If risk classification or policy lookup fails for a consequential action, stop. Do not fall back to automatic execution.

Store the operation that was approved

The approval service should persist the canonical ProposedAction, policy decision, and ApprovalRequirement. Another option is to persist one canonical representation with a consistency digest. The reviewer client receives an approval ID and display data. It does not reconstruct the operation for execution.

Canonical serialization makes a digest reproducible. It does not make a plain cryptographic hash tamper-resistant. An attacker who can modify both the operation and its digest can simply recompute the hash. Use a plain hash only to detect accidental inconsistency inside an already protected store.

If the value must protect integrity across a trust boundary, use an HMAC with a protected server-side key, a digital signature, or a digest stored in a separately protected trust domain. Compute it on the server from one documented serialization format. Property order, number and timestamp formats, Unicode normalization, null handling, and collection ordering must be deterministic.

A useful approval record includes:

approval ID and one-time nonce
operation name and normalized arguments
requester, tenant, environment, and security domain
target resource and expected version or state
decision source ID, policy version, and the typed approval requirement
reviewer, decision, and the reviewer policy that was evaluated
whether self-approval is allowed and any separation-of-duties rule
creation, expiry, decision, and execution-state timestamps
intended executor or delegated identity when relevant
an idempotency or deduplication key for execution

The reviewer needs authorization too. Before accepting the decision, check whether this reviewer may approve this operation for its tenant, environment, resource, and risk class. For some operations, the requester must not approve their own proposal.

public sealed record ApprovalAuthorizationContext(
    ProposedAction Action,
    ApprovalRequirement Requirement);

var approvalContext = new ApprovalAuthorizationContext(
    pendingApproval.ProposedAction,
    pendingApproval.Requirement);

AuthorizationResult reviewerAccess = await authorization.AuthorizeAsync(
    reviewer,
    approvalContext,
    pendingApproval.Requirement.ReviewerPolicy);

if (!reviewerAccess.Succeeded ||
    (!pendingApproval.Requirement.AllowSelfApproval &&
     pendingApproval.ProposedAction.RequesterId == reviewer.GetUserId()))
{
    return ApprovalResult.Denied("Reviewer is not authorized.");
}

return await approvals.RecordDecisionAsync(
    pendingApproval.ApprovalId,
    reviewer.GetUserId(),
    approved,
    cancellationToken);

Reviewer authorization may depend on current resource attributes that are not part of the canonical action, such as ownership, business unit, or classification. Load those attributes through the trusted resource path before evaluating the reviewer policy. Pass the resulting application-defined context to AuthorizeAsync. Do not ask the reviewer client or model to supply it.

RecordDecisionAsync must atomically move the record from Pending to Approved or Rejected. It rejects expired, already decided, or otherwise invalid records. Use a transaction, row version, compare-and-swap update, or an equivalent concurrency mechanism so two reviewers cannot overwrite each other's decision. A check in the client or controller is not enough.

An unauthorized submission is denied. It is not a rejection decision. Reserve Rejected for an authorized reviewer who explicitly chooses not to approve the operation.

The approval UI is a trust boundary. Render the model-generated summary as untrusted text and keep it separate from fields loaded by the application, such as the requester, ticket ID, current priority, environment, and expected effect. Generated markup must not impersonate authoritative UI or conceal part of the operation.

Approval binds to the canonical operation, not to the persuasiveness of the explanation.

For browser-based approval endpoints that use ambient credentials such as authentication cookies, apply the same request-integrity and anti-forgery protections as other state-changing endpoints.

Re-check everything that can change

An automatic Allow and an approved action enter the same execution boundary with different admission evidence. Neither path freezes authorization, policy, or resource state.

The two paths should converge like this:

automatically allowed action
    -> verify the Allow decision and canonical action
    -> reload the resource through the trusted tenant path
    -> re-authorize the requester
    -> compare resource version and expected state
    -> evaluate the current action policy
    -> atomically persist the action and current Allow decision while claiming a durable Executing attempt

approved action
    -> verify binding, expiry, evidence of decision-time reviewer authorization, and decision state
    -> reload the resource through the trusted tenant path
    -> re-authorize every identity whose current authority execution depends on
    -> compare resource version and expected state
    -> evaluate the current action policy and approval compatibility
    -> atomically move Approved to Executing and create a durable attempt

both paths
    -> perform the effect with idempotency or explicit duplicate handling
    -> move Executing to Completed, OutcomeUnknown, FailedRetryable, or FailedFinal
    -> record a result without leaking sensitive content

If a resource change can invalidate execution, condition the claim on the version that was just checked. An optimistic concurrency condition is one option. Another is to prevent incompatible changes until the point of commitment. For an external effect, define that point and decide how to coordinate conflicting state changes.

The durable-attempt and idempotency rules also apply to automatically allowed actions when their side effects require them. Approval changes the admission path, not the reliability requirements of execution.

The automatic path does not need a separate durable Allowed state. Its execution claim atomically persists the canonical action and current Allow decision as admission evidence with the durable attempt.

On the approval path, reviewer authorization is mandatory when the decision is made. Revalidating the reviewer during execution depends on policy. Some systems treat an approval as a valid historical act after the reviewer changes roles. Others require that authority to remain in place until execution. Record which rule your system uses.

The automatic path must still receive Allow when it re-evaluates policy. On the approval path, a current Deny stops execution. A changed approval requirement invalidates the stored approval unless the current policy explicitly accepts approvals created under its recorded version. A later Allow must not silently remove a control that the stored decision required.

The transaction design depends on the side effect. A database update can often move the execution state and apply the change in one transaction with an optimistic concurrency token. An external email or payment cannot share that transaction.

For external effects, use a durable execution record or outbox and an idempotency key when the provider supports one. Recovery must treat an abandoned or stale Executing attempt as potentially applied. Reconcile the external result or rely on a safe idempotency mechanism before retrying. There is no generic exactly-once guarantee across a local database and an arbitrary external system.

Approval-state transitions and idempotency solve different problems. The state machine stops an approval decision or execution claim from being replayed. Idempotency prevents duplicate effects for the same operation. The resource-version check detects changes covered by the resource's versioning policy. Explicit expected-state checks determine whether the security- or business-relevant preconditions still hold.

Before initiating the effect, a failure in reauthorization, current-state loading, policy evaluation, durable-attempt persistence, or approval validation when required must prevent execution. After an external effect may have occurred, a persistence failure cannot turn that effect into a denial. Move the attempt to OutcomeUnknown when possible, block blind retries, and reconcile the external result instead of marking the operation Pending or Rejected.

Structured output narrows the interface

Structured output makes the boundary easier to inspect. It can reject missing properties, wrong types, and unsupported shapes before business logic runs.

It cannot establish intent or permission. This is valid JSON:

{
  "ticketId": "ticket-from-another-tenant",
  "resolutionSummary": "Customer confirmed resolution."
}

Do not ask the model whether the ticket belongs to the current tenant. Load it through the trusted tenant path and evaluate authorization in application code.

Different model-controlled values need different controls. Prefer predefined queries or query builders over arbitrary generated SQL.

For URLs, restrict schemes and destinations. The network endpoint used for the connection must satisfy the DNS/IP policy. Do not validate one resolution and then allow the HTTP client to perform a second, unchecked resolution. Disable redirects or repeat the destination and DNS/IP checks for every redirect target. HttpClientHandler.AllowAutoRedirect defaults to true, so configure it intentionally for server-side fetches.

Resolve file paths against an allowed root and handle traversal and links. Apply destination and data-loss policy to recipient addresses. Map configuration choices to allow-listed identifiers.

There is no generic ValidateModelOutput() call that makes all of those values safe.

Retrieved content and tool results remain data

Enforce tenant and resource authorization inside the trusted retrieval or data-access path before the caller or model can observe protected content or result metadata. A metadata filter is one option. Row-level security, ACL-aware repositories, separate indexes, per-tenant storage, and scoped service identities can enforce the same boundary.

Remove unauthorized candidates inside the trusted authorization boundary. Their content, identifiers, scores, counts, and metadata must not reach an unauthorized component, model, client, cache, log, or trace.

Keep that check as close to candidate generation as practical. A trusted application or retrieval service may create a broader internal candidate set and apply authoritative security trimming, but only when that broader access is legitimate and nothing unauthorized leaves the trusted path.

A matching document can contain malicious instructions, inaccurate statements, or stale facts. Preserve source and version metadata, but do not mistake labels for enforcement. Retrieved content may inform a proposal. It does not authorize the next operation.

Tool results have the same problem. An internal function may return text from an email, website, issue tracker, or database field that another user controls. The function call came from trusted code. Its returned text did not become a trusted instruction because of that.

Indirect prompt injection is one way this boundary fails. No prompt or content filter removes the risk completely, so downstream capabilities still need deterministic controls.

Least privilege limits what a failure can do

A document summarizer does not need permission to send email. A support assistant that drafts refunds may not need permission to issue them. Separate read and write capabilities where practical, and put sensitive operations behind narrow application services instead of exposing broad infrastructure clients to the model-facing layer.

Prefer credentials and data-layer controls that make cross-tenant access impossible. Some systems still use a shared service identity with broad data access. When that is unavoidable, centralize tenant enforcement and test it so model-generated identifiers or queries cannot bypass the trusted path.

Execution identity also needs an explicit design. The application can act with the requester's delegated authority, or it can use a service identity after checking requester authorization and approval. Those choices behave differently when permissions are revoked and when tokens expire. They also produce different audit trails. Record who requested, approved, and executed the operation rather than collapsing all three into "the user."

OWASP describes excessive agency as excessive functionality, permissions, or autonomy. Reducing any one of them limits the damage when the model makes a bad decision or untrusted content influences it.

Review boundaries through failure cases

An architecture diagram names components. A boundary review should say what happens when an assumption fails.

For each transition, record:

Which values cross it?
Which property must be true first?
Which deterministic component checks that property?
What happens when the check fails?
What evidence is logged without exposing protected content?
What can the component still do if this check has a bug?

For the ticket example:

Input	Required property	Enforcement	Failure behavior
Ticket ID	Exists inside the caller's tenant	Tenant-scoped repository	Reject without revealing another tenant's ticket
Resolution summary	Meets storage and display rules	Application validator	Return a controlled validation error
Close request	Requester may close this resource	Resource authorization policy	Reject and audit the denial
Approval decision	Reviewer may approve this risk class	Reviewer authorization policy	Reject the decision
Approval decision	Record is `Pending` and unchanged	Approval service with atomic transition	Reject expired or already decided records
Current ticket	Version and state still match	Repository and action policy	Invalidate the stale approval
Execution attempt	Allowed or approved operation is claimed once	Durable execution state machine	Recover, reconcile, return the recorded result, or block retry

That is more useful than writing "validate AI output" beside a model box. It names the property, owner, and failure contract.

When this level of control is useful

Map explicit trust boundaries when an AI feature:

retrieves protected or tenant-scoped data
consumes uploaded, external, or user-controlled content
turns model output into routing or workflow decisions
sends data to an external provider, tool, plugin, or telemetry system
writes application state or communicates with people
affects money, permissions, production resources, or regulated data

A small drafting feature with no protected data and no side effects needs fewer controls. It still needs safe rendering and appropriate storage rules.

Sometimes the honest answer is that an LLM should not make the decision. If deterministic checks cannot make the transition safe enough for its consequence, keep that operation outside the AI feature.

Practical boundary checklist

Before shipping, check that:

identity, delegation, and tenant scope come from authenticated application context
retrieval authorization runs inside the trusted authorization path before protected content or result metadata becomes observable outside it
model-provider and telemetry boundaries minimize and protect sensitive data
retrieved content and tool results remain untrusted data
structured output receives validation appropriate to the value type and destination
authorization runs against the current protected resource
application policy chooses Allow, Deny, or RequireApproval
mandatory approval is verified outside the prompt
the reviewer is authorized for the operation and cannot bypass separation-of-duties rules
approval binds to the canonical operation, resource version, policy decision, typed requirement, and expiry
approval decisions use an atomic Pending to Approved or Rejected transition
automatically allowed and approved actions enter the execution boundary through a durable attempt
recovery treats stale Executing attempts as potentially applied and blocks unsafe retries
the executor re-checks current authorization, resource state, and current policy
when approval was required, the executor also validates policy compatibility and approval state
retries are idempotent or deduplicated where possible, and blocked when safe retry cannot be guaranteed
execution credentials follow least privilege and tenant enforcement cannot be influenced by model input
approval and policy failures before effect initiation stop consequential execution
logs record decisions without copying sensitive context by default
each client applies output handling for its destination, such as contextual encoding, restricted Markdown, or URL validation

The model can interpret, rank, summarize, and propose. The application owns the decision to act.

DEV Community