DEV Community: Lukas Walter

Querying Qdrant from .NET

Lukas Walter — Tue, 29 Sep 2026 15:30:00 +0000

This example turns a question into an embedding, applies a payload filter, and asks Qdrant for up to three closest eligible points. It then prints the stored text and score for each result.

The code continues the same QdrantCollections console project and three support notes from Indexing Your First Documents. It reuses the collection, embedding profile, point IDs, and payload fields from that example. This time, the program reads instead of writing.

Continue from the indexed collection

Start the Qdrant container from Your First Qdrant Collection if it is not already running:

docker start -a qdrant-local

Keep it running and use another terminal for the .NET application. The collection should contain the three points added at the end of the indexing article:

Document ID	Title
`worker-shutdown`	Stop a background service cleanly
`http-client-lifetime`	Create HTTP clients through the factory
`bounded-work-queue`	Limit queued background work

If the collection is empty, run the indexing program first. This query example does not create the collection or add points.

Continue with the same embedding setup that produced the stored vectors:

Setting	Azure OpenAI	Ollama locally
Model	`text-embedding-3-small`	`embeddinggemma:300m`
Dimensions	1,536	768
Collection	`support-documents-v1`	`support-documents-embeddinggemma-v1`
Profile ID	`support-text-v1`	`support-embeddinggemma-v1`
Query format	Plain question text	`task: search result \

The query vector and stored document vectors must belong to the same embedding space. A 768-dimensional EmbeddingGemma query cannot search the 1,536-dimensional Azure OpenAI collection. Matching dimensions would not be enough either. Two models can return the same number of values while producing incompatible vectors.

Configure the query embedding

Replace {% raw %}Program.cs with one of the following setup blocks. Then append the shared query code from the next section.

Azure OpenAI

Use this block if you indexed the notes into support-documents-v1. It reuses the packages, deployment, and environment variables from the indexing article.

using Microsoft.Extensions.AI;
using Azure.AI.OpenAI;
using System.ClientModel;
using Qdrant.Client;
using Qdrant.Client.Grpc;
using static Qdrant.Client.Grpc.Conditions;

const string collectionName = "support-documents-v1";
const string embeddingProfileId = "support-text-v1";
const int vectorSize = 1536;

string endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT")
    ?? throw new InvalidOperationException("Set AZURE_OPENAI_ENDPOINT.");
string deploymentName = Environment.GetEnvironmentVariable(
    "AZURE_OPENAI_EMBEDDING_DEPLOYMENT")
    ?? throw new InvalidOperationException("Set AZURE_OPENAI_EMBEDDING_DEPLOYMENT.");
string apiKey = Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY")
    ?? throw new InvalidOperationException("Set AZURE_OPENAI_API_KEY.");

using IEmbeddingGenerator<string, Embedding<float>> generator =
    new AzureOpenAIClient(new Uri(endpoint), new ApiKeyCredential(apiKey))
        .GetEmbeddingClient(deploymentName)
        .AsIEmbeddingGenerator();

Func<string, string> prepareQuery = query => query;

The query is embedded as plain text because the documents were embedded as plain text. Keep the deployment on the same text-embedding-3-small model and compatible model version used during indexing.

Ollama with EmbeddingGemma

Use this block if you indexed the notes into support-documents-embeddinggemma-v1:

using Microsoft.Extensions.AI;
using OllamaSharp;
using Qdrant.Client;
using Qdrant.Client.Grpc;
using static Qdrant.Client.Grpc.Conditions;

const string collectionName = "support-documents-embeddinggemma-v1";
const string embeddingProfileId = "support-embeddinggemma-v1";
const int vectorSize = 768;

using IEmbeddingGenerator<string, Embedding<float>> generator =
    new OllamaApiClient(
        new Uri("http://localhost:11434"),
        "embeddinggemma:300m");

Func<string, string> prepareQuery = query =>
    $"task: search result | query: {query}";

The indexing example formatted EmbeddingGemma documents as title: ... | text: .... Retrieval queries use the corresponding task: search result | query: ... format from the model card. This is not decorative prompt text. It is part of the embedding profile and affects the vector the model produces.

Embed the question and query Qdrant

Append this shared code after the chosen setup block:

using var executionTimeout =
    new CancellationTokenSource(TimeSpan.FromMinutes(2));
CancellationToken token = executionTimeout.Token;

using var client = new QdrantClient(host: "127.0.0.1", port: 6334);

CollectionInfo collection = await client.GetCollectionInfoAsync(
    collectionName, token);

VectorsConfig? config = collection.Config.Params.VectorsConfig;
if (config is null ||
    config.ConfigCase != VectorsConfig.ConfigOneofCase.Params ||
    config.Params.Size != (ulong)vectorSize ||
    config.Params.Distance != Distance.Cosine)
{
    throw new InvalidOperationException(
        $"Collection '{collectionName}' does not use the expected " +
        $"unnamed {vectorSize}-dimension cosine vector.");
}

if (!collection.Config.Metadata.TryGetValue(
        "embedding-profile-id", out Value? storedProfile) ||
    storedProfile.StringValue != embeddingProfileId)
{
    throw new InvalidOperationException(
        $"Collection '{collectionName}' does not match profile '{embeddingProfileId}'.");
}

const string question =
    "How do I stop a background service without abandoning I/O?";

ReadOnlyMemory<float> queryVector = await generator.GenerateVectorAsync(
    prepareQuery(question), cancellationToken: token);

if (queryVector.Length != vectorSize)
{
    throw new InvalidOperationException(
        $"Expected {vectorSize} values, received {queryVector.Length}.");
}

string[] eligibleDocumentIds =
    ["worker-shutdown", "bounded-work-queue"];
Filter eligibilityFilter = Match("document_id", eligibleDocumentIds);

IReadOnlyList<ScoredPoint> results = await client.QueryAsync(
    collectionName: collectionName,
    query: queryVector.ToArray(),
    filter: eligibilityFilter,
    limit: 3,
    payloadSelector: new[] { "document_id", "title", "text" },
    vectorsSelector: false,
    cancellationToken: token);

Console.WriteLine($"Question: {question}");

foreach (ScoredPoint result in results)
{
    if (!result.Payload.TryGetValue(
            "document_id", out Value? documentId) ||
        !result.Payload.TryGetValue("title", out Value? title) ||
        !result.Payload.TryGetValue("text", out Value? text))
    {
        throw new InvalidOperationException(
            $"Point '{result.Id}' is missing the expected payload.");
    }

    Console.WriteLine();
    Console.WriteLine($"Score: {result.Score:F4}");
    Console.WriteLine($"Document: {documentId.StringValue}");
    Console.WriteLine($"Title: {title.StringValue}");
    Console.WriteLine($"Text: {text.StringValue}");
}

Run the program:

dotnet run

worker-shutdown is expected to rank first for this sample. I have not hardcoded the scores because they can differ between embedding providers and model versions. With the current filter, http-client-lifetime cannot appear even if its vector is close to the query.

The metadata check above has a deliberate limit. The profile ID says which embedding profile the collection is supposed to contain. It does not verify the model artifact that produced the stored vectors. That distinction matters for mutable model tags such as embeddinggemma:300m.

What the query does

The read path is short:

Validate the expected unnamed vector size and distance, plus the declared embedding profile.
Prepare the question according to that profile.
Generate one query vector.
Apply the payload eligibility constraint during vector search.
Return the highest-scoring eligible points and the payload fields needed by the caller.

The call uses Qdrant's QueryAsync API. Passing the float array as query requests nearest-neighbor search against the collection's unnamed dense vector. The collection already defines cosine distance, so the request does not choose another metric.

payloadSelector returns only document_id, title, and text. vectorsSelector: false makes it explicit that the application does not want the stored 768- or 1,536-value vectors returned. Qdrant query responses already omit vectors by default, so the argument documents the intended response shape. A retrieval result normally needs a stable source identity and displayable content, not another copy of the indexed vector.

The payload guard in the loop checks that those three fields exist. It does not validate their Qdrant value types. That is enough for this controlled sample, where the indexing code owns the payload. If several writers can produce points, validate the payload schema at that boundary instead of assuming every stored value is a string.

The result count is an upper bound. limit: 3 does not guarantee three points. Only two of the sample documents pass the allowlist, so at most two can be returned.

Filters decide eligibility and vectors decide order

I keep the allowlist in the example instead of hiding it behind a helper method:

string[] eligibleDocumentIds =
    ["worker-shutdown", "bounded-work-queue"];
Filter eligibilityFilter = Match("document_id", eligibleDocumentIds);

In an application, that list would come from trusted policy and application state rather than raw user input. The same boundary can represent tenant membership, permissions, language, publication state, or document type.

Only points that satisfy the filter are eligible for the returned results. Qdrant integrates that constraint into its query planning and search instead of retrieving an unfiltered top-k and removing points afterward. With HNSW, Qdrant can use filter-aware graph traversal rather than building a filtered set and starting a separate vector search.

Filtering after retrieval is a different design. It can leave the caller with too few allowed results because ineligible points have already occupied places in the unfiltered top-k. It also puts a mandatory policy check outside the retrieval request.

This local collection has three points and no payload index, which is enough to demonstrate the request shape. For a larger collection, create payload indexes for fields used frequently in filters. A keyword index on document_id supports efficient exact matching and gives Qdrant better information for query planning.

Create those indexes before bulk ingestion when you already know the filter contract. Payload indexes also inform Qdrant's filter-aware vector indexing, so defining them before the HNSW index is built avoids rebuilding that structure later. Qdrant Cloud also enables strict mode by default for new collections and rejects retrieval filters on unindexed fields.

The allowlist is an example, not a complete authorization system. The application still owns how it derives the eligible IDs and whether the caller may see the returned payload.

A score is not a relevance decision

ScoredPoint.Score lets you compare the returned points for this query. With the cosine collection used here, a higher score ranks ahead of a lower score. It does not prove that the first document answers the question.

Avoid copying a threshold from another project. For this dense-vector stage, its useful range depends on the embedding model, corpus, query distribution, and distance metric. Calibrate it with labeled queries and expected results, including cases where the correct outcome is no result.

Thresholds are stage-specific. A value calibrated for dense cosine search does not carry over to hybrid-fusion or reranker scores. Those stages produce different score spaces.

The three-note corpus is useful for checking the mechanics, not for measuring retrieval quality. Try at least these changes before building context for a model:

Rephrase the shutdown question and confirm that worker-shutdown remains near the top.
Ask about HTTP connection reuse, then add http-client-lifetime to the eligible IDs and inspect its rank.
Remove the relevant document from the allowlist and confirm that it does not appear.
Ask an unrelated question and observe that Qdrant still returns the nearest eligible points unless a calibrated acceptance rule rejects them.

That last case matters. A vector search asks which eligible stored vectors are closest under the configured metric. Depending on the collection and search path, the returned points may be approximate nearest neighbors. This query has no acceptance criterion, so it cannot determine whether any result is relevant enough to use. Even with a calibrated score threshold, the cutoff remains an application decision rather than proof of relevance.

When to use this query path

Use this example when the application already has a small Qdrant collection populated with one compatible embedding profile and needs a direct dense-vector query with mandatory payload filtering. It is also a good boundary to wrap behind an application retrieval interface later.

Do not treat the code as a complete RAG pipeline. It does not chunk long documents, deduplicate overlapping results, group chunks by source, rerank candidates, calibrate a relevance threshold, or assemble a context budget for a model. It also does not create the payload indexes needed for a larger filtered workload.

The next implementation step is to replace the whole-document points with deliberate chunks and see how those boundaries change what the query can retrieve. After that, the ranked chunks still need to be selected, cited, and fitted into model context.

Sources

Qdrant: Similarity search and the Query API
Qdrant: Filtering
Qdrant: Payload indexing
Qdrant: Fast approximate search with HNSW
Qdrant: Configure a Cloud cluster
Qdrant: .NET client
Microsoft Learn: Use the IEmbeddingGenerator interface
Google: EmbeddingGemma model card and input formats

Building the .NET AI App Foundation

Lukas Walter — Thu, 24 Sep 2026 15:30:00 +0000

If you already keep provider SDKs at the edge and model calls behind a use-case service, the next problem is assembling those decisions into a working ASP.NET Core application without inventing an AI platform.

This article builds the smallest foundation I would use for that application: one concrete operation, validated model configuration, an instrumented IChatClient pipeline, and explicit buffered and streaming HTTP contracts.

Retrieval, tools, evaluation, and runtime controls stay out of this first version. The code still needs a sensible place for each one when it becomes a real requirement.

Keep the dependency direction small

The request path in this example is deliberately boring:

HTTP endpoint
    -> SupportReplyService
        -> IChatClient

composition root
    -> configuration
    -> Azure OpenAI client
    -> IChatClient pipeline

The articles on provider abstraction and AI service layers explain these two boundaries in detail. I will treat them as settled here and focus on the host around them.

I care more about those arrows than the number of projects. Everything can live in one ASP.NET Core project at first:

AiAppFoundation/
├── Program.cs
├── AI/
│   └── SupportModelOptions.cs
└── SupportReplies/
    ├── SupportReplyContracts.cs
    └── SupportReplyService.cs

Five class libraries would add more project references and more files to navigate without improving this small dependency graph. Keep the direction clear in code first. Add an assembly when you need to enforce dependencies at compile time, reuse a component independently, or reflect a clear ownership boundary. An independently deployed application will usually have its own assembly, but a class library is not a deployment boundary by itself.

Start with one actual operation

The sample drafts a reply to a customer message. It is intentionally unglamorous. Put the application contract in SupportReplies/SupportReplyContracts.cs:

namespace AiAppFoundation.SupportReplies;

public sealed record DraftReplyRequest(string CustomerMessage);

public sealed record DraftReply(string Text);

The endpoint does not accept a system prompt, arbitrary tools, a model name, or provider options. A caller asks for a support reply. It does not get to reconfigure the feature.

Put the service in SupportReplies/SupportReplyService.cs. It owns the instructions and builds fresh request objects for each operation:

using System.Runtime.CompilerServices;
using Microsoft.Extensions.AI;

namespace AiAppFoundation.SupportReplies;

public sealed class SupportReplyService(IChatClient chatClient)
{
    private const string Instructions = """
        Draft a concise support reply.
        Use only facts present in the customer message.
        If information is missing, ask for it instead of inventing it.
        """;

    public async Task<DraftReply> DraftAsync(
        DraftReplyRequest request,
        CancellationToken cancellationToken)
    {
        ChatResponse response = await chatClient.GetResponseAsync(
            CreateMessages(request),
            CreateOptions(),
            cancellationToken);

        return new DraftReply(response.Text);
    }

    public async IAsyncEnumerable<string> StreamAsync(
        DraftReplyRequest request,
        [EnumeratorCancellation] CancellationToken cancellationToken)
    {
        await foreach (ChatResponseUpdate update in
            chatClient.GetStreamingResponseAsync(
                CreateMessages(request),
                CreateOptions(),
                cancellationToken))
        {
            if (!string.IsNullOrEmpty(update.Text))
            {
                yield return update.Text;
            }
        }
    }

    private static List<ChatMessage> CreateMessages(
        DraftReplyRequest request) =>
    [
        new(ChatRole.System, Instructions),
        new(ChatRole.User, request.CustomerMessage)
    ];

    private static ChatOptions CreateOptions() => new()
    {
        MaxOutputTokens = 500
    };
}

SupportReplyService is registered as a concrete type on purpose. The operation, request, and result already give callers a useful boundary. Add a feature-specific interface when you need to replace the service itself, not simply to hide IChatClient. Designing an AI Service Layer explains why a generic IAiService adds little.

Create the messages and ChatOptions for every call because client implementations may mutate them. Dependency Injection for AI Components covers the broader lifetime and scoped-dependency rules.

Wire configuration, model access, and tracing

The code here targets .NET 10. I tested it with Microsoft.Extensions.AI 10.10.0, Microsoft.Extensions.AI.OpenAI 10.10.0, Azure.AI.OpenAI 2.1.0, Azure.Identity 1.21.0, and OpenTelemetry 1.18.0. These are tested pins, not a claim that they will still be current when you start a new project.

dotnet new web -n AiAppFoundation -f net10.0
cd AiAppFoundation
dotnet add package Microsoft.Extensions.AI --version 10.10.0
dotnet add package Microsoft.Extensions.AI.OpenAI --version 10.10.0
dotnet add package Azure.AI.OpenAI --version 2.1.0
dotnet add package Azure.Identity --version 1.21.0
dotnet add package OpenTelemetry.Extensions.Hosting --version 1.18.0
dotnet add package OpenTelemetry.Instrumentation.AspNetCore --version 1.18.0
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol --version 1.18.0

Put the provider-facing settings in AI/SupportModelOptions.cs:

using System.ComponentModel.DataAnnotations;

namespace AiAppFoundation.AI;

public sealed class SupportModelOptions
{
    public const string SectionName = "AI:SupportReplies";

    [Required]
    public Uri Endpoint { get; init; } = null!;

    [Required]
    public string Deployment { get; init; } = "";
}

Add the corresponding routing information to appsettings.json. It does not contain a credential:

{
  "AI": {
    "SupportReplies": {
      "Endpoint": "https://example.openai.azure.com/",
      "Deployment": "support-replies"
    }
  }
}

using AiAppFoundation.AI;
using AiAppFoundation.SupportReplies;
using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Options;
using OpenTelemetry.Trace;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
const string AiActivitySourceName = "AiAppFoundation.AI";

builder.Services
    .AddOptions<SupportModelOptions>()
    .BindConfiguration(SupportModelOptions.SectionName)
    .ValidateDataAnnotations()
    .ValidateOnStart();

builder.Services
    .AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource(AiActivitySourceName)
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter());

builder.Services
    .AddChatClient(services =>
    {
        SupportModelOptions options = services
            .GetRequiredService<IOptions<SupportModelOptions>>()
            .Value;

        var azureClient = new AzureOpenAIClient(
            options.Endpoint,
            new DefaultAzureCredential());

        return azureClient
            .GetChatClient(options.Deployment)
            .AsIChatClient();
    })
    .UseOpenTelemetry(
        sourceName: AiActivitySourceName,
        configure: telemetry =>
            telemetry.EnableSensitiveData = false);

builder.Services.AddScoped<SupportReplyService>();

WebApplication app = builder.Build();

ValidateOnStart catches missing values before the first user request. It cannot prove that Azure is reachable or that the deployment supports every capability the application might request later. Cover those questions with deployment checks and integration tests.

DefaultAzureCredential keeps secrets out of this configuration. For local development, authenticate with a supported developer credential before starting the application. With Azure CLI:

az login

If your account belongs to more than one Microsoft Entra tenant, sign in to the tenant that contains the Azure OpenAI resource:

az login --tenant <tenant-id>

The signed-in identity also needs access to the resource. For inference, Cognitive Services OpenAI User is one suitable role. Check the operations your application performs and grant only the permissions they require. Microsoft's keyless authentication guidance covers other local and deployed credential options.

AddChatClient registers the pipeline as a singleton by default. Keep request state, scoped database contexts, and caller-specific tools out of its factory. Bind those dependencies inside the operation that owns them.

UseOpenTelemetry is easy to overread. It instruments the chat-client pipeline, but it does not collect or export those activities by itself. A host without a tracing provider subscribed to that source still receives nothing. In this sample, AddSource subscribes to the MEAI source, ASP.NET Core instrumentation supplies the surrounding request span, and the OTLP exporter forwards both. Without exporter settings, it uses the local default for the selected OTLP protocol. Point it to a remote collector through the normal OpenTelemetry configuration for your environment, such as OTEL_EXPORTER_OTLP_ENDPOINT.

The explicit EnableSensitiveData = false keeps raw prompts, responses, function arguments, and function results out of MEAI telemetry even if OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT is set elsewhere in the environment. Metadata such as token counts can still be recorded.

The generative AI semantic conventions are still experimental, so individual telemetry attributes should not become application contracts. This sample traces the HTTP request and model call. Retrieval, tools, retries, and persistence will need their own spans when they arrive.

Choose the HTTP contract: buffered or streaming

Continue in Program.cs with the route mappings. My default for this operation is a buffered JSON response:

app.MapPost("/support/replies", async (
    DraftReplyRequest request,
    SupportReplyService replies,
    CancellationToken cancellationToken) =>
{
    if (string.IsNullOrWhiteSpace(request.CustomerMessage))
    {
        return Results.BadRequest(new
        {
            error = "customerMessage is required."
        });
    }

    DraftReply result = await replies.DraftAsync(
        request,
        cancellationToken);

    return Results.Ok(result);
});

The complete draft exists before the response starts. The application can validate it, then choose the final status code and JSON response as one unit.

Streaming can make a longer generation feel faster. It also changes the API contract. I would give it a separate endpoint rather than quietly changing the buffered one:

app.MapPost("/support/replies/stream", async (
    DraftReplyRequest request,
    SupportReplyService replies,
    HttpResponse response,
    CancellationToken cancellationToken) =>
{
    if (string.IsNullOrWhiteSpace(request.CustomerMessage))
    {
        response.StatusCode = StatusCodes.Status400BadRequest;
        await response.WriteAsJsonAsync(
            new { error = "customerMessage is required." },
            cancellationToken);
        return;
    }

    response.ContentType = "text/plain; charset=utf-8";

    await foreach (string text in replies.StreamAsync(
        request,
        cancellationToken))
    {
        await response.WriteAsync(text, cancellationToken);
        await response.Body.FlushAsync(cancellationToken);
    }
});

app.Run();

Flushing after every update favors immediate delivery in this small example. Model updates can contain only a few characters, so a production endpoint may buffer tiny deltas and flush them in larger chunks to reduce write and frame overhead. Measure the effect against the latency the UI needs.

Once the response starts, the endpoint cannot replace it with a neat JSON error. The client also needs framing when a stream carries citations, tool progress, usage, or a final outcome. Plain text works here only because text is the entire contract.

Server-Sent Events or another framed protocol will often suit a browser application better. The important decision is earlier: stream because the product benefits from partial output, not because the model API happens to expose it.

Both endpoints pass ASP.NET Core's request cancellation token through SupportReplyService to the model call. Keep that chain intact. Cancellation is cooperative: it cannot guarantee that provider work or billing stops immediately, and it cannot undo a side effect that has already happened.

Run the application on a fixed local URL:

dotnet run --urls http://localhost:5000

This starts the HTTP server. It does not print a model response in the terminal. Once the application is listening, send a buffered request from another terminal:

curl -X POST http://localhost:5000/support/replies \
  -H "Content-Type: application/json" \
  -d '{"customerMessage":"My order has not arrived yet. What should I do?"}'

The response is JSON with a text property. Its wording depends on the configured model. For the streaming route, tell curl not to buffer the response:

curl --no-buffer -X POST http://localhost:5000/support/replies/stream \
  -H "Content-Type: application/json" \
  -d '{"customerMessage":"My order has not arrived yet."}'

Leave room without building placeholders

I would not create interfaces for retrieval, evaluation, and tools on day one. I would know where each concern belongs before adding it.

Capability	Where it attaches	Boundary to preserve
Retrieval	A capability injected into `SupportReplyService` before message construction	Application code chooses allowed data and filters before anything reaches the model.
Tools	Per-operation `ChatOptions`, with function-invocation middleware in the client pipeline	A singleton client must not capture scoped authorization or data services.
Structured output	The service's response contract and model call	Validate the deserialized result before returning or persisting it.
Evaluation	Tests and production feedback around the feature contract	Keep deterministic application tests separate from model-quality evaluation.
Observability	The request trace and the `IChatClient` pipeline	Correlate the complete operation, including work outside the provider call.
Runtime controls	The endpoint, service, and client middleware	Carry one request budget through retries, tools, and model calls.

Treat this as an attachment map, not a backlog. Add a capability when the feature requires it. Retrieval should arrive as an authorized application dependency. Bind tools for the current operation. Start evaluation with a concrete quality question rather than an evaluator package.

What this foundation deliberately does not solve

This sample is an application shape, not a production-ready AI module. It does not yet define:

authentication or authorization for the endpoint
maximum input size and request rate
structured output and response acceptance rules
provider timeouts, recovery, or overload behavior
content-safety policy
tool authorization and side-effect handling
retrieval filters and data-access rules
evaluation datasets and release gates
telemetry redaction and retention
deployment health checks or fallback behavior

Those omissions are deliberate. Their design depends on the product, its data, and the consequences of failure. This foundation does not answer those questions. It gives their eventual answers a clear place in the application.

When this shape earns its keep

Use this foundation when a model experiment is becoming an application feature with an HTTP contract, shared configuration, operational requirements, or a likely path toward data and tools. A disposable console experiment can remain smaller and call IChatClient directly.

Before adding retrieval or tools, I would check these points:

Provider SDK types appear only at the composition edge.
Missing configuration stops the application at startup.
The endpoint accepts an application request, not an arbitrary prompt or model name.
One use-case service owns the instructions, messages, and result contract.
Messages and call options are local to one operation.
Buffered and streaming endpoints have explicit response contracts.
The caller's cancellation token reaches the model call.
The tracing provider subscribes to the MEAI activity source and has an exporter.
MEAI sensitive-data capture is disabled explicitly.

At that point, add the next real requirement where it belongs. The application does not need an AI platform first.

Sources

Microsoft Learn: Microsoft.Extensions.AI libraries
Microsoft Learn: Use the IChatClient interface
Microsoft Learn: IChatClient API and concurrency contract
Microsoft Learn: AddChatClient registration and default lifetime
Microsoft Learn: GetStreamingResponseAsync
Microsoft Learn: Options validation
Microsoft Learn: Use HttpContext and RequestAborted
Microsoft Learn: Sign in interactively with Azure CLI
Microsoft Learn: Azure OpenAI with keyless authentication
Microsoft Learn: OpenTelemetryChatClient and the experimental GenAI conventions
Microsoft Learn: OpenTelemetryChatClient.EnableSensitiveData
OpenTelemetry: Getting started with traces in ASP.NET Core
NuGet: Microsoft.Extensions.AI 10.10.0
NuGet: Microsoft.Extensions.AI.OpenAI 10.10.0
NuGet: OpenTelemetry.Extensions.Hosting 1.18.0
NuGet: OpenTelemetry.Exporter.OpenTelemetryProtocol 1.18.0

Indexing Your First Documents

Lukas Walter — Tue, 22 Sep 2026 15:30:00 +0000

This example writes a short support document into Qdrant, reads it back, and lets you run the same code again without creating another point for that document.

In Your First Qdrant Collection, we created support-documents-v1 with an unnamed dense vector and checked its embedding profile. That collection is ready for the Azure OpenAI path below. If you prefer local embeddings, the optional Ollama setup creates a separate collection on the same Qdrant server. Both paths use the same indexing code.

Continue from the collection example

Use the QdrantCollections console project from the collection article.

If you stopped the existing container, start it again:

docker start -a qdrant-local

Keep it running and use another terminal for the .NET commands. If you are starting here, follow the local setup and collection example first. It includes the Docker command, project creation, and collection validation.

Choose one embedding setup. Azure OpenAI continues with the existing collection; Ollama needs the preparation in the next section.

Setting	Azure OpenAI	Ollama locally
Model	`text-embedding-3-small`	`embeddinggemma:300m`
Dimensions	1,536	768
Distance	Cosine	Cosine
Collection	`support-documents-v1`	`support-documents-embeddinggemma-v1`
Profile ID	`support-text-v1`	`support-embeddinggemma-v1`

The local path uses a separate collection because EmbeddingGemma produces a different vector space. The later query must target the collection for its embedding profile.

Use Azure OpenAI in Microsoft Foundry

The package commands pin exact versions so you can reproduce the setup. The Microsoft.Extensions.AI dependencies stay at 10.9.0 to match the earlier embedding example. Newer compatible versions may also work.

In the project directory, add the Microsoft.Extensions.AI OpenAI integration used in Embeddings in .NET, plus the Azure OpenAI client:

dotnet add package Microsoft.Extensions.AI.OpenAI --version 10.9.0
dotnet add package Azure.AI.OpenAI --version 2.1.0

Deploy text-embedding-3-small in Microsoft Foundry. This example connects through the Azure OpenAI resource endpoint shown below. Keep its default 1,536-dimensional output. Set these environment variables in the terminal that runs the application:

Variable	Value
`AZURE_OPENAI_ENDPOINT`	Your Azure OpenAI resource endpoint, such as `https://your-resource.openai.azure.com/`
`AZURE_OPENAI_EMBEDDING_DEPLOYMENT`	The name you assigned to the `text-embedding-3-small` deployment, such as `support-embeddings`
`AZURE_OPENAI_API_KEY`	A key for that Azure OpenAI resource

Use the resource endpoint for this AzureOpenAIClient example, not a Foundry project URL or an endpoint with /openai/v1 appended. The client builds the deployment-specific request path. Keep the key outside source control. This example uses key authentication. The Azure SDK also supports Microsoft Entra credentials.

The embedding request sends the text to your Azure OpenAI deployment and incurs Azure usage charges. Start with the sample notes below.

Replace Program.cs with this setup block, then append the shared code under Index and read back the document. Together, the two blocks form the complete program.

using Microsoft.Extensions.AI;
using Azure.AI.OpenAI;
using System.ClientModel;
using Qdrant.Client;
using Qdrant.Client.Grpc;

const string collectionName = "support-documents-v1";
const string embeddingProfileId = "support-text-v1";
const int vectorSize = 1536;

string endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT")
    ?? throw new InvalidOperationException("Set AZURE_OPENAI_ENDPOINT.");
string deploymentName = Environment.GetEnvironmentVariable(
    "AZURE_OPENAI_EMBEDDING_DEPLOYMENT")
    ?? throw new InvalidOperationException("Set AZURE_OPENAI_EMBEDDING_DEPLOYMENT.");
string apiKey = Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY")
    ?? throw new InvalidOperationException("Set AZURE_OPENAI_API_KEY.");

using IEmbeddingGenerator<string, Embedding<float>> generator =
    new AzureOpenAIClient(new Uri(endpoint), new ApiKeyCredential(apiKey))
        .GetEmbeddingClient(deploymentName)
        .AsIEmbeddingGenerator();

Func<SourceDocument, string> prepareDocument = document => document.Text;

GetEmbeddingClient takes the Azure deployment name, which may differ from the model name. Confirm that the deployment uses text-embedding-3-small. Pointing the variable at another model does not make its vectors compatible.

The Azure OpenAI path embeds document.Text exactly as supplied. That keeps the existing support-text-v1 profile: plain text with the default 1,536-dimensional output. We store the title in the payload for display. The empty collection from the first article can be reused with this configuration. If you have already populated it through another endpoint, verify model-version compatibility before mixing vectors. Matching dimensions alone is insufficient.

Optional: generate embeddings locally with Ollama

Skip this section if you chose Azure OpenAI.

Install Ollama and start it. On a command-line installation, run ollama serve in a separate terminal. If the desktop app already runs the server, leave that instance running.

Download the embedding model:

ollama pull embeddinggemma:300m

EmbeddingGemma on Ollama requires Ollama 0.11.10 or later. After downloading the model, this path generates embeddings on your machine without an Azure subscription or cloud API key. Use the explicit 300m tag for this sample. Tags can change, so record the model digest shown by ollama list when reproducing results.

Before replacing the collection program, change its three constants to:

const string collectionName = "support-documents-embeddinggemma-v1";
const string embeddingProfileId = "support-embeddinggemma-v1";
const ulong vectorSize = 768;

Leave cosine distance and the rest of that program unchanged. Run dotnet run to create and validate the local model's collection. The indexing program below does not create collections. It expects this setup step to have completed successfully. The different name preserves support-documents-v1 if it already contains Azure OpenAI vectors.

Here, support-embeddinggemma-v1 declares embeddinggemma:300m, the full 768-dimensional output, cosine distance, and the document/query formatting described below. For the local Ollama path, the sample checks the declared profile ID, but it does not enforce model-artifact identity. If the tag later points to a different digest, the collection check still passes. Before deploying this pipeline, record the expected model digest in the profile and compare it with the locally installed model before writing or querying. Stop on a mismatch and review whether the new artifact requires reindexing.

Add OllamaSharp and pin the abstraction package used by this sample:

dotnet add package Microsoft.Extensions.AI --version 10.9.0
dotnet add package OllamaSharp --version 5.4.30

OllamaSharp already depends on Microsoft.Extensions.AI. The explicit reference pins it to 10.9.0.

Some .NET 10 SDK versions may report CS9057 for OllamaSharp's optional source generator. This example does not use its generated [OllamaTool] support.

Use this setup block instead of the Azure OpenAI block at the top of Program.cs. Append the same shared indexing code that follows.

using Microsoft.Extensions.AI;
using OllamaSharp;
using Qdrant.Client;
using Qdrant.Client.Grpc;

const string collectionName = "support-documents-embeddinggemma-v1";
const string embeddingProfileId = "support-embeddinggemma-v1";
const int vectorSize = 768;

using IEmbeddingGenerator<string, Embedding<float>> generator =
    new OllamaApiClient(
        new Uri("http://localhost:11434"),
        "embeddinggemma:300m");

Func<SourceDocument, string> prepareDocument = document =>
    $"title: {document.Title} | text: {document.Text}";

OllamaApiClient implements IEmbeddingGenerator<string, Embedding<float>> directly. The indexing code uses that interface in either setup.

The title: ... | text: ... prefix follows Google's document format. Later, a search query should use task: search result | query: ... with the same model. These prefixes belong to the embedding profile. They are not part of the original document text stored in the payload.

EmbeddingGemma has a 2,048-token input limit, including the title and document prefix. Ollama's embedding API defaults to truncate: true. This sample leaves that default in place, so an oversized input can lose its ending without the request failing. The short notes below fit, but before accepting larger documents, add token-aware size checks or chunking. You can also set the API's truncate option to false to reject oversized inputs.

Give the source a stable identity

The first document is a short note about stopping a background service. Its source ID is worker-shutdown, and it has one chunk named whole-document. Keeping the entire note in one point makes the first write easy to inspect.

The Qdrant point ID is a fixed UUID stored with that source record. Qdrant accepts UUIDs or unsigned 64-bit integers for point IDs, so a source slug such as worker-shutdown belongs in the payload rather than directly in the point ID field.

For this one-chunk example, each document has one persistent point ID. If a document later produces several chunks, each chunk needs its own stable point ID. The document_id and chunk_id payload fields describe the relationship. They do not enforce uniqueness.

I would keep the first example this small. A file parser and a chunking strategy can come after we have a document we can write and retrieve.

Index and read back the document

Append this code directly after your chosen setup block. Do not keep the old collection program in the same file.

using var executionTimeout =
    new CancellationTokenSource(TimeSpan.FromMinutes(2));
CancellationToken token = executionTimeout.Token;

using var client = new QdrantClient(host: "127.0.0.1", port: 6334);

CollectionInfo collection = await client.GetCollectionInfoAsync(
    collectionName, token);

VectorsConfig? config = collection.Config.Params.VectorsConfig;
if (config is null ||
    config.ConfigCase != VectorsConfig.ConfigOneofCase.Params ||
    config.Params.Size != (ulong)vectorSize ||
    config.Params.Distance != Distance.Cosine)
{
    throw new InvalidOperationException(
        $"Collection '{collectionName}' does not match the vector configuration.");
}

if (!collection.Config.Metadata.TryGetValue(
        "embedding-profile-id", out Value? storedProfile) ||
    storedProfile.StringValue != embeddingProfileId)
{
    throw new InvalidOperationException(
        $"Collection '{collectionName}' does not match profile '{embeddingProfileId}'.");
}

SourceDocument[] documents =
[
    new(
        PointId: Guid.Parse("19e8e940-0562-4b6b-bca5-1f99ea154f2e"),
        DocumentId: "worker-shutdown",
        Title: "Stop a background service cleanly",
        Text: "Pass the stopping token to asynchronous I/O in a BackgroundService. " +
              "When shutdown starts, let the operation observe cancellation " +
              "and finish within the host shutdown timeout.")
];

foreach (SourceDocument document in documents)
{
    if (string.IsNullOrWhiteSpace(document.Text))
    {
        throw new InvalidOperationException(
            $"Document '{document.DocumentId}' has no text.");
    }

    ReadOnlyMemory<float> vector = await generator.GenerateVectorAsync(
        prepareDocument(document), cancellationToken: token);

    if (vector.Length != vectorSize)
    {
        throw new InvalidOperationException(
            $"Expected {vectorSize} values, received {vector.Length}.");
    }

    var point = new PointStruct
    {
        Id = document.PointId,
        Vectors = vector.ToArray(),
        Payload =
        {
            ["document_id"] = document.DocumentId,
            ["chunk_id"] = "whole-document",
            ["title"] = document.Title,
            ["text"] = document.Text
        }
    };

    await client.UpsertAsync(
        collectionName: collectionName,
        points: [point],
        wait: true,
        cancellationToken: token);

    var stored = await client.RetrieveAsync(
        collectionName: collectionName,
        id: document.PointId,
        withPayload: true,
        withVectors: false,
        cancellationToken: token);

    if (stored.Count != 1 ||
        !stored[0].Payload.TryGetValue("text", out Value? storedText) ||
        storedText.StringValue != document.Text ||
        !stored[0].Payload.TryGetValue("title", out Value? storedTitle))
    {
        throw new InvalidOperationException(
            $"Read-back failed for '{document.DocumentId}'.");
    }

    Console.WriteLine($"Indexed '{document.DocumentId}' into '{collectionName}'.");
    Console.WriteLine($"Point ID: {document.PointId}");
    Console.WriteLine($"Title: {storedTitle.StringValue}");
    Console.WriteLine($"Text: {storedText.StringValue}");
}

public sealed record SourceDocument(
    Guid PointId,
    string DocumentId,
    string Title,
    string Text);

The program checks the existing collection before making an embedding request. A missing collection fails at GetCollectionInfoAsync. Create it with the setup program first. A wrong size, distance, or declared profile ID stops the write. Matching that ID does not prove which model artifact generated a vector. As in the collection article, these checks cover the simple unnamed dense-vector setup, not every schema option Qdrant supports.

GenerateVectorAsync returns ReadOnlyMemory<float>. The official Qdrant client accepts the array assigned to Vectors as the unnamed vector. Each point also carries the source text because an embedding cannot reconstruct the document when we need to display it later.

The upsert explicitly uses wait: true, so the read-back follows a completed update rather than merely an acknowledged request. This verifies that the point can be retrieved by ID and its text payload matches the source. The code also checks that the title field exists before printing it. It does not read the vector back or validate the other payload values. The check says nothing about search quality and does not require an HNSW index for this tiny collection.

The two-minute timeout bounds this small console run, including model loading for the local path. It is a sample setting, not a recommended ingestion deadline. The code passes the same cancellation token through the embedding and database calls.

Run it, then change the text

Run the program:

dotnet run

For the Azure OpenAI setup, expected output is:

Indexed 'worker-shutdown' into 'support-documents-v1'.
Point ID: 19e8e940-0562-4b6b-bca5-1f99ea154f2e
Title: Stop a background service cleanly
Text: Pass the stopping token to asynchronous I/O in a BackgroundService. When shutdown starts, let the operation observe cancellation and finish within the host shutdown timeout.

The Ollama setup prints support-documents-embeddinggemma-v1 instead. Open the local Qdrant dashboard, select that collection, and inspect the point's payload.

Run the program again with the same input. Qdrant replaces the point with that ID, so the collection still has one point if it was empty before this exercise. Now change the note's text and run it once more. The program generates a new vector and replaces the point's vector and payload together.

Keep the UUID unchanged during that edit. Calling Guid.NewGuid() inside the indexing loop would turn each run into another point. In a real source system, persist the assigned UUID or derive point IDs reproducibly from stable source and chunk identities.

An upsert replaces the existing point, including its payload. Send the complete payload you want to retain. This sample does not use a partial payload update. Also, the repeat run still calls the embedding model. Stable IDs prevent duplicate points, but they do not skip embedding costs or implement change detection.

Add two more documents

Replace the documents array with this version. The loop stays unchanged:

SourceDocument[] documents =
[
    new(
        Guid.Parse("19e8e940-0562-4b6b-bca5-1f99ea154f2e"),
        "worker-shutdown",
        "Stop a background service cleanly",
        "Pass the stopping token to asynchronous I/O in a BackgroundService. " +
        "When shutdown starts, let the operation observe cancellation " +
        "and finish within the host shutdown timeout."),
    new(
        Guid.Parse("81852b6d-d848-4a62-a777-e45b75f56755"),
        "http-client-lifetime",
        "Create HTTP clients through the factory",
        "Use IHttpClientFactory to create named HTTP clients. " +
        "The factory manages handler lifetimes so callers can dispose " +
        "their clients without creating a new connection pool each time."),
    new(
        Guid.Parse("fcad478c-cdae-428e-8784-c07d7ec9b8c3"),
        "bounded-work-queue",
        "Limit queued background work",
        "Use a bounded Channel to limit queued work. With FullMode set " +
        "to Wait, producers await available capacity instead of letting " +
        "an in-memory backlog grow without a limit.")
];

After a successful run, the collection contains these three points, plus any unrelated points already present. Repeating the run preserves the same three IDs.

Each pass through the loop embeds one note, writes it, and checks the stored text. If the second note fails, the first point remains in Qdrant. There is no transaction around the whole array. For a larger job, I would handle batching and recovery explicitly.

When to use this indexing path

Use this example to populate a local collection with a few short documents and verify that the application can write and retrieve their payloads. It also leaves a small corpus for the next step: embedding a question and searching the chosen collection.

Do not use the loop unchanged for long files or continuous source synchronization. Long documents need chunking and input-size checks. A changing corpus needs a policy for updates, removed documents, and interrupted runs. Removing an item from this array does not delete its existing Qdrant point.

When you add search, use the same embedding profile that produced these points. That includes the model identity, dimensions, distance, and the document/query formatting defined by the profile. Azure OpenAI queries stay with plain text. EmbeddingGemma queries need its retrieval query prefix. Then try a question whose answer you know is in one of the notes. Reading a point by ID cannot tell you whether semantic search will find it.

Sources

Qdrant: Points, IDs, and updates
Qdrant: .NET client
Microsoft Learn: Use the IEmbeddingGenerator interface
Ollama: Embeddings
Google: EmbeddingGemma model card and input formats

What Should You Actually Evaluate?

Lukas Walter — Thu, 17 Sep 2026 15:30:00 +0000

Evaluate whether the application delivered the promised outcome, within its permissions and runtime limits. A good answer is one part of that result.

A support assistant can write an accurate reply and save it to the wrong ticket. Or it can cite an obsolete policy and give advice that sounds well supported. Even a correct save can finish after the caller has timed out. The final text alone will not tell you which of these happened.

I would write down what the feature promises, then decide what evidence would show that it kept that promise. Choose the metrics from there.

Start with one outcome you can verify

Consider a support assistant with this contract:

Prepare a reply from the ticket and the support policy applicable to that ticket. Save it as a draft only when requested. Never send it.

The requester must have access to the ticket. If the available policy does not support an answer, the assistant should explain the gap and request review. If saving fails, it must not claim that a draft exists.

For this worked example, each evaluation case identifies the applicable policy version and the effective timestamp used to select it. "Current" means current for that case, not whatever document happens to be newest when the suite runs. Keep those reference facts fixed when comparing candidates. Historical cases keep their original policy version. Review any new or changed cases intended to represent a newer policy. The contract gives us these checks:

Boundary	What to verify	Evidence to inspect
Reply	Addresses the request, includes required conditions, makes no unsupported commitments	Ticket, applicable policy, generated reply
Retrieval quality	Surfaces the information needed to answer	Labeled relevant passages, retrieved IDs, final model context
Policy applicability	Uses the policy that applies to this ticket at the case's effective time	Policy version, effective timestamp, ticket facts
Authorization	Permits disclosure to the caller or model and any protected state change	Requester identity, resource metadata, authorization decisions, disclosure and write records
Tools	Saves only when requested, against the right ticket, without sending	Tool requests, authorization decisions, persisted state
Workflow	Returns an outcome consistent with what actually happened	Application result, dependency outcomes, state after execution
Safety	Keeps protected data and disallowed actions outside the permitted path	Adversarial cases, boundary decisions, exposed context and outputs
Runtime	Meets the caller's deadline and the backend execution budget	Caller and backend outcomes and timings, usage, retries, and tool calls

These checks need different methods. Some require judgment about language. Others are ordinary assertions over IDs, state, and side effects.

The eval-first article covers how to bring these checks into the development workflow. First, decide which behaviors need them.

Separate answer correctness from groundedness

A grounded answer follows the supplied context. That does not establish that the context is current, complete, or applicable to this customer.

Suppose an old policy says customers have 30 days to request a replacement. The current policy says 14 days, with an exception for damaged deliveries. An answer based on the old document might be perfectly grounded and still wrong for the task.

I would score these questions separately:

Criterion	Question
Correctness	Do the material claims and conclusion match the applicable policy and ticket facts?
Groundedness	Does the evidence supplied to the model support its factual claims?
Completeness	Does the reply include the conditions or exceptions the customer needs to act?
Relevance	Does it answer this customer's request?

A reference answer can help, but acceptable wording should not be limited to one sentence. Write the required facts and forbidden claims explicitly.

For a damaged-delivery case, a useful rubric might require the reply to identify the exception, ask for missing evidence required by the policy, and avoid promising approval before that evidence is reviewed. A fluent reply that promises an immediate replacement fails the task.

Evaluate abstention separately. On cases where the policy cannot answer the question, does the assistant acknowledge the gap? On answerable cases, does it unnecessarily refuse? Combining those groups can reward an assistant that avoids mistakes by declining everything.

Tone matters when the product requires it. For this assistant, I would prioritize unsupported commitments and omitted conditions before spending time distinguishing polished prose from slightly awkward prose.

Check retrieval before blaming the prompt

An incorrect answer leaves several possible causes. The relevant policy may be absent from the corpus, missed by search, removed during context assembly, or ignored by the model.

Inspect both the search results and the context actually passed to generation. Finding the right document is insufficient if the passage containing the exception never reaches the model.

With labeled relevant passages, recall at a chosen result count measures the fraction of relevant passages retrieval found. Precision measures the fraction of returned passages that are relevant. Define the relevance unit first: document, chunk, or required fact. Ten overlapping chunks from one paragraph should not look like ten independent pieces of useful evidence.

For the support example, I would also check whether the final context contains every policy fact needed to answer the ticket. A missing exception then counts as a failure even if the rest of the result list looks reasonable.

Check authorization and policy applicability separately from retrieval quality. A relevant passage from another tenant must not reach the caller or model without permission. A policy that was obsolete at the case's effective time is unsuitable evidence even if it ranks first.

To investigate a failure, compare the original context with a manually verified context in repeated, paired runs. Keep the model, instructions, generation settings, and scoring rubric fixed. A consistent improvement with verified context is stronger evidence of a retrieval or context-assembly problem than one successful rerun. If both conditions keep failing, inspect the instructions and generation behavior too. This experiment helps narrow the search. It does not establish a single cause or replace evaluation of the complete application.

Check what the tools changed

Calling SaveDraft looks correct when the user asks to save a reply. The tool name alone tells us little.

Check which ticket ID and body reached the tool, whether access was enforced, and what was stored. A valid JSON payload can contain the wrong ticket ID. A successful tool response can also be followed by an inaccurate message to the user.

For this assistant, the important assertions include:

A request to prepare a reply without saving produces no write.
A save request stores the intended body against the authorized ticket.
Retrying the same logical save operation with the same operation or idempotency identity does not create another draft. Independent save requests follow the product's rules for new drafts or revisions.
No execution sends a reply.
A failed or uncertain save does not produce a confirmed-success result.

Run side-effecting scenarios against isolated test resources or controlled tool implementations. A recording fake can show what the application attempted. An integration test is needed to verify the real adapter and storage behavior. These are ordinary software tests, but they still belong in the evaluation map when their results determine whether the AI feature fulfilled its contract.

Inspect the sequence where order matters. Authorization must succeed before protected contents reach the caller or model, and before a protected state change. The application may need to load a resource or its authorization metadata to make that decision. ASP.NET Core supports this through resource-based authorization. When trusted tenant identity is already available, constrain retrieval queries to that tenant before fetching content. Independent lookups can still run in either order. One successful trace should not become the only permitted sequence.

Microsoft's agent evaluation documentation distinguishes task outcomes from the process used to reach them, including tool usage. That is a useful distinction, but scoring tool messages does not replace checking the application's actual side effects.

Make failure outcomes part of task success

Successful task completion depends on the scenario. When policy evidence is missing, requesting review can be the correct outcome. When the user lacks access, refusing the operation is correct.

Build these cases from the feature's failure modes: empty retrieval, provider timeout, malformed output, unavailable storage, cancellation, and a write whose outcome is uncertain.

For each case, assert the expected application result and what may already have happened. If storage commits but the acknowledgment is lost, a generic retry test that only counts exceptions misses the important question: did recovery create a duplicate?

Keep two views in the report. One measures whether the system handled each scenario according to its contract. The other measures how often users received the useful outcome they wanted. A service that handles every outage gracefully can still be unavailable too often.

Keep safety failures visible

For this support assistant, safety cases should include a ticket containing instructions to send data elsewhere, a request for another tenant's ticket, and a request to send the reply despite the feature's draft-only contract.

Inspect the evidence at each boundary. Did the application apply the known tenant restriction to the query and authorize disclosure before ticket content reached the caller or model? Did instructions inside the ticket trigger an unauthorized tool attempt, and did the executor block it?

Record an unsafe proposal separately from an executed action. A blocked attempt shows that a control worked, while still revealing behavior worth investigating. A reassuring final answer does not erase an unauthorized disclosure or prohibited operation earlier in the workflow.

Content safety checks may also matter, depending on the feature. A classifier for harmful language cannot verify tenant isolation or write authorization. Those require application-level evidence.

I would make any observed unauthorized disclosure or prohibited side effect block this feature's release. Keep those failures visible rather than averaging them with relevance or tone scores. Zero observed failures in a test set is useful evidence, not proof that no other input can cross the boundary.

Measure latency and cost across the complete task

Measure the caller's wait and outcome separately from backend completion time and outcome. A caller might time out after 12 seconds while the backend saves the draft at 19 seconds. The timeout is a user-visible failure. The late save is a side effect that recovery must account for. Correlate both records with the same logical operation. Include retrieval, retries, tool execution, and persistence in the backend measurement.

For an interactive feature, inspect median and tail latency, such as the 95th percentile, under stated load. For streaming, distinguish the first visible output from completion of the usable result. A quick opening sentence does not mean the draft is ready.

Report failures and timeouts beside latency. Separate user-initiated cancellation, deadline or budget cancellation, and system or dependency cancellation. Record an unknown cause when you cannot establish it. A user choosing to stop is different from an execution exhausting its budget. Show which outcomes the latency sample includes so a candidate cannot look faster merely by dropping slow, failed requests from the report.

Cost should cover the boundaries the feature pays for. Name model-only estimates accordingly, and keep missing usage visible. Include failed attempts and retries rather than counting only the final successful call.

For cost per successful task, first decide which outcomes count as success. If the denominator counts cases handled according to contract, a correct access denial counts. If it counts useful user outcomes, that denial does not. Report the two ratios separately when you need both, alongside their success rates and per-execution cost. A ratio alone can hide expensive outliers or changes in the case mix.

Choose limits from the product promise. The runtime budgets article covers enforcement. Evaluation checks whether realistic workloads and failures stay within those limits while still producing useful outcomes.

Match the evaluator to the evidence

Use deterministic checks for schema validity, identifiers, permissions, persisted state, and counts. Use an explicit rubric with human review or a model judge for semantic questions such as whether the reply omitted a policy condition.

In .NET, Microsoft.Extensions.AI.Evaluation supplies relevance, completeness, and groundedness evaluators that work with existing test infrastructure. Its CompletenessEvaluator compares the response against supplied ground truth. For groundedness, explicitly pass the grounding evidence from the final context the generation model actually received through GroundednessEvaluatorContext. The evaluator does not recover that context automatically. Decide what evidence and pass criteria you need before choosing either.

Give the judge the evidence its rubric asks it to assess: the supplied context for groundedness, or the applicable reference facts for correctness. You still need to inspect storage to verify that a draft exists. The phrase "Draft saved" proves nothing about database state.

Review a sample of judge decisions against human judgments, including failures and borderline cases. Record disagreements and tighten ambiguous rubrics. If the evaluator errors or lacks required evidence, report the result as unscored. Require a valid result that meets the pass criteria at the release gate. Missing or inconclusive results need an explicit decision.

Keep the component results beside the result for the complete workflow. They should let you trace a failed task back to a missing passage, an unsupported claim, or an incorrect write without losing sight of what the user received.

Decide what would block the release

A single overall score hides the reason a candidate improved or regressed. Report results by scenario: routine answers, policy exceptions, missing evidence, denied access, and dependency failures. Include the number of cases in each group.

For example, a higher average answer score is not enough to accept a candidate that now invents commitments on missing-policy cases. A cheaper candidate is not an improvement if it saves the wrong draft. Define these acceptance rules before reviewing the candidate's results.

Compare candidates on the same evaluation protocol and inspect per-case changes. Repeat cases where model variation could change the decision. Keep results tied to their versions using the evaluation provenance guidance rather than relying on a score copied into a pull request.

Production evaluation can reveal cases the offline suite missed. Track corrections, abandoned tasks, and sampled quality judgments with their context. User acceptance alone does not establish correctness. The production evaluation tip covers how to collect those judgments outside the user response path.

When to use this approach

Use this system-level evaluation map when an AI feature retrieves information, invokes tools, changes state, or has meaningful failure and runtime constraints. Start with one workflow and the failures that would change a release decision. Add dimensions as the feature gains responsibilities.

When a smaller evaluation is enough

A low-risk text transformation may need only output validity, preservation of required meaning, and latency. It has no retrieval or tool path to evaluate. A local experiment may begin with a few manually reviewed examples, provided that limited evidence is not treated as production readiness.

For your next change, write one sentence describing the promised outcome. List the unacceptable results, identify the evidence that would expose each one, and choose a check for each. Then add the quality and runtime measures needed to compare acceptable candidates.

For the support assistant, that means checking the reply against the applicable policy, inspecting the saved draft, and confirming what the caller saw. Those results give you something concrete to review before shipping.

Sources

Microsoft Learn: Resource-based authorization in ASP.NET Core
Microsoft Learn: Agent evaluators in Microsoft Foundry
Microsoft Learn: The Microsoft.Extensions.AI.Evaluation libraries
Microsoft Learn: CompletenessEvaluator
Microsoft Learn: GroundednessEvaluatorContext constructor

Dependency Injection for AI Components

Lukas Walter — Tue, 15 Sep 2026 15:30:00 +0000

An IChatClient is expected to support concurrent requests unless its implementation documents otherwise. A tool bound to one user's database context cannot safely follow it into a singleton registration.

I would check that boundary before adding more registrations. A tool delegate can quietly retain a caller's state for as long as the object that holds it.

This article assumes you already know the .NET DI container. It uses Microsoft.Extensions.AI to show where chat clients, prompts, retrieval, and tools belong in the dependency graph. The lifetime decisions also apply when an agent framework sits above the client.

Start with the objects that hold state

Consider a support assistant with a shared model connection and an order lookup tool. The lookup reads through a scoped service that knows which orders the current caller may access.

If startup code captures that service in a tool delegate and saves the tool on a singleton, the delegate retains a reference to the scoped service. If the service came from a temporary scope, later calls can reach it after that scope has disposed its disposable services. Holding a reference does not prevent that disposal.

The intended graph is small:

Application lifetime
    IChatClient pipeline
    immutable prompt templates

Request or job scope
    support service
        -> shared IChatClient
        -> authorized retrieval / order lookup

One model operation
    fresh messages
    fresh ChatOptions
    tools bound to this scope

The service-layer article explains which behavior belongs behind the support feature. Once those dependencies exist, each needs an owner and a lifetime.

Component	Starting lifetime	What changes the decision
Chat client and shared middleware	Singleton	Every component must support concurrent use and avoid retaining caller state.
Fixed prompt template	Constant or immutable singleton	A renderer that depends on scoped services or retains scoped state must follow that scope.
Application service	Scoped when it uses scoped capabilities	A stateless service with suitable dependencies can be transient.
Retrieval or tool that depends on scoped caller or database state	Scoped	Follow the lifetime and concurrency limits of those dependencies.
Messages, call options, bound tool list	Local to one operation	Conversation history needs explicit storage and ownership across operations.

IChatClient expects concurrent use unless an implementation says otherwise. Its arguments have a different contract: implementations may mutate messages or options. Allocate those per operation. The interface remarks describe both points.

Register one client at the composition root

The following sample uses .NET 10, Microsoft.Extensions.Hosting 10.0.11, Microsoft.Extensions.AI 10.9.0, and OllamaSharp 5.4.30. Ollama keeps credentials out of this local example. Wire a hosted provider's adapter and authentication at the composition root. The client factory can resolve a separately registered application-wide credential.

Create a console project:

dotnet new console -n AiComposition -f net10.0
cd AiComposition
dotnet add package Microsoft.Extensions.Hosting --version 10.0.11
dotnet add package Microsoft.Extensions.AI --version 10.9.0
dotnet add package OllamaSharp --version 5.4.30

Replace Program.cs with this startup code, then append the types from the next two sections:

using Microsoft.Extensions.AI;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using OllamaSharp;

using IHost host = Host.CreateDefaultBuilder(args)
    .UseDefaultServiceProvider(options =>
    {
        options.ValidateScopes = true;
        options.ValidateOnBuild = true;
    })
    .ConfigureServices((context, services) =>
    {
        services.AddOptions<SupportModelOptions>()
            .Bind(context.Configuration.GetSection("AI:Support"))
            .Validate(o => Uri.TryCreate(
                    o.Endpoint, UriKind.Absolute, out var uri)
                && (uri.Scheme == "http" || uri.Scheme == "https"),
                "AI:Support:Endpoint must be an HTTP(S) URI.")
            .Validate(o => !string.IsNullOrWhiteSpace(o.Model),
                "AI:Support:Model is required.")
            .ValidateOnStart();

        services.AddSingleton<IChatClient>(sp =>
        {
            var settings = sp
                .GetRequiredService<IOptions<SupportModelOptions>>()
                .Value;

            IChatClient client = new OllamaApiClient(
                new Uri(settings.Endpoint), settings.Model);

            return client.AsBuilder()
                .UseFunctionInvocation()
                .Build();
        });

        services.AddScoped<IOrderStatusReader, DemoOrderStatusReader>();
        services.AddScoped<SupportAssistant>();
    })
    .Build();

await host.StartAsync();

await using (var scope = host.Services.CreateAsyncScope())
{
    var assistant = scope.ServiceProvider
        .GetRequiredService<SupportAssistant>();

    Console.WriteLine("AI service graph resolved.");

    if (args.Contains("--ask"))
    {
        using var timeout = new CancellationTokenSource(
            TimeSpan.FromSeconds(30));

        Console.WriteLine(await assistant.AnswerAsync(
            "What is the status of order DEMO-42?",
            timeout.Token));
    }
}

await host.StopAsync();

public sealed class SupportModelOptions
{
    public string Endpoint { get; set; } = "";
    public string Model { get; set; } = "";
}

The factory constructs the complete client pipeline. It resolves application-wide settings, but never an order reader or current user. UseFunctionInvocation supplies the execution mechanism. The feature supplies tools for each call. Microsoft's tool-calling example shows this same separation between the pipeline and call options.

I use an explicit singleton factory to make ownership visible. AddChatClient is also available when you want the library's DI builder API.

ValidateOnStart checks the configured values when the host starts. It does not prove that the endpoint is reachable or that the model supports tools. The options documentation explains that validation lifecycle. This client reads its settings once. Changing configuration later will not rebuild it. Use a deliberate replacement strategy if you need live model changes.

Bind tools inside the operation

Append this service to Program.cs:

public sealed class SupportAssistant(
    IChatClient chatClient,
    IOrderStatusReader orders)
{
    private const string Instructions = """
        Help with order status. Use lookup_order_status for order facts.
        If the lookup returns no result, say the status is unavailable.
        Do not invent a status.
        """;

    public async Task<string> AnswerAsync(
        string question,
        CancellationToken cancellationToken)
    {
        ArgumentException.ThrowIfNullOrWhiteSpace(question);

        List<ChatMessage> messages =
        [
            new(ChatRole.System, Instructions),
            new(ChatRole.User, question)
        ];

        ChatOptions options = new()
        {
            Tools =
            [
                AIFunctionFactory.Create(
                    orders.FindVisibleAsync,
                    name: "lookup_order_status",
                    description: "Look up an order status by order number.")
            ]
        };

        ChatResponse response = await chatClient.GetResponseAsync(
            messages, options, cancellationToken);

        return response.Text;
    }
}

The method creates a tool bound to this scope's orders instance. The caller awaits the entire operation before disposing its scope, so the tool can use its scoped dependencies until the call completes. A streaming implementation must defer scope disposal until enumeration finishes too.

The fixed instructions need no DI registration. If several features share an immutable template catalog, registering it as a singleton can be reasonable. Rendered messages still belong to the operation. Putting them on the catalog would mix one caller's content with another's.

A scoped tool is not automatically safe for parallel invocation. For example, two tools in one scope could share the same EF Core DbContext. Keep their execution sequential or give each parallel operation its own correctly initialized scope and context. FunctionInvokingChatClient.AllowConcurrentInvocation defaults to false, so the middleware executes function calls within a request sequentially. Separate requests can still invoke their tools concurrently, as the concurrency remarks explain.

This sample returns plain text to keep the lifetime example readable. A real support feature still needs response acceptance rules and a tool-loop budget. The prompt cannot enforce either requirement.

Give retrieval and tools their application dependencies

Append a small lookup contract and a demo implementation:

public interface IOrderStatusReader
{
    Task<string?> FindVisibleAsync(
        string orderNumber,
        CancellationToken cancellationToken);
}

public sealed class DemoOrderStatusReader : IOrderStatusReader
{
    public Task<string?> FindVisibleAsync(
        string orderNumber,
        CancellationToken cancellationToken)
    {
        cancellationToken.ThrowIfCancellationRequested();

        return Task.FromResult<string?>(
            orderNumber == "DEMO-42" ? "Dispatched" : null);
    }
}

The demo contains one public fixture and has no authentication or database. In an application, replace it with a service that checks the authenticated caller's access before loading an order. Keep that service scoped when it depends on scoped caller context or a scoped database context. The tool accepts an order number. Trusted application context supplies identity and tenant scope.

Register an authorized retrieval service the same way when it depends on a scoped database context or caller context. Its lower-level vector client or embedding generator may be reusable across scopes if the implementation supports that. A shared connection does not require a shared authorization context. Database access alone does not dictate a scoped lifetime: a service can use IDbContextFactory<TContext> to create and dispose a context for each operation. Its own lifetime still depends on the other state and dependencies it holds.

Scope also has a precise limit: in ASP.NET Core, it normally means one HTTP request. It does not mean one conversation. Store conversation history under an authorized conversation ID and load the allowed messages for each operation. A scoped List<ChatMessage> will disappear with the request. A singleton list will be shared across callers.

Let the owner dispose the client

The DI container owns the client returned by the singleton factory. At shutdown, disposing the outer pipeline also disposes its inner client. Consumers should await their work and leave disposal to that owner. See the .NET disposal guidance and DelegatingChatClient.Dispose.

Do not put an injected IChatClient in a using block inside AnswerAsync. That would dispose the shared instance while other requests may still need it. For the same reason, avoid separately registering and owning the inner client when a disposing wrapper already owns it.

Creating a scope around a tool delegate and then returning the delegate does not transfer ownership. The delegate still refers to the scoped instance after disposal. Keep the operation inside the scope instead.

A background worker needs to create that scope explicitly. Hosted services have no automatic per-job scope. Resolve the feature inside a fresh async scope for each job and await completion before disposing it, as in Microsoft's scoped worker example. A new scope does not supply an HTTP user. Rebuild execution context from validated job identity data and re-evaluate authorization where permissions may have changed since enqueueing. Stored identity alone does not establish current permission.

Check composition without asking a model

Set the endpoint and model through configuration:

export AI__Support__Endpoint=http://localhost:11434
export AI__Support__Model=llama3.1
dotnet run

Without --ask, the sample starts the host and resolves the scoped feature. It constructs the client without sending a model request. You should see AI service graph resolved. among the host logs.

Remove the model value and startup should fail options validation. Temporarily register SupportAssistant as a singleton and scope validation should reject its scoped order reader. These checks answer useful composition questions before a provider enters the picture.

ValidateOnBuild cannot inspect arbitrary code inside registration factories. Resolve the feature graph as well, as the sample does, and test any factory branches your application uses. Avoid calling BuildServiceProvider inside registration code to get dependencies early. Use the provider passed to the registration factory.

For an optional live check, run Ollama locally, pull llama3.1 (or whatever model you like), then run dotnet run -- --ask. That exercises the model path. If the model is missing, Ollama returns a 404 error. This sample does not pull it automatically. The live check is separate from the composition check and does not establish answer quality.

When adding a second client, choose it explicitly at the feature boundary with a keyed registration or a typed adapter. Keep keys fixed in composition code. A user-supplied string should not choose a privileged model connection.

When to use this structure

Use a shared client with scoped application capabilities when an AI feature reads caller-specific data or exposes tools backed by scoped services. Build fresh messages and tool bindings for every operation, and dispose the scope only after all work finishes.

Before shipping, trace one tool delegate back to its target object. Check who owns that object, when it is disposed, and whether two invocations can reach it concurrently. That small review catches problems a successful chat response will never reveal.

When not to add more DI machinery

A short console experiment can construct and dispose its client directly. A fixed prompt can remain a constant. Add a factory, interface, or registration extension when it owns a real variation or lifecycle decision.

If the lifetime is hard to choose, look at the fields and captured delegates. A stored message list or a reference to the current caller often explains why an otherwise reusable component cannot be shared.

Sources

Microsoft Learn: IChatClient concurrency and argument-mutation contract
Microsoft Learn: Tool calling with IChatClient
Microsoft Learn: FunctionInvokingChatClient and tool concurrency
Microsoft Learn: Dependency injection and disposal ownership
Microsoft Learn: DelegatingChatClient.Dispose
Microsoft Learn: Options validation and ValidateOnStart
Microsoft Learn: Using scoped services in a background worker
Microsoft Learn: Creating EF Core contexts with IDbContextFactory
Ollama: API errors, including missing models
NuGet: Microsoft.Extensions.Hosting 10.0.11

Human Approval as a System Boundary

Lukas Walter — Thu, 10 Sep 2026 12:30:00 +0000

A user approves a support email. Before the worker sends it, the agent rewrites the body and adds another recipient. The application still has Approved = true.

Human approval is an authorization control at the execution boundary. It must bind the decision to an exact operation: its destination, payload, and execution conditions. The flag says nothing about which version the user accepted.

I would design that agreement before adding the approval button. Once the reviewer clicks it, the worker needs enough stored information to finish without asking the model what the user meant.

Decide where human judgment helps

Suppose a support assistant can read a ticket, draft a reply, and send it. Company policy requires a support lead to review outbound replies because they can make commitments on the company's behalf. The application enforces that requirement.

Authorization first establishes which operations the requester may propose and for which resources. Application policy then chooses Allow, Deny, or RequireApproval. Approval is part of authorization: a support lead can authorize a reply that the requester cannot send alone. That decision must stay within the operation's permitted scope and cannot override a hard policy denial. This example uses one reviewer. Workflows with several reviewers need separate decisions and an aggregate policy result.

The broader enforcement design is in Trust Boundaries Around AI Features. This article follows one operation through review and dispatch.

The useful human decision is whether this reply should go to this customer. Routine authorized lookups need no interruption. The approval tip covers that distinction, though a read that exports protected data can still require stricter controls.

Prepare something the reviewer can approve

The assistant should finish the permitted preparation and validation before asking. That means a reply with resolved recipients and the intended attachments. Asking "May I send a response?" before one exists leaves the consequential details undecided.

Store the proposal as an immutable operation revision and use it for both the review screen and executor. That lets the application check the content and destinations it submits to the provider against what the person approved. Downstream processing and the recipient's mail client can still change the delivered message or its appearance.

For a hypothetical ticket, the reviewer might see:

Field	Value shown for review
Operation	Send a customer reply
Account and environment	Example Support, production
Ticket	T-1842, version 17
Sender	support@example.com
Recipients	To: customer@example.net; no CC or BCC; provider envelope has the same single destination
Subject	Replacement for order O-731
Body	Full text, including the promised replacement date
Attachments	replacement-details.pdf, with access to the exact stored file
Approval scope	Authorize one logical dispatch of this revision
Decide by	2026-09-06 10:15 Europe/Berlin (UTC+02:00)
Commit for dispatch by	2026-09-06 10:20 Europe/Berlin (UTC+02:00)

These fields come from the stored operation. A generated explanation can help the reviewer understand them, but must not replace or hide them. The review screen needs safe text rendering and access to the attachments.

Store attachments as immutable versions or protected snapshots. Finish transformations that can change approved details before review. Later transformations, such as provider footers or tracking-link rewriting, need fixed behavior within the approval scope or documented treatment outside it. A filename alone cannot bind approval to a file's contents.

Show actual addresses as well as display names. Header recipients and the SMTP or API envelope can differ, so the review must cover both. If approval depends on individual recipients, resolve mutable groups or aliases into fixed destinations before review. Approving a group address alone cannot establish who its members will be later.

This follows the transaction-review principle in the OWASP Transaction Authorization Cheat Sheet: show the significant operation data, protect it from modification, and enforce the decision on the server.

Bind the decision to a stored revision

An approval record needs a stable ID and a reference to the immutable operation revision. Record the requester and tenant, policy version, reviewer requirement, and decision. Store DecideBy and ExecuteBy as absolute UTC instants. Display the reviewer's time zone and offset. Keep the logical operation ID stable across attempts to dispatch that revision.

An executor deployment must preserve the meaning of stored operations. Record the operation type and a schema or handler-contract version when backward-compatible interpretation is not otherwise guaranteed. The review client submits identifiers, and the server loads the stored revision rather than accepting a replacement payload.

The model supplies candidate content. Authenticated context supplies identity and tenant scope. Application policy sets the approval requirement. The review request only identifies the stored operation and decision:

public enum ReviewDecision
{
    Approve,
    Reject
}

public sealed record SubmitReview(
    Guid ApprovalId,
    long ExpectedOperationRevision,
    ReviewDecision Decision);

This DTO sketches the request. The endpoint must validate the decision, load the record inside the authenticated tenant, and authorize the reviewer, including any rule against self-approval. The client supplies neither reviewer identity nor replacement tool arguments.

Accept the decision only while the record is pending, the operation revision matches, and server time is strictly before DecideBy. Enforce this at the write, with separate approval-record concurrency control. ExpectedOperationRevision identifies the payload, not the ticket version, policy version, or concurrency token. Concurrent approve and reject submissions must commit at most one decision. Knowing an approval ID grants no authority to use it.

If the reviewer edits the body, save a new operation revision and supersede the old approval request. Present the final edited payload for confirmation. An explicit "Save and approve" flow can combine those steps only if it binds the decision to the exact edited revision the reviewer sees.

After approval, the worker loads that revision directly. Asking the model to reconstruct the email from conversation history would create another proposal.

Give waiting its own lifecycle

A reviewer may answer after a restart. Persist the pending request, return control instead of keeping a model request open, and provide a page or endpoint for checking its status. Give resumed execution its own runtime budget.

CommitForDispatch is the durable business transition that admits one logical operation to the dispatch process. A worker later claims an individual attempt. That gives the worker ownership of work already authorized by this gate:

Decision:
Pending -> Approved | Rejected | DecisionExpired | Superseded | Withdrawn

CommitForDispatch requires:
decision == Approved
and operation revision is still current and intact
and server time < ExecuteBy
and current policy and resource conditions permit execution

An approved decision stays in the history even when ExecuteBy passes or a new proposal supersedes its operation revision. If the operation has not yet been admitted, it can no longer pass this gate. The record still shows who approved it. DecisionExpired means nobody decided in time. Withdrawn means an authorized requester withdrew the pending review without submitting a replacement.

Withdrawing the pending review prevents later admission. If the withdrawal ends the proposed operation, its logical state becomes Cancelled. The review remains Withdrawn to show that no reviewer decided.

The deadlines can differ: approval at 10:14 may authorize admission at 10:19, but not at 10:20. Check each deadline synchronously at its transition. A delayed cleanup job must not extend either one.

ExecuteBy bounds admission, not when the provider receives the request. An operation admitted at 10:19 could remain queued past 10:20. If freshness must hold until provider handoff, enforce a separate deadline and the relevant freshness checks before starting the first provider call. If nobody approves, the reply stays unsent. Reminders cannot authorize it.

Recheck before committing to dispatch

At 10:00, a support lead approves the reply for ticket version 17. At 10:02, another employee corrects the customer's email address. At 10:03, the worker picks up the approved send.

The stored decision is still useful history. The worker must now establish whether it can execute under the approved conditions.

Before admission, the application should:

Load the approved operation and check its integrity, revision, decision, and ExecuteBy deadline.
Reload the affected resources through the trusted tenant path and apply the configured requester and reviewer authority rules.
Compare relevant resource versions and evaluate current policy, including whether it still accepts the recorded approval.
Persist CommitForDispatch atomically with the approval and relevant resource checks that share its transactional store.

In this example, a relevant ticket change blocks the old send and requires a refreshed proposal. Do not silently send to the new address under an approval for the old one. If you choose a narrower version check than the entire ticket row, document which fields invalidate approval and make sure changes to them advance that version.

For this support workflow, I would require both requester and reviewer to retain their relevant authority until admission commits. Another workflow may accept the request and approval as historical acts, then execute under organizational or service authority. Each identity needs an explicit policy rule, and the executor needs authority to act. A current hard denial still stops execution.

When approval and relevant resource state share a transactional store, the admission transaction can conditionally check both. When the business effect and result also live in that transaction, they can commit together. EF Core concurrency tokens make updates conditional on the original version and report conflicts. They do not pull another service's state into the transaction. Blindly retrying a conflict with new values would discard the approved precondition.

If the ticket belongs to another service, use its conditional-operation or reservation support where available. A local admission transaction cannot atomically validate that remote ticket or an independent identity service. Without coordination, a time-of-check-to-time-of-use window remains. Document the risk and restrict or block the operation if policy cannot tolerate it.

Sending the email is external even when ticket and approval share a database. In this example, CommitForDispatch commits the business decision to dispatch the fixed revision. Coordinate conflicting local changes through that point. If policy must allow cancellation until provider handoff, the dispatch path needs an additional coordinated cancellation check. A queued record alone cannot provide that guarantee.

Once dispatch has begun, revoking approval cannot reliably recall the email. The UI should report the actual execution state and offer only cancellation or recovery actions the system can still honor.

Approval does not make retries safe

Two clicks on "Approve" must not create two logical dispatches. Record one decision, admit the operation once, and return its existing status when a request repeats. Each worker attempt gets a separate ID under that same logical operation.

Logical operation:
NotStarted --CommitForDispatch--> InProgress
NotStarted -> Blocked
NotStarted -> Cancelled
InProgress -> Succeeded | Failed | OutcomeUnknown
InProgress -> Blocked | Cancelled (before first provider handoff)
OutcomeUnknown -> Succeeded | Failed (new evidence)
OutcomeUnknown -> InProgress (safe retry permitted)

Succeeded means confirmed provider acceptance of the approved request. Mailbox delivery and bounce tracking are separate. SMTP also distinguishes acceptance from later delivery or failure notification in RFC 5321, section 6.1. Failed means failure is established and the workflow will make no more attempts. A definitely failed network attempt does not by itself fail the logical operation. A new attempt may run while the logical operation remains InProgress.

Blocked means policy, freshness, or preconditions prevent the first provider call. Cancelled records a supported cancellation that succeeds before handoff. Both transitions must coordinate with dispatch so no worker can still start that call. Once a call may have been accepted, a policy change or cancellation request cannot establish either outcome. Keep an unresolved call in OutcomeUnknown until evidence establishes what happened.

Suppose the provider accepts the request and the worker crashes before saving the response. After restart, local state cannot establish acceptance. Blindly scheduling another attempt could submit the email again. Worker ownership does not resolve that uncertainty: an expired lease is no proof that a provider call stopped.

Recovery needs durable provider evidence or an idempotency contract that makes another attempt safe. Reuse the logical operation's idempotency key with the same payload, within the provider's scope and retention window. A new attempt ID must not become a new key. An outbox cannot supply deduplication that the provider lacks.

A saved provider reference or queryable client operation ID may establish what happened. The lost response may also have contained the only usable reference. Without evidence or safe idempotency, keep OutcomeUnknown and stop automatic retries. Investigation may never resolve it.

Admission before ExecuteBy can permit safe recovery of the same logical operation afterward if policy allows it. Current hard denials may stop future attempts, but they cannot undo a call the provider may already have accepted or change OutcomeUnknown to Blocked.

Retries Are Not a Recovery Strategy covers attempt recovery in more detail. In this example, the approval record establishes permission to dispatch the reply. It cannot tell the restarted worker whether the provider accepted the request whose response was lost.

Test what happens after the click

I would test the approval service and executor directly, without a model in the test. A prompt change should not affect any of these results:

Test	Required result
Bypass approval or submit as a reviewer from another tenant	Reject without sending or exposing the operation
Edit or tamper with the reviewed payload	Require approval of a valid new revision, or reject an integrity failure
Duplicate approval, or concurrent approve and reject	Commit one decision and admit at most one logical dispatch
Approve at or after `DecideBy`, or after supersession	Reject even if cleanup has not run
Withdraw a pending review, then approve from a stale review page	Reject the decision and never admit the operation
Approve in time; attempt admission at or after `ExecuteBy`	Preserve `Approved` history; reject admission
Requester or reviewer loses required authority before admission	Block under this example's policy; test historical-authority policies separately
Change local ticket state between validation and admission	Reject the stale version in the shared transaction
Change remote state where conditional execution is supported	The remote operation rejects the stale version
Change remote state after final validation but before local admission, with no coordination primitive	Block if policy requires atomic freshness; otherwise assert admission succeeds against the validated snapshot despite the change, as policy explicitly permits
Restart after a provider call whose result was not saved	Recover using durable evidence or valid idempotency; otherwise preserve `OutcomeUnknown` and do not resend
Deny the operation or miss its configured pre-handoff deadline after admission, but before the first provider call	Record `Blocked` and prevent the first provider call
Cancel an admitted operation through a supported path before first handoff	Record `Cancelled` only after preventing dispatch
Deny or request cancellation while a provider call has an unknown outcome	Stop future attempts; preserve `OutcomeUnknown` until evidence resolves the call

For diagnosis, record approval, logical operation, and attempt IDs alongside revision, policy version, identities, deadlines, and state transitions. Keep sensitive payloads in protected storage with appropriate retention rather than copying email bodies into general application logs.

Also inspect the review experience. Track waiting time, edits and rejections, and approvals that miss ExecuteBy. Those observations help identify a queue that nobody can keep up with. An approval rate alone says little about whether anyone had enough context to review the action.

When human approval helps

Use it when an authorized person can assess a concrete operation before its consequences occur. A support lead can check whether the promised replacement is justified and whether the reply says what the company intends to promise. Give that reviewer enough context and time to decide, and define what the application does when nobody responds.

Do not add it to routine work already covered by an explicit automatic-execution policy. Do not offer an approval button for an action policy forbids. If the reviewer cannot inspect the payload or understand its effect, reduce the operation's scope or keep execution in an established manual process.

For an existing agent feature, start with one consequential tool. Write down the exact operation a person will review, what changes invalidate that decision, and how the executor proves it is carrying out the approved revision. Then test a changed payload and a worker crash: the first must block the old approval, and the second must preserve enough state to investigate or safely resume the original operation.

Sources

OWASP: Transaction Authorization Cheat Sheet
Microsoft Learn: Handling concurrency conflicts in EF Core
IETF: RFC 5321, section 6.1: Reliable delivery and replies by email

Your First Qdrant Collection

Lukas Walter — Tue, 08 Sep 2026 12:30:00 +0000

In Embeddings in .NET, I stopped at the vector-store boundary. The article generated embeddings and treated the model, dimensions, distance, and input preparation as one contract. It did not create an index for those vectors.

This article picks up there with a smaller goal: run Qdrant locally, connect from .NET, create one collection, and verify the parts of its configuration that the application depends on. Nothing gets indexed yet.

What a collection owns

A Qdrant collection is the container for points that you want to search together. Each point can hold one or more vector representations and an optional payload. This article uses one unnamed dense vector. Later, the payload can carry application data such as the source ID, text, language, or tenant ID.

This first example will not write any points. It only creates the collection that will receive them later.

For one unnamed dense vector, the collection schema needs two decisions:

vector size
distance function

Both come from the embedding contract. The example uses 1,536 dimensions and cosine distance because that matches the default text-embedding-3-small profile from the previous article. If your model returns 768 or 3,072 values, use that number instead. If the model documentation recommends a different distance function, follow it.

Matching the vector length is necessary, but it does not prove that two vectors are compatible. Qdrant can reject a vector with the wrong length. It cannot tell whether a correctly sized vector came from the wrong embedding model.

Run Qdrant locally

The example uses Qdrant 1.19.0 in Docker. Create a named volume so the collection remains available after the container stops:

docker volume create qdrant_storage

docker run --name qdrant-local \
  -p 127.0.0.1:6333:6333 \
  -p 127.0.0.1:6334:6334 \
  -v qdrant_storage:/qdrant/storage \
  qdrant/qdrant:v1.19.0

Qdrant exposes its HTTP API and dashboard on port 6333. The gRPC API listens on 6334, which is the port used by the .NET client below. Both ports are bound to 127.0.0.1, so they are available only from the Docker host. Without that address, Docker publishes ports on every host interface by default.

Leave that terminal open and use a second one for the .NET commands. Open http://127.0.0.1:6333/dashboard to confirm that the server is available. With a newly created, empty volume, the Collections page should initially be empty. docker volume create reuses an existing volume with the same name, so old collections can reappear.

When you stop the container, start it again with:

docker start -a qdrant-local

This setup is for local development. It has no API key, TLS configuration, backups, or production-grade network controls. Do not expose it to an untrusted network.

Create the .NET project

Create a console application and add the official Qdrant client:

dotnet new console --framework net10.0 --name QdrantCollections
cd QdrantCollections
dotnet add package Qdrant.Client --version 1.19.0

I compiled the following sample with .NET 10 and Qdrant.Client 1.19.0. The client uses gRPC for its high-level API, so it connects to port 6334 rather than the dashboard port.

Replace Program.cs with:

using Qdrant.Client;
using Qdrant.Client.Grpc;

const string collectionName = "support-documents-v1";
const string embeddingProfileId = "support-text-v1";
const ulong vectorSize = 1536;

using var executionTimeout =
    new CancellationTokenSource(TimeSpan.FromSeconds(10));
using var client = new QdrantClient(host: "127.0.0.1", port: 6334);

if (!await client.CollectionExistsAsync(collectionName, executionTimeout.Token))
{
    await client.CreateCollectionAsync(
        collectionName: collectionName,
        vectorsConfig: new VectorParams
        {
            Size = vectorSize,
            Distance = Distance.Cosine
        },
        metadata: new()
        {
            ["embedding-profile-id"] = embeddingProfileId
        },
        cancellationToken: executionTimeout.Token);
}

CollectionInfo collection = await client.GetCollectionInfoAsync(
    collectionName,
    executionTimeout.Token);

VectorsConfig? vectorsConfig = collection.Config.Params.VectorsConfig;

if (vectorsConfig is null ||
    vectorsConfig.ConfigCase != VectorsConfig.ConfigOneofCase.Params)
{
    throw new InvalidOperationException(
        $"Collection '{collectionName}' does not use an unnamed " +
        "dense-vector configuration.");
}

VectorParams vectors = vectorsConfig.Params;

if (vectors.Size != vectorSize || vectors.Distance != Distance.Cosine)
{
    throw new InvalidOperationException(
        $"Collection '{collectionName}' does not match the expected " +
        $"{vectorSize}-dimension cosine configuration.");
}

if (!collection.Config.Metadata.TryGetValue(
        "embedding-profile-id",
        out Value? storedProfile) ||
    storedProfile.StringValue != embeddingProfileId)
{
    throw new InvalidOperationException(
        $"Collection '{collectionName}' does not declare the expected " +
        $"embedding profile '{embeddingProfileId}'.");
}

Console.WriteLine(
    $"Collection '{collectionName}' is {collection.Status} with " +
    $"{vectors.Size} dimensions, {vectors.Distance} distance, and " +
    $"embedding profile '{embeddingProfileId}'.");

Run it while the Qdrant container is available:

dotnet run

The output should look similar to this:

Collection 'support-documents-v1' is Green with 1536 dimensions, Cosine distance, and embedding profile 'support-text-v1'.

Refresh the Qdrant dashboard and the collection should appear there too.

Creating once is not enough

The existence check makes the sample convenient to run more than once. It does not assume that an existing collection is correct. This shorter version would be unsafe:

if (await client.CollectionExistsAsync(collectionName))
{
    return;
}

An older collection might have been created with 768 dimensions, dot-product distance, or a different embedding-profile ID. Returning early would allow the application to start against the wrong collection contract.

The sample reads the stored configuration back. It checks that the dense-vector configuration is unnamed and uses the expected size and distance. It also checks the application-defined embedding-profile ID stored in the collection metadata.

Qdrant does not infer which embedding model or preprocessing rules produced a vector. Collection metadata can hold an application-defined profile ID, but the application still owns what that ID means. Here, support-text-v1 refers to OpenAI text-embedding-3-small, 1,536 dimensions, cosine distance, and plain-text-v1 preprocessing.

The validation remains deliberately narrow. It does not reject additional sparse vectors, a multivector configuration, a non-default vector datatype, or every storage setting Qdrant supports. Add checks for those properties when application behavior depends on them. The current checks cover the unnamed dense-vector variant, size, distance, and profile ID.

Two application instances can also observe a missing collection and race to create it. That is acceptable for this local console app. For a deployed system, I would provision collections through a controlled deployment or migration step instead of letting every application instance own schema creation.

Keep the first collection boring

Qdrant exposes settings for shards, replication, HNSW, quantization, on-disk storage, named vectors, sparse vectors, and more. None of them is needed to learn the collection boundary.

The sample deliberately keeps Qdrant's defaults and creates:

one collection
one unnamed dense vector
one vector size
one distance function

For this local exercise, the defaults are sufficient. In production, replication and sharding follow availability, scale, and isolation requirements. HNSW and quantization choices should come from measurements against a representative workload.

The collection name includes v1 because its vectors belong to a particular embedding space. If a later profile uses an incompatible vector space, a new collection such as support-documents-v2 gives the migration an explicit target. It also avoids the temptation to delete and recreate a populated collection during application startup.

Do not replace this with RecreateCollectionAsync as a general startup shortcut. Recreating a collection deletes the existing collection and its points. That can be handy in a disposable test, but it is the wrong default once the data matters.

Before indexing the first point

The application has verified the dense vector's size and distance and the collection's embedding-profile ID. It has not decided how documents become stable point IDs, which payload fields to store, or where the original text lives. Those choices belong to the indexing path.

Before adding that path, write down four answers:

support-text-v1 owns this collection and identifies the embedding contract defined above.
support-documents-v1 is the collection name and migration target, not the embedding profile.
In a deployed system, collection creation belongs to a controlled provisioning or migration step.
Original text and payload design are intentionally undecided in this article.

Qdrant is a reasonable fit when a dedicated vector database suits the workload and the application can own another data boundary. If the source data already lives in PostgreSQL and the retrieval workload is modest, keeping vectors beside that data with pgvector may be simpler. RAG with EF Core and pgvector covers that route.

The next implementation step is to define stable point IDs and payloads, then write the first documents. HNSW and quantization can stay unchanged until retrieval measurements give a reason to revisit them.

Sources

AI Systems Need Runtime Budgets

Lukas Walter — Thu, 03 Sep 2026 15:30:00 +0000

A five-second model timeout does not give you a five-second AI feature.

Retrieval may take one second. The first model call takes four. A tool takes another two, and the next model call gets its own five-second timeout. Every dependency stayed within its local limit, yet the request took 12 seconds and paid for two model calls.

The complete use case needs one runtime budget. Its remaining allowance follows every model call, tool invocation, and retry.

Timeouts still matter. They are just one line in the budget.

Budget the execution, not each dependency

Consider a support-answer workflow:

load the ticket
    -> retrieve policy documents
    -> ask the model
    -> call a customer-history tool
    -> ask the model again
    -> validate the answer

Each dependency can have a sensible timeout while the workflow still runs too long. The same applies to tokens and cost. Three calls with a 1,000-token output cap can generate 3,000 tokens. A retry repeats input, and tool results enlarge the next prompt.

A runtime budget gives the workflow one shared envelope:

Dimension	What it bounds	What happens when it is exhausted
Wall-clock time	Total useful lifetime of the execution	Request cancellation and return a deadline result
Input tokens	Context sent across all model calls	Reduce context before the call or stop
Output tokens	Generated tokens across all model calls	Lower the next call's cap or stop
Model calls	Initial calls, follow-ups, repairs, and retries	Do not start another call
Tool calls	Automatic and application-directed invocations	Stop the loop before another tool runs
Retries	Repeated attempts after transient failures	Return the chosen failure or recovery outcome
Estimated cost	Accumulated cost of metered work	Stop or use a fallback whose cost already fits

These limits belong together because the dimensions interact. A retry can spend more time, tokens, and money. A tool call spends time and may make the next prompt larger. A longer output consumes more of the deadline.

A budget will stop some work that might have succeeded. Good. A late or overpriced result is not a successful execution for that product.

Define one execution policy

Keep the configured limits immutable. The live ledger belongs to one execution, which may or may not be an HTTP request.

public sealed record RuntimeBudget(
    TimeSpan MaxDuration,
    TimeSpan MaxModelAttemptDuration,
    TimeSpan MinUsefulModelAttemptDuration,
    TimeSpan CompletionHeadroom,
    int MaxModelCalls,
    int MaxToolCalls,
    int MaxRetries,
    long MaxInputTokens,
    long MaxOutputTokens,
    decimal MaxEstimatedCostUsd);

Validate the policy when configuration is loaded. The three attempt and execution durations must be positive. Headroom, counters, token limits, and cost limits cannot be negative. Require a positive output cap. The time limits must satisfy both invariants:

MinUsefulModelAttemptDuration <= MaxModelAttemptDuration
MinUsefulModelAttemptDuration + CompletionHeadroom <= MaxDuration

Apply the same checks to reservations. Negative estimates must never create allowance. Use checked arithmetic or subtraction helpers to avoid overflow.

Keep counters private and protect every compound read or mutation with the same lock. Return immutable snapshots created under that lock. Making one reservation method atomic does not make unrelated public properties thread-safe.

Start the deadline and ledger together

I use four separate cancellation sources:

caller cancellation
host shutdown
the execution deadline
a per-attempt timeout

The execution scope owns its deadline. Both elapsed-time accounting and cancellation use the same TimeProvider and start when the scope is created:

long startedAt = timeProvider.GetTimestamp();
using var deadlineCts = new CancellationTokenSource(
    limits.MaxDuration,
    timeProvider);

TimeSpan RemainingTime()
{
    TimeSpan remaining = limits.MaxDuration
        - timeProvider.GetElapsedTime(startedAt);

    return remaining > TimeSpan.Zero ? remaining : TimeSpan.Zero;
}

Using one clock keeps tests honest when they use FakeTimeProvider. A system timer should not expire independently from the fake clock used by the ledger.

The per-attempt timeout must fit inside the time still available. If an execution has 1.4 seconds left, starting a model call with its normal five-second timeout is dishonest. Reserve completion headroom first, then calculate:

attempt timeout = min(
    configured attempt timeout,
    remaining execution time - completion headroom)

Reject the call when the result is below MinUsefulModelAttemptDuration. A one-millisecond attempt is traffic, not a serious chance of success.

For each dependency call, create a time-provider-aware attempt source and link the sources only for delivery to that dependency:

using var attemptCts = new CancellationTokenSource(
    reservation.AttemptTimeout,
    timeProvider);

using var callCts = CancellationTokenSource.CreateLinkedTokenSource(
    callerToken,
    hostStoppingToken,
    deadlineCts.Token,
    attemptCts.Token);

Keep the original tokens. The linked token asks observing work to stop, but it does not record which source won. In this example, classification precedence is host shutdown, caller cancellation, execution deadline, then attempt timeout. Pick an order deliberately and use it everywhere.

Cancellation is cooperative. It does not prove that local work stopped immediately, or that a remote provider stopped processing and billing.

Reserve before handoff and reconcile afterward

Post-call accounting is too late. The workflow has already spent the money and time.

A model reservation needs a stable ID, estimated input tokens, an output cap, estimated cost, and an attempt timeout. Create it under the ledger lock only when every required dimension has enough allowance.

The reservation lifecycle matters:

Reserved -> Started -> Reconciled

Release Reserved work only when the application never handed it to the inner client or tool.
Reconcile Started work when required usage fields are available.
When a started call returns uncertain or partial usage, charge the unresolved fields against the reservation or stop with an unknown-usage outcome.

At this abstraction level, Started means control was handed to the next layer. Local validation, serialization, or option conversion may still fail before a network request. Precise dispatch tracking requires instrumentation inside the provider or tool adapter. Without it, charge pessimistically after handoff. A caller timeout does not prove that the provider consumed nothing.

The reservation operation should validate its arguments and calculate the attempt timeout inside the same lock that reserves calls, tokens, and cost. The core calculation is:

available attempt time = remaining execution time - completion headroom
attempt timeout = min(configured attempt timeout, available attempt time)
output allowance = limit - committed output - reserved output

Reject the reservation when the attempt timeout is below the minimum or the output allowance is zero. Subtract validated non-negative values and return zero once committed or reserved usage reaches the limit. Use the same approach for input and cost.

The reservation supplies the per-call ChatOptions.MaxOutputTokens. That option limits one model request when the underlying client honors it. It is not a cumulative workflow limit.

Put enforcement inside automatic tool loops

FunctionInvokingChatClient.GetResponseAsync(...) can make several requests to its inner client. It sends tool results back to that inner client and continues until the tool loop ends. The final response aggregates available usage from those turns.

One reservation around the outer call is wrong. It counts one model call even when the wrapper makes three, and its MaxOutputTokens value can be applied to each inner request. It also sees none of the locally invoked tools.

Place model-call enforcement inside the function-invocation wrapper. Put any retry middleware you control outside that budgeting boundary so each attempt must reserve again:

application
    -> FunctionInvokingChatClient
        -> application retry layer
            -> BudgetingChatClient
                -> provider SDK

There is a lifetime trap here. AddChatClient registers its pipeline as a singleton by default. A budgeting adapter that captures one execution's ledger or cancellation sources must not live in that singleton pipeline. Create the budgeting layer and its FunctionInvokingChatClient for each execution, or keep the adapter stateless and pass the current ledger through explicit per-call context.

The following fragment is illustrative pseudocode. Types such as PromptEstimate, ModelReservation, and the ledger are application contracts, not a companion library. callCts.Token is the linked per-attempt token from the previous section.

List<ChatMessage> messageList = [.. messages];
ChatOptions callOptions = options?.Clone() ?? new ChatOptions();

PromptEstimate estimate = promptEstimator.Estimate(
    messageList,
    callOptions,
    modelProfile);

ModelReservation reservation = budget.ReserveModelCall(
    estimate,
    callOptions.MaxOutputTokens);

callOptions.MaxOutputTokens = reservation.OutputTokenCap;

budget.MarkStarted(reservation); // Controlled handoff, not confirmed dispatch.

ChatResponse response = await innerClient.GetResponseAsync(
    messageList,
    callOptions,
    callCts.Token);

budget.ReconcileModel(reservation, response.Usage);

Materializing messages once avoids enumerating an arbitrary IEnumerable<ChatMessage> twice. The estimator needs more than message text. Depending on the provider, the effective prompt can include ChatOptions.Instructions, tool declarations and JSON schemas, structured-output schemas, multimodal content, and provider-specific framing. Estimate is the honest name because local preflight cannot always produce an exact count.

Stateful clients make that limit clearer. If ConversationId refers to history stored by the provider, the application may not possess the complete context. Reserve conservatively or disable local claims of exact input accounting for that path.

After the controlled handoff, charge the reservation on an exception unless reliable usage data allows reconciliation. The fragment omits that branch and the streaming override. A production adapter needs both.

Guard locally executed tools through FunctionInvokingChatClient.FunctionInvoker. Validate known preconditions before reserving, then mark the reservation Started immediately before context.Function.InvokeAsync(context.Arguments, cancellationToken).

On success or a known failure, reconcile from the request, response metadata, usage record, and failure details available to that tool adapter. If those sources cannot resolve the cost after handoff, charge the reservation. This is deliberately pessimistic. Parallel tool calls need atomic reservations before any task starts.

Hard tool-call limits apply only to execution you control. Provider-managed or server-side tools can be opaque. In that case, enforce the controls the provider exposes, account for reported usage, and avoid claiming that the application counted every tool call.

An application-owned tool loop is the other valid design. It is more code, but it makes every model and tool boundary visible without relying on wrapper placement.

Reconcile usage field by field

ChatResponse.Usage being non-null does not mean every count is present. InputTokenCount, OutputTokenCount, cached input, and reasoning counts are nullable. Reconcile each field required by your policy. For a missing required field, charge its reservation or stop with unknown usage.

Per-call interception avoids another ambiguity. FunctionInvokingChatClient adds the non-null usage objects returned by its inner calls. If one turn omits usage and another reports it, the outer response can contain a plausible but incomplete total.

Do not add cached input tokens to InputTokenCount again. Do not add reasoning tokens to OutputTokenCount again. Those detail counts classify pricing; the documented totals already include them.

Cost estimation needs a reviewed pricing key, not only a model ID. Depending on the provider, it may include the provider, deployment or model snapshot, region, service tier, modality, batch mode, and cached-token rules.

An application-estimated cost budget is not a provider billing ceiling. Rounding, missing usage, and price changes can make the invoice differ. Use provider-side quotas and billing controls separately, and know whether they alert or block spend.

The total cost dimension must include every paid boundary the article claims to budget. Retrieval services, rerankers, paid tools, and external APIs need their own reservations. If the ledger only accounts for model calls, name the limit MaxEstimatedModelCostUsd instead.

Retries spend the original execution budget

A retry does not receive a fresh deadline or token allowance. Do not start one that cannot finish inside the remaining time or would exceed another dimension.

Put application retry middleware outside BudgetingChatClient, as shown above. Each retry attempt then crosses the budgeting boundary and needs a new reservation.

Before repeating an attempt, the retry layer must atomically reserve one retry slot. The inner budgeting client separately reserves the repeated model or dependency call.

Retries inside the provider SDK are different. Configure or disable them when possible. If the SDK retries internally without an attempt callback, the application cannot claim an exact retry or model-call count. Treat that layer as opaque and reserve conservatively.

A cost-exhausted execution may use a fallback only when that fallback is free, has separately reserved capacity, or still fits inside the remaining allowance.

Budget exhaustion is an application result

OperationCanceledException alone does not explain what happened. It might mean the user disconnected, the host stopped, a dependency attempt timed out, or the complete runtime deadline expired.

Select a stop reason from the original source tokens, not from the linked token. Apply the same precedence at every boundary. This is a policy decision based on the tokens observed at inspection time; it does not prove which token caused the cancellation first.

For example, a SelectStopReason helper can check host shutdown first, then caller cancellation, the execution deadline, and finally the attempt timer. Map those to unavailable, cancelled, time-budget exhaustion, and dependency timeout results. Ledger failures should carry the exhausted dimension directly rather than masquerading as cancellation.

The user-facing message need not mention tokens or internal prices. Keep the reason in the application result for telemetry, support, and fallback decisions.

A partial answer needs its own policy. Some features can label and return one. Others should return nothing because an incomplete answer would be misleading.

Observe the allowance and the spend

Record both sides of the ledger. A trace that says model.duration = 3.2s does not say whether 3.2 seconds was healthy for this feature.

For one execution, I would record:

the budget policy or a stable policy version
elapsed and remaining time at each boundary
model calls, tool calls, and retries used
reported input and output tokens
estimated cost and the price configuration version
the exhausted dimension, when execution stops

Do not copy prompts, retrieved documents, or tool results into telemetry just to explain the numbers. Operational metadata usually answers the budget question without duplicating sensitive content.

Per-execution budgets stop one workflow from running away. They do nothing about ten thousand executions that each stay inside budget. Rate limits, concurrency limits, tenant quotas, and dependency-level retry budgets protect that shared capacity.

Choose numbers from the product promise

There is no universal runtime budget. A chat interaction and an overnight evaluation run have different definitions of "too late." Start with how long the caller will wait, what one result may cost, which tools it needs, and whether a fallback still has value.

Test the complete path under normal load and injected failure. Measure tail latency as well as averages. Include prompt growth, tool output, throttling, and retry delays. Test the effects of opaque SDK retries even when the SDK does not expose an exact attempt count.

Leave headroom between the internal deadline and the external timeout. The application still needs to map the result, write telemetry, and respond. If both expire together, the caller may receive a connection failure instead of the intended outcome.

Treat the first values as a hypothesis. Production traces and cost data will show where the budget is too loose, too strict, or spent on the wrong step.

When runtime budgets are useful

Use an execution-scoped budget when a workflow can make more than one metered or remote call, invoke tools, retry, or fan out. It gives the orchestration layer one answer to the question: may this execution start more work?

For interactive work, that execution normally observes caller cancellation. Deliberate background work needs an independent budget instead of silently outliving the request. Durable workflows also need persisted reservations and idempotent reconciliation so a restart cannot reset or double-charge the ledger.

When a full budget object is unnecessary

A local spike or one low-risk model call may only need cancellation, a timeout, and an output cap. Add a shared ledger when the workflow can multiply time or spend, not when the object would add ceremony without changing a decision.

Practical takeaway

Pick one AI use case and write down the maximum wall-clock time, cumulative tokens, model calls, tool calls, retries, and estimated cost for one execution.

Put those limits in one policy. Create one ledger when the execution begins. Before every new attempt, reserve what it can spend. After each metered operation, reconcile its reservation with the available usage or cost data. When a required dimension lacks enough allowance, return a named application outcome instead of starting one more call.

The next model call now starts only when the complete execution can still afford it.

Sources

Microsoft Learn: ChatOptions.MaxOutputTokens
.NET source: ChatOptions.Clone()
Microsoft Learn: ChatResponse.Usage
Microsoft Learn: AddChatClient service lifetime
Microsoft Learn: FunctionInvokingChatClient
.NET source: FunctionInvokingChatClient.cs
Microsoft Learn: FunctionInvokingChatClient.FunctionInvoker
Microsoft Learn: UsageDetails, including CachedInputTokenCount and ReasoningTokenCount
Microsoft Learn: TimeProvider overview
Microsoft Learn: Cancellation in managed threads
Microsoft Learn: Best practices for transient fault handling

Provider Independence from Day One

Lukas Walter — Mon, 31 Aug 2026 15:30:00 +0000

Provider independence starts before the second provider arrives. Draw the provider boundary when the first integration is built. Alternate implementations can wait until a real need appears.

IChatClient gives .NET applications a common API for messages, responses, streaming, tools, and structured output. That removes a lot of SDK coupling. The remaining differences still need a place in the design. JSON Schema support, tool behavior, context limits, model names, deployment names, and provider-only options do not disappear because two clients implement the same interface.

Provider wiring stays at the composition boundary. A dedicated factory selects and constructs the Azure OpenAI or Ollama client. The ticket-routing feature uses one C# service for either backend. A small capability declaration selects native JSON Schema or prompt-guided schema instructions. A qualification check confirms that the configured structured-output path works before the backend is promoted.

That is the level of independence I trust: one application path with the differences exposed and backed by tests against each configured backend.

There is a complete example available as a .NET 10 file-based app. It runs offline by default and includes configuration examples for Ollama and Azure OpenAI.

Define the portable behavior first

Start with the feature, not a list of providers.

This example routes a support ticket to one of three queues. The model call needs:

system and user messages
a typed JSON result
cancellation
no tools, images, or provider-managed conversation state

Those requirements fit IChatClient. The application result does not need to expose chat messages or SDK response types:

public enum TicketQueue
{
    Billing,
    TechnicalSupport,
    AccountAccess
}

public sealed record TicketRoutingDecision(
    TicketQueue Queue,
    bool RequiresHumanReview,
    string Reason);

This is the portability contract for the first version. If the feature later requires hosted file search or another provider-only API, the contract has changed. Pretending otherwise would only move the coupling into a loosely typed options dictionary.

Install the provider integrations

The sample uses Azure OpenAI in a hosted environment and Ollama for a local backend:

dotnet add package Microsoft.Extensions.AI
dotnet add package Microsoft.Extensions.AI.OpenAI
dotnet add package Azure.AI.OpenAI
dotnet add package Azure.Identity
dotnet add package OllamaSharp

I compiled the code in this article with .NET 10, Microsoft.Extensions.AI 10.9.0, Microsoft.Extensions.AI.OpenAI 10.9.0, Azure.AI.OpenAI 2.1.0, Azure.Identity 1.21.0, and OllamaSharp 5.4.30. Check the current package documentation when using newer versions.

The dependency direction stays small:

TicketRouter
    -> SupportModelClient
        -> IChatClient

composition root
    -> SupportModelClientFactory
        -> AzureOpenAIClient or OllamaApiClient

TicketRouter has no reference to either provider SDK.

Keep model and deployment configuration separate

Azure OpenAI addresses a deployment. Ollama addresses a model available at its endpoint. The application should not use either value as its stable service name.

Use one options type for the selected support-routing backend:

using System.ComponentModel.DataAnnotations;

public enum StructuredOutputMode
{
    Unspecified,
    NativeJsonSchema,
    PromptedJsonSchema
}

public sealed class SupportModelOptions
{
    [Required]
    public string Provider { get; init; } = "";

    [Required]
    public string Model { get; init; } = "";

    public string? Deployment { get; init; }

    [Required]
    public Uri Endpoint { get; init; } = null!;

    public StructuredOutputMode StructuredOutput { get; init; }
}

An Azure OpenAI configuration can look like this:

{
  "AI": {
    "SupportRouting": {
      "Provider": "AzureOpenAI",
      "Endpoint": "https://example.openai.azure.com/",
      "Model": "your-model-id",
      "Deployment": "support-routing-prod",
      "StructuredOutput": "NativeJsonSchema"
    }
  }
}

The deployment is the Azure OpenAI deployment name used by the SDK. It identifies a model deployment inside the Azure OpenAI resource. Model is the expected model identity the team records with traces, evaluations, and release decisions. They may have the same text, but they own different concerns.

For local Ollama development, only the provider-specific settings change:

{
  "AI": {
    "SupportRouting": {
      "Provider": "Ollama",
      "Endpoint": "http://localhost:11434/",
      "Model": "your-qualified-model",
      "StructuredOutput": "PromptedJsonSchema"
    }
  }
}

StructuredOutput describes a tested capability of the model, backend, and integration. It is not a permanent fact about the provider. Native JSON Schema may work well enough for one combination of model, backend, and integration but not another. Set NativeJsonSchema only after checking the exact combination you will deploy. The probe later in this article checks that the path works, but it cannot prove schema enforcement. Unspecified exists so missing configuration fails instead of silently selecting the stronger mode.

Do not copy the placeholder model names and capability values into production. Qualify the model you intend to run.

Build one provider-neutral client holder

The application needs the chat client and the capability declaration together. A small holder keeps them consistent. The DI container owns that holder and disposes the selected IChatClient when the container shuts down:

using Microsoft.Extensions.AI;

public sealed class SupportModelClient(
    IChatClient chatClient,
    string declaredModelIdentity,
    StructuredOutputMode structuredOutput) : IDisposable
{
    public IChatClient ChatClient { get; } = chatClient;
    public string DeclaredModelIdentity { get; } = declaredModelIdentity;
    public StructuredOutputMode StructuredOutput { get; } = structuredOutput;

    public void Dispose() => ChatClient.Dispose();
}

I would not call this class AiClient. It is the selected client for one application purpose. A document summarizer or an agent may have a different model, capability set, and operating policy.

DeclaredModelIdentity is configuration, not proof of what an Azure deployment currently serves. A deployment can move to another model or version while the application setting becomes stale. Record model metadata returned by the provider as an observed identity when it is available, and compare it with the declared value in telemetry or deployment checks.

Create the provider clients at the edge

The factory is allowed to know the provider SDKs. That is its job.

using Azure.AI.OpenAI;
using Azure.Identity;
using Microsoft.Extensions.AI;
using OllamaSharp;

public static class SupportModelClientFactory
{
    public static SupportModelClient Create(SupportModelOptions options)
    {
        if (options.StructuredOutput is not (
            StructuredOutputMode.NativeJsonSchema or
            StructuredOutputMode.PromptedJsonSchema))
        {
            throw new InvalidOperationException(
                "A supported structured-output mode is required.");
        }

        return options.Provider switch
        {
            "AzureOpenAI" => CreateAzureOpenAI(options),
            "Ollama" => CreateOllama(options),
            _ => throw new InvalidOperationException(
                $"Unsupported AI provider '{options.Provider}'.")
        };
    }

    private static SupportModelClient CreateAzureOpenAI(
        SupportModelOptions options)
    {
        if (string.IsNullOrWhiteSpace(options.Deployment))
        {
            throw new InvalidOperationException(
                "AI:SupportRouting:Deployment is required for Azure OpenAI.");
        }

        var azureClient = new AzureOpenAIClient(
            options.Endpoint,
            new DefaultAzureCredential());

        IChatClient chatClient = azureClient
            .GetChatClient(options.Deployment)
            .AsIChatClient();

        return new SupportModelClient(
            chatClient,
            options.Model,
            options.StructuredOutput);
    }

    private static SupportModelClient CreateOllama(
        SupportModelOptions options)
    {
        IChatClient chatClient = new OllamaApiClient(
            options.Endpoint,
            options.Model);

        return new SupportModelClient(
            chatClient,
            options.Model,
            options.StructuredOutput);
    }
}

Azure authentication belongs in CreateAzureOpenAI. Ollama endpoint and model setup belong in CreateOllama. TicketRouter sees neither SDK.

The switch runs once while the container creates its singleton. It does not appear in every request path, controller, or application service.

using Microsoft.Extensions.Options;

builder.Services
    .AddOptions<SupportModelOptions>()
    .BindConfiguration("AI:SupportRouting")
    .ValidateDataAnnotations()
    .Validate(
        options => options.Provider is "AzureOpenAI" or "Ollama",
        "Provider must be AzureOpenAI or Ollama.")
    .Validate(
        options => options.StructuredOutput is
            StructuredOutputMode.NativeJsonSchema or
            StructuredOutputMode.PromptedJsonSchema,
        "StructuredOutput must be declared explicitly.")
    .Validate(
        options =>
            options.Provider != "AzureOpenAI" ||
            !string.IsNullOrWhiteSpace(options.Deployment),
        "Deployment is required for Azure OpenAI.")
    .ValidateOnStart();

builder.Services.AddSingleton<SupportModelClient>(services =>
{
    SupportModelOptions options = services
        .GetRequiredService<IOptions<SupportModelOptions>>()
        .Value;

    return SupportModelClientFactory.Create(options);
});

builder.Services.AddTransient<TicketRouter>();

The custom validators reject an unsupported provider, a missing capability declaration, and a missing Azure OpenAI deployment during startup. The factory keeps defensive checks for callers that bypass this registration path. A production application should also validate allowed endpoint schemes, the selected authentication mode, and unsupported option combinations.

Implement the feature once

The model-facing DTO uses strings so the application can reject unknown queue values deliberately. Deserializing directly into an enum can make an unsupported value look like a serialization detail instead of an invalid model decision.

internal sealed record TicketRoutingOutput(
    string? Queue,
    bool? RequiresHumanReview,
    string? Reason);

RequiresHumanReview is nullable only at the model boundary. If the model omits the property, deserialization leaves it as null and the application can reject the incomplete decision instead of treating the missing value as false.

The router asks for a typed response, then validates every field that crosses the model boundary:

using Microsoft.Extensions.AI;

public sealed class TicketRouter(SupportModelClient model)
{
    private const string Instructions = """
        Route the support ticket to exactly one queue:
        Billing, TechnicalSupport, or AccountAccess.
        Set requiresHumanReview to true when the ticket is ambiguous.
        Keep the reason to 120 characters or fewer.
        Return JSON only.
        """;

    public async Task<TicketRoutingDecision> RouteAsync(
        string ticket,
        CancellationToken cancellationToken)
    {
        if (string.IsNullOrWhiteSpace(ticket))
        {
            throw new ArgumentException(
                "Ticket text is required.",
                nameof(ticket));
        }

        ChatResponse<TicketRoutingOutput> response =
            await model.ChatClient.GetResponseAsync<TicketRoutingOutput>(
                [
                    new ChatMessage(ChatRole.System, Instructions),
                    new ChatMessage(ChatRole.User, ticket)
                ],
                useJsonSchemaResponseFormat:
                    model.StructuredOutput is
                        StructuredOutputMode.NativeJsonSchema,
                cancellationToken: cancellationToken);

        if (!response.TryGetResult(out TicketRoutingOutput? output) ||
            output is null ||
            output.RequiresHumanReview is null ||
            string.IsNullOrWhiteSpace(output.Reason) ||
            output.Reason.Length > 120)
        {
            throw new InvalidOperationException(
                "The model returned an invalid routing decision.");
        }

        TicketQueue queue = output.Queue switch
        {
            nameof(TicketQueue.Billing) => TicketQueue.Billing,
            nameof(TicketQueue.TechnicalSupport) =>
                TicketQueue.TechnicalSupport,
            nameof(TicketQueue.AccountAccess) => TicketQueue.AccountAccess,
            _ => throw new InvalidOperationException(
                "The model returned an invalid queue.")
        };

        return new TicketRoutingDecision(
            queue,
            output.RequiresHumanReview.Value,
            output.Reason);
    }
}

Both modes use the same DTO and application validation. The generated schema describes the serialized DTO shape. The explicit checks still own domain rules such as required fields, the allowed queue names, and the maximum reason length.

GetResponseAsync<T> derives a JSON Schema from T in both modes. With NativeJsonSchema, the helper sends that schema through the backend's structured-response mechanism. With PromptedJsonSchema, it requests JSON and appends another user message containing the generated schema. The second mode guides the model through the prompt instead of relying on native schema enforcement. The application applies the same post-deserialization validation either way.

An invalid result should become an explicit application outcome in a complete service layer. I use an exception here to keep the sample focused on provider composition. The production decision may be to return InvalidGeneration, make one bounded repair attempt, or require review. It should not return the unvalidated text as a routing decision.

Treat capabilities as claims that need evidence

A capability flag can become another lie in configuration. Verify it against the exact model, deployment, integration package, and endpoint used by the environment.

A small qualification probe can check the structured-output path:

using Microsoft.Extensions.AI;

internal sealed record CapabilityProbe(string? Value);

public static class SupportModelQualification
{
    public static async Task VerifyAsync(
        SupportModelClient model,
        CancellationToken cancellationToken)
    {
        ChatResponse<CapabilityProbe> response =
            await model.ChatClient.GetResponseAsync<CapabilityProbe>(
                "Return JSON with value set to ready.",
                useJsonSchemaResponseFormat:
                    model.StructuredOutput is
                        StructuredOutputMode.NativeJsonSchema,
                cancellationToken: cancellationToken);

        if (!response.TryGetResult(out CapabilityProbe? result) ||
            result?.Value != "ready")
        {
            throw new InvalidOperationException(
                $"Model '{model.DeclaredModelIdentity}' failed the " +
                "support-routing structured-output check.");
        }
    }
}

Run this as a deployment smoke test or structured-output contract test. I would not make every application startup depend on a paid model call. The test should run whenever the model, deployment, provider integration, or relevant configuration changes.

This probe proves that the request succeeds and the helper can deserialize a compatible result for this example. A model could return {"value":"ready"} without obeying the supplied schema, so the probe does not prove native schema enforcement. It also says nothing about routing quality. Keep a small golden dataset for that question and evaluate every candidate backend against the same cases. Provider independence without comparable output quality is not useful independence.

Keep provider-only behavior in a named adapter

ChatOptions covers common settings. It also exposes AdditionalProperties and RawRepresentationFactory for options understood by a specific provider. Those escape hatches are useful, but the type system cannot turn them into portable behavior.

If TicketRouter creates an OpenAI ChatCompletionOptions through RawRepresentationFactory, that call is OpenAI-specific even though the outer dependency is IChatClient. Put it in an infrastructure adapter with a name that says what it requires, or accept that the feature is tied to that provider.

The same rule applies to hosted file search, provider-managed conversation state, batch APIs, safety configuration, and model-specific reasoning controls. Do not flatten those features into an AiOptions bag shared by the whole application.

Sometimes the honest design has two paths:

portable ticket routing
    -> IChatClient

provider-hosted file search
    -> purpose-specific application interface
        -> provider SDK adapter

That is still a well-contained architecture. Independence is about controlling the dependency, not eliminating every provider reference from the repository.

What changes during a provider switch

With this structure, moving the ticket router to another backend still requires engineering work. The work is contained:

Add the provider package and one factory branch.
Configure endpoint, model identity, deployment identity when applicable, and the qualified capability mode.
Run the structured-output contract check.
Run the routing evaluation dataset and compare quality, latency, and cost.
Review operational differences such as authentication, throttling, telemetry metadata, and content retention.

TicketRouter changes only if the feature contract changes. A new provider that cannot satisfy the existing contract is not a drop-in replacement. It may still be usable behind a deliberately reduced mode, but the application must name and test that behavior.

When this approach is enough

Use this pattern when several backends can satisfy the same bounded operation through IChatClient. Classification, extraction, summarization, and constrained generation are good candidates when their required inputs and outputs fit the shared abstraction.

It also works well when production uses one provider and local development uses another. That setup catches accidental SDK leakage early, even if the production provider never changes.

When provider independence is the wrong goal

Keep the provider SDK visible when the feature exists because of a provider-only capability, or when the abstraction would discard information the application needs. A thin, purpose-specific adapter is better than a generic interface full of string properties and raw objects.

Do not pay a permanent complexity tax for a switch the application will never make. One provider behind a clean composition boundary is already a sensible design. Add alternate implementations and capability negotiation when an environment-specific requirement, resilience plan, procurement constraint, or local-development need justifies them.

Practical takeaway

IChatClient is useful only for behavior it can represent honestly. Keep provider construction at the edge and attach the evidence-based capability mode to the selected client.

Provider-specific code will remain. Keep it easy to find and change, with its purpose stated in the adapter. The application feature should have no reason to care which SDK sits underneath.

Sources

Microsoft Learn: Use the IChatClient interface
Microsoft Learn: IChatClient.GetResponseAsync
Microsoft Learn: Structured-output extensions for IChatClient
dotnet/extensions: GetResponseAsync<T> structured-output implementation
Microsoft Learn: ChatOptions.RawRepresentationFactory
Microsoft Learn: AddOptions
Microsoft Learn: ValidateOnStart
Microsoft Learn: Azure OpenAI client library for .NET
Microsoft Learn: Azure OpenAI model deployments
dotnet/extensions: Microsoft.Extensions.AI.OpenAI examples
OllamaSharp: IChatClient integration
NuGet: Microsoft.Extensions.AI 10.9.0
NuGet: Microsoft.Extensions.AI.OpenAI 10.9.0
NuGet: OllamaSharp 5.4.30

Retries Are Not a Recovery Strategy

Lukas Walter — Thu, 27 Aug 2026 15:30:00 +0000

A retry answers a narrow question: might the same operation succeed if I attempt it again?

Recovery has a harder job. It must bring the original business operation to a known, valid outcome after something went wrong. Getting there may require another attempt, a status lookup, resuming from persisted state, or compensation. If the system cannot resolve the operation safely, it must hand it to a person.

This difference matters as soon as an AI workflow does more than return text. If it retrieves data, calls tools, writes state, or continues after the HTTP request ends, adding three retries around the workflow is not a recovery design. It is three more chances to spend money, repeat a side effect, or lose track of what already happened.

A retry repeats an attempt

Suppose a support feature performs this workflow:

load the ticket and approved policy
    -> generate a reply
    -> validate the reply
    -> save it as a draft

The policy read returns 503 Service Unavailable with an applicable Retry-After response, and the dependency contract classifies it as transient. No application business state changed, and the request still has time left. A delayed retry may be reasonable.

Now suppose the draft save times out after the request reached the database. The caller cannot tell whether the write committed. Repeating the complete workflow creates a new model response and may save a second draft. Retrying only the write is safe when the write is naturally idempotent, or when the boundary can recognize the retry as the same logical operation. Otherwise, the second attempt may create another draft.

Both failures may appear as a timeout or dependency exception in application code. They do not have the same effect.

What happened	What is known	Suitable response
A transient policy read failed before returning data	No application business state changed	Retry the read within its budget
The model endpoint rejected an invalid request	The same request will fail again	Stop and fix the request or contract
The model call timed out	No application state changed, but provider work and cost may already have occurred	Retry only if the result is still useful and budget remains
A draft save was submitted and its response was lost	The draft may already exist	Look up the original operation or retry it with the same idempotency identity
A multi-step workflow stopped after some steps completed	The operation is partially complete	Resume, reconcile, compensate, or escalate according to persisted workflow state

Retry belongs to the first few rows. Recovery begins when the system must find out what happened or restore a valid business state.

A timeout does not mean the operation failed

When the caller's timeout expires, it tells the caller that it stopped waiting. It does not prove that the callee stopped working, rolled back, or never received the request. The remote operation may never have started, may have failed, may still be running, or may have completed while its response was lost.

Those states should not collapse into one Failed result. I find it useful to separate the operation status from its business outcome:

public enum OperationStatus
{
    Pending,
    Running,
    PartiallyCompleted,
    OutcomeUnknown,
    Terminal,
    NeedsReview
}

public enum OperationOutcome
{
    Succeeded,
    Rejected,
    Failed,
    Compensated
}

public sealed record OperationState(
    OperationStatus Status,
    OperationOutcome? Outcome);

This type is deliberately simplified. As written, it permits invalid combinations such as Running with Succeeded, or Terminal with no outcome. Production code should enforce those invariants through its type design or validated construction. OperationStatus is a top-level recovery status, not a complete per-step workflow model. A workflow can be partially complete while the outcome of its latest step is still unknown.

Failed is a terminal unsuccessful outcome supported by authoritative evidence. A payment decline, for example, is a completed request with a Rejected business outcome. OutcomeUnknown means the application cannot yet determine the terminal outcome. NeedsReview is not a business outcome at all. It transfers ownership from automatic recovery to a person.

That extra state is mildly inconvenient. Good. The uncertainty already exists in the system, whether the type admits it or not. Encoding it gives the API, UI, recovery worker, and support tooling something honest to work with.

For the draft save, the application can allocate an operation ID before it submits the write:

Guid operationId = Guid.NewGuid();

var command = new SaveDraftCommand(
    OperationId: operationId,
    TicketId: ticketId,
    Reply: reply);

await draftOperations.CreateAsync(
    operationId,
    command,
    cancellationToken);

try
{
    await draftStore.SaveAsync(command, cancellationToken);

    await draftOperations.MarkTerminalAsync(
        operationId,
        OperationOutcome.Succeeded,
        cancellationToken);
}
catch (DraftSaveOutcomeUnknownException)
{
    using var recoveryWriteCts = new CancellationTokenSource(
        TimeSpan.FromSeconds(2));

    await draftOperations.MarkOutcomeUnknownAsync(
        operationId,
        recoveryWriteCts.Token);

    throw;
}

The example uses DraftSaveOutcomeUnknownException for a specific conclusion from the storage adapter: the attempt crossed the submission boundary, but the adapter cannot determine whether the write committed. A plain OperationCanceledException is not enough. Cancellation may happen before anything was submitted, in which case the write outcome is known.

Production code must still distinguish caller cancellation from an attempt timeout. It also has to handle failures while recording the state transition and prevent an older attempt from overwriting a newer result. Create the durable identity before the ambiguous boundary, then keep it for lookup and any safe retry.

The persistence contract must enforce uniqueness for the operation ID and reject reuse with different command data. Merely logging a GUID does not make the write idempotent.

Retrying the workflow is usually the wrong scope

Broad retry policies are attractive because they are easy to add around an endpoint or orchestration method. They also repeat every successful step before the failure.

If the support workflow fails while saving the draft, a workflow-level retry may retrieve the same policy again, pay for another model call, produce a different reply, and run validation again before it reaches the uncertain write. The retry has changed the thing being recovered.

AI workflows make this especially awkward:

Model calls are metered and can be slow.
A repeated generation is not guaranteed to return the same output.
Tool calls can hide writes behind an interface that looks like an ordinary function call.
SDKs, HTTP clients, queues, and workflow code may each have their own retry behavior.

Agent loops add another failure mode. Consider an approved workflow that is allowed to charge a customer:

model selects ChargeCustomer(...)
    -> application validates and authorizes the tool call
    -> payment succeeds
    -> tool response is lost
    -> agent loop starts again
    -> model produces another ChargeCustomer(...) call

The second model run may use a new tool-call ID or produce slightly different arguments. Approval and argument validation do not tell the payment boundary that this is the same charge. The application needs a stable business operation ID that exists independently of the model output and survives another model turn. Otherwise, the retry has changed both the reasoning and the command being recovered.

That last point can turn a small setting into a large amount of traffic. Three attempts in one layer and three in the next can produce up to nine calls to the struggling dependency. Under load, the retries may prolong the incident they were supposed to smooth over.

Per-request limits may still allow too much retry traffic when many requests fail together. A dependency-level retry budget caps the aggregate retries generated by concurrent requests. Once the budget is spent, requests get no additional attempts. Their initial attempts are a separate decision. A circuit breaker or admission-control policy can make new requests fail fast.

Retry the smallest operation whose failure is known to be transient and whose repetition is safe. Do not restart a workflow simply because one of its steps threw an exception.

Recovery starts with durable operation state

An in-memory retry loop can help the current request, but it does not provide durable ownership of the operation. A process restart or deployment can destroy its state, and request-scoped work may end when the caller disconnects.

When the result must still be resolved later, the application needs a durable record of the logical operation. That record needs enough information to identify and continue the same work:

a stable operation ID, plus any boundary-specific idempotency identities needed for safe replay
the command or a durable reference to it
a command fingerprint used to detect and reject the same key being reused for different work
the current state and attempt identity
the next eligible recovery time
enough outcome information for status lookup and support

A recovery worker can then process unresolved operations without inventing a new logical request:

Pending
    -> submit with the original idempotency key

OutcomeUnknown
    -> query the authoritative status
    -> retry the original command only when the contract makes that safe

PartiallyCompleted
    -> resume the next incomplete step or run a domain-specific compensation

Still unresolved after the recovery deadline
    -> NeedsReview

Recovery does not always mean forcing the requested state change to succeed. It may confirm a payment decline and close the operation with a Rejected outcome. If an external reservation succeeded but a later step failed, recovery may cancel the reservation and record Compensated. If neither action is safe to automate, the operation moves to NeedsReview and a person owns the next decision.

Recovery should produce a terminal, explainable business outcome or transfer ownership to manual review. An unresolved operation should not simply disappear when the retry loop stops.

Idempotency protects one boundary

An idempotency key tells a boundary that repeated submissions refer to one logical command. The boundary must atomically claim or enforce uniqueness for that command, including when two requests arrive concurrently. It can prevent the second request from applying the side effect again and, once available, return or reference the recorded result.

While the first request is still running, there is no result to return. The contract may expose the current operation status, wait for completion, or report that the operation is already in progress. A separate check followed by an insert leaves a race in which both requests can execute.

That protection is local to the boundary that enforces it. A database uniqueness constraint does not deduplicate an email already sent by another service. A job ID does not protect a payment call unless the same identity reaches the payment provider and the provider honors it.

Each side-effect boundary needs its own guarantee. A transactional outbox can make the handoff from a business transaction to messaging durable, but the relay may publish a message more than once. The receiving boundary still needs an inbox, an idempotent consumer, a provider-supported idempotency key, or a domain-specific deduplication rule.

Idempotency also does not answer these questions:

Should the operation still run after the caller has gone away?
How long should the system keep trying?
What happens when status lookup remains unavailable?
Can completed steps be reversed?
Who owns an operation that never reaches a terminal state?

Those are recovery decisions.

A retry policy still needs boundaries

Retries are useful when the failure is plausibly transient, repeating the attempt is safe, and another result would still arrive in time to matter.

For each retryable interaction, define:

Which outcomes are transient. Do not retry validation errors, authorization denials, or other requests that cannot succeed unchanged.
The scope of the attempt. Retry the failed read or model call, not an entire sequence of completed work.
The per-attempt timeout and total budget. A new attempt should not start when there is no useful time left.
The maximum attempts and delay. Respect Retry-After where available, and use backoff with jitter rather than synchronizing every instance.
The behavior after the attempts end. Return a specific application outcome, let the circuit-breaker policy update its failure state, degrade explicitly, or hand the operation to durable recovery.

Also inspect retry behavior already present in SDKs and infrastructure. Adding an application policy without accounting for lower layers makes attempt counts and latency hard to predict.

Record the attempt number, duration, outcome category, and operation ID in telemetry. A request that succeeds on its third attempt still represents two failed dependency calls, more latency, and more cost. Occasional transient failures are normal. A rising retry rate or an increasing share of requests that need several attempts may show that the dependency is degrading. Final success alone hides that change.

When retries are enough

A bounded retry policy is often enough for an idempotent read or another side-effect-free call when the failure is clearly transient. The work stays inside the current request, no state is uncertain, and exhausting the attempts can return an honest failure to the caller.

Examples include a throttled metadata lookup, a connection failure where the client can establish that the request was never submitted, or a model call that has no tools and whose repeated cost is acceptable inside the remaining request budget.

When you need a recovery path

Design recovery when an operation changes state, spans several independently committed steps, or must reach a terminal result after the current request ends. You also need it when a response can disappear after a side effect commits, or when an external action requires compensation or human review.

Recovery costs more to build. It needs persisted state, ownership, deadlines, concurrency control, telemetry, and support procedures. That cost is a reason to keep workflows small and side effects explicit. It is not a reason to call a retry loop recovery.

Practical takeaway

Pick one state-changing AI workflow and find every retry around it, including retries inside SDKs and infrastructure.

For each one, write down what is known after the failed attempt, whether state may have changed, which identity the next attempt uses, and what happens after attempts are exhausted. Returning an error is a valid outcome when no state changed and the application has no promise to continue the work.

If state may already have changed, the operation is partially complete, or the business contract requires a terminal result after the request ends, "return an error and hope the user tries again" is not enough. That workflow still needs a recovery path.

Use an explicit OutcomeUnknown path when the application loses the response after submitting a side effect and cannot determine whether it completed. Resolve that state through authoritative lookup, a safe retry with the original identity, compensation, or manual review.

Sources

Microsoft Learn: Retry pattern
Microsoft Learn: Best practices for transient fault handling
Microsoft Learn: Compensating Transaction pattern
Microsoft Learn: Retry Storm antipattern
Microsoft Learn: Azure Cosmos DB design pattern: Transactional Outbox

Embeddings in .NET

Lukas Walter — Mon, 24 Aug 2026 15:30:00 +0000

Generating one in .NET is a short API call. The part worth designing is the contract around that call: which model or encoder produced each vector, how many values it contains, which comparison metric will be used during search, and whether indexed documents and runtime queries use compatible rules.

Microsoft.Extensions.AI gives .NET applications a provider-neutral IEmbeddingGenerator<TInput, TEmbedding> abstraction. It keeps provider SDK types out of the retrieval code, but provider neutrality does not imply vector-space compatibility.

Before the first document is indexed, four choices need to be fixed together: the model, vector shape, comparison metric, and input preparation. The examples below use dense text embeddings for semantic retrieval.

An embedding is a model output, not application data

An embedding model, together with its configuration, maps an input into a vector of numbers with a defined dimensionality.

"stop a worker during shutdown"
-> embedding model
-> [0.018, -0.042, 0.007, ...]

The vector is a compact representation learned by the model. Inputs the model represents as similar should end up near one another under a comparison metric appropriate for that embedding space.

The individual coordinates do not have stable application meanings. Position 37 is not "shutdown behavior," and a larger value at that position does not make one passage more relevant. The vector works as a whole.

This also means an embedding is lossy. You cannot reconstruct the source passage reliably from the vector, and the vector does not retain the document ID, permissions, version, or source URL. Keep the original text and its metadata. The embedding is an index representation of that source, not a replacement for it.

Similarity scores need context too. Read a cosine score within the embedding model, retrieval setup, and corpus that produced it. Adding candidates does not change the score between two vectors. It can change where a passage ranks and whether it remains in the top results. The score distribution across a corpus also affects whether a fixed threshold is useful. The score is not a universal confidence value, a probability that the passage answers the question, or proof that the passage is correct.

Embeddings help rank candidates. They do not own truth or authorization.

Documents and queries must enter the same vector space

A basic semantic retrieval path has two sides.

During ingestion:

document chunk
-> embedding generator
-> document vector
-> vector index

During a query:

user question
-> same embedding contract
-> query vector
-> nearest-neighbor search

The search works because both vectors are coordinates in the same space. Their text does not need to contain the same words. The configured model or encoders should place semantically related inputs close enough for the search step to find them.

"Same embedding contract" is more precise than "same number of floats." Two models can return vectors with equal dimensions and still use unrelated coordinate systems. Vectors from independently chosen models generally cannot be compared meaningfully. They are compatible only when the models or encoders are explicitly designed to produce representations in the same retrieval space.

This is the first boundary to keep clear after deciding that a feature really needs retrieval. Why Retrieval Exists covers that earlier decision. Once vector search is part of the retrieval path, the embedding contract becomes part of the index schema.

Generate an embedding through `IEmbeddingGenerator`

For an OpenAI-backed example, install the Microsoft.Extensions.AI.OpenAI package:

dotnet add package Microsoft.Extensions.AI.OpenAI --version 10.9.0

I compiled the sample below with .NET 10 and Microsoft.Extensions.AI.OpenAI 10.9.0 from the public NuGet feed. It uses the IEmbeddingGenerator API. Check package versions before copying it into an application.

using Microsoft.Extensions.AI;
using OpenAI;

string apiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY")
    ?? throw new InvalidOperationException(
        "Set OPENAI_API_KEY before running the sample.");

IEmbeddingGenerator<string, Embedding<float>> generator =
    new OpenAIClient(apiKey)
        .GetEmbeddingClient("text-embedding-3-small")
        .AsIEmbeddingGenerator();

ReadOnlyMemory<float> queryVector = await generator.GenerateVectorAsync(
    "How do I stop a background service cleanly?",
    cancellationToken: CancellationToken.None);

Console.WriteLine($"Generated {queryVector.Length} values.");

GenerateVectorAsync is a convenience method for one input. It returns the vector as ReadOnlyMemory<float>, which can be passed to a vector-store client or copied into a persistence type when the client requires an array.

The provider-specific client is created at the composition boundary. Application code can depend on IEmbeddingGenerator<string, Embedding<float>> instead of OpenAIClient. Another provider can supply the same interface, but its vectors still have to belong to the index's embedding space. Changing to an incompatible model, model revision, or embedding configuration requires reindexing.

Keep the API key in the environment, a development secret store, or a managed secret service. Do not put it in source code or configuration committed to the repository.

Generate document embeddings in batches

Ingestion rarely embeds one passage at a time. Send a supported batch so the provider can process several inputs in one request.

string[] passages =
[
    "Use CancellationToken to stop long-running work cooperatively.",
    "Register hosted services with AddHostedService.",
    "A bounded channel can apply backpressure to producers."
];

(string Value, Embedding<float> Embedding)[] embeddedPassages =
    await generator.GenerateAndZipAsync(
        passages,
        cancellationToken: cancellationToken);

foreach ((string passage, Embedding<float> embedding) in
    embeddedPassages)
{
    await index.UpsertAsync(
        text: passage,
        vector: embedding.Vector,
        cancellationToken);
}

GenerateAndZipAsync keeps each input beside its returned embedding. That pairing matters once the pipeline also carries stable chunk IDs, document versions, and payload metadata.

Batch size is an operational setting, not a constant copied from this example. Providers impose different input-count and token limits. Start with a conservative size, pass cancellation through the complete ingestion path, and observe latency and throttling before increasing concurrency.

For a real pipeline, the value sent to UpsertAsync should also include a stable point ID and the metadata needed for filtering and traceability. Keep the collection schema and indexing details in the vector-store layer.

Treat the embedding profile as an index contract

The C# type only says that the vector contains float values. It does not tell the compiler which embedding space, dimensions, or comparison rules belong to those values.

Make that missing context explicit as an embedding profile.

public sealed record EmbeddingProfile(
    string Id,
    string Provider,
    string ModelIdentity,
    int Dimensions,
    string ComparisonMetric,
    string InputMode,
    string PreprocessingVersion);

public static class SupportEmbeddings
{
    public static readonly EmbeddingProfile V1 = new(
        Id: "support-text-v1",
        Provider: "OpenAI",
        ModelIdentity: "text-embedding-3-small",
        Dimensions: 1536,
        ComparisonMetric: "Cosine",
        InputMode: "default",
        PreprocessingVersion: "plain-text-v1");
}

The model and 1,536 dimensions match the default output used above. Cosine is the comparison metric selected for the index and is consistent with OpenAI's recommendation. If you request a different output size or select another model, the profile must change.

The profile does not need to be this record. It can live in typed options plus deployment metadata, as long as the application can answer which contract produced an index.

ModelIdentity should identify the model and, when the provider exposes one, the specific revision or snapshot. A deployment identifier can be tracked separately when it is operationally useful, but it does not define compatibility by itself.

Provider is provenance, not a compatibility rule. Keep it because operators may need to trace how an index was produced. Changing that value alone does not prove that the vector space changed.

InputMode and PreprocessingVersion sound similar, but they own different decisions. InputMode records model-specific behavior such as query and document roles or prefixes. The OpenAI example uses default because it does not set a model-specific mode. PreprocessingVersion identifies how the application transforms the source text.

I would still give the complete profile a stable ID. A model name on its own leaves too much unsaid.

Keep compatibility decisions and useful provenance together:

Constraint	What must stay compatible	Typical failure
Embedding space	Document and query vectors must come from the same embedding contract or from encoders explicitly designed to produce compatible representations.	Search can run successfully while ranking becomes meaningless.
Dimensions	Generated vectors must match the index schema.	The vector store rejects writes or queries.
Comparison metric	The index must use a metric appropriate for the embedding space.	Score semantics can change, and ranking quality can degrade for some models.
Input mode	Query and document prefixes or task modes must follow the model's instructions.	Retrieval quality drops even though dimensions match.
Preprocessing version	Normalization and content preparation must remain reproducible.	Reindexed and older records represent different inputs.

Qdrant, for example, requires vectors stored under one vector configuration to share a dimensionality and comparison metric. Named vectors can define separate configurations inside the same collection. Its collection documentation also notes that the metric should follow how the encoder was trained.

For the OpenAI model in this example, metric choice needs one more qualification. OpenAI recommends cosine similarity and states that its embeddings are normalized to length 1. Cosine similarity can therefore be calculated with a dot product. Cosine and Euclidean comparison also produce the same ranking for these embeddings, although their score values have different semantics.

The database can reject a vector with the wrong length. It cannot detect that a 1,536-value query came from the wrong 1,536-dimensional model. That failure is more dangerous because the request succeeds.

Validate the contract before indexing

Configuration drift should fail early, not after a retrieval-quality incident.

One small startup or integration check can verify the output shape:

public static async Task VerifyProfileAsync(
    IEmbeddingGenerator<string, Embedding<float>> generator,
    EmbeddingProfile profile,
    CancellationToken cancellationToken)
{
    ReadOnlyMemory<float> probe = await generator.GenerateVectorAsync(
        "embedding-profile-check",
        cancellationToken: cancellationToken);

    if (probe.Length != profile.Dimensions)
    {
        throw new InvalidOperationException(
            $"Embedding profile '{profile.Id}' expects " +
            $"{profile.Dimensions} dimensions but the configured " +
            $"generator returned {probe.Length}.");
    }
}

This check catches a dimensional mismatch. It does not prove that the configured model is the intended one because two models may return the same size. Verify the model identity and embedding configuration as well. Track the deployment separately when it matters for operations.

Also check the vector-store schema against the same profile before starting ingestion. The generator, collection, and query path should not each carry their own independent copy of the expected dimension.

Treat an incompatible model change as a data migration

An incompatible embedding model uses a different coordinate system from the existing index.

If only the query-side generator changes, its vectors cannot be compared meaningfully with the existing document vectors. If only new documents use the new profile, the collection ends up split across two spaces. Neither failure has to produce an exception.

For this kind of change, I create a new embedding profile and re-embed the corpus. The cost is visible. Mixing vector spaces can fail quietly, which is worse.

A migration usually needs these stages:

Create storage for the new profile with its dimensions and comparison metric.
Backfill existing source records through the new generator.
Write new or changed records to both profiles while the backfill runs, if the application must stay current.
Evaluate the new index with known queries and expected sources.
Switch reads to the new profile.
Retire the old vectors after a rollback window.

Qdrant supports migrations through a new collection and alias swap. Qdrant 1.18 and later can also add a named vector for a second model, which allows backfilling inside the existing collection. Its embedding-model migration guide shows the dual-write sequence.

Do not compare raw similarity scores from the old and new profiles as if they shared a scale. Evaluate whether each profile retrieves the expected sources at the rank positions your application uses.

Test retrieval, not attractive coordinates

Printing a few vector values proves that the API returned numbers. A two-dimensional chart can make a demo easier to understand. Neither one tells you whether the model works for your corpus.

Build a small evaluation set before committing to a profile:

representative user questions
the chunks or documents that should be retrieved
difficult wording differences and domain terms
cases where lexical search should beat semantic search
languages the application must support

Then measure whether the expected sources appear in the top results. Compare candidate models, preprocessing rules, and hybrid-search choices against the same set.

Model descriptions and public benchmarks can narrow the options. Your retrieval questions decide whether the profile is good enough for the application.

When embeddings fit

Use embeddings when the application needs semantic ranking across a candidate corpus and exact wording is unreliable. Typical examples include finding a runbook from a problem description, locating related support cases, or selecting documentation passages for RAG.

Do not use an embedding when the request already contains an exact identifier, when structured filters or SQL express the question directly, or when application code owns the decision. Keep permissions and tenant boundaries in trusted filters. A close vector is not an authorization result.

Embeddings are also a poor substitute for source metadata. Store the document ID, version, language, permissions, and original text beside the vector or in a system that the indexed point references.

Keep the contract visible

The minimal .NET call is simple:

ReadOnlyMemory<float> vector =
    await generator.GenerateVectorAsync(text, cancellationToken: token);

The production code needs to keep more than the returned values:

a provider-neutral IEmbeddingGenerator at the application boundary
an explicit embedding profile for the embedding space and dimensions
a comparison metric chosen for that embedding space
reproducible input preparation
a migration path that re-embeds existing data
retrieval tests with expected sources

If those decisions are visible, the next step is mechanical: create a vector-store collection that matches the profile, then index chunks with stable IDs and useful payload data.

What Happens When Your AI Feature Fails?

Lukas Walter — Wed, 19 Aug 2026 15:30:00 +0000

The first design pass for an AI feature should describe how it fails.

That can feel backwards when the feature does not work yet. But a timeout is already part of the design. So are a retrieval miss, malformed output, and a write that may have completed after the caller gave up waiting.

Start with one user-visible flow. At every boundary, ask what can go wrong, what may already have happened, what the application can still promise, and how anyone will know which case occurred.

Keep the answers in a failure-mode table and use it to shape the API contract, control flow, telemetry, and tests. Writing that table after the first incident is rather late.

The happy path hides product decisions

Consider a support feature that prepares a reply and optionally saves it as a draft:

resolve ticket and evaluate access
    -> retrieve authorized ticket content and policy context
    -> call the model
    -> validate the generated reply
    -> save the draft when requested
    -> return the result

The sequence is easy to understand. It also leaves almost every difficult question unanswered.

What does the user see when retrieval returns no policy documents? Does the model get a chance to answer anyway? If output validation fails, can the application repair it? If saving times out, is the draft absent, stored, or still being processed? Can a retry create a second draft? Which failure should wake an operator?

Those are part of the feature contract. If the team postpones them until implementation, the answers tend to become whatever falls out of an exception handler or SDK default.

I would rather make the awkward cases visible first. Once those decisions are written down, implementing the successful case tends to be the easy part.

Begin with the promise and the unacceptable outcomes

Before listing infrastructure failures, write down what the user believes the operation does.

For the support feature, the promise might be:

Prepare a reply from the ticket and approved support policy. Save it as a draft only when the user asks.

Now write the outcomes the system must not present as success:

a reply based on another tenant's data
a policy claim with no approved source
a valid-looking reply that failed the output contract
a "saved" result when persistence was never confirmed
a duplicate draft created by retrying an ambiguous write

This short list changes the design. Missing retrieval context cannot silently become a normal model request. A model response cannot become a persisted draft before it passes the output contract. Save success requires a confirmed write, not the absence of an exception.

The unacceptable outcomes matter more than a generic goal such as "handle errors gracefully." They state which properties the application must preserve when part of the flow fails.

Strictly speaking, not every row in this exercise is a system failure. An authorization denial can be correct behavior. NoEvidence can be a valid domain outcome. I include those adverse outcomes because they can still prevent the feature from keeping its promise, and the application needs an explicit response for them.

Start from the interaction map

If you already mapped the dependencies around the model, use that interaction map as the input. Otherwise, sketch the important calls and state changes for this one flow. Another inventory of Azure services, SDK clients, and databases will not tell you how the feature should behave.

A service can fail differently in different interactions. Reading policy context may permit a reduced result. Authorizing access does not. A database read and a draft write may use the same database but have different consequences when their outcomes are unknown.

Take one interaction at a time and ask:

How can this interaction be slow, unavailable, wrong, stale, unauthorized, duplicated, or ambiguous?
What is the effect on the user-visible operation?
Could data or external state already have changed?
What response preserves the feature's promise?
Which signal distinguishes this case in production?
How will a test force it to happen?

Use the category list to jog your memory; stop when more rows would describe the same effect and response. A model call can return valid JSON with an unsupported claim. A write can succeed while its acknowledgement is lost. Caller cancellation can arrive after work started. These cases are more useful than another row that simply says "dependency unavailable".

Build a table that can change the code

For the support flow, a first pass could look like this:

Interaction and adverse condition	Effect	Side-effect state uncertainty	Application response	Signal	Test
Ticket authorization denies access	Protected ticket content must not be disclosed or used beyond what is necessary for the authorization decision	None	Return a generic not-found or forbidden result according to application policy	Authorization outcome and controlled resource identifier	Denied principal
Policy retrieval returns no approved context	A grounded policy answer cannot be produced	None	Return `NoEvidence`; do not ask the model to fill the gap	Retrieval outcome, filters, result count	Empty retrieval result
Policy retrieval returns stale context	The reply may use obsolete rules	None	Reject the context or mark the feature temporarily unavailable	Source version and age, without document contents	Expired test document
Model call exceeds its time budget	No reply is available inside the request budget	The provider may still be processing, but no application state changed	Stop waiting, request cancellation when supported, and return `TimedOut`; retry behavior is decided separately	Attempt, elapsed time, cancellation reason	Delayed fake client
Model returns malformed structured output	The reply cannot be validated	None	Reject it or make one bounded repair attempt when the contract permits	Schema version and validation category	Invalid JSON and missing fields
Model cites a source outside the retrieved set	The reply is unsupported	None	Reject the output	Returned source IDs and validation outcome	Unknown source ID
Draft save times out after submission	The application cannot confirm whether the draft committed, so it cannot report `Saved`	Draft may already exist	Return `Unconfirmed` with the operation ID created before submission; query or reconcile that identity before another write	Operation ID, attempt, last known state	Commit succeeds, response is dropped
Best-effort observability export fails	Diagnosis becomes harder	Business state is unchanged	Continue and record the exporter failure locally when possible	Exporter health and dropped-item count	Disabled collector
Required audit or security record cannot be durably written	The feature cannot satisfy its operating obligation	A protected action may already have occurred if recording is not atomic	Apply the audit policy. When the business state and record share a transaction boundary, commit them together. Otherwise prevent the action where possible or reconcile an ambiguous outcome	Audit-write outcome, policy decision, and operation ID where applicable	Rejected audit write before and after the protected action

If the cells do not affect implementation, the table is busywork. "Log the error and retry" leaves open whether retrying is safe, what the caller receives, and when the operation stops.

An external effect and a local audit record do not share a transaction boundary. Record durable intent before the effect. Afterward, reconcile and record the final outcome. Depending on the operation, that recovery path may also need an idempotency key or compensation.

Do not try to enumerate every exception type. Group failures when they have the same effect and response. Split them when the system must behave differently.

Separate failure, effect, and response

Teams often jump from a technical symptom to a resilience mechanism:

timeout -> retry
invalid output -> retry
dependency unavailable -> fallback

That skips the decision that matters: what did the failure do to the operation?

A timeout on an idempotent policy read is not the same as a timeout after a draft write was submitted. Both may present as a timeout at the application boundary. The read has no side effect and might be attempted again within the remaining budget. The write has an ambiguous outcome. Retrying it with a new identity may create a duplicate.

Choose the response from the effect and the known state. The exception name is only one input.

For each row, I use one of a small set of response shapes:

stop and return a specific application outcome
continue with an explicitly reduced capability
make another bounded attempt when the operation is safe and time remains
move unresolved work to a durable recovery path
require human review before a consequential action
fail closed because authorization, integrity, or policy cannot be proven

The exact set belongs to the application. Callers should receive application outcomes without having to interpret exception text.

Put the outcomes in the application contract

Once the failure table stabilizes, encode the outcomes that the endpoint or UI needs to handle.

public enum ReplyOutcome
{
    Prepared,
    NoEvidence,
    InvalidGeneration,
    TimedOut,
    TemporarilyUnavailable
}

public enum DraftSaveOutcome
{
    NotRequested,
    Saved,
    RejectedByPolicy,
    Failed,
    Unconfirmed
}

public sealed record PrepareReplyResult(
    ReplyOutcome ReplyOutcome,
    string? Reply,
    DraftSaveOutcome SaveOutcome,
    Guid? SaveOperationId);

RejectedByPolicy means the application deliberately refused the save before persistence. Failed means the application knows that persistence did not commit. Unconfirmed means the write may have committed, but the application has not verified the result.

Allocate the operation ID before the write:

Guid operationId = Guid.NewGuid();

var command = new SaveDraftCommand(
    OperationId: operationId,
    TicketId: ticketId,
    Reply: reply);

DraftSaveResult saveResult = await draftStore.SaveAsync(
    command,
    cancellationToken);

The store must persist OperationId with the draft or an operation record and enforce uniqueness. Reusing the ID for a different command must fail. Persist enough command identity, such as the ticket ID and a canonical command fingerprint, to detect conflicting reuse. If the response disappears after commit, the application queries that identity, and any safe retry reuses it. Otherwise, the ID gives a worker nothing authoritative to reconcile.

The compact result type still permits invalid combinations. Production code may use factory methods or a result hierarchy to prevent them. Even so, it makes two decisions explicit: reply generation and draft persistence have separate outcomes, and an unconfirmed save is not reported as either success or failure.

The endpoint can now map those outcomes deliberately. The UI can say that a reply was prepared but its save status is still being checked. A background worker can reconcile the same operation ID. Telemetry can record stable outcome names instead of provider-specific exception messages.

Prioritize without pretending the numbers are precise

A full failure-mode analysis can grow quickly. Do not give every row equal attention.

In a review, I start with three questions:

How much harm can this cause to users, data, security, or an operational promise?
How plausible is the case in this design and environment?
How likely are we to detect it before the user or an operator has to report it?

I prefer simple priority labels such as critical, high, normal, and low over multiplying guessed scores into a number that looks scientific. Unknown write outcomes, cross-tenant data exposure, silent use of stale policy, and failures that look like success deserve attention before a clean provider error that the application already exposes honestly.

Keep the treatment decision separate: mitigate, accept, or defer. A high-priority risk can still be accepted when the team understands the consequence and decides that mitigation is not justified. "We do not support draft recovery in the first release" can be a legitimate decision if the UI never claims an unconfirmed write succeeded and the consequence is acceptable. An undocumented gap is not the same thing as an accepted risk.

Turn every important row into a forced failure

For every important row, the team should be able to force or simulate the relevant condition and effect. Some provider and infrastructure failures cannot be reproduced exactly, but their application-visible behavior usually can.

You do not need a production-scale chaos platform for the first pass. Use fake clients and controlled test doubles to return no context, delay a model call, produce malformed output, reject authorization, or lose a write acknowledgement after commit.

For each high-priority row, verify four things:

The application returns the intended outcome.
It preserves the required data and authorization properties.
It emits enough context to identify the interaction and failure category without leaking sensitive content.
Any retry or recovery path preserves side-effect safety.

Keep at least the deterministic cases in the normal test suite. Run slower dependency and recovery drills on a schedule that the team will actually maintain.

The same rows guide the code and tests. During an incident, they also tell the operator which behavior was intentional.

Give retries and runtime budgets their own pass

The analysis should identify where another attempt may be useful. Decide retry behavior holistically across the operation afterward. That does not mean retrying the whole workflow.

Whether an attempt is safe depends on idempotency, the failure class, remaining time, provider throttling, and the scope of any side effect. A retry can be a valid response to one row and make the next row much worse.

A five-second model timeout tells you little on its own. It has to fit inside the request budget for retrieval, model calls, validation, tools, persistence, and any synchronous recovery attempts. Durable recovery that outlives the request needs its own deadline or operational objective.

During the failure pass, note which rows need a retry or budget decision. Resist inventing a local retry count or timeout just to fill the cell.

When to use this approach

Use a failure-first pass when an AI feature reads protected data, relies on retrieval, makes authoritative claims, invokes tools, changes state, or creates a meaningful promise about latency and availability. It is also useful before changing a prompt, model, or provider when that change can alter output contracts or runtime behavior.

When a lighter pass is enough

A disposable local experiment with synthetic data and no side effects does not need a workshop or a large register. Write down the few cases that could invalidate the experiment, keep the error visible, and move on.

Keep the method proportional to the feature. A small experiment needs a short list, not a reliability program.

Practical takeaway

Choose one important AI flow and spend 45 minutes on its unhappy paths before adding more happy-path code.

Write the user promise and the outcomes that must never appear as success. Walk each interaction, record the effect of plausible failures, and assign an application response, a diagnostic signal, and a way to force the case in a test.

If a row has no defined response or cannot be tested, it is unfinished design work.

Sources

Microsoft Learn: Architecture strategies for performing failure mode analysis
Microsoft Learn: Transient fault handling
Microsoft Learn: Connection resiliency in Entity Framework Core
Microsoft Learn: Architecture strategies for designing a reliability testing strategy

DEV Community: Lukas Walter

Querying Qdrant from .NET

Continue from the indexed collection

Configure the query embedding

Azure OpenAI

Ollama with EmbeddingGemma

Embed the question and query Qdrant

What the query does

Filters decide eligibility and vectors decide order

A score is not a relevance decision

When to use this query path

Related reading

Sources

Building the .NET AI App Foundation

Keep the dependency direction small

Start with one actual operation

Wire configuration, model access, and tracing

Choose the HTTP contract: buffered or streaming

Leave room without building placeholders

What this foundation deliberately does not solve

When this shape earns its keep

Related reading

Sources

Indexing Your First Documents

Continue from the collection example

Use Azure OpenAI in Microsoft Foundry

Optional: generate embeddings locally with Ollama

Give the source a stable identity

Index and read back the document

Run it, then change the text

Add two more documents

When to use this indexing path

Related reading

Sources

What Should You Actually Evaluate?

Start with one outcome you can verify

Separate answer correctness from groundedness

Check retrieval before blaming the prompt

Check what the tools changed

Make failure outcomes part of task success

Keep safety failures visible

Measure latency and cost across the complete task

Match the evaluator to the evidence

Decide what would block the release

When to use this approach

When a smaller evaluation is enough

Related reading

Sources

Dependency Injection for AI Components

Start with the objects that hold state

Register one client at the composition root

Bind tools inside the operation

Give retrieval and tools their application dependencies

Let the owner dispose the client

Check composition without asking a model

When to use this structure

When not to add more DI machinery

Related reading

Sources

Human Approval as a System Boundary

Decide where human judgment helps

Prepare something the reviewer can approve

Bind the decision to a stored revision

Give waiting its own lifecycle

Recheck before committing to dispatch

Approval does not make retries safe

Test what happens after the click

When human approval helps

Related reading

Sources

Your First Qdrant Collection

What a collection owns

Run Qdrant locally

Create the .NET project

Creating once is not enough

Keep the first collection boring

Before indexing the first point

Related reading

Sources

AI Systems Need Runtime Budgets

Generate an embedding through `IEmbeddingGenerator`