Before the First Pixel Reaches the Model: 6 Questions for Image Input in .NET 10

The happy path for vision in .NET is short now. With Microsoft.Extensions.AI, an image is just another content part on a ChatMessage, and any vision-capable IChatClient can take it. You can have a receipt reader working in an afternoon.

Then it meets real users. Someone uploads a 9 MB phone photo, someone else sends a GIF, a phone labels its JPEG image/jpg, and someone uploads a picture of their lunch to the receipt endpoint. None of that is the model's problem. It's yours, and it's cheaper to solve before the bytes leave your process.

This post is the list of questions I now answer before any image reaches a model. It comes from a receipt and invoice extraction sample I built on Microsoft.Extensions.AI 10.11 in .NET 10: an input-policy middleware, typed extraction, a multipart minimal API, a CLI, and 25 tests (19 unit, 6 integration) that run offline with no API key. The complete project is in Tech Skill Builder.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10.

Question 1: What does an "image message" actually look like?

No special image type. A user message holds a list of content parts, so you put the instruction and the bytes side by side:

var message = new ChatMessage(ChatRole.User,
[
    new TextContent(prompt),
    new DataContent(imageBytes, mediaType) { Name = fileName },
]);

DataContent is for bytes you already have (uploads, camera captures, test fixtures). UriContent points at an image somewhere else. Setting Name costs nothing and makes provider payloads easier to read later. That's the whole multimodal API surface you need; everything else in this post is about what surrounds it.

Question 2: Where do the limits live?

My first instinct was the upload endpoint. Check the size and type there, done. The problem is that the endpoint is only one caller. A background worker or another service that gets the IChatClient from DI never goes through it.

So the limits live in the chat pipeline itself, as a DelegatingChatClient that sits outside the provider and inspects every message before passing it on:

public sealed class VisionInputPolicyChatClient : DelegatingChatClient
{
    public override Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
    {
        var list = messages as IReadOnlyList<ChatMessage> ?? messages.ToList();
        Enforce(list); // throws VisionInputPolicyException
        return base.GetResponseAsync(list, options, cancellationToken);
    }

    // GetStreamingResponseAsync must call Enforce too, or streaming becomes the bypass.
    // Enforce: MIME allowlist, bytes per image, images per request (omitted here)
}

The defaults in the sample are PNG, JPEG and WebP, 2 MB per image, and 3 images per request. They're options, so you tune them per product. It's wired up with ChatClientBuilder, where the first Use is the outermost layer, so policy runs before anything else.

I still check the file size at the API edge too, before buffering the upload, because there's no point reading 50 MB into memory just so the middleware can reject it. Two checks, one at the door and one in the pipeline.

One thing I'd tighten: the middleware only inspects parts whose media type starts with image/. A DataContent labeled application/pdf isn't counted or checked by it at all. In the sample that's covered at the HTTP edge, where a FluentValidation rule only allows the three image types. But if other callers can reach the pipeline, decide explicitly what happens to non-image parts instead of letting them pass by default.

Question 3: Is image/jpg a JPEG?

Not officially. image/jpeg is the registered type, but phones, browsers and file-extension guessers disagree, and some send image/png; charset=binary with a parameter on the end. If your allowlist compares raw strings, you'll reject valid photos and nobody will know why.

Normalize once, in shared code that both the validator and the middleware use:

internal static string NormalizeMediaType(string? mediaType)
{
    // trim, drop anything after ';', map image/jpg to image/jpeg, lowercase
    // (implementation omitted here)
}

Two unit tests pin it: an image/jpg JPEG passes the policy, and image/png; charset=binary normalizes to image/png.

Question 4: What counts as a failure?

There are two kinds of "no", and mixing them makes support miserable:

• The caller broke the rules: wrong type, too big, too many images. The policy middleware throws VisionInputPolicyException, and the API maps it to 400. The provider is never called.
• The model gave you nothing usable: empty response, JSON that won't deserialize, an extraction with no fields. The extraction service returns a failed outcome with a clear error instead of a half-filled object that looks almost right in a UI.

For the typed call, pass AIJsonUtilities.DefaultOptions and ask for a JSON schema response format:

var response = await chatClient.GetResponseAsync<ReceiptExtraction>(
    message, AIJsonUtilities.DefaultOptions, useJsonSchemaResponseFormat: true, cancellationToken: ct);
// TryGetResult, a fallback parse of response.Text, and the "empty extraction" check (omitted here)

Use those options rather than a fresh new JsonSerializerOptions { ... }. A bare instance without a TypeInfoResolver throws once the library marks it read-only, which is not an error you want to meet in production.

And here's the case that surprised me: a successful extraction is not the same as a receipt. Send the sample a blank photo and the result is a success with documentKind: "Unrelated" and no totals. Technically correct, the model looked and reported what it saw. But if your caller only checks "success", it'll happily file a photo of lunch as an expense. Check the document kind and the confidence before you act on the numbers.

Question 5: How do you test this without a vision model?

Calling a paid vision API from unit tests is slow, costs money, and gives you answers you can't assert on. So the bottom of the pipeline in CI is a fake IChatClient, and the trick is making it "recognize" images deterministically:

1. Create tiny valid PNGs (under 200 bytes each in the sample).
2. Embed an ASCII marker in each one, such as FIXTURE:RECEIPT_CAFE, in a PNG text chunk.
3. The fake scans the DataContent bytes for the marker and returns a fixed extraction for it, with a SHA-256 lookup of the exact checked-in files as a fallback in case a re-save drops the text chunk.

internal static string? IdentifyData(ReadOnlySpan<byte> bytes)
{
    if (ContainsAscii(bytes, MarkerReceipt)) return MarkerReceipt;
    // invoice and blank-scene markers, then the SHA-256 fallback (omitted here)
}

Same bytes, same JSON, on every machine. That makes real assertions possible: the receipt fixture gives "Cairo Corner Cafe" with two line items, the invoice fixture gives an invoice, and the blank one gives "Unrelated". Keep the fake even after you plug in a real model. It's what catches policy and parsing bugs in milliseconds.

Be honest in your README about what the fake is: a fixture matcher, not a classifier. For UriContent, mine just looks for "receipt" or "invoice" in the URL.

Question 6: Should you accept image URLs at all?

UriContent is handy for images on a CDN you control. For user-supplied URLs, think twice:

• You can't check the size until something downloads it, so the policy can only enforce the type and the count.
• If you accept any URL from a user, you've built a way to make your system (or your provider) fetch arbitrary addresses.

My rule: user uploads go in as DataContent. UriContent only for hosts on an allowlist, and never on a public endpoint without one.

The checklist

• [ ] Are the image limits inside the chat pipeline, not only on one endpoint?
• [ ] Does the streaming overload enforce the same rules?
• [ ] Is the media type normalized once and shared?
• [ ] Is it decided what happens to non-image parts?
• [ ] Are "caller broke the rules" and "model gave nothing usable" separate outcomes?
• [ ] Does the caller check the document kind, not just success?
• [ ] Do the tests run offline against fixtures, with no API key?
• [ ] Are image URLs restricted to hosts you control?

What I didn't verify

All 25 tests run against the offline fake, including the 6 WebApplicationFactory integration tests, so no real vision model was called. The real-model path is a matter of passing a vision-capable IChatClient as the inner client, but I haven't measured extraction quality with one.

Get the complete project

The snippets above are the shape of it. The complete, working .NET 10 solution is in Tech Skill Builder: the policy middleware with both overloads, the pipeline factory and DI registration, the typed extraction service with its fail-closed checks, the offline fixture-based fake, the fixtures themselves, a multipart minimal API (POST /api/vision/extract, GET /api/health) with FluentValidation, a CLI demo, and all 25 tests, plus a step-by-step PDF guide. No API key needed.

Tech Skill Builder gives you complete, working, tested .NET projects with full source code and step-by-step PDF guides, with new ones added regularly. Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. Join Tech Skill Builder

What's the strangest image a user has sent to your AI feature, and did your code notice?

Comments

[AI Flip Room]

Great checklist. One I'd add from the room-photo side: apply the EXIF orientation before the bytes go anywhere. A phone photo that looks upright in the browser can reach the model sideways, and the model will happily describe a room lying on its side. Also +1 on "success is not the same as a receipt". The same check matters for any domain: a photo of a screen or a floor plan sent where a room photo was expected should fail loudly, not quietly succeed.