I Built a Production AI Layer Inside a Legacy ASP.NET Core App — and It Broke in Ways Tutorials Never Mention

Sharad Kumar — Tue, 12 May 2026 00:34:14 +0000

Introduction

Most LLM tutorials assume you start from nothing. A blank project. A clean architecture. No constraints. No legacy code. No deployment history. No production traffic.

That is not how real systems work.

I spent one week integrating a production-grade AI service layer into an existing ASP.NET Core MVC e-commerce system that was already live, already structured, and already dependent on architectural decisions I couldn't change. The challenge wasn't calling an LLM API. It was designing an AI layer that could survive inside a real backend system without breaking testability, without leaking cost, without becoming tightly coupled to the domain, and without collapsing the moment the model or provider behaved unexpectedly.

The feature itself, a tone-aware product description generator, is simple. The system design behind it is not. This article is about the architectural decisions, the production failure modes, and the assumptions that broke the moment an LLM entered a real backend system.

AI System Design

The Provider/Domain Seam: The Most Important Structural Decision

The first instinct when adding AI to any backend is to create one service class that does everything. One class, one interface, done.

That instinct produces a system you cannot test, cannot swap, and cannot extend without touching everything.

The correct design draws a hard seam between two concerns:

Provider layer (IAIService / AzureOpenAIService): knows how to talk to an LLM. Accepts two strings (system prompt, user prompt), returns a string. Knows nothing about what a book or product is.
Domain layer (IProductAIService / BookAIService): knows what a description request looks like, what tone means, and how prompts should be constructed. Knows nothing about Azure, HTTP, or Semantic Kernel.

Above that seam: domain logic, testable with zero real network calls. Below it: provider mechanics, swappable without touching anything above. Swap Azure OpenAI for Ollama, write one new class, and change one registration. The domain layer doesn't notice.

"Naming is a design smell detector. My original class was called OpenAIService and it implemented both interfaces. When I tried to give it an honest name, I couldn't because it was doing two things."

The Generic Wrapper That Eliminates Try/Catch Everywhere

With the seam in place, the next question was: what does every AI call return? My first draft returned plain strings or domain objects directly. That looked fine until I had to handle failures, and I started writing the same try/catch block in three different places.

Every AI call in the system returns AIResponse<T>, not a raw result type. The wrapper carries:

public class AIResponse<T>
{
    public bool    Success      { get; init; }
    public T?      Data         { get; init; }
    public string? ErrorMessage { get; init; }
    public bool    FromCache    { get; init; }
    public int     TokensUsed   { get; init; }

    public static AIResponse<T> Ok(T data, int tokens = 0, bool fromCache = false) => ...
    public static AIResponse<T> Fail(string error) => ...
}

The payoff is architectural: no caller above the provider layer ever writes a try/catch. Every feature just checks result.Success. Error handling is decided once, at the boundary where it's caught, and that discipline holds automatically across every AI feature you add later.

AIResponse<string> at the provider layer. AIResponse<ProductDescriptionResult> at the domain layer. AIResponse<ChatResult> when the chatbot arrives in Week 3. Same envelope, different payloads.

The static factory methods aren't just convenience they make invalid states unrepresentable. You cannot call Ok() and get Success = false. You cannot call Fail() and accidentally leave ErrorMessage null.

The Inheritance Trap I Almost Fell Into

My first instinct was to write ProductDescriptionResult : AIResponse. It compiles. It works. But the moment you try to store a ProductDescriptionResult in a database or pass it across a service boundary, it drags Success, ErrorMessage, and TokensUsed with it, infrastructure concerns that mean nothing in the domain layer.

Composition is correct here. The result class is a pure data payload. The wrapper is the envelope. Neither inherits from the other.

System Prompts as Architectural Contracts

This is where prompt engineering actually lives in the codebase, not scattered across controllers, not inline in HTTP calls, but centralised in a single switch expression:

private static string BuildSystemPrompt(DescriptionTone tone) => tone switch
{
    DescriptionTone.Professional =>
        "You are a professional copywriter for a premium book retailer. " +
        "Write concise, authoritative product descriptions. " +
        "Never fabricate awards, authors, or facts not provided.",
    DescriptionTone.Casual =>
        "You are a friendly book recommender. Keep it warm and enthusiastic.",
    // ...
};

Two things worth calling out:

BuildSystemPrompt() and BuildUserPrompt() are separate private methods by design. Developer rules and user input must never accidentally merge. That separation is what makes the service layer both testable and secure.
Hardcoded system prompts are a deliberate decision, not laziness. They represent fixed feature contracts that don't change at runtime, they don't bleed across features, and they're easy to audit.

The temperature misconception is worth correcting explicitly: temperature is not a safety control. It's a creativity dial. The system prompt is where your behavioural constraints live. Three controls, three separate jobs: system prompt = instructions, temperature = style, max_tokens = budget.

ChatHistory: Single-Turn Today, Multi-Turn Ready Tomorrow

Most beginners send one big string to an LLM API. Using Semantic Kernel's ChatHistory correctly is a signal that you understand how chat models actually work:

var chatHistory = new ChatHistory();
chatHistory.AddSystemMessage(systemPrompt);
chatHistory.AddUserMessage(userPrompt);

The description generator creates a new ChatHistory per call by design. When the chatbot arrives in Week 3, the same pattern extends with no architecture change, just persistence added.

CancellationToken: The Hidden Cost Leak

Every async AI method in the chain accepts and propagates a CancellationToken. This isn't just politeness; it's cost control. If a user closes their browser tab mid-request and the token isn't propagated all the way to the Azure SDK call, your app completes the API call and gets billed for a response nobody receives.

A token accepted but not passed to the next await is worse than not accepting it at all; it gives false confidence that cancellation is handled. The chain must be unbroken: Controller → Service → Provider → Azure SDK call.

Response Caching: Proving It Works in the UI

Caching is wired at the provider layer with a hash-based key:

var cacheKey = $"ai:text:{systemPrompt.GetHashCode()}:{userPrompt.GetHashCode()}";

The AIResponse<T> wrapper carries a FromCache boolean all the way to the admin UI, where it renders as ⚡ Cached vs ✨ Generated. That's not decoration, it's verification. You can prove your caching is working in a live demo without opening Application Insights.

EnableCaching is a feature flag in AISettings, not a hardcoded true. You can flip it in the Azure App Service configuration without redeploying. That's a production pattern, not a tutorial habit.

The AI Cost Dashboard: Why Almost Nobody Builds This

Most AI portfolio projects show features. Mine also shows cost. The admin dashboard tracks tokens per feature per day, cost per request, and cache hit rate. This makes the economics of AI visible and is the difference between someone who built a feature and someone who shipped one.

Concrete number: approximately $15 total API spend over 8 weeks of development, with gpt-4o-mini during development (roughly 15x cheaper than gpt-4o) and caching in place.

Supporting Architecture Decisions

Three decisions that shaped how the AI layer was placed and wired were included because they caused real problems, not because they're interesting trivia.

AI infrastructure belongs at the application root, not inside feature folders. The app used area-based routing. The instinct was to put Services/AI/ inside an Area. Wrong call, UI grouping doesn't determine where cross-cutting infrastructure lives. AI features span the whole application. Scoping them to an Area implies boundaries that don't exist and creates coupling that becomes painful when the chatbot and search features arrive.

All AI wiring lives in one extension method. One line in Program.cs: builder.Services.AddAIServices(builder.Configuration). Everything else is encapsulated. The subtle thing to know: if you register the same concrete class twice under different interfaces without careful use of GetRequiredService, you can end up with two separate instances per request instead of one. Most tutorials never flag this.

Secrets never touch the config file. ApiKey exists in the settings class but is absent from appsettings.json. User Secrets fill it locally, App Service Application Settings fill it in production. The application code is identical in both environments everything is injected through the same configuration pipeline.

Challenges & Learnings

The Debugging Arc: 404 → 400 → 200

Getting the AI endpoint working produced three sequential errors, each from a different layer:

404 : routing was treating AI as an area name. Fix: explicit [Route("AI")] on the controller.
400 : the JavaScript payload sends "Professional" as a string; the C# model declares a DescriptionTone enum. ASP.NET Core's default deserializer returns a silent 400 without bridging them. Fix: add JsonStringEnumConverter. This is an AI-specific integration point — the LLM-facing tone selector has to round-trip correctly through the API layer.
200 : success.

Each error was a different subsystem. Real integrations rarely have a single root cause.

The Cascade Failure at Deployment

At this point, the feature was working locally. Deployment looked straightforward.

It wasn't.

After deploying, MaxTokens was reading as 0. Completely unrelated to the actual cause: a missing Azure App Settings entry was causing AIServiceExtensions.cs to crash at startup, leaving the settings object in a zeroed-out state. A NullReferenceException deep in Semantic Kernel setup was the symptom. A missing config key was the cause.

The lesson applies specifically to AI service registration: validate that your AI configuration is present and well-formed at startup, loudly, before any service gets built. A ?? throw on the config read gives you a readable failure message at the right location, not a cryptic error three layers downstream.

The Two Hours I Lost to a URL

This one stings to write. I spent the better part of an afternoon convinced my Semantic Kernel wiring was broken. Checked the DI registration twice. Re-read the SK docs. Added logging everywhere. Silent failure, no exception, no response.

The actual problem: I had copied the full REST endpoint URL from the Azure portal, something like https://your-resource.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-01 directly into my config. Semantic Kernel only wants the base domain. It constructs the rest itself. One wrong URL format, zero helpful error messages, two hours gone.

I'm including this because the Azure portal genuinely shows you the full path and it looks correct. It isn't.

Use https://your-resource.openai.azure.com/, the base domain only. Semantic Kernel builds the path from there based on your deployment name and API version.

LLM Error Messages Are a Security Leak

A raw Azure OpenAI 401 or timeout exception carries endpoint URLs, subscription hints, and SDK internals in ex.Message. None of that should reach a client. The AI service boundary catches the exception, logs it internally with full context, and returns a sanitised message to the caller. One line of difference, significant surface reduction.

The TinyMCE Silent Failure

The AI card writes to the product description field via textarea.value = text. This silently does nothing when TinyMCE is active. TinyMCE replaces the DOM element entirely. No error, no feedback. The button appears to work, but nothing changes. Lesson: The AI integration point in the UI needs to know what the UI is actually made of.

Key Takeaways

Draw the seam before you write any code. The provider/domain split is the foundational decision. Everything else, testability, swappability, and cost control, follows from it.

Make failure a return value, not a control flow mechanism. An AI system has expected failure modes: timeouts, rate limits, and degraded responses. AIResponse<T> handles them at the boundary. No caller above it needs a try/catch.

System prompts are architectural contracts, not strings. They live in one place, they don't change at runtime, and they're the only place your business rules for AI behaviour exist. Treat them with the same discipline as interface contracts.

Temperature is a creativity dial, not a safety control. The system prompt is where constraints live. Knowing the difference affects every feature you build on top of the same provider.

Propagate CancellationToken all the way to the LLM call. A broken chain gives false confidence and silently leaks API cost when users abandon requests.

Make cache hits observable. The FromCache signal in the response wrapper isn't decoration; it's the only way to verify your caching is working without opening a monitoring dashboard.

Cost is a first-class concern, not an afterthought. Token tracking per feature per day is what separates a portfolio project from a production system. Almost no one builds this. Build it anyway.

AI startup failures should be loud and specific. A missing config key that crashes deep inside Semantic Kernel setup is far worse than a clean, early "AI configuration is missing" exception at the service boundary.

Conclusion & What's Next

Week 1 is one AI feature with a production-grade system behind it: a layered architecture with a hard provider/domain seam, a generic response envelope, observable caching, structured logging, graceful degradation, and cost tracking. The feature is modest. The system thinking is not.

Week 2 is RAG semantic search using text-embedding-3-small and Azure SQL Vector, with hybrid search (vector + keyword in parallel) to handle exact matches that pure vector retrieval handles poorly. The agentic chatbot in Weeks 3 - 4 depends on RAG to ground its responses, which is why RAG comes first. Dependency-first sequencing isn't just planning hygiene; it prevents retrofitting at the seam where two features meet.

The design decisions in Week 1 were made with Week 4 in mind. That's what production AI system design actually is.

Built on: ASP.NET Core MVC · Azure OpenAI · Semantic Kernel · Microsoft.Extensions.AI · gpt-4.1-mini · IMemoryCache

🤝 Connect with Me

If you're building AI into .NET or just following along, let's connect.

💼 LinkedIn
🐙 GitHub

DEV Community: Sharad Kumar