Amitesh0512

Posted on Sep 6 Originally published at amiteshsurwar.com

Building a Production Agent Harness in ASP.NET Core: The Five‑Layer Blueprint

#aspnetcore #agenticai #semantickernel #azure

Quick Answer

building a production agent harness in asp.net core: A 5‑layer ASP.NET Core agent harness uses Redis, Cosmos DB, Azure Functions, and event‑driven orchestration to scale, observe, and control costs in production.

Prototype Failure: Architecture Pitfalls

You’ve spun up a quick ASP.NET Core API, wired up Semantic Kernel, and the first 100 requests finish in under a second. The next 200 hit a 504. The culprit isn’t the LLM; it’s the architecture you used to glue the pieces together. In production, a single point of failure—no retry policy, shared in‑memory state, or unbounded request queue—turns a prototype into a service that stalls and crashes.

Real‑World Example: Order‑Processing Bots in an E‑Commerce Platform

An online retailer runs a fleet of micro‑services: inventory, payments, shipping, and a new “Agentic” layer that answers customer queries, places orders, and suggests upsells. The agent layer is built on Semantic Kernel, calling Azure-openai-service-vs-gpt4-api-for-net-microservices-a-deepdive-for-architects-20260830" class="internal-link">Azure OpenAI for natural language understanding and Azure Cosmos DB for persistent state. When a surge of 10k concurrent users hits the system during a flash sale, the orchestrator’s in‑process queue back‑pressures, the Redis cache is saturated, and the LLM throttles. The result: a cascade of timeouts, degraded user experience, and a spike in support tickets.

Trade‑Offs in a Production Agent Harness

State Management – In‑memory dictionary is fast but not shareable; Redis gives consistency at the cost of network latency.
LLM Invocation – Direct calls from the API keep latency low but tie the user’s request to a single LLM instance; a serverless function off‑load decouples traffic but introduces cold start.
Orchestrator Design – A monolithic orchestrator simplifies code but becomes a bottleneck; a distributed event‑driven coordinator scales horizontally but adds operational complexity.
Vector Store Choice – Azure Cognitive Search offers managed scaling and vector search, but its cost per query scales with the size of the index; an in‑house Qdrant cluster gives lower per‑query cost but requires self‑management.
Observability Granularity – Full OpenTelemetry traces provide root‑cause visibility but add ~10 % CPU overhead; lightweight metrics reduce overhead but may miss subtle race conditions.

When This Fails in Production

State Desynchronization – Multiple pods read and write the same conversation context without a lock, leading to lost messages and inconsistent responses.
LLM Throttling – Azure OpenAI’s request limits are hit during a traffic spike; the API returns 429 without a back‑off strategy.
Cache Eviction – Redis’ default eviction policy removes recent conversation windows, causing the agent to re‑query the database and double the latency.
Vector Search Latency – A growing vector index in Azure Cognitive Search causes query times to climb from 20 ms to 200 ms, pushing the overall request beyond the SLA.
Orchestrator Bottleneck – A single orchestrator instance becomes a single point of failure; its thread pool is exhausted under load, leading to thread starvation.

Common Mistakes Engineers Make

Assuming the ASP.NET Core request pipeline can handle LLM calls inline; in reality, the LLM is a long‑running I/O operation that should be decoupled.
Storing conversation history in a single Cosmos DB container without partition keys; this results in RU spikes and poor query performance.
Using the default Redis eviction policy (volatile-lru) for short‑term context; high traffic pushes recent data out of cache.
Neglecting to instrument MCP actions; without correlation IDs, troubleshooting a multi‑step conversation becomes impossible.
Deploying the orchestrator as a stateless Web API; it ends up holding the conversation context in memory, which is lost when a pod is rescheduled.

Better Approach Based on Experience

State Layer – Use Redis for short‑term context with a maxmemory-policy set to volatile-ttl and a TTL of 30 min. Persist only facts that survive beyond a session to Cosmos DB with a conversationId partition key.
LLM Off‑load – Wrap LLM calls in Azure Functions Premium. The function receives the conversation window via a Service Bus queue, processes it, and pushes the result back to the orchestrator. This isolates the LLM latency from the API request.
Event‑Driven Orchestrator – Replace the in‑process orchestrator with a lightweight event bus (Azure Service Bus or Kafka). Each agent publishes a ToolRequest event; the orchestrator subscribes and aggregates responses. This decouples the orchestrator from the agent lifecycle.
Vector Store Strategy – Keep a small, hot slice of the vector index in Redis (using RediSearch) for the most recent 1k turns; push older vectors to Azure Cognitive Search asynchronously.
Observability – Emit a single correlation ID per conversation, propagate it through all downstream calls, and log MCP actions with the ID. Use OpenTelemetry to capture token counts and LLM latency, and expose a Grafana dashboard with alerting on > 150 ms average.
Scaling – Deploy the orchestrator in an AKS cluster with HPA based on queue length, not CPU. Use a sidecar for Redis connection pooling to avoid per‑pod connection churn.
Cost Control – Cache LLM embeddings in Redis to avoid re‑calling the embedding endpoint. Use Azure Cost Management to monitor LLM usage per tenant and apply throttling budgets.

Use-Case Architecture Trade-Offs

Use‑Case	Recommended Pattern	Key Trade‑Offs
High‑throughput customer support (<10k RPS)	Event‑driven orchestrator + Azure Functions LLM	Adds cold‑start latency but scales linearly; requires more DevOps overhead.
Real‑time order placement (latency < 200 ms)	Synchronous orchestrator with in‑process LLM calls	Simpler, but LLM throttling directly affects SLA.
Hybrid: occasional heavy queries + frequent light traffic	Hybrid orchestrator: in‑process for short tasks, off‑load heavy tasks to Functions	Complexity in routing; requires careful cache invalidation.
Multi‑tenant SaaS with strict cost limits	Per‑tenant Redis shards + shared Azure Cognitive Search	Higher operational cost for Redis but tighter cost control per tenant.

Performance & Scaling Notes

Cache the LLM prompt template and tool definitions; avoid re‑serialization per request.
Batch vector embeddings for a batch of 50 turns to reduce per‑embedding API calls.
Use a connection pool for Redis (e.g., StackExchange.Redis) and keep a single ConnectionMultiplexer per pod.
Implement a TokenLimiter that throttles requests per tenant to stay within Azure OpenAI quota.
Measure LLM token usage per conversation and expose it as a metric; set alerts on anomalous token spikes.
Deploy Azure Functions with a Premium plan and keep warm-up triggers (e.g., a scheduled ping) to reduce cold starts.
Use Azure Managed Identities for all service-to-service calls to eliminate credential rotation overhead.
Enable az acr repository show-tags --repository for automated image scanning; avoid shipping containers with vulnerable dependencies.

What are the primary reasons prototype agent harnesses fail in production?

They lack retry policies, use shared in‑memory state, have unbounded request queues, and expose a single orchestrator point of failure, causing stalls and crashes under load.

How should I manage conversation state across multiple pods?

Store short‑term context in Redis with a volatile‑ttl policy and 30‑minute TTL; persist long‑term facts in Cosmos DB using a conversationId partition key for consistency.

What is the recommended approach for decoupling LLM calls from the API request?

Offload LLM work to Azure Functions Premium, enqueue the conversation window on Service Bus, process it asynchronously, and return the result to the orchestrator via a callback.

Which observability patterns should I implement for a production agent harness?

Emit a single correlation ID per conversation, propagate it through all services, use OpenTelemetry traces and metrics, and expose dashboards with alerts on >150 ms latency or token spikes.

When should I choose an event‑driven orchestrator over an in‑process orchestrator?

Use event‑driven when you need high throughput, burst handling, or multi‑tenant isolation; choose in‑process for low‑latency, simple workloads where LLM throttling is acceptable.

Building a Resilient Cost‑Efficient Agent Harness

A production agent harness is not a glorified prototype; it’s a distributed system with state, retries, observability, and cost controls baked in. By treating the orchestrator as a decoupled service, persisting state in Redis and Cosmos DB, and off‑loading LLM calls to serverless functions, you can build a system that survives traffic spikes, scales horizontally, and stays within budget. The trade‑offs you make today—between latency, complexity, and cost—will determine whether your agent layer is a competitive advantage or a silent bottleneck.

What to Ship

Add a health‑check endpoint that aggregates status of all five layers and exposes it on /health.
Wrap external API calls in Polly circuit breakers with a 3‑second timeout and exponential back‑off, and expose metrics via app.Metrics.
Configure Hangfire or Quartz.NET to schedule bot jobs, ensuring each job runs in its own scoped service provider and logs start/finish timestamps.
Add a Redis‑backed distributed lock around order‑processing bots to prevent duplicate processing across instances.
Enable Azure App Service or Kubernetes autoscaling based on CPU usage of the “Execution” layer, and set a max concurrency limit on the “Execution” worker.

DEV Community