Amitesh0512

Posted on Oct 14 Originally published at amiteshsurwar.com

Connecting Claude and Other LLMs to a Custom .NET MCP Server – A Production‑Ready Guide

#aiarchitecture #semantickernel #mcpserverintegration #azureopenai

Quick Answer

connecting Claude and other LLMs to a custom .NET MCP server: Learn how to architect, secure, and optimize a custom .NET MCP server for Claude, Azure OpenAI, and other LLMs. Real‑world code, performance tricks, and production pitfalls included.

Black-Box LLM Pitfalls

When you bolt Claude, GPT‑4 or any other LLM straight into a .NET endpoint, the first 200 requests look great. But as soon as you hit a few hundred concurrent users, you start seeing a cascade of symptoms: 200‑plus millisecond latency, sporadic 429s, cache pollution, and a brittle tool‑calling pipeline that breaks on the slightest provider change. The root cause is that most quick‑starts treat the LLM as a black‑box HTTP call and ignore the Model Context Protocol (MCP), the contract that lets you swap providers, reuse KV caches, and enforce tenant isolation without touching business logic.

Real‑World Example

Consider a SaaS company that offers a knowledge‑base chatbot to its customers. They ship a minimal API that forwards every user query to Claude. Within a week the product is used by 12,000 concurrent sessions. Suddenly the team sees:

Average response latency jumps from 800 ms to 3.2 s.
Rate‑limit errors from Claude surface 12% of the time.
Tenant A receives answers that belong to Tenant B.
Tool calls (e.g., SQL lookup) start timing out because the adapter no longer preserves the tenantId in the cache key.

Under the hood, the API was a simple wrapper around ClaudeClient.ChatAsync and a naive MemoryCache. There was no MCP layer, no unified tool registry, no observability, and no failover strategy. The result: a production system that can’t scale, is insecure, and is hard to debug.

Trade‑offs

Protocol‑agnostic vs. provider‑specific optimisations

Pros of MCP: One codebase for all LLMs, centralised caching, consistent observability, easier compliance checks.
Cons: You lose provider‑level tuning knobs (e.g., Claude’s parallel_requests or Azure-ai-foundry-vs-azure-openai-service-observability-in-production-20261007" class="internal-link">Azure’s function_call Semantics) unless you expose them in the MCP schema. Adding a new field means updating every adapter and potentially breaking existing deployments.

Centralised cache vs. distributed cache

Pros: A single Redis cluster simplifies key design and eviction policies.
Cons: Network latency to Redis can dominate inference time if the cluster is in a different region. A per‑tenant cache shard can mitigate this but increases operational complexity.

Strict tenant isolation vs. performance

Pros: Prefixing keys with {tenantId}: guarantees no cross‑tenant leakage.
Cons: It reduces cache hit rates for shared system prompts. A shared cache with a deterministic hash of the system prompt and tenant ID can mitigate but requires careful key design.

Custom MCP vs Provider SDK Comparison

Use the following matrix to decide whether to adopt a custom MCP Server or stick with provider SDKs.

Criterion	Custom MCP	Provider SDK
Multi‑tenant isolation required	✓	✗
Need to switch providers at runtime	✓	✗
Observability across providers	✓	✗
Latency budget < 1 s	✓ (with careful caching)	✓ (but no cache reuse)
Operational overhead acceptable	✓	✗ (less code but more provider lock‑in)

In most SaaS or enterprise scenarios, the benefits of a MCP Server outweigh the extra operational burden.

When This Fails in Production

Even with a well‑designed MCP Server, you can hit failure modes:

Cache stampede – If many tenants hit a cache miss for the same system prompt, the server will flood the LLM backend. Mitigate with LazyCache or a SemaphoreSlim per key.
Provider drift – When a provider updates its API (e.g., Claude adds a new field), the adapter can silently break. Implement a versioned MCP schema and a health‑check that validates adapter compatibility.
Back‑pressure misconfiguration – A bounded channel that’s too small will drop requests under load; too large will consume memory. Tune the channel size based on observed QPS and average inference latency.
Security mis‑configuration – If tenant isolation is disabled or the API key is leaked to a malicious tenant, the KV cache can be read or corrupted. Use Azure AD scopes that map 1 to tenant IDs and enforce them in every request.

Common Mistakes Engineers Make

Ignoring the Metadata bag – Developers often forget to propagate tenant IDs into the adapter, leading to cross‑tenant cache pollution.
Hard‑coding system prompts – Embedding the system prompt in code means every provider change requires a redeploy. Store system prompts in a shared KV store and reference them by hash.
Over‑caching the entire conversation – Caching the full chat history can quickly exhaust Redis memory. Cache only the last N tokens or the prompt + first assistant message.
Not handling provider rate limits – A 429 from Claude is treated as a fatal error. Implement back‑off and a fallback policy (e.g., switch to Claude Instant).
Missing observability hooks – Without tracing at the adapter level, you can’t correlate latency spikes to specific providers or tool calls.

Better Approach Based on Experience

In a recent production rollout for a financial services firm, we adopted the following pattern that cut latency by 35% and eliminated cross‑tenant leaks:

Use RedisCache with HashTags to group keys by tenant. The key format is {tenantId}:prompt:{hash} and the hash is a SHA‑256 of the concatenated system prompt + user message.
Implement a RequestQueue that is a bounded Channel<McpChatRequest> with a back‑pressure strategy that returns HTTP 429 when full.
Expose a ProviderRegistry that maps provider names to adapters and includes a HealthCheck endpoint that pings each adapter’s /health route. If a provider is unhealthy, the registry marks it as degraded and routes traffic to a fallback provider.
Instrument the MCP server with OpenTelemetry, exporting traces to Azure Monitor. Use span.SetAttribute("provider") and span.SetAttribute("tenant") to filter dashboards by tenant.
For tool execution, use a ToolExecutorFactory that lazily loads executors from a DI container. This keeps the startup footprint small and allows adding new tools without redeploying the MCP server.
Deploy the MCP server as a stateless container in Azure Container Apps with a scale‑out rule based on CPU% > 70% and QPS > 200. Use a sidecar Redis instance in the same subnet to reduce network hops.

Performance Considerations

Inference latency – The adapter adds ~5 ms overhead. Cache hits reduce token count by 15–25%, cutting provider cost per request.
Throughput – With a 30 ms per request average, a single container can handle ~3k QPS. Scale horizontally by adding more replicas; use Kubernetes HorizontalPodAutoscaler with custom metrics.
Cache hit ratio – A 70% hit ratio on the KV cache keeps token usage low. Monitor CacheHitRate in Redis and adjust TTLs accordingly.
Network hops – Keep the MCP server, Redis, and LLM provider in the same region to avoid 200–300 ms round‑trips. If you must cross regions, use Azure Front Door to route traffic and reduce latency.

Scaling Notes

When scaling a MCP server, remember:

Statelessness – All state must be external (Redis, blob storage). This allows the autoscaler to spin up new instances without losing context.
Connection pooling – Use HttpClientFactory to pool connections to LLM providers. Disable KeepAlive for providers that don’t support it to avoid socket exhaustion.
Graceful shutdown – Implement IHostedService.StopAsync that drains the request queue before terminating. This prevents orphaned requests that never hit the LLM backend.
Observability – Expose a /metrics endpoint for Prometheus. Include metrics for request_latency_seconds, cache_hit_rate, provider_errors_total, and tenant_requests_total.
Cost control – Use Azure Cost Management to set alerts on token usage. Combine this with a RateLimiter that caps requests per tenant to avoid runaway costs.

What to Ship

Expose a single, unified /chat REST endpoint that accepts a JSON body with model, prompt, temperature, max_tokens and returns a standardized JSON response containing content, usage, provider, and cost_estimate.
Implement a provider‑router that maps the model field to the corresponding provider SDK via a shared interface, and add a fallback path to an alternative provider when the primary returns an error or times out.
Add per‑request cost estimation headers (e.g., X-Estimated-Cost) derived from live provider pricing tables, and emit provider‑specific latency metrics to your observability stack (Prometheus/CloudWatch).
Integrate a circuit‑breaker that trips after three consecutive failures for a provider, temporarily redirecting traffic to a safe fallback or returning a 503 with a Retry-After header.
Store all prompt templates in a versioned, cache‑able repository; load them at startup and expose a /prompts/{name} endpoint that returns the current template and its hash to detect drift.
Deploy a feature‑flag (e.g., LaunchDarkly, Split.io) that toggles between the custom MCP implementation and the provider SDK directly, enabling controlled roll‑outs and instant rollback if issues surface.

Conclusion

Connecting Claude and other LLMs to a custom .NET MCP server is not a novelty; it’s a maturity step that turns a brittle prototype into a production‑grade platform. The trade‑offs are clear: you pay operational overhead for a unified protocol, but you gain tenant isolation, observability, and the flexibility to orchestrate multiple providers. By following the patterns above—centralised adapters, bounded queues, tenant‑aware caching, and a robust observability stack—you can build a system that scales, stays secure, and remains maintainable even as you add new LLMs, tool calls, or vector database integrations.

DEV Community