Quick Answer
Learn how to embed verification loops for AI agents in .NET, integrate with Semantic Kernel, and avoid the hidden pitfalls that break production systems.
Verification Loops for AI Agents in .NET: A Production‑Ready Guide
When you hand an LLM the keys to a customer‑facing API, the first thing you want to guarantee is that every response is safe to expose. In a .NET micro‑service that powers banking or insurance, a single hallucinated policy number can trigger regulatory alerts, a data‑leak audit, or a costly downstream API call. The engineering challenge isn’t how to call the model; it’s how to weave a deterministic, observable, and cost‑aware verification loop around every LLM turn.
Assumptions of Trustworthiness in Chat Endpoints
Most teams ship a “chat” endpoint that forwards user text to Azure OpenAI, then streams the raw token stream back to the client. The implicit assumption is that the model’s output is trustworthy. In practice, the model can:
- Invent policy IDs that never exist.
- Inject commands that bypass your own guardrails.
- Return a response that triggers an expensive downstream call, only to be rejected later.
These failures manifest as SLA violations, compliance fines, or runaway cloud spend. A verification loop turns the LLM response into a contract that the rest of your system can rely on.
Real‑World Example: A US Insurance Chatbot
In 2023 a mid‑size insurer deployed a .NET Core chatbot to answer claim status queries. Within 10 days, the bot was sending non‑existent policy numbers to the claims engine, raising a compliance alarm. The root cause: an unchecked hallucination from GPT‑4. The fix involved a hybrid verification loop that combined rule‑based checks against the policy database with a secondary LLM judge. After deployment the hallucination rate dropped from 12 % to 0.4 %, and the average added latency was only 180 ms per turn.
Trade‑Offs
| Strategy | Latency Impact | Cost per Turn | Safety Coverage | Operational Complexity |
|---|---|---|---|---|
| Regex + whitelist | ≈5 ms | $0.0001 | Low – only known patterns | Low |
| Secondary LLM (e.g., GPT‑4 as judge) | ≈200–400 ms | $0.002 | High – contextual reasoning | Medium – need retry/back‑off logic |
| Hybrid (rule → LLM fallback) | ≈100–150 ms | $0.001 | Medium‑High – 70‑80 % rule coverage, LLM covers edge cases | Medium – orchestrator needed |
The trade‑off is always between latency vs safety vs cost. For latency‑sensitive services (e.g., real‑time support), a rule‑first approach keeps the round‑trip under 200 ms. For compliance‑heavy domains (e.g., finance), the extra 200 ms of LLM judging is acceptable if it eliminates a single policy‑breach incident.
- When I’d choose rule‑first: The downstream system is tolerant of a 50 ms SLA hit and you have a well‑defined schema.
- When I’d choose full LLM judging: The cost of a single compliance incident (legal, regulatory, or brand damage) outweighs the extra 200 ms latency.
- When I’d choose hybrid: You want a safety net for edge cases while keeping most traffic fast.
Risk, Latency, and Cost Evaluation
- Assess risk tolerance. If a single hallucination could trigger a regulatory fine, lean toward a hybrid or full LLM verifier.
- Measure baseline latency. Run a pilot with the model’s token‑level latency and add the verification step; if you’re already near your SLA, you may need to move the verifier to a background queue.
- Profile cost. Use Azure’s token pricing to estimate the incremental cost of a secondary LLM call. If the cost outweighs the benefit of a single compliance incident, consider rule‑based only.
- Plan observability. Instrument every validator with OpenTelemetry traces and metrics; the verification pass rate should be a KPI.
- Scale accordingly. If the verifier becomes a bottleneck, shard the state store or move to a serverless function that can scale on demand.
- When to queue vs inline. If the primary LLM is already at capacity, queue the verifier so you don’t double‑stack the load on the same endpoint.
- When to expose health endpoints. A simple /health endpoint that reports the status of each validator helps operators spot outages before user impact.
When This Fails in Production
- Validator Outage. If the secondary LLM or the rule database is down, the orchestrator may fall back to a generic “I’m sorry” message, causing a drop in user satisfaction.
- Infinite Retry Loops. A naive implementation that retries forever on transient failures will tie up request threads and blow up the queue.
- Stale Context. Without a proper cache of conversation history, the LLM may hallucinate again because it lacks the latest user intent.
- Race Conditions in Shared State. Concurrent requests updating a shared retry counter can produce inconsistent decisions.
Common Mistakes Engineers Make
- Using a single
Task.Delayfor all retries, ignoring the nature of the failure. - Storing verification state in a local in‑memory dictionary, which breaks when scaling out.
- Assuming the LLM will always produce a valid rewrite; in reality the rewrite can still violate policy.
- Neglecting to expose verification metrics, making it hard to spot a rising failure rate.
- Treating the verifier as a black box; not logging the validator’s decision or the reason for failure.
- When I’d avoid storing state locally: In a multi‑instance deployment you’ll lose state across restarts.
- When I’d avoid a single retry strategy: Different failure modes (network vs token limit) need distinct back‑off curves.
Better Approach Based on Experience
Below is a production‑ready skeleton that addresses the pitfalls above. It uses Polly for robust retry/back‑off, Redis for distributed state, and a circuit breaker around the secondary LLM.
public class VerificationPipeline
{
private readonly Kernel _kernel;
private readonly IValidator _policyValidator;
private readonly IValidator _factValidator;
private readonly IDistributedCache _cache;
private readonly AsyncPolicy _retryPolicy;
private readonly AsyncPolicy _circuitBreaker;
public VerificationPipeline(Kernel kernel, IValidator policy, IValidator fact, IDistributedCache cache)
{
_kernel = kernel;
_policyValidator = policy;
_factValidator = fact;
_cache = cache;
_retryPolicy = Policy.Handle<TransientValidationException>()
.WaitAndRetryAsync(3, attempt => TimeSpan.FromMilliseconds(100 * Math.Pow(2, attempt)));
_circuitBreaker = Policy.Handle<Exception>().CircuitBreakerAsync(5, TimeSpan.FromSeconds(30));
}
public async Task<string> RunAsync(string conversationId, string userPrompt)
{
// Load or initialize context
var context = await LoadContextAsync(conversationId);
// 1️⃣ Primary LLM call
var raw = await _kernel.InvokeAsync("Chat", new() { ["input"] = userPrompt });
var response = raw.GetValue<string>();
// 2️⃣ Verification loop
await _retryPolicy.ExecuteAsync(async () =>
{
var policyResult = await _policyValidator.ValidateAsync(response);
var factResult = await _factValidator.ValidateAsync(response);
if (policyResult.IsSuccess && factResult.IsSuccess)
{
await StoreContextAsync(conversationId, context);
return;
}
if (policyResult.IsTransient || factResult.IsTransient)
throw new TransientValidationException();
// Non‑transient failure – rewrite
response = await _kernel.InvokeAsync("Rewrite", new() { ["badResponse"] = response });
});
// 3️⃣ Final fallback if verifier still fails
return response;
}
private async Task<VerificationContext> LoadContextAsync(string id)
{
var data = await _cache.GetStringAsync(id);
return string.IsNullOrEmpty(data) ? new VerificationContext() : JsonSerializer.Deserialize<VerificationContext>(data);
}
private async Task StoreContextAsync(string id, VerificationContext ctx)
{
var data = JsonSerializer.Serialize(ctx);
await _cache.SetStringAsync(id, data, new DistributedCacheEntryOptions { SlidingExpiration = TimeSpan.FromMinutes(10) });
}
}
Performance Considerations
- Token Budget. Every LLM call consumes tokens. The verifier should be lightweight; a rule‑based check is O(1) in token cost. The LLM judge should only be invoked on a small subset of responses.
-
Parallelism. Run the policy and fact validators concurrently with
Task.WhenAll. This halves the verifier’s latency if both are I/O bound. - Cold Starts. If you host the verifier in a serverless function, warm the kernel on startup by pre‑loading the model weights.
- Cache Reuse. Store the last successful rewrite in Redis and skip the LLM rewrite step if the same prompt appears within a 30‑second window.
- Back‑pressure. Use a bounded channel to queue verification tasks; if the queue exceeds 10k items, reject new requests with a 429 status.
- When to isolate the verifier. If the primary LLM is CPU‑bound, run the verifier in a separate process or container to avoid resource contention.
- When to prefetch embeddings. For fact‑based checks that rely on vector similarity, cache the embeddings of frequent policy numbers to shave milliseconds per query.
Scaling Notes
- State Store. A single Redis instance can become a bottleneck. Partition the cache by conversation ID hash or use Azure Cache for Redis Cluster.
- LLM Throughput. Azure OpenAI limits the number of concurrent calls per deployment. Use a token bucket algorithm to throttle primary and secondary calls separately.
-
Observability. Ship OpenTelemetry traces to Azure Monitor. The
verification_latency_msmetric should be split by validator type; this lets you see which step is the true bottleneck. - Cost Control. Implement a quota per user that caps the number of secondary LLM calls per day. When the quota is hit, fall back to a pre‑validated static answer.
- Resilience. Wrap the entire pipeline in a circuit breaker. If the secondary LLM fails for >5 consecutive attempts, the system should stop invoking it until a health probe reports success.
- When to move to serverless. If traffic is bursty and you want pay‑as‑you‑go, deploy the verifier to Azure Functions Premium to scale automatically.
- When to shard. For millions of concurrent conversations, shard the Redis cache by key prefix to distribute load and reduce contention.
What to Ship
- Wrap every LLM call in a reusable
VerificationLoopcomponent that retries up tomaxAttempts(e.g., 3) until the response satisfies a safety/accuracy policy; expose the policy as a delegate so it can be swapped per endpoint. - Configure a fallback path that routes the user to a human or a simpler rule‑based agent if the loop fails after
maxAttempts; expose the fallback as an optionalOnFailurecallback. - Instrument the loop with structured logs that include
iteration,promptId,responseScore, andlatencyMs; send these to your observability platform so you can track iteration distribution and cost per request. - Set up a real‑time alert that fires when the average number of loop iterations per request exceeds a threshold (e.g., 2.5) or when total latency per request crosses the SLA (e.g., 800 ms).
- Package the loop and fallback logic as a .NET middleware that can be injected into any ASP.NET Core or SignalR chat pipeline, so you don’t duplicate code across services.
Conclusion
Verification loops aren’t a luxury; they’re a contractual layer that turns an untrustworthy LLM into a production‑grade service. The right design balances latency, cost, and safety, uses distributed state to survive scale, and instruments every decision so you can spot regressions before they hit customers.
If you skip verification, you’re effectively outsourcing compliance to the LLM, which is not acceptable for regulated workloads.
Related Articles
- NVIDIA NOOA vs Semantic Kernel for .NET developers: Deep Dive into Agentic AI Architecture
- Kafka producer configuration for .NET real-time streams: Tuning for Low Latency & High Throughput
- Avoiding Multi-Agent Orchestration Architecture Mistakes in Cloud‑Native .NET Systems
- Advanced Shell Scripting for .NET Build Automation: Production‑Ready Patterns
- Claude Agent SDK vs Semantic Kernel: Debugging Function‑Call Failures
Top comments (0)