DEV Community: Dishant Sethi

Why Your RAG Pipeline Is Failing in Production (And How to Fix It)

Dishant Sethi — Wed, 27 May 2026 16:19:11 +0000

Originally published on prodinit.com

Key Takeaways

80% of RAG failures trace back to the ingestion layer, not the LLM — fix chunking and indexing before tuning your prompts

Chunk size alone can swing retrieval precision by 20–40%; there is no universal right answer, and the correct value depends on your document type and query pattern

Adding a cross-encoder reranker on top of vector search typically lifts answer correctness by 15–25% with minimal latency cost

Stale indexes are invisible in standard monitoring: a document updated 3 months ago may still be answering queries from its old content

Teams without an eval loop discover regressions 4–8× slower than teams with automated retrieval quality checks running on every deployment

A RAG pipeline looks straightforward on paper: retrieve relevant chunks, stuff them into a prompt, get an answer. Teams wire it up in a weekend, the demo works, and they ship it. Then, weeks later, users start complaining that the system returns outdated information, misses obvious answers, or confidently cites the wrong document.

RAG pipeline debugging starts at the retrieval layer, not the LLM. The five failure modes that break production RAG systems — bad chunking, missing reranking, stale indexes, no hybrid retrieval, and no eval loop — are all fixable at the data and infrastructure layer. None require changing your model or rewriting your application.

Why RAG Fails Silently in Production

The LLM itself is almost always fine. The retrieval layer is what's broken — and most observability tooling points at the model, not the retriever. You can spend days tweaking system prompts and temperature settings while the root cause sits in how you chunked your documents three months ago. Production RAG failure leaves no stack trace.

There is no exception, no 500 error, no latency spike. The system continues to return answers. They are just wrong, or incomplete, or stale. Without an explicit eval loop tied to retrieval quality, you will not know until a user tells you.

This guide covers the five failure modes Prodinit encounters most often when auditing RAG systems in production, with diagnosis steps and fixes for each.

Failure Mode 1: Bad Chunking Strategy

Fixed-size character splitting destroys retrieval quality for anything beyond plain prose. A 512-token chunk of a legal contract may split a clause mid-sentence; a 512-token chunk of code may span four unrelated functions. Neither produces embeddings specific enough to surface the right document for a precise query.

Why it breaks

Chunking is the most consequential decision in a RAG pipeline and the one teams spend the least time on. The default in most frameworks is a fixed-size character or token split with a small overlap. This works in demos. In production, it destroys retrieval quality for anything that isn't plain prose.

The problem with fixed-size chunking:

A 512-token chunk of a legal contract may split a clause mid-sentence, leaving neither chunk with enough context to be retrieved correctly
A 512-token chunk of code may contain four unrelated functions, causing the entire chunk to match queries loosely but none of them precisely
Tables, structured data, and numbered lists lose their semantics when split by character count

When your chunks are semantically incoherent, your embeddings are noisy. Noisy embeddings produce low-confidence nearest-neighbor results. The retriever returns tangentially related chunks, the LLM hallucinates to fill the gap, and the answer looks plausible but wrong.

Diagnosis

Check what your chunks actually look like:

import json
def audit_chunks(chunks: list[str], sample_size: int = 20) -> dict:
    import random
    sample = random.sample(chunks, min(sample_size, len(chunks)))
    stats = {
        "avg_tokens": sum(len(c.split()) for c in chunks) / len(chunks),
        "min_tokens": min(len(c.split()) for c in chunks),
        "max_tokens": max(len(c.split()) for c in chunks),
        "truncated_sentences": sum(
            1 for c in sample
            if not c.strip().endswith((".", "?", "!", "```

", "}"))
        ),
        "sample": sample[:3],
    }
    return stats

Red flags: truncated_sentences above 30%, average tokens below 100 or above 600, or chunks that end mid-code-block.

Fix

Switch to semantic chunking. For prose documents, split on sentence boundaries and merge until a semantic similarity threshold is crossed. For structured content, use document-aware splitters that respect headings, tables, and code blocks.


python
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Document-aware splitter that respects structure
splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", ". ", "! ", "? ", " ", ""],
    chunk_size=600,          # tokens, not characters
    chunk_overlap=60,        # ~10% overlap for context continuity
    length_function=len,
    is_separator_regex=False,
)

# For code: use language-aware splitters
from langchain.text_splitter import Language, RecursiveCharacterTextSplitter

code_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=800,
    chunk_overlap=80,
)

There is no universal correct chunk size. Run retrieval precision benchmarks at 256, 512, and 1024 tokens against a sample of real queries. Pick the size that maximises the percentage of queries where the correct answer appears in the top-3 retrieved chunks.

Failure Mode 2: Missing Reranking

Vector similarity search retrieves the right candidates but ranks them poorly. The chunk with the highest cosine similarity is not always the most useful chunk for the specific query — it is the closest in embedding space, not the most relevant to the question. Without a cross-encoder reranker, you are systematically passing the wrong context to your LLM.

Why it breaks

Vector similarity search is excellent at candidate retrieval. It is poor at ranking. Cosine similarity between two high-dimensional embeddings captures semantic proximity, not answer relevance for a specific query. The top result by cosine distance is not always the most useful chunk for the question at hand.

Teams that skip reranking are essentially treating their retrieval problem as solved after the first-stage ANN search. In practice, the chunk that best answers the query is often ranked 3rd or 5th by embedding similarity — close enough to retrieve, not close enough to surface first.

If your system passes the top-1 or top-2 chunks to the LLM without reranking and truncates the rest, you are systematically dropping the best answers.

Diagnosis

Run a relevance audit on your retrieval results:


python
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def audit_retrieval_rank(query: str, retrieved_chunks: list[str], 
                          ground_truth_chunk: str) -> dict:
    scores = reranker.predict(
        [(query, chunk) for chunk in retrieved_chunks]
    )
    reranked = sorted(
        enumerate(retrieved_chunks), 
        key=lambda x: scores[x[0]], 
        reverse=True
    )

    vector_rank = retrieved_chunks.index(ground_truth_chunk) + 1
    reranked_rank = next(
        i + 1 for i, (orig_idx, _) in enumerate(reranked)
        if retrieved_chunks[orig_idx] == ground_truth_chunk
    )

    return {
        "query": query,
        "vector_rank": vector_rank,
        "reranked_rank": reranked_rank,
        "improved": reranked_rank < vector_rank,
    }

If reranked rank is better than vector rank on more than 30% of your test queries, you have a reranking gap that is actively hurting answer quality.

Fix

Add a cross-encoder reranker as a second-pass filter. Retrieve k=20 candidates from your vector store, rerank them, and pass the top-3 to your LLM. The cross-encoder sees the full query and each chunk together, which lets it score relevance directly rather than proximity in embedding space.


python
from sentence_transformers import CrossEncoder
from typing import List

class RerankedRetriever:
    def __init__(self, vector_store, reranker_model: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"):
        self.vector_store = vector_store
        self.reranker = CrossEncoder(reranker_model)

    def retrieve(self, query: str, top_k: int = 3, candidate_k: int = 20) -> List[str]:
        # First-stage: broad vector retrieval
        candidates = self.vector_store.similarity_search(query, k=candidate_k)

        # Second-stage: cross-encoder reranking
        pairs = [(query, doc.page_content) for doc in candidates]
        scores = self.reranker.predict(pairs)

        ranked = sorted(
            zip(candidates, scores),
            key=lambda x: x[1],
            reverse=True
        )
        return [doc.page_content for doc, _ in ranked[:top_k]]

Cross-encoder reranking adds 50–200ms of latency for a 20-candidate set. For most production RAG workloads, that is an acceptable trade for a 15–25% improvement in answer correctness.

Failure Mode 3: Stale Index

Your embedding index is a snapshot of your documents at indexing time. When a policy is updated, a product spec is revised, or a pricing page changes, the index does not update automatically — queries continue retrieving content from weeks or months ago, with no error signal to indicate the problem.

Why it breaks

Stale index is insidious because it is invisible in standard observability. Query latency is normal. Embedding lookups return results. The system appears healthy. Users are just silently receiving outdated information.

The problem compounds with time. A document indexed 6 months ago and updated 3 times since is a liability, not an asset.

Diagnosis

Implement index freshness tracking:


python
import hashlib
from datetime import datetime
from dataclasses import dataclass

@dataclass
class IndexedDocument:
    doc_id: str
    content_hash: str
    indexed_at: datetime
    source_updated_at: datetime

def audit_index_freshness(indexed_docs: list[IndexedDocument], 
                           max_age_days: int = 30) -> dict:
    now = datetime.utcnow()
    stale = []

    for doc in indexed_docs:
        age = (now - doc.indexed_at).days
        if age > max_age_days:
            stale.append({"id": doc.doc_id, "age_days": age})

        if doc.source_updated_at > doc.indexed_at:
            stale.append({
                "id": doc.doc_id, 
                "reason": "source_updated_after_index",
                "gap_hours": (doc.source_updated_at - doc.indexed_at).seconds // 3600,
            })

    return {
        "total_documents": len(indexed_docs),
        "stale_count": len(stale),
        "stale_pct": round(len(stale) / len(indexed_docs) * 100, 1),
        "stale_docs": stale[:10],
    }

Fix

Implement incremental re-indexing on document change, not on a fixed schedule. Track content hashes. When a source document's hash changes, queue it for re-embedding immediately.


python
import hashlib
from datetime import datetime

class IncrementalIndexer:
    def __init__(self, vector_store, embedder):
        self.vector_store = vector_store
        self.embedder = embedder
        self.index_registry: dict[str, str] = {}  # doc_id -> content_hash

    def _content_hash(self, content: str) -> str:
        return hashlib.sha256(content.encode()).hexdigest()

    def upsert_document(self, doc_id: str, content: str, metadata: dict):
        new_hash = self._content_hash(content)

        if self.index_registry.get(doc_id) == new_hash:
            return  # Content unchanged, skip re-indexing

        self.vector_store.delete(filter={"doc_id": doc_id})

        chunks = self.chunk(content)
        embeddings = self.embedder.embed_documents(chunks)

        self.vector_store.add_embeddings(
            texts=chunks,
            embeddings=embeddings,
            metadatas=[{**metadata, "doc_id": doc_id, "indexed_at": datetime.utcnow().isoformat()}
                       for _ in chunks],
        )

        self.index_registry[doc_id] = new_hash

Wire this to your content management system's webhook or change-data-capture stream. Every document update should trigger an upsert within minutes, not the next scheduled batch run.

Failure Mode 4: No Hybrid Retrieval (BM25 + Vector)

Pure vector search fails on exact-match queries. When a user searches for a specific error code, API endpoint, or product identifier, vector similarity often surfaces semantically related content that never contains the exact string. BM25 handles rare-term and exact-match queries precisely — hybrid retrieval combines both and consistently outperforms either approach alone.

Why it breaks

Pure vector search excels at semantic similarity. It is poor at exact matching. When a user queries for a specific product code, a person's name, an API endpoint, or an error message, vector search often surfaces semantically related but lexically different results. The chunk containing the exact string ERR_QUOTA_EXCEEDED may score lower than a chunk about "error handling" that never mentions the specific code.

BM25 (the algorithm behind classic keyword search) handles exact and rare-term matching extremely well. It rewards documents that contain the query terms, with inverse document frequency weighting meaning that rare, specific terms get boosted. What BM25 misses is paraphrase, synonym, and conceptual matching — exactly what vector search handles.

Teams that use only vector search leave a meaningful precision gap for queries with specific identifiers. Teams that use only BM25 miss semantic intent. Hybrid retrieval combines both, and on standard retrieval benchmarks it consistently outperforms either approach alone.

Diagnosis

Run a query set that mixes semantic queries ("how does the refund policy work?") and exact-match queries ("what is the timeout value for API_GATEWAY_CONNECT?"). Compare top-3 precision for vector-only versus BM25-only versus hybrid across both query types. If vector-only precision on exact-match queries is more than 15 percentage points lower than on semantic queries, you have a pure-vector blind spot.

Fix

Implement reciprocal rank fusion (RRF) to merge vector and BM25 rankings:


python
from rank_bm25 import BM25Okapi
import numpy as np
from typing import List

class HybridRetriever:
    def __init__(self, vector_store, documents: List[str], 
                 rrf_k: int = 60, alpha: float = 0.5):
        self.vector_store = vector_store
        self.documents = documents
        self.alpha = alpha         # 0 = BM25 only, 1 = vector only
        self.rrf_k = rrf_k

        tokenized = [doc.lower().split() for doc in documents]
        self.bm25 = BM25Okapi(tokenized)

    def _rrf_score(self, rank: int) -> float:
        return 1.0 / (self.rrf_k + rank)

    def retrieve(self, query: str, top_k: int = 5) -> List[str]:
        vector_results = self.vector_store.similarity_search(query, k=top_k * 4)

        tokenized_query = query.lower().split()
        bm25_scores = self.bm25.get_scores(tokenized_query)
        bm25_ranked = np.argsort(bm25_scores)[::-1][:top_k * 4]

        rrf_scores: dict[str, float] = {}

        for rank, doc in enumerate(vector_results):
            doc_id = doc.metadata["id"]
            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + self.alpha * self._rrf_score(rank)

        for rank, idx in enumerate(bm25_ranked):
            doc_id = f"doc_{idx}"
            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + (1 - self.alpha) * self._rrf_score(rank)

        sorted_ids = sorted(rrf_scores, key=rrf_scores.get, reverse=True)
        return sorted_ids[:top_k]

Start with alpha=0.5 (equal weight) and tune based on your query distribution. If your users ask mostly exact-product or identifier queries, shift toward alpha=0.3 to weight BM25 more heavily.

Failure Mode 5: No Eval Loop

Without an automated eval loop, every regression in your RAG pipeline is invisible until a user complaint surfaces it. Teams without retrieval quality checks running on every deployment discover degradation 4–8× slower than teams that do — and by then, the root cause is typically tangled across multiple changes and hard to isolate.

Why it breaks

You cannot improve what you do not measure. RAG systems degrade over time as documents are updated, query patterns shift, and underlying model versions change. Without an automated eval loop tied to retrieval quality metrics, every one of these changes is invisible until a user complaint surfaces it.

The eval loop is not optional. It is the mechanism that keeps your RAG pipeline honest over its operational lifetime.

Diagnosis

Check whether your deployment pipeline currently runs any of these:

Retrieval precision@k (what fraction of ground-truth relevant chunks appear in the top-k retrieved?)
Answer faithfulness (does the generated answer stay within the retrieved context, or does it hallucinate beyond it?)
Answer relevance (does the generated answer actually address the query?)
Context recall (does the retrieved set contain all the information needed to answer correctly?)

If none of these are tracked per deployment, you are operating blind.

Fix

Build a retrieval eval suite using a golden query set and run it in CI on every deployment:


python
from dataclasses import dataclass
from typing import List, Optional

@dataclass
class EvalCase:
    query: str
    expected_doc_ids: List[str]
    expected_answer_contains: Optional[str] = None

def precision_at_k(retrieved_ids: List[str], relevant_ids: List[str], k: int) -> float:
    top_k = retrieved_ids[:k]
    hits = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return hits / k

def run_retrieval_eval(retriever, eval_cases: List[EvalCase], k: int = 3) -> dict:
    results = []

    for case in eval_cases:
        retrieved = retriever.retrieve(case.query, top_k=k)
        retrieved_ids = [r["id"] for r in retrieved]

        precision = precision_at_k(retrieved_ids, case.expected_doc_ids, k)
        recall = sum(
            1 for doc_id in case.expected_doc_ids if doc_id in retrieved_ids
        ) / len(case.expected_doc_ids)

        results.append({
            "query": case.query,
            f"precision@{k}": precision,
            "recall": recall,
        })

    avg_precision = sum(r[f"precision@{k}"] for r in results) / len(results)
    avg_recall = sum(r["recall"] for r in results) / len(results)

    return {
        f"avg_precision@{k}": round(avg_precision, 3),
        "avg_recall": round(avg_recall, 3),
        "per_query": results,
    }

def ci_gate(current_metrics: dict, baseline_metrics: dict, 
             relative_threshold: float = 0.05) -> bool:
    baseline_p = baseline_metrics["avg_precision@3"]
    current_p = current_metrics["avg_precision@3"]
    regression = (baseline_p - current_p) / baseline_p

    if regression > relative_threshold:
        print(f"FAIL: precision@3 regressed {regression:.1%} (baseline={baseline_p:.3f}, current={current_p:.3f})")
        return False
    return True

Run this eval suite against a golden set of 50–200 query/relevant-document pairs on every deploy. Gate the deployment if precision@3 drops more than 5% relative to the last passing run.

RAG Pipeline Debugging Checklist

Run this before spending time on prompt engineering or model tuning. These five failure modes are sequential — chunking problems corrupt every downstream step, so work top to bottom. If any item below fails, fix it before moving to the next row.

Check	Tool / Signal	Pass Condition
Chunk quality	Run `audit_chunks()`	`truncated_sentences` < 30%, avg tokens 200–600
Chunk strategy	Manual inspection	Chunks are semantically coherent units
Reranker present	Code review	Cross-encoder reranker on first-stage candidates
Reranker improves rank	`audit_retrieval_rank()`	Ground-truth rank improves in > 30% of queries
Index freshness	Hash comparison	No document indexed > 30 days without change check
CDC / webhook	Infrastructure review	Document updates trigger re-index within minutes
Hybrid retrieval	Code review	BM25 + vector fusion implemented
Hybrid alpha tuned	Precision comparison	Hybrid P@3 ≥ max(vector-only, BM25-only) P@3
Eval suite exists	CI pipeline	Retrieval eval runs on every deployment
Regression gate	CI config	Deploy blocked if precision drops > 5% relative

Building Production Voice AI Agents: Latency, Architecture, and What Nobody Tells You

Dishant Sethi — Wed, 27 May 2026 16:10:44 +0000

Originally published on prodinit.com

Key Takeaways

Sub-300ms end-to-end latency is the human-conversation threshold for voice AI.

The latency budget breaks into four layers: STT (80–120ms), LLM first-token (150–250ms), TTS first-chunk (60–100ms), and network transport (20–60ms). Missing target in any one layer pushes the total over 500ms.

WebRTC with ICE Trickle is the correct transport for browser and mobile clients. SIP is the right choice for PSTN integration and legacy telephony.

LiveKit SFU reduces media server complexity by forwarding encoded streams rather than decoding and re-mixing them, and its hosted tier removes the need to operate a media server fleet entirely.

Why Voice AI Fails in Production

Voice AI demos look deceptively easy. A GPT-4o API call, a TTS response, a microphone input — connected together in 200 lines of Python, the thing works. Then you put it in front of real users and it fails.

The failure is almost never the model. It is the architecture.

In production at 2000+ calls per day — the scale Prodinit operates for a healthcare scheduling platform — three classes of failure dominate: latency spikes that destroy conversational flow, audio glitches from unmanaged WebRTC sessions, and compliance gaps where customer PII surfaces in LLM provider logs. None of these appear in a notebook demo. All of them have architecture solutions.

This guide walks through the complete production stack: what latency target you are actually trying to hit, how the budget breaks across each layer, the transport architecture that achieves it, and the security and observability instrumentation that keeps it running without surprises.

What Latency Is Acceptable for Voice AI?

Sub-300ms end-to-end latency is the human-conversation threshold. Conversational linguistics research places the average human response gap at 200ms; gaps up to 500ms are within the natural range. Beyond 500ms, listeners register the pause. Beyond 1,500ms, they start to speak again — or hang up.

The practical production target is under 800ms at p95, with a p50 below 400ms. This is not a soft target — these numbers correlate directly with call completion rates and CSAT scores.

End-to-end latency in a voice AI agent is the sum of five contributors:

Audio capture and VAD (voice activity detection) — 10–30ms
STT transcription — 80–120ms with streaming
LLM first-token latency — 150–250ms with low-latency models
TTS first-audio-chunk — 60–100ms with streaming
Network transport and jitter buffer — 20–60ms

Total target: 320–560ms. That is achievable. The mistakes that push it over 1,000ms are predictable and avoidable.

Latency Budget by Layer

Voice Activity Detection (10–30ms)

VAD decides when the user has stopped speaking and the pipeline should fire. A misconfigured VAD is the single easiest way to add 500ms of latency without touching any model. Most implementations default to a trailing silence window of 500–800ms — that pause sits entirely in the user experience before a single API call fires.

In production, configure VAD with:

Silence threshold: 300ms for call center contexts, 200ms for high-tempo applications
Endpointing: fire on silence, not on a fixed timer
Echo cancellation: required whenever the agent speaks; browser getUserMedia handles this with echoCancellation: true

Deepgram's streaming STT includes built-in VAD endpointing via endpointing=300 — use this rather than a separate VAD layer, as it eliminates an additional round-trip.

STT: Streaming Transcription (80–120ms)

Batch transcription — send audio, wait for full transcript — adds 600–1,200ms before your LLM call even starts. This alone makes sub-300ms unreachable. The solution is streaming STT with interim results.

Deepgram Nova-2 delivers streaming transcription with a first-word latency around 80ms over WebSocket. You do not wait for the complete transcript; you begin processing on is_final: true utterances:

User audio → WebSocket → Deepgram Nova-2 (streaming)
                              ↓
                    interim results (ignored)
                              ↓
                    is_final: true → LLM pipeline fires

Critical configuration: punctuate=true, smart_format=true, and endpointing=300. Without endpointing set, Deepgram uses server-side silence detection that defaults longer than your VAD window.

LLM Reasoning (150–250ms)

LLM first-token latency is the hardest constraint to optimize. GPT-4 in streaming mode cannot reliably hit sub-200ms first-token in typical network conditions. The model choices that achieve 150–250ms in practice:

GPT-4o-mini — ~150ms first-token median; suitable for most voice turn completions
GPT-4o — ~200–300ms first-token; higher quality for complex reasoning turns
Claude Haiku 4.5 — ~120–180ms first-token; strong instruction-following, well-suited for structured voice turns
Groq-hosted Llama — sub-100ms first-token via custom hardware; lower model quality ceiling

Stream the response. Pass tokens to TTS as they arrive — do not buffer the full LLM output before starting TTS synthesis. The overlap between LLM generation and TTS synthesis recovers 100–200ms of total latency.

Prompt engineering for voice: system prompts should be shorter than for text chatbots. Strip all markdown formatting instructions — the output goes to TTS and formatted text degrades audio. Keep total context under 2,000 tokens where possible; token count has a near-linear relationship with first-token latency.

TTS: Streaming Synthesis (60–100ms)

ElevenLabs streaming delivers first-audio-chunk in 60–100ms on their Flash tier versus 200–400ms on standard. The difference is significant enough that choosing the wrong tier consumes your entire latency budget on TTS alone.

Use streaming TTS: do not wait for the complete audio file before playback. The client should begin playing as soon as the first audio chunk arrives. For browser clients, the Web Audio API handles chunked playback natively; for telephony, use RTP packetization.

The TTS configuration that matters for latency:

Model: eleven_flash_v2_5 for minimum latency
Streaming: set stream=true
Output format: pcm_16000 for telephony, mp3_44100_128 for browser
Streaming latency optimization: optimize_streaming_latency=4 (aggressive mode)

Network Transport (20–60ms)

With a well-configured WebRTC connection, transport adds 20–40ms round-trip. With a WebSocket-only approach through a distant cloud region, transport alone can add 200ms in the tail. This is where the transport choice has the most impact.

Full Stack Architecture

The production architecture for a sub-300ms voice AI agent:

The agent worker sits between the media plane and the model APIs. It receives raw audio frames from LiveKit, streams them to Deepgram, fires the LLM on final utterances, and pushes TTS audio frames back into the LiveKit room. The client never calls model APIs directly — this is essential for PII control and rate-limit management.

ICE Trickle and the LiveKit SFU Pattern

Why ICE Trickle Matters

WebRTC connection establishment uses Interactive Connectivity Establishment (ICE) to find a network path between peers. In the naive implementation — wait for all ICE candidates before signaling — setup latency adds 500–2,000ms to every call start. This is invisible in demos and very visible in production.

ICE Trickle solves this: candidates are sent to the remote peer as they are gathered, and connectivity checks begin immediately. Call setup time drops to 100–400ms in most network conditions.

LiveKit implements ICE Trickle automatically. What you need to deploy:

STUN servers — used for reflexive candidate discovery; stun.l.google.com:19302 works for most cases; deploy your own for HIPAA environments to keep traffic off third-party infrastructure
TURN servers — required for clients behind symmetric NAT, common in enterprise networks; LiveKit's hosted tier includes TURN, or deploy coturn yourself
Signaling — LiveKit's built-in signaling server handles offer/answer exchange; no separate WebSocket signaling server required

LiveKit SFU Pattern

A Selective Forwarding Unit receives encoded media streams and forwards them to participants without decoding and re-encoding. For voice AI, this matters because:

The agent worker receives RTP packets from the SFU rather than raw WebRTC — simpler to handle in server-side Python or Node.js code
Multiple agents or observers can subscribe to the same audio stream without additional encoding cost
The SFU handles DTLS/SRTP complexity; the agent sees plain RTP internally

The LiveKit room model maps cleanly to a voice call session:

from livekit import agents, rtc
import asyncio

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    async for event in ctx.room.on("track_subscribed"):
        if event.track.kind == rtc.TrackKind.KIND_AUDIO:
            audio_stream = rtc.AudioStream(event.track)
            asyncio.create_task(process_audio(audio_stream, ctx.room))

async def process_audio(stream: rtc.AudioStream, room: rtc.Room):
    async for frame in stream:
        await pipeline.push_frame(frame)

LiveKit's agent framework handles room lifecycle, track subscription, and RTP framing. Application code focuses on pipeline logic.

WebRTC vs SIP: Which Transport to Use

This is the question that trips up most teams evaluating voice AI infrastructure. They are not competing choices — they solve different integration problems.

Use WebRTC when you control the client — a web app, mobile app, or embedded SDK. It gives you wideband Opus audio (meaningfully better STT accuracy), lower setup latency, and direct control over the media path.

Use SIP when the caller is on a real phone number — inbound calls to a support line, outbound dialer campaigns, or integration with an existing contact center (Genesys, Five9, Twilio PSTN). Twilio's Media Streams provides a WebSocket bridge from PSTN to your agent worker, which avoids running a full SIP stack yourself.

The G.711 codec limitation of PSTN calls has an underappreciated consequence: STT accuracy on 8kHz narrowband audio is meaningfully lower than on 16kHz+ wideband. For healthcare or fintech agents where transcription accuracy directly affects outcomes, browser/mobile WebRTC with Opus gives a material accuracy advantage over telephone calls.

A production voice AI WebRTC architecture typically uses both: WebRTC for app callers and a SIP trunk or Twilio Media Streams for inbound phone calls, with the same agent worker behind both paths.

Observability: What to Instrument

Voice AI pipelines fail silently. A WebRTC ICE failure looks like a dropped call. A Deepgram WebSocket disconnect looks like the agent not hearing the user. A TTS timeout manifests as silence on the line. Without structured observability, every incident is a multi-hour debugging session across three services.

Instrument the following at minimum:

Per-call latency histogram — record wall-clock time from VAD endpoint event to first TTS audio chunk, broken down by component: stt_latency_ms, llm_first_token_ms, tts_first_chunk_ms. Alert on p95 > 800ms for any single component.

Per-call transcription confidence — Deepgram returns a confidence score per utterance. Log confidence distributions; a degradation in median confidence correlates with audio quality issues, codec mismatches, or background noise problems before callers start complaining.

WebRTC ICE connection state — log ICE state transitions (checking → connected → disconnected → failed). Track failed rates by client region. Elevated failure rates in a specific geography usually indicate TURN server coverage gaps.

STT WebSocket reconnections — Deepgram WebSocket connections drop under load or network events. Count reconnections per call. A call with 3+ reconnections will have visible transcription gaps; flag and review these separately.

LLM error rates — log 4xx/5xx rates from your LLM provider independently from total call failure. A 429 spike during peak hours needs a different response (add capacity, queue calls) than a 500 (inspect payloads, contact provider).

Use structured logging with a call_id field on every log event. Voice AI incidents always span Deepgram, your agent worker, and your SFU. Without a consistent call_id, joining those log lines across services is impossible.

How to Deploy on Air-Gapped AWS EKS for Regulated Financial Services

Dishant Sethi — Wed, 27 May 2026 16:05:29 +0000

Originally published on prodinit.com

Financial services data breaches cost an average of $6.08 million per incident — 22% above the global average across all industries (IBM Cost of a Data Breach 2024, 2024). For regulated institutions, the answer isn't just better firewalls. It's network architecture that eliminates the attack surface at the infrastructure level.

Air-gapped AWS EKS deployments — where private subnets have zero internet egress and all traffic routes through VPC endpoints — are becoming the standard for regulated financial services workloads. This guide walks through the full architecture, from VPC design to CI/CD pipeline, based on a real deployment we executed for a fintech platform at Prodinit.

Key Takeaways

Air-gapped EKS requires VPC interface and gateway endpoints for every AWS service your workloads touch — there's no fallback to the public internet

Your CI/CD pipeline must be redesigned from scratch: images push to private ECR via VPC endpoint, and deployment runs through Systems Manager or a bastion inside the VPC

Kubernetes External Secrets Operator + AWS Secrets Manager is the cleanest pattern for pod-level secret injection without exposing credentials in manifests

Every data store (RDS, DynamoDB, ElastiCache) must live in private subnets, accessed via security group rules — no public endpoints

What Does "Air-Gapped" Mean in AWS Context?

An air-gapped VPC means your private subnets have no route to the internet — no NAT Gateway in private subnets, no internet gateway attachment to private route tables. All communication between your workloads and AWS services (S3, ECR, Secrets Manager, CloudWatch, Bedrock) must route through VPC endpoints.

AWS supports two endpoint types (AWS VPC Endpoints documentation):

Gateway endpoints — for S3 and DynamoDB only; free, added as route table entries
Interface endpoints — for all other AWS services via AWS PrivateLink; billed per hour per AZ

For a typical EKS deployment, you'll need interface endpoints for: ECR API, ECR Docker, S3 (or gateway), Secrets Manager, Systems Manager, CloudWatch Logs, STS, ELB, Bedrock (if using AI services), and Transcribe (if using speech-to-text).

Why This Matters for Regulated Workloads

Network isolation is a hard requirement under FFIEC guidelines and SEC cybersecurity rules for financial institutions. An air-gapped VPC enforces this at the infrastructure layer — there's no misconfigured security group that can accidentally allow outbound internet access, because the route simply doesn't exist.

On the Client deployment, we discovered mid-project that several Helm charts we'd planned to use for in-cluster controllers (ALB Ingress Controller, cluster-autoscaler) attempt to pull their own images from public registries at install time. We had to mirror every controller image into private ECR before the cluster could bootstrap. This is a class of problem that only surfaces when you actually try to deploy — not in planning.

How Do You Design a Zero-Egress VPC for AWS EKS?

Regulated financial services environments under FFIEC and SEC cybersecurity guidelines require network isolation enforced at the infrastructure level — not as a policy overlay but as a structural property of the network. A multi-AZ VPC with private subnets carrying no internet route, combined with VPC interface endpoints for every AWS service, eliminates the outbound internet path entirely rather than restricting it.

Start with a multi-AZ VPC with distinct public and private subnet tiers.

Subnet Design

VPC: 10.0.0.0/16
├── Public Subnets (one per AZ)
│   ├── 10.0.1.0/24 (us-east-1a)
│   ├── 10.0.2.0/24 (us-east-1b)
│   └── NAT Gateways (for public-subnet resources only)
└── Private Subnets (one per AZ)
    ├── 10.0.10.0/24 (us-east-1a)
    └── 10.0.11.0/24 (us-east-1b)
        — No internet route in route table
        — VPC endpoints attached

Private subnet route tables should contain exactly two entries: the local VPC CIDR, and gateway endpoint routes for S3/DynamoDB. Nothing else.

VPC Endpoints to Provision

Create interface endpoints in your private subnets for each required AWS service:

# ECR — required for image pulls
aws ec2 create-vpc-endpoint --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.ecr.api \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-xxx subnet-yyy \
  --security-group-ids sg-endpoints

aws ec2 create-vpc-endpoint --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.ecr.dkr \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-xxx subnet-yyy \
  --security-group-ids sg-endpoints

Create a dedicated security group for endpoints that allows HTTPS (443) inbound from your VPC CIDR. Don't open it wider than needed.

IAM Roles

Set up three distinct IAM role categories before touching EKS:

Developer access role — scoped to read operations, no production deploy permissions
CI/CD role — ECR push, EKS kubectl apply, Secrets Manager read
Node instance role — ECR pull, CloudWatch logging, S3 read for application buckets

Use IAM Roles for Service Accounts (IRSA) for in-cluster components. This ties Kubernetes service accounts to IAM roles without storing credentials anywhere.

How Do You Run an EKS Cluster with No Public Internet Path?

As of 2024, 80% of organizations run Kubernetes in production (CNCF Annual Survey 2024). The hard part isn't Kubernetes — it's running it without any public internet path.

Cluster Creation

When creating the EKS cluster, set the API server endpoint access to private only:

eksctl create cluster \
  --name fintech-prod \
  --region us-east-1 \
  --vpc-private-subnets subnet-xxx,subnet-yyy \
  --node-private-networking \
  --endpoint-private-access true \
  --endpoint-public-access false

With --endpoint-public-access false, kubectl only works from inside the VPC. This is intentional. Access the cluster via a bastion host or AWS Systems Manager Session Manager.

Node Groups

Place node groups in private subnets with autoscaling enabled:

# nodegroup.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: fintech-prod
  region: us-east-1
managedNodeGroups:
  - name: workers
    instanceTypes: ["m6i.xlarge", "m6i.2xlarge"]
    minSize: 2
    maxSize: 10
    desiredCapacity: 3
    privateNetworking: true
    iam:
      withAddonPolicies:
        autoScaler: true
        cloudWatch: true

Bootstrapping In-Cluster Controllers

This is where most air-gapped deployments stall. cluster-autoscaler and the AWS Load Balancer Controller both try to pull images from public registries during helm install. You must mirror them to ECR first:

# Pull, retag, and push to private ECR
docker pull registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0
docker tag registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 \
  123456789.dkr.ecr.us-east-1.amazonaws.com/cluster-autoscaler:v1.29.0
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/cluster-autoscaler:v1.29.0

Override the image in Helm values to point to your private ECR before installing.

How Do You Build a CI/CD Pipeline Without Internet Access?

Standard GitHub Actions hosted runners and most CI/CD platforms assume outbound internet access for image pulls and API calls — assumptions that silently break the moment you remove internet egress. The working architecture requires a self-hosted runner deployed inside the VPC, all image pushes to private ECR via VPC endpoint, and all cluster deployments executed through AWS Systems Manager Session Manager with zero inbound ports.

Standard CI/CD tooling assumes internet access. GitHub Actions' hosted runners can't reach a private EKS API endpoint. CodePipeline agents can't pull from Docker Hub. You need a fundamentally different pipeline architecture.

The pattern that works:

Developer pushes code
        ↓
GitHub Actions (or CodePipeline)
        ↓
Build image in CI environment (with internet access)
        ↓
Push image to Private ECR via VPC endpoint
        ↓
Trigger deployment (CodePipeline or self-hosted runner in VPC)
        ↓
kubectl/Helm apply via Systems Manager Session Manager
        ↓
EKS pulls image from private ECR (no internet needed)
        ↓
ALB routes traffic

Self-Hosted Runner in VPC

If using GitHub Actions, deploy a self-hosted runner inside the VPC. It can reach the private EKS API endpoint and ECR via VPC endpoints:

# .github/workflows/deploy.yml
jobs:
  deploy:
    runs-on: self-hosted  # runner inside VPC
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/cicd-deploy-role
          aws-region: us-east-1

      - name: Login to ECR
        run: |
          aws ecr get-login-password | docker login \
            --username AWS \
            --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com

      - name: Deploy to EKS
        run: |
          aws eks update-kubeconfig --name fintech-prod
          helm upgrade --install my-app ./helm/my-app \
            --set image.tag=${{ github.sha }}

First Deployment Validation

Don't call the pipeline "working" until you've traced the full path end to end: image push → ECR → EKS pod pull → running pod → ALB health check → live traffic. Each hop can fail independently in an air-gapped setup.

How Should Data Stores Be Configured in an Air-Gapped VPC?

Every data store in a regulated EKS deployment — RDS PostgreSQL, DynamoDB, and ElastiCache Redis — must be provisioned in private subnets with publicly_accessible: false set explicitly at the resource level, not just through security group rules. Security groups can be modified; the publicly_accessible flag removes the public DNS endpoint entirely, closing the exposure regardless of any future policy drift.

All data stores must be provisioned in private subnets with no public endpoint exposure.

RDS PostgreSQL with pgvector

For AI-augmented fintech applications, pgvector enables vector similarity search inside Postgres — useful for semantic search over transaction data, document embeddings, or fraud pattern matching.

# Terraform
resource "aws_db_instance" "postgres" {
  identifier           = "fintech-postgres"
  engine               = "postgres"
  engine_version       = "16.1"
  instance_class       = "db.r6g.large"
  multi_az             = true
  db_subnet_group_name = aws_db_subnet_group.private.name
  vpc_security_group_ids = [aws_security_group.rds.id]
  publicly_accessible  = false
  storage_encrypted    = true

  # Enable pgvector via parameter group
  parameter_group_name = aws_db_parameter_group.postgres_pgvector.name
}

Install the extension after provisioning:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops);

ElastiCache Redis and DynamoDB

Both run entirely in private subnets. ElastiCache Redis requires a subnet group scoped to private subnets. DynamoDB uses a gateway endpoint (free) — no interface endpoint needed.

How Do You Manage Secrets and Security Without Exposing Credentials?

Kubernetes Secret objects are base64-encoded, not encrypted — any cluster administrator with RBAC read access can decode them with a single command. In regulated environments, AWS External Secrets Operator resolves this by pulling credentials from AWS Secrets Manager at pod startup and syncing them into ephemeral Kubernetes Secrets. Credentials never appear in manifest files, Git history, or container image layers.

AWS WAF on the ALB

Attach a Web ACL to your Application Load Balancer with at minimum:

Core Rule Set (CRS) — protects against OWASP Top 10
Known Bad Inputs — blocks common injection payloads

The ALB sits in public subnets (it receives external traffic), but the security group only allows 443 inbound. Backend EKS nodes only allow traffic from the ALB security group.

Kubernetes Secrets Without Kubernetes Secrets

Storing secrets as Kubernetes Secret objects is fine for development, but they're base64-encoded, not encrypted, and cluster admins can read them. In a regulated environment, use External Secrets Operator to pull from AWS Secrets Manager instead:

# ExternalSecret — pulls from Secrets Manager into a Kubernetes Secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: fintech/prod/db
        property: password

Pods mount the resulting Kubernetes Secret normally — but the actual credential lives in Secrets Manager, has rotation enabled, and never touches a manifest file.

TLS and HTTPS Enforcement

Provision an ACM certificate for your domain and configure the ALB to redirect HTTP to HTTPS. Set HTTPS-only enforcement at the ALB listener level — don't rely on application code to enforce it.

How Do You Run Amazon Bedrock and Transcribe in an Air-Gapped Cluster?

Amazon Bedrock and Amazon Transcribe both support VPC interface endpoints, meaning all LLM inference and speech-to-text requests from private EKS workloads never leave the AWS network. For regulated industries, this keeps AI processing within the same network boundary as the rest of the application — data residency compliance is maintained without routing inference traffic through the public internet.

For platforms using Amazon Bedrock (LLM inference) or Amazon Transcribe (speech-to-text), both services support VPC interface endpoints — meaning model inference requests never leave the AWS network.

# Bedrock VPC endpoint
aws ec2 create-vpc-endpoint \
  --service-name com.amazonaws.us-east-1.bedrock-runtime \
  --vpc-endpoint-type Interface \
  --vpc-id vpc-xxx \
  --subnet-ids subnet-xxx subnet-yyy \
  --security-group-ids sg-endpoints

IAM policies for Bedrock should be scoped to specific model ARNs — don't grant bedrock:* broadly.

How to Evaluate LLM Outputs: Building Evals That Actually Catch Regressions

Dishant Sethi — Wed, 27 May 2026 15:50:09 +0000

Originally published on prodinit.com

Key Takeaways

Most LLM eval setups fail for three structural reasons: evaluating on metrics that don't reflect production failure modes, using golden datasets that have silently rotted, and running evals on a separate schedule from deployments

The four-layer eval stack — unit, reference, rubric, and behavioral — catches different regression types; shipping without all four leaves blind spots

GPT-4 as judge agrees with human experts 85% of the time on general tasks (Zheng et al., NeurIPS 2023), but that agreement drops to 60–68% in expert domains — calibrate before you trust it

A February 2025 Amazon study found INT4 quantization caused a 39.46% accuracy drop on Llama-3.3 70B — silent regressions from "safe" model changes are real and statistically detectable (Kübler et al., arXiv 2025)

Block deployments on rubric regressions ≥2% relative to the last passing run; warn on everything else

Why Most LLM Eval Setups Miss Regressions

42% of companies abandoned the majority of their AI initiatives in 2025, up from 17% in 2024 (S&P Global Market Intelligence, 2025). The default explanation is ROI. The technical explanation, in most cases, is that the system shipped fine and then quietly got worse — and nobody caught it until a customer did.

Three structural failure modes explain most missed regressions in production LLM systems.

Failure mode 1: Proxy metrics that don't predict production failure. Teams instrument BLEU score, exact match, or perplexity because those are easy to compute. A customer-facing summarisation model can maintain a BLEU score of 0.74 while its summaries become subtly contradictory after a retrieval change. BLEU measures token overlap; it doesn't measure factual consistency. The metric passed. The feature regressed.

Failure mode 2: Golden datasets that have silently rotted. A golden dataset built during initial evaluation captures the distribution of inputs that existed at that moment. Six months later, real traffic has drifted: new document formats, new query patterns, edge cases the original set never covered. Evaluating against a stale golden set produces a green score against a test that no longer represents the problem you're actually solving.

Failure mode 3: Evals that don't run at deployment time. Evaluation suites that run weekly, on a separate schedule from code deploys, detect regressions after they've been live for days. The culprit PR has already been merged and three others have been built on top of it. What you needed was a gate, not a report.

The Four-Layer Eval Stack

The single strongest change you can make to your eval setup is adding layers. Each layer catches different failure modes; each is cheap to run for what it surfaces. Shipping any one layer in isolation leaves a class of regression invisible.

Layer 1: Unit Evals

Unit evals test individual capabilities in isolation: does the model correctly extract a date from a structured input? Does it refuse an off-topic request? Does it stay within a 200-word limit when instructed to? These are deterministic — the answer is either correct or it isn't.

Unit evals run in milliseconds, require no LLM calls for evaluation, and give you a precise signal when a model update breaks a capability it previously had. They are the first gate in the pipeline: cheap to fail, cheap to fix.

Layer 2: Reference Evals

Reference evals compare model output against a gold-standard answer using a similarity metric. They're appropriate when outputs have a correct or near-correct form: code generation, factual Q&A with a known answer, structured extraction against a schema.

The weakness: reference evals degrade with output diversity. A model that answers correctly but in different words than the reference will score low. Use them where correctness has a tight definition. Avoid them for open-ended generation where paraphrase is acceptable.

Layer 3: Rubric Evals (LLM-as-Judge)

Rubric evals ask a separate LLM to score the output against a defined rubric. This is the only practical approach for evaluating coherence, helpfulness, or factual consistency at scale — human annotation doesn't scale to continuous deployment. Stanford's HELM benchmark applies seven evaluation metrics across 42 real-world scenarios using a comparable rubric-based approach at research scale.

Rubric evals are powerful but require calibration. See the LLM-as-Judge section below for the documented failure modes.

Layer 4: Behavioral Evals

Behavioral evals test system-level properties that don't reduce to a single output score: does the system stay in character across a 10-turn conversation? Does it escalate correctly when the user indicates distress? Does retrieval-augmented generation cite only sources it actually retrieved?

These require end-to-end test harnesses or carefully instrumented integration tests. They're more expensive to run but catch a class of regression that the other three layers cannot: failures that only manifest across interactions or under specific system conditions. They also run slower — which matters for your CI blocking policy, covered below.

Golden Datasets: How They Rot and How to Refresh Them

A golden dataset is the most valuable artifact your evaluation pipeline owns, and it has an expiry date nobody writes down.

Datasets rot in three ways. Input drift: real user queries evolve — new terminology, new intents, new edge cases — and your golden set stops representing them. Label rot: the correct answer changes. A customer service bot's golden dataset might contain ideal answers that reference a product feature that no longer exists. Coverage gaps: your initial dataset captured the happy path. Production traffic eventually surfaces the long tail that was never represented.

The practical fix is a two-track refresh strategy.

Track 1: Scheduled review. Every 90 days, pull a stratified sample of real production inputs — at minimum, 200 examples per major intent cluster — and manually verify that the golden labels are still correct. Flag rows where the ideal answer has changed. Retire rows from deprecated flows. Statsig's research on golden dataset maintenance recommends marking rows stale after 90 days unless re-verified; persistent drift is a signal the dataset no longer reflects reality.

Track 2: Failure-driven refresh. When a customer-reported regression reaches you, trace it back to the eval suite. If the failing case wasn't in the golden set, add it — annotated with why it failed and what the correct output should have been. A regression that reaches production is, at minimum, a contribution to the golden dataset. Don't waste the signal.

One diagnostic worth running: if your eval suite consistently scores above 90% but your support tickets are increasing, the dataset has drifted past the real problem space. That 90% is measuring something — it's just no longer measuring the right thing.

LLM-as-Judge: When It Works, When It Lies

LLM-as-judge is a necessary tool for evaluating open-ended outputs at scale. It's also unreliable in specific, documented ways. Use it without understanding those ways, and your rubric evals will give you false confidence.

What works. GPT-4 as judge achieves 85% agreement with human expert evaluators on general-task benchmarks (MT-Bench), and 83–87% agreement on Chatbot Arena evaluations (Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," NeurIPS 2023, via Eugene Yan). For general-purpose, non-expert tasks, LLM-as-judge is a defensible substitute for human annotation if you validate the judge against your specific rubric before deploying it.

What lies.

Verbosity bias. Both GPT-3.5 and Claude (v1) preferred longer responses over shorter ones more than 90% of the time, independent of correctness (Zheng et al., NeurIPS 2023). If your outputs tend to be long and verbose, a verbosity-biased judge will score them well even when they're wrong. Mitigate by normalising output length in your rubric prompt or running paired length-controlled evaluations.

Self-preference bias. GPT-4 as judge gave a 10% win-rate advantage to GPT-4-generated outputs; Claude v1 showed a 25% self-preference bias (Zheng et al., NeurIPS 2023). If your production model and your judge share a model family, expect inflated scores. Use a different model family for the judge.

Expert domain degradation. Agreement between LLM judges and human domain experts drops to 60–68% in fields like dietetics and mental health (ACL/EMNLP 2024, via ACM DL). If you're evaluating a healthcare, legal, or highly specialized technical application, LLM-as-judge is not a substitute for domain expert annotation on the rubric dimensions that matter most.

Calibration process. Before deploying a rubric eval in CI: (1) define explicit scoring criteria with labelled examples for each score level; (2) run the judge on 50–100 human-annotated examples and measure agreement; (3) if agreement is below 75% on your specific rubric, revise the rubric or change the judge model. The 2024 survey on LLM-as-a-Judge provides a comprehensive bias taxonomy useful as a calibration checklist. Treat LLM-as-judge as a probabilistic instrument you've validated — not a ground truth.

Wiring Evals into CI: What to Block On, What to Warn On

Running evals in CI without a blocking policy produces reports, not gates. The purpose of CI eval integration is to make a shipping decision: does this diff change behavior in a way that crosses a regression threshold?

The integration pattern that works in production:

# eval_pipeline.py — framework-agnostic eval runner
# Runs on every PR against main; blocks merge if BLOCK conditions fail

def run_eval_suite(model_version, golden_dataset, thresholds):
    results = {}

    # Layer 1: Unit evals — run all, block on any failure
    results["unit"] = run_unit_evals(model_version)

    # Layer 2: Reference evals — block if accuracy drops below floor
    results["reference"] = run_reference_evals(
        model_version,
        golden_dataset,
        metric="exact_match_normalized"
    )

    # Layer 3: Rubric evals — block on relative regression vs baseline
    results["rubric"] = run_rubric_evals(
        model_version,
        golden_dataset,
        judge_model="gpt-4o",   # different family from production model
        rubric=RUBRIC_CONFIG
    )

    # Layer 4: Behavioral evals — warn only; too slow to block on every PR
    results["behavioral"] = run_behavioral_evals(model_version)

    return evaluate_thresholds(results, thresholds)


THRESHOLDS = {
    "unit":       {"block_on_any_failure": True},
    "reference":  {"block_if_below": 0.92},
    "rubric":     {"block_if_regression_vs_baseline": 0.02},  # 2% relative
    "behavioral": {"warn_only": True},
}

What to block on. Any unit eval failure. Reference accuracy falling below your defined floor. Rubric score dropping more than 2% relative to the last passing run on main. These signals have high signal-to-noise ratio — when they fire, they reliably indicate a regression rather than measurement variance.

What to warn on. Behavioral eval regressions (too slow and too variable to block every PR), single-dimension rubric drops that don't cross the aggregate threshold, and latency increases above your SLO. Warnings go into the PR review, not the merge gate.

The baseline problem. Your blocking threshold needs a reference point. Store eval results in a persistent store — a JSON file in the repo works; a purpose-built eval tracking system works better — and compare each run to the last green run on main. Don't compare to a fixed absolute. Compare to a rolling baseline that advances with intentional quality improvements.

Our AI Infrastructure & LLMOps service wires eval pipelines directly into deployment workflows so that model updates, retrieval changes, and prompt edits all pass through the same gate before reaching production.

A Regression That Slipped Through (and the Eval That Would Have Caught It)

A retrieval-augmented clinical documentation system was producing accurate outputs in testing. Production ROUGE-L scores were stable at 0.81. An infrastructure team updated the vector database and reindexed the embeddings corpus. No model weights changed. The migration was flagged as non-breaking.

Two weeks later: escalating complaints from clinical staff. Summaries were citing facts from adjacent patient records in a multi-tenant environment. The retrieval had started returning higher-cosine-similarity results from nearby tenant partitions due to an index partitioning bug introduced in the new release.

What the eval suite had: ROUGE-L score on golden summaries (Layer 2).

What it didn't have: a cross-tenant citation check (Layer 4 behavioral), or a factual grounding check verifying that every claimed fact appeared in the retrieved source documents (Layer 3 rubric).

The eval that would have caught it: A rubric eval scoring "all factual claims in the output are supported by at least one retrieved source document" — rated by an LLM judge with access to both the output and the retrieved context. This would have flagged outputs immediately: claims were present in the generation, but the supporting documents in context were from different records.

A behavioral eval running 20 end-to-end test cases with known tenant isolation requirements would have caught the regression in the first CI run after the index migration.

Neither eval existed because both required knowing what to test before the failure occurred. The lesson isn't that you should anticipate every specific bug. It's that behavioral evals should cover the properties your system must hold regardless of what changes — tenant isolation, citation grounding, output fidelity are invariants, not features. They belong in the eval suite from day one, not after the first production incident.

A related pattern appears in model optimisation: the Amazon arXiv study (February 2025) found that INT4 quantization — routinely treated as a cost-reduction step with negligible quality impact — caused a 1.73% accuracy drop on Llama-3.1 8B and a 39.46% drop on Llama-3.3 70B. The study also showed the McNemar statistical test can detect accuracy degradations as small as 0.3% — meaning you don't need large regressions to justify measurement. You just need to be measuring.