DEV Community: Alok Ranjan Daftuar

Context Engineering: The Discipline That Determines What Your LLM Actually Sees

Alok Ranjan Daftuar — Mon, 29 Jun 2026 08:11:02 +0000

Prompt engineering asks: how do I phrase this instruction? Context engineering asks: what information does the model need, in what form, in what order, and how much of it — to produce a correct answer?

For a long time, the implicit mental model was: give the LLM more context and it performs better. This is wrong. A 20,000-token window stuffed with weakly relevant content produces worse answers than a 4,000-token window with precisely curated information. Larger windows do not eliminate context quality problems — they amplify them.

The Context Window Is a Budget

Treat it as a budget with competing line items, not a container you fill. Start with the total window, subtract fixed allocations (system prompt, output reserve, safety margin), and what remains is your dynamic budget split across retrieved chunks, conversation history, and memory.

The first question should always be: "can we get better at selecting less, rather than including more?"

Four Memory Types, Four Purposes

Episodic — conversation history. Highest priority for continuity. Grows unbounded — needs compression.
Semantic — durable facts about the user (role, team, preferences). Compact, injected in system prompt before retrieved content.
Procedural — reusable workflows and SOPs. Retrieved selectively when the query type matches.
Working — intermediate results within a single request (agentic loop output). Ephemeral, request-scoped.

Each type has different durability, update frequency, and token cost. Conflating them into a single undifferentiated store is the most common memory architecture mistake.

Structured Injection Patterns

XML tags for section boundaries (<documents>, <user_context>, <instructions>) — gives the model clear anchors for where information types begin and end
Indexed documents — label chunks with indices so citations can be traced
Ordering matters — most relevant content first (primacy effect), user query last (recency effect)
Grounding instruction is not optional — explicit instruction to use only provided context and signal when insufficient

Lost-in-the-Middle

Models attend more strongly to content near the beginning and end of the context window. Information buried in the middle receives less attention. Mitigations:

Relevance-ordered injection (highest score first)
Sandwich pattern (critical content at both start and end)
Active relevance filtering (exclude low-scoring chunks even if they fit)
Smaller, tighter windows (fewer high-quality chunks > more mediocre chunks)

Conversation Compression

A 100-turn conversation consumes your entire retrieved context budget. Naive truncation loses critical early constraints. Solutions:

Sliding window with pinned turns — critical turns (user constraints, decisions) never truncated
Progressive summarization — compress old segments into 3-5 sentence summaries using Haiku (cheap, mechanical task)

Context Assembly Is Testable

Unit test your assembly layer: budget compliance, ordering preserved, critical turns survive truncation, no mid-chunk truncation. Every assembly failure produces a predictable RAGAS metric signature — context precision drops point to noisy inclusion, faithfulness drops point to contradictions.

Read the Full Article

This is a summary of my deep dive into context engineering. The full article covers the complete discipline with production implementations:

👉 Context Engineering: The Discipline That Determines What Your LLM Actually Sees — Full Article

The full article includes:

Context window budget accounting with Python dataclasses
Four memory types with implementation patterns (episodic, semantic, procedural, working)
Working memory bridge from agentic retrieval loops
XML-structured injection with document indexing
Primacy/recency ordering strategy
Progressive summarization with critical turn pinning
Lost-in-the-middle mitigation (4 strategies with code)
Contradiction detection and resolution
Noise taxonomy (stale, tangential, redundant, over-retrieved)
Unit testing context assembly
AssemblyMetadata integration with RAGAS eval pipeline
RAGAS metric → assembly failure mapping table
Production checklist (19 items)

Agentic RAG: Designing Self-Correcting Retrieval Loops for Production

Alok Ranjan Daftuar — Mon, 22 Jun 2026 05:59:20 +0000

Standard RAG retrieves once and hopes for the best. Agentic RAG retrieves, reflects, decides it was wrong, and tries again — without being told to.

Single-pass RAG has a fundamental flaw: it commits to its first retrieval attempt and generates forward regardless. It has no mechanism to check whether the retrieved chunks actually contain the answer. This works for simple factual queries. It breaks on multi-hop questions, ambiguous intent, and analytical queries requiring sequenced lookups.

The Architecture

An agentic RAG system treats retrieval as a tool available to a reasoning loop. The LLM decides what to retrieve, evaluates what came back, and determines when to stop.

The key component: a reflection agent sits between retrieval and generation. It evaluates the quality and sufficiency of accumulated context and either terminates the loop or sends it back with a refined query.

Three patterns in increasing complexity:

Iterative Query Refinement — single tool, query rewritten per pass
Multi-Tool Orchestration — agent selects between keyword, semantic, hybrid, and filtered search
Hierarchical Decomposition — planner splits multi-hop queries into dependent sub-queries

Routing: The Most Important Decision

Sending every query through the agentic path is the most common mistake. Agentic retrieval adds 2-8s latency and 4-12x cost. Simple factual queries (60-75% of typical traffic) get no quality improvement from it.

Use a hybrid router: deterministic rules first (regex patterns, length heuristics, keyword signals), LLM classification only for ambiguous cases. Use Haiku for routing — it's a classification task, not a reasoning task.

Reflection Agent: Deciding When to Stop

The reflection agent's judgment quality determines the entire system's utility. Calibrate it against real queries:

Iteration 1: 65-75% of queries should terminate (simple queries succeeding on first pass)
Iteration 2: 15-20% (needed one refinement)
Iteration 3: 5-10% (multi-hop or genuinely ambiguous)
Iteration 4+: <5% (forced termination — investigate these)

If significant traffic hits max iterations, either routing is broken or your corpus has coverage gaps.

Failure Isolation and Loop Bounding

Without explicit bounding, misbehaving loops drive latency and cost to unacceptable levels. Non-negotiable limits:

max_iterations: 4 — never exceed
timeout: 12s — wall-clock for entire loop
min_new_chunks_per_iteration: 1 — if retrieval returns nothing new, break immediately
context token budget — stop accepting chunks beyond the budget

On timeout or max iterations: generate with accumulated context + caveat, never return a 500 error.

Cost Reality

Single-pass RAG:     ~$0.003/request
Agentic (2 iter):    ~$0.006/request (2x)
Agentic (4 iter):    ~$0.010/request (3-4x)

If 25% of traffic goes agentic at 2.5x cost → 37% total increase (acceptable). If 75% goes agentic → costs triple (likely unacceptable). The router controls your bill.

The Key Insight

An agentic system with no observability is not an improvement over single-pass — it's a more expensive pipeline that's harder to debug. The loop delivers quality improvement only when it is instrumented, bounded, and its behavior is understood at the query level.

Agency without accountability is just unpredictability.

Read the Full Article

This is a summary of my deep dive into agentic RAG architecture. The full article covers the complete system with production implementations:

👉 Designing Self-Correcting Retrieval Loops for Production — Full Article

The full article includes:

Full agentic RAG architecture diagram (router → planner → loop → generation)
Query planner implementation with multi-hop decomposition (Python/Anthropic)
Iterative retrieval loop with async timeout and dedup
Reflection agent prompt and calibration patterns
Multi-tool orchestration with Claude tool-use API
Hybrid router (rules-first + LLM fallback)
Loop bounding with five hard limits
Graceful degradation with context caveats
Per-request cost model (single-pass vs 2-iter vs 4-iter)
Latency budget breakdown and streaming response pattern
Structured loop telemetry with structlog
Alerting metrics for agentic systems
Production deployment checklist

LLM Evaluation in Production: Building the Eval Pipeline That Runs on Every Deploy

Alok Ranjan Daftuar — Wed, 17 Jun 2026 13:22:36 +0000

Everyone ships the RAG system. Almost nobody ships the eval system that tells them when the RAG system starts lying.

You updated the embedding model. Tweaked the system prompt. Swapped the re-ranker. Metrics look fine. Three weeks later, support tickets arrive — the system is drawing inferences the source documents never made. No alarm fired. No test failed. The system drifted silently.

This is not a model quality problem. It is an evaluation infrastructure problem.

The Four Metrics That Matter

Faithfulness — of the claims in the response, what fraction are directly supported by the retrieved context? Your primary hallucination guard. Does not require ground truth.

Answer Relevance — how directly does the response address the user's question? Catches the "technically correct but useless" failure mode. Does not require ground truth.

Context Precision — of the retrieved chunks, what fraction were actually relevant? Requires ground truth. Belongs in offline CI eval.

Answer Correctness — how factually accurate vs the reference answer? Most expensive, requires curated ground truth. Pre-deploy regression suite only.

Operational rule: Faithfulness and Answer Relevance run on every deploy and on sampled production traffic. Context Precision and Answer Correctness run in CI against the golden dataset.

LLM-as-Judge: The Pattern and Pitfalls

RAGAS uses an LLM to evaluate LLM output — the only practical way to evaluate semantic quality at scale.

Pitfalls to manage:

Positional bias — randomize order in pairwise comparisons
Verbosity bias — judge rates longer answers higher even when less accurate
Self-preference — use a different model family as judge than the one generating answers
Calibration drift — pin judge model to a specific version; treat upgrades as baseline resets

Calibrate against human labels using Cohen's Kappa on 50-100 examples. Below 0.4 means your judge prompt needs revision.

CI/CD Integration

The eval pipeline triggers on every PR touching RAG code, prompts, or model configuration:

Run RAG pipeline against golden dataset (100+ curated questions)
Score with RAGAS (faithfulness, relevance, precision, correctness)
Compare against baseline — block deploy if regression exceeds threshold
Post results as PR comment with per-metric scores and pass/fail status

Cost: ~$0.50-$2.00 per full eval run at Claude Sonnet pricing. On PRs, run only faithfulness + relevance (cheapest). Full suite runs nightly.

Production Sampling

CI catches regressions from code changes. Production sampling catches drift from corpus staleness, query distribution shift, and model behavior changes.

Sample 5% of live traffic for async evaluation. Never evaluate synchronously — judge calls add 2-5s per request. Track 7-day rolling faithfulness and answer relevance. Alert when they drop >0.05 from monthly baseline.

The Key Insight

LLM systems do not have stable, deterministic behavior. They drift through corpus changes, model updates, prompt evolution, and query distribution shift. Evaluation is not a checkpoint — it is continuous infrastructure.

Build the eval system before you need it. By the time you need it, it is already too late — you will be debugging a production quality regression with no historical baseline and no automated detection.

Read the Full Article

This is a summary of my deep dive into LLM evaluation infrastructure. The full article covers the complete eval stack with implementation examples:

👉 LLM Evaluation in Production — Full Article

The full article includes:

Evaluation stack architecture (retrieval layer vs generation layer)
Four metrics with RAGAS Python implementations
LLM-as-Judge faithfulness prompt with claim-level scoring
Judge calibration against human labels (Cohen's Kappa)
RAGAS configuration with Claude as judge model
Regression threshold framework (absolute + delta from baseline)
Golden dataset generation, versioning, and holdout partitions
Full GitHub Actions eval pipeline (YAML + runner scripts)
Production sampling with async eval queue worker
Eval observability dashboard schema (PostgreSQL)
Eight failure modes in eval systems and mitigations
Production deployment checklist

Building Reliable RAG Pipelines: From Prototype to Production

Alok Ranjan Daftuar — Wed, 10 Jun 2026 06:20:00 +0000

Most teams get RAG working in a notebook over a weekend. Very few get it working reliably in production. The gap is not model quality — it is engineering discipline.

The RAG prototype is fifty lines of Python. It works. Then production happens — users ask unexpected questions, retrieval degrades as the corpus grows, and the model confidently synthesizes wrong answers from bad context. Nobody knows, because there is no instrumentation to catch it.

Chunking: The Foundation

A poor chunking strategy cannot be compensated for downstream. If relevant information is split across chunks or diluted into one too large, no retrieval algorithm will recover it.

Hierarchical chunking is the production-grade pattern: maintain parent chunks (full sections) and child chunks (sentences/short paragraphs). Retrieve at child granularity for precision. Return parent text as LLM context for completeness.

Every chunk must carry metadata — source document ID, version, content hash, embedding model version. content_hash tells you when a chunk needs re-embedding because the source changed.

Retrieval: Hybrid Is the Default

Neither BM25 nor vector search alone is sufficient. Hybrid retrieval with Reciprocal Rank Fusion (RRF) is the baseline for production RAG.

The pipeline:

Dense retrieval (vector similarity) + Sparse retrieval (BM25 keywords) in parallel
RRF merge — rank-based fusion without score normalization
Cross-encoder re-ranker — precision pass on top candidates

Skipping the re-ranker is the most common mistake. Initial retrieval optimizes for recall. The re-ranker optimizes for precision — critical when your context window only fits top-5 chunks.

Context Assembly: Where Pipelines Quietly Break

Token budget management — hard ceiling, never rely on hope that chunks fit
Deduplication — hierarchical chunking and hybrid retrieval can surface the same content via multiple paths
Source attribution — every chunk in context must carry its source ID for citation

The "I Don't Know" Instruction Is Not Optional

Without explicit grounding instructions, LLMs fill context gaps with plausible hallucinations. Your system prompt must instruct the model to acknowledge when context is insufficient — and to cite sources for every factual claim.

Evaluate Retrieval Independently

The most common RAG debugging mistake: assuming a bad answer is a generation failure. Most bad RAG answers are retrieval failures — the right chunk was not in the context.

Measure Recall@K and MRR against a ground truth dataset of 50-100 queries. Fix retrieval before you blame the model.

Production Observability

A RAG pipeline without observability is a black box that silently degrades. Key signals:

"I don't know" rate — drops below 80% signals retrieval degradation
Chunks dropped rate — rising means context window pressure
Retrieval latency p99 — vector index performance
Corpus staleness — content hash mismatches between source docs and stored chunks

Read the Full Article

This is a summary of my deep dive into production RAG engineering. The full article covers every pipeline component with implementation examples:

👉 Building Reliable RAG Pipelines — Full Article

The full article includes:

Full pipeline architecture diagram (9 stages)
Three chunking approaches with Python implementations (fixed, semantic, hierarchical)
Hybrid retrieval with RRF implementation (Qdrant)
Cross-encoder re-ranking (self-hosted and Cohere API)
Context assembly with token budget management and deduplication
Prompt construction with grounding and guardrails
Retrieval evaluation framework (Recall@K, MRR, context relevance)
Per-request tracing schema and aggregate alerting metrics
Corpus staleness detection implementation
Graceful degradation with BM25 fallback
Production deployment checklist

Event-Driven Architecture: The Dual Write Problem and How to Solve It

Alok Ranjan Daftuar — Thu, 04 Jun 2026 06:32:28 +0000

You have a well-designed order service. It writes to the database and publishes an event to Kafka. Clean, decoupled, event-driven. Then Kafka has a brief network hiccup. The database write succeeds. The event publish fails. The order exists. Fulfillment never hears about it. No alert fires. Just a quietly broken order going nowhere.

This is the dual write problem — an architectural correctness problem that exists the moment you write to two separate systems without a coordination mechanism.

The Problem

A dual write occurs when your application writes to two separate systems as part of a single logical operation without atomicity across both. The dangerous failure modes are silent — the HTTP response returns 200, the client gets a success, and nothing downstream happens.

The naive fixes don't work:

Try/catch with retry — introduces duplicate events; consumers must be idempotent
Publish first, then write DB — just reverses which failure mode you're exposed to
Distributed transactions (2PC) — sacrifices availability and introduces distributed locking

The real solution: reduce to a single atomic write and derive the event from it.

Solution 1: Transactional Outbox Pattern

Write the event as a row in an outbox table in the same database transaction as your business data. A separate relay process reads from the outbox and publishes to the broker.

Both writes succeed or fail together (single DB transaction)
Relay publishes and marks messages as published
Guarantees at-least-once delivery — consumers must be idempotent

Best for: greenfield services, full control over event schema, teams wanting simplicity.

Solution 2: Change Data Capture (Debezium)

Read directly from the database's transaction log (WAL/binlog). Every committed write is captured and streamed to Kafka automatically. No application code changes required.

Sub-second publish latency (WAL-based, no polling)
Captures all state changes including DB migrations and admin tools
Requires infrastructure for Kafka Connect + Debezium

Best for: legacy systems, high-throughput services, capturing all state changes without code modification.

Solution 3: Event Sourcing

The event log is the source of truth. The database is a derived projection. There is no dual write because there is only one write — appending events to the event store.

Eliminates the problem entirely
Introduces significant complexity (schema versioning, aggregate rehydration, eventual consistency)

Best for: domains where history of state changes matters (financial systems, audit-heavy domains).

Operational Non-Negotiables

Consumer idempotency — at-least-once delivery means duplicates will arrive. Deduplicate on event ID.
Outbox housekeeping — purge published messages; don't let the table grow unbounded.
Replication slot monitoring — for CDC, a stuck connector causes WAL accumulation and disk exhaustion.

Read the Full Article

This is a summary of my deep dive into the dual write problem. The full article covers all three solutions with production implementation examples:

👉 The Dual Write Problem and How to Solve It — Full Article

The full article includes:

Four failure scenarios with a dual write matrix
Transactional Outbox Pattern implementation (.NET with EF Core)
Polling relay vs log-tailing relay comparison
Debezium PostgreSQL connector configuration
Event Sourcing with aggregate pattern (C#)
Decision matrix for choosing between the three solutions
Operational concerns: housekeeping, replication slot monitoring, consumer idempotency
Production deployment checklist

AI-Assisted Data Reconciliation at Scale: Patterns for Distributed Systems

Alok Ranjan Daftuar — Mon, 01 Jun 2026 08:41:05 +0000

In any sufficiently large distributed system, data reconciliation is the dark matter of engineering — invisible, pervasive, and holding everything together through mechanisms nobody fully understands.

Rule-based reconciliation works until it doesn't. Rule engines break on ambiguity, cannot handle semantic equivalence across schema versions, and generate false positives at scale that overwhelm operations teams. AI — specifically embedding-based similarity and LLM classification — fills the gap. Not as a replacement, but as a layer that handles what rules cannot.

Where Traditional Reconciliation Breaks Down

Eventual consistency windows — a naive reconciliation job that diffs at a point in time generates thousands of false positives that are self-healing within seconds. The rule engine cannot distinguish transient inconsistency from legitimate divergence.

Cross-service schema drift — Service A stores an address as { street, city, state, zip }. Service B stores it as { addressLine1, municipality, postalCode }. Semantically equivalent. A field-level comparator flags every record as mismatched.

Semantic equivalence in free-text — "Acme Corporation" vs "ACME Corp." vs "Acme Corp (formerly Roadrunner Supplies)". Rule-based systems cannot reason about semantic identity at scale.

Volume-driven false positive fatigue — at millions of records per day, even 0.1% false positives generate thousands of alerts. Real issues get buried. The reconciliation system becomes theater.

The Architecture: Rules First, AI at the Boundary

The pattern is not AI-first. It is rules-first, AI at the boundary:

Deterministic mismatch detection — checksums, field comparisons, primary key matching
High-confidence matches/mismatches — auto-resolve or route to correction (no AI needed)
Ambiguous cases → AI classification layer:
- Embedding similarity — detect semantic equivalence across schema variations
- LLM classification — reason about why a mismatch exists

Embedding-Based Similarity

Serialize records into schema-agnostic text representations before embedding. Compute cosine similarity. Calibrate thresholds against labeled data:

≥ 0.95 → auto-resolve as equivalent
0.80 – 0.95 → route to LLM classification
< 0.80 → high-confidence mismatch, route to correction

The thresholds are not universal — calibrate against 500–1000 manually classified record pairs from your actual data.

LLM Classification for the Ambiguous Band

For the 5–15% of mismatches that fall in the ambiguous range, an LLM reasons about context that vector distance cannot capture. Classifications: equivalent, stale_copy, legitimate_divergence, data_corruption.

Cost management: route only the ambiguous band to the LLM. Batch where latency allows. Cache results for record pairs re-evaluated in subsequent cycles.

Where AI Should Never Be Trusted

Financial and compliance records — dollar amount disagreements are correctness errors, not semantic questions
Primary key and identity resolution — AI suggestions acceptable; auto-resolution without human sign-off is not
Any decision that must be explainable to a regulator — "87% confidence" is not an audit-compliant explanation

The Key Insight

AI in reconciliation is a judgment layer, not a trust layer. It handles ambiguous cases that rules cannot, reduces volume reaching human review, and provides structured reasoning. The deterministic foundation must remain intact.

A reconciliation system you cannot audit is worse than one that generates false positives. Build the observability before you build the AI.

Read the Full Article

This is a summary of my deep dive into AI-assisted data reconciliation. The full article covers the complete architecture with implementation examples:

👉 AI-Assisted Data Reconciliation at Scale — Full Article

The full article includes:

Where traditional reconciliation breaks down (4 failure modes)
Full architecture diagram with rules-first, AI-at-boundary pattern
Embedding-based similarity implementation (Python, OpenAI embeddings)
LLM classification prompt pattern with structured JSON output
Observation window pattern for filtering eventual consistency false positives
Hard boundaries where AI should never auto-resolve
Observability patterns with structured logging
Production deployment checklist

Why Lift-and-Shift Fails Quietly: Architectural Smells That Appear After Migration

Alok Ranjan Daftuar — Fri, 29 May 2026 07:34:23 +0000

Every cloud migration starts with a promise: "We'll get onto cloud first, optimize later." That sentence is where the trouble begins.

Lift-and-shift leaves on-premises assumptions baked into a system operating in a fundamentally different environment. The failure doesn't arrive on day one. It arrives three months later, in a Slack alert at 2am, or in an invoice that made a VP ask uncomfortable questions.

1. Latency Amplification

On a physical LAN, a service call is sub-millisecond. In a cloud VPC, even same-AZ calls incur 1-3ms. A service making 40 synchronous downstream calls goes from ~4ms network overhead to ~160ms — without any code change.

Same call graph. Same code. 8x more latency — purely from network topology.

Fix: consolidate reads with batch APIs, introduce async messaging for non-critical paths, add caching for hot reference data.

2. Chatty Services

The N+1 problem at infrastructure scale. A service making 60 per-entity HTTP calls to render a dashboard is annoying on LAN. In cloud, it's a 300-600ms tax on every page load.

Chatty patterns also exhaust connection pools faster — each call traverses the network and holds an open connection during transit.

Fix: batch endpoints on all internal APIs, DataLoader pattern, connection pool profiling under realistic concurrency.

3. Cost Surprises

The PoC cost $340. The first production month is $8,200. Nobody changed the architecture.

Data egress — free on-prem, metered in cloud. Cross-AZ, cross-region, and internet egress all bill.
Over-provisioning — on-prem sizing instincts (buy for 3-5 years) don't translate. Cloud charges per idle CPU cycle.
Idle infrastructure — dev/staging environments left running 24/7.

4. Stateful Assumptions

In-memory session state works with a single server. The moment you auto-scale, 33% of requests hit instances with no session. Filesystem dependencies break when containers reschedule or pods restart.

Fix: externalize session to Redis. Replace local filesystem writes with object storage at the upload boundary.

5. The Observability Void

On-prem monitoring (Nagios, Zabbix) watches hardware metrics that mean nothing in cloud. What you need to observe is different: cold start times, managed service throttling, connection pool utilization, cost-per-request.

The danger window is immediately after migration when legacy monitoring reports "all green" while user-facing metrics degrade invisibly.

6. The Monolith in Microservice Clothing

Containerized and deployed to Kubernetes with separate deployments per service. On the surface: microservices. Underneath: shared database schemas, synchronous HTTP chains, coordinated deployments. A distributed monolith you think is clean is a production incident waiting to happen.

A Realistic Migration Philosophy

Lift-and-shift is not a failure state. It's a phase. The mistake is treating it as a destination. Every migrated workload should have a documented list of known architectural debts, an owner for each, and a timeline to address them — agreed before the migration.

Moving to cloud does not modernize your architecture. It gives you a new environment in which your existing architectural decisions — good and bad — will be amplified.

Read the Full Article

This is a summary of my deep dive into post-migration architectural smells. The full article covers all six patterns with diagnostics, mitigations, and a pre-migration review checklist:

👉 Why Lift-and-Shift Fails Quietly — Full Article

The full article includes:

Latency amplification with SVG architecture diagram (on-prem vs cloud)
Chatty services with before/after code examples and connection pool diagnostics
Cost surprise breakdown with egress pricing tables
Stateful assumptions with session externalization code (Node.js/Redis)
Observability void with Prometheus recording rules for post-migration signals
Distributed monolith diagnostic patterns
Complete pre-migration architecture review checklist

Designing Cloud-Native Systems That Survive Region-Level Failures

Alok Ranjan Daftuar — Wed, 20 May 2026 10:56:50 +0000

Most teams design for instance and zone failures but treat region-level outages as someone else's problem. Region-level failures are rare — but they are not theoretical. AWS us-east-1 has had multiple significant incidents. Azure AD suffered a global authentication outage in 2023. Google Cloud's europe-west9 went offline due to a data center fire.

When a region fails, the blast radius is not one service. It is every workload, every database, every queue, and every control plane operation scoped to that region.

Multi-AZ Does Not Protect Against Regional Failures

Multi-AZ protects against data center failures. It does not protect against:

Regional control plane failures — the API that manages your resources is regional. If it degrades, you cannot scale or deploy.
Regional service outages — SQS, Lambda, DynamoDB, Cosmos DB are all regional.
Shared fate dependencies — IAM, Secrets Manager, Key Vault are regional. If your app cannot retrieve secrets, it doesn't matter that compute is healthy across three AZs.

The December 2021 AWS us-east-1 incident demonstrated this. Services in unaffected AZs experienced degradation because their dependencies were not AZ-independent.

Multi-Region Architecture Patterns

Pilot Light — secondary region has minimum infrastructure (DB replicas, networking). Compute provisioned on failover. RTO: 15-60 min. Cost: ~10-15% of primary.

Warm Standby — secondary runs a scaled-down but fully functional copy. On failover, scale up and promote DB. RTO: 5-15 min. Cost: ~25-40% of primary.

Active-Active — both regions serve traffic simultaneously. No failover needed. Requires multi-region writes (DynamoDB Global Tables, Cosmos DB) and conflict resolution. RTO: near-zero. Cost: ~80-100%+ of primary.

Data Replication: The Hardest Problem

Synchronous — zero data loss, but adds 50-150ms to every write. Impractical for most workloads.
Asynchronous — no write latency impact, but creates a replication lag window where data can be lost if primary fails.

For active-active with async replication, you need a conflict resolution strategy. Last-writer-wins works for profiles and preferences. It silently drops writes for counters and balances — use application-level merge or CRDTs there.

Practical rule: if you cannot define a conflict resolution strategy for a data entity, route its writes to a single primary region.

Failover Automation

Manual failover is not failover. Under the stress of a region-level incident, manual steps fail or take far longer than practiced.

DNS-based failover (Route 53, Traffic Manager) with health checks on actual regional functionality — not just process liveness
Database promotion automated via API (Aurora Global: under 1 minute, RDS cross-region: 5-10 minutes)
Test quarterly — not a tabletop exercise, an actual failover. Measure real RTO. Fix the gaps.

Common Mistakes

Untested failover — an assumption, not a plan
Hidden regional dependencies — auth provider or secrets manager pinned to one region
Same deployment pipeline for both regions — a bad deploy takes down both simultaneously
No capacity planning — secondary region hits service quotas during scale-up

Read the Full Article

This is a summary of my deep dive into multi-region resilience. The full article covers all patterns with AWS and Azure architecture sketches, cost analysis, and a decision framework:

👉 Designing Cloud-Native Systems That Survive Region-Level Failures — Full Article

The full article includes:

Multi-AZ vs multi-region — what each actually protects (and what it doesn't)
Three patterns with RTO/RPO/cost profiles (Pilot Light, Warm Standby, Active-Active)
AWS and Azure architecture sketches for active-active
Data replication deep dive (sync vs async, managed DB options, conflict resolution)
Failover automation with Route 53 config and health check design
Cost vs resilience decision framework with workload tiering
Six common mistakes that break multi-region architectures

Designing for Partial Failure: Why 'Everything is Highly Available' Is a Myth

Alok Ranjan Daftuar — Mon, 11 May 2026 08:41:55 +0000

Your system will fail. The question is whether it fails completely or gracefully — and that answer is decided at design time, not incident time.

High availability is not a property of your system — it is an emergent behavior of how your system handles the inevitable failure of its parts. A cluster of five-nines components can still produce a zero-nines system if you haven't designed for what happens when one of them degrades.

Partial Failure Is the Normal State

The CAP theorem is clean in theory. In production, it manifests as partial unavailability — some nodes respond, some don't, and your system has to decide what to do about it.

Real-world partitions are never clean:

A replica is reachable but 800ms slower than normal
A downstream service responds to health checks but times out on actual requests
A database primary is alive, but replication lag has grown to 45 seconds

Degraded systems are harder to reason about than failed ones. A service that returns errors is visible. A service that returns stale data silently is far more dangerous.

Practical rule: for every external dependency, explicitly answer — "What does my service do when this dependency is unavailable for 30 seconds? For 5 minutes? For 30 minutes?"

How Cascading Failures Actually Propagate

Cascading failures rarely start big. They start with one slow service and end with everything down.

The classic thread pool exhaustion cascade:

A payment service starts responding in 4s instead of 200ms
Threads pile up — connection pool goes from 20% to 80%
Latency bleeds upstream — unrelated operations slow down because they share the same thread pool
Connection pools hit their limit — new requests fail immediately
Health checks fail — load balancer removes the service from rotation

Your entire checkout is down because a payment service was slow — not even failed.

Cascade enablers to eliminate:

Synchronous chains longer than 2–3 hops
Shared thread pools across dependencies
Missing or oversized timeouts (a 30s default in a p99 200ms service is a loaded gun)
Retry storms without backoff
No bulkhead isolation between operations

Graceful Degradation Patterns

Circuit Breaker — monitors failure rate and opens the circuit above a threshold, immediately returning a fallback instead of attempting the call. States: Closed → Open → Half-Open.

Bulkhead Isolation — prevents one failing dependency from consuming all shared resources. Isolate thread/connection pools per downstream service. At the Kubernetes level, namespace ResourceQuotas enforce cluster-level isolation.

Timeout Hierarchy — every downstream call's timeout must be shorter than the upstream caller's timeout. A 5s payment timeout inside a 20s checkout timeout inside a 30s user request timeout.

Fallback Responses — return degraded but functional responses rather than errors. Pricing service down? Return last-cached price with a "price may vary" indicator. Feature flags unavailable? Return safe defaults.

Retry with Exponential Backoff + Jitter — retries without backoff amplify load on degraded services. Jitter prevents synchronized retry storms. Only retry on transient failures (5xx, timeouts) — never on 4xx.

Observability During Partial Failure

Graceful degradation can mask serious problems for hours. The four signals that matter most:

Circuit breaker state transitions — a circuit that opens at 2am with no alert is silent failure accumulating debt
Per-dependency error rates — aggregate 0.5% looks fine; per-dependency 40% on payment tells you exactly what's wrong
Queue depth and consumer lag — the leading indicator before errors surface at the API layer
Fallback invocation rate — 0.1% is noise; 15% is a dependency in chronic distress being silently masked

The Key Insight

Most architecture reviews start with the happy path. Flip that. Design your degraded states first — what does this system look like when the payment service is down? When a network partition isolates one AZ?

If you can answer with specific, tested, observable behaviors — you have a resilient system. If the answer is "it depends on what fails" — you have a system that will surprise you in production.

Partial failure is not an edge case. It is the normal operating condition of any distributed system at scale.

Read the Full Article

This is a summary of my deep dive into designing for partial failure. The full article covers each pattern with production implementation examples, real-world cascade case studies, and a complete resilience checklist:

👉 Designing for Partial Failure — Full Article

The full article includes:

CAP theorem applied to partial unavailability scenarios
Real-world cascade case studies (AWS us-east-1 2021, Facebook 2021)
Circuit breaker implementation in .NET (Polly) and Node.js (opossum)
Bulkhead isolation with Kubernetes ResourceQuotas
Timeout hierarchy design with named HttpClient factories
Stale cache fallback implementation with Redis
Retry with exponential backoff and jitter (TypeScript)
Structured logging patterns for degraded responses
Production resilience checklist

The CAP Theorem in Practice: Making the Right Trade-offs at Scale

Alok Ranjan Daftuar — Mon, 04 May 2026 05:59:51 +0000

Every distributed system you build is already taking a side in the CAP trade-off. The question is whether you made that choice deliberately or discover it during an incident.

CAP states that a distributed system can guarantee at most two of three properties: Consistency, Availability, and Partition Tolerance. The critical insight most teams miss — P is not optional. Networks fail. Pods crash. AZs go dark. You are choosing between CP and AP. Full stop.

CP vs AP: What You're Actually Trading

CP systems (etcd, ZooKeeper, CockroachDB) refuse to serve requests during a partition rather than return stale data. Leader-based consensus ensures correctness. Choose CP for financial ledgers, inventory reservation, distributed locks — any domain where stale reads are more dangerous than errors.

AP systems (Cassandra, DynamoDB, DNS) continue serving requests during a partition, accepting diverging state. Reconciliation happens later. Choose AP for user feeds, shopping carts, session data — any domain where temporary inconsistency is tolerable and availability is a hard SLA.

Neither is universally correct. What is unacceptable is having no defined behavior.

PACELC: The Model That Actually Matches Production

CAP only describes behavior during partitions. Your system spends most of its time healthy. PACELC extends CAP: even during normal operation, you are trading Latency against Consistency.

A CP system with synchronous replication pays a latency tax on every write — all the time, not just during incidents. DynamoDB offers eventual consistency by default (low latency) or strong consistency per read (higher latency). The trade-off is continuous, not just during failures.

Architectural Patterns Shaped by CAP

Saga Pattern — inherently AP. Each local transaction commits immediately (available). Global consistency is eventual. Compensating transactions are your consistency guarantee, not your database.

CQRS + Event Sourcing — assigns CP to commands (strong consistency via transactional aggregate root) and AP to queries (eventual consistency via denormalized projections). You are not picking one model — you are assigning different models per use case.

Tunable Consistency (Cassandra) — CONSISTENCY QUORUM on reads and writes achieves CP behavior. CONSISTENCY ONE maximizes AP. Tune per operation, not per cluster. User profile reads can tolerate eventual consistency. Payment status reads cannot.

Common Mistakes

Treating CAP as a database property — it is a system property. Your retry logic, caching, and timeout behavior all participate in the trade-off.
Assuming strong consistency is always safer — CP under partition returns errors. Cascading timeouts from a blocked write path can cause a larger outage than serving stale data.
One consistency model across the entire system — your order service (CP), product catalog (AP), session store (AP), and audit log (CP) should not share a single strategy.

Read the Full Article

This is a summary of my deep dive into CAP theorem trade-offs. The full article covers CP vs AP with canonical examples, the PACELC model, architectural patterns, and a decision framework:

👉 The CAP Theorem in Practice — Full Article

The full article includes:

Detailed CP vs AP comparison with canonical system examples
PACELC model with system-by-system partition and normal operation behavior
Saga and CQRS patterns analyzed through the CAP lens
Tunable consistency deep dive with Cassandra
Real-world decision framework for architecture reviews
Four common mistakes architects make with CAP trade-offs

BM25 vs. Vector Search: Choosing the Right Retrieval Strategy for Production Systems

Alok Ranjan Daftuar — Tue, 07 Apr 2026 05:53:00 +0000

Search is deceptively complex. You can stand up Elasticsearch in an afternoon and have something that works. Whether it surfaces the right document when a user asks "how do I reset my subscription?" instead of typing "subscription reset steps" is an entirely different problem.

The two dominant retrieval paradigms — BM25 and Vector Search — are both mature and production-proven. The real question is why one fails where the other succeeds, and how to combine them.

BM25: The Probabilistic Workhorse

BM25 scores documents using term frequency, inverse document frequency, and document length normalization. It is still the default ranking algorithm in Elasticsearch and OpenSearch.

Excels at: exact keyword matching (SKUs, error codes, CLI flags), transparent debuggable ranking, sub-millisecond latency at scale, zero GPU cost.

Breaks down at: vocabulary mismatch ("cancel membership" vs "terminate subscription"), semantic intent, cross-language queries, conceptual similarity.

BM25 is fundamentally a bag-of-words model. It has no understanding of meaning.

Vector Search: Semantic Retrieval via Embeddings

Vector search transforms text into dense numerical vectors where semantically similar content is geometrically close. Retrieval becomes nearest-neighbor search in that space.

Excels at: semantic equivalence, natural language questions, cross-lingual retrieval, RAG pipelines where LLMs generate natural language queries.

Breaks down at: exact term matching (NullPointerException, order IDs), infrastructure cost (GPU for embedding, RAM for HNSW indexes), staleness when documents change frequently.

The quality of vector search is entirely determined by your embedding model — domain fit, dimensionality, and max token length all matter.

Hybrid Search: The Production Reality

In most real-world systems, neither alone is sufficient. The answer is hybrid retrieval — running both in parallel and combining scores.

Reciprocal Rank Fusion (RRF) merges ranked lists without score normalization. Documents that rank well in both lists get a substantial boost. It is rank-based, not score-based — no need to normalize cosine similarity against BM25 scores.

The full pipeline:

BM25 + Vector Search in parallel → top-K candidates each
RRF merge → combined ranked list
Cross-encoder re-ranker → precision pass on top candidates
Final top-N → LLM context or search results

Skipping the re-ranker is a common mistake. Initial retrieval optimizes for recall. The re-ranker optimizes for precision — especially critical when context window limits mean you can only pass top-5 to an LLM.

Common Architecture Mistakes

Vector search alone for RAG — misses exact-match cases that BM25 catches, creating systematic blind spots
Ignoring chunk boundaries — embedding a 5000-token document as a single vector destroys retrieval specificity
General-purpose embedding model on specialized corpus — domain-specific models significantly outperform on legal, medical, or code retrieval
Not evaluating retrieval independently — teams blame the LLM when retrieval is the actual failure point. Measure Recall@K and MRR before debugging generation.

Read the Full Article

This is a summary of my deep dive into retrieval architecture. The full article covers BM25 mechanics, vector search internals, the complete tooling landscape, chunking strategies, and a decision framework:

👉 BM25 vs. Vector Search — Full Article

The full article includes:

BM25 scoring formula breakdown and tuning parameters
Embedding model evaluation criteria and provider comparison
Head-to-head scenario table showing when each wins
Hybrid architecture pattern with RRF and re-ranking
Complete tooling landscape (Pinecone, Qdrant, Weaviate, pgvector, Elasticsearch 8.x)
Chunking strategy deep dive (fixed, semantic, hierarchical)
Decision framework for choosing your retrieval architecture

Saga Orchestration vs. Choreography: Making the Right Trade-off in Event-Driven Systems

Alok Ranjan Daftuar — Fri, 27 Mar 2026 09:04:14 +0000

The saga pattern looks straightforward in diagrams. It becomes genuinely complex the moment you operate it in production.

The central question — orchestration or choreography — carries consequences that ripple through your codebase, your operational posture, and your team's cognitive load for years.

This is not a "use orchestration for complex sagas, choreography for simple ones" post. The real trade-offs are more specific.

The Baseline: What Both Approaches Must Solve

Before choosing an approach, every saga implementation must handle:

Atomicity at step boundaries — commit the database write and publish the event in the same transaction (transactional outbox or CDC)
Idempotent consumers — at-least-once delivery means your steps will be invoked more than once
Compensation correctness — compensating transactions are not rollbacks; they undo changes in a world that has moved on
Observability — correlation IDs, structured logging, and queryable saga state

These are table stakes, not optional concerns.

Orchestration: Central Control

A dedicated orchestrator drives the saga — it knows the sequence, issues commands, waits for responses, and drives compensation.

Shines when:

Workflows have complex conditional branching
Long-running sagas involve human steps or wait states
Operational visibility and debugging matter most
Compensation must be guaranteed and sequenced

Breaks down when:

The orchestrator becomes a throughput bottleneck
Tight temporal coupling conflicts with event-driven decoupling goals
Business logic gravitates into the orchestrator (god-object risk)

Choreography: Decentralized Reactions

No central coordinator. Each service listens for events, performs its local transaction, and publishes events that others react to. The saga is an emergent property.

Shines when:

Services are genuinely independent
Throughput is high and latency requirements are strict
The workflow is stable and simple
Independent deployability is valued over centralized visibility

Breaks down when:

Saga state is implicit and debugging requires forensic log analysis
Business logic is distributed across every participating service
Compensation failures go undetected — no component knows a step was missed

Failure Modes That Catch Teams Off Guard

Lost compensation events — a compensating transaction fails, lands in a DLQ, and the system stays inconsistent until someone investigates.

Pivot transaction ambiguity — misidentifying the point of no return leads to compensating steps that cannot actually be reversed.

Saga timeouts and orphaned state — sagas that time out without completing compensation leave the system in a partially-applied state.

Event schema evolution — a schema change breaks consumers silently, causing sagas to process with incorrect data.

Making the Decision

Most large systems use both — choreography for high-throughput, loosely-coupled flows; orchestration for complex, stateful, business-critical workflows.

The key insight: neither approach eliminates the need for idempotent consumers, transactional outboxes, schema governance, DLQ monitoring, or explicit compensation design. The approach determines where control and visibility live — not whether your system is correct.

Get the baseline right. Then choose the approach that fits your operational context — not the one that looked better in the last conference talk you attended.

Read the Full Article

This is a summary of my deep dive into saga patterns. The full article covers orchestration and choreography in detail with production failure scenarios, compensation strategies, and decision frameworks:

👉 Saga Orchestration vs. Choreography — Full Article

The full article includes:

Detailed comparison of both approaches with subsections on how they work, where they shine, and where they break down
Four critical failure modes that affect both approaches
Practical decision heuristics for choosing the right approach
Baseline requirements every saga implementation must handle