DEV Community: ameya joshi

Can we have consistency, availability, AND partition tolerance ?

ameya joshi — Sun, 07 Jun 2026 12:05:42 +0000

Everyone in a distributed systems interview can recite CAP theorem like a reflex. "You can only pick two." Head nods all around. The conversation moves on.

But here's the thing — I've been building distributed infrastructure for over a decade. Kubernetes operators, multi-region control planes, private cloud deployments that bridge on-prem racks to cloud APIs. And after enough production incidents, I started asking a different question.

What if everyone is just wrong about CAP? What if you CAN have all three?

Let me try to prove it. And then watch it fall apart. In an ideal world, a CP and an AP system would look the same if there are no network partitions.

The Math That Makes You Confident

Before I play devil's advocate, let me establish what makes the "we have consistency" argument so seductive.

The formula: W + R > N.

You have N replicas. A write acknowledges W of them. A read queries R of them. If W + R exceeds N, the write set and read set must overlap by the pigeonhole principle — there aren't enough nodes to keep them apart. That shared node already absorbed the write, so the read will find it.

With N=3, W=2, R=2: overlap = W + R - N = 1. Exactly one node is guaranteed to carry the fresh value. That's all you need.

One thing most people gloss over: the overlap puts the fresh value somewhere in your R replies. It doesn't pick it for you. You still need resolution logic — compare timestamps, apply vector clocks, pick last-write-wins. If your read just returns the first reply or trusts a majority count, the math bought you nothing.

And tunable consistency? It's not a different guarantee — it's just latency redistribution:

Config	Trade-off
W=1, R=N	Cheap writes, expensive reads
W=N, R=1	Expensive writes, cheap reads
W=2, R=2, N=3	Balanced

Same consistency ceiling across all three. The W and R knobs only move where the cost lands. Still solid. So far, my attempt to break CAP has no ammunition.

My Argument: CAP Is Beatable

Here's the pitch I built for myself.

"What if I just add more servers? More copies of the data means more redundancy. Even if some servers go offline, enough will still be reachable to keep things consistent AND available. W + R > N still holds. Partitions handled. Everyone goes home happy."

It sounds airtight. So let me try to break it — and I'll use a bank to do it, because banks make this painfully concrete.

The Bank Branch Thought Experiment

Imagine you run a bank with 5 branches across a city. Your rule is simple: a transaction is only official once at least 3 branches have recorded it. This way, if a branch burns down or goes offline, the other 4 can still operate. You've built in redundancy. Feels bulletproof.

Now a storm hits and knocks out communication between two parts of the city. Your 5 branches split into two isolated groups with no way to talk to each other:

Group A (minority): Branch 1 and Branch 2 — only 2 branches
Group B (majority): Branch 3, Branch 4, and Branch 5 — 3 branches

Group B is fine. Three branches can still reach agreement among themselves. Business as usual on that side of town.

But a customer walks into Branch 1 — stuck in the isolated group — and wants to withdraw ₹50,000. Branch 1 can't reach the other branches to get the required 3-branch confirmation.

You now face a choice. And this is where CAP becomes real.

Option A — Turn them away

You tell the customer: "Sorry, our systems are down. We can't process this until we reconnect."

Your records stay accurate. Nobody ends up with a wrong balance. But Branch 1 and Branch 2 are effectively closed. You protected consistency — and sacrificed availability.

Option B — Serve them anyway

You process the withdrawal. Branch 1 records it locally and adds a note: "Sync this with the other branches once the connection is back."

The customer is happy. The branch is open. Availability preserved. ✅

But here's what just happened across town: someone at Branch 4 checks that same account. Branch 4 doesn't know about the withdrawal yet. It returns the old balance — before the ₹50,000 left. Two branches now have different answers for the same account.

You preserved availability — and sacrificed consistency.

There is no Option C.

No matter how many branches you open, the moment a communication breakdown splits them into groups that can't talk to each other, you face this exact same binary. You either close the cut-off branches (lose availability) or let them operate independently (lose consistency).

I added more servers thinking I could escape this. I didn't. I just delayed the moment the storm arrived.

The Sloppy Quorum Trap

To stay available through the partition, the system can't just refuse requests on the minority side. It uses hinted handoff: it parks the write on substitute nodes — nodes that are reachable but are not the intended replicas — with a sticky note: "Deliver this to Node1 and Node2 once they reconnect."

The write returns W acknowledgments. ✅ Success, from the client's perspective.

But those acknowledgments came from the wrong nodes — not the real quorum members.

A read comes in. It hits the real replicas. Those replicas have no record of the write. They return stale data, with complete confidence, and no indication that anything is wrong. The W + R > N formula is technically satisfied — the numbers add up — but the sets are no longer intersecting on the nodes you assumed they were. The shared node in the overlap is a ghost.

The math didn't lie. The assumption underneath it did.

Why Availability Always Wins the Partition Fight

Here's the precise statement of why you cannot keep all three:

Partition tolerance is not optional. Networks partition. Nodes crash. Packets get dropped inside data centers, not just across regions. If you run anything at meaningful scale, a partition is not a hypothetical — it's a calendar event you haven't received yet.

Given that P is non-negotiable, your real choice resolves to:

CP — Reject requests when quorum is unreachable. Clients see errors or timeouts. Consistency is preserved. Availability is sacrificed.
AP — Serve requests always. Writes land on hinted nodes. Stale reads happen. Consistency is sacrificed. Availability is preserved.

And this is the part no one says plainly in interviews: the moment you choose to stay available during a partition, you have already abandoned strong consistency — even if W + R > N is still satisfied on paper.

Systems like Cassandra and DynamoDB make this honest. They're AP by design. Sloppy quorums, eventual consistency, read-repair, anti-entropy — all explicit mechanisms for healing state after divergence. They're not broken. They're correctly prioritizing availability and accepting the consistency cost.

Systems like etcd and ZooKeeper go the other way. They'll tell you "I don't know" rather than lie to you. Minority partitions get no writes. That's CP.

Both are correct choices for their use cases. What's not a valid choice is claiming you have all three.

What I Actually Learned

I came into this exercise wanting to find the edge case that breaks CAP. I left with something more useful: a clearer intuition for where exactly the guarantee breaks.

W + R > N is a strong and correct guarantee — conditional on one assumption that the formula itself doesn't state: every write lands on the real replicas in the real quorum. The moment a partition forces you to route around that assumption to stay available, the guarantee is void.

The candidates who can draw this line in a system design interview aren't just reciting theory. They've internalized that distributed systems don't fail loudly. The cluster stays "up." The writes return success. The formula adds up. And the stale data quietly becomes someone else's incident at 2am.

The CAP theorem doesn't tell you what to build. It tells you what question you're actually answering when you make your availability decision under partition. Answer it deliberately.

I work on distributed Kubernetes infrastructure and private cloud systems. Most of what I understand about distributed systems came from things that broke in ways the math didn't predict.

Building a LangGraph RAG Agent from Scratch — with a Live UI That Shows Every Step

ameya joshi — Sat, 06 Jun 2026 20:06:02 +0000

I built a learning project that teaches LangChain and LangGraph step by step — starting from a raw LLM call and ending with a full ReAct agent backed by RAG, streamed over SSE to a React UI that visualises every node in the agent loop in real time.

This post walks through the whole thing: what each concept does, how it connects to the next, and how the live pipeline view works.

What we're building

frontend/   ← React + Vite chat UI (live agent loop visualisation)
backend/    ← FastAPI server wrapping the RAG agent
step*.py    ← 6 progressive learning files

The agent answers questions about rate limiting algorithms. That's just the domain — the real goal is to understand how LangChain and LangGraph fit together.

The 6-step learning path

File	Concept introduced
`step1_llm_basics.py`	Chat models, messages, `.invoke()`, statelessness
`step2_prompts_and_chains.py`	Prompt templates, LCEL `\
{% raw %}`step3_tools.py`	`@tool` decorator, `bind_tools()`, manual tool loop
`step4_langgraph_intro.py`	`StateGraph`, nodes, edges, conditional routing
`step5_full_agent.py`	Full ReAct loop with `ToolNode`
`step6_rag_agent.py`	RAG — FAISS, HuggingFace embeddings, retriever tool

Step 1 — Raw LLM call

The simplest possible thing: call a model and read the reply.

from langchain_groq import ChatGroq
from langchain_core.messages import SystemMessage, HumanMessage

llm = ChatGroq(model="llama-3.3-70b-versatile")

messages = [
    SystemMessage(content="You are a rate limiting expert."),
    HumanMessage(content="What is token bucket?"),
]

response = llm.invoke(messages)
print(response.content)

Key insight: The LLM is stateless. Every call is independent. You manage the conversation history yourself by passing the full message list each time.

Step 2 — Prompt templates and LCEL chains

LangChain Expression Language (LCEL) lets you compose components with the | pipe operator — the same way Unix pipes work.

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a rate limiting expert."),
    ("human", "{question}"),
])

# Chain: prompt → LLM
chain = prompt | llm

# Invoke
response = chain.invoke({"question": "Compare token bucket and leaky bucket"})

# Stream tokens as they arrive
for chunk in chain.stream({"question": "What is sliding window log?"}):
    print(chunk.content, end="", flush=True)

Key insight: LCEL chains are lazy. .stream() and .batch() are first-class — no extra code needed.

Step 3 — Tools and tool calling

Tools let the LLM take actions. The @tool decorator turns a Python function into something the model can call.

from langchain_core.tools import tool
from langchain_groq import ChatGroq

@tool
def get_algorithm_info(algorithm: str) -> str:
    """Return a brief description of a rate limiting algorithm."""
    descriptions = {
        "token_bucket":    "Tokens refill at a fixed rate up to a capacity cap. Allows bursts.",
        "fixed_window":    "Counts requests in fixed time windows. Simple but has boundary spikes.",
        "sliding_window":  "Precise per-request log. High memory, no boundary spikes.",
        "leaky_bucket":    "Queue drains at a constant rate. Smooths traffic, no bursts allowed.",
    }
    return descriptions.get(algorithm, "Unknown algorithm.")

# Bind tools to the model — it now knows what tools exist and their signatures
llm_with_tools = ChatGroq(model="meta-llama/llama-4-scout-17b-16e-instruct").bind_tools(
    [get_algorithm_info]
)

response = llm_with_tools.invoke("Tell me about token bucket")
# response.tool_calls → [{"name": "get_algorithm_info", "args": {"algorithm": "token_bucket"}}]

Key insight: bind_tools() sends the tool schemas to the model. The model returns a structured tool_calls list — it does not execute the tools itself. You run them and send the results back.

Step 4 — LangGraph basics

LangGraph models the agent as a state machine. You define:

State — a typed dict that flows through the graph
Nodes — Python functions that receive state and return updates
Edges — connections between nodes (including conditional branches)

from langgraph.graph import StateGraph, END
from langgraph.graph.message import add_messages
from typing import Annotated
from typing_extensions import TypedDict

class State(TypedDict):
    messages: Annotated[list, add_messages]  # reducer: appends, never replaces

def node_a(state: State):
    return {"messages": ["Hello from node A"]}

def node_b(state: State):
    return {"messages": ["Hello from node B"]}

def route(state: State):
    return "b" if len(state["messages"]) < 3 else END

graph = StateGraph(State)
graph.add_node("a", node_a)
graph.add_node("b", node_b)
graph.set_entry_point("a")
graph.add_conditional_edges("a", route, {"b": "b", END: END})
graph.add_edge("b", "a")

app = graph.compile()

Key insight: add_messages is a reducer. When a node returns {"messages": [new_msg]}, LangGraph appends it to the list instead of replacing it. This is how the conversation history accumulates automatically.

Step 5 — Full ReAct agent

The ReAct pattern (Reason + Act) is: LLM decides what to do → tools execute it → LLM sees the result → repeat.

LangGraph's ToolNode handles the execution side automatically.

from langgraph.graph import StateGraph, END
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode
from langchain_groq import ChatGroq
from langchain_core.messages import HumanMessage
from typing import Annotated
from typing_extensions import TypedDict

tools = [get_algorithm_info, recommend_algorithm, calculate_token_bucket]
llm   = ChatGroq(model="meta-llama/llama-4-scout-17b-16e-instruct").bind_tools(tools)

class State(TypedDict):
    messages: Annotated[list, add_messages]

def llm_node(state: State):
    return {"messages": [llm.invoke(state["messages"])]}

def tools_condition(state: State):
    return "tools" if state["messages"][-1].tool_calls else END

graph = StateGraph(State)
graph.add_node("llm",   llm_node)
graph.add_node("tools", ToolNode(tools))
graph.set_entry_point("llm")
graph.add_conditional_edges("llm", tools_condition)
graph.add_edge("tools", "llm")  # always loop back after tool execution

agent = graph.compile()

result = agent.invoke({"messages": [HumanMessage(content="What algorithm for bursty traffic?")]})
print(result["messages"][-1].content)

The loop:

START → [llm] → has tool_calls? → YES → [tools] → back to [llm]
                                → NO  → END

Step 6 — Adding RAG

Retrieval-Augmented Generation (RAG) gives the agent long-form knowledge from documents. We embed documents into a FAISS vector store and expose it as a tool.

from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.tools import tool

# Index documents once at startup
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

docs = load_knowledge_base()           # returns list of Document objects
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

vectorstore = FAISS.from_documents(chunks, embeddings)
retriever   = vectorstore.as_retriever(search_kwargs={"k": 3})

# Expose retrieval as a tool
@tool
def search_knowledge_base(query: str) -> str:
    """Search the rate limiting knowledge base for relevant information."""
    docs = retriever.invoke(query)
    return "\n---\n".join(d.page_content for d in docs)

Key insight: RAG is just a tool from the agent's perspective. The LLM decides when to call it based on the question. The retriever converts the query to an embedding, finds the nearest chunks in FAISS, and returns them as context.

The FastAPI backend — SSE streaming

The backend wraps the agent in a FastAPI server. The interesting part is the streaming endpoint, which uses agent.astream_events() — a granular async generator that fires events for every internal state change in the graph.

from fastapi.responses import StreamingResponse
from langchain_core.messages import HumanMessage
import json

@app.post("/chat/stream")
async def chat_stream(request: ChatRequest):
    async def generate():
        llm_call_count = 0
        graph_started  = False

        async for event in agent.astream_events(
            {"messages": [HumanMessage(content=request.message)]},
            version="v2",
        ):
            kind = event["event"]
            node = event.get("metadata", {}).get("langgraph_node", "")

            # LLM node starting
            if kind == "on_chat_model_start" and node == "llm":
                if not graph_started:
                    graph_started = True
                    yield sse({"type": "pipeline", "phase": "graph_start"})
                llm_call_count += 1
                yield sse({"type": "pipeline", "phase": "llm_start", "call": llm_call_count})

            # LLM done — emit routing decision
            elif kind == "on_chat_model_end" and node == "llm":
                output     = event["data"].get("output")
                tool_calls = getattr(output, "tool_calls", []) if output else []
                yield sse({
                    "type":       "pipeline",
                    "phase":      "llm_end",
                    "decision":   "tools" if tool_calls else "answer",
                    "tool_names": [tc["name"] for tc in tool_calls],
                })

            # Tool executing
            elif kind == "on_tool_start":
                yield sse({"type": "pipeline", "phase": "tool_start",
                           "tool": event["name"], "args": event["data"].get("input", {})})

            # Tool done
            elif kind == "on_tool_end":
                out     = event["data"].get("output", "")
                content = out.content if hasattr(out, "content") else str(out)
                yield sse({"type": "pipeline", "phase": "tool_end",
                           "tool": event["name"], "preview": content[:120]})

            # Individual LLM output tokens (final answer only)
            elif kind == "on_chat_model_stream" and node == "llm":
                chunk = event["data"]["chunk"]
                if chunk.content and not getattr(chunk, "tool_call_chunks", []):
                    yield sse({"type": "token", "content": chunk.content})

        yield sse({"type": "pipeline", "phase": "graph_end"})
        yield "data: [DONE]\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

Why astream_events instead of astream?

astream() gives you one event per node that completes — coarse-grained. astream_events(version="v2") fires for every internal lifecycle hook: model start/stream/end, tool start/end, chain start/end. This is what lets us show individual tokens and the routing decision in real time.

The React UI — live agent loop visualisation

Every assistant response shows a collapsible Agent Loop panel. Each node card appears and updates live as the corresponding event arrives from the SSE stream.

🚀 StateGraph Initialized          [langgraph]
   StateGraph.compile() · add_messages reducer
   ↓
🧠 LLM Node — Call #1  ⟳           [langchain]   ← spinning while active
   ChatGroq(llama-4-scout) · bind_tools(4)
   AIMessage has tool_calls → selected: search_knowledge_base
   ↓
◆  Conditional Edge → tools node   [langgraph]
   add_conditional_edges · tools_condition(state)
   has tool_calls → route to tools
   ↓
🔍 ToolNode: search_knowledge_base ⟳ [langchain]
   FAISS vector search · HuggingFace embeddings
   query: HTTP headers rate limiting
   → Retrieved 3 relevant chunk(s)
   ↓
🧠 LLM Node — Call #2  ✓           [langchain]
   LLM sees ToolMessage in state
   no tool_calls → generating final answer
   ↓
◆  Conditional Edge → END          [langgraph]
   no tool_calls → route to END
   ↓
🏁 Graph END                       [langgraph]
   messages[-1].content → response

Nodes are colour-coded:

Blue border + spinner = currently active
Dark green border + ✓ = completed
Dashed border = routing / conditional edge

Badges identify which framework is responsible: langgraph (purple) vs langchain (orange).

Typewriter effect

Tokens from the LLM arrive in bursts over SSE. Rather than applying them immediately, a character queue drains at a fixed pace (18ms/char) so the text types out at a readable speed:

const CHAR_DELAY = 18  // ms per character

// When a token event arrives, push each character into the queue
if (ev.type === 'token') {
  tokenQueue.current.push(...ev.content.split(''))
  startTicker(assistantId)
}

// Ticker drains one char at a time
const startTicker = (id) => {
  tickerRef.current = setInterval(() => {
    if (!tokenQueue.current.length) return
    const ch = tokenQueue.current.shift()
    setMessages(prev => prev.map(m =>
      m.id === id ? { ...m, content: (m.content || '') + ch } : m
    ))
  }, CHAR_DELAY)
}

Stack

Layer	Technology
LLM	Groq — `llama-4-scout-17b` (tool calling), `llama-3.3-70b` (text)
Agent framework	LangGraph — `StateGraph`, `ToolNode`, `add_conditional_edges`
RAG	LangChain + HuggingFace `all-MiniLM-L6-v2` embeddings + FAISS
Streaming	`astream_events(version="v2")` → Server-Sent Events
Backend	FastAPI + uvicorn
Frontend	React 18 + Vite + react-markdown

Running it

# Python deps (uses uv to avoid system Python issues)
uv venv .venv --python 3.12
uv pip install -r requirements.txt

# Frontend deps
cd frontend && npm install && cd ..

# Terminal 1 — backend
cd backend
GROQ_API_KEY=your_key uvicorn main:app --port 8000 --reload

# Terminal 2 — frontend
cd frontend && npm run dev

Open http://localhost:5173. The first run downloads the embedding model (~90 MB) and caches it.

What I learned

LangChain gives you the building blocks: models, prompt templates, tools, LCEL chains, vector stores.

LangGraph gives you the control flow: a state machine where you decide the loop, the branching, and when to stop.

The two fit together naturally — LangGraph nodes call LangChain components, and LangChain tools feed results back into LangGraph state via add_messages.

The most clarifying thing was building the UI that shows the loop. When you watch the graph execute in real time — LLM node lights up, routing decision fires, ToolNode spins, LLM node fires again — the ReAct pattern stops being abstract and becomes something you can see.

The full source is on GitHub. The step.py files are designed to be read in order — each one is self-contained and introduces exactly one new concept.*