This guide assumes you know nothing about LLMs beyond "I can chat with ChatGPT/Claude." Every concept is introduced with a real-world analogy first, then explained technically, then backed by runnable code.
Table of Contents
- Tool Use Fundamentals
- Tool Schema Design
- Parallel Tool Calls and Failure Handling
- Hybrid Search (Dense + Sparse Retrieval)
- Metadata Filtering
- Reranking: Cross-Encoders vs Bi-Encoders
- Query Rewriting with HyDE
- Semantic Caching
- Full System Design: RAG with 10M Documents, Zero Hallucinations
- Appendix A: Streaming Tool Calls
- Appendix B: Evaluating RAG with RAGAS
- Glossary
- "Why This Matters in Production" Summary
1. Tool Use Fundamentals
1.1 The simple example first
Imagine you ask a friend, "Hey, what's the weather in Tokyo right now?" Your friend doesn't know this off the top of their head — nobody just knows live weather. So your friend pulls out their phone, opens a weather app, types "Tokyo," reads the result, and tells you: "It's 24°C and cloudy."
Notice what happened: your friend (the "brain" doing the reasoning) never became a weather sensor. They recognized they needed external, live information, used a tool (the weather app) to fetch it, and then used that fetched information to answer you in natural language.
This is exactly what "tool use" (also called "function calling") means for an LLM like Claude or GPT. The model is the friend. It's great at reasoning and language, but it has no live connection to today's weather, your company's database, or a calculator that never makes arithmetic mistakes. So we give it a list of "apps" (tools) it's allowed to ask for, and something else — your application code — actually goes and uses those apps on its behalf.
1.2 Technical explanation
Tool use extends a model's capabilities to call "external systems" — anything outside the model's own weights and training data. Examples from the notes:
- Given a city, get the current temperature.
- Given a math expression, solve it precisely.
- Run a bash command.
These are all things a language model is fundamentally bad at doing internally (it can't refresh live weather data, and it's notoriously unreliable at exact arithmetic), so instead of asking the model to guess, we let it delegate to a real, deterministic function.
How it's wired up:
- You provide the model with a list of tools (each tool = a function name + a description + a schema of its parameters) alongside your prompt.
- The model reads your message and decides, based on the user's request, whether an external call is needed, and if so, which tool and with what arguments.
- This decision is called a tool_use request — the model doesn't execute anything, it just emits a structured message saying "I'd like to call
get_weatherwith{"city": "Tokyo"}."
The single most important sentence in this whole section, straight from the notes:
"The model never calls a function itself!"
The model's role is reasoning and argument construction only. It cannot reach out to the internet, touch your database, or execute code. Everything it does is emit text/JSON. Your application is the one with hands.
The full lifecycle (the loop), step by step:
- Tools are defined as a list of functions + metadata (name, description, parameter schema). This is sent to the model as part of the conversation (typically bundled with — or alongside — the user's message).
- The model reads the message and decides: is an external call needed? If yes, which tool, and with what arguments?
- Application intercepts the response. Your code (not the model) receives this tool_use request.
-
Application invokes the function. Your code actually runs
get_weather("Tokyo")for real — hits a weather API, database, calculator, whatever the tool represents. - Application sends the result back to the model as a "tool-result message."
- Model proceeds — now that it has the real data, it continues generating its final natural-language answer to the user.
Along the way your application also has to handle practical realities like rate limits on the tools you're calling (e.g., a weather API only allows so many requests per minute) — that's your application's job too, not the model's.
1.3 Code: the full request/response loop
Below is a minimal, close-to-runnable example using the Anthropic SDK style referenced in the notes. It defines a get_weather tool, sends a prompt, receives a tool_use block, executes a mock function (standing in for a real weather API), and sends the result back so the model can finish its answer.
# pip install anthropic --break-system-packages
import anthropic
import json
# Step 0: create a client (assumes ANTHROPIC_API_KEY is set as an environment variable)
client = anthropic.Anthropic()
# ---------------------------------------------------------------------------
# Step 1: Define the tool as "a function + metadata"
# This is NOT the real function. It's a JSON description that tells the model
# "this function exists, here's its name, what it does, and what arguments it needs."
# ---------------------------------------------------------------------------
tools = [
{
"name": "get_weather",
"description": "Get the current weather for a given city.",
"input_schema": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name, e.g. Tokyo"},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit to return",
},
},
"required": ["city"], # unit is optional
},
}
]
# ---------------------------------------------------------------------------
# Step 2: This is the REAL function that will actually run.
# In production this would call a weather API (e.g. OpenWeatherMap).
# Here we mock it so the example is runnable without external dependencies.
# ---------------------------------------------------------------------------
def get_weather(city: str, unit: str = "celsius") -> dict:
"""Pretend weather lookup — replace with a real HTTP call in production."""
fake_database = {"Tokyo": 24, "London": 14, "Delhi": 33}
temp_c = fake_database.get(city, 20)
temp = temp_c if unit == "celsius" else (temp_c * 9 / 5) + 32
return {"city": city, "temperature": temp, "unit": unit}
# ---------------------------------------------------------------------------
# Step 3: Send the user's question + the tool list to the model.
# The model will read the question and DECIDE if it needs the tool.
# ---------------------------------------------------------------------------
messages = [{"role": "user", "content": "What is the weather in Tokyo?"}]
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
tools=tools,
messages=messages,
)
# ---------------------------------------------------------------------------
# Step 4: "Application intercepts the response."
# We look through the model's response blocks. If it contains a "tool_use"
# block, that means the model wants us to run a function on its behalf.
# The model itself did NOT run anything — it only asked.
# ---------------------------------------------------------------------------
tool_use_block = None
for block in response.content:
if block.type == "tool_use":
tool_use_block = block
break
if tool_use_block:
print(f"Model wants to call: {tool_use_block.name} with {tool_use_block.input}")
# Step 5: "Application invokes the function." We actually run the real code.
if tool_use_block.name == "get_weather":
result = get_weather(**tool_use_block.input)
else:
result = {"error": "unknown tool"}
# Step 6: "Application sends the result back to model" as a tool_result message.
# We must echo back the ORIGINAL assistant message (containing the tool_use)
# plus a new "user" message containing the tool_result, so the model has
# full context of what it asked for and what it got back.
messages.append({"role": "assistant", "content": response.content})
messages.append(
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use_block.id,
"content": json.dumps(result),
}
],
}
)
# Step 7: "Model proceeds..." — call the model again so it can turn the
# raw tool result into a natural-language final answer.
final_response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
tools=tools,
messages=messages,
)
for block in final_response.content:
if block.type == "text":
print("Final answer:", block.text)
else:
# The model answered directly without needing a tool.
for block in response.content:
if block.type == "text":
print("Final answer:", block.text)
Notice the loop shape: model decides → app intercepts → app invokes → app returns result → model proceeds. That five-step handoff is the entire mental model for tool use, and it never changes no matter how complex the tool is.
2. Tool Schema Design
2.1 The simple example first
Imagine you're hiring a new employee and you hand them a one-line job description: "Search for products." That's it — no other instructions.
On day one, a customer asks them "how much does this cost?" and the new employee, trying to be helpful, uses the "search for products" tool because it's the only tool they were told about, even though there's actually a dedicated pricing tool sitting right next to it that they didn't know they should prefer. They also aren't told whether "search" means searching by name, by category, or by a product code — so they guess, and sometimes guess wrong.
Now imagine instead you hand them a proper SOP: "Search for products by name, category, or SKU. Use this only when the customer is explicitly trying to find or filter catalog items. Do NOT use this for pricing questions or stock checks — those have their own dedicated tools." Now the employee has almost no room to misinterpret their job.
This is the difference between a bad tool schema and a good one. The LLM is that new employee, and the schema is the only "training" it gets before being thrown into the job.
2.2 Technical explanation
From the notes: "JSON schema is [the] contract between your intent and the model's behaviour." And critically: "Ambiguous schema is [the] most common root cause of tool failure and is 'invisible' until production." It's invisible because in your own testing you tend to ask clean, obvious questions — production users ask messy, ambiguous, unexpected ones, and that's where a vague schema quietly breaks.
There are three rules for designing a good tool schema:
Rule 1 — Specify semantics, not just a description.
Don't just say what the function is named or roughly does. Say when to use it, when not to use it, and what it should never be confused with. The notes' worked example:
- ❌ Bad:
"search for products" - ✅ Good:
"Search for products by name, category, or SKU. Use only when user is explicitly looking to find or filter catalog items. Do not use for pricing questions or stock checks — those have other dedicated tools."
Rule 2 — Be deliberate about required vs. optional parameters.
If a parameter is optional and you don't clearly explain what happens when it's omitted, the model may hallucinate a value for it just to fill the slot (marked with a red ✗ in the notes). Conversely, if a parameter is marked mandatory but the model doesn't have a value for it, it may pass nothing at all or an invalid placeholder (also marked ✗). The fix from the notes: provide default values for the model to use so there's no ambiguity about what an "empty" optional parameter should look like.
Rule 3 — Enums are the cheapest reliability primitive you have.
Whenever a parameter has a fixed, known set of valid values (a category, a unit, a status), constrain it with an enum instead of leaving it as a free-form string. This "keeps things readable and reduces hallucination" — the model can only pick from the list you gave it, it can't invent "CategoryXYZ" out of thin air.
The "too many tools" problem, and its fix — dynamic tool loading.
The notes flag: "large number of tools: model gets confused" — when a model is handed 50+ tools at once, two hard questions creep in: "What category tool?" and "Which specific tool?" The model starts guessing between similar-looking tools. The fix is dynamic tool loading: instead of always sending the model your entire tool catalog, your application first figures out (from context, intent classification, or the conversation's domain) which subset of tools is actually relevant, and only sends that subset to the model for this particular turn. Fewer, more relevant choices means fewer wrong guesses.
2.3 Code: bad schema vs. good schema, side by side
# =============================================================================
# BAD SCHEMA
# Vague description, no enum for category, ambiguous optional parameter.
# =============================================================================
bad_tool = {
"name": "search_products",
"description": "Search for products", # <-- too vague: search how? for what purpose?
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"category": {"type": "string"}, # <-- free text: model may invent categories
"include_out_of_stock": {"type": "boolean"}, # <-- optional, no default explained
},
"required": ["query"],
},
}
# Problems this causes in production:
# 1. Model may call this tool for pricing questions, since nothing tells it not to.
# 2. Model may pass category="Electronics & Gadgets" when your real categories
# are ["electronics", "apparel", "home"] -> zero results, silent failure.
# 3. Model may omit include_out_of_stock, or guess True/False randomly, since
# there's no guidance on what happens if it's left out.
# =============================================================================
# GOOD SCHEMA
# Specifies semantics (when to use / not use), enum for category,
# explicit default behaviour for optional params.
# =============================================================================
good_tool = {
"name": "search_products",
"description": (
"Search for products by name, category, or SKU. "
"Use only when the user is explicitly looking to find or filter catalog "
"items. Do NOT use this for pricing questions or stock-level checks — "
"those have their own dedicated tools (get_price, check_stock)."
),
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Free-text search term, e.g. a product name or SKU.",
},
"category": {
"type": "string",
# Enum = the model can ONLY pick from these exact values.
"enum": ["electronics", "apparel", "home", "books", "toys"],
"description": "Optional. Restrict results to one catalog category.",
},
"include_out_of_stock": {
"type": "boolean",
"description": (
"Whether to include out-of-stock items. "
"If omitted, DEFAULTS to false (only show in-stock items)."
),
},
},
"required": ["query"],
},
}
# The good schema tells the model: what it's for, what it's NOT for, exactly
# which category strings are valid, and exactly what "not specifying" means.
# That triple clarity is what the notes call the "contract" between your
# intent and the model's behaviour.
2.4 Code sketch: dynamic tool loading
# A tiny illustration of picking a relevant SUBSET of tools before calling
# the model, instead of always sending your entire tool catalog.
ALL_TOOLS = {
"search_products": {...}, # schema dicts omitted for brevity
"get_price": {...},
"check_stock": {...},
"get_weather": {...},
"run_bash_command": {...},
"search_hr_docs": {...},
}
# A simple keyword-based intent router. In production this is often a small
# classifier model or an embedding-similarity lookup, not just keywords —
# but the principle (narrow the toolset BEFORE calling the main model) is identical.
def select_relevant_tools(user_message: str) -> list[str]:
msg = user_message.lower()
if any(word in msg for word in ["price", "cost", "how much"]):
return ["get_price"]
if any(word in msg for word in ["stock", "available", "in stock"]):
return ["check_stock"]
if any(word in msg for word in ["find", "search", "looking for"]):
return ["search_products"]
if "weather" in msg:
return ["get_weather"]
# Fallback: give a small default set rather than everything
return ["search_products", "get_price"]
def build_tools_for_request(user_message: str) -> list[dict]:
relevant_names = select_relevant_tools(user_message)
return [ALL_TOOLS[name] for name in relevant_names if name in ALL_TOOLS]
# Usage: only the relevant tools go into the API call, not all 6+.
# tools_for_this_call = build_tools_for_request("How much does the red mug cost?")
# -> only sends the get_price tool, so the model can't confuse it with search_products.
3. Parallel Tool Calls and Failure Handling
3.1 The simple example first
Suppose you ask a travel-planning friend: "Can you check the weather in Paris, Tokyo, and New York for me?" A slow friend would check Paris, wait, tell you, then check Tokyo, wait, tell you, then check New York. A smart friend instead opens three browser tabs at once, checks all three simultaneously, and gives you all three answers together. Same total work, far less waiting.
Now imagine one of those three lookups fails — say the New York weather site is down. Does your friend refuse to tell you about Paris and Tokyo just because New York failed? Or do they hand you what they do have, and mention New York didn't work? That's a judgment call — and it's the same judgment call your application has to make about tool calls.
3.2 Technical explanation
Parallel tool calls: the notes describe a model that can emit multiple tool-call requests in a single turn — e.g., "compare weather across N cities" or a "trip plan" that needs both search_flight and search_hotel at once. Your orchestration layer (the application code, not the model) is responsible for "fanning out" these calls concurrently, collecting all the results, and sending them back together. The direct payoff: it improves overall time to completion — you pay for the slowest of the N calls, not the sum of all N. The notes' guidance: use this whenever the operations are independent of one another (one call's result doesn't depend on another's).
Failure handling — "it depends". The notes are refreshingly honest here: "Answer to any question could well be IT DEPENDS." There isn't one universal right answer; it's a judgment call based on your product's tolerance for incompleteness vs. correctness. Three named strategies:
- Prevent — stop the bad outcome before it happens. E.g., validate inputs up front, apply timeouts, use circuit breakers so a known-broken downstream tool isn't even attempted.
- Absorb — let the failure happen, but contain the blast radius: send back only the successful tool-call responses and quietly drop/flag the failed ones, so the model can still produce a partial, useful answer rather than crashing the whole turn.
- Fail gracefully — if absorbing isn't safe (e.g., you can't answer any meaningful part of the question without the missing piece), abort the entire execution and return a clear, honest error/refusal rather than a partial or misleading answer.
The notes give a concrete example of when parallel fan-out matters: "4 websearch calls, lookup across multiple knowledge bases." If a question needs facts from 4 different knowledge bases, firing all 4 lookups concurrently instead of one after another is the entire difference between a fast, responsive system and a sluggish one.
3.3 Code: concurrent tool calls with all three failure strategies
import asyncio
import random
# ---------------------------------------------------------------------------
# Mock tool: fetch weather for one city. Sometimes fails, to demonstrate
# failure handling (simulating a flaky downstream API).
# ---------------------------------------------------------------------------
async def fetch_weather(city: str) -> dict:
await asyncio.sleep(random.uniform(0.1, 0.5)) # simulate network latency
if city == "New York" and random.random() < 0.5:
raise ConnectionError(f"Weather API timed out for {city}")
return {"city": city, "temp_c": random.randint(10, 35)}
# =============================================================================
# STRATEGY 1: PREVENT
# Validate/guard BEFORE attempting the call, so we never even try a call
# we already know is unsafe (e.g. an unsupported city, a rate-limited tool).
# =============================================================================
SUPPORTED_CITIES = {"Paris", "Tokyo", "New York", "London"}
async def call_with_prevention(city: str) -> dict:
if city not in SUPPORTED_CITIES:
# We "prevent" the failure by never calling the API for a city we
# know isn't supported, instead of letting it fail downstream.
return {"city": city, "error": "unsupported_city", "skipped_call": True}
return await fetch_weather(city)
# =============================================================================
# STRATEGY 2: ABSORB
# Run all calls concurrently; if one fails, catch the exception, keep going,
# and return only the successful results (plus a note about what failed).
# =============================================================================
async def call_with_absorption(cities: list[str]) -> dict:
tasks = [fetch_weather(city) for city in cities]
# return_exceptions=True means a failed task returns its Exception object
# instead of crashing the whole asyncio.gather() call.
results = await asyncio.gather(*tasks, return_exceptions=True)
successes, failures = [], []
for city, result in zip(cities, results):
if isinstance(result, Exception):
failures.append({"city": city, "error": str(result)})
else:
successes.append(result)
return {"successful_results": successes, "failed_cities": failures}
# =============================================================================
# STRATEGY 3: FAIL GRACEFULLY
# If ANY call fails and a partial answer would be misleading (e.g. this tool
# call was mandatory context for the user's question), abort entirely and
# surface a clear, honest message instead of a silently incomplete answer.
# =============================================================================
async def call_with_graceful_failure(cities: list[str]) -> dict:
tasks = [fetch_weather(city) for city in cities]
try:
results = await asyncio.gather(*tasks) # no return_exceptions: first failure raises
return {"ok": True, "results": results}
except Exception as exc:
return {
"ok": False,
"refused": True,
"reason": f"Could not complete weather comparison: {exc}",
}
async def main():
cities = ["Paris", "Tokyo", "New York"]
print("PREVENT strategy (one unsupported city):")
print(await asyncio.gather(*[call_with_prevention(c) for c in ["Paris", "Atlantis"]]))
print("\nABSORB strategy (best-effort, partial results ok):")
print(await call_with_absorption(cities))
print("\nFAIL GRACEFULLY strategy (all-or-nothing):")
print(await call_with_graceful_failure(cities))
if __name__ == "__main__":
asyncio.run(main())
Which strategy to pick is a product decision, not a technical one — that's the "it depends" in the notes. A weather-comparison feature can tolerate "absorb" (2 out of 3 cities is still useful). A financial transaction pipeline probably needs "fail gracefully" (a half-completed transfer is dangerous, not just incomplete).
4. Hybrid Search (Dense + Sparse Retrieval)
4.1 The simple example first
Say you're searching your email for something. If you search "birthday party invite," a smart search that understands meaning (semantic search) will find an email titled "Come celebrate turning 30!" even though none of your exact words appear in it — because it understands the concept of a birthday party.
But now say you search for an exact error code, like 1099-MISC. A meaning-based search might get confused and return generic tax documents about "how do I reset my password" style topics — it doesn't realize you need an exact string match, not a conceptual one. A classic old-school keyword search (like Ctrl+F, but smarter) would nail this instantly because it's built for exact-term matching.
Neither approach alone is good at everything. The fix: run both kinds of search at the same time, and combine (merge and rerank) their results — a bit like asking two friends with different strengths for restaurant recommendations, and combining their two rankings into one better, more trustworthy ranking.
4.2 Technical explanation
Dense retrieval (semantic search): your query and your documents are both converted into vectors ("embeddings") that capture meaning. Two pieces of text that mean similar things end up as vectors that are close together in space, even if they don't share any exact words. This is great when there's genuine semantic similarity — the notes' example: "How do I reset my password?" naturally matches a document titled "Account Recovery Steps," even with zero overlapping words.
Sparse retrieval / BM25: this is the notes' name for traditional keyword-based search ("BM25" = "Best Matching 25," a well-known statistical formula scoring how well a document's exact terms match a query's exact terms, weighted by term rarity and frequency). BM25 is the opposite strength of dense search: it's excellent at exact terms, codes, IDs, acronyms — things like "error code 1099-MISC" — but it has no concept of meaning, so it's poor at "How do I reset my password?" style natural-language questions where the right document doesn't share your exact wording.
Hybrid search = dense + sparse, run together. The notes' rule of thumb: "20|50 documents certainly having the right answer" — hybrid retrieval improves recall (the fraction of genuinely relevant documents that get retrieved at all), because you're covering both failure modes (semantic gap AND exact-term gap) simultaneously. The fix line from the notes: "Run both and merge and rerank!"
Reciprocal Rank Fusion (RRF) is the merging technique. Analogy: imagine two friends each independently rank the same 10 restaurants from 1 (best) to 10 (worst). To get one combined ranking that reflects both opinions, you don't just average their scores (their internal scoring "scales" may be totally different: one friend rates out of 5 stars, another out of 100 points). Instead, RRF says: only the rank position matters, not the internal score — a restaurant ranked #1 by both friends should shoot to the top of the combined list, even if their raw scores were on incompatible scales. This "brings reciprocal ranks together" as the notes put it.
The formula, straight from the notes:
RRF(d) = Σ 1 / (k + rank_r(d))
r∈R
Read this piece by piece:
-
dis a particular document you're scoring. -
Ris the set of ranked result lists you're fusing (in our case: the dense-search ranking and the BM25 ranking). -
rank_r(d)is the position of documentdin rankingr(1st place, 2nd place, etc.). -
kis a small constant, usually 60 (per the notes) — it dampens the effect of very high ranks so a #1 spot doesn't dominate too extremely, and it also means documents that don't appear at all in one of the lists aren't catastrophically penalized. - You take
1 / (k + rank)for each list the document appears in, and sum those fractions across all lists. A document ranked highly (small rank number) in multiple lists gets a bigger combined score.
The notes' visual intuition: dense list gives fractions like 1/1, 1/2, ... and sparse list gives 1/61, 1/62, ... (i.e., 1/(60+1), 1/(60+2), ...) — "Fusion is better" because a document doesn't need to win outright in either individual list; it just needs to consistently rank reasonably well across both to accumulate a high combined score.
One more practical note from the notes: Weaviate and Elasticsearch (ES) support hybrid search natively — meaning they've built RRF-style fusion directly into the database. Otherwise, you have to do it yourself with a scatter-gather pattern (fire off both queries yourself, gather both result sets back in your application, then fuse them).
4.3 Code: RRF fusion from scratch (no external libraries)
# =============================================================================
# Reciprocal Rank Fusion (RRF) — implemented from the raw formula.
# RRF(d) = sum over each ranked list r containing d of 1 / (k + rank_r(d))
# =============================================================================
def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
"""
ranked_lists: a list of ranked result lists. Each inner list is already
sorted best-to-worst, e.g. ["doc3", "doc1", "doc9"].
k: the RRF damping constant (60 is the standard default from the notes).
Returns a list of (doc_id, fused_score) sorted from best to worst.
"""
scores: dict[str, float] = {}
for ranked_list in ranked_lists:
for position, doc_id in enumerate(ranked_list):
rank = position + 1 # ranks start at 1, not 0
# This is the exact formula: 1 / (k + rank)
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
# Sort documents by their fused score, highest first
fused_ranking = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
return fused_ranking
# --- Example usage ---------------------------------------------------------
# Dense (semantic) search results for "how do I reset my password?"
dense_results = ["account_recovery_steps", "login_faq", "security_settings", "billing_faq"]
# BM25 (keyword) search results for the SAME query — note it ranks things
# differently because it only sees exact word overlap
sparse_results = ["login_faq", "account_recovery_steps", "terms_of_service", "billing_faq"]
fused = reciprocal_rank_fusion([dense_results, sparse_results], k=60)
for doc_id, score in fused:
print(f"{doc_id}: {score:.5f}")
# Expected behaviour: "account_recovery_steps" and "login_faq" both rank
# highly in BOTH lists, so their fused scores rise above documents that only
# appeared in one list or ranked poorly in both.
4.4 Code: a BM25 call + a vector similarity call, combined
# This shows the SHAPE of a real hybrid search call: one sparse (BM25) query,
# one dense (vector) query, run independently, then fused with RRF.
# (rank_bm25 and sentence-transformers are common lightweight libraries.)
# pip install rank_bm25 sentence-transformers --break-system-packages
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
import numpy as np
documents = [
"To reset your password, go to Account Recovery and click 'Forgot Password'.",
"Error code 1099-MISC indicates a mismatch in the tax reporting form.",
"Our billing FAQ covers refunds, invoices, and subscription changes.",
]
doc_ids = ["account_recovery", "error_1099_misc", "billing_faq"]
query = "how do I reset my password"
# --- Sparse (BM25) leg: pure keyword/token overlap scoring -----------------
tokenized_corpus = [doc.lower().split() for doc in documents]
bm25 = BM25Okapi(tokenized_corpus)
bm25_scores = bm25.get_scores(query.lower().split())
sparse_ranking = [doc_ids[i] for i in np.argsort(bm25_scores)[::-1]]
# --- Dense (semantic) leg: embed query + docs, rank by cosine similarity ---
embedder = SentenceTransformer("all-MiniLM-L6-v2") # small, fast embedding model
doc_embeddings = embedder.encode(documents, normalize_embeddings=True)
query_embedding = embedder.encode(query, normalize_embeddings=True)
cosine_scores = doc_embeddings @ query_embedding # dot product of normalized vectors = cosine similarity
dense_ranking = [doc_ids[i] for i in np.argsort(cosine_scores)[::-1]]
print("Sparse (BM25) ranking:", sparse_ranking)
print("Dense (semantic) ranking:", dense_ranking)
# --- Fuse both rankings with RRF (reusing the function from section 4.3) ---
final_ranking = reciprocal_rank_fusion([dense_ranking, sparse_ranking], k=60)
print("Final hybrid ranking:", final_ranking)
5. Metadata Filtering
5.1 The simple example first
Imagine you're searching your email for a specific message and you know it arrived last week. You could (a) tell your email client "only show me emails from last week," and then search the text — that's fast and narrow. Or you could (b) search the text across your entire mailbox history, get back thousands of results, and manually throw away everything not from last week — that's slow and wasteful, since your search engine did a ton of unnecessary work scanning old emails it was always going to discard.
Now a second, more serious example: at a company, an HR employee should be able to see every department's HR documents, but a regular software engineer should only see documents relevant to their own team — they should never even see, let alone search inside, another department's confidential HR files. This isn't just a nice-to-have; it's a hard security requirement, often called RBAC (Role-Based Access Control — a plain-English one-liner: "restricting what a user can see or do based on their assigned role").
5.2 Technical explanation
Metadata filtering applies hard constraints (like date ranges, document owner, department, tags) either before or after the vector search runs, on top of whatever relevance ranking your retrieval produces. The notes' example query: "What was our Q1 2026 revenue?" — you'd want to filter to date range = Q1 2026 before even letting semantic search loose on the whole corpus.
Beyond just retrieval quality, the notes stress this is especially important for Enterprise RAG and RBAC: an employee simply must not be able to retrieve another department's HR docs, no matter how semantically relevant those docs might look to their query. This is a security boundary, not just a relevance tweak.
There are two places you can apply the filter:
- Pre-filter — you extract the filter conditions (date range, department, tags) and pass them into the vector search itself, so the search engine only ever looks at the allowed subset of documents. The notes call this faster, but note it requires that "the vector db should support metadata filter" natively (not every vector database can efficiently combine filtering with approximate nearest-neighbor search).
- Post-filter — you run the vector search across the entire corpus first, get back your top-K results, and then throw away any results that don't match your filter. The notes call this a waste of retrieval budget: you spent compute finding results you were always going to discard, and worse, you might discard so many that you end up with fewer than K valid results left over (imagine 8 of your top 10 semantic matches belong to a forbidden department — post-filtering leaves you with only 2 usable results).
5.3 Code: pre-filter (vector DB metadata filter) vs. post-filter (Python)
# =============================================================================
# PRE-FILTERING: the filter is passed INTO the vector database query itself.
# The DB never even considers documents outside the RBAC/date boundary, so
# ranking + filtering happen together, efficiently, in one pass.
# (Qdrant-style filter syntax, as referenced in the notes' prototype.)
# =============================================================================
# pip install qdrant-client --break-system-packages
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
client = QdrantClient(url="http://localhost:6333") # assumes a running Qdrant instance
query_vector = [0.12, 0.98, 0.31] # a real embedding would be much longer, e.g. 1536-dim
results = client.search(
collection_name="hr_documents",
query_vector=query_vector,
query_filter=Filter(
must=[
# RBAC constraint: only documents this user's department may see
FieldCondition(key="department", match=MatchValue(value="engineering")),
# Date constraint: only Q1 2026 documents
FieldCondition(key="date", range=Range(gte="2026-01-01", lte="2026-03-31")),
]
),
limit=10,
)
# Qdrant applies the filter WHILE searching, so all 10 results returned are
# guaranteed to satisfy both constraints — no wasted compute, no leakage risk.
# =============================================================================
# POST-FILTERING: search everything first, THEN discard invalid results in
# plain Python. Shown for contrast — this is what the notes call "wasteful."
# =============================================================================
def post_filter_search(all_search_results: list[dict], user_department: str) -> list[dict]:
"""
all_search_results: e.g. the top-K matches from an UNFILTERED vector search
across the entire corpus, regardless of department.
"""
filtered = [
doc for doc in all_search_results
if doc["department"] == user_department
and "2026-01-01" <= doc["date"] <= "2026-03-31"
]
return filtered
# Danger illustrated: if the unfiltered top-10 search returns mostly documents
# from OTHER departments (because they happened to be semantically similar),
# post-filtering might leave you with just 1-2 usable results, or worse —
# if the filtering logic has a bug, a forbidden document could slip through
# into the LLM's context before being filtered, which is a real security risk
# with RBAC-sensitive data. Pre-filtering avoids this entirely.
6. Reranking: Cross-Encoders vs. Bi-Encoders
6.1 The simple example first
Bi-encoder analogy: imagine two strangers each write their own short bio independently — "I like hiking, sci-fi movies, and cooking Italian food" and "I enjoy the outdoors, watching space films, and making pasta." You then compare these two bios after the fact to guess how compatible these two people might be. It's fast (you can pre-write and store millions of bios ahead of time and just compare later), but it's a bit shallow — the two people never actually talked to each other, so subtle nuances get missed.
Cross-encoder analogy: now imagine instead you sit those same two strangers down together for an actual conversation. They can react to each other in real time, clarify ambiguous statements, and pick up on nuance that never would have shown up in two separately-written bios. You get a much better read on true compatibility — but it only works one pair at a time, and a live conversation takes real time and effort per pair. You couldn't have every person in a stadium have a real conversation with every other person — it simply wouldn't scale.
6.2 Technical explanation
Bi-encoders (used for the first stage of dense retrieval, described back in Section 4): the query is turned into an embedding, and each document is turned into an embedding, completely independently of each other. Relevance is then just vector proximity — how close the two embeddings are (measured via cosine similarity). Because documents can be embedded once, offline, ahead of time, bi-encoders are fast at query time and scalable — this is why they're "good enough for most" use cases, per the notes.
Their key weakness, from the notes: "No cross attention, i.e. query and doc never see each other." Because the two embeddings were produced in isolation, the model can't attend to specific interactions between query terms and document terms. This makes bi-encoders weaker on negation, exact terms, etc. — e.g., a bi-encoder might not clearly distinguish "the flight is NOT delayed" from "the flight IS delayed," because both sentences produce very similar embeddings overall (they share almost all the same words).
Cross-encoders, by contrast, take the <query, document> pair as a single input — literally concatenating the query and the candidate document together — and output one single relevance score for that specific pair. Because "every token in the query attends every token of the document" (full cross-attention, across the whole concatenated input), the model can compare query and document together, catching subtleties bi-encoders miss. This: improves precision, and it genuinely sees the query and document together rather than as two isolated summaries.
The notes draw a small attention diagram illustrating the "lost in the middle" problem: attention/relevance tends to concentrate most heavily near the query token and decays as you move deeper into a long document — meaning information buried in the middle of a long document or a long context window can get comparatively under-weighted relative to information at the start or end. This is a known failure mode of transformer attention over long inputs, and it's part of why reranking with a small, focused candidate set (rather than dumping a giant unranked context at the LLM) genuinely helps.
Cross-encoders: much higher accuracy — they can compare every token in the query to every token in the document, so they handle negation, exact matches, etc. far better than bi-encoders. Example model named in the notes: cross-encoder/ms-marco-MiniLM-L-6-v2.
The catch — cross-encoders are slow and don't scale. From the notes: they work well on "50 doc[s] v[s] million doc[s] per query" — meaning cross-encoders are only practical on a small shortlist, never the entire corpus. There's no indexing possible (because the score depends on the specific query, you can't precompute it ahead of time the way you can with bi-encoder embeddings), and each pair costs roughly ~10ms of latency.
The latency math, worked out step by step (from the notes):
- Assume you have 1 million documents and it costs 20ms per
<query, doc>pair to score with a cross-encoder. - Total time for one query = 1,000,000 docs × 20ms = 20,000,000ms = 20,000 seconds.
- Converting to hours: 20,000 seconds ÷ 3,600 seconds/hour ≈ 5.5 hours per single query.
That's obviously unusable in a real product. The fix, stated directly in the notes: "could be done on candidate docs only." This is exactly why the two techniques are used together in a pipeline, not as alternatives: use the cheap, scalable bi-encoder (plus BM25/hybrid search) to narrow millions of documents down to a small shortlist (say, the top 40), and then run the expensive, high-accuracy cross-encoder reranker only on that small shortlist. You get bi-encoder speed at scale, and cross-encoder accuracy where it matters most — the final ranking of the few documents that actually make it into the LLM's context.
6.3 Code: bi-encoder vs. cross-encoder on the same toy dataset
# pip install sentence-transformers --break-system-packages
from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np
query = "Why is my API returning a 500 error under high load?"
candidate_docs = [
"Throughput degradation under load can trigger unhandled exceptions, "
"which surface to callers as HTTP 500 errors.",
"To reset your password, click 'Forgot Password' on the login screen.",
"500 errors are NOT related to client-side input validation issues.",
"Our billing FAQ covers refunds, invoices, and subscription cancellations.",
]
# =============================================================================
# STAGE 1 (fast, scalable): BI-ENCODER — embed query and docs SEPARATELY,
# then compare with cosine similarity. This is what you'd run across
# millions of documents to get an initial shortlist.
# =============================================================================
bi_encoder = SentenceTransformer("all-MiniLM-L6-v2")
query_vec = bi_encoder.encode(query, normalize_embeddings=True)
doc_vecs = bi_encoder.encode(candidate_docs, normalize_embeddings=True)
# cosine similarity = dot product of normalized vectors
bi_encoder_scores = doc_vecs @ query_vec
print("Bi-encoder ranking (fast, first-pass):")
for idx in np.argsort(bi_encoder_scores)[::-1]:
print(f" {bi_encoder_scores[idx]:.3f} {candidate_docs[idx][:60]}...")
# =============================================================================
# STAGE 2 (slow, accurate): CROSS-ENCODER — feed the query+doc TOGETHER as a
# single pair, get one relevance score per pair. Only run this on the small
# shortlist that stage 1 already narrowed down (here, all 4, since it's tiny).
# =============================================================================
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [[query, doc] for doc in candidate_docs] # literal <query, doc> pairing
cross_encoder_scores = cross_encoder.predict(pairs)
print("\nCross-encoder ranking (slow, high-precision rerank):")
for idx in np.argsort(cross_encoder_scores)[::-1]:
print(f" {cross_encoder_scores[idx]:.3f} {candidate_docs[idx][:60]}...")
# You'll typically notice the cross-encoder more confidently separates the
# TRUE match (the throughput/500-error doc) from the negation trap (doc #3,
# which mentions "500 errors" but explicitly says they are NOT related to
# the topic) — a distinction bi-encoders are known to struggle with.
7. Query Rewriting with HyDE
7.1 The simple example first
Imagine you ask a librarian, "Why is my code slow?" The librarian's catalog, though, doesn't use casual words like "slow" — the technical manuals on the shelf are indexed under formal terms like "throughput degradation," "latency regression," or "bottleneck analysis." If the librarian searches the catalog using your exact casual phrase, they might get nothing useful back, even though the perfect manual for your problem exists on the shelf. The words just don't line up.
A clever librarian's trick: before searching, they first jot down (in their head) roughly what the ideal, technical-sounding manual would probably say about your problem — then they search the catalog using that imagined, more formal description instead of your casual phrasing. That imagined "ideal answer" acts as a bridge between your everyday words and the library's formal vocabulary.
That's exactly what HyDE (Hypothetical Document Embedding) does for a RAG system.
7.2 Technical explanation
The notes call the "Query-Document Gap" the most persistent failure mode in RAG. It shows up because user queries tend to be short and conversational, while the underlying documents/corpus are often semantically distant in their vocabulary — the classic example: "why is my pipeline slow?" (user's words) vs. "Throughput degradation is..." (the document's actual wording). Even though these mean almost the same thing conceptually, their raw embeddings may not be close enough for a naive semantic search to reliably connect them.
There are two named fixes in the notes:
1. Expand or rephrase the user query. Simply ask an LLM: "generate 3 alternate phrasings..." of the user's question, and search using all of them (or a combination), increasing the odds that at least one phrasing's embedding lands close to the real document.
2. HyDE — Hypothetical Document Embedding. Instead of embedding the user's raw, short question, you first ask an LLM to write a short document that would answer this question — the exact prompt style from the notes: "Write a short document that will answer this question. Be factual. If you don't know the answer, write what the answer would typically look like." You then take that generated hypothetical document — not the original question — and embed it. Because this hypothetical answer is written in a style and vocabulary much closer to how real documents in your corpus are likely written, its embedding tends to land much closer to the real relevant document's embedding than the original short, casual question would have.
Technically, HyDE reframes retrieval as a similarity search between d and d ∈ D — that is, comparing a (hypothetical) document to the real documents in your corpus D, rather than comparing a short question to documents (which is a fundamentally harder, more mismatched comparison).
When does this actually help? The notes are precise about this: HyDE "works when corpus vocab is very different than user query." Good candidates: scientific papers, legal texts, internal jargon-heavy documentation — domains where the "real" documents are written in dense, formal, or highly technical language that a typical user's question would never naturally use.
7.3 Code: HyDE retrieval function
# pip install anthropic sentence-transformers --break-system-packages
import anthropic
from sentence_transformers import SentenceTransformer
import numpy as np
client = anthropic.Anthropic()
embedder = SentenceTransformer("all-MiniLM-L6-v2")
def generate_hypothetical_answer(user_query: str) -> str:
"""
Ask an LLM to write a short, FACTUAL-SOUNDING document that would answer
the user's question. This is the exact prompt shape from the notes:
"Write a short document that will answer this question. Be factual.
If you don't know the answer, write what the answer would typically
look like."
We use this generated text purely to bridge the vocabulary gap — we do
NOT show this hypothetical text to the end user; it's an internal
retrieval-only artifact.
"""
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=300,
messages=[
{
"role": "user",
"content": (
"Write a short document that will answer this question. "
"Be factual. If you don't know the answer, write what the "
f"answer would typically look like.\n\nQuestion: {user_query}"
),
}
],
)
return response.content[0].text
def hyde_retrieve(user_query: str, doc_texts: list[str], doc_ids: list[str], top_k: int = 3):
"""
Full HyDE retrieval: generate a hypothetical answer, embed THAT instead
of the raw query, then find the closest real documents to it.
"""
# Step 1: generate the hypothetical, jargon-matched answer
hypothetical_doc = generate_hypothetical_answer(user_query)
# Step 2: embed the HYPOTHETICAL document, not the raw user query
hyde_embedding = embedder.encode(hypothetical_doc, normalize_embeddings=True)
# Step 3: embed the real corpus documents (in production this is pre-computed
# and stored in a vector DB, not re-computed on every query)
doc_embeddings = embedder.encode(doc_texts, normalize_embeddings=True)
# Step 4: rank real documents by similarity to the hypothetical embedding
similarities = doc_embeddings @ hyde_embedding
top_indices = np.argsort(similarities)[::-1][:top_k]
return [(doc_ids[i], doc_texts[i], float(similarities[i])) for i in top_indices]
# --- Example -----------------------------------------------------------------
corpus_texts = [
"Throughput degradation under sustained load is typically caused by "
"connection pool exhaustion or unindexed database queries.",
"Our refund policy allows cancellations within 30 days of purchase.",
"Latency regressions in distributed systems often trace back to N+1 "
"query patterns or missing caching layers.",
]
corpus_ids = ["perf_doc_1", "refund_policy", "perf_doc_2"]
results = hyde_retrieve("why is my code slow?", corpus_texts, corpus_ids)
for doc_id, text, score in results:
print(f"{score:.3f} {doc_id}: {text[:70]}...")
# Without HyDE, embedding the raw casual query "why is my code slow?" might
# rank the perf docs only weakly, since they never use the word "slow."
# With HyDE, the generated hypothetical answer naturally uses words like
# "throughput," "latency," and "degradation" -- much closer to the real docs.
8. Semantic Caching
8.1 The simple example first
Imagine ten different customers each phrase the exact same underlying question in slightly different words: "How do I integrate libX?", "I want help integrating libX," "How can I get libX working with my app?" A naive cache (the kind that only matches exact strings) would treat these as three totally unrelated questions and pay to compute a fresh, expensive answer for every single one — even though the right answer is identical for all three.
A smarter cache recognizes that all three questions mean the same thing, and serves the same cached, already-computed answer to all of them — saving time and money on the second and third askers, without them ever noticing anything different.
8.2 Technical explanation
The core motivating fact from the notes: "Every call to LLM cost[s] latency and tokens" — i.e., time and money. A semantic cache sits in front of the LLM: it intercepts a query before it reaches the model and, if a semantically similar (not necessarily an exact-text-match) prior query exists in the cache, it returns the cached response instead of paying for a new LLM call.
When this works best, per the notes: static data with no personalization — situations where the "right" answer genuinely doesn't change per-user or over short timeframes. Two named examples:
- Asking an AI assistant about your API/integration documentation ("How do I integrate libX?" / "I want help to integrate libX" — both map to the same content already published on libX's website).
- A customer-support chatbot with a bounded query space — the notes cite that in such settings, roughly 60% of incoming queries can be near-duplicates of previously-asked ones, making caching extremely high-leverage.
Implementation, step by step (from the notes):
- Take the incoming query, and either normalize it (lowercase, strip punctuation, etc.) or store it as-is.
- Index it in a vector database along with its (eventual) response — so both the question's embedding and its answer live together.
- During lookup, find similar queries — embed the new incoming query, and search the cache's vector index for anything above a chosen similarity threshold.
The core tension: cost vs. correctness, governed entirely by what similarity threshold you pick and how you set it. The notes are blunt about the two failure directions:
- Threshold too low → wrong answers. You'll match queries that only look superficially similar but actually mean something different (e.g., "How do I cancel my subscription?" vs. "How do I change my subscription plan?" — related, but the correct action is different), and you'll confidently serve the wrong cached answer.
- Threshold too high → hit rate collapses. You become so strict that almost nothing ever matches, and you rarely benefit from caching at all, defeating the entire purpose.
The reliable way to pick a threshold, per the notes: don't guess — empirically find your "sweet spot" on your own query distribution. As a reasonable starting point, the notes suggest 0.85 cosine similarity, but this must be calibrated, not treated as gospel, because every product's query patterns differ.
8.3 Code: a minimal semantic cache class
# pip install sentence-transformers --break-system-packages
from sentence_transformers import SentenceTransformer
import numpy as np
class SemanticCache:
"""
A minimal in-memory semantic cache: get() looks for a semantically
similar prior query above `similarity_threshold`; set() stores a new
query + response pair for future lookups.
"""
def __init__(self, similarity_threshold: float = 0.85):
self.embedder = SentenceTransformer("all-MiniLM-L6-v2")
self.similarity_threshold = similarity_threshold
self.cached_queries: list[str] = []
self.cached_embeddings: list[np.ndarray] = []
self.cached_responses: list[str] = []
def get(self, query: str) -> str | None:
"""Return a cached response if a semantically similar query exists, else None."""
if not self.cached_queries:
return None
query_embedding = self.embedder.encode(query, normalize_embeddings=True)
similarities = np.array(self.cached_embeddings) @ query_embedding
best_idx = int(np.argmax(similarities))
best_score = float(similarities[best_idx])
if best_score >= self.similarity_threshold:
print(f"[cache HIT] similarity={best_score:.3f} matched: "
f"'{self.cached_queries[best_idx]}'")
return self.cached_responses[best_idx]
print(f"[cache MISS] best similarity was only {best_score:.3f}")
return None
def set(self, query: str, response: str) -> None:
"""Store a new query + response pair in the cache."""
embedding = self.embedder.encode(query, normalize_embeddings=True)
self.cached_queries.append(query)
self.cached_embeddings.append(embedding)
self.cached_responses.append(response)
# --- Demonstration: threshold too low vs. too high --------------------------
cache = SemanticCache(similarity_threshold=0.85)
cache.set("How do I integrate libX?", "To integrate libX, install it via pip and call libx.init()...")
print("\n--- Reasonable threshold (0.85) ---")
cache.get("I want help integrating libX") # should HIT: same intent, different phrasing
cache.get("What's the weather in Tokyo?") # should MISS: unrelated question
print("\n--- Threshold set TOO LOW (0.50): risks wrong answers ---")
loose_cache = SemanticCache(similarity_threshold=0.50)
loose_cache.set("How do I cancel my subscription?", "Go to Settings > Billing > Cancel Plan.")
# This is a RELATED but MEANINGFULLY DIFFERENT question -- with a low
# threshold it may incorrectly match and serve the wrong cached answer.
loose_cache.get("How do I upgrade my subscription plan?")
print("\n--- Threshold set TOO HIGH (0.99): hit rate collapses ---")
strict_cache = SemanticCache(similarity_threshold=0.99)
strict_cache.set("How do I integrate libX?", "To integrate libX, install it via pip...")
# Even a near-identical paraphrase may now fail to match because embeddings
# are almost never PERFECTLY identical between two different sentences.
strict_cache.get("I want help to integrate libX")
The prototype in the notes ("Semantic Caching and Similarity Calibration") describes exactly this experiment at scale: 500 queries → embeddings → index, then sweeping thresholds from 0.75 to 0.95, populating the cache first, then computing hit rate (fraction of queries found in cache) and accuracy/precision. It uses three types of test queries: seed (the original query, e.g. "Explain the ending of movie Arrival"), paraphrase (same intent, different words, e.g. "Please clarify the conclusion of the movie Arrival" — a correct match), and near-miss (looks similar but has a genuinely different intent, e.g. "Who directed the movie Arrival" — a trap, since matching this to the seed's answer would be a false positive / incorrect hit). The accuracy formula from the notes:
accuracy = correct hits (paraphrase matching the seed) / total hits
Below is a small script that runs this exact style of calibration experiment:
def evaluate_threshold(cache: SemanticCache, seed_paraphrase_pairs, near_miss_queries):
"""
seed_paraphrase_pairs: list of (seed_query, paraphrase_query, response) tuples
-- paraphrase SHOULD correctly hit the seed's cached entry.
near_miss_queries: list of queries that LOOK similar but have different intent
-- these SHOULD ideally miss (or if they hit, that's a false positive).
"""
for seed, response in {(s, r) for s, _, r in seed_paraphrase_pairs}:
cache.set(seed, response)
total_hits, correct_hits = 0, 0
for seed, paraphrase, _ in seed_paraphrase_pairs:
result = cache.get(paraphrase)
if result is not None:
total_hits += 1
correct_hits += 1 # paraphrase hitting its own seed = correct
false_positives = 0
for near_miss in near_miss_queries:
result = cache.get(near_miss)
if result is not None:
total_hits += 1
false_positives += 1 # near-miss hitting ANY cached entry = wrong
accuracy = correct_hits / total_hits if total_hits else 0
return {"hit_rate_relevant": correct_hits, "false_positives": false_positives, "accuracy": accuracy}
9. Full System Design: RAG with 10M Documents, Zero Hallucinations
9.1 The simple example first
Imagine building a "help desk" for a huge company with 10 million internal documents. A visitor walks up and asks a question in plain English. Before answering, your help desk clerk must: (1) understand what's really being asked, (2) go search both the exact-keyword index and the meaning-based index of every document, (3) combine and re-sort those results by true relevance, (4) double-check that the results are actually confident and trustworthy (not just "vaguely related"), (5) draft an answer strictly grounded in what was found, and — critically — (6) go back and check every single sentence of that draft answer against the source documents before saying it out loud, refusing to say anything it can't verify. That's the whole system in one sentence: retrieve carefully, answer carefully, and never say something you can't prove.
9.2 Technical explanation: requirements and why they matter
What the system does: Answers natural language queries. The one hard guarantee that shapes the entire design: "the system must not assert any claim that cannot be traced to [a] retrieved passage." This single sentence is why "correctness" (not just relevance, not just speed) drives nearly every architectural decision below. Why this guarantee matters: an internal knowledge system that confidently states wrong facts is often worse than one that admits "I don't know," because wrong confident answers erode trust and can cause real harm (wrong policy info, wrong financial numbers, etc.).
What the system explicitly is NOT: a document management system, an ingestion pipeline product, etc. — the notes flag this so the design stays scoped: we are not building a general-purpose CMS, just the "ask questions about documents that already exist somewhere" layer.
Scale numbers (why they matter):
- 10 million documents, roughly 2KB each → this sets the raw data size baseline (worked out below).
- 200 QPS (queries per second) → this is very read-heavy traffic, which shapes almost every capacity and replication decision (lots of replicas for reads, less concern about write throughput).
- ~1,000 documents updated per day → writes are comparatively rare, which is why the system can tolerate eventual consistency rather than needing instant read-your-writes guarantees.
- We do not need to serve data from a just-updated document immediately — an SLO (service level objective, a plain-English promise about how good the system must be) of roughly ~15 minutes eventual consistency is acceptable. Why this matters: it means the write path doesn't need to be synchronously blocking or ultra-low-latency; it can batch, chunk, and index asynchronously.
9.3 The nine solutioning questions (from the notes)
The notes lay out the exact ordered checklist a systems-design interview or real design doc would walk through:
- How do we store the docs, and where?
- What does the query API look like?
- What does the response look like?
- Which databases, and why?
- Capacity estimation: query and storage.
- Read path and query pseudocode.
- What should be returned?
- What are the failure modes in the read path?
- What does the write path look like (S3 → databases)?
- What are we missing? (a deliberate closing gut-check)
Let's walk through each, grounding every architectural choice in a "why" before the "what."
1 & 2 — Storage location and Query API. Documents live in an S3 bucket (e.g. s3://myrag/doc1.txt), organized per the notes with some inspiration from Git's object storage model (content-addressed, organized by structure like date/git-style hashing prefixes, e.g. folders ab/, ac/, ad/ — a common trick to avoid dumping literally 10 million flat files into one folder, which many filesystems and object stores handle poorly). The query API is a simple HTTP endpoint: GET /query? — the notes sketch this as @app.get('/query?').
3 — Response shape. From the notes, a single respond() call returns a structured object containing:
question: <string>
filters: { date_from, date_to, source_domains, tag, topics, etc. }
answer: <string>
refused: <true/false>
refusal_reason: <string, if refused>
passages: [ {doc_id, passage_id, text, score, highlight_idx}, ... ]
confidence: <number>
query_id: <string> -- used for tracing/debugging a specific request later
Why this shape: it's not enough to return just an answer string. You need the passages (so a user or auditor can verify the claim was actually grounded), a refused boolean plus reason (so "I don't know" is a first-class, explicit outcome, not a vague hedge buried in the answer text), a confidence score (so downstream systems or UI can decide whether to trust/display the answer), and a query_id for tracing — being able to look up exactly what happened for a specific request later, which is essential for debugging a "why did it hallucinate here" incident.
4 — Which databases, and why:
- S3 — the durable source of truth for raw documents.
- Postgres (PG Master + Replica) — stores document and passage metadata (id, text, title, URL, date, tags, is-deleted flag for docs; id, doc_id, chunk_index, text, char_start, char_end, token_count for passages). Why Postgres: this is classic structured, relational metadata — dates, tags, foreign keys between docs and their passages — exactly what a relational database is built for, plus JSONB support for flexible tag/topic fields and strong replication for read-heavy traffic.
- Qdrant (vector database) — stores the dense embeddings for approximate nearest-neighbor (ANN) search. Why a dedicated vector DB: embeddings need specialized indexing structures (like HNSW — Hierarchical Navigable Small World, a plain-English one-liner: "a graph-based index structure that lets you find approximately the closest vectors in high-dimensional space without comparing against every single vector") that general-purpose databases don't do efficiently at this scale.
- ElasticSearch (ES) — holds the BM25 inverted index for the sparse/keyword leg of hybrid search (Section 4), using an English analyzer with stemming and stop-word removal (so "running" and "run" are treated as the same term, and words like "the"/"is" don't dilute matching).
- Redis (cache) — caches passages and query results (this is the semantic-caching layer from Section 8), reducing repeat load on the expensive LLM-generation step.
5 — Capacity estimation, worked step by step (numbers straight from the notes' math):
Query load:
50,000 DAU (daily active users) × 8 queries/day/user × 4.5 (peak multiplier)
────────────────────────────────────────────────────────────────────────── = 200 QPS
86,400 seconds/day
So: 50,000 × 8 = 400,000 total queries/day. 400,000 ÷ 86,400 seconds ≈ 4.6 QPS average. Multiply by a peak multiplier of 4.5 (because traffic isn't spread evenly across 24 hours — there are busy hours) to get roughly 200 QPS at peak. The team then designs for a bit of headroom: 250 QPS target capacity.
Storage — raw documents:
10,000,000 docs × 2KB each = 20GB raw data
This is small enough to be almost trivial to store redundantly — it's the derived data (embeddings, indexes) that actually dominates storage, as shown next.
Storage — vector embeddings:
10 × 10^6 documents × 8 passages/doc (avg) × 1536 dimensions × 4 bytes/float32
= 10×10^6 × 8 × 1536 × 4 bytes
≈ 491 GB (just the raw embedding vectors)
Then add HNSW graph overhead (the extra data structure the ANN index needs to do fast approximate search) — roughly 1.5x — bringing:
Total vector index storage ≈ 491GB × 1.5 ≈ 700GB
This is why the notes conclude: "Hence, we need a distributed vector database" — 700GB (spread across replicas for redundancy and read throughput) is well beyond what a single machine comfortably handles for a low-latency, always-on service. The notes also mention Product Quantization can reduce this by roughly 8x, but "at [the] cost of recall" — i.e., you trade some retrieval accuracy for a much smaller memory footprint, a classic space/accuracy tradeoff.
ElasticSearch (BM25 index) sizing: roughly 20GB × 1.5 ≈ 30GB for the inverted index, run as a 3-node cluster with high availability (HA) plus 3 replicas, ≈ 90GB total, each node with 30GB RAM.
Postgres sizing: the 20GB of metadata plus JSONB overhead and replication comes out to roughly 60-80GB, served with 3 read replicas (since this is read-heavy traffic) with a target replication lag of under 100ms.
Redis: a 3-node cluster with 3 replicas, caching passages and query results — this is explicitly conditioned on the earlier semantic-caching insight: it works best "if no personalization & data is static."
6, 7, 8 — Read path, what's returned, and failure handling are covered together via the code walkthrough in Section 9.4 below (the notes' own respond() pseudocode).
9 — Write path is covered in Section 9.5 below.
9.4 Code: the read path (respond() function), cleaned up and fully commented
The notes' own hand-drawn pseudocode for the read path is the heart of the "zero hallucinations" guarantee. Here it is translated into clean, fully-commented Python:
from dataclasses import dataclass, field
@dataclass
class RespondResult:
answer: str
passages: list
refused: bool
refusal_reason: str | None = None
confidence: float | None = None
def respond(question: str) -> RespondResult:
"""
The full "zero hallucinations" read path:
embed -> hybrid search (dense + sparse) -> RRF fusion -> cross-encoder
rerank -> confidence threshold -> LLM generate -> claim extraction ->
NLI fact-check -> final answer OR refusal.
Every stage is a chance to refuse rather than guess -- that's the whole
point of the "must not assert unsupported claims" guarantee.
"""
# -------------------------------------------------------------------
# Step 1: Embed the question (turn text into a dense vector).
# -------------------------------------------------------------------
query_vector = embed(question)
# -------------------------------------------------------------------
# Step 2: Hybrid search -- run dense (vector) and sparse (BM25) search
# CONCURRENTLY (this is the "parallel tool calls" idea from Section 3
# applied to retrieval itself), each returning their own top-40 shortlist.
# -------------------------------------------------------------------
dense_results = qdrant_ann_search(query_vector, top_k=40)
sparse_results = elasticsearch_bm25(question, top_k=40)
# -------------------------------------------------------------------
# Step 3: Fuse both ranked lists with Reciprocal Rank Fusion (Section 4).
# -------------------------------------------------------------------
candidates = rrf_fusion(dense_results, sparse_results)
# -------------------------------------------------------------------
# Step 4: Rerank the fused candidates with a cross-encoder (Section 6)
# for much higher-precision final ordering. Cross-encoders are too slow
# to run on millions of docs, but perfectly fine on ~40-80 candidates.
# -------------------------------------------------------------------
ranked = cross_encoder_rerank(question, candidates, top_k=40)
# -------------------------------------------------------------------
# Step 5: CONFIDENCE GATE #1 -- if even the BEST passage scores too low,
# we don't have real evidence to answer with. Refuse rather than guess.
# -------------------------------------------------------------------
if not any(passage.score > 0.4 for passage in ranked):
return refused_response("low_retrieval_confidence")
# -------------------------------------------------------------------
# Step 6: Take only the top 5 passages -- this keeps the LLM's context
# small, focused, and less prone to the "lost in the middle" problem
# (Section 6) that hurts long, noisy contexts.
# -------------------------------------------------------------------
top_passages = ranked[:5]
prompt = build_prompt(question, top_passages)
# -------------------------------------------------------------------
# Step 7: Ask the LLM to generate an answer, GROUNDED ONLY in top_passages.
# temperature=0 for maximum determinism/factuality (no creative sampling).
# -------------------------------------------------------------------
raw_answer = llm_generate(prompt, temperature=0)
# -------------------------------------------------------------------
# Step 8: CONFIDENCE GATE #2 -- the prompt instructs the LLM to say
# "INSUFFICIENT_CONTEXT" if the retrieved passages don't actually answer
# the question. If it says so, we honor that and refuse cleanly.
# -------------------------------------------------------------------
if raw_answer.startswith("INSUFFICIENT_CONTEXT"):
return refused_response("no_relevant_passages")
# -------------------------------------------------------------------
# Step 9: CLAIM EXTRACTION + FACT-CHECKING -- this is the actual
# "zero hallucinations" enforcement mechanism. Break the raw answer
# into small, individually-checkable "atomic claims" (e.g. one claim
# per sentence or per fact), then verify EACH claim is actually
# entailed (logically supported) by the retrieved passages using an
# NLI (Natural Language Inference) model -- a plain-English one-liner:
# "a model that decides whether a piece of text logically follows from,
# contradicts, or is unrelated to another piece of text."
# If ANY claim can't be verified, we refuse rather than let an
# unsupported claim slip through to the user.
# -------------------------------------------------------------------
claims = extract_atomic_claims(raw_answer)
for claim in claims:
if not nli_entailed(claim, top_passages):
return refused_response("unverified_claim")
# -------------------------------------------------------------------
# All gates passed: the answer is grounded, confident, and every claim
# has been individually verified against the retrieved evidence.
# -------------------------------------------------------------------
return RespondResult(answer=raw_answer, passages=top_passages, refused=False)
def refused_response(reason: str) -> RespondResult:
"""A first-class, explicit refusal -- NOT a vague hedge buried in text."""
return RespondResult(answer="", passages=[], refused=True, refusal_reason=reason)
# NOTE: embed(), qdrant_ann_search(), elasticsearch_bm25(), rrf_fusion(),
# cross_encoder_rerank(), build_prompt(), llm_generate(), extract_atomic_claims(),
# and nli_entailed() are each real functions you'd implement using the
# techniques from Sections 4, 6, and standard NLI models (e.g. a model
# fine-tuned on MNLI/SNLI for entailment classification) -- they are stubbed
# here to keep the read-path LOGIC the star of this example.
9.5 Code: a simple hierarchical chunking function (sentence-boundary + token overlap)
The write path (S3 → chunk ingestor → hierarchical chunking → embeddings → storage) needs a chunker. The notes specify sentence-level boundaries (using spaCy) plus 32-token overlap between consecutive chunks (overlap ensures a sentence that would otherwise get awkwardly split across two chunks still has enough surrounding context in at least one of them).
# pip install spacy --break-system-packages
# python -m spacy download en_core_web_sm
import spacy
nlp = spacy.load("en_core_web_sm")
def chunk_document(text: str, max_tokens_per_chunk: int = 256, overlap_tokens: int = 32) -> list[dict]:
"""
Hierarchical, sentence-boundary-respecting chunking with token overlap.
Why sentence boundaries: splitting mid-sentence produces chunks that are
semantically broken and embed poorly. Why overlap: a fact stated right at
a chunk boundary shouldn't lose its surrounding context in EVERY chunk
that references it.
"""
doc = nlp(text)
sentences = [sent.text.strip() for sent in doc.sents]
chunks = []
current_chunk_sentences: list[str] = []
current_token_count = 0
char_cursor = 0
def rough_token_count(s: str) -> int:
# A simple approximation (real systems use a proper tokenizer, e.g.
# tiktoken, matching whatever embedding model will consume the chunk).
return len(s.split())
for sentence in sentences:
sentence_tokens = rough_token_count(sentence)
# If adding this sentence would overflow the chunk, close the
# current chunk out and start a new one, carrying over the last
# `overlap_tokens` worth of sentences for continuity.
if current_token_count + sentence_tokens > max_tokens_per_chunk and current_chunk_sentences:
chunk_text = " ".join(current_chunk_sentences)
chunks.append({
"text": chunk_text,
"token_count": current_token_count,
"char_start": char_cursor,
"char_end": char_cursor + len(chunk_text),
})
char_cursor += len(chunk_text)
# Build the overlap: keep trailing sentences until we've got
# roughly `overlap_tokens` worth, to seed the next chunk.
overlap_sentences, overlap_count = [], 0
for sent in reversed(current_chunk_sentences):
overlap_sentences.insert(0, sent)
overlap_count += rough_token_count(sent)
if overlap_count >= overlap_tokens:
break
current_chunk_sentences = overlap_sentences
current_token_count = overlap_count
current_chunk_sentences.append(sentence)
current_token_count += sentence_tokens
# Flush the final chunk
if current_chunk_sentences:
chunk_text = " ".join(current_chunk_sentences)
chunks.append({
"text": chunk_text,
"token_count": current_token_count,
"char_start": char_cursor,
"char_end": char_cursor + len(chunk_text),
})
return chunks
# --- Example ---------------------------------------------------------------
sample_doc = (
"Throughput degradation under sustained load is a common issue. "
"It is typically caused by connection pool exhaustion. "
"Another frequent cause is unindexed database queries. "
"Monitoring dashboards should track p50, p95, and p99 latency. "
"Alerts should fire when p99 crosses a defined SLO threshold."
)
for i, chunk in enumerate(chunk_document(sample_doc, max_tokens_per_chunk=15, overlap_tokens=5)):
print(f"Chunk {i}: {chunk['text']}")
Each resulting chunk (a "passage" in the notes' schema) would then get an embedding computed (using one of the models the notes list — microsoft/hamier, Qwen3 (8B), BGE-M3, or an OpenAI/OSS embedding API), stored in Qdrant (the vector), Postgres (the metadata: doc_id, chunk_index, char_start, char_end, token_count), and ElasticSearch (the BM25 inverted index).
9.6 The write path, explained in words
Flow: S3 (raw doc storage) → Chunk Ingestor → Hierarchical Chunking → Embeddings → Storage (Postgres, Qdrant, ElasticSearch, Redis).
Step by step: a document lands in S3. A chunk ingestor process picks it up (typically via an event notification or polling), performs three sub-steps the notes list explicitly (list files, download, read), then runs hierarchical chunking (Section 9.5's sentence-boundary + overlap logic) to split it into passages. Each passage is embedded, and the results are fanned out to storage: Postgres gets the passage/doc metadata, Qdrant gets the embedding vectors, and ElasticSearch gets the text for its BM25 inverted index. This asynchronous pipeline is exactly why the system can tolerate the ~15-minute eventual consistency SLO mentioned earlier — the write path doesn't need to be instantaneous, it needs to be reliable and complete within a bounded, acceptable delay.
The notes also flag a timebox mechanism on the read side with two named fallback behaviors — cascade and fallback — meaning if some part of the read path takes too long (e.g., the cross-encoder rerank step or the LLM generation call, described as "the long pole" of end-to-end latency), the system has a time budget after which it cascades to a simpler/cheaper strategy or falls back to a safe default (like a refusal) rather than hanging indefinitely. This connects directly back to the query latency numbers given in the notes: p50 = 1.5 sec, p99 = 5 sec, with the LLM inference call explicitly called out as the long pole (the slowest, bottleneck stage) — largely because the very first token the user sees (their Time To First Token, covered next in the Appendix) is gated on that call finishing its reasoning.
10. Appendix A: Streaming Tool Calls
10.1 The simple example first
Think about the difference between getting a text message that appears all at once after a long pause, versus watching someone type it live, word by word, right in front of you. Even though the total message might take the same amount of time to fully arrive, watching it appear progressively feels far less frustrating — you get immediate feedback that something is happening, instead of staring at a blank screen wondering if anything is working at all.
That's the entire motivation for streaming: even if the total response time is identical, showing partial output as it's generated dramatically improves perceived responsiveness.
10.2 Technical explanation
The most important metric here, per the notes: TTFT — Time To First Token (a plain-English one-liner: "how long the user waits before they see ANY output at all, even if the full response isn't done yet"). If a tool call takes real time to execute (a slow database query, a slow external API), and the user has to wait through the entire tool call before seeing anything, that's flagged directly in the notes as bad UX — "like everything else," slow, silent waiting erodes trust in the product.
The tricky part specific to streaming tool calls: "what if tool input arrives as partial JSON fragments?" When a model streams its response token by token, and part of that response is a tool_use block (e.g., building up {"city": "Tok ... yo"} character by character), you cannot execute the tool with a half-formed, syntactically-invalid JSON blob. The notes' answer: "wait until the entire input is available" — i.e., keep buffering (accumulating) the partial JSON text as it streams in, and only attempt to parse/execute the tool call once the model signals that this particular content block is fully complete.
The notes include a cautionary, very concrete illustration of why this matters: imagine the model is streaming a dangerous tool call like "delete that record..." If your system tried to act on a half-baked, incomplete tool input (say, a delete call where the id argument hasn't fully arrived yet), it could either error out unpredictably or — worse — "no input could be fired" i.e., act on a malformed/empty argument. Half-baked input / tool calls with no input could be fired is exactly the failure mode the buffering strategy exists to prevent.
10.3 Code: the notes' Anthropic SDK streaming example, cleaned up and explained line by line
import anthropic
import json
client = anthropic.Anthropic()
# The tool definition is identical in shape to Section 1 -- streaming
# doesn't change WHAT a tool looks like, only HOW the response arrives.
tools = [{
"name": "get_weather",
"description": "Get current weather for a location",
"input_schema": {
"type": "object",
"properties": {
"location": {"type": "string"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
}]
def execute_tool(tool_name: str, tool_input: dict) -> dict:
"""Stand-in for a real tool implementation (see Section 1 for a fuller mock)."""
if tool_name == "get_weather":
return {"location": tool_input["location"], "temp": 24, "condition": "sunny"}
return {"error": "unknown tool"}
with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=1024,
tools=tools,
messages=[{"role": "user", "content": "What is the weather in Tokyo?"}],
) as stream:
# These two variables ACCUMULATE state across streamed events -- this IS
# the "buffering" the notes insist on. We never act on partial state.
current_tool_input = ""
current_tool_name = None
for event in stream:
# -------------------------------------------------------------
# "content_block_start": a NEW block (either plain text or a
# tool_use block) has begun streaming. If it's a tool_use block,
# note its NAME immediately, but its INPUT starts empty --
# arguments arrive incrementally via later delta events.
# -------------------------------------------------------------
if event.type == "content_block_start":
if event.content_block.type == "tool_use":
current_tool_name = event.content_block.name
current_tool_input = "" # reset the buffer for this new block
# -------------------------------------------------------------
# "content_block_delta": an incremental CHUNK of the current
# block has arrived. For tool_use blocks, this is a fragment of
# partial JSON (event.delta.type == "input_json_delta") -- we
# must ACCUMULATE it, not parse it yet, since it's incomplete.
# For plain text blocks, we can print immediately since partial
# text is still readable (this is what gives the "word by word"
# streaming feel for normal chat responses).
# -------------------------------------------------------------
elif event.type == "content_block_delta":
if event.delta.type == "input_json_delta":
current_tool_input += event.delta.partial_json # <- Accumulate
elif event.delta.type == "text_delta":
print(event.delta.text, end="", flush=True)
# -------------------------------------------------------------
# "content_block_stop": the CURRENT block (text or tool_use) has
# finished streaming completely. If it was a tool_use block, the
# buffered JSON string is now guaranteed complete and valid --
# THIS is the safe moment to parse it and actually execute the
# tool, never before.
# -------------------------------------------------------------
elif event.type == "content_block_stop":
if current_tool_name:
parsed_input = json.loads(current_tool_input) # <- input complete
result = execute_tool(current_tool_name, parsed_input) # <- execute
print(f"\n[Executed {current_tool_name} -> {result}]")
current_tool_name = None # reset for the next potential tool block
The three event types to remember: content_block_start (a new block began — note its type/name), content_block_delta (a fragment arrived — accumulate if it's tool JSON, print immediately if it's plain text), and content_block_stop (the block is fully complete — only now is it safe to parse and execute a tool call).
11. Appendix B: Evaluating RAG with RAGAS
11.1 The simple example first
Context Precision, simple example: think of a search-results page with 5 blue links. Some of them are genuinely useful and directly relevant to what you searched for (imagine green dots next to them); others are irrelevant filler or noise (red dots). A good search engine doesn't just include some relevant results somewhere on the page — it puts them near the top. A search engine that buries the one truly relevant result at position #5, behind four irrelevant ones, is worse than one that puts it at position #1, even if both technically "found" it somewhere on the page.
Faithfulness, simple example: imagine grading a student's essay by checking, sentence by sentence, whether each individual claim they made is actually backed up by the assigned textbook. If the student writes five sentences and four of them are directly supported by the textbook but the fifth one is something they simply made up (that isn't in the textbook at all), you wouldn't give them full credit — you'd score the essay based on the fraction of sentences that are actually grounded in the source material. That's exactly what "faithfulness" measures for an LLM's generated answer.
11.2 Technical explanation
The motivation, per the notes: naive RAG evaluation is just qualitative — "does it seem right?" — and manual feedback loops are slow. That doesn't scale, and it's subjective. RAGAS (a plain-English one-liner: "a framework and library for evaluating RAG pipelines using automated, LLM-assisted metrics instead of manual human judgment") fixes two specific pain points named in the notes: no need for human-annotated ground truth, and it lets you verify that changes/optimizations to your pipeline don't quietly downgrade quality (a regression-testing mindset applied to RAG quality, not just code correctness).
Context Precision — measures the retriever's (or reranker's) ability to rank relevant chunks higher, and it directly penalizes systems that bury relevant content below noise — this is precisely the "lost in the middle" problem flagged back in Section 6. The method: use an LLM to label each retrieved chunk as RELEVANT or NOT relevant, and then, for every position k in the ranked list, compute precision@k — the fraction of the top k results that are relevant.
The notes' worked example, using a 5-result list marked (green = relevant, red = not relevant): 🟢 🔴 🟢 🔴 🔴
- p@1 = 1 — the top 1 result: 1 out of 1 is relevant → 1/1 = 1.
-
p@2 = 0 — wait, this looks odd at first: the notes explicitly write
p@2 = 0. Reading the notes' own math carefully: precision@k is only counted (included in the final average) at positions where the k-th result itself is relevant — i.e., you only "credit" a position if the newly added result at that rank is itself a hit, otherwise that position contributes 0 to the running average (this is the standard "average precision" style computation, where precision at non-relevant ranks doesn't get counted toward the final score). - p@3 = 2/3 — of the top 3 results, 2 are relevant (the 1st and 3rd) → 2/3 ≈ 0.67.
- p@4 = 0, p@5 = 0 — same logic: the 4th and 5th results are themselves not relevant, so they don't contribute.
Final Context Precision formula (from the notes' own arithmetic):
Context Precision = (p@1 + p@3) / (# relevant chunks)
= (1 + 0.67) / 2
= 1.67 / 2
= 0.835
In words: you sum up the precision@k values at each rank where a relevant chunk appears, then divide by the total number of relevant chunks in the list (here, 2 relevant chunks out of 5 total). This rewards systems that put their relevant chunks early — a relevant chunk found at rank 1 contributes a full 1.0, while one found deeper in the list contributes a smaller fraction, exactly reflecting the "don't bury the good stuff" intuition from the simple example.
Faithfulness Score — measures factual consistency of the generated answer, and is explicitly called a "measure of hallucination" in the notes. Method: break the answer into atomic claims (using an LLM — the same "claim extraction" step from the read-path pseudocode in Section 9.4), then for each claim, score it 1 if the retrieved context supports the claim, or 0 if the context does not support the claim.
Faithfulness formula:
faithfulness = (# claims supported) / (# claims)
This is the exact same underlying idea as the nli_entailed() check baked into the "zero hallucinations" respond() function from Section 9 — faithfulness is essentially the evaluation-time metric version of the same runtime guardrail.
11.3 Code: Context Precision and Faithfulness, computed from scratch
# =============================================================================
# CONTEXT PRECISION -- from a list of relevant/irrelevant flags
# (the "green dot / red dot" example from the notes).
# =============================================================================
def context_precision(relevance_flags: list[bool]) -> float:
"""
relevance_flags: e.g. [True, False, True, False, False]
-- True = relevant chunk, False = not relevant,
in the ORDER they were retrieved/ranked (best first).
Returns the RAGAS-style Context Precision score.
"""
total_relevant = sum(relevance_flags)
if total_relevant == 0:
return 0.0 # no relevant chunks at all -- precision is undefined/zero
running_relevant_count = 0
precision_sum = 0.0
for k, is_relevant in enumerate(relevance_flags, start=1):
if is_relevant:
running_relevant_count += 1
# precision@k = (# relevant chunks in top k) / k
precision_at_k = running_relevant_count / k
precision_sum += precision_at_k
# if NOT relevant, this rank contributes 0 and is skipped, matching
# the notes' worked example where p@2, p@4, p@5 = 0
return precision_sum / total_relevant
# --- Example matching the notes exactly: 🟢 🔴 🟢 🔴 🔴 ----------------------
flags = [True, False, True, False, False]
score = context_precision(flags)
print(f"Context Precision: {score:.3f}") # -> should print 0.835
# =============================================================================
# FAITHFULNESS -- given a list of claims and whether each is supported.
# =============================================================================
def faithfulness_score(claim_support_labels: list[bool]) -> float:
"""
claim_support_labels: e.g. [True, True, True, False]
-- True = context supports this atomic claim,
False = context does NOT support it (hallucinated).
"""
if not claim_support_labels:
return 1.0 # no claims made -- vacuously faithful (nothing to check)
supported = sum(claim_support_labels)
total = len(claim_support_labels)
return supported / total
# --- Example: an answer with 4 atomic claims, 1 of which is unsupported ----
claims_supported = [True, True, True, False]
faith_score = faithfulness_score(claims_supported)
print(f"Faithfulness: {faith_score:.3f}") # -> 0.75
11.4 Code: using the real RAGAS library
# pip install ragas datasets --break-system-packages
from ragas import evaluate
from ragas.metrics import context_precision, faithfulness
from datasets import Dataset
# RAGAS expects a dataset shaped like this: one row per question, with the
# retrieved contexts and the generated answer already filled in.
eval_dataset = Dataset.from_dict({
"question": ["Why is my API returning a 500 error under high load?"],
"answer": ["500 errors under high load are typically caused by connection "
"pool exhaustion or unhandled exceptions from resource contention."],
"contexts": [[
"Throughput degradation under sustained load can trigger unhandled "
"exceptions, which surface to callers as HTTP 500 errors.",
"Connection pool exhaustion is a common cause of request failures "
"under high concurrency.",
]],
# ground_truth is optional for some metrics but required for others (e.g. recall)
"ground_truth": ["500 errors under load are usually caused by connection "
"pool exhaustion or unhandled exceptions."],
})
results = evaluate(eval_dataset, metrics=[context_precision, faithfulness])
print(results)
# This runs the SAME conceptual computation as the from-scratch functions
# above, but uses an LLM internally to (a) judge chunk relevance for
# context_precision, and (b) extract + verify atomic claims for faithfulness.
12. Glossary
- Tool use / function calling — letting an LLM request that your application run a specific, predefined function on its behalf, using arguments the model constructs.
- Tool schema — the JSON description (name, description, parameters) that tells the model what a tool does and how to call it; the model never sees or runs the actual code.
- Enum — a parameter constrained to a fixed list of valid values, so a model can't invent arbitrary strings.
- RBAC (Role-Based Access Control) — restricting what a user can see or do based on their assigned role (e.g., department, seniority).
- Dense retrieval — semantic search using embeddings; finds documents with similar meaning, even without shared words.
- Sparse retrieval / BM25 — traditional keyword-based search; scores documents by exact term overlap, weighted by term rarity/frequency.
- Hybrid search — running dense and sparse retrieval together and merging the results, to cover both of their blind spots.
- RRF (Reciprocal Rank Fusion) — a method for merging multiple ranked lists into one, based only on each item's rank position in each list, not its raw score.
- Metadata filtering — applying hard constraints (date, department, tags) before or after a search, on top of relevance ranking.
- Pre-filtering — applying metadata constraints inside the search query itself, so the database never considers disallowed documents.
- Post-filtering — running an unfiltered search first, then discarding invalid results afterward in application code.
- Bi-encoder — embeds the query and each document independently, then compares them via vector similarity; fast and scalable but can't compare them in context.
- Cross-encoder — feeds the query and a single document together as one input, producing one precise relevance score; accurate but slow and non-scalable.
- "Lost in the middle" problem — the tendency for information in the middle of a long context to receive less effective attention than information at the start or end.
- HyDE (Hypothetical Document Embedding) — generating a fake, ideal-sounding answer to a query first, then using that text's embedding for retrieval, to bridge a vocabulary gap between casual queries and formal documents.
- Semantic caching — caching LLM responses keyed by meaning (embedding similarity) rather than exact text match, so paraphrased questions can still hit the cache.
- NLI (Natural Language Inference) — a model task/technique for deciding whether one piece of text logically follows from (entails), contradicts, or is unrelated to another.
- HNSW (Hierarchical Navigable Small World) — a graph-based index structure used by vector databases to find approximately-nearest vectors quickly without comparing against every single one.
- TTFT (Time To First Token) — how long a user waits before seeing any output at all, even if the full response is still being generated.
- RAGAS — a framework/library for evaluating RAG pipelines with automated, LLM-assisted metrics instead of manual human review.
- Context Precision — a RAGAS metric measuring whether relevant retrieved chunks are ranked near the top rather than buried lower down.
- Faithfulness — a RAGAS metric measuring what fraction of an LLM's generated claims are actually supported by the retrieved context (a direct measure of hallucination).
13. "Why This Matters in Production" — One-Liner Per Section
- Tool Use Fundamentals — without a disciplined intercept/execute/return loop, you either let a model hallucinate data it can't actually access, or you build a system that silently breaks the moment a tool call needs real-world side effects.
- Tool Schema Design — a vague schema is invisible in your own testing and only breaks in production, when real users ask messy questions your clean test prompts never covered.
- Parallel Tool Calls and Failure Handling — sequential tool calls make your product feel painfully slow at scale, and having no explicit failure strategy means one flaky dependency can silently corrupt or crash every response.
- Hybrid Search — relying on dense-only or sparse-only retrieval guarantees you'll systematically miss either exact-term queries or natural-language queries, and users will experience this as "the search is broken" without knowing why.
- Metadata Filtering — get pre- vs post-filtering wrong and you either waste retrieval budget or, worse, leak data across security boundaries like RBAC in enterprise settings.
- Reranking (Cross-Encoders vs Bi-Encoders) — skipping reranking means your final answer is only as good as noisy first-pass retrieval; running cross-encoders on your whole corpus means your product times out.
- Query Rewriting with HyDE — without bridging the query-document vocabulary gap, your retrieval will keep failing on exactly the domains (legal, scientific, internal jargon) where correctness matters most.
- Semantic Caching — every uncached LLM call costs real latency and real money; miscalibrating the threshold either serves wrong answers or barely saves anything at all.
- Full RAG System Design — at 10M-document scale, "correctness" has to be engineered in at every stage (retrieval, reranking, generation, fact-checking), because a single loose stage can let an unsupported claim reach the user.
- Streaming Tool Calls — users abandon or distrust products that feel frozen; but streaming without proper buffering risks executing dangerous operations (like deletes) on incomplete input.
- Evaluating RAG with RAGAS — without automated, repeatable metrics, you can't tell whether a "quality improvement" you shipped actually helped or quietly made hallucinations worse.
Top comments (0)